core Package

core Package

This module contains all of the core components of fastr. It has the classes to create networks and work with them.

cardinality Module

class fastr.core.cardinality.AnyCardinalitySpec(parent)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

validate(payload, cardinality)[source]

Validate cardinality given a payload and cardinality

Parameters
  • payload (dict) – Payload of the corresponding job

  • cardinality (int) – Cardinality to validate

Return type

bool

Returns

Validity of the cardinality given the spec and payload

class fastr.core.cardinality.AsCardinalitySpec(parent, target)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, target)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters

key – Key for which the cardinality is calculated

Return type

Optional[int]

Returns

calculated cardinality

calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type

Optional[int]

calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters

node – Node for which the cardinality is calculated

Return type

Optional[int]

Returns

calculated cardinality

get_ordereddict_cardinality()[source]
get_target()[source]
Return type

str

property node
property predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

class fastr.core.cardinality.CardinalitySpec(parent)[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.core.cardinality', '__init__': <function CardinalitySpec.__init__>, '__str__': <function CardinalitySpec.__str__>, '__repr__': <function CardinalitySpec.__repr__>, '__eq__': <function CardinalitySpec.__eq__>, '__ne__': <function CardinalitySpec.__ne__>, 'predefined': <property object>, 'validate': <function CardinalitySpec.validate>, '_validate': <function CardinalitySpec._validate>, 'calculate_planning_cardinality': <function CardinalitySpec.calculate_planning_cardinality>, 'calculate_execution_cardinality': <function CardinalitySpec.calculate_execution_cardinality>, 'calculate_job_cardinality': <function CardinalitySpec.calculate_job_cardinality>, '__dict__': <attribute '__dict__' of 'CardinalitySpec' objects>, '__weakref__': <attribute '__weakref__' of 'CardinalitySpec' objects>, '__doc__': None, '__hash__': None})
abstract __eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__ne__(other)[source]

Return self!=value.

__repr__()[source]

Console representation of the cardinality spec

Return type

str

abstract __str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

__weakref__

list of weak references to the object (if defined)

calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters

key – Key for which the cardinality is calculated

Return type

Optional[int]

Returns

calculated cardinality

calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type

Optional[int]

calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters

node – Node for which the cardinality is calculated

Return type

Optional[int]

Returns

calculated cardinality

property predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

validate(payload, cardinality, planning=True)[source]

Validate cardinality given a payload and cardinality

Parameters
  • payload (Optional[dict]) – Payload of the corresponding job

  • cardinality (int) – Cardinality to validate

Return type

bool

Returns

Validity of the cardinality given the spec and payload

class fastr.core.cardinality.ChoiceCardinalitySpec(parent, options)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, options)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

class fastr.core.cardinality.IntCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

calculate_execution_cardinality(node)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters

key – Key for which the cardinality is calculated

Return type

int

Returns

calculated cardinality

calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type

Optional[int]

calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters

node – Node for which the cardinality is calculated

Return type

int

Returns

calculated cardinality

property predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

class fastr.core.cardinality.MaxCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

class fastr.core.cardinality.MinCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

class fastr.core.cardinality.RangeCardinalitySpec(parent, min, max)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, min, max)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

class fastr.core.cardinality.ValueCardinalitySpec(parent, target)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, target)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type

str

calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters

key – Key for which the cardinality is calculated

Return type

Optional[int]

Returns

calculated cardinality

calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type

Optional[int]

property node
fastr.core.cardinality.create_cardinality(desc, parent)[source]

Create simplified description of the cardinality. This changes the string representation to a tuple that is easier to check at a later time.

Parameters

desc (str) – the string version of the cardinality

Parent

the parent input or output to which this cardinality spec belongs

Return type

CardinalitySpec

Returns

the simplified cardinality description

Raises

FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

The translation works with the following table:

cardinality string

cardinality spec

description

"*", any

``(‘any’,)

Any cardinality is allowed

"N"

('int', N)

A cardinality of N is required

"N-M"

('range', N, M)

A cardinality between N and M is required

"*-M"

('max', M)

A cardinality of maximal M is required

"N-*"

('min', N)

A cardinality of minimal N is required

"[M,N,...,O,P]"

('choice', [M,N,...,O,P])

The cardinality should one of the given options

"as:input_id"

('as', 'input_id')

The cardinality should match the cardinality of the given Input

"val:input_id"

('val', 'input_id')

The cardinliaty should match the value of the given Input

Note

The maximumu, minimum and range are inclusive

dimension Module

class fastr.core.dimension.Dimension(name, size)[source]

Bases: object

A class representing a dimension. It contains the name and size of the dimension.

__eq__(other)[source]

Dimension is the same if the name and size are the same

__hash__ = None
__init__(name, size)[source]

The constructor for the dimension.

Parameters
  • name (str) – Name of the dimension

  • size (int or Symbol) – Size fo the dimension

__module__ = 'fastr.core.dimension'
__ne__(other)[source]

The not equal test is simply the inverse of the equal test

__repr__()[source]

String representation of a Dimension

__slots__ = ('_name', '_size')
copy()[source]

Get a copy object of a Dimension

property name
property size
update_size(value)[source]
class fastr.core.dimension.ForwardsDimensions[source]

Bases: fastr.core.dimension.HasDimensions

Class of objects that have dimensions not because they contain data with dimensions but forward them (optionally with changes via combine_dimensions)

__abstractmethods__ = frozenset({'combine_dimensions', 'source'})
__module__ = 'fastr.core.dimension'
abstract combine_dimensions(dimensions)[source]

Method to combine/manipulate the dimensions

Parameters

dimensions – the input dimensions from the source

Returns

dimensions manipulated for this object

Return type

tuple of dimensions

property dimensions

The dimensions of the object based on the forwarding

abstract property source

The source object from which the dimensions are forwarded

Returns

the object from which the dimensions are forwarded

Return type

HasDimensions

class fastr.core.dimension.HasDimensions[source]

Bases: object

A Mixin class for any object that has a notion of dimensions and size. It uses the dimension property to expose the dimension name and size.

__abstractmethods__ = frozenset({'dimensions'})
__dict__ = mappingproxy({'__module__': 'fastr.core.dimension', '__doc__': '\n A Mixin class for any object that has a notion of dimensions and size. It\n uses the dimension property to expose the dimension name and size.\n ', 'dimensions': <abc.abstractproperty object>, 'dimnames': <property object>, 'size': <property object>, 'ndims': <property object>, '__dict__': <attribute '__dict__' of 'HasDimensions' objects>, '__weakref__': <attribute '__weakref__' of 'HasDimensions' objects>, '__abstractmethods__': frozenset({'dimensions'}), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 58})
__module__ = 'fastr.core.dimension'
__weakref__

list of weak references to the object (if defined)

abstract property dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns

dimensions

Return type

tuple

property dimnames

A tuple containing the dimension names of this object. All items of the tuple are of type str.

property ndims

The number of dimensions in this object

property size

A tuple containing the size of this object. All items of the tuple are of type int or Symbol.

interface Module

A module that describes the interface of a Tool. It specifies how a set of

input values will be translated to commands to be executed. This creates a generic interface to different ways of executing underlying software.

class fastr.core.interface.InputSpec(id_, cardinality, datatype, required=False, description='', default=None, hidden=False)[source]

Bases: fastr.core.interface.InputSpec

Class containing the information about an Input Specification, this is essentially a data class (but

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Input Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, 'asdict': <function InputSpec.asdict>, '__dict__': <attribute '__dict__' of 'InputSpec' objects>})
__module__ = 'fastr.core.interface'
static __new__(cls, id_, cardinality, datatype, required=False, description='', default=None, hidden=False)[source]

Create new instance of InputSpec(id, cardinality, datatype, required, description, default, hidden)

asdict()[source]
fastr.core.interface.InputSpecBase

alias of fastr.core.interface.InputSpec

class fastr.core.interface.Interface[source]

Bases: fastr.abc.baseplugin.Plugin, fastr.abc.serializable.Serializable

Abstract base class of all Interfaces. Defines the minimal requirements for all Interface implementations.

__abstractmethods__ = frozenset({'__getstate__', '__setstate__', 'execute', 'expanding', 'inputs', 'outputs'})
abstract __getstate__()[source]

Retrieve the state of the Interface

Returns

the state of the object

Rtype dict

__module__ = 'fastr.core.interface'
abstract __setstate__(state)[source]

Set the state of the Interface

abstract execute(target, payload)[source]

Execute the interface given the a target and payload. The payload should have the form:

{
  'input': {
    'input_id_a': (value, value),
    'input_id_b': (value, value)
  },
  'output': {
    'output_id_a': (value, value),
    'output_id_b': (value, value)
  }
}
Parameters
  • target – the target to call

  • payload – the payload to use

Returns

the result of the execution

Return type

(tuple of) InterfaceResult

abstract property expanding

Indicates whether or not this Interface will result in multiple samples per run. If the flow is unaffected, this will be zero, if it is nonzero it means that number of dimension will be added to the sample array.

abstract property inputs

OrderedDict of Inputs connected to the Interface. The format should be {input_id: InputSpec}.

abstract property outputs

OrderedDict of Output connected to the Interface. The format should be {output_id: OutputSpec}.

classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.core.interface.InterfaceResult(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]

Bases: object

The class in which Interfaces should wrap their results to be picked up by fastr

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n The class in which Interfaces should wrap their results to be picked up by fastr\n ', '__init__': <function InterfaceResult.__init__>, '__dict__': <attribute '__dict__' of 'InterfaceResult' objects>, '__weakref__': <attribute '__weakref__' of 'InterfaceResult' objects>})
__init__(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.interface'
__weakref__

list of weak references to the object (if defined)

class fastr.core.interface.OutputSpec(id_, cardinality, datatype, automatic=True, required=False, description='', hidden=False)[source]

Bases: fastr.core.interface.OutputSpec

Class containing the information about an Output Specification, this is essentially a data class (but

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Output Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, 'asdict': <function OutputSpec.asdict>, '__dict__': <attribute '__dict__' of 'OutputSpec' objects>})
__module__ = 'fastr.core.interface'
static __new__(cls, id_, cardinality, datatype, automatic=True, required=False, description='', hidden=False)[source]

Create new instance of OutputSpec(id, cardinality, datatype, automatic, required, description, hidden)

asdict()[source]
fastr.core.interface.OutputSpecBase

alias of fastr.core.interface.OutputSpec

ioplugin Module

This module contains the manager class for IOPlugins and the base class for all IOPlugins

class fastr.core.ioplugin.IOPlugin[source]

Bases: fastr.abc.baseplugin.Plugin

IOPlugins are used for data import and export for the sources and sinks. The main use of the IOPlugins is during execution (see Execution). The IOPlugins can be accessed via fastr.ioplugins, but generally there should be no need for direct interaction with these objects. The use of is mainly via the URL used to specify source and sink data.

__abstractmethods__ = frozenset({'scheme'})
__init__()[source]

Initialization for the IOPlugin

Returns

newly created IOPlugin

__module__ = 'fastr.core.ioplugin'
cleanup()[source]

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters

url (str) – url to expand

Returns

the resulting url(s), a tuple if multiple, otherwise a str

Return type

str or tuple of str

fetch_url(inurl, outfile)[source]

(abstract) Fetch a file from an external data source.

Parameters
  • inurl – url to the item in the data store

  • outpath – path where to store the fetch data locally

fetch_value(inurl)[source]

(abstract) Fetch a value from an external data source.

Parameters

inurl – the url of the value to retrieve

Returns

the fetched value

static isurl(string)[source]

Test if given string is an url.

Parameters

string (str) – string to test

Returns

True if the string is an url, False otherwise

Return type

bool

path_to_url(path, mountpoint=None)[source]

(abstract) Construct an url from a given mount point and a relative path to the mount point.

Parameters
  • path (str) – the path to determine the url for

  • mountpoint (str or None) – the mount point to use, will be automatically detected if None is given

Returns

url matching the path

Return type

str

static print_result(result)[source]

Print the result of the IOPlugin to stdout to be picked up by the tool

Parameters

result – value to print as a result

Returns

None

pull_source_data(inurl, outdir, sample_id, datatype=None)[source]

Transfer the source data from inurl to be available in outdir.

Parameters
  • inurl (str) – the input url to fetch data from

  • outdir (str) – the directory to write the data to

  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer

Returns

None

push_sink_data(inpath, outurl, datatype=None)[source]

Write out the sink data from the inpath to the outurl.

Parameters
  • inpath (str) – the path of the data to be pushed

  • outurl (str) – the url to write the data to

  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer

Returns

None

put_url(inpath, outurl)[source]

(abstract) Put the files to the external data store.

Parameters
  • inpath – path to the local data

  • outurl – url to where to store the data in the external data store.

put_value(value, outurl)[source]

(abstract) Put the files to the external data store.

Parameters
  • value – the value to store

  • outurl – url to where to store the data in the external data store.

abstract property scheme

(abstract) This abstract property is to be overwritten by a subclass to indicate the url scheme associated with the IOPlugin.

setup(*args, **kwargs)[source]

(abstract) Setup before data transfer. This can be any function that needs to be used to prepare the plugin for data transfer.

url_to_path(url)[source]

(abstract) Get the path to a file from a url.

Parameters

url (str) – the url to retrieve the path for

Returns

the corresponding path

Return type

str

provenance Module

class fastr.core.provenance.Provenance(host=None)[source]

Bases: object

The Provenance object keeps track of everything that happens to a data object.

__dict__ = mappingproxy({'__module__': 'fastr.core.provenance', '__doc__': '\n The Provenance object keeps track of everything that happens to a data object.\n ', '__init__': <function Provenance.__init__>, '_add_namespace': <function Provenance._add_namespace>, 'agent': <function Provenance.agent>, 'activity': <function Provenance.activity>, 'entity': <function Provenance.entity>, 'init_provenance': <function Provenance.init_provenance>, 'collect_provenance': <function Provenance.collect_provenance>, 'collect_input_argument_provenance': <function Provenance.collect_input_argument_provenance>, 'data_uri': <staticmethod object>, 'get_parent_provenance': <staticmethod object>, 'serialize': <function Provenance.serialize>, '__dict__': <attribute '__dict__' of 'Provenance' objects>, '__weakref__': <attribute '__weakref__' of 'Provenance' objects>})
__init__(host=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.provenance'
__weakref__

list of weak references to the object (if defined)

activity(identifier, start_time=None, end_time=None, other_attributes=None)[source]
agent(identifier, other_attributes=None)[source]
collect_input_argument_provenance(input_argument)[source]
collect_provenance(job, advanced_flow=False)[source]

Collect the provenance for this job

static data_uri(value, job)[source]
entity(identifier, other_attributes=None)[source]
static get_parent_provenance(value)[source]

Find the provenance of the parent job

Parameters

value (str) – url for the value for which to find the job

Returns

the provenance of the job that created the value

Raises
init_provenance(job)[source]

Create initial provenance document

serialize(filename, format)[source]

resourcelimit Module

Module for the management of resource limits of compute resources

class fastr.core.resourcelimit.ResourceLimit(cores=1, memory='2G', time=None)[source]

Bases: object

__eq__(other)[source]

Check if two resource limits are equal

Parameters

other – resource limit to test against

Return type

bool

__getstate__()[source]
Return type

dict

__hash__ = None
__init__(cores=1, memory='2G', time=None)[source]

An object describing resource requirements/limits for a node

Parameters
  • cores (Optional[int]) – number of cores

  • memory (Union[str, int, None]) – memory specification, can be int with number of megabytes or a string with numbers ending on M, G, T, P for megabytes, gigabytes, terrabytes or petabytes. Note that the number has to be an integer, e.g. 1500M would work, whereas 1.5G would be invalid

  • time (Union[str, int, None]) – run time specification, this can be an int with the number of seconds or a string in the HH:MM:SS, MM:SS, or SS format. Where HH, MM, and SS are integers representing the number of hours, minutes and seconds.

__module__ = 'fastr.core.resourcelimit'
__ne__(other)[source]

Check if two resource limits are not equal

Parameters

other – resource limit to test against

Return type

bool

__setstate__(state)[source]
__slots__ = ('_cores', '_memory', '_time')
copy()[source]

Return a copy of current resource limit object

Return type

ResourceLimit

property cores

The required number of gpus

Return type

int

property memory

The required memory in megabytes

Return type

int

property time

The required time in seconds

Return type

int

samples Module

This package holds the classes for working with samples.

class fastr.core.samples.ContainsSamples[source]

Bases: fastr.core.samples.HasSamples

__abstractmethods__ = frozenset({'samples'})
__getitem__(item)[source]
__module__ = 'fastr.core.samples'
__setitem__(key, value)[source]
property dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns

dimensions

Return type

tuple

abstract property samples
class fastr.core.samples.HasSamples[source]

Bases: fastr.core.dimension.HasDimensions

Base class for all classes that supply samples. This base class allows to only define __getitem__ and size and get all other basic functions mixed in so that the object behaves similar to a Mapping.

__abstractmethods__ = frozenset({'__getitem__', 'dimensions'})
__contains__(item)[source]
abstract __getitem__(item)[source]
__iter__()[source]
__module__ = 'fastr.core.samples'
ids()[source]
indexes()[source]
items()[source]
iteritems()[source]
class fastr.core.samples.SampleBaseId(*args)[source]

Bases: tuple

This class represents a sample id. A sample id is a multi-dimensional id that has a simple, consistent string representation.

__add__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way.

__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample id. A sample id is a multi-dimensional\n id that has a simple, consistent string representation.\n ', '_element_type': <class 'NoneType'>, '__new__': <staticmethod object>, '__getnewargs__': <function SampleBaseId.__getnewargs__>, '__repr__': <function SampleBaseId.__repr__>, '__str__': <function SampleBaseId.__str__>, '__add__': <function SampleBaseId.__add__>, '__radd__': <function SampleBaseId.__radd__>, '__dict__': <attribute '__dict__' of 'SampleBaseId' objects>})
__getnewargs__()[source]

Get new args gives the arguments to use to re-create this object, This is used for serialization.

__module__ = 'fastr.core.samples'
static __new__(cls, *args)[source]

Create a new SampleId

Parameters

args (iterator/iterable of element type or element type) – the strings to make sample id for

__radd__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way. This is the right-hand version of the operator.

__repr__()[source]

Get a string representation for the SampleBaseId

Returns

the string representation

Return type

str

__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns

the string version

Return type

str

class fastr.core.samples.SampleCollection(dimnames, parent)[source]

Bases: collections.abc.MutableMapping, fastr.core.dimension.HasDimensions

The SampleCollections is a class that contains the data including a form of ordering. Each sample is reachable both by its SampleId and a SampleIndex. The object is sparse, so not all SampleId have to be defined allowing for non-rectangular data shapes.

Note

This object is meant to replace both the SampleIdList and the ValueStorage.

__abstractmethods__ = frozenset({})
__contains__(item)[source]

Check if an item is in the SampleCollection. The item can be a SampleId or SampleIndex. If the item is a slicing SampleIndex, then check if it would return any data (True) or no data (False)

Parameters

item (SampleId, SampleIndex) – the item to check for

Returns

flag indicating item is in the collections

Return type

bool

__delitem__(key)[source]

Remove an item from the SampleCollection

Parameters

key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to remove

__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n The SampleCollections is a class that contains the data including a form\n of ordering. Each sample is reachable both by its SampleId and a\n SampleIndex. The object is sparse, so not all SampleId have to be defined\n allowing for non-rectangular data shapes.\n\n .. note::\n\n This object is meant to replace both the SampleIdList and the\n ValueStorage.\n ', '__init__': <function SampleCollection.__init__>, '__repr__': <function SampleCollection.__repr__>, '__contains__': <function SampleCollection.__contains__>, '__getitem__': <function SampleCollection.__getitem__>, '__setitem__': <function SampleCollection.__setitem__>, '__delitem__': <function SampleCollection.__delitem__>, '__iter__': <function SampleCollection.__iter__>, '__len__': <function SampleCollection.__len__>, 'dimensions': <property object>, 'ndims': <property object>, 'parent': <property object>, 'fullid': <property object>, '__dict__': <attribute '__dict__' of 'SampleCollection' objects>, '__weakref__': <attribute '__weakref__' of 'SampleCollection' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 58})
__getitem__(item)[source]

Retrieve (a) SampleItem(s) from the SampleCollection using the SampleId or SampleIndex. If the item is a tuple, it should be valid tuple for constructing either a SampleId or SampleIndex.

Parameters

item (SampleId, SampleIndex, or tuple) – the identifier of the item to retrieve

Returns

the requested item

Return type

SampleItem

Raises
__init__(dimnames, parent)[source]

Createa a new SampleCollection

__iter__()[source]

Iterate over the indices

__len__()[source]

Get the number of samples in the SampleCollections.

__module__ = 'fastr.core.samples'
__repr__()[source]

Return repr(self).

__setitem__(key, value)[source]

Set an item to the SampleCollection. The key can be a SampleId, SampleIndex or a tuple containing a SampleId and SampleIndex. The value can be a SampleItem (with the SampleId and SampleIndex matching), a tuple with values (assuming no depending jobs), or a with a list of values and a set of depending jobs.

Parameters
  • key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to store

  • value (SampleItem, tuple of values, or tuple of tuple of values and set of depending jobs) – the value of the SampleItem to store

Raises
__weakref__

list of weak references to the object (if defined)

property dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns

dimensions

Return type

tuple

property fullid

The full defining ID for the SampleIdList

property ndims

The number of dimensions in this SampleCollection

property parent

The parent object holding the SampleCollection

class fastr.core.samples.SampleId(*args)[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using human readable strings

__module__ = 'fastr.core.samples'
class fastr.core.samples.SampleIndex(*args)[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using the location in the N-d data structure.

__module__ = 'fastr.core.samples'
__repr__()[source]

Get a string representation for the SampleIndex

Returns

the string representation

Return type

str

__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns

the string version

Return type

str

expand(size)[source]

Function expanding a slice SampleIndex into a list of non-slice SampleIndex objects

Parameters

size – the size of the collection to slice

property isslice

Flag indicating that the SampleIndex is a slice (as opposed to a simple single index).

class fastr.core.samples.SampleItem(index, id, data, jobs=None, failed_annotations=None)[source]

Bases: fastr.core.samples.SampleItemBase

__module__ = 'fastr.core.samples'
static __new__(cls, index, id, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters
  • index (tuple, slice) – the sample index

  • id (SampleId) – the sample id

  • data (SampleValue, Mapping) – the data values

  • jobs (set) – set of jobs on which this SampleItems data depends.

  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).

class fastr.core.samples.SampleItemBase(index, id, data, jobs=None, failed_annotations=None)[source]

Bases: tuple

This class represents a sample item, a combination of a SampleIndex, SampleID, value and required jobs. The SampleItem based on a named tuple and has some extra methods to combine SampleItems easily.

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters

other (SampleItem) – The other item to add to this one

Returns

the combined SampleItem

Return type

SampleItem

__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample item, a combination of a SampleIndex,\n SampleID, value and required jobs. The SampleItem based on a named\n tuple and has some extra methods to combine SampleItems easily.\n ', '__new__': <staticmethod object>, '__repr__': <function SampleItemBase.__repr__>, '__getnewargs__': <function SampleItemBase.__getnewargs__>, '__add__': <function SampleItemBase.__add__>, 'combine': <staticmethod object>, 'index': <property object>, 'id': <property object>, 'data': <property object>, 'jobs': <property object>, 'failed_annotations': <property object>, 'cardinality': <property object>, 'dimensionality': <property object>, '__dict__': <attribute '__dict__' of 'SampleItemBase' objects>})
__getnewargs__()[source]

Get new args gives the arguments to use to re-create this object, This is used for serialization.

__module__ = 'fastr.core.samples'
static __new__(cls, index, id, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters
  • index (tuple, slice) – the sample index

  • id (SampleId) – the sample id

  • data (SampleValue, Mapping) – the data values

  • jobs (set) – set, tuple or list of jobs on which this SampleItems data depends.

  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).

__repr__()[source]

Get a string representation for the SampleItem

Returns

the string representation

Return type

str

property cardinality

The cardinality of this Sample

static combine(*args)[source]

Combine a number of SampleItems into a new one.

Parameters

args (iterable of SampleItems) – the SampleItems to combine

Returns

the combined SampleItem

Return type

SampleItem

It is possible to both give multiple arguments, where each argument is a SampleItem, or a single argument which is an iterable yielding SampleItems.

# variables a, b, c, d are SampleItems to combine
# These are all valid ways of combining the SampleItems
comb1 = SampleItem.combine(a, b, c, d)  # Using multiple arguments
l = [a, b, c, d]
comb2 = SampleItem.combine(l)  # Using a list of arguments
comb3 = SampleItem.combine(l.__iter__())  # Using an iterator
property data

The data SampleValue of the SampleItem

Returns

The value of this SampleItem

Return type

SampleValue

property dimensionality

The dimensionality of this Sample

property failed_annotations
property id

The sample id of the SampleItem

Returns

The id of this SampleItem

Return type

SampleId

property index

The index of the SampleItem

Returns

The index of this SampleItem

Return type

SampleIndex

property jobs

The set of the jobs on which this SampleItem depends

Returns

The jobs that generated the data for this SampleItem

Return type

set

class fastr.core.samples.SamplePayload(index, id, data, jobs=None, failed_annotations=None)[source]

Bases: fastr.core.samples.SampleItemBase

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters

other (SampleItem) – The other item to add to this one

Returns

the combined SamplePayload

Return type

SamplePayload

__module__ = 'fastr.core.samples'
static __new__(cls, index, id, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters
  • index (tuple, slice) – the sample index

  • id (SampleId) – the sample id

  • data (SampleValue, Mapping) – the data values

  • jobs (set) – set of jobs on which this SampleItems data depends.

  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).

class fastr.core.samples.SampleValue(*args, **kwargs)[source]

Bases: collections.abc.MutableMapping

A collection containing the content of a sample

__abstractmethods__ = frozenset({})
__add__(other)[source]
__delitem__(key)[source]
__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n A collection containing the content of a sample\n ', '_key_type': (<class 'int'>, <class 'str'>), '__init__': <function SampleValue.__init__>, '__repr__': <function SampleValue.__repr__>, '__getitem__': <function SampleValue.__getitem__>, '__setitem__': <function SampleValue.__setitem__>, '__getstate__': <function SampleValue.__getstate__>, '__setstate__': <function SampleValue.__setstate__>, '__delitem__': <function SampleValue.__delitem__>, '__len__': <function SampleValue.__len__>, '__iter__': <function SampleValue.__iter__>, 'is_sequence': <property object>, 'is_mapping': <property object>, 'sequence_part': <function SampleValue.sequence_part>, 'mapping_part': <function SampleValue.mapping_part>, 'cast': <function SampleValue.cast>, 'iterelements': <function SampleValue.iterelements>, '__radd__': <function SampleValue.__radd__>, '__add__': <function SampleValue.__add__>, '__dict__': <attribute '__dict__' of 'SampleValue' objects>, '__weakref__': <attribute '__weakref__' of 'SampleValue' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 58})
__getitem__(item)[source]
__getstate__()[source]
__init__(*args, **kwargs)[source]

Initialize self. See help(type(self)) for accurate signature.

__iter__()[source]
__len__()[source]
__module__ = 'fastr.core.samples'
__radd__(other)[source]
__repr__()[source]

Return repr(self).

__setitem__(key, value)[source]
__setstate__(state)[source]
__weakref__

list of weak references to the object (if defined)

cast(datatype)[source]
property is_mapping
property is_sequence
iterelements()[source]
mapping_part()[source]
sequence_part()[source]

target Module

The module containing the classes describing the targets.

class fastr.core.target.ProcessUsageCollection[source]

Bases: collections.abc.Sequence

__abstractmethods__ = frozenset({})
__dict__ = mappingproxy({'__module__': 'fastr.core.target', 'usage_type': <class 'fastr.core.target.SystemUsageInfo'>, '__init__': <function ProcessUsageCollection.__init__>, '__len__': <function ProcessUsageCollection.__len__>, '__getitem__': <function ProcessUsageCollection.__getitem__>, 'append': <function ProcessUsageCollection.append>, 'aggregate': <function ProcessUsageCollection.aggregate>, '__dict__': <attribute '__dict__' of 'ProcessUsageCollection' objects>, '__weakref__': <attribute '__weakref__' of 'ProcessUsageCollection' objects>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 58})
__getitem__(item)[source]
__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

__len__()[source]
__module__ = 'fastr.core.target'
__weakref__

list of weak references to the object (if defined)

aggregate(number_of_points)[source]
append(value)[source]
usage_type

alias of SystemUsageInfo

class fastr.core.target.SubprocessBasedTarget[source]

Bases: fastr.core.target.Target

Abstract based class for targets which call the target via a subprocess. Supplies a call_subprocess which executes the command and profiles the resulting subprocess.

__abstractmethods__ = frozenset({'run_command'})
__module__ = 'fastr.core.target'
call_subprocess(command)[source]

Call a subprocess with logging/timing/profiling

Parameters

command (list) – the command to execute

Returns

execution info

Return type

dict

monitor_process(process, resources)[source]

Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.

Parameters
  • process (subproces.Popen) – process to monitor

  • resources (ProcessUsageCollection) – list to append measurements to

class fastr.core.target.SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__module__ = 'fastr.core.target'
static __new__(_cls, timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Create new instance of SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

__repr__()

Return a nicely formatted representation string

__slots__ = ()
property cpu_percent

Alias for field number 1

property read_bytes

Alias for field number 4

property rmem

Alias for field number 3

property timestamp

Alias for field number 0

property vmem

Alias for field number 2

property write_bytes

Alias for field number 5

class fastr.core.target.Target[source]

Bases: fastr.abc.baseplugin.Plugin

The abstract base class for all targets. Execution with a target should follow the following pattern:

>>> with Target() as target:
...     target.run_commmand(['sleep', '10'])

The Target context operator will set the correct paths/initialization. Within the context command can be ran and when leaving the context the target reverts the state before.

__abstractmethods__ = frozenset({'run_command'})
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment where needed

__module__ = 'fastr.core.target'
abstract run_command(command)[source]

Run a command with the target

Return type

TargetResult

classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.core.target.TargetResult(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.core.target', '__init__': <function TargetResult.__init__>, 'as_dict': <function TargetResult.as_dict>, '__dict__': <attribute '__dict__' of 'TargetResult' objects>, '__weakref__': <attribute '__weakref__' of 'TargetResult' objects>, '__doc__': None})
__init__(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]

Class to formalize the resulting data of a Target

Parameters
  • return_code (int) – the return code of the process

  • stdout (Union[str, bytes]) – the stdout generated by the process

  • stderr (Union[str, bytes]) – the stderr generated by the process

  • command (List[Union[str, bytes]]) – the command executed

  • resource_usage (List[SystemUsageInfo]) – the resource use during execution

  • time_elapsed (int) – time used (in seconds)

__module__ = 'fastr.core.target'
__weakref__

list of weak references to the object (if defined)

as_dict()[source]

A dictionary of the data in the object (meant for serialization)

Return type

Dict[str, Union[int, str, List]]

tool Module

A module to maintain a tool.

Exported classes:

  • Tool – A class encapsulating a tool.

  • ParameterDescription – The base class containing the shared description of a parameter (both input and ouput).

  • InputParameterDescription – A class containing the description of an input parameter.

  • Output ParameterDescription – A class containing the description of an output parameter.

class fastr.core.tool.Tool(doc=None)[source]

Bases: fastr.abc.serializable.Serializable

The class encapsulating a tool.

DEFAULT_TARGET_CLASS = {'MacroNode': 'MacroTarget'}
TOOL_REFERENCE_FILE_NAME = '__fastr_tool_ref__.json'
TOOL_RESULT_FILE_NAME = '__fastr_tool_result.pickle.gz'
__dataschemafile__ = 'Tool.schema.json'
__eq__(other)[source]

Compare two Tool instances with each other.

Parameters

other (Tool) – the other instances to compare to

Returns

True if equal, False otherwise

__getstate__()[source]

Retrieve the state of the Tool

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(doc=None)[source]

Create a new Tool :param doc: path of toolfile or a dict containing the tool data :type doc: str or dict

__module__ = 'fastr.core.tool'
__repr__()[source]

Get a string representation for the Tool. This will show the inputs and output defined in a table-like structure.

Returns

the string representation

Return type

str

__setstate__(state)[source]

Set the state of the Tool by the given state.

Parameters

state (dict) – The state to populate the object with

__str__()[source]

Get a string version for the Tool

Returns

the string version

Return type

str

authors

List of authors of the tool. These people wrapped the executable but are not responsible for executable itself.

cite

This holds the citation you should use when publishing something based on this Tool

command

Command is a dictionary contain information about the command which is called by this Tool: command[‘interpreter’] holds the (possible) interpreter to use command[‘targets’] holds a per os/arch dictionary of files that should be executed command[‘url’] is the webpage of the command to be called command[‘version’] is the version of the command used command[‘description’] can help a description of the command command[‘authors’] lists the original authors of the command

property command_version
static compare_output_data(current_output_data, reference_output_data, validation_result, output)[source]
create_reference(input_data, output_directory, mount_name='__ref_tmp__', copy_input=True)[source]
description

Description of the tool and it’s functionality

execute(payload=None, **kwargs)[source]

Execute a Tool given the payload for a single run

Parameters

payload – the data to execute the Tool with

Returns

The result of the execution

Return type

InterFaceResult

property fullid

The full id of this tool

property hash
help

Man page for the Tool. Here usage and examples can be described in detail

property id
property inputs
name

Name of the tool, this should be a descriptive, human readable name.

namespace

The namespace this tools lives in, this will be set by the ToolManager on load

node_class

Class for of the Node to use

property ns_id

The namespace and id of the Tool

property outputs
property path

The path of the directory in which the tool definition file was located.

references

A list of documents and in depth reading about the methods used in this tool

requirements

Requirements for this Tool

Warning

Not yet implemented

serialize()[source]

Prepare data for serialization, this removes some fields from the state that are not needed when serializing to a file

tags

List of tags for this tool

property target

The OS and arch matched target definition.

test(reference=None)[source]

Run the tests for this tool

test_spec

alias of TestSpecification

classmethod test_tool(reference_data_dir, tool=None, input_data=None)[source]

Execute the tool with the input data specified and test the results against the refence data. This effectively tests the tool execution.

Parameters
  • reference_data_dir (str) – The path or vfs url of reference data to compare with

  • source_data (dict) – The source data to use

url

URL to website where this tool can be downloaded from

version

Version of the tool, not of the underlying software

version Module

Module containing the class that represent versions

class fastr.core.version.Version(*version)[source]

Bases: tuple

Class representing a software version definition. Allows for sorting and extraction of parts.

__dict__ = mappingproxy({'__module__': 'fastr.core.version', '__doc__': '\n Class representing a software version definition. Allows for sorting and\n extraction of parts.\n ', 'version_matcher': re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)'), 'date_version_matcher': re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)'), '__new__': <staticmethod object>, '__str__': <function Version.__str__>, '__repr__': <function Version.__repr__>, 'major': <property object>, 'minor': <property object>, 'extra': <property object>, 'extra_string': <property object>, 'status': <property object>, 'build': <property object>, 'suffix': <property object>, '__dict__': <attribute '__dict__' of 'Version' objects>})
__module__ = 'fastr.core.version'
static __new__(cls, *version)[source]

Class containing a version

Can be constructed by:

Version( 'major.$minor.$extra[0].$extra[1]$seperator$status$build$suffix' )
Version( major, minor, extra, status, build, suffix, seperator )
Version( (major, minor, extra, status, build, suffix, seperator) )
Version( [major, minor, extra, status, build, suffix, seperator] )
Parameters
  • major (int) – interger giving major version

  • minor (int) – is an integer (required)

  • extra (list of int) – is a list of integers

  • status (str) – can be “a”, “alpha”, “b”, “beta”, “rc”, or “r”

  • build (int) – is an integer

  • suffix (str) – can contain any combination of alpha-numeric character and “._-“

  • seperator (str) – is any of “.”, “-“, or “_”, which is located between $extra and $build

Note

The method based on strings is the recommended method. For strings the major and minor version are required, where for tuple and list constructors all seven elements are optional.

Examples:

>>> a = Version('0.1')
>>> print(tuple(a))
(0, 1, None, None, None, '', None)
>>> b = Version('2.5.3-rc2')
>>> print(tuple(b))
(2, 5, [3], 'rc', 2, '', '-')
>>> c = Version('1.2.3.4.5.6.7-beta8_with_suffix')
>>> print(tuple(c))
(1, 2, [3, 4, 5, 6, 7], 'beta', 8, '_with_suffix', '-')
__repr__()[source]

Return a in-editor representation of the version

__str__()[source]

Return a string representation of the version

property build

the build number, this is following the status (e.g. for 3.2-beta4, this would be 4)

date_version_matcher = re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)')
property extra

extra version extension as a list

property extra_string

extra version extension as a string

property major

major version

property minor

minor version

property status

the status of the version (a, alpha, b, beta, rc or r)

property suffix

the remainder of the version which was not formatted in a known way

version_matcher = re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)')

vfs Module

This module contains the virtual file system code. This is internally used object as used as base class for the IOPlugin.

class fastr.core.vfs.VirtualFileSystem[source]

Bases: object

The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the vfs:// url scheme.

A typical virtual filesystem url is formatted as vfs://mountpoint/relative/dir/from/mount.ext

Where the mountpoint is defined in the Config file. A list of the currently known mountpoints can be found in the fastr.config object

>>> fastr.config.mounts
{'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',
 'home': '/home/username/',
 'tmp': '/home/username/FastrTemp'}

This shows that a url with the mount home such as vfs://home/tempdir/testfile.txt would be translated into /home/username/tempdir/testfile.txt.

There are a few default mount points defined by Fastr (that can be changed via the config file).

mountpoint

default location

home

the users home directory (expanduser('~/'))

tmp

the fastr temprorary dir, defaults to tempfile.gettempdir()

example_data

the fastr example data directory, defaults $FASTRDIR/example/data

__dict__ = mappingproxy({'__module__': 'fastr.core.vfs', '__doc__': "\n The virtual file system class. This is an IOPlugin, but also heavily used\n internally in fastr for working with directories. The VirtualFileSystem\n uses the ``vfs://`` url scheme.\n\n A typical virtual filesystem url is formatted as ``vfs://mountpoint/relative/dir/from/mount.ext``\n\n Where the ``mountpoint`` is defined in the :ref:`config-file`. A list of\n the currently known mountpoints can be found in the ``fastr.config`` object\n\n .. code-block:: python\n\n >>> fastr.config.mounts\n {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',\n 'home': '/home/username/',\n 'tmp': '/home/username/FastrTemp'}\n\n This shows that a url with the mount ``home`` such as\n ``vfs://home/tempdir/testfile.txt`` would be translated into\n ``/home/username/tempdir/testfile.txt``.\n\n There are a few default mount points defined by Fastr (that can be changed\n via the config file).\n\n +--------------+-----------------------------------------------------------------------------+\n | mountpoint | default location |\n +==============+=============================================================================+\n | home | the users home directory (:py:func:`expanduser('~/') <os.path.expanduser>`) |\n +--------------+-----------------------------------------------------------------------------+\n | tmp | the fastr temprorary dir, defaults to ``tempfile.gettempdir()`` |\n +--------------+-----------------------------------------------------------------------------+\n | example_data | the fastr example data directory, defaults ``$FASTRDIR/example/data`` |\n +--------------+-----------------------------------------------------------------------------+\n\n ", '_status': (<PluginState.loaded: '\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m'>, ''), 'abstract': False, '__init__': <function VirtualFileSystem.__init__>, 'scheme': <property object>, 'setup': <function VirtualFileSystem.setup>, 'fetch_url': <function VirtualFileSystem.fetch_url>, 'fetch_value': <function VirtualFileSystem.fetch_value>, 'put_url': <function VirtualFileSystem.put_url>, 'put_value': <function VirtualFileSystem.put_value>, 'expand_url': <function VirtualFileSystem.expand_url>, 'url_to_path': <function VirtualFileSystem.url_to_path>, 'path_to_url': <function VirtualFileSystem.path_to_url>, 'copy_file_dir': <staticmethod object>, '_correct_separators': <staticmethod object>, '__dict__': <attribute '__dict__' of 'VirtualFileSystem' objects>, '__weakref__': <attribute '__weakref__' of 'VirtualFileSystem' objects>})
__init__()[source]

Instantiate the VFS plugin

Returns

the VirtualFileSysten plugin

__module__ = 'fastr.core.vfs'
__weakref__

list of weak references to the object (if defined)

abstract = False
static copy_file_dir(inpath, outpath)[source]

Helper function, copies a file or directory not caring what the inpath actually is

Parameters
  • inpath – path of the things to be copied

  • outpath – path of the destination

Returns

the result of shutil.copy2 or shutil.copytree (depending on inpath pointing to a file or directory)

expand_url(url)[source]

Try to expand the url. For vfs with will return the original url.

Parameters

url – url to expand

Returns

the expanded url (same as url)

fetch_url(inurl, outpath)[source]

Fetch the files from the vfs.

Parameters
  • inurl – url to the item in the data store, starts with vfs://

  • outpath – path where to store the fetch data locally

fetch_value(inurl)[source]

Fetch a value from an external vfs file.

Parameters

inurl – url of the value to read

Returns

the fetched value

path_to_url(path, mountpoint=None, scheme=None)[source]

Construct an url from a given mount point and a relative path to the mount point.

Parameters

path (str) – the path to find the url for

Mountpoint str

mountpoint the url should be under

Returns

url of the

put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters
  • inpath – path of the local data

  • outurl – url to where to store the data, starts with vfs://

put_value(value, outurl)[source]

Put the value in the external data store.

Parameters
  • value – value to store

  • outurl – url to where to store the data, starts with vfs://

property scheme
setup()[source]

The plugin setup, does nothing but needs to be implemented

url_to_path(url, scheme=None)[source]

Get the path to a file from a vfs url

Parameters

url (str) – url to get the path for

Returns

the matching path

Return type

str

Raises

Example (the mountpoint tmp points to /tmp):

>>> fastr.vfs.url_to_path('vfs://tmp/file.ext')
'/tmp/file.ext'