core Package

core Package

This module contains all of the core components of fastr. It has the classes to create networks and work with them.

cardinality Module

class fastr.core.cardinality.AnyCardinalitySpec(parent)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
validate(payload, cardinality)[source]

Validate cardinality given a payload and cardinality

Parameters:
  • payload (dict) – Payload of the corresponding job
  • cardinality (int) – Cardinality to validate
Return type:

bool

Returns:

Validity of the cardinality given the spec and payload

class fastr.core.cardinality.AsCardinalitySpec(parent, target)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, target)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters:key – Key for which the cardinality is calculated
Return type:Optional[int]
Returns:calculated cardinality
calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type:Optional[int]
calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters:node – Node for which the cardinality is calculated
Return type:Optional[int]
Returns:calculated cardinality
node
predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

class fastr.core.cardinality.CardinalitySpec(parent)[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.core.cardinality', '__init__': <function CardinalitySpec.__init__>, '__str__': <function CardinalitySpec.__str__>, '__repr__': <function CardinalitySpec.__repr__>, '__eq__': <function CardinalitySpec.__eq__>, '__ne__': <function CardinalitySpec.__ne__>, 'predefined': <property object>, 'validate': <function CardinalitySpec.validate>, '_validate': <function CardinalitySpec._validate>, 'calculate_planning_cardinality': <function CardinalitySpec.calculate_planning_cardinality>, 'calculate_execution_cardinality': <function CardinalitySpec.calculate_execution_cardinality>, 'calculate_job_cardinality': <function CardinalitySpec.calculate_job_cardinality>, '__dict__': <attribute '__dict__' of 'CardinalitySpec' objects>, '__weakref__': <attribute '__weakref__' of 'CardinalitySpec' objects>, '__doc__': None, '__hash__': None})
__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__ne__(other)[source]

Return self!=value.

__repr__()[source]

Console representation of the cardinality spec

Return type:str
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
__weakref__

list of weak references to the object (if defined)

calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters:key – Key for which the cardinality is calculated
Return type:Optional[int]
Returns:calculated cardinality
calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type:Optional[int]
calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters:node – Node for which the cardinality is calculated
Return type:Optional[int]
Returns:calculated cardinality
predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

validate(payload, cardinality, planning=True)[source]

Validate cardinality given a payload and cardinality

Parameters:
  • payload (Optional[dict]) – Payload of the corresponding job
  • cardinality (int) – Cardinality to validate
Return type:

bool

Returns:

Validity of the cardinality given the spec and payload

class fastr.core.cardinality.ChoiceCardinalitySpec(parent, options)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, options)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
class fastr.core.cardinality.IntCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
calculate_execution_cardinality(node)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters:key – Key for which the cardinality is calculated
Return type:int
Returns:calculated cardinality
calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type:Optional[int]
calculate_planning_cardinality()[source]

Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.

Parameters:node – Node for which the cardinality is calculated
Return type:int
Returns:calculated cardinality
predefined

Indicate whether the cardinality is predefined or can only be calculated after execution

class fastr.core.cardinality.MaxCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
class fastr.core.cardinality.MinCardinalitySpec(parent, value)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, value)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
class fastr.core.cardinality.RangeCardinalitySpec(parent, min, max)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, min, max)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
class fastr.core.cardinality.ValueCardinalitySpec(parent, target)[source]

Bases: fastr.core.cardinality.CardinalitySpec

__eq__(other)[source]

Test for equality

__hash__ = None
__init__(parent, target)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.cardinality'
__str__()[source]

String version of the cardinality spec, should be parseable by create_cardinality

Return type:str
calculate_execution_cardinality(key=None)[source]

Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.

Parameters:key – Key for which the cardinality is calculated
Return type:Optional[int]
Returns:calculated cardinality
calculate_job_cardinality(payload)[source]

Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.

Return type:Optional[int]
node
fastr.core.cardinality.create_cardinality(desc, parent)[source]

Create simplified description of the cardinality. This changes the string representation to a tuple that is easier to check at a later time.

Parameters:desc (str) – the string version of the cardinality
Parent:the parent input or output to which this cardinality spec belongs
Return type:CardinalitySpec
Returns:the simplified cardinality description
Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

The translation works with the following table:

cardinality string cardinality spec description
"*", any ``(‘any’,) Any cardinality is allowed
"N" ('int', N) A cardinality of N is required
"N-M" ('range', N, M) A cardinality between N and M is required
"*-M" ('max', M) A cardinality of maximal M is required
"N-*" ('min', N) A cardinality of minimal N is required
"[M,N,...,O,P]" ('choice', [M,N,...,O,P]) The cardinality should one of the given options
"as:input_id" ('as', 'input_id') The cardinality should match the cardinality of the given Input
"val:input_id" ('val', 'input_id') The cardinliaty should match the value of the given Input

Note

The maximumu, minimum and range are inclusive

dimension Module

class fastr.core.dimension.Dimension(name, size)[source]

Bases: object

A class representing a dimension. It contains the name and size of the dimension.

__eq__(other)[source]

Dimension is the same if the name and size are the same

__hash__ = None
__init__(name, size)[source]

The constructor for the dimension.

Parameters:
  • name (str) – Name of the dimension
  • size (int or sympy.Symbol) – Size fo the dimension
__module__ = 'fastr.core.dimension'
__ne__(other)[source]

The not equal test is simply the inverse of the equal test

__repr__()[source]

String representation of a Dimension

__slots__ = ('_name', '_size')
copy()[source]

Get a copy object of a Dimension

name
size
update_size(value)[source]
class fastr.core.dimension.ForwardsDimensions[source]

Bases: fastr.core.dimension.HasDimensions

Class of objects that have dimensions not because they contain data with dimensions but forward them (optionally with changes via combine_dimensions)

__abstractmethods__ = frozenset({'source', 'combine_dimensions'})
__module__ = 'fastr.core.dimension'
combine_dimensions(dimensions)[source]

Method to combine/manipulate the dimensions

Parameters:dimensions – the input dimensions from the source
Returns:dimensions manipulated for this object
Return type:tuple of dimensions
dimensions

The dimensions of the object based on the forwarding

source

The source object from which the dimensions are forwarded

Returns:the object from which the dimensions are forwarded
Return type:HasDimensions
class fastr.core.dimension.HasDimensions[source]

Bases: object

A Mixin class for any object that has a notion of dimensions and size. It uses the dimension property to expose the dimension name and size.

__abstractmethods__ = frozenset({'dimensions'})
__dict__ = mappingproxy({'__module__': 'fastr.core.dimension', '__doc__': '\n A Mixin class for any object that has a notion of dimensions and size. It\n uses the dimension property to expose the dimension name and size.\n ', 'dimensions': <abc.abstractproperty object>, 'dimnames': <property object>, 'size': <property object>, 'ndims': <property object>, '__dict__': <attribute '__dict__' of 'HasDimensions' objects>, '__weakref__': <attribute '__weakref__' of 'HasDimensions' objects>, '__abstractmethods__': frozenset({'dimensions'}), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})
__module__ = 'fastr.core.dimension'
__weakref__

list of weak references to the object (if defined)

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
dimnames

A tuple containing the dimension names of this object. All items of the tuple are of type str.

ndims

The number of dimensions in this object

size

A tuple containing the size of this object. All items of the tuple are of type int or sympy.Symbol.

interface Module

A module that describes the interface of a Tool. It specifies how a set of
input values will be translated to commands to be executed. This creates a generic interface to different ways of executing underlying software.
class fastr.core.interface.InputSpec[source]

Bases: fastr.core.interface.InputSpec

Class containing the information about an Input Specification, this is essentially a data class (but

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Input Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, '__dict__': <attribute '__dict__' of 'InputSpec' objects>})
__module__ = 'fastr.core.interface'
static __new__(cls, id_, cardinality, datatype, required=False, description='', default=None, hidden=False)[source]

Create new instance of InputSpec(id, cardinality, datatype, required, description, default, hidden)

fastr.core.interface.InputSpecBase

alias of fastr.core.interface.InputSpec

class fastr.core.interface.Interface[source]

Bases: fastr.abc.baseplugin.Plugin, fastr.abc.serializable.Serializable

Abstract base class of all Interfaces. Defines the minimal requirements for all Interface implementations.

__abstractmethods__ = frozenset({'execute', 'expanding', 'inputs', '__setstate__', '__getstate__', 'outputs'})
__getstate__()[source]

Retrieve the state of the Interface

Returns:the state of the object
Rtype dict:
__module__ = 'fastr.core.interface'
__setstate__(state)[source]

Set the state of the Interface

execute(target, payload)[source]

Execute the interface given the a target and payload. The payload should have the form {‘input’: {‘input_id_a’: (value, value), ‘input_id_b’: (value, value)},

‘output’: {‘output_id_a’: (value, value), ‘output_id_b’: (value, value)}}
Parameters:
  • target – the target to call
  • payload – the payload to use
Returns:

the result of the execution

Return type:

(tuple of) InterfaceResult

expanding

Indicates whether or not this Interface will result in multiple samples per run. If the flow is unaffected, this will be zero, if it is nonzero it means that number of dimension will be added to the sample array.

inputs

OrderedDict of Inputs connected to the Interface. The format should be {input_id: InputSpec}.

outputs

OrderedDict of Output connected to the Interface. The format should be {output_id: OutputSpec}.

classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.core.interface.InterfaceResult(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]

Bases: object

The class in which Interfaces should wrap their results to be picked up by fastr

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n The class in which Interfaces should wrap their results to be picked up by fastr\n ', '__init__': <function InterfaceResult.__init__>, '__dict__': <attribute '__dict__' of 'InterfaceResult' objects>, '__weakref__': <attribute '__weakref__' of 'InterfaceResult' objects>})
__init__(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.interface'
__weakref__

list of weak references to the object (if defined)

class fastr.core.interface.OutputSpec[source]

Bases: fastr.core.interface.OutputSpec

Class containing the information about an Output Specification, this is essentially a data class (but

__dict__ = mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Output Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, '__dict__': <attribute '__dict__' of 'OutputSpec' objects>})
__module__ = 'fastr.core.interface'
static __new__(cls, id_, cardinality, datatype, automatic=True, required=False, description='', hidden=False)[source]

Create new instance of OutputSpec(id, cardinality, datatype, automatic, required, description, hidden)

fastr.core.interface.OutputSpecBase

alias of fastr.core.interface.OutputSpec

ioplugin Module

This module contains the manager class for IOPlugins and the base class for all IOPlugins

class fastr.core.ioplugin.IOPlugin[source]

Bases: fastr.abc.baseplugin.Plugin

IOPlugins are used for data import and export for the sources and sinks. The main use of the IOPlugins is during execution (see Execution). The IOPlugins can be accessed via fastr.ioplugins, but generally there should be no need for direct interaction with these objects. The use of is mainly via the URL used to specify source and sink data.

__abstractmethods__ = frozenset({'scheme'})
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.core.ioplugin'
cleanup()[source]

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
fetch_url(inurl, outfile)[source]

(abstract) Fetch a file from an external data source.

Parameters:
  • inurl – url to the item in the data store
  • outpath – path where to store the fetch data locally
fetch_value(inurl)[source]

(abstract) Fetch a value from an external data source.

Parameters:inurl – the url of the value to retrieve
Returns:the fetched value
static isurl(string)[source]

Test if given string is an url.

Parameters:string (str) – string to test
Returns:True if the string is an url, False otherwise
Return type:bool
path_to_url(path, mountpoint=None)[source]

(abstract) Construct an url from a given mount point and a relative path to the mount point.

Parameters:
  • path (str) – the path to determine the url for
  • mountpoint (str or None) – the mount point to use, will be automatically detected if None is given
Returns:

url matching the path

Return type:

str

static print_result(result)[source]

Print the result of the IOPlugin to stdout to be picked up by the tool

Parameters:result – value to print as a result
Returns:None
pull_source_data(inurl, outdir, sample_id, datatype=None)[source]

Transfer the source data from inurl to be available in outdir.

Parameters:
  • inurl (str) – the input url to fetch data from
  • outdir (str) – the directory to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

push_sink_data(inpath, outurl, datatype=None)[source]

Write out the sink data from the inpath to the outurl.

Parameters:
  • inpath (str) – the path of the data to be pushed
  • outurl (str) – the url to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

put_url(inpath, outurl)[source]

(abstract) Put the files to the external data store.

Parameters:
  • inpath – path to the local data
  • outurl – url to where to store the data in the external data store.
put_value(value, outurl)[source]

(abstract) Put the files to the external data store.

Parameters:
  • value – the value to store
  • outurl – url to where to store the data in the external data store.
scheme

(abstract) This abstract property is to be overwritten by a subclass to indicate the url scheme associated with the IOPlugin.

setup(*args, **kwargs)[source]

(abstract) Setup before data transfer. This can be any function that needs to be used to prepare the plugin for data transfer.

url_to_path(url)[source]

(abstract) Get the path to a file from a url.

Parameters:url (str) – the url to retrieve the path for
Returns:the corresponding path
Return type:str

provenance Module

class fastr.core.provenance.Provenance(host=None)[source]

Bases: object

The Provenance object keeps track of everything that happens to a data object.

__dict__ = mappingproxy({'__module__': 'fastr.core.provenance', '__doc__': '\n The Provenance object keeps track of everything that happens to a data object.\n ', '__init__': <function Provenance.__init__>, '_add_namespace': <function Provenance._add_namespace>, 'agent': <function Provenance.agent>, 'activity': <function Provenance.activity>, 'entity': <function Provenance.entity>, 'init_provenance': <function Provenance.init_provenance>, 'collect_provenance': <function Provenance.collect_provenance>, 'collect_input_argument_provenance': <function Provenance.collect_input_argument_provenance>, 'data_uri': <staticmethod object>, 'get_parent_provenance': <staticmethod object>, 'serialize': <function Provenance.serialize>, '__dict__': <attribute '__dict__' of 'Provenance' objects>, '__weakref__': <attribute '__weakref__' of 'Provenance' objects>})
__init__(host=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.core.provenance'
__weakref__

list of weak references to the object (if defined)

activity(identifier, start_time=None, end_time=None, other_attributes=None)[source]
agent(identifier, other_attributes=None)[source]
collect_input_argument_provenance(input_argument)[source]
collect_provenance(job, advanced_flow=False)[source]

Collect the provenance for this job

static data_uri(value, job)[source]
entity(identifier, other_attributes=None)[source]
static get_parent_provenance(value)[source]

Find the provenance of the parent job

Parameters:

value (str) – url for the value for which to find the job

Returns:

the provenance of the job that created the value

Raises:
init_provenance(job)[source]

Create initial provenance document

serialize(filename, format)[source]

resourcelimit Module

Module for the management of resource limits of compute resources

class fastr.core.resourcelimit.ResourceLimit(cores=1, memory='2G', time='01:00:00')[source]

Bases: object

__eq__(other)[source]

Check if two resource limits are equal

Parameters:other – resource limit to test against
Return type:bool
__getstate__()[source]
Return type:dict
__hash__ = None
__init__(cores=1, memory='2G', time='01:00:00')[source]

An object describing resource requirements/limits for a node

Parameters:
  • cores (Optional[int]) – number of cores
  • memory (Union[str, int, None]) – memory specification, can be int with number of megabytes or a string with numbers ending on M, G, T, P for megabytes, gigabytes, terrabytes or petabytes. Note that the number has to be an integer, e.g. 1500M would work, whereas 1.5G would be invalid
  • time (Union[str, int, None]) – run time specification, this can be an int with the number of seconds or a string in the HH:MM:SS, MM:SS, or SS format. Where HH, MM, and SS are integers representing the number of hours, minutes and seconds.
__module__ = 'fastr.core.resourcelimit'
__ne__(other)[source]

Check if two resource limits are not equal

Parameters:other – resource limit to test against
Return type:bool
__setstate__(state)[source]
__slots__ = ('_cores', '_memory', '_time')
copy()[source]

Return a copy of current resource limit object

Return type:ResourceLimit
cores

The required number of gpus

Return type:int
memory

The required memory in megabytes

Return type:int
time

The required time in seconds

Return type:int

samples Module

This package holds the classes for working with samples.

class fastr.core.samples.ContainsSamples[source]

Bases: fastr.core.samples.HasSamples

__abstractmethods__ = frozenset({'samples'})
__getitem__(item)[source]
__module__ = 'fastr.core.samples'
__setitem__(key, value)[source]
dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
samples
class fastr.core.samples.HasSamples[source]

Bases: fastr.core.dimension.HasDimensions

Base class for all classes that supply samples. This base class allows to only define __getitem__ and size and get all other basic functions mixed in so that the object behaves similar to a Mapping.

__abstractmethods__ = frozenset({'__getitem__', 'dimensions'})
__contains__(item)[source]
__getitem__(item)[source]
__iter__()[source]
__module__ = 'fastr.core.samples'
ids()[source]
indexes()[source]
items()[source]
iteritems()[source]
class fastr.core.samples.SampleBaseId[source]

Bases: tuple

This class represents a sample id. A sample id is a multi-dimensional id that has a simple, consistent string representation.

__add__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way.

__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample id. A sample id is a multi-dimensional\n id that has a simple, consistent string representation.\n ', '_element_type': <class 'NoneType'>, '__new__': <staticmethod object>, '__repr__': <function SampleBaseId.__repr__>, '__str__': <function SampleBaseId.__str__>, '__add__': <function SampleBaseId.__add__>, '__radd__': <function SampleBaseId.__radd__>, '__dict__': <attribute '__dict__' of 'SampleBaseId' objects>})
__module__ = 'fastr.core.samples'
static __new__(cls, *args)[source]

Create a new SampleId

Parameters:args (iterator/iterable of element type or element type) – the strings to make sample id for
__radd__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way. This is the right-hand version of the operator.

__repr__()[source]

Get a string representation for the SampleBaseId

Returns:the string representation
Return type:str
__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns:the string version
Return type:str
class fastr.core.samples.SampleCollection(dimnames, parent)[source]

Bases: collections.abc.MutableMapping, fastr.core.dimension.HasDimensions

The SampleCollections is a class that contains the data including a form of ordering. Each sample is reachable both by its SampleId and a SampleIndex. The object is sparse, so not all SampleId have to be defined allowing for non-rectangular data shapes.

Note

This object is meant to replace both the SampleIdList and the ValueStorage.

__abstractmethods__ = frozenset()
__contains__(item)[source]

Check if an item is in the SampleCollection. The item can be a SampleId or SampleIndex. If the item is a slicing SampleIndex, then check if it would return any data (True) or no data (False)

Parameters:item (SampleId, SampleIndex) – the item to check for
Returns:flag indicating item is in the collections
Return type:bool
__delitem__(key)[source]

Remove an item from the SampleCollection

Parameters:key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to remove
__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n The SampleCollections is a class that contains the data including a form\n of ordering. Each sample is reachable both by its SampleId and a\n SampleIndex. The object is sparse, so not all SampleId have to be defined\n allowing for non-rectangular data shapes.\n\n .. note::\n\n This object is meant to replace both the SampleIdList and the\n ValueStorage.\n ', '__init__': <function SampleCollection.__init__>, '__repr__': <function SampleCollection.__repr__>, '__contains__': <function SampleCollection.__contains__>, '__getitem__': <function SampleCollection.__getitem__>, '__setitem__': <function SampleCollection.__setitem__>, '__delitem__': <function SampleCollection.__delitem__>, '__iter__': <function SampleCollection.__iter__>, '__len__': <function SampleCollection.__len__>, 'dimensions': <property object>, 'ndims': <property object>, 'parent': <property object>, 'fullid': <property object>, '__dict__': <attribute '__dict__' of 'SampleCollection' objects>, '__weakref__': <attribute '__weakref__' of 'SampleCollection' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})
__getitem__(item)[source]

Retrieve (a) SampleItem(s) from the SampleCollection using the SampleId or SampleIndex. If the item is a tuple, it should be valid tuple for constructing either a SampleId or SampleIndex.

Parameters:

item (SampleId, SampleIndex, or tuple) – the identifier of the item to retrieve

Returns:

the requested item

Return type:

SampleItem

Raises:
__init__(dimnames, parent)[source]

Createa a new SampleCollection

__iter__()[source]

Iterate over the indices

__len__()[source]

Get the number of samples in the SampleCollections.

__module__ = 'fastr.core.samples'
__repr__()[source]

Return repr(self).

__setitem__(key, value)[source]

Set an item to the SampleCollection. The key can be a SampleId, SampleIndex or a tuple containing a SampleId and SampleIndex. The value can be a SampleItem (with the SampleId and SampleIndex matching), a tuple with values (assuming no depending jobs), or a with a list of values and a set of depending jobs.

Parameters:
  • key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to store
  • value (SampleItem, tuple of values, or tuple of tuple of values and set of depending jobs) – the value of the SampleItem to store
Raises:
__weakref__

list of weak references to the object (if defined)

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
fullid

The full defining ID for the SampleIdList

ndims

The number of dimensions in this SampleCollection

parent

The parent object holding the SampleCollection

class fastr.core.samples.SampleId[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using human readable strings

__module__ = 'fastr.core.samples'
class fastr.core.samples.SampleIndex[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using the location in the N-d data structure.

__module__ = 'fastr.core.samples'
__repr__()[source]

Get a string representation for the SampleIndex

Returns:the string representation
Return type:str
__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns:the string version
Return type:str
expand(size)[source]

Function expanding a slice SampleIndex into a list of non-slice SampleIndex objects

Parameters:size – the size of the collection to slice
isslice

Flag indicating that the SampleIndex is a slice (as opposed to a simple single index).

class fastr.core.samples.SampleItem[source]

Bases: fastr.core.samples.SampleItemBase

__module__ = 'fastr.core.samples'
static __new__(cls, index, id_, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set of jobs on which this SampleItems data depends.
  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
class fastr.core.samples.SampleItemBase[source]

Bases: tuple

This class represents a sample item, a combination of a SampleIndex, SampleID, value and required jobs. The SampleItem based on a named tuple and has some extra methods to combine SampleItems easily.

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters:other (SampleItem) – The other item to add to this one
Returns:the combined SampleItem
Return type:SampleItem
__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample item, a combination of a SampleIndex,\n SampleID, value and required jobs. The SampleItem based on a named\n tuple and has some extra methods to combine SampleItems easily.\n ', '__new__': <staticmethod object>, '__repr__': <function SampleItemBase.__repr__>, '__getnewargs__': <function SampleItemBase.__getnewargs__>, '__add__': <function SampleItemBase.__add__>, 'combine': <staticmethod object>, 'index': <property object>, 'id': <property object>, 'data': <property object>, 'jobs': <property object>, 'failed_annotations': <property object>, 'cardinality': <property object>, 'dimensionality': <property object>, '__dict__': <attribute '__dict__' of 'SampleItemBase' objects>})
__getnewargs__()[source]

Get new args gives the arguments to use to re-create this object, This is used for serialization.

__module__ = 'fastr.core.samples'
static __new__(cls, index, id_, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set, tuple or list of jobs on which this SampleItems data depends.
  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
__repr__()[source]

Get a string representation for the SampleItem

Returns:the string representation
Return type:str
cardinality

The cardinality of this Sample

static combine(*args)[source]

Combine a number of SampleItems into a new one.

Parameters:*args

the SampleItems to combine

Returns:the combined SampleItem
Return type:SampleItem

It is possible to both give multiple arguments, where each argument is a SampleItem, or a single argument which is an iterable yielding SampleItems.

# variables a, b, c, d are SampleItems to combine
# These are all valid ways of combining the SampleItems
comb1 = SampleItem.combine(a, b, c, d)  # Using multiple arguments
l = [a, b, c, d]
comb2 = SampleItem.combine(l)  # Using a list of arguments
comb3 = SampleItem.combine(l.__iter__())  # Using an iterator
data

The data SampleValue of the SampleItem

Returns:The value of this SampleItem
Return type:SampleValue
dimensionality

The dimensionality of this Sample

failed_annotations
id

The sample id of the SampleItem

Returns:The id of this SampleItem
Return type:SampleId
index

The index of the SampleItem

Returns:The index of this SampleItem
Return type:SampleIndex
jobs

The set of the jobs on which this SampleItem depends

Returns:The jobs that generated the data for this SampleItem
Return type:set
class fastr.core.samples.SamplePayload[source]

Bases: fastr.core.samples.SampleItemBase

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters:other (SampleItem) – The other item to add to this one
Returns:the combined SamplePayload
Return type:SamplePayload
__module__ = 'fastr.core.samples'
static __new__(cls, index, id_, data, jobs=None, failed_annotations=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set of jobs on which this SampleItems data depends.
  • failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
class fastr.core.samples.SampleValue(*args, **kwargs)[source]

Bases: collections.abc.MutableMapping

A collection containing the content of a sample

__abstractmethods__ = frozenset()
__add__(other)[source]
__delitem__(key)[source]
__dict__ = mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n A collection containing the content of a sample\n ', '_key_type': (<class 'int'>, <class 'str'>), '__init__': <function SampleValue.__init__>, '__repr__': <function SampleValue.__repr__>, '__getitem__': <function SampleValue.__getitem__>, '__setitem__': <function SampleValue.__setitem__>, '__getstate__': <function SampleValue.__getstate__>, '__setstate__': <function SampleValue.__setstate__>, '__delitem__': <function SampleValue.__delitem__>, '__len__': <function SampleValue.__len__>, '__iter__': <function SampleValue.__iter__>, 'is_sequence': <property object>, 'is_mapping': <property object>, 'sequence_part': <function SampleValue.sequence_part>, 'mapping_part': <function SampleValue.mapping_part>, 'cast': <function SampleValue.cast>, 'iterelements': <function SampleValue.iterelements>, '__radd__': <function SampleValue.__radd__>, '__add__': <function SampleValue.__add__>, '__dict__': <attribute '__dict__' of 'SampleValue' objects>, '__weakref__': <attribute '__weakref__' of 'SampleValue' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})
__getitem__(item)[source]
__getstate__()[source]
__init__(*args, **kwargs)[source]

Initialize self. See help(type(self)) for accurate signature.

__iter__()[source]
__len__()[source]
__module__ = 'fastr.core.samples'
__radd__(other)[source]
__repr__()[source]

Return repr(self).

__setitem__(key, value)[source]
__setstate__(state)[source]
__weakref__

list of weak references to the object (if defined)

cast(datatype)[source]
is_mapping
is_sequence
iterelements()[source]
mapping_part()[source]
sequence_part()[source]

target Module

The module containing the classes describing the targets.

class fastr.core.target.ProcessUsageCollection[source]

Bases: collections.abc.Sequence

__abstractmethods__ = frozenset()
__dict__ = mappingproxy({'__module__': 'fastr.core.target', 'usage_type': <class 'fastr.core.target.SystemUsageInfo'>, '__init__': <function ProcessUsageCollection.__init__>, '__len__': <function ProcessUsageCollection.__len__>, '__getitem__': <function ProcessUsageCollection.__getitem__>, 'append': <function ProcessUsageCollection.append>, 'aggregate': <function ProcessUsageCollection.aggregate>, '__dict__': <attribute '__dict__' of 'ProcessUsageCollection' objects>, '__weakref__': <attribute '__weakref__' of 'ProcessUsageCollection' objects>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})
__getitem__(item)[source]
__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

__len__()[source]
__module__ = 'fastr.core.target'
__weakref__

list of weak references to the object (if defined)

aggregate(number_of_points)[source]
append(value)[source]
usage_type

alias of SystemUsageInfo

class fastr.core.target.SubprocessBasedTarget[source]

Bases: fastr.core.target.Target

Abstract based class for targets which call the target via a subprocess. Supplies a call_subprocess which executes the command and profiles the resulting subprocess.

__abstractmethods__ = frozenset({'run_command'})
__module__ = 'fastr.core.target'
call_subprocess(command)[source]

Call a subprocess with logging/timing/profiling

Parameters:command (list) – the command to execute
Returns:execution info
Return type:dict
monitor_process(process, resources)[source]

Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.

Parameters:
  • process (subproces.Popen) – process to monitor
  • resources (ProcessUsageCollection) – list to append measurements to
class fastr.core.target.SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__module__ = 'fastr.core.target'
static __new__(_cls, timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Create new instance of SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

__repr__()

Return a nicely formatted representation string

__slots__ = ()
cpu_percent

Alias for field number 1

read_bytes

Alias for field number 4

rmem

Alias for field number 3

timestamp

Alias for field number 0

vmem

Alias for field number 2

write_bytes

Alias for field number 5

class fastr.core.target.Target[source]

Bases: fastr.abc.baseplugin.Plugin

The abstract base class for all targets. Execution with a target should follow the following pattern:

>>> with Target() as target:
...     target.run_commmand(['sleep', '10'])

The Target context operator will set the correct paths/initialization. Within the context command can be ran and when leaving the context the target reverts the state before.

__abstractmethods__ = frozenset({'run_command'})
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment where needed

__module__ = 'fastr.core.target'
run_command(command)[source]

Run a command with the target

Return type:TargetResult
classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.core.target.TargetResult(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.core.target', '__init__': <function TargetResult.__init__>, 'as_dict': <function TargetResult.as_dict>, '__dict__': <attribute '__dict__' of 'TargetResult' objects>, '__weakref__': <attribute '__weakref__' of 'TargetResult' objects>, '__doc__': None})
__init__(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]

Class to formalize the resulting data of a Target

Parameters:
  • return_code (int) – the return code of the process
  • stdout (Union[str, bytes]) – the stdout generated by the process
  • stderr (Union[str, bytes]) – the stderr generated by the process
  • command (List[Union[str, bytes]]) – the command executed
  • resource_usage (List[SystemUsageInfo]) – the resource use during execution
  • time_elapsed (int) – time used (in seconds)
__module__ = 'fastr.core.target'
__weakref__

list of weak references to the object (if defined)

as_dict()[source]

A dictionary of the data in the object (meant for serialization)

Return type:Dict[str, Union[int, str, List[~T]]]

tool Module

A module to maintain a tool.

Exported classes:

  • Tool – A class encapsulating a tool.
  • ParameterDescription – The base class containing the shared description of a parameter (both input and ouput).
  • InputParameterDescription – A class containing the description of an input parameter.
  • Output ParameterDescription – A class containing the description of an output parameter.
class fastr.core.tool.Tool(doc=None)[source]

Bases: fastr.abc.serializable.Serializable

The class encapsulating a tool.

DEFAULT_TARGET_CLASS = {'MacroNode': 'MacroTarget'}
TOOL_REFERENCE_FILE_NAME = '__fastr_tool_ref__.json'
TOOL_RESULT_FILE_NAME = '__fastr_tool_result.pickle.gz'
__dataschemafile__ = 'Tool.schema.json'
__eq__(other)[source]

Compare two Tool instances with each other.

Parameters:other (Tool) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the Tool

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(doc=None)[source]

Create a new Tool :param doc: path of toolfile or a dict containing the tool data :type doc: str or dict

__module__ = 'fastr.core.tool'
__repr__()[source]

Get a string representation for the Tool. This will show the inputs and output defined in a table-like structure.

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Tool by the given state.

Parameters:state (dict) – The state to populate the object with
__str__()[source]

Get a string version for the Tool

Returns:the string version
Return type:str
authors = None

List of authors of the tool. These people wrapped the executable but are not responsible for executable itself.

cite = None

This holds the citation you should use when publishing something based on this Tool

command = None

Command is a dictionary contain information about the command which is called by this Tool: command[‘interpreter’] holds the (possible) interpreter to use command[‘targets’] holds a per os/arch dictionary of files that should be executed command[‘url’] is the webpage of the command to be called command[‘version’] is the version of the command used command[‘description’] can help a description of the command command[‘authors’] lists the original authors of the command

command_version
static compare_output_data(current_output_data, reference_output_data, validation_result, output)[source]
create_reference(input_data, output_directory, mount_name='__ref_tmp__', copy_input=True)[source]
description = None

Description of the tool and it’s functionality

execute(payload=None, **kwargs)[source]

Execute a Tool given the payload for a single run

Parameters:payload – the data to execute the Tool with
Returns:The result of the execution
Return type:InterFaceResult
fullid

The full id of this tool

hash
help = None

Man page for the Tool. Here usage and examples can be described in detail

id
inputs
name = None

Name of the tool, this should be a descriptive, human readable name.

namespace = None

The namespace this tools lives in, this will be set by the ToolManager on load

node_class = None

Class for of the Node to use

ns_id

The namespace and id of the Tool

outputs
path

The path of the directory in which the tool definition file was located.

references = None

A list of documents and in depth reading about the methods used in this tool

requirements = None

Requirements for this Tool

Warning

Not yet implemented

tags = None

List of tags for this tool

target

The OS and arch matched target definition.

test(reference=None)[source]

Run the tests for this tool

test_spec

alias of TestSpecification

classmethod test_tool(reference_data_dir, tool=None, input_data=None)[source]

Execute the tool with the input data specified and test the results against the refence data. This effectively tests the tool execution.

Parameters:
  • reference_data_dir (str) – The path or vfs url of reference data to compare with
  • source_data (dict) – The source data to use
url = None

URL to website where this tool can be downloaded from

version = None

Version of the tool, not of the underlying software

version Module

Module containing the class that represent versions

class fastr.core.version.Version[source]

Bases: tuple

Class representing a software version definition. Allows for sorting and extraction of parts.

__dict__ = mappingproxy({'__module__': 'fastr.core.version', '__doc__': '\n Class representing a software version definition. Allows for sorting and\n extraction of parts.\n ', 'version_matcher': re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)'), 'date_version_matcher': re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)'), '__new__': <staticmethod object>, '__str__': <function Version.__str__>, '__repr__': <function Version.__repr__>, 'major': <property object>, 'minor': <property object>, 'extra': <property object>, 'extra_string': <property object>, 'status': <property object>, 'build': <property object>, 'suffix': <property object>, '__dict__': <attribute '__dict__' of 'Version' objects>})
__module__ = 'fastr.core.version'
static __new__(cls, *version)[source]

Class containing a version

Can be constructed by:

Version( 'major.$minor.$extra[0].$extra[1]$seperator$status$build$suffix' )
Version( major, minor, extra, status, build, suffix, seperator )
Version( (major, minor, extra, status, build, suffix, seperator) )
Version( [major, minor, extra, status, build, suffix, seperator] )
Parameters:
  • major (int) – interger giving major version
  • minor (int) – is an integer (required)
  • extra (list of int) – is a list of integers
  • status (str) – can be “a”, “alpha”, “b”, “beta”, “rc”, or “r”
  • build (int) – is an integer
  • suffix (str) – can contain any combination of alpha-numeric character and “._-“
  • seperator (str) – is any of “.”, “-“, or “_”, which is located between $extra and $build

Note

The method based on strings is the recommended method. For strings the major and minor version are required, where for tuple and list constructors all seven elements are optional.

Examples:

>>> a = Version('0.1')
>>> print(tuple(a))
(0, 1, None, None, None, '', None)
>>> b = Version('2.5.3-rc2')
>>> print(tuple(b))
(2, 5, [3], 'rc', 2, '', '-')
>>> c = Version('1.2.3.4.5.6.7-beta8_with_suffix')
>>> print(tuple(c))
(1, 2, [3, 4, 5, 6, 7], 'beta', 8, '_with_suffix', '-')
__repr__()[source]

Return a in-editor representation of the version

__str__()[source]

Return a string representation of the version

build

the build number, this is following the status (e.g. for 3.2-beta4, this would be 4)

date_version_matcher = re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)')
extra

extra version extension as a list

extra_string

extra version extension as a string

major

major version

minor

minor version

status

the status of the version (a, alpha, b, beta, rc or r)

suffix

the remainder of the version which was not formatted in a known way

version_matcher = re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)')

vfs Module

This module contains the virtual file system code. This is internally used object as used as base class for the IOPlugin.

class fastr.core.vfs.VirtualFileSystem[source]

Bases: object

The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the vfs:// url scheme.

A typical virtual filesystem url is formatted as vfs://mountpoint/relative/dir/from/mount.ext

Where the mountpoint is defined in the Config file. A list of the currently known mountpoints can be found in the fastr.config object

>>> fastr.config.mounts
{'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',
 'home': '/home/username/',
 'tmp': '/home/username/FastrTemp'}

This shows that a url with the mount home such as vfs://home/tempdir/testfile.txt would be translated into /home/username/tempdir/testfile.txt.

There are a few default mount points defined by Fastr (that can be changed via the config file).

mountpoint default location
home the users home directory (expanduser('~/'))
tmp the fastr temprorary dir, defaults to tempfile.gettempdir()
example_data the fastr example data directory, defaults $FASTRDIR/example/data
__dict__ = mappingproxy({'__module__': 'fastr.core.vfs', '__doc__': "\n The virtual file system class. This is an IOPlugin, but also heavily used\n internally in fastr for working with directories. The VirtualFileSystem\n uses the ``vfs://`` url scheme.\n\n A typical virtual filesystem url is formatted as ``vfs://mountpoint/relative/dir/from/mount.ext``\n\n Where the ``mountpoint`` is defined in the :ref:`config-file`. A list of\n the currently known mountpoints can be found in the ``fastr.config`` object\n\n .. code-block:: python\n\n >>> fastr.config.mounts\n {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',\n 'home': '/home/username/',\n 'tmp': '/home/username/FastrTemp'}\n\n This shows that a url with the mount ``home`` such as\n ``vfs://home/tempdir/testfile.txt`` would be translated into\n ``/home/username/tempdir/testfile.txt``.\n\n There are a few default mount points defined by Fastr (that can be changed\n via the config file).\n\n +--------------+-----------------------------------------------------------------------------+\n | mountpoint | default location |\n +==============+=============================================================================+\n | home | the users home directory (:py:func:`expanduser('~/') <os.path.expanduser>`) |\n +--------------+-----------------------------------------------------------------------------+\n | tmp | the fastr temprorary dir, defaults to ``tempfile.gettempdir()`` |\n +--------------+-----------------------------------------------------------------------------+\n | example_data | the fastr example data directory, defaults ``$FASTRDIR/example/data`` |\n +--------------+-----------------------------------------------------------------------------+\n\n ", '_status': (<PluginState.loaded: '\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m'>, ''), 'abstract': False, '__init__': <function VirtualFileSystem.__init__>, 'scheme': <property object>, 'setup': <function VirtualFileSystem.setup>, 'fetch_url': <function VirtualFileSystem.fetch_url>, 'fetch_value': <function VirtualFileSystem.fetch_value>, 'put_url': <function VirtualFileSystem.put_url>, 'put_value': <function VirtualFileSystem.put_value>, 'expand_url': <function VirtualFileSystem.expand_url>, 'url_to_path': <function VirtualFileSystem.url_to_path>, 'path_to_url': <function VirtualFileSystem.path_to_url>, 'copy_file_dir': <staticmethod object>, '_correct_separators': <staticmethod object>, '__dict__': <attribute '__dict__' of 'VirtualFileSystem' objects>, '__weakref__': <attribute '__weakref__' of 'VirtualFileSystem' objects>})
__init__()[source]

Instantiate the VFS plugin

Returns:the VirtualFileSysten plugin
__module__ = 'fastr.core.vfs'
__weakref__

list of weak references to the object (if defined)

abstract = False
static copy_file_dir(inpath, outpath)[source]

Helper function, copies a file or directory not caring what the inpath actually is

Parameters:
  • inpath – path of the things to be copied
  • outpath – path of the destination
Returns:

the result of shutil.copy2 or shutil.copytree (depending on inpath pointing to a file or directory)

expand_url(url)[source]

Try to expand the url. For vfs with will return the original url.

Parameters:url – url to expand
Returns:the expanded url (same as url)
fetch_url(inurl, outpath)[source]

Fetch the files from the vfs.

Parameters:
  • inurl – url to the item in the data store, starts with vfs://
  • outpath – path where to store the fetch data locally
fetch_value(inurl)[source]

Fetch a value from an external vfs file.

Parameters:inurl – url of the value to read
Returns:the fetched value
path_to_url(path, mountpoint=None, scheme=None)[source]

Construct an url from a given mount point and a relative path to the mount point.

Parameters:path (str) – the path to find the url for
Mountpoint str:mountpoint the url should be under
Returns:url of the
put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters:
  • inpath – path of the local data
  • outurl – url to where to store the data, starts with vfs://
put_value(value, outurl)[source]

Put the value in the external data store.

Parameters:
  • value – value to store
  • outurl – url to where to store the data, starts with vfs://
scheme
setup()[source]

The plugin setup, does nothing but needs to be implemented

url_to_path(url, scheme=None)[source]

Get the path to a file from a vfs url

Parameters:

url (str) – url to get the path for

Returns:

the matching path

Return type:

str

Raises:

Example (the mountpoint tmp points to /tmp):

>>> fastr.vfs.url_to_path('vfs://tmp/file.ext')
'/tmp/file.ext'