core Package¶
core
Package¶
This module contains all of the core components of fastr. It has the classes to create networks and work with them.
cardinality
Module¶
-
class
fastr.core.cardinality.
AnyCardinalitySpec
(parent)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
AsCardinalitySpec
(parent, target)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
Return type: str
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
Parameters: key – Key for which the cardinality is calculated Return type: Optional
[int
]Returns: calculated cardinality
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
Return type: Optional
[int
]
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
Parameters: node – Node for which the cardinality is calculated Return type: Optional
[int
]Returns: calculated cardinality
-
node
¶
-
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
CardinalitySpec
(parent)[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.core.cardinality', '__init__': <function CardinalitySpec.__init__>, '__str__': <function CardinalitySpec.__str__>, '__repr__': <function CardinalitySpec.__repr__>, '__eq__': <function CardinalitySpec.__eq__>, '__ne__': <function CardinalitySpec.__ne__>, 'predefined': <property object>, 'validate': <function CardinalitySpec.validate>, '_validate': <function CardinalitySpec._validate>, 'calculate_planning_cardinality': <function CardinalitySpec.calculate_planning_cardinality>, 'calculate_execution_cardinality': <function CardinalitySpec.calculate_execution_cardinality>, 'calculate_job_cardinality': <function CardinalitySpec.calculate_job_cardinality>, '__dict__': <attribute '__dict__' of 'CardinalitySpec' objects>, '__weakref__': <attribute '__weakref__' of 'CardinalitySpec' objects>, '__doc__': None, '__hash__': None})¶
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
Return type: str
-
__weakref__
¶ list of weak references to the object (if defined)
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
Parameters: key – Key for which the cardinality is calculated Return type: Optional
[int
]Returns: calculated cardinality
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
Return type: Optional
[int
]
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
Parameters: node – Node for which the cardinality is calculated Return type: Optional
[int
]Returns: calculated cardinality
-
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
ChoiceCardinalitySpec
(parent, options)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
IntCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
Return type: str
-
calculate_execution_cardinality
(node)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
Parameters: key – Key for which the cardinality is calculated Return type: int
Returns: calculated cardinality
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
Return type: Optional
[int
]
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
Parameters: node – Node for which the cardinality is calculated Return type: int
Returns: calculated cardinality
-
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
MaxCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
MinCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
RangeCardinalitySpec
(parent, min, max)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
ValueCardinalitySpec
(parent, target)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
Return type: str
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
Parameters: key – Key for which the cardinality is calculated Return type: Optional
[int
]Returns: calculated cardinality
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
Return type: Optional
[int
]
-
node
¶
-
-
fastr.core.cardinality.
create_cardinality
(desc, parent)[source]¶ Create simplified description of the cardinality. This changes the string representation to a tuple that is easier to check at a later time.
Parameters: desc ( str
) – the string version of the cardinalityParent: the parent input or output to which this cardinality spec belongs Return type: CardinalitySpec
Returns: the simplified cardinality description Raises: FastrCardinalityError – if the Input/Output has an incorrect cardinality description. The translation works with the following table:
cardinality string cardinality spec description "*"
,any
``(‘any’,) Any cardinality is allowed "N"
('int', N)
A cardinality of N is required "N-M"
('range', N, M)
A cardinality between N and M is required "*-M"
('max', M)
A cardinality of maximal M is required "N-*"
('min', N)
A cardinality of minimal N is required "[M,N,...,O,P]"
('choice', [M,N,...,O,P])
The cardinality should one of the given options "as:input_id"
('as', 'input_id')
The cardinality should match the cardinality of the given Input "val:input_id"
('val', 'input_id')
The cardinliaty should match the value of the given Input Note
The maximumu, minimum and range are inclusive
dimension
Module¶
-
class
fastr.core.dimension.
Dimension
(name, size)[source]¶ Bases:
object
A class representing a dimension. It contains the name and size of the dimension.
-
__hash__
= None¶
-
__module__
= 'fastr.core.dimension'¶
-
__slots__
= ('_name', '_size')¶
-
name
¶
-
size
¶
-
-
class
fastr.core.dimension.
ForwardsDimensions
[source]¶ Bases:
fastr.core.dimension.HasDimensions
Class of objects that have dimensions not because they contain data with dimensions but forward them (optionally with changes via combine_dimensions)
-
__abstractmethods__
= frozenset({'source', 'combine_dimensions'})¶
-
__module__
= 'fastr.core.dimension'¶
-
combine_dimensions
(dimensions)[source]¶ Method to combine/manipulate the dimensions
Parameters: dimensions – the input dimensions from the source Returns: dimensions manipulated for this object Return type: tuple of dimensions
-
dimensions
¶ The dimensions of the object based on the forwarding
-
source
¶ The source object from which the dimensions are forwarded
Returns: the object from which the dimensions are forwarded Return type: HasDimensions
-
-
class
fastr.core.dimension.
HasDimensions
[source]¶ Bases:
object
A Mixin class for any object that has a notion of dimensions and size. It uses the dimension property to expose the dimension name and size.
-
__abstractmethods__
= frozenset({'dimensions'})¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.dimension', '__doc__': '\n A Mixin class for any object that has a notion of dimensions and size. It\n uses the dimension property to expose the dimension name and size.\n ', 'dimensions': <abc.abstractproperty object>, 'dimnames': <property object>, 'size': <property object>, 'ndims': <property object>, '__dict__': <attribute '__dict__' of 'HasDimensions' objects>, '__weakref__': <attribute '__weakref__' of 'HasDimensions' objects>, '__abstractmethods__': frozenset({'dimensions'}), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.dimension'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
Returns: dimensions Return type: tuple
-
dimnames
¶ A tuple containing the dimension names of this object. All items of the tuple are of type str.
-
ndims
¶ The number of dimensions in this object
-
size
¶ A tuple containing the size of this object. All items of the tuple are of type int or sympy.Symbol.
-
interface
Module¶
- A module that describes the interface of a Tool. It specifies how a set of
- input values will be translated to commands to be executed. This creates a generic interface to different ways of executing underlying software.
-
class
fastr.core.interface.
InputSpec
[source]¶ Bases:
fastr.core.interface.InputSpec
Class containing the information about an Input Specification, this is essentially a data class (but
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Input Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, '__dict__': <attribute '__dict__' of 'InputSpec' objects>})¶
-
__module__
= 'fastr.core.interface'¶
-
-
fastr.core.interface.
InputSpecBase
¶ alias of
fastr.core.interface.InputSpec
-
class
fastr.core.interface.
Interface
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
,fastr.abc.serializable.Serializable
Abstract base class of all Interfaces. Defines the minimal requirements for all Interface implementations.
-
__abstractmethods__
= frozenset({'inputs', '__getstate__', 'expanding', '__setstate__', 'outputs', 'execute'})¶
-
__getstate__
()[source]¶ Retrieve the state of the Interface
Returns: the state of the object Rtype dict:
-
__module__
= 'fastr.core.interface'¶
-
execute
(target, payload)[source]¶ Execute the interface given the a target and payload. The payload should have the form {‘input’: {‘input_id_a’: (value, value), ‘input_id_b’: (value, value)},
‘output’: {‘output_id_a’: (value, value), ‘output_id_b’: (value, value)}}Parameters: - target – the target to call
- payload – the payload to use
Returns: the result of the execution
Return type: (tuple of) InterfaceResult
-
expanding
¶ Indicates whether or not this Interface will result in multiple samples per run. If the flow is unaffected, this will be zero, if it is nonzero it means that number of dimension will be added to the sample array.
-
inputs
¶ OrderedDict of Inputs connected to the Interface. The format should be {input_id: InputSpec}.
-
outputs
¶ OrderedDict of Output connected to the Interface. The format should be {output_id: OutputSpec}.
-
-
class
fastr.core.interface.
InterfaceResult
(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]¶ Bases:
object
The class in which Interfaces should wrap their results to be picked up by fastr
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n The class in which Interfaces should wrap their results to be picked up by fastr\n ', '__init__': <function InterfaceResult.__init__>, '__dict__': <attribute '__dict__' of 'InterfaceResult' objects>, '__weakref__': <attribute '__weakref__' of 'InterfaceResult' objects>})¶
-
__init__
(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
__module__
= 'fastr.core.interface'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
fastr.core.interface.
OutputSpec
[source]¶ Bases:
fastr.core.interface.OutputSpec
Class containing the information about an Output Specification, this is essentially a data class (but
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Output Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, '__dict__': <attribute '__dict__' of 'OutputSpec' objects>})¶
-
__module__
= 'fastr.core.interface'¶
-
-
fastr.core.interface.
OutputSpecBase
¶ alias of
fastr.core.interface.OutputSpec
ioplugin
Module¶
This module contains the manager class for IOPlugins and the base class for all IOPlugins
-
class
fastr.core.ioplugin.
IOPlugin
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
IOPlugins
are used for data import and export for the sources and sinks. The main use of theIOPlugins
is during execution (see Execution). TheIOPlugins
can be accessed viafastr.ioplugins
, but generally there should be no need for direct interaction with these objects. The use of is mainly via the URL used to specify source and sink data.-
__abstractmethods__
= frozenset({'scheme'})¶
-
__module__
= 'fastr.core.ioplugin'¶
-
cleanup
()[source]¶ (abstract)
Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.
-
expand_url
(url)[source]¶ (abstract)
Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.Parameters: url (str) – url to expand Returns: the resulting url(s), a tuple if multiple, otherwise a str Return type: str or tuple of str
-
fetch_url
(inurl, outfile)[source]¶ (abstract)
Fetch a file from an external data source.Parameters: - inurl – url to the item in the data store
- outpath – path where to store the fetch data locally
-
fetch_value
(inurl)[source]¶ (abstract)
Fetch a value from an external data source.Parameters: inurl – the url of the value to retrieve Returns: the fetched value
-
static
isurl
(string)[source]¶ Test if given string is an url.
Parameters: string (str) – string to test Returns: True
if the string is an url,False
otherwiseReturn type: bool
-
path_to_url
(path, mountpoint=None)[source]¶ (abstract)
Construct an url from a given mount point and a relative path to the mount point.Parameters: Returns: url matching the path
Return type:
-
static
print_result
(result)[source]¶ Print the result of the IOPlugin to stdout to be picked up by the tool
Parameters: result – value to print as a result Returns: None
-
pull_source_data
(inurl, outdir, sample_id, datatype=None)[source]¶ Transfer the source data from inurl to be available in outdir.
Parameters: Returns: None
-
push_sink_data
(inpath, outurl, datatype=None)[source]¶ Write out the sink data from the inpath to the outurl.
Parameters: Returns: None
-
put_url
(inpath, outurl)[source]¶ (abstract)
Put the files to the external data store.Parameters: - inpath – path to the local data
- outurl – url to where to store the data in the external data store.
-
put_value
(value, outurl)[source]¶ (abstract)
Put the files to the external data store.Parameters: - value – the value to store
- outurl – url to where to store the data in the external data store.
-
scheme
¶ (abstract)
This abstract property is to be overwritten by a subclass to indicate the url scheme associated with the IOPlugin.
-
provenance
Module¶
-
class
fastr.core.provenance.
Provenance
(host=None)[source]¶ Bases:
object
The Provenance object keeps track of everything that happens to a data object.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.provenance', '__doc__': '\n The Provenance object keeps track of everything that happens to a data object.\n ', '__init__': <function Provenance.__init__>, '_add_namespace': <function Provenance._add_namespace>, 'agent': <function Provenance.agent>, 'activity': <function Provenance.activity>, 'entity': <function Provenance.entity>, 'init_provenance': <function Provenance.init_provenance>, 'collect_provenance': <function Provenance.collect_provenance>, 'collect_input_argument_provenance': <function Provenance.collect_input_argument_provenance>, 'data_uri': <staticmethod object>, 'get_parent_provenance': <staticmethod object>, 'serialize': <function Provenance.serialize>, '__dict__': <attribute '__dict__' of 'Provenance' objects>, '__weakref__': <attribute '__weakref__' of 'Provenance' objects>})¶
-
__module__
= 'fastr.core.provenance'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
static
get_parent_provenance
(value)[source]¶ Find the provenance of the parent job
Parameters: value (str) – url for the value for which to find the job
Returns: the provenance of the job that created the value
Raises: - FastrKeyError – if the deferred is not available (yet)
- FastrValueError – if the value is not a valid deferred url
-
resourcelimit
Module¶
Module for the management of resource limits of compute resources
-
class
fastr.core.resourcelimit.
ResourceLimit
(cores=1, memory='2G', time='01:00:00')[source]¶ Bases:
object
-
__eq__
(other)[source]¶ Check if two resource limits are equal
Parameters: other – resource limit to test against Return type: bool
-
__hash__
= None¶
-
__init__
(cores=1, memory='2G', time='01:00:00')[source]¶ An object describing resource requirements/limits for a node
Parameters: - cores (
Optional
[int
]) – number of cores - memory (
Union
[str
,int
,None
]) – memory specification, can be int with number of megabytes or a string with numbers ending on M, G, T, P for megabytes, gigabytes, terrabytes or petabytes. Note that the number has to be an integer, e.g. 1500M would work, whereas 1.5G would be invalid - time (
Union
[str
,int
,None
]) – run time specification, this can be an int with the number of seconds or a string in the HH:MM:SS, MM:SS, or SS format. Where HH, MM, and SS are integers representing the number of hours, minutes and seconds.
- cores (
-
__module__
= 'fastr.core.resourcelimit'¶
-
__ne__
(other)[source]¶ Check if two resource limits are not equal
Parameters: other – resource limit to test against Return type: bool
-
__slots__
= ('_cores', '_memory', '_time')¶
-
copy
()[source]¶ Return a copy of current resource limit object
Return type: ResourceLimit
-
samples
Module¶
This package holds the classes for working with samples.
-
class
fastr.core.samples.
ContainsSamples
[source]¶ Bases:
fastr.core.samples.HasSamples
-
__abstractmethods__
= frozenset({'samples'})¶
-
__module__
= 'fastr.core.samples'¶
-
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
Returns: dimensions Return type: tuple
-
samples
¶
-
-
class
fastr.core.samples.
HasSamples
[source]¶ Bases:
fastr.core.dimension.HasDimensions
Base class for all classes that supply samples. This base class allows to only define __getitem__ and size and get all other basic functions mixed in so that the object behaves similar to a Mapping.
-
__abstractmethods__
= frozenset({'dimensions', '__getitem__'})¶
-
__module__
= 'fastr.core.samples'¶
-
-
class
fastr.core.samples.
SampleBaseId
[source]¶ Bases:
tuple
This class represents a sample id. A sample id is a multi-dimensional id that has a simple, consistent string representation.
-
__add__
(other)[source]¶ Add another SampleId, this allows to add parts to the SampleId in a convenient way.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample id. A sample id is a multi-dimensional\n id that has a simple, consistent string representation.\n ', '_element_type': <class 'NoneType'>, '__new__': <staticmethod object>, '__repr__': <function SampleBaseId.__repr__>, '__str__': <function SampleBaseId.__str__>, '__add__': <function SampleBaseId.__add__>, '__radd__': <function SampleBaseId.__radd__>, '__dict__': <attribute '__dict__' of 'SampleBaseId' objects>})¶
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, *args)[source]¶ Create a new SampleId
Parameters: args (iterator/iterable of element type or element type) – the strings to make sample id for
-
__radd__
(other)[source]¶ Add another SampleId, this allows to add parts to the SampleId in a convenient way. This is the right-hand version of the operator.
-
-
class
fastr.core.samples.
SampleCollection
(dimnames, parent)[source]¶ Bases:
collections.abc.MutableMapping
,fastr.core.dimension.HasDimensions
The SampleCollections is a class that contains the data including a form of ordering. Each sample is reachable both by its SampleId and a SampleIndex. The object is sparse, so not all SampleId have to be defined allowing for non-rectangular data shapes.
Note
This object is meant to replace both the SampleIdList and the ValueStorage.
-
__abstractmethods__
= frozenset()¶
-
__contains__
(item)[source]¶ Check if an item is in the SampleCollection. The item can be a SampleId or SampleIndex. If the item is a slicing SampleIndex, then check if it would return any data (True) or no data (False)
Parameters: item (SampleId, SampleIndex) – the item to check for Returns: flag indicating item is in the collections Return type: bool
-
__delitem__
(key)[source]¶ Remove an item from the SampleCollection
Parameters: key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to remove
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n The SampleCollections is a class that contains the data including a form\n of ordering. Each sample is reachable both by its SampleId and a\n SampleIndex. The object is sparse, so not all SampleId have to be defined\n allowing for non-rectangular data shapes.\n\n .. note::\n\n This object is meant to replace both the SampleIdList and the\n ValueStorage.\n ', '__init__': <function SampleCollection.__init__>, '__repr__': <function SampleCollection.__repr__>, '__contains__': <function SampleCollection.__contains__>, '__getitem__': <function SampleCollection.__getitem__>, '__setitem__': <function SampleCollection.__setitem__>, '__delitem__': <function SampleCollection.__delitem__>, '__iter__': <function SampleCollection.__iter__>, '__len__': <function SampleCollection.__len__>, 'dimensions': <property object>, 'ndims': <property object>, 'parent': <property object>, 'fullid': <property object>, '__dict__': <attribute '__dict__' of 'SampleCollection' objects>, '__weakref__': <attribute '__weakref__' of 'SampleCollection' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__getitem__
(item)[source]¶ Retrieve (a) SampleItem(s) from the SampleCollection using the SampleId or SampleIndex. If the item is a tuple, it should be valid tuple for constructing either a SampleId or SampleIndex.
Parameters: item (SampleId, SampleIndex, or tuple) – the identifier of the item to retrieve
Returns: the requested item
Return type: Raises: - FastrTypeError – if the item parameter is of incorrect type
- KeyError – if the item is not found
-
__module__
= 'fastr.core.samples'¶
-
__setitem__
(key, value)[source]¶ Set an item to the SampleCollection. The key can be a SampleId, SampleIndex or a tuple containing a SampleId and SampleIndex. The value can be a SampleItem (with the SampleId and SampleIndex matching), a tuple with values (assuming no depending jobs), or a with a list of values and a set of depending jobs.
Parameters: - key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to store
- value (SampleItem, tuple of values, or tuple of tuple of values and set of depending jobs) – the value of the SampleItem to store
Raises: - FastrTypeError – if the key or value types are incorrect
- FastrValueError – if the id or values are incorrectly formed
-
__weakref__
¶ list of weak references to the object (if defined)
-
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
Returns: dimensions Return type: tuple
-
fullid
¶ The full defining ID for the SampleIdList
-
ndims
¶ The number of dimensions in this SampleCollection
-
parent
¶ The parent object holding the SampleCollection
-
-
class
fastr.core.samples.
SampleId
[source]¶ Bases:
fastr.core.samples.SampleBaseId
SampleId is an identifier for data using human readable strings
-
__module__
= 'fastr.core.samples'¶
-
-
class
fastr.core.samples.
SampleIndex
[source]¶ Bases:
fastr.core.samples.SampleBaseId
SampleId is an identifier for data using the location in the N-d data structure.
-
__module__
= 'fastr.core.samples'¶
-
__repr__
()[source]¶ Get a string representation for the SampleIndex
Returns: the string representation Return type: str
-
__str__
()[source]¶ Get a string version for the SampleId, joins the SampleId with __ to create a single string version.
Returns: the string version Return type: str
-
expand
(size)[source]¶ Function expanding a slice SampleIndex into a list of non-slice SampleIndex objects
Parameters: size – the size of the collection to slice
-
isslice
¶ Flag indicating that the SampleIndex is a slice (as opposed to a simple single index).
-
-
class
fastr.core.samples.
SampleItem
[source]¶ Bases:
fastr.core.samples.SampleItemBase
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
Parameters: - index (tuple, slice) – the sample index
- id (
SampleId
) – the sample id - data (SampleValue, Mapping) – the data values
- jobs (set) – set of jobs on which this SampleItems data depends.
- failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
-
class
fastr.core.samples.
SampleItemBase
[source]¶ Bases:
tuple
This class represents a sample item, a combination of a SampleIndex, SampleID, value and required jobs. The SampleItem based on a named tuple and has some extra methods to combine SampleItems easily.
-
__add__
(other)[source]¶ The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.
Parameters: other (SampleItem) – The other item to add to this one Returns: the combined SampleItem Return type: SampleItem
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample item, a combination of a SampleIndex,\n SampleID, value and required jobs. The SampleItem based on a named\n tuple and has some extra methods to combine SampleItems easily.\n ', '__new__': <staticmethod object>, '__repr__': <function SampleItemBase.__repr__>, '__getnewargs__': <function SampleItemBase.__getnewargs__>, '__add__': <function SampleItemBase.__add__>, 'combine': <staticmethod object>, 'index': <property object>, 'id': <property object>, 'data': <property object>, 'jobs': <property object>, 'failed_annotations': <property object>, 'cardinality': <property object>, 'dimensionality': <property object>, '__dict__': <attribute '__dict__' of 'SampleItemBase' objects>})¶
-
__getnewargs__
()[source]¶ Get new args gives the arguments to use to re-create this object, This is used for serialization.
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
Parameters: - index (tuple, slice) – the sample index
- id (
SampleId
) – the sample id - data (SampleValue, Mapping) – the data values
- jobs (set) – set, tuple or list of jobs on which this SampleItems data depends.
- failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
__repr__
()[source]¶ Get a string representation for the SampleItem
Returns: the string representation Return type: str
-
cardinality
¶ The cardinality of this Sample
-
static
combine
(*args)[source]¶ Combine a number of SampleItems into a new one.
Parameters: *args – the SampleItems to combine
Returns: the combined SampleItem Return type: SampleItem It is possible to both give multiple arguments, where each argument is a SampleItem, or a single argument which is an iterable yielding SampleItems.
# variables a, b, c, d are SampleItems to combine # These are all valid ways of combining the SampleItems comb1 = SampleItem.combine(a, b, c, d) # Using multiple arguments l = [a, b, c, d] comb2 = SampleItem.combine(l) # Using a list of arguments comb3 = SampleItem.combine(l.__iter__()) # Using an iterator
-
data
¶ The data SampleValue of the SampleItem
Returns: The value of this SampleItem Return type: SampleValue
-
dimensionality
¶ The dimensionality of this Sample
-
failed_annotations
¶
-
index
¶ The index of the SampleItem
Returns: The index of this SampleItem Return type: SampleIndex
-
-
class
fastr.core.samples.
SamplePayload
[source]¶ Bases:
fastr.core.samples.SampleItemBase
-
__add__
(other)[source]¶ The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.
Parameters: other (SampleItem) – The other item to add to this one Returns: the combined SamplePayload Return type: SamplePayload
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
Parameters: - index (tuple, slice) – the sample index
- id (
SampleId
) – the sample id - data (SampleValue, Mapping) – the data values
- jobs (set) – set of jobs on which this SampleItems data depends.
- failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
-
class
fastr.core.samples.
SampleValue
(*args, **kwargs)[source]¶ Bases:
collections.abc.MutableMapping
A collection containing the content of a sample
-
__abstractmethods__
= frozenset()¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n A collection containing the content of a sample\n ', '_key_type': (<class 'int'>, <class 'str'>), '__init__': <function SampleValue.__init__>, '__repr__': <function SampleValue.__repr__>, '__getitem__': <function SampleValue.__getitem__>, '__setitem__': <function SampleValue.__setitem__>, '__getstate__': <function SampleValue.__getstate__>, '__setstate__': <function SampleValue.__setstate__>, '__delitem__': <function SampleValue.__delitem__>, '__len__': <function SampleValue.__len__>, '__iter__': <function SampleValue.__iter__>, 'is_sequence': <property object>, 'is_mapping': <property object>, 'sequence_part': <function SampleValue.sequence_part>, 'mapping_part': <function SampleValue.mapping_part>, 'cast': <function SampleValue.cast>, 'iterelements': <function SampleValue.iterelements>, '__radd__': <function SampleValue.__radd__>, '__add__': <function SampleValue.__add__>, '__dict__': <attribute '__dict__' of 'SampleValue' objects>, '__weakref__': <attribute '__weakref__' of 'SampleValue' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.samples'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
is_mapping
¶
-
is_sequence
¶
-
target
Module¶
The module containing the classes describing the targets.
-
class
fastr.core.target.
ProcessUsageCollection
[source]¶ Bases:
collections.abc.Sequence
-
__abstractmethods__
= frozenset()¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.target', 'usage_type': <class 'fastr.core.target.SystemUsageInfo'>, '__init__': <function ProcessUsageCollection.__init__>, '__len__': <function ProcessUsageCollection.__len__>, '__getitem__': <function ProcessUsageCollection.__getitem__>, 'append': <function ProcessUsageCollection.append>, 'aggregate': <function ProcessUsageCollection.aggregate>, '__dict__': <attribute '__dict__' of 'ProcessUsageCollection' objects>, '__weakref__': <attribute '__weakref__' of 'ProcessUsageCollection' objects>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.target'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
usage_type
¶ alias of
SystemUsageInfo
-
-
class
fastr.core.target.
SubprocessBasedTarget
[source]¶ Bases:
fastr.core.target.Target
Abstract based class for targets which call the target via a subprocess. Supplies a call_subprocess which executes the command and profiles the resulting subprocess.
-
__abstractmethods__
= frozenset({'run_command'})¶
-
__module__
= 'fastr.core.target'¶
-
call_subprocess
(command)[source]¶ Call a subprocess with logging/timing/profiling
Parameters: command (list) – the command to execute Returns: execution info Return type: dict
-
monitor_process
(process, resources)[source]¶ Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.
Parameters: - process (subproces.Popen) – process to monitor
- resources (ProcessUsageCollection) – list to append measurements to
-
-
class
fastr.core.target.
SystemUsageInfo
(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)¶ Bases:
tuple
-
__getnewargs__
()¶ Return self as a plain tuple. Used by copy and pickle.
-
__module__
= 'fastr.core.target'¶
-
static
__new__
(_cls, timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)¶ Create new instance of SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)
-
__repr__
()¶ Return a nicely formatted representation string
-
__slots__
= ()¶
-
cpu_percent
¶ Alias for field number 1
-
read_bytes
¶ Alias for field number 4
-
rmem
¶ Alias for field number 3
-
timestamp
¶ Alias for field number 0
-
vmem
¶ Alias for field number 2
-
write_bytes
¶ Alias for field number 5
-
-
class
fastr.core.target.
Target
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
The abstract base class for all targets. Execution with a target should follow the following pattern:
>>> with Target() as target: ... target.run_commmand(['sleep', '10'])
The Target context operator will set the correct paths/initialization. Within the context command can be ran and when leaving the context the target reverts the state before.
-
__abstractmethods__
= frozenset({'run_command'})¶
-
__module__
= 'fastr.core.target'¶
-
run_command
(command)[source]¶ Run a command with the target
Return type: TargetResult
-
-
class
fastr.core.target.
TargetResult
(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.core.target', '__init__': <function TargetResult.__init__>, 'as_dict': <function TargetResult.as_dict>, '__dict__': <attribute '__dict__' of 'TargetResult' objects>, '__weakref__': <attribute '__weakref__' of 'TargetResult' objects>, '__doc__': None})¶
-
__init__
(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]¶ Class to formalize the resulting data of a Target
Parameters: - return_code (
int
) – the return code of the process - stdout (
Union
[str
,bytes
]) – the stdout generated by the process - stderr (
Union
[str
,bytes
]) – the stderr generated by the process - command (
List
[Union
[str
,bytes
]]) – the command executed - resource_usage (
List
[SystemUsageInfo
]) – the resource use during execution - time_elapsed (
int
) – time used (in seconds)
- return_code (
-
__module__
= 'fastr.core.target'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
tool
Module¶
A module to maintain a tool.
Exported classes:
- Tool – A class encapsulating a tool.
- ParameterDescription – The base class containing the shared description of a parameter (both input and ouput).
- InputParameterDescription – A class containing the description of an input parameter.
- Output ParameterDescription – A class containing the description of an output parameter.
-
class
fastr.core.tool.
Tool
(doc=None)[source]¶ Bases:
fastr.abc.serializable.Serializable
The class encapsulating a tool.
-
DEFAULT_TARGET_CLASS
= {'MacroNode': 'MacroTarget'}¶
-
TOOL_REFERENCE_FILE_NAME
= '__fastr_tool_ref__.json'¶
-
TOOL_RESULT_FILE_NAME
= '__fastr_tool_result.pickle.gz'¶
-
__dataschemafile__
= 'Tool.schema.json'¶
-
__eq__
(other)[source]¶ Compare two Tool instances with each other.
Parameters: other (Tool) – the other instances to compare to Returns: True if equal, False otherwise
-
__hash__
= None¶
-
__init__
(doc=None)[source]¶ Create a new Tool :param doc: path of toolfile or a dict containing the tool data :type doc: str or dict
-
__module__
= 'fastr.core.tool'¶
-
__repr__
()[source]¶ Get a string representation for the Tool. This will show the inputs and output defined in a table-like structure.
Returns: the string representation Return type: str
-
__setstate__
(state)[source]¶ Set the state of the Tool by the given state.
Parameters: state (dict) – The state to populate the object with
List of authors of the tool. These people wrapped the executable but are not responsible for executable itself.
-
cite
= None¶ This holds the citation you should use when publishing something based on this Tool
-
command
= None¶ Command is a dictionary contain information about the command which is called by this Tool: command[‘interpreter’] holds the (possible) interpreter to use command[‘targets’] holds a per os/arch dictionary of files that should be executed command[‘url’] is the webpage of the command to be called command[‘version’] is the version of the command used command[‘description’] can help a description of the command command[‘authors’] lists the original authors of the command
-
command_version
¶
-
static
compare_output_data
(current_output_data, reference_output_data, validation_result, output)[source]¶
-
description
= None¶ Description of the tool and it’s functionality
-
execute
(payload=None, **kwargs)[source]¶ Execute a Tool given the payload for a single run
Parameters: payload – the data to execute the Tool with Returns: The result of the execution Return type: InterFaceResult
-
fullid
¶ The full id of this tool
-
hash
¶
-
help
= None¶ Man page for the Tool. Here usage and examples can be described in detail
-
id
¶
-
inputs
¶
-
name
= None¶ Name of the tool, this should be a descriptive, human readable name.
-
namespace
= None¶ The namespace this tools lives in, this will be set by the ToolManager on load
-
node_class
= None¶ Class for of the Node to use
-
ns_id
¶ The namespace and id of the Tool
-
outputs
¶
-
path
¶ The path of the directory in which the tool definition file was located.
-
references
= None¶ A list of documents and in depth reading about the methods used in this tool
-
requirements
= None¶ Requirements for this Tool
Warning
Not yet implemented
List of tags for this tool
-
target
¶ The OS and arch matched target definition.
-
test_spec
¶ alias of
TestSpecification
-
classmethod
test_tool
(reference_data_dir, tool=None, input_data=None)[source]¶ Execute the tool with the input data specified and test the results against the refence data. This effectively tests the tool execution.
Parameters:
-
url
= None¶ URL to website where this tool can be downloaded from
-
version
= None¶ Version of the tool, not of the underlying software
-
version
Module¶
Module containing the class that represent versions
-
class
fastr.core.version.
Version
[source]¶ Bases:
tuple
Class representing a software version definition. Allows for sorting and extraction of parts.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.version', '__doc__': '\n Class representing a software version definition. Allows for sorting and\n extraction of parts.\n ', 'version_matcher': re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)'), 'date_version_matcher': re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)'), '__new__': <staticmethod object>, '__str__': <function Version.__str__>, '__repr__': <function Version.__repr__>, 'major': <property object>, 'minor': <property object>, 'extra': <property object>, 'extra_string': <property object>, 'status': <property object>, 'build': <property object>, 'suffix': <property object>, '__dict__': <attribute '__dict__' of 'Version' objects>})¶
-
__module__
= 'fastr.core.version'¶
-
static
__new__
(cls, *version)[source]¶ Class containing a version
Can be constructed by:
Version( 'major.$minor.$extra[0].$extra[1]$seperator$status$build$suffix' ) Version( major, minor, extra, status, build, suffix, seperator ) Version( (major, minor, extra, status, build, suffix, seperator) ) Version( [major, minor, extra, status, build, suffix, seperator] )
Parameters: - major (int) – interger giving major version
- minor (int) – is an integer (required)
- extra (list of int) – is a list of integers
- status (str) – can be “a”, “alpha”, “b”, “beta”, “rc”, or “r”
- build (int) – is an integer
- suffix (str) – can contain any combination of alpha-numeric character and “._-“
- seperator (str) – is any of “.”, “-“, or “_”, which is located between $extra and $build
Note
The method based on strings is the recommended method. For strings the major and minor version are required, where for tuple and list constructors all seven elements are optional.
Examples:
>>> a = Version('0.1') >>> print(tuple(a)) (0, 1, None, None, None, '', None) >>> b = Version('2.5.3-rc2') >>> print(tuple(b)) (2, 5, [3], 'rc', 2, '', '-') >>> c = Version('1.2.3.4.5.6.7-beta8_with_suffix') >>> print(tuple(c)) (1, 2, [3, 4, 5, 6, 7], 'beta', 8, '_with_suffix', '-')
-
build
¶ the build number, this is following the status (e.g. for 3.2-beta4, this would be 4)
-
date_version_matcher
= re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)')¶
-
extra
¶ extra version extension as a list
-
extra_string
¶ extra version extension as a string
-
major
¶ major version
-
minor
¶ minor version
-
status
¶ the status of the version (a, alpha, b, beta, rc or r)
-
suffix
¶ the remainder of the version which was not formatted in a known way
-
version_matcher
= re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)')¶
-
vfs
Module¶
This module contains the virtual file system code. This is internally used object as used as base class for the IOPlugin.
-
class
fastr.core.vfs.
VirtualFileSystem
[source]¶ Bases:
object
The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the
vfs://
url scheme.A typical virtual filesystem url is formatted as
vfs://mountpoint/relative/dir/from/mount.ext
Where the
mountpoint
is defined in the Config file. A list of the currently known mountpoints can be found in thefastr.config
object>>> fastr.config.mounts {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data', 'home': '/home/username/', 'tmp': '/home/username/FastrTemp'}
This shows that a url with the mount
home
such asvfs://home/tempdir/testfile.txt
would be translated into/home/username/tempdir/testfile.txt
.There are a few default mount points defined by Fastr (that can be changed via the config file).
mountpoint default location home the users home directory ( expanduser('~/')
)tmp the fastr temprorary dir, defaults to tempfile.gettempdir()
example_data the fastr example data directory, defaults $FASTRDIR/example/data
-
__dict__
= mappingproxy({'__module__': 'fastr.core.vfs', '__doc__': "\n The virtual file system class. This is an IOPlugin, but also heavily used\n internally in fastr for working with directories. The VirtualFileSystem\n uses the ``vfs://`` url scheme.\n\n A typical virtual filesystem url is formatted as ``vfs://mountpoint/relative/dir/from/mount.ext``\n\n Where the ``mountpoint`` is defined in the :ref:`config-file`. A list of\n the currently known mountpoints can be found in the ``fastr.config`` object\n\n .. code-block:: python\n\n >>> fastr.config.mounts\n {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',\n 'home': '/home/username/',\n 'tmp': '/home/username/FastrTemp'}\n\n This shows that a url with the mount ``home`` such as\n ``vfs://home/tempdir/testfile.txt`` would be translated into\n ``/home/username/tempdir/testfile.txt``.\n\n There are a few default mount points defined by Fastr (that can be changed\n via the config file).\n\n +--------------+-----------------------------------------------------------------------------+\n | mountpoint | default location |\n +==============+=============================================================================+\n | home | the users home directory (:py:func:`expanduser('~/') <os.path.expanduser>`) |\n +--------------+-----------------------------------------------------------------------------+\n | tmp | the fastr temprorary dir, defaults to ``tempfile.gettempdir()`` |\n +--------------+-----------------------------------------------------------------------------+\n | example_data | the fastr example data directory, defaults ``$FASTRDIR/example/data`` |\n +--------------+-----------------------------------------------------------------------------+\n\n ", '_status': (<PluginState.loaded: '\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m'>, ''), 'abstract': False, '__init__': <function VirtualFileSystem.__init__>, 'scheme': <property object>, 'setup': <function VirtualFileSystem.setup>, 'fetch_url': <function VirtualFileSystem.fetch_url>, 'fetch_value': <function VirtualFileSystem.fetch_value>, 'put_url': <function VirtualFileSystem.put_url>, 'put_value': <function VirtualFileSystem.put_value>, 'expand_url': <function VirtualFileSystem.expand_url>, 'url_to_path': <function VirtualFileSystem.url_to_path>, 'path_to_url': <function VirtualFileSystem.path_to_url>, 'copy_file_dir': <staticmethod object>, '_correct_separators': <staticmethod object>, '__dict__': <attribute '__dict__' of 'VirtualFileSystem' objects>, '__weakref__': <attribute '__weakref__' of 'VirtualFileSystem' objects>})¶
-
__module__
= 'fastr.core.vfs'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
abstract
= False¶
-
static
copy_file_dir
(inpath, outpath)[source]¶ Helper function, copies a file or directory not caring what the inpath actually is
Parameters: - inpath – path of the things to be copied
- outpath – path of the destination
Returns: the result of shutil.copy2 or shutil.copytree (depending on inpath pointing to a file or directory)
-
expand_url
(url)[source]¶ Try to expand the url. For vfs with will return the original url.
Parameters: url – url to expand Returns: the expanded url (same as url)
-
fetch_url
(inurl, outpath)[source]¶ Fetch the files from the vfs.
Parameters: - inurl – url to the item in the data store, starts with
vfs://
- outpath – path where to store the fetch data locally
- inurl – url to the item in the data store, starts with
-
fetch_value
(inurl)[source]¶ Fetch a value from an external vfs file.
Parameters: inurl – url of the value to read Returns: the fetched value
-
path_to_url
(path, mountpoint=None, scheme=None)[source]¶ Construct an url from a given mount point and a relative path to the mount point.
Parameters: path (str) – the path to find the url for Mountpoint str: mountpoint the url should be under Returns: url of the
-
put_url
(inpath, outurl)[source]¶ Put the files to the external data store.
Parameters: - inpath – path of the local data
- outurl – url to where to store the data, starts with
vfs://
-
put_value
(value, outurl)[source]¶ Put the value in the external data store.
Parameters: - value – value to store
- outurl – url to where to store the data, starts with
vfs://
-
scheme
¶
-
url_to_path
(url, scheme=None)[source]¶ Get the path to a file from a vfs url
Parameters: url (str) – url to get the path for
Returns: the matching path
Return type: Raises: - FastrMountUnknownError – if the mount in url is unknown
- FastrUnknownURLSchemeError – if the url scheme is not correct
Example (the mountpoint tmp points to /tmp):
>>> fastr.vfs.url_to_path('vfs://tmp/file.ext') '/tmp/file.ext'
-