core Package¶
core
Package¶
This module contains all of the core components of fastr. It has the classes to create networks and work with them.
cardinality
Module¶
-
class
fastr.core.cardinality.
AnyCardinalitySpec
(parent)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
AsCardinalitySpec
(parent, target)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
- Return type
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
-
property
node
¶
-
property
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
CardinalitySpec
(parent)[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.core.cardinality', '__init__': <function CardinalitySpec.__init__>, '__str__': <function CardinalitySpec.__str__>, '__repr__': <function CardinalitySpec.__repr__>, '__eq__': <function CardinalitySpec.__eq__>, '__ne__': <function CardinalitySpec.__ne__>, 'predefined': <property object>, 'validate': <function CardinalitySpec.validate>, '_validate': <function CardinalitySpec._validate>, 'calculate_planning_cardinality': <function CardinalitySpec.calculate_planning_cardinality>, 'calculate_execution_cardinality': <function CardinalitySpec.calculate_execution_cardinality>, 'calculate_job_cardinality': <function CardinalitySpec.calculate_job_cardinality>, '__dict__': <attribute '__dict__' of 'CardinalitySpec' objects>, '__weakref__': <attribute '__weakref__' of 'CardinalitySpec' objects>, '__doc__': None, '__hash__': None})¶
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
abstract
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
- Return type
-
__weakref__
¶ list of weak references to the object (if defined)
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
-
property
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
ChoiceCardinalitySpec
(parent, options)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
IntCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
- Return type
-
calculate_execution_cardinality
(node)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
- Parameters
key – Key for which the cardinality is calculated
- Return type
- Returns
calculated cardinality
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
-
calculate_planning_cardinality
()[source]¶ Calculate the cardinality given the node and spec, for cardinalities that only have validation and not a pre-calculable value, this return None.
- Parameters
node – Node for which the cardinality is calculated
- Return type
- Returns
calculated cardinality
-
property
predefined
¶ Indicate whether the cardinality is predefined or can only be calculated after execution
-
-
class
fastr.core.cardinality.
MaxCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
MinCardinalitySpec
(parent, value)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
RangeCardinalitySpec
(parent, min, max)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
-
class
fastr.core.cardinality.
ValueCardinalitySpec
(parent, target)[source]¶ Bases:
fastr.core.cardinality.CardinalitySpec
-
__hash__
= None¶
-
__module__
= 'fastr.core.cardinality'¶
-
__str__
()[source]¶ String version of the cardinality spec, should be parseable by create_cardinality
- Return type
-
calculate_execution_cardinality
(key=None)[source]¶ Calculate the cardinality given the node and spec, during execution this should be available and not give unknowns once the data is present and the key is given.
-
calculate_job_cardinality
(payload)[source]¶ Calculate the actually cardinality when a job needs to know how many arguments to create for a non-automatic output.
-
property
node
¶
-
-
fastr.core.cardinality.
create_cardinality
(desc, parent)[source]¶ Create simplified description of the cardinality. This changes the string representation to a tuple that is easier to check at a later time.
- Parameters
desc (
str
) – the string version of the cardinality- Parent
the parent input or output to which this cardinality spec belongs
- Return type
- Returns
the simplified cardinality description
- Raises
FastrCardinalityError – if the Input/Output has an incorrect cardinality description.
The translation works with the following table:
cardinality string
cardinality spec
description
"*"
,any
``(‘any’,)
Any cardinality is allowed
"N"
('int', N)
A cardinality of N is required
"N-M"
('range', N, M)
A cardinality between N and M is required
"*-M"
('max', M)
A cardinality of maximal M is required
"N-*"
('min', N)
A cardinality of minimal N is required
"[M,N,...,O,P]"
('choice', [M,N,...,O,P])
The cardinality should one of the given options
"as:input_id"
('as', 'input_id')
The cardinality should match the cardinality of the given Input
"val:input_id"
('val', 'input_id')
The cardinliaty should match the value of the given Input
Note
The maximumu, minimum and range are inclusive
dimension
Module¶
-
class
fastr.core.dimension.
Dimension
(name, size)[source]¶ Bases:
object
A class representing a dimension. It contains the name and size of the dimension.
-
__hash__
= None¶
-
__module__
= 'fastr.core.dimension'¶
-
__slots__
= ('_name', '_size')¶
-
property
name
¶
-
property
size
¶
-
-
class
fastr.core.dimension.
ForwardsDimensions
[source]¶ Bases:
fastr.core.dimension.HasDimensions
Class of objects that have dimensions not because they contain data with dimensions but forward them (optionally with changes via combine_dimensions)
-
__abstractmethods__
= frozenset({'combine_dimensions', 'source'})¶
-
__module__
= 'fastr.core.dimension'¶
-
abstract
combine_dimensions
(dimensions)[source]¶ Method to combine/manipulate the dimensions
- Parameters
dimensions – the input dimensions from the source
- Returns
dimensions manipulated for this object
- Return type
tuple of dimensions
-
property
dimensions
¶ The dimensions of the object based on the forwarding
-
abstract property
source
¶ The source object from which the dimensions are forwarded
- Returns
the object from which the dimensions are forwarded
- Return type
-
-
class
fastr.core.dimension.
HasDimensions
[source]¶ Bases:
object
A Mixin class for any object that has a notion of dimensions and size. It uses the dimension property to expose the dimension name and size.
-
__abstractmethods__
= frozenset({'dimensions'})¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.dimension', '__doc__': '\n A Mixin class for any object that has a notion of dimensions and size. It\n uses the dimension property to expose the dimension name and size.\n ', 'dimensions': <abc.abstractproperty object>, 'dimnames': <property object>, 'size': <property object>, 'ndims': <property object>, '__dict__': <attribute '__dict__' of 'HasDimensions' objects>, '__weakref__': <attribute '__weakref__' of 'HasDimensions' objects>, '__abstractmethods__': frozenset({'dimensions'}), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.dimension'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
abstract property
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
- Returns
dimensions
- Return type
-
property
dimnames
¶ A tuple containing the dimension names of this object. All items of the tuple are of type str.
-
property
ndims
¶ The number of dimensions in this object
-
property
size
¶ A tuple containing the size of this object. All items of the tuple are of type int or sympy.Symbol.
-
interface
Module¶
- A module that describes the interface of a Tool. It specifies how a set of
input values will be translated to commands to be executed. This creates a generic interface to different ways of executing underlying software.
-
class
fastr.core.interface.
InputSpec
[source]¶ Bases:
fastr.core.interface.InputSpec
Class containing the information about an Input Specification, this is essentially a data class (but
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Input Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, 'asdict': <function InputSpec.asdict>, '__dict__': <attribute '__dict__' of 'InputSpec' objects>})¶
-
__module__
= 'fastr.core.interface'¶
-
-
fastr.core.interface.
InputSpecBase
¶ alias of
fastr.core.interface.InputSpec
-
class
fastr.core.interface.
Interface
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
,fastr.abc.serializable.Serializable
Abstract base class of all Interfaces. Defines the minimal requirements for all Interface implementations.
-
__abstractmethods__
= frozenset({'__getstate__', '__setstate__', 'execute', 'expanding', 'inputs', 'outputs'})¶
-
abstract
__getstate__
()[source]¶ Retrieve the state of the Interface
- Returns
the state of the object
- Rtype dict
-
__module__
= 'fastr.core.interface'¶
-
abstract
execute
(target, payload)[source]¶ Execute the interface given the a target and payload. The payload should have the form:
{ 'input': { 'input_id_a': (value, value), 'input_id_b': (value, value) }, 'output': { 'output_id_a': (value, value), 'output_id_b': (value, value) } }
- Parameters
target – the target to call
payload – the payload to use
- Returns
the result of the execution
- Return type
(tuple of) InterfaceResult
-
abstract property
expanding
¶ Indicates whether or not this Interface will result in multiple samples per run. If the flow is unaffected, this will be zero, if it is nonzero it means that number of dimension will be added to the sample array.
-
abstract property
inputs
¶ OrderedDict of Inputs connected to the Interface. The format should be {input_id: InputSpec}.
-
abstract property
outputs
¶ OrderedDict of Output connected to the Interface. The format should be {output_id: OutputSpec}.
-
-
class
fastr.core.interface.
InterfaceResult
(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]¶ Bases:
object
The class in which Interfaces should wrap their results to be picked up by fastr
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n The class in which Interfaces should wrap their results to be picked up by fastr\n ', '__init__': <function InterfaceResult.__init__>, '__dict__': <attribute '__dict__' of 'InterfaceResult' objects>, '__weakref__': <attribute '__weakref__' of 'InterfaceResult' objects>})¶
-
__init__
(result_data, target_result, payload, sample_index=None, sample_id=None, errors=None)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
__module__
= 'fastr.core.interface'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
fastr.core.interface.
OutputSpec
[source]¶ Bases:
fastr.core.interface.OutputSpec
Class containing the information about an Output Specification, this is essentially a data class (but
-
__dict__
= mappingproxy({'__module__': 'fastr.core.interface', '__doc__': '\n Class containing the information about an Output Specification, this is\n essentially a data class (but\n ', '__new__': <staticmethod object>, 'asdict': <function OutputSpec.asdict>, '__dict__': <attribute '__dict__' of 'OutputSpec' objects>})¶
-
__module__
= 'fastr.core.interface'¶
-
-
fastr.core.interface.
OutputSpecBase
¶ alias of
fastr.core.interface.OutputSpec
ioplugin
Module¶
This module contains the manager class for IOPlugins and the base class for all IOPlugins
-
class
fastr.core.ioplugin.
IOPlugin
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
IOPlugins
are used for data import and export for the sources and sinks. The main use of theIOPlugins
is during execution (see Execution). TheIOPlugins
can be accessed viafastr.ioplugins
, but generally there should be no need for direct interaction with these objects. The use of is mainly via the URL used to specify source and sink data.-
__abstractmethods__
= frozenset({'scheme'})¶
-
__module__
= 'fastr.core.ioplugin'¶
-
cleanup
()[source]¶ (abstract)
Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.
-
expand_url
(url)[source]¶ (abstract)
Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.
-
fetch_url
(inurl, outfile)[source]¶ (abstract)
Fetch a file from an external data source.- Parameters
inurl – url to the item in the data store
outpath – path where to store the fetch data locally
-
fetch_value
(inurl)[source]¶ (abstract)
Fetch a value from an external data source.- Parameters
inurl – the url of the value to retrieve
- Returns
the fetched value
-
path_to_url
(path, mountpoint=None)[source]¶ (abstract)
Construct an url from a given mount point and a relative path to the mount point.
-
static
print_result
(result)[source]¶ Print the result of the IOPlugin to stdout to be picked up by the tool
- Parameters
result – value to print as a result
- Returns
None
-
pull_source_data
(inurl, outdir, sample_id, datatype=None)[source]¶ Transfer the source data from inurl to be available in outdir.
-
push_sink_data
(inpath, outurl, datatype=None)[source]¶ Write out the sink data from the inpath to the outurl.
-
put_url
(inpath, outurl)[source]¶ (abstract)
Put the files to the external data store.- Parameters
inpath – path to the local data
outurl – url to where to store the data in the external data store.
-
put_value
(value, outurl)[source]¶ (abstract)
Put the files to the external data store.- Parameters
value – the value to store
outurl – url to where to store the data in the external data store.
-
abstract property
scheme
¶ (abstract)
This abstract property is to be overwritten by a subclass to indicate the url scheme associated with the IOPlugin.
-
provenance
Module¶
-
class
fastr.core.provenance.
Provenance
(host=None)[source]¶ Bases:
object
The Provenance object keeps track of everything that happens to a data object.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.provenance', '__doc__': '\n The Provenance object keeps track of everything that happens to a data object.\n ', '__init__': <function Provenance.__init__>, '_add_namespace': <function Provenance._add_namespace>, 'agent': <function Provenance.agent>, 'activity': <function Provenance.activity>, 'entity': <function Provenance.entity>, 'init_provenance': <function Provenance.init_provenance>, 'collect_provenance': <function Provenance.collect_provenance>, 'collect_input_argument_provenance': <function Provenance.collect_input_argument_provenance>, 'data_uri': <staticmethod object>, 'get_parent_provenance': <staticmethod object>, 'serialize': <function Provenance.serialize>, '__dict__': <attribute '__dict__' of 'Provenance' objects>, '__weakref__': <attribute '__weakref__' of 'Provenance' objects>})¶
-
__module__
= 'fastr.core.provenance'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
static
get_parent_provenance
(value)[source]¶ Find the provenance of the parent job
- Parameters
value (str) – url for the value for which to find the job
- Returns
the provenance of the job that created the value
- Raises
FastrKeyError – if the deferred is not available (yet)
FastrValueError – if the value is not a valid deferred url
-
resourcelimit
Module¶
Module for the management of resource limits of compute resources
-
class
fastr.core.resourcelimit.
ResourceLimit
(cores=1, memory='2G', time=None)[source]¶ Bases:
object
-
__eq__
(other)[source]¶ Check if two resource limits are equal
- Parameters
other – resource limit to test against
- Return type
-
__hash__
= None¶
-
__init__
(cores=1, memory='2G', time=None)[source]¶ An object describing resource requirements/limits for a node
- Parameters
memory (
Union
[str
,int
,None
]) – memory specification, can be int with number of megabytes or a string with numbers ending on M, G, T, P for megabytes, gigabytes, terrabytes or petabytes. Note that the number has to be an integer, e.g. 1500M would work, whereas 1.5G would be invalidtime (
Union
[str
,int
,None
]) – run time specification, this can be an int with the number of seconds or a string in the HH:MM:SS, MM:SS, or SS format. Where HH, MM, and SS are integers representing the number of hours, minutes and seconds.
-
__module__
= 'fastr.core.resourcelimit'¶
-
__ne__
(other)[source]¶ Check if two resource limits are not equal
- Parameters
other – resource limit to test against
- Return type
-
__slots__
= ('_cores', '_memory', '_time')¶
-
samples
Module¶
This package holds the classes for working with samples.
-
class
fastr.core.samples.
ContainsSamples
[source]¶ Bases:
fastr.core.samples.HasSamples
-
__abstractmethods__
= frozenset({'samples'})¶
-
__module__
= 'fastr.core.samples'¶
-
property
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
- Returns
dimensions
- Return type
-
abstract property
samples
¶
-
-
class
fastr.core.samples.
HasSamples
[source]¶ Bases:
fastr.core.dimension.HasDimensions
Base class for all classes that supply samples. This base class allows to only define __getitem__ and size and get all other basic functions mixed in so that the object behaves similar to a Mapping.
-
__abstractmethods__
= frozenset({'__getitem__', 'dimensions'})¶
-
__module__
= 'fastr.core.samples'¶
-
-
class
fastr.core.samples.
SampleBaseId
[source]¶ Bases:
tuple
This class represents a sample id. A sample id is a multi-dimensional id that has a simple, consistent string representation.
-
__add__
(other)[source]¶ Add another SampleId, this allows to add parts to the SampleId in a convenient way.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample id. A sample id is a multi-dimensional\n id that has a simple, consistent string representation.\n ', '_element_type': <class 'NoneType'>, '__new__': <staticmethod object>, '__repr__': <function SampleBaseId.__repr__>, '__str__': <function SampleBaseId.__str__>, '__add__': <function SampleBaseId.__add__>, '__radd__': <function SampleBaseId.__radd__>, '__dict__': <attribute '__dict__' of 'SampleBaseId' objects>})¶
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, *args)[source]¶ Create a new SampleId
- Parameters
args (iterator/iterable of element type or element type) – the strings to make sample id for
-
__radd__
(other)[source]¶ Add another SampleId, this allows to add parts to the SampleId in a convenient way. This is the right-hand version of the operator.
-
-
class
fastr.core.samples.
SampleCollection
(dimnames, parent)[source]¶ Bases:
collections.abc.MutableMapping
,fastr.core.dimension.HasDimensions
The SampleCollections is a class that contains the data including a form of ordering. Each sample is reachable both by its SampleId and a SampleIndex. The object is sparse, so not all SampleId have to be defined allowing for non-rectangular data shapes.
Note
This object is meant to replace both the SampleIdList and the ValueStorage.
-
__abstractmethods__
= frozenset({})¶
-
__contains__
(item)[source]¶ Check if an item is in the SampleCollection. The item can be a SampleId or SampleIndex. If the item is a slicing SampleIndex, then check if it would return any data (True) or no data (False)
- Parameters
item (SampleId, SampleIndex) – the item to check for
- Returns
flag indicating item is in the collections
- Return type
-
__delitem__
(key)[source]¶ Remove an item from the SampleCollection
- Parameters
key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to remove
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n The SampleCollections is a class that contains the data including a form\n of ordering. Each sample is reachable both by its SampleId and a\n SampleIndex. The object is sparse, so not all SampleId have to be defined\n allowing for non-rectangular data shapes.\n\n .. note::\n\n This object is meant to replace both the SampleIdList and the\n ValueStorage.\n ', '__init__': <function SampleCollection.__init__>, '__repr__': <function SampleCollection.__repr__>, '__contains__': <function SampleCollection.__contains__>, '__getitem__': <function SampleCollection.__getitem__>, '__setitem__': <function SampleCollection.__setitem__>, '__delitem__': <function SampleCollection.__delitem__>, '__iter__': <function SampleCollection.__iter__>, '__len__': <function SampleCollection.__len__>, 'dimensions': <property object>, 'ndims': <property object>, 'parent': <property object>, 'fullid': <property object>, '__dict__': <attribute '__dict__' of 'SampleCollection' objects>, '__weakref__': <attribute '__weakref__' of 'SampleCollection' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__getitem__
(item)[source]¶ Retrieve (a) SampleItem(s) from the SampleCollection using the SampleId or SampleIndex. If the item is a tuple, it should be valid tuple for constructing either a SampleId or SampleIndex.
- Parameters
item (SampleId, SampleIndex, or tuple) – the identifier of the item to retrieve
- Returns
the requested item
- Return type
- Raises
FastrTypeError – if the item parameter is of incorrect type
KeyError – if the item is not found
-
__module__
= 'fastr.core.samples'¶
-
__setitem__
(key, value)[source]¶ Set an item to the SampleCollection. The key can be a SampleId, SampleIndex or a tuple containing a SampleId and SampleIndex. The value can be a SampleItem (with the SampleId and SampleIndex matching), a tuple with values (assuming no depending jobs), or a with a list of values and a set of depending jobs.
- Parameters
key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to store
value (SampleItem, tuple of values, or tuple of tuple of values and set of depending jobs) – the value of the SampleItem to store
- Raises
FastrTypeError – if the key or value types are incorrect
FastrValueError – if the id or values are incorrectly formed
-
__weakref__
¶ list of weak references to the object (if defined)
-
property
dimensions
¶ The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.
- Returns
dimensions
- Return type
-
property
fullid
¶ The full defining ID for the SampleIdList
-
property
ndims
¶ The number of dimensions in this SampleCollection
-
property
parent
¶ The parent object holding the SampleCollection
-
-
class
fastr.core.samples.
SampleId
[source]¶ Bases:
fastr.core.samples.SampleBaseId
SampleId is an identifier for data using human readable strings
-
__module__
= 'fastr.core.samples'¶
-
-
class
fastr.core.samples.
SampleIndex
[source]¶ Bases:
fastr.core.samples.SampleBaseId
SampleId is an identifier for data using the location in the N-d data structure.
-
__module__
= 'fastr.core.samples'¶
-
__repr__
()[source]¶ Get a string representation for the SampleIndex
- Returns
the string representation
- Return type
-
__str__
()[source]¶ Get a string version for the SampleId, joins the SampleId with __ to create a single string version.
- Returns
the string version
- Return type
-
expand
(size)[source]¶ Function expanding a slice SampleIndex into a list of non-slice SampleIndex objects
- Parameters
size – the size of the collection to slice
-
property
isslice
¶ Flag indicating that the SampleIndex is a slice (as opposed to a simple single index).
-
-
class
fastr.core.samples.
SampleItem
[source]¶ Bases:
fastr.core.samples.SampleItemBase
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
- Parameters
id (
SampleId
) – the sample iddata (SampleValue, Mapping) – the data values
jobs (set) – set of jobs on which this SampleItems data depends.
failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
-
class
fastr.core.samples.
SampleItemBase
[source]¶ Bases:
tuple
This class represents a sample item, a combination of a SampleIndex, SampleID, value and required jobs. The SampleItem based on a named tuple and has some extra methods to combine SampleItems easily.
-
__add__
(other)[source]¶ The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.
- Parameters
other (SampleItem) – The other item to add to this one
- Returns
the combined SampleItem
- Return type
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n This class represents a sample item, a combination of a SampleIndex,\n SampleID, value and required jobs. The SampleItem based on a named\n tuple and has some extra methods to combine SampleItems easily.\n ', '__new__': <staticmethod object>, '__repr__': <function SampleItemBase.__repr__>, '__getnewargs__': <function SampleItemBase.__getnewargs__>, '__add__': <function SampleItemBase.__add__>, 'combine': <staticmethod object>, 'index': <property object>, 'id': <property object>, 'data': <property object>, 'jobs': <property object>, 'failed_annotations': <property object>, 'cardinality': <property object>, 'dimensionality': <property object>, '__dict__': <attribute '__dict__' of 'SampleItemBase' objects>})¶
-
__getnewargs__
()[source]¶ Get new args gives the arguments to use to re-create this object, This is used for serialization.
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
- Parameters
id (
SampleId
) – the sample iddata (SampleValue, Mapping) – the data values
jobs (set) – set, tuple or list of jobs on which this SampleItems data depends.
failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
__repr__
()[source]¶ Get a string representation for the SampleItem
- Returns
the string representation
- Return type
-
property
cardinality
¶ The cardinality of this Sample
-
static
combine
(*args)[source]¶ Combine a number of SampleItems into a new one.
- Parameters
args (iterable of SampleItems) – the SampleItems to combine
- Returns
the combined SampleItem
- Return type
It is possible to both give multiple arguments, where each argument is a SampleItem, or a single argument which is an iterable yielding SampleItems.
# variables a, b, c, d are SampleItems to combine # These are all valid ways of combining the SampleItems comb1 = SampleItem.combine(a, b, c, d) # Using multiple arguments l = [a, b, c, d] comb2 = SampleItem.combine(l) # Using a list of arguments comb3 = SampleItem.combine(l.__iter__()) # Using an iterator
-
property
data
¶ The data SampleValue of the SampleItem
- Returns
The value of this SampleItem
- Return type
-
property
dimensionality
¶ The dimensionality of this Sample
-
property
failed_annotations
¶
-
property
index
¶ The index of the SampleItem
- Returns
The index of this SampleItem
- Return type
-
-
class
fastr.core.samples.
SamplePayload
[source]¶ Bases:
fastr.core.samples.SampleItemBase
-
__add__
(other)[source]¶ The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.
- Parameters
other (SampleItem) – The other item to add to this one
- Returns
the combined SamplePayload
- Return type
-
__module__
= 'fastr.core.samples'¶
-
static
__new__
(cls, index, id_, data, jobs=None, failed_annotations=None)[source]¶ Create a SampleItem. Data should be an OrderedDict of tuples.
- Parameters
id (
SampleId
) – the sample iddata (SampleValue, Mapping) – the data values
jobs (set) – set of jobs on which this SampleItems data depends.
failed_annotations (set) – set of tuples. The tuple is contructed like follows: (job_id, reason).
-
-
class
fastr.core.samples.
SampleValue
(*args, **kwargs)[source]¶ Bases:
collections.abc.MutableMapping
A collection containing the content of a sample
-
__abstractmethods__
= frozenset({})¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.samples', '__doc__': '\n A collection containing the content of a sample\n ', '_key_type': (<class 'int'>, <class 'str'>), '__init__': <function SampleValue.__init__>, '__repr__': <function SampleValue.__repr__>, '__getitem__': <function SampleValue.__getitem__>, '__setitem__': <function SampleValue.__setitem__>, '__getstate__': <function SampleValue.__getstate__>, '__setstate__': <function SampleValue.__setstate__>, '__delitem__': <function SampleValue.__delitem__>, '__len__': <function SampleValue.__len__>, '__iter__': <function SampleValue.__iter__>, 'is_sequence': <property object>, 'is_mapping': <property object>, 'sequence_part': <function SampleValue.sequence_part>, 'mapping_part': <function SampleValue.mapping_part>, 'cast': <function SampleValue.cast>, 'iterelements': <function SampleValue.iterelements>, '__radd__': <function SampleValue.__radd__>, '__add__': <function SampleValue.__add__>, '__dict__': <attribute '__dict__' of 'SampleValue' objects>, '__weakref__': <attribute '__weakref__' of 'SampleValue' objects>, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.samples'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
property
is_mapping
¶
-
property
is_sequence
¶
-
target
Module¶
The module containing the classes describing the targets.
-
class
fastr.core.target.
ProcessUsageCollection
[source]¶ Bases:
collections.abc.Sequence
-
__abstractmethods__
= frozenset({})¶
-
__dict__
= mappingproxy({'__module__': 'fastr.core.target', 'usage_type': <class 'fastr.core.target.SystemUsageInfo'>, '__init__': <function ProcessUsageCollection.__init__>, '__len__': <function ProcessUsageCollection.__len__>, '__getitem__': <function ProcessUsageCollection.__getitem__>, 'append': <function ProcessUsageCollection.append>, 'aggregate': <function ProcessUsageCollection.aggregate>, '__dict__': <attribute '__dict__' of 'ProcessUsageCollection' objects>, '__weakref__': <attribute '__weakref__' of 'ProcessUsageCollection' objects>, '__doc__': None, '__abstractmethods__': frozenset(), '_abc_registry': <_weakrefset.WeakSet object>, '_abc_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_abc_negative_cache_version': 55})¶
-
__module__
= 'fastr.core.target'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
usage_type
¶ alias of
SystemUsageInfo
-
-
class
fastr.core.target.
SubprocessBasedTarget
[source]¶ Bases:
fastr.core.target.Target
Abstract based class for targets which call the target via a subprocess. Supplies a call_subprocess which executes the command and profiles the resulting subprocess.
-
__abstractmethods__
= frozenset({'run_command'})¶
-
__module__
= 'fastr.core.target'¶
-
monitor_process
(process, resources)[source]¶ Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.
- Parameters
process (subproces.Popen) – process to monitor
resources (ProcessUsageCollection) – list to append measurements to
-
-
class
fastr.core.target.
SystemUsageInfo
(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)¶ Bases:
tuple
-
__getnewargs__
()¶ Return self as a plain tuple. Used by copy and pickle.
-
__module__
= 'fastr.core.target'¶
-
static
__new__
(_cls, timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)¶ Create new instance of SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)
-
__repr__
()¶ Return a nicely formatted representation string
-
__slots__
= ()¶
-
property
cpu_percent
¶ Alias for field number 1
-
property
read_bytes
¶ Alias for field number 4
-
property
rmem
¶ Alias for field number 3
-
property
timestamp
¶ Alias for field number 0
-
property
vmem
¶ Alias for field number 2
-
property
write_bytes
¶ Alias for field number 5
-
-
class
fastr.core.target.
Target
[source]¶ Bases:
fastr.abc.baseplugin.Plugin
The abstract base class for all targets. Execution with a target should follow the following pattern:
>>> with Target() as target: ... target.run_commmand(['sleep', '10'])
The Target context operator will set the correct paths/initialization. Within the context command can be ran and when leaving the context the target reverts the state before.
-
__abstractmethods__
= frozenset({'run_command'})¶
-
__module__
= 'fastr.core.target'¶
-
-
class
fastr.core.target.
TargetResult
(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.core.target', '__init__': <function TargetResult.__init__>, 'as_dict': <function TargetResult.as_dict>, '__dict__': <attribute '__dict__' of 'TargetResult' objects>, '__weakref__': <attribute '__weakref__' of 'TargetResult' objects>, '__doc__': None})¶
-
__init__
(return_code, stdout, stderr, command, resource_usage, time_elapsed)[source]¶ Class to formalize the resulting data of a Target
- Parameters
-
__module__
= 'fastr.core.target'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
tool
Module¶
A module to maintain a tool.
Exported classes:
Tool – A class encapsulating a tool.
ParameterDescription – The base class containing the shared description of a parameter (both input and ouput).
InputParameterDescription – A class containing the description of an input parameter.
Output ParameterDescription – A class containing the description of an output parameter.
-
class
fastr.core.tool.
Tool
(doc=None)[source]¶ Bases:
fastr.abc.serializable.Serializable
The class encapsulating a tool.
-
DEFAULT_TARGET_CLASS
= {'MacroNode': 'MacroTarget'}¶
-
TOOL_REFERENCE_FILE_NAME
= '__fastr_tool_ref__.json'¶
-
TOOL_RESULT_FILE_NAME
= '__fastr_tool_result.pickle.gz'¶
-
__dataschemafile__
= 'Tool.schema.json'¶
-
__eq__
(other)[source]¶ Compare two Tool instances with each other.
- Parameters
other (Tool) – the other instances to compare to
- Returns
True if equal, False otherwise
-
__hash__
= None¶
-
__init__
(doc=None)[source]¶ Create a new Tool :param doc: path of toolfile or a dict containing the tool data :type doc: str or dict
-
__module__
= 'fastr.core.tool'¶
-
__repr__
()[source]¶ Get a string representation for the Tool. This will show the inputs and output defined in a table-like structure.
- Returns
the string representation
- Return type
-
__setstate__
(state)[source]¶ Set the state of the Tool by the given state.
- Parameters
state (dict) – The state to populate the object with
List of authors of the tool. These people wrapped the executable but are not responsible for executable itself.
-
cite
= None¶ This holds the citation you should use when publishing something based on this Tool
-
command
= None¶ Command is a dictionary contain information about the command which is called by this Tool: command[‘interpreter’] holds the (possible) interpreter to use command[‘targets’] holds a per os/arch dictionary of files that should be executed command[‘url’] is the webpage of the command to be called command[‘version’] is the version of the command used command[‘description’] can help a description of the command command[‘authors’] lists the original authors of the command
-
property
command_version
¶
-
static
compare_output_data
(current_output_data, reference_output_data, validation_result, output)[source]¶
-
description
= None¶ Description of the tool and it’s functionality
-
execute
(payload=None, **kwargs)[source]¶ Execute a Tool given the payload for a single run
- Parameters
payload – the data to execute the Tool with
- Returns
The result of the execution
- Return type
InterFaceResult
-
property
fullid
¶ The full id of this tool
-
property
hash
¶
-
help
= None¶ Man page for the Tool. Here usage and examples can be described in detail
-
property
id
¶
-
property
inputs
¶
-
name
= None¶ Name of the tool, this should be a descriptive, human readable name.
-
namespace
= None¶ The namespace this tools lives in, this will be set by the ToolManager on load
-
node_class
= None¶ Class for of the Node to use
-
property
ns_id
¶ The namespace and id of the Tool
-
property
outputs
¶
-
property
path
¶ The path of the directory in which the tool definition file was located.
-
references
= None¶ A list of documents and in depth reading about the methods used in this tool
-
requirements
= None¶ Requirements for this Tool
Warning
Not yet implemented
List of tags for this tool
-
property
target
¶ The OS and arch matched target definition.
-
test_spec
¶ alias of
TestSpecification
-
classmethod
test_tool
(reference_data_dir, tool=None, input_data=None)[source]¶ Execute the tool with the input data specified and test the results against the refence data. This effectively tests the tool execution.
-
url
= None¶ URL to website where this tool can be downloaded from
-
version
= None¶ Version of the tool, not of the underlying software
-
version
Module¶
Module containing the class that represent versions
-
class
fastr.core.version.
Version
[source]¶ Bases:
tuple
Class representing a software version definition. Allows for sorting and extraction of parts.
-
__dict__
= mappingproxy({'__module__': 'fastr.core.version', '__doc__': '\n Class representing a software version definition. Allows for sorting and\n extraction of parts.\n ', 'version_matcher': re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)'), 'date_version_matcher': re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)'), '__new__': <staticmethod object>, '__str__': <function Version.__str__>, '__repr__': <function Version.__repr__>, 'major': <property object>, 'minor': <property object>, 'extra': <property object>, 'extra_string': <property object>, 'status': <property object>, 'build': <property object>, 'suffix': <property object>, '__dict__': <attribute '__dict__' of 'Version' objects>})¶
-
__module__
= 'fastr.core.version'¶
-
static
__new__
(cls, *version)[source]¶ Class containing a version
Can be constructed by:
Version( 'major.$minor.$extra[0].$extra[1]$seperator$status$build$suffix' ) Version( major, minor, extra, status, build, suffix, seperator ) Version( (major, minor, extra, status, build, suffix, seperator) ) Version( [major, minor, extra, status, build, suffix, seperator] )
- Parameters
major (int) – interger giving major version
minor (int) – is an integer (required)
extra (list of int) – is a list of integers
status (str) – can be “a”, “alpha”, “b”, “beta”, “rc”, or “r”
build (int) – is an integer
suffix (str) – can contain any combination of alpha-numeric character and “._-“
seperator (str) – is any of “.”, “-“, or “_”, which is located between $extra and $build
Note
The method based on strings is the recommended method. For strings the major and minor version are required, where for tuple and list constructors all seven elements are optional.
Examples:
>>> a = Version('0.1') >>> print(tuple(a)) (0, 1, None, None, None, '', None) >>> b = Version('2.5.3-rc2') >>> print(tuple(b)) (2, 5, [3], 'rc', 2, '', '-') >>> c = Version('1.2.3.4.5.6.7-beta8_with_suffix') >>> print(tuple(c)) (1, 2, [3, 4, 5, 6, 7], 'beta', 8, '_with_suffix', '-')
-
property
build
¶ the build number, this is following the status (e.g. for 3.2-beta4, this would be 4)
-
date_version_matcher
= re.compile('(\\d+)-(\\d+)-(\\d+)([_\\-\\.])?(.*)')¶
-
property
extra
¶ extra version extension as a list
-
property
extra_string
¶ extra version extension as a string
-
property
major
¶ major version
-
property
minor
¶ minor version
-
property
status
¶ the status of the version (a, alpha, b, beta, rc or r)
-
property
suffix
¶ the remainder of the version which was not formatted in a known way
-
version_matcher
= re.compile('(\\d+)\\.(\\d+)((?:\\.\\d+)+)?([_\\-\\.])?(a(?=\\d)|b(?=\\d)|alpha(?=\\d)|beta(?=\\d)|rc(?=\\d)|r(?=\\d))?(\\d+)?([a-zA-Z0-9\\-_\\.]*)')¶
-
vfs
Module¶
This module contains the virtual file system code. This is internally used object as used as base class for the IOPlugin.
-
class
fastr.core.vfs.
VirtualFileSystem
[source]¶ Bases:
object
The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the
vfs://
url scheme.A typical virtual filesystem url is formatted as
vfs://mountpoint/relative/dir/from/mount.ext
Where the
mountpoint
is defined in the Config file. A list of the currently known mountpoints can be found in thefastr.config
object>>> fastr.config.mounts {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data', 'home': '/home/username/', 'tmp': '/home/username/FastrTemp'}
This shows that a url with the mount
home
such asvfs://home/tempdir/testfile.txt
would be translated into/home/username/tempdir/testfile.txt
.There are a few default mount points defined by Fastr (that can be changed via the config file).
mountpoint
default location
home
the users home directory (
expanduser('~/')
)tmp
the fastr temprorary dir, defaults to
tempfile.gettempdir()
example_data
the fastr example data directory, defaults
$FASTRDIR/example/data
-
__dict__
= mappingproxy({'__module__': 'fastr.core.vfs', '__doc__': "\n The virtual file system class. This is an IOPlugin, but also heavily used\n internally in fastr for working with directories. The VirtualFileSystem\n uses the ``vfs://`` url scheme.\n\n A typical virtual filesystem url is formatted as ``vfs://mountpoint/relative/dir/from/mount.ext``\n\n Where the ``mountpoint`` is defined in the :ref:`config-file`. A list of\n the currently known mountpoints can be found in the ``fastr.config`` object\n\n .. code-block:: python\n\n >>> fastr.config.mounts\n {'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',\n 'home': '/home/username/',\n 'tmp': '/home/username/FastrTemp'}\n\n This shows that a url with the mount ``home`` such as\n ``vfs://home/tempdir/testfile.txt`` would be translated into\n ``/home/username/tempdir/testfile.txt``.\n\n There are a few default mount points defined by Fastr (that can be changed\n via the config file).\n\n +--------------+-----------------------------------------------------------------------------+\n | mountpoint | default location |\n +==============+=============================================================================+\n | home | the users home directory (:py:func:`expanduser('~/') <os.path.expanduser>`) |\n +--------------+-----------------------------------------------------------------------------+\n | tmp | the fastr temprorary dir, defaults to ``tempfile.gettempdir()`` |\n +--------------+-----------------------------------------------------------------------------+\n | example_data | the fastr example data directory, defaults ``$FASTRDIR/example/data`` |\n +--------------+-----------------------------------------------------------------------------+\n\n ", '_status': (<PluginState.loaded: '\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m'>, ''), 'abstract': False, '__init__': <function VirtualFileSystem.__init__>, 'scheme': <property object>, 'setup': <function VirtualFileSystem.setup>, 'fetch_url': <function VirtualFileSystem.fetch_url>, 'fetch_value': <function VirtualFileSystem.fetch_value>, 'put_url': <function VirtualFileSystem.put_url>, 'put_value': <function VirtualFileSystem.put_value>, 'expand_url': <function VirtualFileSystem.expand_url>, 'url_to_path': <function VirtualFileSystem.url_to_path>, 'path_to_url': <function VirtualFileSystem.path_to_url>, 'copy_file_dir': <staticmethod object>, '_correct_separators': <staticmethod object>, '__dict__': <attribute '__dict__' of 'VirtualFileSystem' objects>, '__weakref__': <attribute '__weakref__' of 'VirtualFileSystem' objects>})¶
-
__module__
= 'fastr.core.vfs'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
abstract
= False¶
-
static
copy_file_dir
(inpath, outpath)[source]¶ Helper function, copies a file or directory not caring what the inpath actually is
- Parameters
inpath – path of the things to be copied
outpath – path of the destination
- Returns
the result of shutil.copy2 or shutil.copytree (depending on inpath pointing to a file or directory)
-
expand_url
(url)[source]¶ Try to expand the url. For vfs with will return the original url.
- Parameters
url – url to expand
- Returns
the expanded url (same as url)
-
fetch_url
(inurl, outpath)[source]¶ Fetch the files from the vfs.
- Parameters
inurl – url to the item in the data store, starts with
vfs://
outpath – path where to store the fetch data locally
-
fetch_value
(inurl)[source]¶ Fetch a value from an external vfs file.
- Parameters
inurl – url of the value to read
- Returns
the fetched value
-
path_to_url
(path, mountpoint=None, scheme=None)[source]¶ Construct an url from a given mount point and a relative path to the mount point.
- Parameters
path (str) – the path to find the url for
- Mountpoint str
mountpoint the url should be under
- Returns
url of the
-
put_url
(inpath, outurl)[source]¶ Put the files to the external data store.
- Parameters
inpath – path of the local data
outurl – url to where to store the data, starts with
vfs://
-
put_value
(value, outurl)[source]¶ Put the value in the external data store.
- Parameters
value – value to store
outurl – url to where to store the data, starts with
vfs://
-
property
scheme
¶
-
url_to_path
(url, scheme=None)[source]¶ Get the path to a file from a vfs url
- Parameters
url (str) – url to get the path for
- Returns
the matching path
- Return type
- Raises
FastrMountUnknownError – if the mount in url is unknown
FastrUnknownURLSchemeError – if the url scheme is not correct
Example (the mountpoint tmp points to /tmp):
>>> fastr.vfs.url_to_path('vfs://tmp/file.ext') '/tmp/file.ext'
-