core Package

core Package

This module contains all of the core components of fastr. It has the classes to create networks and work with them.

basemanager Module

This module contains the core class for all managers

class fastr.core.basemanager.BaseManager(path=None, recursive=False)[source]

Bases: _abcoll.MutableMapping

Baseclass for a Manager, subclasses needs to override the following methods:
BaseManager._item_extension, BaseManager._load_item()
_item_extension()

Abstract property that sets the extension of the files to be loaded by the BaseManager. When scanning for items, only files with this extension will be loaded.

Returns:desired extension
Return type:str
Raises:FastrNotImplementedError – if property not reimplemented in subclass
_load_item(filepath, namespace)[source]

Abstract method to load an item of the BaseManager. This function is not implemented and needs to be reimplemented by a subclass.

Parameters:
  • filepath (str) – path of the item to load
  • namespace (str) – the namespace of the item to be loaded
Returns:

the loaded item

Raises:

FastrNotImplementedError – if called without being reimplemented by a subclass

__abstractmethods__ = frozenset(['_load_item', '_item_extension'])
__delitem__(key)[source]

Remove item from the BaseManager

Parameters:key – key of the item to remove
Returns:None
Raises:FastrKeyError – if the key is not found in the BaseManager
__getitem__(key)[source]

Retrieve item from BaseManager

Parameters:key – the key of the item to retrieve
Returns:the value indicated by the key
Raises:FastrKeyError – if the key is not found in the BaseManager
__init__(path=None, recursive=False)[source]

The BaseManager constructor

Parameters:
  • path (str or None) – path to scan for items, or None for no path
  • recursive (bool) – Flag to indicate a recursive search is desired
Returns:

the newly created BaseManager

Return type:

BaseManager

__iter__()[source]

Get an iterator from the BaseManager. The iterator will iterate over the keys of the BaseManager.

Returns:the iterator
Return type:dictionary-keyiterator
__keytransform__(key)[source]

Identity transform for the keys. This function can be reimplemented by a subclass to implement a different key transform.

Parameters:key – key to transform
Returns:the transformed key (in this case the same key as inputted)
__len__()[source]

Return the number of items in the BaseManager

Returns:number of items in the BaseManager
Return type:int
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.basemanager'
__repr__()[source]

Convert the BaseManager to a representation string.

Returns:Representation string
Return type:str
__setitem__(key, value)[source]

Set item in the BaseManager

Parameters:
  • key – the key of the item to store
  • value – the value of the item to store
Returns:

None

data

The actual data dict underlying this Manager

match_filename(filename)[source]

Check if the filename matches the pattern the manager expects.

Parameters:filename – filename to match
Returns:flag indicating that the filename matches
populate()[source]

Populate the manager with the data. This is a method that will be called when the Managers data is first accessed. This way we avoid doing expensive directory scans when the data is never requested.

reload()[source]

Reload entire contents of this manager.

baseplugin Module

The base class for all Plugins in the fastr system

class fastr.core.baseplugin.BasePlugin[source]

Bases: object

Base class for Plugins in the fastr system.

__abstractmethods__ = frozenset([])
__dict__ = dict_proxy({'status': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22dff090>, '__module__': 'fastr.core.baseplugin', '__abstractmethods__': frozenset([]), '__metaclass__': <class 'fastr.core.baseplugin.PluginMeta'>, '_abc_negative_cache': <_weakrefset.WeakSet object at 0x7f3a22dff1d0>, '__str__': <function __str__ at 0x7f3a22dfbc80>, '_instantiate': False, 'module': None, '__dict__': <attribute '__dict__' of 'BasePlugin' objects>, 'fullid': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22df8fd0>, '_source_code': None, 'source_code': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22dff050>, '__weakref__': <attribute '__weakref__' of 'BasePlugin' objects>, 'id': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22df8e10>, '__init__': <function __init__ at 0x7f3a22dfbc08>, '_abc_cache': <_weakrefset.WeakSet object at 0x7f3a22dff190>, 'register_configuration': <classmethod object at 0x7f3a22e5a638>, 'status_message': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22dff110>, 'instantiate': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22dff0d0>, 'configuration_fields': <fastr.utils.classproperty.ClassPropertyDescriptor object at 0x7f3a22df8f90>, '_abc_registry': <_weakrefset.WeakSet object at 0x7f3a22dff150>, 'cleanup': <function cleanup at 0x7f3a22e002a8>, '_abc_negative_cache_version': 30, '__repr__': <function __repr__ at 0x7f3a22dfbcf8>, 'set_status': <classmethod object at 0x7f3a22e5a788>, 'test': <classmethod object at 0x7f3a22e5a7f8>, '_status': (<PluginState.uninitialized: '\x1b[46mUnInitialized\x1b[0m'>, 'Plugin object created', None), 'set_code': <classmethod object at 0x7f3a22e5a7c0>, '__doc__': '\n Base class for Plugins in the fastr system.\n '})
__init__()[source]

The BasePlugin constructor.

Returns:the created plugin
Return type:BasePlugin
Raises:FastrPluginNotLoaded – if the plugin did not load correctly
__metaclass__

alias of PluginMeta

__module__ = 'fastr.core.baseplugin'
__repr__()[source]
__str__()[source]

Creare string representation of the plugin.

Returns:string represenation
Return type:str
__weakref__

list of weak references to the object (if defined)

cleanup()[source]

Perform any cleanup action needed when the plugin use ended. This can be closing files/streams etc.

configuration_fields = {}
fullid = 'fastr://plugins/BasePlugin'
id = 'BasePlugin'
instantiate = False
module = None
classmethod register_configuration()[source]

Register and test the configuation fields of the plugin

classmethod set_code(source_code)[source]

Set the filename and source code of the plugin

Parameters:source_code (str) – the source code of the plugin
classmethod set_status(status, message, exception=None)[source]

Update the status of the plugin

Parameters:
  • status (str) – the new status
  • message (str) – message explaining the status change
  • exception (str) – stacktrace of the exception causing the failed load
status = <PluginState.uninitialized: '\x1b[46mUnInitialized\x1b[0m'>[source]
status_message = 'Plugin object created'
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.core.baseplugin.Plugin[source]

Bases: fastr.core.baseplugin.BasePlugin

__abstractmethods__ = frozenset([])
__module__ = 'fastr.core.baseplugin'
class fastr.core.baseplugin.PluginMeta[source]

Bases: abc.ABCMeta

__module__ = 'fastr.core.baseplugin'
__repr__()[source]
class fastr.core.baseplugin.PluginState[source]

Bases: enum.Enum

__format__(format_spec)
__module__ = 'fastr.core.baseplugin'
static __new__(value)
__reduce_ex__(proto)
__repr__()
__str__()
failed = <PluginState.failed: '\x1b[37m\x1b[41m\x1b[1mFailed\x1b[0m'>
loaded = <PluginState.loaded: '\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m'>
preload = <PluginState.preload: '\x1b[102mPreLoad\x1b[0m'>
uninitialized = <PluginState.uninitialized: '\x1b[46mUnInitialized\x1b[0m'>
unloaded = <PluginState.unloaded: '\x1b[46mUnLoaded\x1b[0m'>

datatypemanager Module

This module manages datatypes. These datatypes are python classes generated from the XML/JSON datatype files.

class fastr.core.datatypemanager.DataTypeManager[source]

Bases: fastr.core.pluginmanager.BasePluginManager

The DataTypeManager hold a mapping of all DataTypes in the fast system and can create new DataTypes from files/data structures.

__abstractmethods__ = frozenset([])
__init__()[source]

The DataTypeManager constructor will create a new DataTypeManager and populate it with all DataTypes it can find in the paths set in fastr.config.types_path.

Returns:the created DataTypeManager
__keytransform__(key)[source]

Key transformation for this mapping. The key transformation allows indexing by both the DataType name as well as the DataType it self.

Parameters:key (fastr.datatypes.BaseDataType or str) – The name of the requested datatype or the datatype itself
Returns:The requested datatype
__module__ = 'fastr.core.datatypemanager'
create_enumtype(type_id, options, name=None)[source]

Create a python class based on an XML file. This function return a completely functional python class based on the contents of a DataType XML file.

Such a class will be of type EnumType.

Parameters:
  • type_id (str) – the id of the new class
  • options (iterable) – an iterable of options, each option should be str
Returns:

the newly created subclass of EnumType

Raises:

FastrTypeError – if the options is not an iterable of str

fullid

The fullid of the datatype manager

get_type(name)[source]

Read a type given a typename. This will scan all directories in types_path and attempt to load the newest version of the DataType.

Parameters:name (str) – Name of the datatype that should be imported in the system
Returns:the datatype with the requested name, or None if datatype is not found

Note

If type is already in TypeManager it will not load anything and return the already loaded version.

guess_type(value, exists=True, options=None, preferred=None)[source]

Guess the DataType based on a value str.

Parameters:
  • value (str) – the value to guess the type for
  • options (TypeGroup, DataType or tuple of DataTypes) – The options that are allowed to be guessed from
  • extists (bool) – Indicate the value exists (if file) and can be checked for validity, if false skip validity check
  • preferred (iterable) – An iterable of preferred types in case multiple types match.
Returns:

The resulting DataType or None if no match was found

Raises:

FastrTypeError – if the options argument is of the wrong type

The function will first create a list of all candidate DataTypes. Subsequently, it will check for each candidate if the value would valid. If there are multiple matches, the config value for preferred types is consulted to break the ties. If non of the DataTypes are in the preferred types list, a somewhat random DataType will be picked as the most optimal result.

has_type(name)[source]

Check if the datatype with requested name exists

Parameters:name (str) – the name of the requested datatype
Returns:flag indicating if the datatype exists
Return type:bool
static isdatatype(item)[source]

Check if item is a valid datatype for the fastr system.

Parameters:item – item to check
Returns:flag indicating if the item is a fastr datatype
Return type:bool
match_types(*args, **kwargs)[source]

Find the match between a list of DataTypes/TypeGroups, see resolve-datatype for details

Parameters:
  • args – A list of DataType/TypeGroup objects to match
  • kwargs – A ‘preferred’ keyword argument can be used to indicate a list of DataTypes to prefer in case of ties (first has precedence over later in list)
Returns:

The best DataType match, or None if no match is possible.

Raises:

FastrTypeError – if not all args are subclasses of BaseDataType

match_types_any(*args)[source]

Find the match between a list of DataTypes/TypeGroups, see resolve-datatype for details

Parameters:args – A list of DataType/TypeGroup objects to match
Returns:A tuple with all DataTypes that match.
Return type:set
Raises:FastrTypeError – if not all args are subclasses of BaseDataType
plugin_class

The PluginClass of the items of the BasePluginManager

poll_datatype(filename)[source]

Poll an xml file to see if there is a definition of a datatype in it.

Parameters:filename (str) – path of the file to poll
Returns:tuple with (id, version, basetype) if a datatype is found or (None, None, None) if no datatype is found
populate()[source]

Populate Manager. After scanning for DataTypes, create the AnyType and set the preferred types

dimension Module

class fastr.core.dimension.Dimension(name, size)[source]

Bases: object

A class representing a dimension. It contains the name and size of the dimension.

__dict__ = dict_proxy({'__dict__': <attribute '__dict__' of 'Dimension' objects>, '__module__': 'fastr.core.dimension', '__weakref__': <attribute '__weakref__' of 'Dimension' objects>, '__doc__': '\n A class representing a dimension. It contains the name and size of the\n dimension.\n ', '__init__': <function __init__ at 0x7f3a104a1aa0>})
__init__(name, size)[source]

The constructor for the dimension.

Parameters:
  • name (str) – Name of the dimension
  • size (int or sympy.Symbol) – Size fo the dimension
__module__ = 'fastr.core.dimension'
__weakref__

list of weak references to the object (if defined)

class fastr.core.dimension.HasDimensions[source]

Bases: object

A Mixin class for any object that has a notion of dimensions and size. It uses the dimension property to expose the dimension name and size.

__abstractmethods__ = frozenset(['dimensions'])
__dict__ = dict_proxy({'__module__': 'fastr.core.dimension', '__metaclass__': <class 'abc.ABCMeta'>, '_abc_negative_cache': <_weakrefset.WeakSet object at 0x7f3a1015ef90>, '__dict__': <attribute '__dict__' of 'HasDimensions' objects>, '__weakref__': <attribute '__weakref__' of 'HasDimensions' objects>, 'dimnames': <property object at 0x7f3a0fdf1ec0>, 'size': <property object at 0x7f3a0fdf1e68>, '_abc_cache': <_weakrefset.WeakSet object at 0x7f3a1015ea50>, 'dimensions': <abc.abstractproperty object at 0x7f3a10c6def0>, '__abstractmethods__': frozenset(['dimensions']), '_abc_negative_cache_version': 30, '_abc_registry': <_weakrefset.WeakSet object at 0x7f3a101af3d0>, '__doc__': '\n A Mixin class for any object that has a notion of dimensions and size. It\n uses the dimension property to expose the dimension name and size.\n '})
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.dimension'
__weakref__

list of weak references to the object (if defined)

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
dimnames

A tuple containing the dimension names of this object. All items of the tuple are of type str.

size

A tuple containing the size of this object. All items of the tuple are of type int or sympy.Symbol.

inputoutput Module

Classes for arranging the input and output for nodes.

Exported classes:

Input – An input for a node (holding datatype). Output – The output of a node (holding datatype and value). ConstantOutput – The output of a node (holding datatype and value).

Warning

Don’t mess with the Link, Input and Output internals from other places. There will be a huge chances of breaking the network functionality!

class fastr.core.inputoutput.AdvancedFlowOutput(node, description)[source]

Bases: fastr.core.inputoutput.Output

__abstractmethods__ = frozenset([])
__module__ = 'fastr.core.inputoutput'
dimnames

The dimnames of AdvancedFlowNodes have the output id appended, as the sizes per output can be different.

class fastr.core.inputoutput.BaseInput(node, description)[source]

Bases: fastr.core.inputoutput.BaseInputOutput

Base class for all inputs.

__abstractmethods__ = frozenset(['_update', 'fullid', '__getitem__', 'num_subinput', 'itersubinputs', 'size'])
__init__(node, description)[source]

Instantiate a BaseInput

Parameters:
  • node – the parent node the input/output belongs to.
  • description – the ParameterDescription describing the input/output.
Returns:

the created BaseInput

Raises:
  • FastrTypeError – if description is not of class ParameterDescription
  • FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the fastr.typelist
__module__ = 'fastr.core.inputoutput'
itersubinputs()[source]

Iterator over the SubInputs

Returns:iterator

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
num_subinput

The number of SubInputs in this Input

class fastr.core.inputoutput.BaseInputOutput(node, description)[source]

Bases: fastr.core.samples.HasSamples, fastr.core.updateable.Updateable, fastr.core.serializable.Serializable

Base class for Input and Output classes. It mainly implements the properties to access the data from the underlying ParameterDescription.

__abstractmethods__ = frozenset(['_update', 'fullid', '__getitem__', 'size'])
__getstate__()[source]

Retrieve the state of the BaseInputOutput

Returns:the state of the object
Rtype dict:
__init__(node, description)[source]

Instantiate a BaseInputOutput

Parameters:
  • node – the parent node the input/output belongs to.
  • description – the ParameterDescription describing the input/output.
Returns:

created BaseInputOutput

Raises:
  • FastrTypeError – if description is not of class ParameterDescription
  • FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the fastr.typelist
__iter__()[source]

This function is blocked to avoid support for iteration using a lecacy __getitem__ method.

Returns:None
Raises:FastrNotImplementedError – always
__module__ = 'fastr.core.inputoutput'
__repr__()[source]

Get a string representation for the Input/Output

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the BaseInputOutput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
cardinality(key=None, job_data=None)[source]

Determine the cardinality of this Input/Output. Optionally a key can be given to determine for a sample.

Parameters:key – key for a specific sample
Returns:the cardinality
Return type:int, sympy.Symbol, or None
check_cardinality(key=None)[source]

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters:key – sample_index (tuple of int) or SampleId for desired sample
Returns:flag indicating that the cardinality is correct
Return type:bool
Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.
datatype

The datatype of this Input/Output

description

The description object of this input/output

fullid

The fullid of the Input/Output, the fullid should be unnique and makes the object retrievable by the network.

id

Id of the Input/Output

node

The Node to which this Input/Output belongs

numel

The number of elements in this Input/Output

required

Flag indicating that the Input/Output is required

size

The size of the Input/Output

class fastr.core.inputoutput.BaseOutput(node, description)[source]

Bases: fastr.core.inputoutput.BaseInputOutput

Base class for all outputs.

__abstractmethods__ = frozenset(['_update', 'fullid', '__getitem__', 'size'])
__init__(node, description)[source]

Instantiate a BaseOutput

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created BaseOutput

Raises:
  • FastrTypeError – if description is not of class ParameterDescription
  • FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the fastr.typelist
__module__ = 'fastr.core.inputoutput'
automatic

Flag indicating that the Output is generated automatically without being specified on the command line

class fastr.core.inputoutput.Input(node, description)[source]

Bases: fastr.core.inputoutput.BaseInput

Class representing an input of a node. Such an input will be connected to the output of another node or the output of an constant node to provide the input value.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Compare two Input instances with each other. This function ignores the parent node and update status, but tests rest of the dict for equality.

Parameters:other (Input) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getitem__(key)[source]

Retrieve an item from this Input.

Parameters:

key (str, SampleId or tuple) – the key of the requested item, can be a key str, sample index tuple or a SampleId

Returns:

the return value depends on the requested key. If the key was an int the corresponding SubInput will be returned. If the key was a SampleId or sample index tuple, the corresponding SampleItem will be returned.

Return type:

SampleItem or SubInput

Raises:
  • FastrTypeError – if key is not of a valid type
  • FastrKeyError – if the key is not found
__getstate__()[source]

Retrieve the state of the Input

Returns:the state of the object
Rtype dict:
__init__(node, description)[source]

Instantiate an input.

Parameters:
  • node (Node) – the parent node of this input.
  • description (ParameterDescription) – the ParameterDescription of the input.
Returns:

the created Input

__module__ = 'fastr.core.inputoutput'
__setitem__(key, value)[source]

Create a link between a SubInput of this Inputs and an Output/Constant

Parameters:
  • key (int, str) – the key of the SubInput
  • value (BaseOutput, list, tuple, dict, OrderedDict) – the target to link, can be an output or a value to create a constant for
Raises:

FastrTypeError – if key is not of a valid type

__setstate__(state)[source]

Set the state of the Input by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Input

Returns:the string version
Return type:str
append(value)[source]

When you want to append a link to an Input, you can use the append property. This will automatically create a new SubInput to link to.

example:

>>> link = node2['input'].append(node1['output'])

will create a new SubInput in node2[‘input’] and link to that.

cardinality(key=None, job_data=None)[source]

Cardinality for an Input is the sum the cardinalities of the SubInputs, unless defined otherwise.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
datatype

The datatype of this Input

dimnames

The list names of the dimensions in this Input. This will be a list of str.

fullid

The full defining ID for the Input

get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this Input

Returns:list of all connected Nodes
Return type:list
get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this Input

Returns:tuple of all connected Outputs
Return type:tuple
get_subinput(key)[source]

Get a requested SubInput

Parameters:key (int) – the index of the SubInput to retrieve
Returns:requested SubInput
index(value)[source]

Find index of a SubInput

Parameters:value (SubInput) – the SubInput to find the index of
Returns:key
Return type:int, str
input_group

The id of the InputGroup this Input belongs to.

insert(index)[source]

Insert a new SubInput at index in the sources list

Parameters:key (int) – positive integer for position in _source list to insert to
Returns:newly inserted SubInput
Return type:SubInput
itersubinputs()[source]

Iterate over the SubInputs in this Input.

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
num_subinput

The number of SubInputs in this Input

prepare(sample_size=None)[source]

This function makes sure the SampleIdList has the correct size.

Parameters:sample_size (tuple of int) – the required size of the SampleIdList. If no size is given, self.size will be used by default.
remove(value)[source]

Remove a SubInput from the SubInputs list based on the connected Link.

Parameters:value (SubInput, <fastr.core.inputoutput.SubInput>`) – the SubInput to removed from this Input
set_subinput(key, value)[source]

Set a specified SubInput.

Parameters:
  • key (int) – positive integer for position in _source list
  • value – new SubInput to assign to the selected location
size

The size of the sample collections that can accessed via this Input.

source

The mapping of SubInputs that are connected and have more than 0 elements.

class fastr.core.inputoutput.Output(node, description)[source]

Bases: fastr.core.inputoutput.BaseOutput

Class representing an output of a node. It holds the output values of the tool ran. Output fields can be connected to inputs of other nodes.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Compare two Output instances with each other. This function ignores the parent node, listeners and update status, but tests rest of the dict for equality.

Parameters:other (Output) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getitem__(key)[source]

Retrieve an item from this Output. The returned value depends on what type of key used:

  • Retrieving data using index tuple: [index_tuple]
  • Retrieving data sample_id str: [SampleId]
  • Retrieving a list of data using SampleId list: [sample_id1, ..., sample_idN]
  • Retrieving a SubOutput using an int or slice: [n] or [n:m]
Parameters:

key (int, slice, SampleId or tuple) – the key of the requested item, can be a number, slice, sample index tuple or a SampleId

Returns:

the return value depends on the requested key. If the key was an int or slice the corresponding SubOutput will be returned (and created if needed). If the key was a SampleId or sample index tuple, the corresponding SampleItem will be returned. If the key was a list of SampleId a tuple of SampleItem will be returned.

Return type:

SubInput or SampleItem or list of SampleItem

Raises:
  • FastrTypeError – if key is not of a valid type
  • FastrKeyError – if the parent Node has not been executed
__getstate__()[source]

Retrieve the state of the Output

Returns:the state of the object
Rtype dict:
__init__(node, description)[source]

Instantiate an Output

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created Output

Raises:
  • FastrTypeError – if description is not of class ParameterDescription
  • FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the fastr.typelist
__module__ = 'fastr.core.inputoutput'
__setitem__(key, value)[source]

Store an item in the Output

Parameters:
  • key (tuple of int or SampleId) – key of the value to store
  • value – the value to store
Returns:

None

Raises:

FastrTypeError – if key is not of correct type

__setstate__(state)[source]

Set the state of the Output by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Output

Returns:the string version
Return type:str
blocking

Flag indicating that this Output will cause blocking in the execution

cardinality(key=None, job_data=None)[source]

Cardinality of this Output, may depend on the inputs of the parent Node.

Parameters:

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:
  • FastrCardinalityError – if cardinality references an invalid Input
  • FastrTypeError – if the referenced cardinality values type cannot be case to int
  • FastrValueError – if the referenced cardinality value cannot be case to int
datatype

The datatype of this Output

dimnames

The list names of the dimensions in this Output. This will be a list of str.

fullid

The full defining ID for the Output

iterconvergingindices(collapse_dims)[source]

Iterate over all data, but collapse certain dimension to create lists of data.

Parameters:collapse_dims (iterable of int) – dimension to collapse
Returns:iterator SampleIndex (possibly containing slices)
listeners

The list of Links connected to this Output.

ndims

The number of dimensions in this Output

preferred_types

The list of preferred DataTypes for this Output.

prepare()[source]

This function makes sure that a value storage will be created

resulting_datatype

The DataType that will the results of this Output will have.

samples

The SampleCollection of the samples in this Output. None if the Node has not yet been executed. Otherwise a SampleCollection.

size

The sample size of the Output

valid

Check if the output is valid, i.e. has a valid cardinality

class fastr.core.inputoutput.SourceOutput(node, description)[source]

Bases: fastr.core.inputoutput.Output

Output for a SourceNode, this type of Output determines the cardinality in a different way than a normal Node.

__abstractmethods__ = frozenset([])
__getitem__(item)[source]

Retrieve an item from this Output. The returned value depends on what type of key used:

  • Retrieving data using index tuple: [index_tuple]
  • Retrieving data sample_id str: [SampleId]
  • Retrieving a list of data using SampleId list: [sample_id1, ..., sample_idN]
  • Retrieving a SubOutput using an int or slice: [n] or [n:m]
Parameters:

key (int, slice, SampleId or tuple) – the key of the requested item, can be a number, slice, sample index tuple or a SampleId

Returns:

the return value depends on the requested key. If the key was an int or slice the corresponding SubOutput will be returned (and created if needed). If the key was a SampleId or sample index tuple, the corresponding SampleItem will be returned. If the key was a list of SampleId a tuple of SampleItem will be returned.

Return type:

SubInput or SampleItem or list of SampleItem

Raises:
  • FastrTypeError – if key is not of a valid type
  • FastrKeyError – if the parent Node has not been executed
__init__(node, description)[source]

Instantiate a FlowOutput

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created FlowOutput

Raises:
  • FastrTypeError – if description is not of class ParameterDescription
  • FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the fastr.typelist
__module__ = 'fastr.core.inputoutput'
__setitem__(key, value)[source]

Store an item in the Output

Parameters:
  • key (tuple of int or SampleId) – key of the value to store
  • value – the value to store
Returns:

None

Raises:

FastrTypeError – if key is not of correct type

cardinality(key=None, job_data=None)[source]

Cardinality of this SourceOutput, may depend on the inputs of the parent Node.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
linearized

A linearized version of the sample data, this is lazily cached linearized version of the underlying SampleCollection.

ndims

The number of dimensions in this SourceOutput

size

The sample size of the SourceOutput

class fastr.core.inputoutput.SubInput(input_)[source]

Bases: fastr.core.inputoutput.BaseInput

This class is used by Input to allow for multiple links to an Input. The SubInput class can hold only a single Link to a (Sub)Output, but behaves very similar to an Input otherwise.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Compare two SubInput instances with each other. This function ignores the parent, node, source and update status, but tests rest of the dict for equality.

Parameters:other (SubInput) – the other instances to compare to
Returns:True if equal, False otherwise
__getitem__(key)[source]

Retrieve an item from this SubInput.

Parameters:key (int, SampleId or SampleIndex) – the key of the requested item, can be a number, sample index tuple or a SampleId
Returns:the return value depends on the requested key. If the key was an int the corresponding SubInput will be returned. If the key was a SampleId or sample index tuple, the corresponding SampleItem will be returned.
Return type:SampleItem or SubInput
Raises:FastrTypeError – if key is not of a valid type

Note

As a SubInput has only one SubInput, only requesting int key 0 or -1 is allowed, and it will return self

__getstate__()[source]

Retrieve the state of the SubInput

Returns:the state of the object
Rtype dict:
__init__(input_)[source]

Instantiate an SubInput.

Parameters:input (Input) – the parent of this SubInput.
Returns:the created SubInput
__module__ = 'fastr.core.inputoutput'
__setstate__(state)[source]

Set the state of the SubInput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the SubInput

Returns:the string version
Return type:str
cardinality(key=None, job_data=None)[source]

Get the cardinality for this SubInput. The cardinality for a SubInputs is defined by the incoming link.

Parameters:key (SampleIndex or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
description
dimnames

List of dimension names for this SubInput

fullid

The full defining ID for the SubInput

get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this SubInput

Returns:list of all connected Nodes
Return type:list
get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this SubInput

Returns:list of all connected Outputs
Return type:list
input_group

The id of the InputGroup this SubInputs parent belongs to.

iteritems()[source]

Iterate over the SampleItems that are in the SubInput.

Returns:iterator yielding SampleItem objects
itersubinputs()[source]

Iterate over SubInputs (for a SubInput it will yield self and stop iterating after that)

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
node

The Node to which this SubInputs parent belongs

num_subinput

The number of SubInputs in this SubInput, this is always 1.

remove(value)[source]

Remove a SubInput from parent Input.

Parameters:value (SubInput) – the SubInput to removed from this Input
size

The sample size of the SubInput

source

A list with the source Link. The list is to be compatible with Input

source_output

The Output linked to this SubInput

class fastr.core.inputoutput.SubOutput(output, index)[source]

Bases: fastr.core.inputoutput.Output

The SubOutput is an Output that represents a slice of another Output.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Compare two SubOutput instances with each other. This function ignores the parent, node and update status, but tests rest of the dict for equality. equality

Parameters:other (SubOutput) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getitem__(key)[source]

Retrieve an item from this SubOutput. The returned value depends on what type of key used:

  • Retrieving data using index tuple: [index_tuple]
  • Retrieving data sample_id str: [SampleId]
  • Retrieving a list of data using SampleId list: [sample_id1, ..., sample_idN]
  • Retrieving a SubOutput using an int or slice: [n] or [n:m]
Parameters:key (int, slice, SampleId or tuple) – the key of the requested item, can be a number, slice, sample index tuple or a SampleId
Returns:the return value depends on the requested key. If the key was an int or slice the corresponding SubOutput will be returned (and created if needed). If the key was a SampleId or sample index tuple, the corresponding SampleItem will be returned. If the key was a list of SampleId a tuple of SampleItem will be returned.
Return type:SubInput or SampleItem or list of SampleItem
Raises:FastrTypeError – if key is not of a valid type
__getstate__()[source]

Retrieve the state of the SubOutput

Returns:the state of the object
Rtype dict:
__init__(output, index)[source]

Instantiate a SubOutput

Parameters:
  • output – the parent output the suboutput slices.
  • index (int or slice) – the way to slice the parent output
Returns:

created SubOutput

Raises:
  • FastrTypeError – if the output argument is not an instance of Output
  • FastrTypeError – if the index argument is not an int or slice
__len__()[source]

Return the length of the Output.

Note

In a SubOutput this is always 1.

__module__ = 'fastr.core.inputoutput'
__setitem__(key, value)[source]

A function blocking the assignment operator. Values cannot be assigned to a SubOutput.

Raises:FastrNotImplementedError – if called
__setstate__(state)[source]

Set the state of the SubOutput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the SubOutput

Returns:the string version
Return type:str
cardinality(key=None, job_data=None)[source]

Cardinality of this SubOutput depends on the parent Output and self.index

Parameters:

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:
  • FastrCardinalityError – if cardinality references an invalid Input
  • FastrTypeError – if the referenced cardinality values type cannot be case to int
  • FastrValueError – if the referenced cardinality value cannot be case to int
datatype

The datatype of this SubOutput

fullid

The full defining ID for the SubOutput

indexrep

Simple representation of the index.

listeners

The list of Links connected to this Output.

node

The Node to which this SubOutput belongs

preferred_types

The list of preferred DataTypes for this SubOutput.

resulting_datatype

The DataType that will the results of this SubOutput will have.

samples

The SampleCollection for this SubOutput

interface Module

A module that describes the interface of a Tool. It specifies how a set of
input values will be translated to commands to be executed. This creates a generic interface to different ways of executing underlying software.
class fastr.core.interface.InputSpec[source]

Bases: fastr.core.interface.InputSpec

__dict__ = dict_proxy({'__dict__': <attribute '__dict__' of 'InputSpec' objects>, '__module__': 'fastr.core.interface', '__new__': <staticmethod object at 0x7f3a22bd4bb0>, '__doc__': None})
__module__ = 'fastr.core.interface'
static __new__(id_, cardinality, datatype, required=False, description='', default=None, hidden=False)[source]
fastr.core.interface.InputSpecBase

alias of InputSpec

class fastr.core.interface.Interface[source]

Bases: fastr.core.baseplugin.Plugin, fastr.core.serializable.Serializable

Abstract base class of all Interfaces. Defines the minimal requirements for all Interface implementations.

__abstractmethods__ = frozenset(['inputs', 'execute', '__setstate__', 'expanding', '__getstate__', 'outputs'])
__getstate__()[source]

Retrieve the state of the Interface

Returns:the state of the object
Rtype dict:
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.interface'
__setstate__(state)[source]

Set the state of the Interface

execute(target, payload)[source]

Execute the interface given the a target and payload. The payload should have the form {‘input’: {‘input_id_a’: (value, value), ‘input_id_b’: (value, value)}, ‘output’: {‘output_id_a’: (value, value), ‘output_id_b’: (value, value)}}

Parameters:
  • target – the target to call
  • payload – the payload to use
Returns:

the result of the execution

Return type:

(tuple of) InterfaceResult

expanding

Indicates whether or not this Interface will result in multiple samples per run. If the flow is unaffected, this will be zero, if it is nonzero it means that number of dimension will be added to the sample array.

inputs

OrderedDict of Inputs connected to the Interface. The format should be {input_id: InputSpec}.

outputs

OrderedDict of Output connected to the Interface. The format should be {output_id: OutputSpec}.

classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.core.interface.InterfacePluginManager[source]

Bases: fastr.core.pluginmanager.PluginSubManager

Container holding all the CollectorPlugins

__abstractmethods__ = frozenset([])
__init__()[source]

Create the Coll :param path: :param recursive: :return:

__module__ = 'fastr.core.interface'
plugin_class

The class of the Plugins in the collection

class fastr.core.interface.InterfaceResult(result_data, log_data, payload, sample_index=None, sample_id=None)[source]

Bases: object

The class in which Interfaces should wrap their results to be picked up by fastr

__dict__ = dict_proxy({'__dict__': <attribute '__dict__' of 'InterfaceResult' objects>, '__module__': 'fastr.core.interface', '__weakref__': <attribute '__weakref__' of 'InterfaceResult' objects>, '__doc__': '\n The class in which Interfaces should wrap their results to be picked up by fastr\n ', '__init__': <function __init__ at 0x7f3a22be0f50>})
__init__(result_data, log_data, payload, sample_index=None, sample_id=None)[source]
__module__ = 'fastr.core.interface'
__weakref__

list of weak references to the object (if defined)

class fastr.core.interface.OutputSpec[source]

Bases: fastr.core.interface.OutputSpec

__dict__ = dict_proxy({'__dict__': <attribute '__dict__' of 'OutputSpec' objects>, '__module__': 'fastr.core.interface', '__new__': <staticmethod object at 0x7f3a22bd4be8>, '__doc__': None})
__module__ = 'fastr.core.interface'
static __new__(id_, cardinality, datatype, automatic=True, required=False, description='', hidden=False)[source]
fastr.core.interface.OutputSpecBase

alias of OutputSpec

ioplugin Module

This module contains the manager class for IOPlugins and the base class for all IOPlugins

class fastr.core.ioplugin.IOPlugin[source]

Bases: fastr.core.baseplugin.Plugin

IOPlugins are used for data import and export for the sources and sinks. The main use of the IOPlugins is during execution (see Execution). The IOPlugins can be accessed via fastr.ioplugins, but generally there should be no need for direct interaction with these objects. The use of is mainly via the URL used to specify source and sink data.

__abstractmethods__ = frozenset(['scheme'])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.ioplugin'
cleanup()[source]

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
fetch_url(inurl, outfile)[source]

(abstract) Fetch a file from an external data source.

Parameters:
  • inurl – url to the item in the data store
  • outpath – path where to store the fetch data locally
fetch_value(inurl)[source]

(abstract) Fetch a value from an external data source.

Parameters:inurl – the url of the value to retrieve
Returns:the fetched value
static isurl(string)[source]

Test if given string is an url.

Parameters:string (str) – string to test
Returns:True if the string is an url, False otherwise
Return type:bool
path_to_url(path, mountpoint=None)[source]

(abstract) Construct an url from a given mount point and a relative path to the mount point.

Parameters:
  • path (str) – the path to determine the url for
  • mountpoint (str or None) – the mount point to use, will be automatically detected if None is given
Returns:

url matching the path

Return type:

str

static print_result(result)[source]

Print the result of the IOPlugin to stdout to be picked up by the tool

Parameters:result – value to print as a result
Returns:None
pull_source_data(inurl, outdir, sample_id, datatype=None)[source]

Transfer the source data from inurl to be available in outdir.

Parameters:
  • inurl (str) – the input url to fetch data from
  • outdir (str) – the directory to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

push_sink_data(inpath, outurl, datatype=None)[source]

Write out the sink data from the inpath to the outurl.

Parameters:
  • inpath (str) – the path of the data to be pushed
  • outurl (str) – the url to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

put_url(inpath, outurl)[source]

(abstract) Put the files to the external data store.

Parameters:
  • inpath – path to the local data
  • outurl – url to where to store the data in the external data store.
put_value(value, outurl)[source]

(abstract) Put the files to the external data store.

Parameters:
  • value – the value to store
  • outurl – url to where to store the data in the external data store.
scheme

(abstract) This abstract property is to be overwritten by a subclass to indicate the url scheme associated with the IOPlugin.

setup(*args, **kwargs)[source]

(abstract) Setup before data transfer. This can be any function that needs to be used to prepare the plugin for data transfer.

url_to_path(url)[source]

(abstract) Get the path to a file from a url.

Parameters:url (str) – the url to retrieve the path for
Returns:the corresponding path
Return type:str
class fastr.core.ioplugin.IOPluginManager[source]

Bases: fastr.core.pluginmanager.PluginSubManager

A mapping containing the IOPlugins known to this system

__abstractmethods__ = frozenset([])
__init__()[source]

Create the IOPluginManager and populate it.

Returns:newly created IOPluginManager
__iter__()[source]
__keytransform__(key)[source]
__module__ = 'fastr.core.ioplugin'
cleanup()[source]

Cleanup all plugins, this closes files, connections and other things that could be left dangling otherwise.

static create_ioplugin_tool()[source]

Create the tools which handles sinks and sources. The command of this tool is the main of core.ioplugin.

expand_url(url)[source]

Expand the url by filling the wildcards. This function checks the url scheme and uses the expand function of the correct IOPlugin.

Parameters:url (str) – url to expand
Returns:list of urls
Return type:list of str
plugin_class

The PluginClass of the items of the BasePluginManager

populate()[source]

Populate the IOPlugins manager. After the default directory scan, add the vfs IOPlugin and create the Tools for the IOPlugins

pull_source_data(url, outdir, sample_id, datatype=None)[source]

Retrieve data from an external source. This function checks the url scheme and selects the correct IOPlugin to retrieve the data.

Parameters:
  • url – url to pull
  • outdir (str) – the directory to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters:
  • inpath – path to the local data
  • outurl – url to where to store the data in the external data store.
static register_url_scheme(scheme)[source]

Register a custom scheme to behave http like. This is needed to parse all things properly with urlparse.

Parameters:scheme – the scheme to register
url_to_path(url)[source]

Retrieve the path for a given url

Parameters:url (str) – the url to parse
Returns:the path corresponding to the input url
Return type:str
fastr.core.ioplugin.main()[source]

The main entry point for command line access to the IOPlugin

network Module

Network module containing Network facilitators and analysers.

class fastr.core.network.Network(id_='unnamed_network', version=None)[source]

Bases: fastr.core.serializable.Serializable

The Network class represents a workflow. This includes all Nodes (including ConstantNodes, SourceNodes and Sinks) and Links.

NETWORK_DUMP_FILE_NAME = '__fastr_network__.json'
SOURCE_DUMP_FILE_NAME = '__source_data__.pickle.gz'
__dataschemafile__ = 'Network.schema.json'
__eq__(other)[source]

Compare two Networks and see if they are equal.

Parameters:other (Network) –
Returns:flag indicating that the Networks are the same
Return type:bool
__getitem__(item)[source]

Get an item by its fullid. The fullid can point to a link, node, input, output or even subinput/suboutput.

Parameters:item (str,unicode) – fullid of the item to retrieve
Returns:the requested item
__getstate__()[source]

Retrieve the state of the Network

Returns:the state of the object
Rtype dict:
__init__(id_='unnamed_network', version=None)[source]

Create a new, empty Network

Parameters:name (str) – name of the Network
Returns:newly created Network
Raises:OSError – if the tmp mount in the config is not a writable directory
__module__ = 'fastr.core.network'
__ne__(other)[source]

Tests for non-equality, this is the negated version __eq__

__repr__()[source]
__setstate__(state)[source]

Set the state of the Network by the given state. This completely overwrites the old state!

Parameters:state (dict) – The state to populate the object with
Returns:None
abort()[source]

Add a Link to the Network. Make sure the link is in the link list and the link parent is set to this Network

Parameters:

link (Link) – link to add

Raises:
  • FastrTypeError – if link is incorrectly typed
  • FastrNetworkMismatchError – if the link already belongs to another Network
add_node(node)[source]

Add a Node to the Network. Make sure the node is in the node list and the node parent is set to this Network

Parameters:node (Node) – node to add
Raises:FastrTypeError – if node is incorrectly typed
add_stepid(stepid, node)[source]

Add a Node to a specific step id

Parameters:
  • stepid (str) – the stepid that the node will be added to
  • node (Node) – the node to add to the stepid
check_id(id_)[source]

Check if an id for an object is valid and unused in the Network. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:
  • FastrValueError – if the id is not correctly formatted
  • FastrValueError – if the id is already in use
create_constant(datatype, data, id_=None, stepid=None, nodegroup=None, sourcegroup=None)[source]

Create a ConstantNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the constant node
  • data (datatype or list of datatype) – The data to hold in the constant node
  • id (str) – The id of the constant node to be created
  • stepid (str) – The stepid to add the created constant node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created constant node

Return type:

ConstantNode

Create a link between two Nodes and add it to the current Network.

Parameters:
  • source (BaseOutput) – the output that is the source of the link
  • target (BaseInput) – the input that is the target of the link
  • id (str) – the id of the link
Returns:

the created link

Type:

Link

create_macro(network, id_=None)[source]
create_node(tool, id_=None, stepid=None, cores=None, memory=None, walltime=None, nodegroup=None)[source]

Create a Node in this Network. The Node will be automatically added to the Network.

Parameters:
  • tool (Tool) – The Tool to base the Node on
  • id (str) – The id of the node to be created
  • stepid (str) – The stepid to add the created node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created node

Return type:

Node

create_reference(source_data, output_directory)[source]
create_sink(datatype, id_=None, stepid=None)[source]

Create a SinkNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the sink node
  • id (str) – The id of the sink node to be created
  • stepid (str) – The stepid to add the created sink node to
Returns:

the newly created sink node

Return type:

SinkNode

create_source(datatype, id_=None, stepid=None, nodegroup=None, sourcegroup=None)[source]

Create a SourceNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the source source_node
  • id (str) – The id of the source source_node to be created
  • stepid (str) – The stepid to add the created source source_node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
  • sourcegroup (str) – DEPRECATED! The nodegroup this SourceNode will be added to
Returns:

the newly created source source_node

Return type:

SourceNode

draw_network(name='network_layout', img_format='svg', draw_dimension=False)[source]

Output a dot file and try to convert it to an image file.

Parameters:img_format (str) – extension of the image format to convert to
Returns:path of the image created or None if failed
Return type:str or None
execute(sourcedata, sinkdata, execution_plugin=None, tmpdir=None, cluster_queue=None)[source]

Execute the Network with the given data. This will analyze the Network, create jobs and send them to the execution backend of the system.

Parameters:
  • sourcedata (dict) – dictionary containing all data for the sources
  • sinkdata (dict) – dictionary containing directives for the sinks
  • execution_plugin (str) – the execution plugin to use (None will use the config value)
Raises:
  • FastrKeyError – if a source has not corresponding key in sourcedata
  • FastrKeyError – if a sink has not corresponding key in sinkdata
fullid

The fullid of the Network

id

The id of the Network. This is a read only property.

is_valid()[source]
job_finished(job, execution_interface)[source]

Call-back handler for when a job is finished. Will collect the results and handle blocking jobs. This function is automatically called when the execution plugin finished a job.

Parameters:job (Job) – the job that finished
remove(value)[source]

Remove an item from the Network.

Parameters:value (Node or Link) – the item to remove
test(reference_data_dir, source_data=None)[source]

Execute the network with the source data specified and test the results against the refence data. This effectively tests the network execution.

Parameters:
  • reference_data_dir (str) – The path or vfs url of reference data to compare with
  • source_data (dict) – The source data to use

networkmanager Module

This module contains the tool manager class

class fastr.core.networkmanager.NetworkManager(path)[source]

Bases: fastr.core.objectmanager.ObjectManager

__abstractmethods__ = frozenset([])
__module__ = 'fastr.core.networkmanager'
get_object_version(obj)[source]
object_class
fastr.core.networkmanager.networklist = Empty NetworkManager

The fastr networklist

node Module

A module to maintain a network node.

Exported classes:

Node – A class encapsulating a tool. ConstantNode – A node encapsulating an Output to set scalar values. SourceNode – A class providing a handle to a file.

class fastr.core.node.AdvancedFlowNode(tool, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Bases: fastr.core.node.FlowNode

__abstractmethods__ = frozenset([])
__module__ = 'fastr.core.node'
execute()[source]

Execute the node and create the jobs that need to run

Returns:list of jobs to run
Return type:list of Jobs
set_result(job)[source]
class fastr.core.node.ConstantNode(datatype, data, id_=None)[source]

Bases: fastr.core.node.SourceNode

Class encapsulating one output for which a value can be set. For example used to set a scalar value to the input of a node.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'ConstantNode.schema.json'
__getstate__()[source]

Retrieve the state of the ConstantNode

Returns:the state of the object
Rtype dict:
__init__(datatype, data, id_=None)[source]

Instantiation of the ConstantNode.

Parameters:
  • datatype – The datatype of the output.
  • data – the prefilled data to use.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a constant using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> source = network.create_source(datatype=fastr.typelist['ITKImageFile'], id_='sourceN')

or alternatively create a constant node by assigning data to an item in an InputDict:

>>> node_a.inputs['in'] = ['some', 'data']

which automatically creates and links a ConstantNode to the specified Input

__module__ = 'fastr.core.node'
__setstate__(state)[source]

Set the state of the ConstantNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
data

The data stored in this constant node

execute()[source]

Execute the constant node and create the jobs that need to run

Returns:list of jobs to run
Return type:list of Jobs
set_data(data=None, ids=None)[source]

Set the data of this constant node in the correct way. This is mainly for compatibility with the parent class SourceNode

Parameters:
  • data (dict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
class fastr.core.node.DefaultInputGroupCombiner(input_groups)[source]

Bases: object

__dict__ = dict_proxy({'__module__': 'fastr.core.node', 'merge_sample_jobs': <function merge_sample_jobs at 0x7f3a2258aed8>, '__weakref__': <attribute '__weakref__' of 'DefaultInputGroupCombiner' objects>, 'merge_sample_id': <function merge_sample_id at 0x7f3a2258ad70>, 'update': <function update at 0x7f3a2258e140>, 'iter_input_groups': <function iter_input_groups at 0x7f3a2258e050>, 'merge': <function merge at 0x7f3a2258ab90>, '__iter__': <function __iter__ at 0x7f3a2258e0c8>, 'merge_payloads': <function merge_payloads at 0x7f3a2258af50>, 'dimnames': <property object at 0x7f3a2258b3c0>, 'outputsize': <property object at 0x7f3a2258b418>, '__dict__': <attribute '__dict__' of 'DefaultInputGroupCombiner' objects>, 'merge_sample_index': <function merge_sample_index at 0x7f3a2258ade8>, 'unmerge': <function unmerge at 0x7f3a2258ac08>, '__doc__': None, '__init__': <function __init__ at 0x7f3a2258ab18>, 'merge_sample_data': <function merge_sample_data at 0x7f3a2258ae60>})
__init__(input_groups)[source]
__iter__()[source]
__module__ = 'fastr.core.node'
__weakref__

list of weak references to the object (if defined)

dimnames
iter_input_groups()[source]
merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine
Returns:combined list
merge_payloads(sample_payloads)[source]
merge_sample_data(list_of_sample_data)[source]
merge_sample_id(list_of_sample_ids)[source]
merge_sample_index(list_of_sample_indexes)[source]
merge_sample_jobs(list_of_sample_jobs)[source]
outputsize
unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> list_of_items = combiner.unmerge(item)
>>> item = combiner.merge(list_of_items.values())
Parameters:item (list) – the item to unmerge
Returns:items
Return type:OrderedDict
update()[source]
class fastr.core.node.FlowNode(tool, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Bases: fastr.core.node.Node

A Flow Node is a special subclass of Nodes in which the amount of samples can vary per Output. This allows non-default data flows.

__abstractmethods__ = frozenset([])
__init__(tool, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Instantiate a flow node.

Parameters:
  • tool (Tool) – The tool to base the node on
  • id (str) – the id of the node
  • parent (Network) – the parent network of the node
Returns:

the newly created FlowNode

__module__ = 'fastr.core.node'
blocking

A FlowNode is (for the moment) always considered blocking.

Returns:True
dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

outputsize

Size of the outputs in this Node

set_result(job)[source]

Incorporate result of a job into the FlowNode.

Parameters:job (Type) – job of which the result to store
class fastr.core.node.InputDict(*args, **kwds)[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Implements helper functions for the easy linking syntax.

__module__ = 'fastr.core.node'
__setitem__(key, value, dict_setitem=<slot wrapper '__setitem__' of 'dict' objects>)[source]

Set an item in the input dictionary. The behaviour depends on the type of the value. For a BaseInput, the input will simply be added to the list of inputs. For a BaseOutput, a link between the output and input will be created.

Parameters:
  • key (str) – id of the input to assign/link
  • value (BaseInput or BaseOutput) – either the input to add or the output to link
  • dict_setitem – the setitem function to use for the underlying OrderedDict insert
class fastr.core.node.InputGroup(*args, **kwargs)[source]

Bases: collections.OrderedDict

A class representing a group of inputs. Input groups allow the

__abstractmethods__ = frozenset([])
__delitem__(*args, **kwargs)

od.__delitem__(y) <==> del od[y]

Note

This is a wrapped version of collections.__delitem__ which triggers an update of the object after being called

__getitem__(key)[source]
__init__(*args, **kwargs)

Create a new InputGroup representation

Parameters:
  • parent (Node) – the parent node
  • id (str) – the id of the input group
Raises:

FastrTypeError – if parent is not a Node

Note

This is a wrapped version of fastr.core.node.__init__ which triggers an update of the object after being called

__metaclass__

alias of UpdateableMeta

__module__ = 'fastr.core.node'
__setitem__(*args, **kwargs)

Assign an input to this input group.

Parameters:
  • key (str) – id of the input
  • value (Input) – the input to assign
Raises:

FastrTypeError – if value of valid type

Note

This is a wrapped version of fastr.core.node.__setitem__ which triggers an update of the object after being called

__updatefunc__()[source]

Update the InputGroup. Triggers when a change is made to the content of the InputGroup. Automatically recalculates the size, primary Input etc.

__updatetriggers__ = ['__init__', '__setitem__', '__delitem__', 'clear', 'pop', 'popitem', 'setdefault', 'update']
clear() → None. Remove all items from od.

Note

This is a wrapped version of collections.clear which triggers an update of the object after being called

dimnames

The names of the dimensions in this InputGroup

empty

Bool indicating that this InputGroup is empty (has no data connected)

classmethod find_source_index(target_size, target_dimnames, source_size, source_dimnames, target_index)[source]
iterinputvalues

Iterate over the item in this InputGroup

Returns:iterator yielding SampleItems
parent

The parent node of this InputGroup

pop(k[, d]) → v, remove specified key and return the corresponding
value. If key is not found, d is returned if given, otherwise KeyError
is raised.

Note

This is a wrapped version of collections.pop which triggers an update of the object after being called

popitem() → (k, v), return and remove a (key, value) pair.

Pairs are returned in LIFO order if last is true or FIFO order if false.

Note

This is a wrapped version of collections.popitem which triggers an update of the object after being called

primary

The primary Input in this InputGroup. The primary Input is the Input that defines the size of this InputGroup. In case of ties it will be the first in the tool definition.

setdefault(k[, d]) → od.get(k,d), also set od[k]=d if k not in od

Note

This is a wrapped version of collections.setdefault which triggers an update of the object after being called

size

The sample size of this InputGroup

classmethod solve_broadcast(target_size, target_dimnames, source_size, source_dimnames, target_index, nodegroups=None)[source]
update([E, ]**F) → None. Update D from mapping/iterable E and F.
If E present and has a .keys() method, does: for k in E: D[k] = E[k]
If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

Note

This is a wrapped version of _abcoll.update which triggers an update of the object after being called

class fastr.core.node.MacroNode(network, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Bases: fastr.core.node.Node

MacroNode encapsulates an entire network in a single node.

__abstractmethods__ = frozenset([])
__getstate__()[source]

Retrieve the state of the MacroNode

Returns:the state of the object
Rtype dict:
__init__(network, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]
Parameters:network (Network) – network to create macronode for
__module__ = 'fastr.core.node'
__setstate__(state)[source]
execute()[source]
class fastr.core.node.MergingInputGroupCombiner(input_groups, merge_dimension)[source]

Bases: fastr.core.node.DefaultInputGroupCombiner

__init__(input_groups, merge_dimension)[source]
__module__ = 'fastr.core.node'
iter_input_groups()[source]
merge(list_of_items)[source]
unmerge(item)[source]
update()[source]
class fastr.core.node.Node(tool, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Bases: fastr.core.updateable.Updateable, fastr.core.serializable.Serializable

The class encapsulating a node in the network. The node is responsible for setting and checking inputs and outputs based on the description provided by a tool instance.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'Node.schema.json'
__eq__(other)[source]

Compare two Node instances with each other. This function ignores the parent and update status, but tests rest of the dict for equality. equality

Parameters:other (Node) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the Node

Returns:the state of the object
Rtype dict:
__init__(tool, id_=None, parent=None, cores=None, memory=None, walltime=None)[source]

Instantiate a node.

Parameters:
  • tool (Tool) – The tool to base the node on
  • id (str) – the id of the node
  • parent (Network) – the parent network of the node
  • cores (int) – number of cores required for executing this Node
  • memory (str) – amount of memory required in the form d+[mMgG] where M is for megabyte and G for gigabyte
  • walltime (str) – amount of time required in second or in the form HOURS:MINUTES:SECOND
Returns:

the newly created Node

__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.node'
__repr__()[source]

Get a string representation for the Node

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Node

Returns:the string version
Return type:str
blocking

Indicate that the results of this Node cannot be determined without first executing the Node, causing a blockage in the creation of jobs. A blocking Nodes causes the Chunk borders.

create_job(sample_id, sample_index, job_data, job_dependencies, jobid=None, outputurl=None, **kwargs)[source]

Create a job based on the sample id, job data and job dependencies.

Parameters:
  • sample_id (SampleId) – the id of the corresponding sample
  • job_data (dict) – dictionary containing all input data for the job
  • job_dependencies – other jobs that need to finish before this job can run
Returns:

the created job

Return type:

Job

classmethod createobj(state, network=None)[source]
dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

execute()[source]

Execute the node and create the jobs that need to run

Returns:list of jobs to run
Return type:list of Jobs
find_source_index(target_index, target, source)[source]
fullid

The full defining ID for the Node

get_sourced_nodes()[source]

A list of all Nodes connected as sources to this Node

Returns:list of all nodes that are connected to an input of this node
id

The id of the Node

id_ = None

The Node id s a unique string identifying the Node

inputgroups
A list of inputgroups for this Node. An input group is InputGroup
object filled according to the Node
inputs = None

A list of inputs of this Node

listeners

All the listeners requesting output of this node, this means the listeners of all Outputs and SubOutputs

merge_dimensions
name

Name of the Tool the Node was based on. In case a Toolless Node was used the class name is given.

nodegroup
outputs = None

A list of outputs of this Node

outputsize

Size of the outputs in this Node

parent

The parent is the Network this Node is part of

prepare()[source]

Prepare the node for execution. It will create a SampleIdList of the correct size and prepare the outputs.

required_cores

Number of cores required for the execution of this Node

required_memory

Amount of memory required for the execution of this Node. Follows the format d+[mMgG] so 500M or 4g would be valid ways to specify 500 megabytes or 4 gigabyte of memory.

required_time

Amount of time required for the execution of this Node. Follows the format of a number of second or H:M:S, with H the number of hours, M the number of minutes and S the number of seconds.

set_result(job)[source]

Incorporate result of a job into the Node.

Parameters:job (Type) – job of which the result to store
status
tool
update_inputgroups()[source]

Update all input groups in this node

class fastr.core.node.OutputDict(*args, **kwds)[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Only checks if the inserted values are actually outputs.

__module__ = 'fastr.core.node'
__setitem__(key, value, dict_setitem=<slot wrapper '__setitem__' of 'dict' objects>)[source]

Set an output.

Parameters:
  • key (str) – the of the item to set
  • value (BaseOutput) – the output to set
  • dict_setitem – the setitem function to use for the underlying OrderedDict insert
class fastr.core.node.SinkNode(datatype, id_=None)[source]

Bases: fastr.core.node.Node

Class which handles where the output goes. This can be any kind of file, e.g. image files, textfiles, config files, etc.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'SinkNode.schema.json'
__getstate__()[source]
__init__(datatype, id_=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The datatype of the output.
  • id – the id of the node to create
Returns:

newly created sink node

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> sink = network.create_sink(datatype=fastr.typelist['ITKImageFile'], id_='SinkN')
__module__ = 'fastr.core.node'
__setstate__(state)[source]
create_job(sample_id, sample_index, job_data, job_dependencies)[source]

Create a job for a sink based on the sample id, job data and job dependencies.

Parameters:
  • sample_id (SampleId) – the id of the corresponding sample
  • job_data (dict) – dictionary containing all input data for the job
  • job_dependencies – other jobs that need to finish before this job can run
Returns:

the created job

Return type:

Job

datatype

The datatype of the data this sink can store.

execute()[source]

Execute the sink node and create the jobs that need to run

Returns:list of jobs to run
Return type:list of Jobs
input

The default input of the sink Node

set_data(data)[source]

Set the targets of this sink node.

Parameters:data (dict or list of urls) – the targets rules for where to write the data

The target rules can include a few fields that can be filled out:

field description
sample_id the sample id of the sample written in string form
cardinality the cardinality of the sample written
ext the extension of the datatype of the written data, including the .
network the id of the network the sink is part of
node the id of the node of the sink
timestamp the iso formatted datetime the network execution started
uuid the uuid of the network run (generated using uuid.uuid1)

An example of a valid target could be:

>>> target = 'vfs://output_mnt/some/path/image_{sample_id}_{cardinality}{ext}'
class fastr.core.node.SourceNode(datatype, id_=None)[source]

Bases: fastr.core.node.FlowNode

Class providing a connection to data resources. This can be any kind of file, stream, database, etc from which data can be received.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'SourceNode.schema.json'
__eq__(other)[source]

Compare two Node instances with each other. This function ignores the parent and update status, but tests rest of the dict for equality. equality

Parameters:other (Node) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the SourceNode

Returns:the state of the object
Rtype dict:
__init__(datatype, id_=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The (id of) the datatype of the output.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a source using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> source = network.create_source(datatype=fastr.typelist['ITKImageFile'], id_='sourceN')
__module__ = 'fastr.core.node'
__setstate__(state)[source]

Set the state of the SourceNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
create_job(sample_id, sample_index, job_data, job_dependencies)[source]
datatype

The datatype of the data this source supplies.

dimnames

Names of the dimensions in the SourceNode output. These will be reflected in the SampleIdLists.

execute()[source]

Execute the source node and create the jobs that need to run

Returns:list of jobs to run
Return type:list of Jobs
output

Shorthand for self.outputs['output']

outputsize

The size of output of this SourceNode

set_data(data, ids=None)[source]

Set the data of this source node.

Parameters:
  • data (dict, OrderedDict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
sourcegroup
valid

This does nothing. It only overloads the valid method of Node(). The original is intended to check if the inputs are connected to some output. Since this class does not implement inputs, it is skipped.

objectmanager Module

This module contains the object manager class

class fastr.core.objectmanager.ObjectManager(path)[source]

Bases: fastr.core.basemanager.BaseManager

Class for managing all the objects loaded in the fastr system

__abstractmethods__ = frozenset(['object_class', 'get_object_version'])
__contains__(key)[source]

Check if an item is in the ObjectManager

Parameters:key (str or tuple) – object id or tuple (Objectid, version)
Returns:flag indicating the item is in the manager
__getitem__(key)[source]

Retrieve a Object from the ObjectManager. You can request by only an id, which results in the newest version of the Object being returned, or request using both an id and a version.

Parameters:key (str or tuple) – object id or tuple (Objectid, version)
Returns:the requested Object
Raises:FastrObjectUnknownError – if a non-existing Object was requested
__init__(path)[source]

Create a ObjectManager and scan path to search for Objects

Parameters:path (str or iterable of str) – the path(s) to scan for Objects
Returns:newly created ObjectManager
__keytransform__(key)[source]

Key transform, used for allowing indexing both by id-only and by (id, version)

Parameters:key – key to transform
Returns:key in form (id, version)
__module__ = 'fastr.core.objectmanager'
get_object_version(obj)[source]

Get the version of a given object

Parameters:object – the object to use
Returns:the version of the object
object_class

The class of the objects to populate the manager with

objectversions(obj)[source]

Return a list of available versions for the object

Parameters:object – The object to check the versions for. Can be either a Object or a str.
Returns:List of version objects. Returns None when the given object is not known.
todict()[source]

Return a dictionary version of the Manager

Returns:manager as a dict

pluginmanager Module

This module contains the Manager class for Plugins in the fastr system

class fastr.core.pluginmanager.BasePluginManager(path=None, recursive=False)[source]

Bases: fastr.core.basemanager.BaseManager

Baseclass for PluginManagers, need to override the self._plugin_class

__abstractmethods__ = frozenset(['plugin_class'])
__getitem__(key)[source]

Retrieve item from BaseManager

Parameters:key – the key of the item to retrieve
Returns:the value indicated by the key
Raises:FastrKeyError – if the key is not found in the BaseManager
__init__(path=None, recursive=False)[source]

Create a BasePluginManager and scan the give path for matching plugins

Parameters:
  • path (str) – path to scan
  • recursive (bool) – flag to indicate a recursive search
Returns:

newly created plugin manager

Raises:

FastrTypeError – if self._plugin_class is set to a class not subclassing BasePlugin

__module__ = 'fastr.core.pluginmanager'
load_plugin(plugin_key)[source]
plugin_class

The class from which the plugins must be subclassed

populate()[source]

Populate the manager with the data. This is a method that will be called when the Managers data is first accessed. This way we avoid doing expensive directory scans when the data is never requested.

class fastr.core.pluginmanager.LazyModule(name, parent, plugin_manager)[source]

Bases: module

A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.

__getattr__(item)[source]

The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.

Parameters:item (str) – attribute to retrieve
Returns:the requested attribute
__init__(name, parent, plugin_manager)[source]
__module__ = 'fastr.core.pluginmanager'
__repr__()[source]
__weakref__

list of weak references to the object (if defined)

class fastr.core.pluginmanager.PluginManager(path=None)[source]

Bases: fastr.core.pluginmanager.BasePluginManager

__abstractmethods__ = frozenset([])
__init__(path=None)[source]
__module__ = 'fastr.core.pluginmanager'
__setitem__(key, value)[source]

Store an item in the BaseManager, will ignore the item if the key is already present in the BaseManager.

Parameters:
  • name – the key of the item to save
  • value – the value of the item to save
Returns:

None

plugin_class

The plugin manager contains any Plugin subclass

class fastr.core.pluginmanager.PluginSubManager(parent, plugin_class)[source]

Bases: fastr.core.pluginmanager.BasePluginManager

A PluginManager that is a selection of a parent plugin manger. It uses the PluginsView to only exponse part of the parent PluginManager. This is used to create plugin managers for only certain types of plugins (e.g. IOPlugins) without loading them multiple times.

__abstractmethods__ = frozenset([])
__init__(parent, plugin_class)[source]
__module__ = 'fastr.core.pluginmanager'
data
plugin_class

PluginSubManagers only expose the plugins of a certain class

class fastr.core.pluginmanager.PluginsView(parent, plugin_class)[source]

Bases: _abcoll.MutableMapping

A collection that acts like view of the plugins of another plugin manager. This is a proxy object that only gives access the plugins of a certain plugin class. It behaves like a mapping and is used as the data object for a PluginSubManager.

__abstractmethods__ = frozenset([])
__delitem__(key)[source]
__getitem__(item)[source]
__init__(parent, plugin_class)[source]

Constructor for the plugins view

Parameters:
  • parent (BasePluginManager) – the parent plugin manager
  • plugin_class (class) – the class of the plugins to expose
__iter__()[source]
__len__()[source]
__module__ = 'fastr.core.pluginmanager'
__setitem__(key, value)[source]
filter_plugin(plugin)[source]
class fastr.core.pluginmanager.plugin_option_type(filename, name, namespace, id)

Bases: tuple

__dict__ = dict_proxy({'__module__': 'fastr.core.pluginmanager', '_make': <classmethod object at 0x7f3a22e0fb78>, '_replace': <function _replace at 0x7f3a22e20410>, '_asdict': <function _asdict at 0x7f3a22e20398>, '__dict__': <property object at 0x7f3a22e1d418>, '__getnewargs__': <function __getnewargs__ at 0x7f3a22e20488>, 'id': <property object at 0x7f3a22e1d628>, 'name': <property object at 0x7f3a22e1d578>, '_fields': ('filename', 'name', 'namespace', 'id'), '__new__': <staticmethod object at 0x7f3a22e0fbe8>, 'namespace': <property object at 0x7f3a22e1d5d0>, 'filename': <property object at 0x7f3a22e1d520>, '__slots__': (), '__repr__': <function __repr__ at 0x7f3a22e20320>, '__getstate__': <function __getstate__ at 0x7f3a22e20500>, '__doc__': 'plugin_option_type(filename, name, namespace, id)'})
__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

__module__ = 'fastr.core.pluginmanager'
static __new__(_cls, filename, name, namespace, id)

Create new instance of plugin_option_type(filename, name, namespace, id)

__repr__()

Return a nicely formatted representation string

__slots__ = ()
filename

Alias for field number 0

id

Alias for field number 3

name

Alias for field number 1

namespace

Alias for field number 2

provenance Module

class fastr.core.provenance.Provenance(parent, host=None)[source]

Bases: object

The Provenance object keeps track of everything that happens to a data object.

__dict__ = dict_proxy({'__module__': 'fastr.core.provenance', '__init__': <function __init__ at 0x7f3a22573410>, 'activity': <function activity at 0x7f3a22573578>, '__dict__': <attribute '__dict__' of 'Provenance' objects>, '_add_namespace': <function _add_namespace at 0x7f3a22573488>, '__weakref__': <attribute '__weakref__' of 'Provenance' objects>, '__doc__': '\n The Provenance object keeps track of everything that happens to a data object.\n ', 'agent': <function agent at 0x7f3a22573500>, 'entity': <function entity at 0x7f3a225735f0>})
__init__(parent, host=None)[source]
__module__ = 'fastr.core.provenance'
__weakref__

list of weak references to the object (if defined)

activity(identifier, start_time=None, end_time=None, other_attributes=None)[source]
agent(identifier, other_attributes=None)[source]
entity(identifier, other_attributes=None)[source]

samples Module

This package holds the classes for working with samples.

class fastr.core.samples.HasSamples[source]

Bases: object

Base class for all classes that supply samples. This base class allows to only define __getitem__ and size and get all other basic functions mixed in so that the object behaves similar to a Mapping.

__abstractmethods__ = frozenset(['__getitem__', 'size'])
__contains__(item)[source]
__dict__ = dict_proxy({'_abc_cache': <_weakrefset.WeakSet object at 0x7f3a22de1150>, '__module__': 'fastr.core.samples', '__metaclass__': <class 'abc.ABCMeta'>, '_abc_negative_cache': <_weakrefset.WeakSet object at 0x7f3a22de11d0>, '__getitem__': <function __getitem__ at 0x7f3a22dda398>, '__contains__': <function __contains__ at 0x7f3a22dda488>, 'items': <function items at 0x7f3a22dda5f0>, '__abstractmethods__': frozenset(['__getitem__', 'size']), 'ids': <function ids at 0x7f3a22dda6e0>, 'indexes': <function indexes at 0x7f3a22dda668>, '_abc_registry': <_weakrefset.WeakSet object at 0x7f3a22de10d0>, '__iter__': <function __iter__ at 0x7f3a22dda578>, '_abc_negative_cache_version': 30, 'iteritems': <function iteritems at 0x7f3a22dda500>, '__dict__': <attribute '__dict__' of 'HasSamples' objects>, '__weakref__': <attribute '__weakref__' of 'HasSamples' objects>, '__doc__': '\n Base class for all classes that supply samples. This base class allows\n to only define __getitem__ and size and get all other basic functions mixed\n in so that the object behaves similar to a Mapping.\n ', 'size': <abc.abstractproperty object at 0x7f3a22dc8b48>})
__getitem__(item)[source]
__iter__()[source]
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.samples'
__weakref__

list of weak references to the object (if defined)

ids()[source]
indexes()[source]
items()[source]
iteritems()[source]
size
class fastr.core.samples.SampleBaseId[source]

Bases: tuple

This class represents a sample id. A sample id is a multi-dimensional id that has a simple, consistent string representation.

__add__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way.

__dict__ = dict_proxy({'__module__': 'fastr.core.samples', '__new__': <staticmethod object at 0x7f3a22dcf600>, '__str__': <function __str__ at 0x7f3a22dda9b0>, '__radd__': <function __radd__ at 0x7f3a22ddaaa0>, '__repr__': <function __repr__ at 0x7f3a22dda938>, '__add__': <function __add__ at 0x7f3a22ddaa28>, '__dict__': <attribute '__dict__' of 'SampleBaseId' objects>, '__doc__': '\n This class represents a sample id. A sample id is a multi-dimensional\n id that has a simple, consistent string representation.\n ', '_element_type': <type 'NoneType'>})
__module__ = 'fastr.core.samples'
static __new__(*args)[source]

Create a new SampleId

Parameters:args (iterator/iterable of element type or element type) – the strings to make sample id for
__radd__(other)[source]

Add another SampleId, this allows to add parts to the SampleId in a convenient way. This is the right-hand version of the operator.

__repr__()[source]

Get a string representation for the SampleBaseId

Returns:the string representation
Return type:str
__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns:the string version
Return type:str
class fastr.core.samples.SampleCollection(dimnames, parent)[source]

Bases: _abcoll.MutableMapping

The SampleCollections is a class that contains the data including a form of ordering. Each sample is reachable both by its SampleId and a SampleIndex. The object is sparse, so not all SampleId have to be defined allowing for non-rectangular data shapes.

Note

This object is meant to replace both the SampleIdList and the ValueStorage.

__abstractmethods__ = frozenset([])
__contains__(item)[source]

Check if an item is in the SampleCollection. The item can be a SampleId or SampleIndex. If the item is a slicing SampleIndex, then check if it would return any data (True) or no data (False)

Parameters:item (SampleId, SampleIndex) – the item to check for
Returns:flag indicating item is in the collections
Return type:bool
__delitem__(key)[source]

Remove an item from the SampleCollection

Parameters:key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to remove
__getitem__(item)[source]

Retrieve (a) SampleItem(s) from the SampleCollection using the SampleId or SampleIndex. If the item is a tuple, it should be valid tuple for constructing either a SampleId or SampleIndex.

Parameters:

item (SampleId, SampleIndex, or tuple) – the identifier of the item to retrieve

Returns:

the requested item

Return type:

SampleItem

Raises:
  • FastrTypeError – if the item parameter is of incorrect type
  • KeyError – if the item is not found
__init__(dimnames, parent)[source]

Createa a new SampleCollection

__iter__()[source]

Iterate over the indices

__len__()[source]

Get the number of samples in the SampleCollections.

__module__ = 'fastr.core.samples'
__repr__()[source]
__setitem__(key, value)[source]

Set an item to the SampleCollection. The key can be a SampleId, SampleIndex or a tuple containing a SampleId and SampleIndex. The value can be a SampleItem (with the SampleId and SampleIndex matching), a tuple with values (assuming no depending jobs), or a with a list of values and a set of depending jobs.

Parameters:
  • key (SampleId, SampleIndex, tuple of both, or SampleItem) – the key of the item to store
  • value (SampleItem, tuple of values, or tuple of tuple of values and set of depending jobs) – the value of the SampleItem to store
Raises:
  • FastrTypeError – if the key or value types are incorrect
  • FastrValueError – if the id or values are incorrectly formed
dimnames

The dimnames of the SampleCollection

fullid

The full defining ID for the SampleIdList

ndims

The number of dimensions in this SampleCollection

parent

The parent object holding the SampleCollection

size

The size of the SampleCollection. The size is the largest index in every dimension. For a 2D SampleCollection with 2 entries (10, 2) and (6, 6) the size would be (10, 6). As that is the rectangular grid that contains all data points.

class fastr.core.samples.SampleId[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using human readable strings

__module__ = 'fastr.core.samples'
class fastr.core.samples.SampleIndex[source]

Bases: fastr.core.samples.SampleBaseId

SampleId is an identifier for data using the location in the N-d data structure.

__module__ = 'fastr.core.samples'
__repr__()[source]

Get a string representation for the SampleIndex

Returns:the string representation
Return type:str
__str__()[source]

Get a string version for the SampleId, joins the SampleId with __ to create a single string version.

Returns:the string version
Return type:str
expand(size)[source]

Function expanding a slice SampleIndex into a list of non-slice SampleIndex objects

Parameters:size – the size of the collection to slice
isslice

Flag indicating that the SampleIndex is a slice (as opposed to a simple single index).

class fastr.core.samples.SampleItem[source]

Bases: fastr.core.samples.SampleItemBase

__module__ = 'fastr.core.samples'
static __new__(index, id_, data, jobs=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set of jobs on which this SampleItems data depends.
class fastr.core.samples.SampleItemBase[source]

Bases: tuple

This class represents a sample item, a combination of a SampleIndex, SampleID, value and required jobs. The SampleItem based on a named tuple and has some extra methods to combine SampleItems easily.

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters:other (SampleItem) – The other item to add to this one
Returns:the combined SampleItem
Return type:SampleItem
__dict__ = dict_proxy({'index': <property object at 0x7f3a22de00a8>, '__module__': 'fastr.core.samples', 'dimensionality': <property object at 0x7f3a22de0260>, 'jobs': <property object at 0x7f3a22de01b0>, '__new__': <staticmethod object at 0x7f3a22dcf558>, 'data': <property object at 0x7f3a22de0158>, '__doc__': '\n This class represents a sample item, a combination of a SampleIndex,\n SampleID, value and required jobs. The SampleItem based on a named\n tuple and has some extra methods to combine SampleItems easily.\n ', 'combine': <staticmethod object at 0x7f3a22dcf520>, '__repr__': <function __repr__ at 0x7f3a22e28cf8>, '__add__': <function __add__ at 0x7f3a22e28de8>, '__dict__': <attribute '__dict__' of 'SampleItemBase' objects>, 'cardinality': <property object at 0x7f3a22de0208>, '__getnewargs__': <function __getnewargs__ at 0x7f3a22e28d70>, 'id': <property object at 0x7f3a22de0100>})
__getnewargs__()[source]

Get new args gives the arguments to use to re-create this object, This is used for serialization.

__module__ = 'fastr.core.samples'
static __new__(index, id_, data, jobs=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set, tuple or list of jobs on which this SampleItems data depends.
__repr__()[source]

Get a string representation for the SampleItem

Returns:the string representation
Return type:str
cardinality

The cardinality of this Sample

static combine(*args)[source]

Combine a number of SampleItems into a new one.

Parameters:*args

the SampleItems to combine

Returns:the combined SampleItem
Return type:SampleItem

It is possible to both give multiple arguments, where each argument is a SampleItem, or a single argument which is an iterable yielding SampleItems.

# variables a, b, c, d are SampleItems to combine
# These are all valid ways of combining the SampleItems
comb1 = SampleItem.combine(a, b, c, d)  # Using multiple arguments
l = [a, b, c, d]
comb2 = SampleItem.combine(l)  # Using a list of arguments
comb3 = SampleItem.combine(l.__iter__())  # Using an iterator
data

The data SampleValue of the SampleItem

Returns:The value of this SampleItem
Return type:SampleValue
dimensionality

The dimensionality of this Sample

id

The sample id of the SampleItem

Returns:The id of this SampleItem
Return type:SampleId
index

The index of the SampleItem

Returns:The index of this SampleItem
Return type:SampleIndex
jobs

The set of the jobs on which this SampleItem depends

Returns:The jobs that generated the data for this SampleItem
Return type:set
class fastr.core.samples.SamplePayload[source]

Bases: fastr.core.samples.SampleItemBase

__add__(other)[source]

The addition operator combines two SampleItems into a single SampleItems. It merges the data and jobs and takes the index and id of the left-hand item.

Parameters:other (SampleItem) – The other item to add to this one
Returns:the combined SamplePayload
Return type:SamplePayload
__module__ = 'fastr.core.samples'
static __new__(index, id_, data, jobs=None)[source]

Create a SampleItem. Data should be an OrderedDict of tuples.

Parameters:
  • index (tuple, slice) – the sample index
  • id (SampleId) – the sample id
  • data (SampleValue, Mapping) – the data values
  • jobs (set) – set of jobs on which this SampleItems data depends.
class fastr.core.samples.SampleValue(*args, **kwargs)[source]

Bases: _abcoll.MutableMapping

A collection containing the content of a sample

__abstractmethods__ = frozenset([])
__add__(other)[source]
__delitem__(key)[source]
__getitem__(item)[source]
__getstate__()[source]
__init__(*args, **kwargs)[source]
__iter__()[source]
__len__()[source]
__module__ = 'fastr.core.samples'
__radd__(other)[source]
__repr__()[source]
__setitem__(key, value)[source]
__setstate__(state)[source]
cast(datatype)[source]
is_mapping
is_sequence
iterelements()[source]
mapping_part()[source]
sequence_part()[source]

serializable Module

This package contains the base class and meta class for all serializable objects in the Fastr system.

class fastr.core.serializable.PassThroughSerializer[source]

Bases: object

__dict__ = dict_proxy({'__module__': 'fastr.core.serializable', 'dumps': <staticmethod object at 0x7f3a22bd4670>, '__dict__': <attribute '__dict__' of 'PassThroughSerializer' objects>, 'loads': <staticmethod object at 0x7f3a22bd46a8>, '__weakref__': <attribute '__weakref__' of 'PassThroughSerializer' objects>, '__doc__': None})
__module__ = 'fastr.core.serializable'
__weakref__

list of weak references to the object (if defined)

static dumps(data)[source]
static loads(data)[source]
class fastr.core.serializable.Serializable[source]

Bases: object

Superclass for all classes that can be serialized.

SERIALIZERS = {'/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/Network.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a125a1290>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/Tool.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a2099d210>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/FastrInterface.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a204c86d0>}
__dict__ = dict_proxy({'load': <classmethod object at 0x7f3a22bd4788>, '__module__': 'fastr.core.serializable', 'SERIALIZERS': {'/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/Network.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a125a1290>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/Tool.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a2099d210>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/schemas/FastrInterface.schema.json': <fastr.utils.jsonschemaparser.Blueprinter object at 0x7f3a204c86d0>}, 'dump': <function dump at 0x7f3a22bdec08>, '_unicode_convert': <classmethod object at 0x7f3a22bd4980>, 'get_serializer': <classmethod object at 0x7f3a22bd46e0>, 'dumpfuncs': {'xml': <module 'fastr.utils.xmltodict' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/utils/xmltodict.pyc'>, 'pickle': <module 'pickle' from '/usr/lib/python2.7/pickle.pyc'>, 'json': <module 'json' from '/usr/lib/python2.7/json/__init__.pyc'>, 'dict': <class 'fastr.core.serializable.PassThroughSerializer'>, 'marshall': <module 'marshal' (built-in)>}, '_dumps': <classmethod object at 0x7f3a22bd4868>, '_dump': <classmethod object at 0x7f3a22bd4830>, '__dict__': <attribute '__dict__' of 'Serializable' objects>, 'loadf': <classmethod object at 0x7f3a22bd47f8>, '_unserialize': <classmethod object at 0x7f3a22bd4718>, '_load': <classmethod object at 0x7f3a22bd4910>, '_dumpf': <classmethod object at 0x7f3a22bd48a0>, '_loads': <classmethod object at 0x7f3a22bd48d8>, 'dumpf': <function dumpf at 0x7f3a22bdecf8>, '__weakref__': <attribute '__weakref__' of 'Serializable' objects>, '__doc__': '\n Superclass for all classes that can be serialized.\n ', 'dumps': <function dumps at 0x7f3a22bdec80>, '_loadf': <classmethod object at 0x7f3a22bd4948>, '_serialize': <function _serialize at 0x7f3a22bdeb90>, 'loads': <classmethod object at 0x7f3a22bd47c0>, '__getstate__': <function __getstate__ at 0x7f3a22bdea28>, 'createobj': <classmethod object at 0x7f3a22bd4750>})
__getstate__()[source]
__module__ = 'fastr.core.serializable'
__weakref__

list of weak references to the object (if defined)

classmethod createobj(state, _=None)[source]

Create object function for generic objects

Parameters:
  • cls – The class to create
  • state – The state to use to create the Link
  • network – the parent Network
Returns:

newly created Link

dump(file_handle, method='json', **kwargs)[source]

Dump the object to a file like object.

Parameters:
  • file_handle – file descriptor to write the data to
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • kwargs – extra arguments passed to the final serializer
dumpf(path, method=None, **kwargs)[source]

Dump the object to a file

Parameters:
  • path – path where to write the file
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • kwargs – extra arguments passed to the final serializer

Note

The dumpf function can determine the method based on the desired output filename. Also, if the filename ends with .gz it will continue search for another extension (so .json.gz could be found) and will then compress the result with gzip.

dumpfuncs = {'xml': <module 'fastr.utils.xmltodict' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/utils/xmltodict.pyc'>, 'pickle': <module 'pickle' from '/usr/lib/python2.7/pickle.pyc'>, 'json': <module 'json' from '/usr/lib/python2.7/json/__init__.pyc'>, 'dict': <class 'fastr.core.serializable.PassThroughSerializer'>, 'marshall': <module 'marshal' (built-in)>}
dumps(method='json', **kwargs)[source]

Dump the object to a string

Parameters:
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • kwargs – extra arguments passed to the final serializer
Returns:

serialization string

Return type:

str

classmethod get_serializer(filename=None)[source]
classmethod load(file_handle, method=None, network=None, **kwargs)[source]

Load the object from a file-like object

Parameters:
  • cls – class of the object
  • file_handle – file descriptor to write the data to
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • network – network in which to place the loaded object
  • kwargs – extra arguments passed to the final serializer
Returns:

newly created object

Warning

Unlike the loadf functions, this function does not automatically detect gzip compression. You read a gzip using the gzip.open method, but not but simply opening a stream and hopeing this function will function.

classmethod loadf(path, method=None, network=None, **kwargs)[source]

Load the object from a file

Parameters:
  • cls – class of the object
  • path – path where to write the file
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • network – network in which to place the loaded object
  • kwargs – extra arguments passed to the final serializer
Returns:

newly created object

Note

The loadf function can determine the method of loading based on the filename. Also it can automatically determine whether a file is gzipped.

classmethod loads(string, method=None, network=None, **kwargs)[source]

Load the object from a string

Parameters:
  • cls – class of the object
  • string (str) – the string containing the serialized data
  • method (str) – method of final serialization to use (e.g. json, xml, pickle)
  • network – network in which to place the loaded object
  • kwargs – extra arguments passed to the final serializer
Returns:

newly created object

target Module

The module containing the classes describing the targets.

class fastr.core.target.DockerTarget(binary, docker_image, **kwargs)[source]

Bases: fastr.core.target.Target

A tool target that is located in a Docker images. Can be run using docker-py.

__abstractmethods__ = frozenset([])
__enter__()[source]
__exit__(exc_type, exc_value, traceback)[source]
__init__(binary, docker_image, **kwargs)[source]

Define a new docker target.

Parameters:docker_image (str) – Docker image to use
__module__ = 'fastr.core.target'
container
docker_api = None

Docker api to use for docker target

monitor_docker(container, resources)[source]

Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.

Parameters:
  • process (subproces.Popen) – process to monitor
  • resources – list to append measurements to
run_command(command)[source]
class fastr.core.target.LocalBinaryTarget(binary, paths=None, environment_variables=None, initscripts=None, modules=None, interpreter=None, **kwargs)[source]

Bases: fastr.core.target.Target

A tool target that is a local binary on the system. Can be found using environmentmodules or vfs-path on the executing machine

DYNAMIC_LIBRARY_PATH_DICT = {'windows': 'PATH', 'darwin': 'DYLD_LIBRARY_PATH', 'linux': 'LD_LIBRARY_PATH'}
__abstractmethods__ = frozenset([])
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment

__init__(binary, paths=None, environment_variables=None, initscripts=None, modules=None, interpreter=None, **kwargs)[source]

Define a new local binary target. Must be defined either using paths and optionally environment_variables and initscripts, or enviroment modules.

__module__ = 'fastr.core.target'
call_subprocess(command)[source]

Call a subprocess with logging/timing/profiling

Parameters:command (list) – the command to execute
Returns:execution info
Return type:dict
monitor_process(process, resources)[source]

Monitor a process and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.

Parameters:
  • process (subproces.Popen) – process to monitor
  • resources – list to append measurements to
run_command(command)[source]
class fastr.core.target.ProcessUsageCollection[source]

Bases: _abcoll.Sequence

__abstractmethods__ = frozenset([])
__getitem__(item)[source]
__init__()[source]
__len__()[source]
__module__ = 'fastr.core.target'
aggregate(number_of_points)[source]
append(value)[source]
usage_type

alias of SystemUsageInfo

class fastr.core.target.SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Bases: tuple

__dict__ = dict_proxy({'__module__': 'fastr.core.target', '_make': <classmethod object at 0x7f3a226c2408>, 'timestamp': <property object at 0x7f3a226697e0>, '_replace': <function _replace at 0x7f3a2266d668>, 'read_bytes': <property object at 0x7f3a22669940>, '_asdict': <function _asdict at 0x7f3a2266d5f0>, 'rmem': <property object at 0x7f3a226698e8>, '__dict__': <property object at 0x7f3a22b287e0>, '__getnewargs__': <function __getnewargs__ at 0x7f3a2266d6e0>, 'write_bytes': <property object at 0x7f3a22669998>, '_fields': ('timestamp', 'cpu_percent', 'vmem', 'rmem', 'read_bytes', 'write_bytes'), '__new__': <staticmethod object at 0x7f3a226c2440>, 'cpu_percent': <property object at 0x7f3a22669838>, 'vmem': <property object at 0x7f3a22669890>, '__slots__': (), '__repr__': <function __repr__ at 0x7f3a2266d578>, '__getstate__': <function __getstate__ at 0x7f3a2266d758>, '__doc__': 'SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)'})
__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

__module__ = 'fastr.core.target'
static __new__(_cls, timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

Create new instance of SystemUsageInfo(timestamp, cpu_percent, vmem, rmem, read_bytes, write_bytes)

__repr__()

Return a nicely formatted representation string

__slots__ = ()
cpu_percent

Alias for field number 1

read_bytes

Alias for field number 4

rmem

Alias for field number 3

timestamp

Alias for field number 0

vmem

Alias for field number 2

write_bytes

Alias for field number 5

class fastr.core.target.Target(**kwargs)[source]

Bases: object

The abstract base class for all targets. Execution with a target should follow the following pattern:

>>> with Target() as target:
...     target.run_commmand(['sleep', '10'])
...     target.run_commmand(['sleep', '10'])
...     target.run_commmand(['sleep', '10'])

The Target context operator will set the correct paths/initialization. Within the context command can be ran and when leaving the context the target reverts the state before.

__abstractmethods__ = frozenset(['run_command'])
__dict__ = dict_proxy({'_abc_cache': <_weakrefset.WeakSet object at 0x7f3a2265fb90>, '__module__': 'fastr.core.target', '__enter__': <function __enter__ at 0x7f3a2266dc08>, '__metaclass__': <class 'abc.ABCMeta'>, '__abstractmethods__': frozenset(['run_command']), '_abc_registry': <_weakrefset.WeakSet object at 0x7f3a2265fb50>, '_NIPYPE_RUN_COMMAND': None, '_abc_negative_cache_version': 30, '__init__': <function __init__ at 0x7f3a2266db90>, '_abc_negative_cache': <_weakrefset.WeakSet object at 0x7f3a2265fbd0>, '__dict__': <attribute '__dict__' of 'Target' objects>, '__exit__': <function __exit__ at 0x7f3a2266dc80>, '__weakref__': <attribute '__weakref__' of 'Target' objects>, '__doc__': "\n The abstract base class for all targets. Execution with a target should\n follow the following pattern:\n\n >>> with Target() as target:\n ... target.run_commmand(['sleep', '10'])\n ... target.run_commmand(['sleep', '10'])\n ... target.run_commmand(['sleep', '10'])\n\n The Target context operator will set the correct paths/initialization.\n Within the context command can be ran and when leaving the context the\n target reverts the state before.\n ", 'run_command': <function run_command at 0x7f3a2266dcf8>})
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment where needed

__init__(**kwargs)[source]
__metaclass__

alias of ABCMeta

__module__ = 'fastr.core.target'
__weakref__

list of weak references to the object (if defined)

run_command(command)[source]

tool Module

A module to maintain a tool.

Exported classes:

  • Tool – A class encapsulating a tool.
  • ParameterDescription – The base class containing the shared description of a parameter (both input and ouput).
  • InputParameterDescription – A class containing the description of an input parameter.
  • Output ParameterDescription – A class containing the description of an output parameter.
class fastr.core.tool.Tool(doc=None)[source]

Bases: fastr.core.serializable.Serializable

The class encapsulating a tool.

__dataschemafile__ = 'Tool.schema.json'
__eq__(other)[source]

Compare two Tool instances with each other.

Parameters:other (Tool) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the Tool

Returns:the state of the object
Rtype dict:
__init__(doc=None)[source]

Create a new Tool :param doc: path of toolfile or a dict containing the tool data :type doc: str or dict

__module__ = 'fastr.core.tool'
__repr__()[source]

Get a string representation for the Tool. This will show the inputs and output defined in a table-like structure.

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Tool by the given state.

Parameters:state (dict) – The state to populate the object with
__str__()[source]

Get a string version for the Tool

Returns:the string version
Return type:str
authors = None

List of authors of the tool. These people wrapped the executable but are not responsible for executable itself.

cite = None

This holds the citation you should use when publishing something based on this Tool

command = None

Command is a dictionary contain information about the command which is called by this Tool: command[‘interpreter’] holds the (possible) interpreter to use command[‘targets’] holds a per os/arch dictionary of files that should be executed command[‘url’] is the webpage of the command to be called command[‘version’] is the version of the command used command[‘description’] can help a description of the command command[‘authors’] lists the original authors of the command

command_version
description = None

Description of the tool and it’s functionality

execute(payload=None, **kwargs)[source]

Execute a Tool given the payload for a single run

Parameters:payload – the data to execute the Tool with
Returns:The result of the execution
Return type:InterFaceResult
fullid

The full id of this tool

hash
help = None

Man page for the Tool. Here usage and examples can be described in detail

inputs
interface_class = None

Create the Interface based on the class specified in the tool file

name = None

Name of the tool, this should be a descriptive, human readable name.

namespace = None

The namespace this tools lives in, this will be set by the ToolManager on load

node_class = None

Class for of the Node to use

ns_id

The namespace and id of the Tool

outputs
path

The path of the directory in which the tool definition file was located.

references = None

A list of documents and in depth reading about the methods used in this tool

regex = None

Identifier for the tool

requirements = None

Requirements for this Tool

Warning

Not yet implemented

tags = None

List of tags for this tool

target

The OS and arch matched target definition.

test()[source]

Run the tests for this tool

test_spec

alias of TestSpecification

tests = None

Test for this tool. A test should be a collection of inputs, parameters and outputs to verify the proper functioning of the Tool.

The format of the tests is a list of namedtuples, that have 3 fields: - input: a dict of the input data - command: a list given the expected command-line arguments - output: a dict of the output data to validate

Warning

Not yet implemented

url = None

URL to website where this tool can be downloaded from

version = None

Version of the tool, not of the underlying software

toolmanager Module

This module contains the tool manager class

class fastr.core.toolmanager.ToolManager(path)[source]

Bases: fastr.core.objectmanager.ObjectManager

__abstractmethods__ = frozenset([])
__module__ = 'fastr.core.toolmanager'
get_object_version(obj)[source]
object_class
populate()[source]
toolversions(tool)[source]

Return a list of available versions for the tool

Parameters:tool – The tool to check the versions for. Can be either a Tool or a str.
Returns:List of version objects. Returns None when the given tool is not known.
fastr.core.toolmanager.toollist = ToolManager fastr.Sink v1.0 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/core/ioplugin.pyc fastr.Source v1.0 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/core/ioplugin.pyc fastr.flow.CrossValidation v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/flow/0.1/crossvalidation.xml fastr.math.Add v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/add.xml fastr.math.AddInt v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/addint.xml fastr.math.Divide v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/divide.xml fastr.math.IntegerDivide v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/intdivide.xml fastr.math.Max v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/max.xml fastr.math.Min v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/min.xml fastr.math.Multiply v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/multiply.xml fastr.math.Subtract v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/subtract.xml fastr.math.Sum v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/math/0.1/sum.xml fastr.util.Delay v0.1 : /home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/1.1.1/local/lib/python2.7/site-packages/fastr-1.1.1-py2.7.egg/fastr/resources/tools/fastr/util/0.1/delay.xml

The fastr toollist

updateable Module

This module contains both the Updateable class and UpdateableMeta meta-class for objects which support updates within the fastr system

class fastr.core.updateable.Updateable[source]

Bases: object

Super class for all classes that can be updated and have a status. These objects can be valid/invalid and ready/not-ready depending on their state. These states are set by the function update. This allows for interactively checking the network.

__abstractmethods__ = frozenset(['_update'])
__dict__ = dict_proxy({'__module__': 'fastr.core.updateable', '_update': <function _update at 0x7f3a22bed938>, '__metaclass__': <class 'fastr.core.updateable.UpdateableMeta'>, '__updatefunc__': <function update at 0x7f3a22bed8c0>, '_abc_negative_cache': <_weakrefset.WeakSet object at 0x7f3a22be9ed0>, 'ready': <property object at 0x7f3a22be6d08>, '__dict__': <attribute '__dict__' of 'Updateable' objects>, '__weakref__': <attribute '__weakref__' of 'Updateable' objects>, '__init__': <function __init__ at 0x7f3a22bed5f0>, '_abc_cache': <_weakrefset.WeakSet object at 0x7f3a22be9e50>, '__abstractmethods__': frozenset(['_update']), 'valid': <property object at 0x7f3a22be6d60>, '__getstate__': <function __getstate__ at 0x7f3a22bed668>, '__doc__': '\n Super class for all classes that can be updated and have a status.\n These objects can be valid/invalid and ready/not-ready depending on their\n state. These states are set by the function update. This allows for\n interactively checking the network.\n ', '__setstate__': <function __setstate__ at 0x7f3a22bed6e0>, 'update': <function update at 0x7f3a22bed8c0>, '__updating__': True, 'messages': <property object at 0x7f3a22be6cb0>, '_abc_negative_cache_version': 30, '__updatetriggers__': [], '__updateinprogress__': <thread.lock object at 0x7f3a22c0b610>, '_abc_registry': <_weakrefset.WeakSet object at 0x7f3a22be9dd0>})
__getstate__()[source]

Retrieve the state of the object, make sure the status is not part of the description as it will not be valid after re-creating the object.

Returns:the state of the object
Rtype dict:
__init__()[source]

Constructor, creates the status field

Returns:newly created object
__metaclass__

alias of UpdateableMeta

__module__ = 'fastr.core.updateable'
__setstate__(state)[source]

Set the state of the object by the given state. This adds a clean status field, making sure it is not unintended, outdated information from before serialization.

Parameters:state (dict) – The state to populate the object with
__updatefunc__(key=None, forward=True, backward=False)

Default function for updating, it can be called without key to have a new update started with a new key.

Parameters:
  • key (int) – a key for this update, should be different than the last update key
  • forward (bool) – flag indicating to update forward in the network
  • backward (bool) – flag indicating to update backward in the network
__updateinprogress__ = <thread.lock object>

Lock to avoid multiple updates happening at the same time

__updatetriggers__ = []

Which methods need to be wrapped to trigger an update. Override this value to have the functions automatically wrapped. E.g. __update_triggers__ = ['append', 'insert', '__setitem__'] to have these functions wrapped.

__updating__ = True

Flag to indicate that this object is allowed to update

__weakref__

list of weak references to the object (if defined)

messages

The messages of the last update

ready

Flag indicating that the object is ready

update(key=None, forward=True, backward=False)[source]

Default function for updating, it can be called without key to have a new update started with a new key.

Parameters:
  • key (int) – a key for this update, should be different than the last update key
  • forward (bool) – flag indicating to update forward in the network
  • backward (bool) – flag indicating to update backward in the network
valid

Flag indicating that the object is valid

class fastr.core.updateable.UpdateableMeta[source]

Bases: abc.ABCMeta

A metaclass for objects which are updateable and need some methods/properties to trigger an update.

__module__ = 'fastr.core.updateable'
static __new__(mcs, name, parents, dct)[source]
classmethod calcmro(mcs, bases)[source]

Calculate the Method Resolution Order of bases using the C3 algorithm.

Suppose you intended creating a class K with the given base classes. This function returns the MRO which K would have, excluding K itself (since it doesn’t yet exist), as if you had actually created the class.

Another way of looking at this, if you pass a single class K, this will return the linearization of K (the MRO of K, including itself).

Parameters:bases – the list of bases for which create the MRO
Returns:the list representing the entire MRO, except the (non-existing) class itself
Note: Taken from http://code.activestate.com/recipes/577748-calculate-the-mro-of-a-class/
Created by Steven D’Aprano and licensed under the MIT license
classmethod find_member(mcs, name, parents, dct)[source]

Find a member of the class in the same way as Python would if it had a given dict and set of bases

Parameters:
  • mcs – metaclass at work
  • name – name of the class to be created
  • parents – list of the bases for the new class
  • dct – the dict of the class being created
Returns:

the firstly resolved member or None if nothing found

static updatetrigger(fnc)[source]

Function decorator to make a function trigger an update after being called. This is a way to easily have function trigger an update after setting a value without writing tons of wrapper functions. The function keeps the original docstring and appends a note to it.

version Module

Module containing the class that represent versions

class fastr.core.version.Version[source]

Bases: tuple

Class representing a software version definition. Allows for sorting and extraction of parts.

__dict__ = dict_proxy({'status': <property object at 0x7f3a22e69d08>, '__module__': 'fastr.core.version', '__new__': <staticmethod object at 0x7f3a22e5a948>, 'extra': <property object at 0x7f3a22e69db8>, '__str__': <function __str__ at 0x7f3a22e006e0>, 'major': <property object at 0x7f3a22e69e68>, '__dict__': <attribute '__dict__' of 'Version' objects>, '__repr__': <function __repr__ at 0x7f3a22e00758>, 'extra_string': <property object at 0x7f3a22e69d60>, 'version_matcher': <_sre.SRE_Pattern object at 0x1d90760>, 'build': <property object at 0x7f3a22e69ba8>, '__doc__': '\n Class representing a software version definition. Allows for sorting and\n extraction of parts.\n ', 'minor': <property object at 0x7f3a22e69e10>, 'suffix': <property object at 0x7f3a22e69b50>})
__module__ = 'fastr.core.version'
static __new__(*version)[source]

Class containing a version

Can be constructed by:

Version( 'major.$minor.$extra[0].$extra[1]$seperator$status$build$suffix' )
Version( major, minor, extra, status, build, suffix, seperator )
Version( (major, minor, extra, status, build, suffix, seperator) )
Version( [major, minor, extra, status, build, suffix, seperator] )
Parameters:
  • major (int) – interger giving major version
  • minor (int) – is an integer (required)
  • extra (list of int) – is a list of integers
  • status (str) – can be “a”, “alpha”, “b”, “beta”, “rc”, or “r”
  • build (int) – is an integer
  • suffix (str) – can contain any combination of alpha-numeric character and ”._-“
  • seperator (str) – is any of ”.”, “-”, or “_”, which is located between $extra and $build

Note

The method based on strings is the recommended method. For strings the major and minor version are required, where for tuple and list constructors all seven elements are optional.

Examples:

>>> a = Version('0.1')
>>> print(tuple(a))
(0, 1, None, None, None, '', None)
>>> b = Version('2.5.3-rc2')
>>> print(tuple(b))
(2, 5, [3], 'rc', 2, '', '-')
>>> c = Version('1.2.3.4.5.6.7-beta8_with_suffix')
>>> print(tuple(c))
(1, 2, [3, 4, 5, 6, 7], 'beta', 8, '_with_suffix', '-')
__repr__()[source]

Return a in-editor representation of the version

__str__()[source]

Return a string representation of the version

build

the build number, this is following the status (e.g. for 3.2-beta4, this would be 4)

extra

extra version extension as a list

extra_string

extra version extension as a string

major

major version

minor

minor version

status

the status of the version (a, alpha, b, beta, rc or r)

suffix

the remainder of the version which was not formatted in a known way

version_matcher = <_sre.SRE_Pattern object at 0x1d90760>

vfs Module

This module contains the virtual file system code. This is both an internally used object as well as an IOPlugin.