fastr Package

fastr Package

FASTR is a top level package which includes all parts required to create networks and edit networks.

class fastr.__init__.Network[source]

The class representing a Network, this is in fact a reference to fastr.core.network.Network.

class fastr.__init__.NodeRun

The class representing a NodeRun, this is in fact a reference to fastr.core.node.NodeRun.

The class representing a Link, this is in fact a reference to fastr.core.link.Link.

class fastr.__init__.SourceNodeRun

The class representing a data source, this is in fact a reference to fastr.core.node.SourceNodeRun.

class fastr.__init__.SinkNodeRun

The class representing a data sink, this is in fact a reference to fastr.core.node.SinkNodeRun.

class fastr.__init__.ConstantNodeRun

The class representing a constant data source, this is in fact a reference to fastr.core.node.ConstantNodeRun.

fastr.__init__.toollist

A fastr.core.toolmanager.ToolManager containing all Tools known to the FASTR environment. The toollist can be accessed in a similar way to a dict. Indexing with a tool id will return the newest version of the Tool. If a specific version of the tool is required a tuple can be used as the index:

>>> import fastr
>>> fastr.toollist['testtool']
<Tool: testtool version: 4.2>
>>> fastr.toollist['testtool', '2.0']
<Tool: testtool version: 2.0>
fastr.__init__.typelist

A fastr.core.datatypemanager.DataTypeManager containing all Types known to the FASTR environment. This is usuable as a dict where the key is the datatype id and the value is the datatype itself.

class fastr.__init__.Network(id_='unnamed_network', version=None, filename=None)[source]

Bases: fastr.core.serializable.Serializable

The NetworkRun contains the entire Run state for a Network execution. It has a working copy of the network, but also includes all temporary data required for the execution. These objects are meant to be single use.

NETWORK_DUMP_FILE_NAME = '__fastr_network__.json'
SINK_DUMP_FILE_NAME = '__sink_data__.json'
SOURCE_DUMP_FILE_NAME = '__source_data__.pickle.gz'
__dataschemafile__ = 'Network.schema.json'
__eq__(other)[source]

Compare two Networks and see if they are equal.

Parameters:other (Network) –
Returns:flag indicating that the Networks are the same
Return type:bool
__getitem__(item)[source]

Get an item by its fullid. The fullid can point to a link, node, input, output or even subinput/suboutput.

Parameters:item (str,unicode) – fullid of the item to retrieve
Returns:the requested item
__getstate__()[source]

Retrieve the state of the Network

Returns:the state of the object
Rtype dict:
__init__(id_='unnamed_network', version=None, filename=None)[source]

Create a new, empty Network

Parameters:name (str) – name of the Network
Returns:newly created Network
Raises:OSError – if the tmp mount in the config is not a writable directory
__module__ = 'fastr.core.network'
__ne__(other)[source]

Tests for non-equality, this is the negated version __eq__

__repr__() <==> repr(x)[source]
__setstate__(state)[source]

Set the state of the Network by the given state. This completely overwrites the old state!

Parameters:state (dict) – The state to populate the object with
Returns:None

Add a Link to the Network. Make sure the link is in the link list and the link parent is set to this Network

Parameters:

link (Link) – link to add

Raises:
add_node(node)[source]

Add a Node to the Network. Make sure the node is in the node list and the node parent is set to this Network

Parameters:node (Node) – node to add
Raises:FastrTypeError – if node is incorrectly typed
add_stepid(stepid, node)[source]

Add a Node to a specific step id

Parameters:
  • stepid (str) – the stepid that the node will be added to
  • node (Node) – the node to add to the stepid
check_id(id_)[source]

Check if an id for an object is valid and unused in the Network. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:
create_constant(datatype, data, id_=None, stepid=None, nodegroup=None, sourcegroup=None)[source]

Create a ConstantNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the constant node
  • data (datatype or list of datatype) – The data to hold in the constant node
  • id (str) – The id of the constant node to be created
  • stepid (str) – The stepid to add the created constant node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created constant node

Return type:

ConstantNode

Create a link between two Nodes and add it to the current Network.

Parameters:
  • source (BaseOutput) – the output that is the source of the link
  • target (BaseInput) – the input that is the target of the link
  • id (str) – the id of the link
Returns:

the created link

Type:

Link

create_macro(network, id_=None)[source]
create_node(tool, id_=None, stepid=None, cores=None, memory=None, walltime=None, nodegroup=None)[source]

Create a Node in this Network. The Node will be automatically added to the Network.

Parameters:
  • tool (Tool) – The Tool to base the Node on
  • id (str) – The id of the node to be created
  • stepid (str) – The stepid to add the created node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created node

Return type:

Node

create_reference(source_data, output_directory)[source]
create_sink(datatype, id_=None, stepid=None)[source]

Create a SinkNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the sink node
  • id (str) – The id of the sink node to be created
  • stepid (str) – The stepid to add the created sink node to
Returns:

the newly created sink node

Return type:

SinkNode

create_source(datatype, id_=None, stepid=None, nodegroup=None, sourcegroup=None)[source]

Create a SourceNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the source source_node
  • id (str) – The id of the source source_node to be created
  • stepid (str) – The stepid to add the created source source_node to
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
  • sourcegroup (str) – DEPRECATED! The nodegroup this SourceNode will be added to
Returns:

the newly created source source_node

Return type:

SourceNode

draw(name=None, image_format=None, draw_dimensions=False, context=None, graph=None, expand_macro=False)[source]
draw_network(name='network_layout', img_format='svg', draw_dimension=False, expand_macro=False)[source]

Output a dot file and try to convert it to an image file.

Parameters:img_format (str) – extension of the image format to convert to
Returns:path of the image created or None if failed
Return type:str or None
execute(sourcedata, sinkdata, blocking=True, **kwargs)[source]
fullid

The fullid of the Network, within the network scope

global_id

The global id of the Network, this is different for networks used in macronodes, as they still have parents.

id

The id of the Network. This is a read only property.

is_valid()[source]
namespace = None

The namespace this network lives in, this will be set by the NetworkManager on load

nodegroups

Give an overview of the nodegroups in the network

remove(value)[source]

Remove an item from the Network.

Parameters:value (Node or Link) – the item to remove
classmethod test(reference_data_dir, network=None, source_data=None, force_remove_temp=False)[source]

Execute the network with the source data specified and test the results against the refence data. This effectively tests the network execution.

Parameters:
  • reference_data_dir (str) – The path or vfs url of reference data to compare with
  • source_data (dict) – The source data to use
class fastr.__init__.Link(source, target, parent, id_=None, collapse=None, expand=None)[source]

Bases: fastr.core.dimension.HasDimensions, fastr.core.updateable.Updateable, fastr.core.serializable.Serializable

Class for linking outputs (BaseOutput) to inputs (BaseInput)

Examples:

>>> import fastr
>>> network = fastr.Network()
>>> link1 = network.create_link( n1.ouputs['out1'], n2.inputs['in2'] )

link2 = Link()
link2.source = n1.ouputs['out1']
link2.target = n2.inputs['in2']
__abstractmethods__ = frozenset([])
__dataschemafile__ = 'Link.schema.json'
__eq__(other)[source]

Test for equality between two Links

Parameters:other (Link) – object to test against
Returns:True for equality, False otherwise
Return type:bool
__getstate__()[source]

Retrieve the state of the Link

Returns:the state of the object
Rtype dict:
__init__(source, target, parent, id_=None, collapse=None, expand=None)[source]

Create a new Link in a Network.

Parameters:
  • source (BaseOutput) – the source output
  • target (BaseInput) – the target input
  • parent (Network or None) – the parent network
  • id (str or None) – the id of the link, if no id_ is given, the id will be in the form of “link_{:d}”
  • collapse (int, str, or tuple of int/str) – the dimensions that the link has to collapse on
  • expand (bool) – Does this link need to expand the cardinality into a new sample dimension
Returns:

newly created Link

Raises:
__module__ = 'fastr.core.link'
__repr__()[source]

Get a string representation for the Link

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Link by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
Raises:FastrValueError – if the parent network is not set
cardinality(index=None)[source]

Cardinality for a Link is given by source Output and the collapse/expand settings

Parameters:key (SampleIndex) – key for a specific sample (can be only a sample index!)
Returns:the cardinality
Return type:int, sympy.Symbol
Raises:FastrIndexError – if the index length does not match the number of dimension in the data
collapse

The converging dimensions of this link. Collapsing changes some dimensions of sample lists into cardinality, reshaping the data.

Collapse can be set to a tuple or an int/str, in which case it will be automatically wrapped in a tuple. The int will be seen as indices of the dimensions to collapse. The str will be seen as the name of the dimensions over which to collapse.

Raises:FastrTypeError – if assigning a collapse value of a wrong type
collapse_indexes

The converging dimensions of this link as integers. Dimension names are replaces with the corresponding int.

Collapsing changes some dimensions of sample lists into cardinality, reshaping the data

classmethod createobj(state, network=None)[source]

Create object function for Link

Parameters:
  • cls – The class to create
  • state – The state to use to create the Link
  • network – the parent Network
Returns:

newly created Link

destroy()[source]

The destroy function of a link removes all default references to a link. This means the references in the network, input and output connected to this link. If there is no references in other places in the code, it will destroy the link (reference count dropping to zero).

This function is called when a source for an input is set to another value and the links becomes disconnected. This makes sure there is no dangling links.

dimensions

The dimensions of the data delivered by the link. This can be different from the source dimensions because the link can make data collapse or expand.

draw(context, graph)[source]
expand

Flag indicating that the link will expand the cardininality into a new sample dimension to be created.

fullid

The full defining ID for the Input

parent

The Network to which this Link belongs.

source

The source BaseOutput of the Link. Setting the source will automatically register the Link with the source BaseOutput. Updating source will also make sure the Link is unregistered with the previous source.

Raises:FastrTypeError – if assigning a non BaseOutput
status
target

The target BaseInput of the Link. Setting the target will automatically register the Link with the target BaseInput. Updating target will also make sure the Link is unregistered with the previous target.

Raises:FastrTypeError – if assigning a non BaseInput
class fastr.__init__.Node(tool, id_=None, node_class=None, parent=None, cores=None, memory=None, walltime=None, nodegroup=None)[source]

Bases: fastr.core.dimension.HasDimensions, fastr.core.updateable.Updateable, fastr.core.serializable.Serializable

The class encapsulating a node in the network. The node is responsible for setting and checking inputs and outputs based on the description provided by a tool instance.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'Node.schema.json'
__eq__(other)[source]

Check two Node instances for equality.

Parameters:other (Node) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the Node

Returns:the state of the object
Rtype dict:
__init__(tool, id_=None, node_class=None, parent=None, cores=None, memory=None, walltime=None, nodegroup=None)[source]

Instantiate a node.

Parameters:
  • tool (Tool) – The tool to base the node on
  • id (str) – the id of the node
  • node_class (str) – The class of the NodeRun to create (e.g. SourceNodeRun, NodeRun)
  • parent (Network) – the parent network of the node
  • cores (int) – number of cores required for executing this Node
  • memory (str) – amount of memory required in the form d+[mMgG] where M is for megabyte and G for gigabyte
  • walltime (str) – amount of time required in second or in the form HOURS:MINUTES:SECOND
Returns:

the newly created Node

__metaclass__

alias of abc.ABCMeta

__module__ = 'fastr.core.node'
__ne__(other)[source]

Check two Node instances for inequality. This is the inverse of __eq__

Parameters:other (Node) – the other instances to compare to
Returns:True if unequal, False otherwise
__repr__()[source]

Get a string representation for the Node

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Node

Returns:the string version
Return type:str
blocking

Indicate that the results of this Node cannot be determined without first executing the Node, causing a blockage in the creation of jobs. A blocking Nodes causes the Chunk borders.

classmethod createobj(state, network=None)[source]

Create object function for generic objects

Parameters:
  • cls – The class to create
  • state – The state to use to create the Link
  • network – the parent Network
Returns:

newly created Link

dimensions
dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

draw(context, graph, color=None)[source]
draw_id(context)[source]
find_source_index(target_index, target, source)[source]
fullid

The full defining ID for the Node inside the network

get_sourced_nodes()[source]

A list of all Nodes connected as sources to this Node

Returns:list of all nodes that are connected to an input of this node
global_id

The global defining ID for the Node from the main network (goes out of macro nodes to root network)

id

The id of the Node

input_groups
A list of input groups for this Node. An input group is InputGroup
object filled according to the Node
inputs = None

A list of inputs of this Node

listeners

All the listeners requesting output of this node, this means the listeners of all Outputs and SubOutputs

merge_dimensions
name

Name of the Tool the Node was based on. In case a Toolless Node was used the class name is given.

nodegroup
outputs = None

A list of outputs of this Node

outputsize

The size of output of this SourceNode

parent

The parent is the Network this Node is part of

required_cores

Number of cores required for the execution of this Node

required_memory

Amount of memory required for the execution of this Node. Follows the format d+[mMgG] so 500M or 4g would be valid ways to specify 500 megabytes or 4 gigabyte of memory.

required_time

Amount of time required for the execution of this Node. Follows the format of a number of second or H:M:S, with H the number of hours, M the number of minutes and S the number of seconds.

status
tool
update_input_groups()[source]

Update all input groups in this node

class fastr.__init__.ConstantNode(datatype, data, id_=None, parent=None, nodegroup=None)[source]

Bases: fastr.core.node.SourceNode

Class encapsulating one output for which a value can be set. For example used to set a scalar value to the input of a node.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'ConstantNode.schema.json'
__getstate__()[source]

Retrieve the state of the ConstantNode

Returns:the state of the object
Rtype dict:
__init__(datatype, data, id_=None, parent=None, nodegroup=None)[source]

Instantiation of the ConstantNode.

Parameters:
  • datatype – The datatype of the output.
  • data – the prefilled data to use.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a constant using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> source = network.create_source(datatype=fastr.typelist['ITKImageFile'], id_='sourceN')

or alternatively create a constant node by assigning data to an item in an InputDict:

>>> node_a.inputs['in'] = ['some', 'data']

which automatically creates and links a ConstantNode to the specified Input

__module__ = 'fastr.core.node'
__setstate__(state)[source]

Set the state of the ConstantNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
data

The data stored in this constant node

draw(context, graph, color=None)[source]
set_data(data=None, ids=None)[source]

Set the data of this constant node in the correct way. This is mainly for compatibility with the parent class SourceNode

Parameters:
  • data (dict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
class fastr.__init__.SourceNode(datatype, id_=None, parent=None, nodegroup=None)[source]

Bases: fastr.core.node.FlowNode

Class providing a connection to data resources. This can be any kind of file, stream, database, etc from which data can be received.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'SourceNode.schema.json'
__getstate__()[source]

Retrieve the state of the SourceNode

Returns:the state of the object
Rtype dict:
__init__(datatype, id_=None, parent=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The (id of) the datatype of the output.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a source using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> source = network.create_source(datatype=fastr.typelist['ITKImageFile'], id_='sourceN')
__module__ = 'fastr.core.node'
__setstate__(state)[source]

Set the state of the SourceNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
datatype

The datatype of the data this source supplies.

dimensions

The dimensions in the SourceNode output. These will be reflected in the SampleIdLists.

draw(context, graph, color=None)[source]
nodegroup
output

Shorthand for self.outputs['output']

set_data(data, ids=None)[source]

Set the data of this source node.

Parameters:
  • data (dict, OrderedDict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
sourcegroup
valid

This does nothing. It only overloads the valid method of Node(). The original is intended to check if the inputs are connected to some output. Since this class does not implement inputs, it is skipped.

class fastr.__init__.SinkNode(datatype, id_=None, parent=None, nodegroup=None)[source]

Bases: fastr.core.node.Node

Class which handles where the output goes. This can be any kind of file, e.g. image files, textfiles, config files, etc.

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'SinkNode.schema.json'
__getstate__()[source]

Retrieve the state of the Node

Returns:the state of the object
Rtype dict:
__init__(datatype, id_=None, parent=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The datatype of the output.
  • id – the id of the node to create
Returns:

newly created sink node

usage example:

>>> import fastr
>>> network = fastr.Network()
>>> sink = network.create_sink(datatype=fastr.typelist['ITKImageFile'], id_='SinkN')
__module__ = 'fastr.core.node'
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
datatype

The datatype of the data this sink can store.

draw(context, graph, color=None)[source]
input

The default input of the sink Node

configmanager Module

This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.

class fastr.configmanager.Config(*configfiles)[source]

Bases: object

Class contain the fastr configuration

DEFAULT_FIELDS = {'debug': (<type 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<type 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'filesynchelper_url': (<type 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<type 'str'>, 'non_failed', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'non_failed', <function <lambda> at 0x7fdcee12b1b8>), 'logdir': (<type 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<type 'dict'>, {}, 'Python logger config'), 'loglevel': (<type 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<type 'str'>, 'default', 'Type of logging to use'), 'mounts': (<type 'dict'>, {'tmp': '/tmp', 'home': '/home/docs/', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/examples/data'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'home': '~/', 'example_data': '$systemdir/examples/data'}), 'networks_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'pim_host': (<type 'str'>, '', 'Host of the PIM server to report to'), 'plugins_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<type 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<type 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'resourcesdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<type 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr', 'Fastr installation directory'), 'tools_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<type 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<type 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<type 'str'>, 'localhost', 'The hostname to expose the web app for'), 'web_port': (<type 'int'>, 5000, 'The port to expose the web app on'), 'web_secret_key': (<type 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app')}
__dict__ = dict_proxy({'plugins_path': <property object>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'warn_develop': <property object>, 'DEFAULT_FIELDS': {'resourcesdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'plugins_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'executionscript': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'protected_modules': (<type 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'warn_develop': (<type 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'preferred_types': (<type 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'pim_host': (<type 'str'>, '', 'Host of the PIM server to report to'), 'logtype': (<type 'str'>, 'default', 'Type of logging to use'), 'systemdir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr', 'Fastr installation directory'), 'execution_plugin': (<type 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'logdir': (<type 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'web_secret_key': (<type 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app'), 'types_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'logging_config': (<type 'dict'>, {}, 'Python logger config'), 'filesynchelper_url': (<type 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'debug': (<type 'bool'>, False, 'Flag to enable/disable debugging'), 'tools_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'source_job_limit': (<type 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'web_port': (<type 'int'>, 5000, 'The port to expose the web app on'), 'loglevel': (<type 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'web_hostname': (<type 'str'>, 'localhost', 'The hostname to expose the web app for'), 'networks_path': (<type 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'userdir': (<type 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'schemadir': (<type 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'mounts': (<type 'dict'>, {'tmp': '/tmp', 'home': '/home/docs/', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/examples/data'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'home': '~/', 'example_data': '$systemdir/examples/data'}), 'job_cleanup_level': (<type 'str'>, 'non_failed', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'non_failed', <function <lambda>>)}, 'slurm_partition': <property object>, 'preferred_types': <property object>, 'pim_host': <property object>, 'examplesdir': <property object>, '_field_property': <staticmethod object>, 'logdir': <property object>, 'get_field': <function get_field>, 'read_config': <function read_config>, '__doc__': '\n Class contain the fastr configuration\n ', 'logging_config': <property object>, 'tools_path': <property object>, '_create_field_properties': <classmethod object>, '_update_logging': <function _update_logging>, 'source_job_limit': <property object>, 'web_hostname': <property object>, 'schemadir': <property object>, 'debug': <property object>, 'x': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins', 'resourcesdir': <property object>, '__module__': 'fastr.configmanager', 'executionscript': <property object>, 'protected_modules': <property object>, 'web_url': <function web_url>, '__init__': <function __init__>, '_deep_update': <function _deep_update>, 'logtype': <property object>, 'systemdir': <property object>, 'register_fields': <function register_fields>, 'set_field': <function set_field>, 'execution_plugin': <property object>, 'job_cleanup_level': <property object>, 'process_pool_worker_number': <property object>, 'web_secret_key': <property object>, 'types_path': <property object>, 'filesynchelper_url': <property object>, 'mounts': <property object>, 'slurm_job_check_interval': <property object>, 'web_port': <property object>, 'read_config_string': <function read_config_string>, 'loglevel': <property object>, 'networks_path': <property object>, 'userdir': <property object>, '__repr__': <function __repr__>})
__init__(*configfiles)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

__module__ = 'fastr.configmanager'
__repr__() <==> repr(x)[source]
__weakref__

list of weak references to the object (if defined)

debug
examplesdir
execution_plugin
executionscript
filesynchelper_url
get_field(item)[source]
job_cleanup_level
logdir
logging_config
loglevel
logtype
mounts
networks_path
pim_host
plugins_path
preferred_types
process_pool_worker_number
protected_modules
read_config(filename)[source]

Read a configuration and update the configuration object accordingly

Parameters:filename – the configuration file to read
read_config_files = None

Trace of the config files read by this object

read_config_string(value)[source]
register_fields(fields_spec)[source]

Register extra fields to the configuration manager.

resourcesdir
schemadir
set_field(item, value)[source]
slurm_job_check_interval
slurm_partition
source_job_limit
systemdir
tools_path
types_path
userdir
warn_develop
web_hostname
web_port
web_secret_key
web_url()[source]

Construct a fqdn from the web[‘hostname’] and web[‘port’] settings. :return: FQDN :rtype: str

x = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins'
class fastr.configmanager.EmptyDefault(data=None)[source]

Bases: object

Empty defaultdict.

__add__(right)[source]
__delitem__(key)[source]
__dict__ = dict_proxy({'extend': <function extend>, '__delitem__': <function __delitem__>, '__module__': 'fastr.configmanager', '__getitem__': <function __getitem__>, 'asdict': <function asdict>, 'update': <function update>, '__iadd__': <function __iadd__>, '__radd__': <function __radd__>, '__setitem__': <function __setitem__>, '__init__': <function __init__>, '__add__': <function __add__>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, 'prepend': <function prepend>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>, '__doc__': ' Empty defaultdict. ', 'append': <function append>, 'aslist': <function aslist>})
__getitem__(item)[source]
__iadd__(right)[source]
__init__(data=None)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

__module__ = 'fastr.configmanager'
__radd__(other)[source]
__setitem__(key, value)[source]
__weakref__

list of weak references to the object (if defined)

append(value)[source]
asdict()[source]
aslist()[source]
extend(other)[source]
prepend(value)[source]
update(other)[source]

datatypes Module

The datatypes module holds all DataTypes generated by fastr and all the base classes for these datatypes.

class fastr.datatypes.AnalyzeImageFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
classmethod content(invalue, outvalue=None)[source]

Give the contents of a URLType, this is generally useful for filetypes that consists of multiple files (e.g. AnalyzeImageFile, DICOM). The value will indicate the main file, and the contents function can determine all files that form a single data value.

Parameters:
  • inval – a value to figure out contents for this type
  • outval – the place where the copy should point to
Returns:

a list of all files part of the value (e.g. header and data file)

Return type:

list

description = 'Analyze Image file formate'
extension = 'hdr'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/AnalyzeImageFile.py'
module = <module 'AnalyzeImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/AnalyzeImageFile.pyc'>
class fastr.datatypes.AnyFile(value=None)[source]

Bases: fastr.datatypes.TypeGroup

Special Datatype in fastr that is a TypeGroup with all known DataTypes as its members.

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'TypeGroup AnyFile\nAnyFile (AnyFile) is a group of consisting of all URLTypes known by fastr, currently:\n - <URLType: NiftiImageFileCompressed class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: ProvNFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: NiftiImageFileUncompressed class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: FilePrefix class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: MetaImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: Directory class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: AnalyzeImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: TifImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: TxtFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: NrrdImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: JsonFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>'
class fastr.datatypes.AnyType(value=None)[source]

Bases: fastr.datatypes.TypeGroup

Special Datatype in fastr that is a TypeGroup with all known DataTypes as its members.

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'TypeGroup AnyType\nAnyType (AnyType) is a group of consisting of all DataTypes known by fastr, currently:\n - <URLType: ProvNFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: NiftiImageFileCompressed class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <DataType: Deferred class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: NiftiImageFileUncompressed class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <ValueType: Float class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: MetaImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <ValueType: Int class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: NrrdImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <ValueType: Boolean class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <ValueType: UnsignedInt class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: Directory class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: JsonFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: TifImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <ValueType: String class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: TxtFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: FilePrefix class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>\n - <URLType: AnalyzeImageFile class [\x1b[37m\x1b[42m\x1b[1mLoaded\x1b[0m]>'
class fastr.datatypes.BaseDataType(value=None, format_=None)[source]

Bases: fastr.core.baseplugin.BasePlugin

The base class for all datatypes in the fastr type system.

__abstractmethods__ = frozenset(['__init__'])
__eq__(other)[source]

Test the equality of two DataType objects

Parameters:other (DataType) – the object to compare against
Returns:flag indicating equality
Return type:bool
__getstate__()[source]
__init__(value=None, format_=None)[source]

The BaseDataType constructor.

Parameters:
  • value – value to assign to the new BaseDataType object
  • format – the format used for the ValueType
Returns:

new BaseDataType object

Raises:

FastrNotImplementedError – if id, name, version or description is None

__module__ = 'fastr.datatypes'
__ne__(other)[source]

Test if two objects are not equal. This is by default done by negating the __eq__ operator

Parameters:other (DataType) – the object to compare against
Returns:flag indicating equality
Return type:bool
__reduce_ex__(*args, **kwargs)[source]

helper for pickle

__repr__()[source]

Returns string representation of the BaseDataType

Returns:string represenation
Return type:str
__setstate__(state)[source]
__str__()[source]

Returns the string version of the BaseDataType

Returns:string version
Return type:str
checksum()[source]

Generate a checksum for the value of this DataType

Returns:the checksum of the value
Return type:str
description = ''

Description of the DataType

dot_extension = None
extension = None

Extension related to the Type

filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/datatypes.pyc'
fullid = 'fastr://typelist/BaseDataType'
id = 'BaseDataType'
classmethod isinstance(value)[source]

Indicate whether value is an instance for this DataType.

Returns:the flag indicating the value is of this DataType
Return type:bool
name = 'BaseDataType'
parent = DataTypeManager AnalyzeImageFile : <URLType: AnalyzeImageFile> AnyFile : <TypeGroup: AnyFile> AnyType : <TypeGroup: AnyType> Boolean : <ValueType: Boolean> Deferred : <DataType: Deferred> Directory : <URLType: Directory> FilePrefix : <URLType: FilePrefix> Float : <ValueType: Float> ITKImageFile : <TypeGroup: ITKImageFile> Int : <ValueType: Int> JsonFile : <URLType: JsonFile> MetaImageFile : <URLType: MetaImageFile> NiftiImageFile : <TypeGroup: NiftiImageFile> NiftiImageFileCompressed : <URLType: NiftiImageFileCompressed> NiftiImageFileUncompressed : <URLType: NiftiImageFileUncompressed> NrrdImageFile : <URLType: NrrdImageFile> Number : <TypeGroup: Number> ProvNFile : <URLType: ProvNFile> String : <ValueType: String> TifImageFile : <URLType: TifImageFile> TxtFile : <URLType: TxtFile> UnsignedInt : <ValueType: UnsignedInt> VirtualFileSystem : <IOPlugin: VirtualFileSystem>
parsed_value

The parsed value of object instantiation of this DataType.

raw_value

The raw value of object instantiation of this DataType. For datatypes that override value (like Deferred) this is the way to access the _value field.

classmethod test()[source]

Define the test for the BasePluginManager. Make sure we are not one of the base classes

valid

A boolean flag that indicates weather or not the value assigned to this DataType is valid. This property is generally overwritten by implementation of specific DataTypes.

value

The value of object instantiation of this DataType.

version = <Version: 1.0>

Version of the DataType definition

class fastr.datatypes.Boolean(value=None, format_=None)

Bases: fastr.datatypes.ValueType

Datatype representing a boolean

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
__str__()[source]

Returns the string version of the BaseDataType

Returns:string version
Return type:str
description = 'A boolean value (True of False)'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Boolean.py'
module = <module 'Boolean' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Boolean.pyc'>
value

The value of object instantiation of this DataType.

class fastr.datatypes.DataType(value=None, format_=None)[source]

Bases: fastr.datatypes.BaseDataType

This class is the base class for all DataTypes that can hold a value.

__abstractmethods__ = frozenset(['__init__'])
__init__(value=None, format_=None)[source]

The DataType constructor.

Parameters:
  • value – value to assign to the new DataType object
  • format – the format used for the ValueType
Returns:

new DataType object

__module__ = 'fastr.datatypes'
action(name)[source]

This function can be overwritten by subclasses to implement certain action that should be performed. For example, the Directory DataType has an action ensure. This method makes sure the Directory exists. A Tool can indicate an action that should be called for an Output which will be called before execution.

Parameters:name (str) – name of the action to execute
Returns:None
class fastr.datatypes.Deferred(value=None, format_=None)[source]

Bases: fastr.datatypes.DataType

__abstractmethods__ = frozenset([])
__getstate__()[source]
__init__(value=None, format_=None)[source]

The Deferred constructor.

Parameters:
  • value – value to assign to the new DataType object
  • format – This is ignore but here for compatibility
Returns:

new Deferred object

__module__ = 'fastr.datatypes'
__repr__()[source]

Returns string representation of the BaseDataType

Returns:string represenation
Return type:str
__setstate__(state)[source]
checksum()[source]

Generate a checksum for the value of this DataType

Returns:the checksum of the value
Return type:str
job
classmethod lookup(value)[source]

Look up the deferred target and return that object

Param:

value

Returns:

The value the deferred points to

Return type:

DataType

Raises:
parsed_value

The value of object instantiation of this DataType.

provenance
target

Target object for this deferred.

Raises:
value

The value of object instantiation of this DataType.

class fastr.datatypes.Directory(value=None, format_=None)

Bases: fastr.datatypes.URLType

DataType representing a directory.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Directories are equal by default as long as the validatity matches.

Parameters:other (Directory) – other to compare against
Returns:equality flag
__module__ = 'fastr.datatypes'
action(name)[source]

This method makes sure the Directory exists. A Tool can indicate an action that should be called for an Output which will be called before execution.

Parameters:name (str) – name of the action to execute
Returns:None
description = 'A directory on the disk'
extension = None
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Directory.py'
module = <module 'Directory' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Directory.pyc'>
class fastr.datatypes.EnumType(value=None, format_=None)[source]

Bases: fastr.datatypes.DataType

The EnumType is the base for DataTypes that can have a value which is an option from a predefined set of possibilities (similar to an enum type in many programming languages).

__abstractmethods__ = frozenset([])
__init__(value=None, format_=None)[source]

The EnumType constructor.

Parameters:
  • value – value to assign to the new EnumType object
  • format – the format used for the ValueType
Returns:

new EnumType object

Raises:

FastrDataTypeNotInstantiableError – if not subclassed

__module__ = 'fastr.datatypes'
__reduce_ex__(*args, **kwargs)[source]

helper for pickle

description = 'EnumType (EnumType) is a enumerate type with options:\n\n\nEnumType can take the value of any of the option, but any other value is considered invalid.'
options = frozenset([])
version = <Version: 1.0>

Enums always have version 1.0

class fastr.datatypes.FilePrefix(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
checksum()[source]

Return the checksum of this URL type

Returns:checksum string
Return type:str
description = 'Prefix for another file, including the path'
extension = None
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/FilePrefix.py'
module = <module 'FilePrefix' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/FilePrefix.pyc'>
class fastr.datatypes.Float(value=None, format_=None)

Bases: fastr.datatypes.ValueType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'A floating point value'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Float.py'
module = <module 'Float' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Float.pyc'>
value

The value of object instantiation of this DataType.

class fastr.datatypes.ITKImageFile(value=None)

Bases: fastr.datatypes.TypeGroup

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'Text file to store point coordinates'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/ITKImageFile.py'
module = <module 'ITKImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/ITKImageFile.pyc'>
class fastr.datatypes.Int(value=None, format_=None)

Bases: fastr.datatypes.ValueType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'an integer value'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Int.py'
module = <module 'Int' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Int.pyc'>
value

The value of object instantiation of this DataType.

class fastr.datatypes.JsonFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
checksum()[source]

Return the checksum of this URL type

Returns:checksum string
Return type:str
description = 'json file'
extension = 'json'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/JsonFile.py'
module = <module 'JsonFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/JsonFile.pyc'>
class fastr.datatypes.MetaImageFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Test the equality of two DataType objects

Parameters:other (URLType) – the object to compare against
Returns:flag indicating equality
Return type:bool
__module__ = 'fastr.datatypes'
checksum()[source]

Return the checksum of this MetaImageFile

Returns:checksum string
Return type:str
classmethod content(invalue, outvalue=None)[source]

Give the contents of a URLType, this is generally useful for filetypes that consists of multiple files (e.g. AnalyzeImageFile, DICOM). The value will indicate the main file, and the contents function can determine all files that form a single data value.

Parameters:
  • inval – a value to figure out contents for this type
  • outval – the place where the copy should point to
Returns:

a list of all files part of the value (e.g. header and data file)

Return type:

list

description = 'Meta Image file format'
extension = 'mhd'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/MetaImageFile.py'
module = <module 'MetaImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/MetaImageFile.pyc'>
class fastr.datatypes.NiftiImageFile(value=None)

Bases: fastr.datatypes.TypeGroup

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'NiftiTypeGroup'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFile.py'
module = <module 'NiftiImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFile.pyc'>
class fastr.datatypes.NiftiImageFileCompressed(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
checksum()[source]

Return the checksum of this URL type

Returns:checksum string
Return type:str
description = 'Compressed Nifti Image File format'
extension = 'nii.gz'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFileCompressed.py'
module = <module 'NiftiImageFileCompressed' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFileCompressed.pyc'>
class fastr.datatypes.NiftiImageFileUncompressed(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'Nifti Image File format'
extension = 'nii'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFileUncompressed.py'
module = <module 'NiftiImageFileUncompressed' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NiftiImageFileUncompressed.pyc'>
class fastr.datatypes.NrrdImageFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'Nrrd Image File format'
extension = 'nrrd'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NrrdImageFile.py'
module = <module 'NrrdImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/NrrdImageFile.pyc'>
class fastr.datatypes.Number(value=None)

Bases: fastr.datatypes.TypeGroup

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'an numeric value'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Number.py'
module = <module 'Number' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/Number.pyc'>
class fastr.datatypes.ProvNFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'Provenance file in the prov N format'
extension = 'provn'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/ProvNFile.py'
module = <module 'ProvNFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/ProvNFile.pyc'>
class fastr.datatypes.String(value=None, format_=None)

Bases: fastr.datatypes.ValueType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'A simple string value'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/String.py'
module = <module 'String' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/String.pyc'>
class fastr.datatypes.TifImageFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'Tif Image File format'
extension = ('tiff', 'tif')
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/TifImageFile.py'
module = <module 'TifImageFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/TifImageFile.pyc'>
class fastr.datatypes.TxtFile(value=None, format_=None)

Bases: fastr.datatypes.URLType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'General text file'
extension = 'txt'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/TxtFile.py'
module = <module 'TxtFile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/TxtFile.pyc'>
class fastr.datatypes.TypeGroup(value=None)[source]

Bases: fastr.datatypes.BaseDataType

The TypeGroup is a special DataType that does not hold a value of its own but is used to group a number of DataTypes. For example ITK has a list of supported file formats that all tools build on ITK support. A group can be used to conveniently specify this in multiple Tools that use the same set DataTypes.

__abstractmethods__ = frozenset(['_members'])
__init__(value=None)[source]

Dummy constructor. TypeGroups are not instantiable and cannot hold a value of its own.

Raises:FastrDataTypeNotInstantiableError – if called
__module__ = 'fastr.datatypes'
static __new__(cls, value=None, format_=None)[source]

Instantiate a TypeGroup. This will for match the value to the best matching type and instantiate that. Not that the returned object will not be of type TypeGroup but one of the TypeGroup members.

classmethod isinstance(value)[source]

Indicate whether value is an instance for this DataType.

Returns:the flag indicating the value is of this DataType
Return type:bool
members

A descriptor that can act like a property for a class.

preference

A descriptor that can act like a property for a class.

class fastr.datatypes.URLType(value=None, format_=None)[source]

Bases: fastr.datatypes.DataType

The URLType is the base for DataTypes that point to a resource somewhere else (typically a filesystem). The true value is actually the resource referenced by the value in this object.

__abstractmethods__ = frozenset([])
__eq__(other)[source]

Test the equality of two DataType objects

Parameters:other (URLType) – the object to compare against
Returns:flag indicating equality
Return type:bool
__init__(value=None, format_=None)[source]

The URLType constructor

Parameters:
  • value – value to assign to the new URLType
  • format – the format used for the ValueType
Returns:

new URLType object

__module__ = 'fastr.datatypes'
checksum()[source]

Return the checksum of this URL type

Returns:checksum string
Return type:str
classmethod content(inval, outval=None)[source]

Give the contents of a URLType, this is generally useful for filetypes that consists of multiple files (e.g. AnalyzeImageFile, DICOM). The value will indicate the main file, and the contents function can determine all files that form a single data value.

Parameters:
  • inval – a value to figure out contents for this type
  • outval – the place where the copy should point to
Returns:

a list of all files part of the value (e.g. header and data file)

Return type:

list

parsed_value

The parsed value of object instantiation of this DataType.

valid

A boolean flag that indicates weather or not the value assigned to this DataType is valid. This property is generally overwritten by implementation of specific DataTypes.

class fastr.datatypes.UnsignedInt(value=None, format_=None)

Bases: fastr.datatypes.ValueType

__abstractmethods__ = frozenset([])
__module__ = 'fastr.datatypes'
description = 'an unsigned integer value'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/UnsignedInt.py'
module = <module 'UnsignedInt' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/datatypes/UnsignedInt.pyc'>
value

The value of object instantiation of this DataType.

class fastr.datatypes.ValueType(value=None, format_=None)[source]

Bases: fastr.datatypes.DataType

The ValueType is the base for DataTypes that hold simple values (not an EnumType and not a file/URL). The values is generally represented by a string.

__abstractmethods__ = frozenset([])
__init__(value=None, format_=None)[source]

The ValueType constructor

Parameters:
  • value – value to assign to the new ValueType
  • format – the format used for the ValueType
Returns:

new ValueType object

__module__ = 'fastr.datatypes'
fastr.datatypes.fastr_isinstance(obj, datatype)[source]

Check if an object is of a specific datatype.

Parameters:
  • obj – Object to inspect
  • datatype (tuple, BaseDataType) – The datatype(s) to check
Returns:

flag indicating object is of datatype

Return type:

bool

exceptions Module

This module contains all Fastr-related Exceptions

exception fastr.exceptions.FastrAttributeError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.AttributeError

AttributeError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrCannotChangeAttributeError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Attempting to change an attribute of an object that can be set only once.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrCardinalityError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The description of the cardinality is not valid.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrCollectorError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Cannot collect the results from a Job because of an error

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrDataTypeFileNotReadable(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Could not read the datatype file.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrDataTypeMismatchError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

When using a DataType as the key for the DataTypeManager, the DataTypeManager found another DataType with the same name already in the DataTypeManager. The means fastr has two version of the same DataType in the system, which should never happen!

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrDataTypeNotAvailableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The DataType requested is not found by the fastr system. Typically this means that no matching DataType is found in the DataTypeManager.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrDataTypeNotInstantiableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The base classes for DataTypes cannot be instantiated and should always be sub-classed.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrDataTypeValueError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

This value in fastr did not pass the validation specificied for its DataType, typically means that the data is missing or corrupt.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrError(*args, **kwargs)[source]

Bases: exceptions.Exception

This is the base class for all fastr related exceptions. Catching this class of exceptions should ensure a proper execution of fastr.

__init__(*args, **kwargs)[source]

Constructor for all exceptions. Saves the caller object fullid (if found) and the file, function and line number where the object was created.

__module__ = 'fastr.exceptions'
__str__()[source]

String representation of the error

Returns:error string
Return type:str
__weakref__

list of weak references to the object (if defined)

excerpt()[source]

Return a excerpt of the Error as a tuple.

exception fastr.exceptions.FastrErrorInSubprocess(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

Encountered an error in the subprocess started by the execution script

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrExecutableNotFoundError(executable=None, *args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

The executable could not be found!

__init__(executable=None, *args, **kwargs)[source]

Constructor for all exceptions. Saves the caller object fullid (if found) and the file, function and line number where the object was created.

__module__ = 'fastr.exceptions'
__str__()[source]

String representation of the error

exception fastr.exceptions.FastrExecutionError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Base class for all fastr execution related errors

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrIOError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.IOError

IOError in the fastr system

__module__ = 'fastr.exceptions'
__weakref__

list of weak references to the object (if defined)

exception fastr.exceptions.FastrImportError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.ImportError

ImportError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrIndexError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.IndexError

IndexError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrIndexNonexistent(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrIndexError

This is an IndexError for samples requested from a sparse data array. The sample is not there but is probably not there because of sparseness rather than being a missing sample (e.g. out of bounds).

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrKeyError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.KeyError

KeyError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrLookupError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Could not find specified object in the fastr environment.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrMountUnknownError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Trying to access an undefined mount

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNetworkMismatchError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Two interacting objects belong to different fastr network.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNetworkUnknownError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Reference to a Tool that is not recognised by the fastr system. This typically means the specific id/version combination of the requested tool has not been loaded by the ToolManager.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNoValidTargetError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Cannot find a valid target for the tool

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNodeAreadyPreparedError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrStateError

A attempt is made at preparing a NodeRun for the second time. This is not allowed as it would wipe the current execution data and cause data-loss.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNodeNotPreparedError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrStateError

When trying to access executation data of a NodeRun, the NodeRun must be prepare. The NodeRun has not been prepared by the execution, so the data is not available!

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNodeNotValidError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrStateError

A NodeRun is not in a valid state where it should be, typically an invalid NodeRun is passed to the executor causing trouble.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNotExecutableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

The command invoked by subprocess is not executable on the system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrNotImplementedError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.NotImplementedError

This function/method has not been implemented on purpose (e.g. should be overwritten in a sub-class)

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrOSError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.OSError

OSError in the fastr system

__module__ = 'fastr.exceptions'
__weakref__

list of weak references to the object (if defined)

exception fastr.exceptions.FastrObjectUnknownError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Reference to a Tool that is not recognised by the fastr system. This typically means the specific id/version combination of the requested tool has not been loaded by the ToolManager.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrOptionalModuleNotAvailableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrNotImplementedError

A optional modules for Fastr is needed for this function, but is not available on the current python installation.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrOutputValidationError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

An output of a Job does not pass validation

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrParentMismatchError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Two interactive objects have different parent where they should be the same

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrPluginCapabilityNotImplemented(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrNotImplementedError

A plugin did not implement a capability that it advertised.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrPluginNotAvailable(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Indicates that a requested Plugin was not found on the system.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrPluginNotLoaded(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrStateError

The plugin was not successfully loaded. This means the plugin class cannot be instantiated.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrResultFileNotFound(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

Could not found the result file of job that finished. This means the executionscript process was killed during interruption. Generally this means a scheduler killed it because of resource shortage.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrScriptNotFoundError(interpreter=None, script=None, paths=None, *args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

Script could not be found

__init__(interpreter=None, script=None, paths=None, *args, **kwargs)[source]

Constructor for all exceptions. Saves the caller object fullid (if found) and the file, function and line number where the object was created.

__module__ = 'fastr.exceptions'
__str__()[source]

String representation of the error

Returns:error string
Return type:str
exception fastr.exceptions.FastrSerializationError(message, serializer, original_exception=None)[source]

Bases: fastr.exceptions.FastrError

The serialization encountered a serious problem

__init__(message, serializer, original_exception=None)[source]

Constructor for all exceptions. Saves the caller object fullid (if found) and the file, function and line number where the object was created.

__module__ = 'fastr.exceptions'
__repr__()[source]

Simple string representation of the exception

__str__()[source]

Advanced string representation of the exception including the data about where in the schema things went wrong.

exception fastr.exceptions.FastrSerializationIgnoreDefaultError(message, serializer, original_exception=None)[source]

Bases: fastr.exceptions.FastrSerializationError

The value and default are both None, so the value should not be serialized.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSerializationInvalidDataError(message, serializer, original_exception=None)[source]

Bases: fastr.exceptions.FastrSerializationError

Encountered data to serialize that is invalid given the serialization schema.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSerializationMethodError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

The desired serialization method does not exist.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSinkDataUnavailableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Could not find the Sink data for the desire sink.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSizeInvalidError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The given size cannot be valid.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSizeMismatchError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The size of two object in fastr is not matching where it should.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSizeUnknownError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The size of object is not (yet) known and only a theoretical estimate is available at the moment.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSourceDataUnavailableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Could not find the Source data for the desire source.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrStateError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

An object is in an invalid/unexpected state.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrSubprocessNotFinished(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrExecutionError

Encountered an error before the subprocess call by the execution script was properly finished.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrToolNotAvailableError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

The tool used is not available on the current platform (OS and architecture
combination) and cannot be used.
__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrToolTargetNotFound(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError

Could not determine the location of the tools target binary/script. The tool cannot be used.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrToolUnknownError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Reference to a Tool that is not recognised by the fastr system. This typically means the specific id/version combination of the requested tool has not been loaded by the ToolManager.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrTypeError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.TypeError

TypeError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrUnknownURLSchemeError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrKeyError

Fastr encountered a data URL with a scheme that was not recognised by the IOPlugin manager.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrValueError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrError, exceptions.ValueError

ValueError in the fastr system

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrVersionInvalidError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrValueError

The string representation of the version is malformatted.

__module__ = 'fastr.exceptions'
exception fastr.exceptions.FastrVersionMismatchError(*args, **kwargs)[source]

Bases: fastr.exceptions.FastrValueError

There is a mismatch between different parts of the Fastr environment and integrity is compromised.

__module__ = 'fastr.exceptions'

plugins Module

The plugins module holds all plugins loaded by Fastr. It is empty on start and gets filled by the BasePluginManager

class fastr.plugins.BlockingExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

The blocking execution plugin is a special plugin which is meant for debug purposes. It will not queue jobs but immediately execute them inline, effectively blocking fastr until the Job is finished. It is the simplest execution plugin and can be used as a template for new plugins or for testing purposes.

__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/blockingexecution.py'
module = <module 'blockingexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/blockingexecution.pyc'>
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.CommaSeperatedValueFile

Bases: fastr.core.ioplugin.IOPlugin

The CommaSeperatedValueFile an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs.

The csv:// URL is a vfs:// URL with a number of query variables available. The URL mount and path should point to a valid CSV file. The query variable then specify what column(s) of the file should be used.

The following variable can be set in the query:

variable usage
value the column containing the value of interest, can be int for index or string for key
id the column containing the sample id (optional)
header indicates if the first row is considered the header, can be true or false (optional)
delimiter the delimiter used in the csv file (optional)
quote the quote character used in the csv file (optional)
reformat a reformatting string so that value = reformat.format(value) (used before relative_path)
relative_path indicates the entries are relative paths (for files), can be true or false (optional)

The header is by default false if the neither the value and id are set as a string. If either of these are a string, the header is required to define the column names and it automatically is assumed true

The delimiter and quota characters of the file should be detected automatically using the Sniffer, but can be forced by setting them in the URL.

Example of valid csv URLs:

# Use the first column in the file (no header row assumed)
csv://mount/some/dir/file.csv?value=0

# Use the images column in the file (first row is assumed header row)
csv://mount/some/dir/file.csv?value=images

# Use the segmentations column in the file (first row is assumed header row)
# and use the id column as the sample id
csv://mount/some/dir/file.csv?value=segmentations&id=id

# Use the first column as the id and the second column as the value
# and skip the first row (considered the header)
csv://mount/some/dir/file.csv?value=1&id=0&header=true

# Use the first column and force the delimiter to be a comma
csv://mount/some/dir/file.csv?value=0&delimiter=,
__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/commaseperatedvaluefile.py'
module = <module 'commaseperatedvaluefile' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/commaseperatedvaluefile.pyc'>
scheme = 'csv'
class fastr.plugins.CrossValidation

Bases: flowinterface.FlowPlugin

Advanced flow plugin that generated a cross-validation data flow. The node need an input with data and an input number of folds. Based on that the outputs test and train will be supplied with a number of data sets.

__abstractmethods__ = frozenset([])
__module__ = 'fastr.plugins'
static execute(payload)[source]
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/flowplugins/crossvalidation.py'
module = <module 'crossvalidation' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/flowplugins/crossvalidation.pyc'>
class fastr.plugins.DRMAAExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

A DRMAA execution plugin to execute Jobs on a Grid Engine cluster. It uses a configuration option for selecting the queue to submit to. It uses the python drmaa package.

Note

To use this plugin, make sure the drmaa package is installed and that the execution is started on an SGE submit host with DRMAA libraries installed.

Note

This plugin is at the moment tailored to SGE, but it should be fairly easy to make different subclasses for different DRMAA supporting systems.

CANCELS_DEPENDENCIES = False
GE_NATIVE_SPEC = {'CWD': '-cwd', 'DEPENDS': '-hold_jid {hold_list}', 'DEPENDS_SEP': ',', 'ERRORLOG': '-e {errorlog}', 'HOLD': '-h', 'MEMORY': '-l h_vmem={memory}', 'NCORES': '-pe smp {ncores:d}', 'OUTPUTLOG': '-o {outputlog}', 'QUEUE': '-q {queue}', 'WALLTIME': '-l h_rt={walltime}'}
NATIVE_SPEC = {'grid_engine': {'QUEUE': '-q {queue}', 'DEPENDS': '-hold_jid {hold_list}', 'HOLD': '-h', 'MEMORY': '-l h_vmem={memory}', 'NCORES': '-pe smp {ncores:d}', 'OUTPUTLOG': '-o {outputlog}', 'DEPENDS_SEP': ',', 'ERRORLOG': '-e {errorlog}', 'CWD': '-cwd', 'WALLTIME': '-l h_rt={walltime}'}, 'torque': {'QUEUE': '-q {queue}', 'DEPENDS': '-W depend=afterok:{hold_list}', 'HOLD': '-h', 'MEMORY': '-l mem={memory}', 'NCORES': '-l procs={ncores:d}', 'OUTPUTLOG': '-o {outputlog}', 'DEPENDS_SEP': ':', 'ERRORLOG': '-e {errorlog}', 'CWD': '', 'WALLTIME': '-l walltime={walltime}'}}
SUPPORTS_CANCEL = True
SUPPORTS_DEPENDENCY = True
SUPPORTS_HOLD_RELEASE = True
TORQUE_NATIVE_SPEC = {'CWD': '', 'DEPENDS': '-W depend=afterok:{hold_list}', 'DEPENDS_SEP': ':', 'ERRORLOG': '-e {errorlog}', 'HOLD': '-h', 'MEMORY': '-l mem={memory}', 'NCORES': '-l procs={ncores:d}', 'OUTPUTLOG': '-o {outputlog}', 'QUEUE': '-q {queue}', 'WALLTIME': '-l walltime={walltime}'}
__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
collect_jobs()[source]
configuration_fields = {'drmaa_engine': (<type 'str'>, 'grid_engine', 'The engine to use (options: grid_engine, torque'), 'drmaa_job_check_interval': (<type 'int'>, 900, 'The interval in which the job checker will startto check for stale jobs'), 'drmaa_max_jobs': (<type 'int'>, 0, 'The maximum jobs that can be send to the scheduler at the same time (0 for no limit)'), 'drmaa_queue': (<type 'str'>, 'week', 'The default queue to use for jobs send to the scheduler')}
create_native_spec(queue, walltime, memory, ncores, outputLog, errorLog, hold_job, hold)[source]

Create the native spec for the DRMAA scheduler. Needs to be implemented in the subclasses

Parameters:
  • queue (str) – the queue to submit to
  • walltime (str) – walltime specified
  • memory (str) – memory requested
  • ncores (int) – number of cores requested
  • outputLog (str) – the location of the stdout log
  • errorLog (str) – the location of stderr log
  • hold_job (list) – list of jobs to depend on
  • hold (bool) – flag if job should be submitted in hold mode
Returns:

current_jobs
dispatch_callbacks()[source]
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/drmaaexecution.py'
job_regression_check()[source]
module = <module 'drmaaexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/drmaaexecution.pyc'>
send_job(command, arguments, queue=None, walltime=None, job_name=None, memory=None, ncores=None, joinLogFiles=False, outputLog=None, errorLog=None, hold_job=None, hold=False)[source]
spec_fields
submit_jobs()[source]
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.DockerTarget(binary, docker_image)

Bases: fastr.core.target.Target

A tool target that is located in a Docker images. Can be run using docker-py. A docker target only need two variables: the binary to call within the docker container, and the docker container to use.

{
  "arch": "*",
  "os": "*",
  "binary": "bin/test.py",
  "docker_image": "fastr/test"
}
<target os="*" arch="*" binary="bin/test.py" docker_image="fastr/test">
__abstractmethods__ = frozenset([])
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment where needed

__init__(binary, docker_image)[source]

Define a new docker target.

Parameters:docker_image (str) – Docker image to use
__module__ = 'fastr.plugins'
container
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/dockertarget.py'
module = <module 'dockertarget' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/dockertarget.pyc'>
monitor_docker(container, resources)[source]

Monitor a docker container and profile the cpu, memory and io use. Register the resource use every _MONITOR_INTERVAL seconds.

Parameters:
  • container (ContainerCollection) – process to monitor
  • resources (ProcessUsageCollection) – list to append measurements to
run_command(command)[source]
class fastr.plugins.FastrInterface(id_, document)

Bases: fastr.core.interface.Interface

The default Interface for fastr. For the command-line Tools as used by fastr. It build a commandline call based on the input/output specification.

The fields that can be set in the interface:

Attribute Description
id The id of this Tool (used internally in fastr)
inputs[]   List of Inputs that can are accepted by the Tool
id ID of the Input
name Longer name of the Input (more human readable)
datatype The ID of the DataType of the Input [1]
enum[] List of possible values for an EnumType (created on the fly by fastr) [1]
prefix Commandline prefix of the Input (e.g. –in, -i)
cardinality Cardinality of the Input
repeat_prefix Flag indicating if for every value of the Input the prefix is repeated
required Flag indicating if the input is required
nospace Flag indicating if there is no space between prefix and value (e.g. –in=val)
format For DataTypes that have multiple representations, indicate which one to use
default Default value for the Input
description Long description for an input
outputs[]   List of Outputs that are generated by the Tool (and accessible to fastr)
id ID of the Output
name Longer name of the Output (more human readable)
datatype The ID of the DataType of the Output [1]
enum[] List of possible values for an EnumType (created on the fly by fastr) [1]
prefix Commandline prefix of the Output (e.g. –out, -o)
cardinality Cardinality of the Output
repeat_prefix Flag indicating if for every value of the Output the prefix is repeated
required Flag indicating if the input is required
nospace Flag indicating if there is no space between prefix and value (e.g. –out=val)
format For DataTypes that have multiple representations, indicate which one to use
description Long description for an input
action Special action (defined per DataType) that needs to be performed before creating output value (e.g. ‘ensure’ will make sure an output directory exists)
automatic Indicate that output doesn’t require commandline argument, but is created automatically by a Tool [2]
method The collector plugin to use for the gathering automatic output, see the Collector plugins
location Definition where to an automatically, usage depends on the method [2]

Footnotes

[1](1, 2, 3, 4) datatype and enum are conflicting entries, if both specified datatype has presedence
[2](1, 2) More details on defining automatica output are given in [TODO]
__abstractmethods__ = frozenset([])
__dataschemafile__ = 'FastrInterface.schema.json'
__eq__(other)[source]

x.__eq__(y) <==> x==y

__getstate__()[source]

Get the state of the FastrInterface object.

Returns:state of interface
Return type:dict
__init__(id_, document)[source]

The BasePlugin constructor.

Returns:the created plugin
Return type:BasePlugin
Raises:FastrPluginNotLoaded – if the plugin did not load correctly
__module__ = 'fastr.plugins'
__setstate__(state)[source]

Set the state of the Interface

check_input_id(id_)[source]

Check if an id for an object is valid and unused in the Tool. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:
check_output_id(id_)[source]

Check if an id for an object is valid and unused in the Tool. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:
static collect_errors(result)[source]

Special error collection for fastr interfaces

collect_results(result)[source]

Collect all results of the interface

collector_plugin_type

alias of fastrinterface.CollectorPlugin

collectors = CollectorPluginManager Loaded json : <CollectorPlugin: JsonCollector> Loaded path : <CollectorPlugin: PathCollector> Loaded stdout : <CollectorPlugin: StdoutCollector>
execute(target, payload)[source]

Execute the interface using a specific target and payload (containing a set of values for the arguments)

Parameters:
  • target (SampleId) – the target to use
  • payload (dict) – the values for the arguments
Returns:

result of the execution

Return type:

InterfaceResult

expanding
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/fastrinterface.py'
get_arguments(values)[source]

Get the argument list for this interface

Returns:return list of arguments
get_command(target, payload)[source]
get_specials(payload, output, cardinality_nr)[source]

Get special attributes. Returns tuples for specials, inputs and outputs that are used for formatting substitutions.

Parameters:
  • output – Output for which to get the specials
  • cardinality_nr (int) – the cardinality number
inputs
module = <module 'fastrinterface' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/fastrinterface.pyc'>
outputs
class fastr.plugins.FileSystem

Bases: fastr.core.ioplugin.IOPlugin

The FileSystem plugin is create to handle file:// type or URLs. This is generally not a good practice, as this is not portable over between machines. However, for test purposes it might be useful.

The URL scheme is rather simple: file://host/path (see wikipedia for details)

We do not make use of the host part and at the moment only support localhost (just leave the host empty) leading to file:/// URLs.

Warning

This plugin ignores the hostname in the URL and does only accept driver letters on Windows in the form c:/

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
fetch_url(inurl, outpath)[source]

Fetch the files from the file.

Parameters:
  • inurl – url to the item in the data store, starts with file://
  • outpath – path where to store the fetch data locally
fetch_value(inurl)[source]

Fetch a value from an external file file.

Parameters:inurl – url of the value to read
Returns:the fetched value
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/filesystem.py'
module = <module 'filesystem' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/filesystem.pyc'>
path_to_url(path, mountpoint=None)[source]

Construct an url from a given mount point and a relative path to the mount point.

put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters:
  • inpath – path of the local data
  • outurl – url to where to store the data, starts with file://
put_value(value, outurl)[source]

Put the value in the external data store.

Parameters:
  • value – value to store
  • outurl – url to where to store the data, starts with file://
scheme = 'file'
url_to_path(url)[source]

Get the path to a file from a url. Currently supports the file:// scheme

Examples:

>>> 'file:///d:/data/project/file.ext'
'd:\data\project\file.ext'

Warning

file:// will not function cross platform and is mainly for testing

class fastr.plugins.FlowInterface(id_, document)

Bases: fastr.core.interface.Interface

The Interface use for AdvancedFlowNodes to create the advanced data flows that are not implemented in the fastr. This allows nodes to implement new data flows using the plugin system.

The definition of FlowInterfaces are very similar to the default FastrInterfaces.

Note

A flow interface should be using a specific FlowPlugin

__abstractmethods__ = frozenset([])
__dataschemafile__ = 'FastrInterface.schema.json'
__eq__(other)[source]

x.__eq__(y) <==> x==y

__getstate__()[source]

Get the state of the FastrInterface object.

Returns:state of interface
Return type:dict
__init__(id_, document)[source]

The BasePlugin constructor.

Returns:the created plugin
Return type:BasePlugin
Raises:FastrPluginNotLoaded – if the plugin did not load correctly
__module__ = 'fastr.plugins'
__setstate__(state)[source]

Set the state of the Interface

execute(target, payload)[source]

Execute the interface given the a target and payload. The payload should have the form {‘input’: {‘input_id_a’: (value, value), ‘input_id_b’: (value, value)}, ‘output’: {‘output_id_a’: (value, value), ‘output_id_b’: (value, value)}}

Parameters:
  • target – the target to call
  • payload – the payload to use
Returns:

the result of the execution

Return type:

(tuple of) InterfaceResult

expanding
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/flowinterface.py'
flow_plugin_type

alias of flowinterface.FlowPlugin

flow_plugins = FlowPluginManager Loaded CrossValidation : <FlowPlugin: CrossValidation>
inputs
module = <module 'flowinterface' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/flowinterface.pyc'>
outputs
class fastr.plugins.LinearExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

An execution engine that has a background thread that executes the jobs in order. The queue is a simple FIFO queue and there is one worker thread that operates in the background. This plugin is meant as a fallback when other plugins do not function properly. It does not multi-processing so it is safe to use in environments that do no support that.

__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
exec_worker()[source]
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/linearexecution.py'
module = <module 'linearexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/linearexecution.pyc'>
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.LocalBinaryTarget(binary, paths=None, environment_variables=None, initscripts=None, modules=None, interpreter=None, **kwargs)

Bases: fastr.core.target.SubprocessBasedTarget

A tool target that is a local binary on the system. Can be found using environmentmodules or a path on the executing machine. A local binary target has a number of fields that can be supplied:

  • binary (required): the name of the binary/script to call, can also be called bin for backwards compatibility.
  • modules: list of modules to load, this can be environmentmodules or lmod modules. If modules are given, the paths, environment_variables and initscripts are ignored.
  • paths: a list of paths to add following the structure {"value": "/path/to/dir", "type": "bin"}. The types can be bin if the it should be added to $PATH or lib if it should be added to te library path (e.g. $LD_LIBRARY_PATH for linux).
  • environment_variables: a dictionary of environment variables to set.
  • initscript: a list of script to run before running the main tool
  • interpreter: the interpreter to use to call the binary e.g. python

The LocalBinaryTarget will first check if there are modules given and the module subsystem is loaded. If that is the case it will simply unload all current modules and load the given modules. If not it will try to set up the environment itself by using the following steps:

  1. Prepend the bin paths to $PATH
  2. Prepend the lib paths to the correct environment variable
  3. Setting the other environment variables given ($PATH and the system library path are ignored and cannot be set that way)
  4. Call the initscripts one by one

The definition of the target in JSON is very straightforward:

{
  "binary": "bin/test.py",
  "interpreter": "python",
  "paths": [
    {
      "type": "bin",
      "value": "vfs://apps/test/bin"
    },
    {
      "type": "lib",
      "value": "./lib"
    }
  ],
  "environment_variables": {
    "othervar": 42,
    "short_var": 1,
    "testvar": "value1"
  },
  "initscripts": [
    "bin/init.sh"
  ],
  "modules": ["elastix/4.8"]
}

In XML the definition would be in the form of:

<target os="linux" arch="*" modules="elastix/4.8" bin="bin/test.py" interpreter="python">
  <paths>
    <path type="bin" value="vfs://apps/test/bin" />
    <path type="lib" value="./lib" />
  </paths>
  <environment_variables short_var="1">
    <testvar>value1</testvar>
    <othervar>42</othervar>
  </environment_variables>
  <initscripts>
    <initscript>bin/init.sh</initscript>
  </initscripts>
</target>
DYNAMIC_LIBRARY_PATH_DICT = {'darwin': 'DYLD_LIBRARY_PATH', 'linux': 'LD_LIBRARY_PATH', 'windows': 'PATH'}
__abstractmethods__ = frozenset([])
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment

__init__(binary, paths=None, environment_variables=None, initscripts=None, modules=None, interpreter=None, **kwargs)[source]

Define a new local binary target. Must be defined either using paths and optionally environment_variables and initscripts, or enviroment modules.

__module__ = 'fastr.plugins'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/localbinarytarget.py'
module = <module 'localbinarytarget' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/localbinarytarget.pyc'>
run_command(command)[source]
class fastr.plugins.NipypeInterface(id_, nipype_cls=None, document=None)

Bases: fastr.core.interface.Interface

Experimental interfaces to using nipype interfaces directly in fastr tools, only using a simple reference.

To create a tool using a nipype interface just create an interface with the correct type and set the nipype argument to the correct class. For example in an xml tool this would become:

<interface class="NipypeInterface">
  <nipype_class>nipype.interfaces.elastix.Registration</nipype_class>
</interface>

Note

To use these interfaces nipype should be installed on the system.

Warning

This interface plugin is basically functional, but highly experimental!

__abstractmethods__ = frozenset([])
__eq__(other)[source]

x.__eq__(y) <==> x==y

__getstate__()[source]

Retrieve the state of the Interface

Returns:the state of the object
Rtype dict:
__init__(id_, nipype_cls=None, document=None)[source]

The BasePlugin constructor.

Returns:the created plugin
Return type:BasePlugin
Raises:FastrPluginNotLoaded – if the plugin did not load correctly
__module__ = 'fastr.plugins'
__setstate__(state)[source]

Set the state of the Interface

execute(target, payload)[source]

Execute the interface using a specific target and payload (containing a set of values for the arguments)

Parameters:
  • target (SampleId) – the target to use
  • payload (dict) – the values for the arguments
Returns:

result of the execution

Return type:

InterfaceResult

expanding
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/nipypeinterface.py'
get_type(trait)[source]
inputs
module = <module 'nipypeinterface' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/interfaceplugins/nipypeinterface.pyc'>
outputs
classmethod test()[source]

Test the plugin, interfaces do not need to be tested on import

class fastr.plugins.Null

Bases: fastr.core.ioplugin.IOPlugin

The Null plugin is create to handle null:// type or URLs. These URLs are indicating the sink should not do anything. The data is not written to anywhere. Besides the scheme, the rest of the URL is ignored.

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/null.py'
module = <module 'null' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/null.pyc'>
put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters:
  • inpath – path of the local data
  • outurl – url to where to store the data, starts with file://
put_value(value, outurl)[source]

Put the value in the external data store.

Parameters:
  • value – value to store
  • outurl – url to where to store the data, starts with file://
scheme = 'null'
class fastr.plugins.ProcessPoolExecution(finished_callback=None, cancelled_callback=None, status_callback=None, nr_of_workers=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

A local execution plugin that uses multiprocessing to create a pool of worker processes. This allows fastr to execute jobs in parallel with true concurrency. The number of workers can be specified in the fastr configuration, but the default amount is the number of cores - 1 with a minimum of 1.

Warning

The ProcessPoolExecution does not check memory requirements of jobs and running many workers might lead to memory starvation and thus an unresponsive system.

__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None, nr_of_workers=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
configuration_fields = {'process_pool_worker_number': (<type 'int'>, 3, 'Number of workers to use in a process pool')}
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/processpoolexecution.py'
job_finished_callback(result)[source]

Reciever for the callback, it will split the result tuple and call job_finished

Parameters:result (tuple) – return value of run_job
module = <module 'processpoolexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/processpoolexecution.pyc'>
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.RQExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

A execution plugin based on Redis Queue. Fastr will submit jobs to the redis queue and workers will peel the jobs from the queue and process them.

This system requires a running redis database and the database url has to be set in the fastr configuration.

Note

This execution plugin required the redis and rq packages to be installed before it can be loaded properly.

__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
check_finished()[source]
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
configuration_fields = {'rq_host': (<type 'str'>, 'redis://localhost:6379/0', 'The url of the redis serving the redis queue'), 'rq_queue': (<type 'str'>, 'default', 'The redis queue to use')}
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/rqexecution.py'
module = <module 'rqexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/rqexecution.pyc'>
classmethod run_job(job_id, job_command, job_stdout, job_stderr)[source]
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.Reference

Bases: fastr.core.ioplugin.IOPlugin

The Reference plugin is create to handle ref:// type or URLs. These URLs are to make the sink just write a simple reference file to the data. The reference file contains the DataType and the value so the result can be reconstructed. It for files just leaves the data on disk by reference. This plugin is not useful for production, but is used for testing purposes.

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/reference.py'
module = <module 'reference' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/reference.pyc'>
push_sink_data(value, outurl, datatype=None)[source]

Write out the sink data from the inpath to the outurl.

Parameters:
  • value (str) – the path of the data to be pushed
  • outurl (str) – the url to write the data to
  • datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer
Returns:

None

scheme = 'ref'
class fastr.plugins.SingularityTarget(binary, container, interpreter=None)

Bases: fastr.core.target.SubprocessBasedTarget

A tool target that is run using a singularity container, see the singulary website

  • binary (required): the name of the binary/script to call, can also be called bin for backwards compatibility.
  • container (required): the singularity container to run, this can be in url form for singularity
    pull or as a path to a local container
  • interpreter: the interpreter to use to call the binary e.g. python
SINGULARITY_BIN = 'singularity'
__abstractmethods__ = frozenset([])
__enter__()[source]

Set the environment in such a way that the target will be on the path.

__exit__(exc_type, exc_value, traceback)[source]

Cleanup the environment

__init__(binary, container, interpreter=None)[source]

Define a new local binary target. Must be defined either using paths and optionally environment_variables and initscripts, or enviroment modules.

__module__ = 'fastr.plugins'
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/singularitytarget.py'
module = <module 'singularitytarget' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/targetplugins/singularitytarget.pyc'>
run_command(command)[source]
classmethod test()[source]

Test if singularity is availble on the path

class fastr.plugins.SlurmExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

SBATCH = 'sbatch'
SCANCEL = 'scancel'
SCONTROL = 'scontrol'
SQUEUE = 'squeue'
SQUEUE_FORMAT = '{"id": %.18i, "status": "%.2t"}'
STATUS_MAPPING = {' F': <JobState.failed: ('failed', 'done', True)>, ' R': <JobState.running: ('running', 'in_progress', False)>, 'CA': <JobState.cancelled: ('cancelled', 'done', True)>, 'CD': <JobState.finished: ('finished', 'done', False)>, 'CF': <JobState.running: ('running', 'in_progress', False)>, 'CG': <JobState.running: ('running', 'in_progress', False)>, 'NF': <JobState.failed: ('failed', 'done', True)>, 'PD': <JobState.queued: ('queued', 'idle', False)>, 'RV': <JobState.cancelled: ('cancelled', 'done', True)>, 'SE': <JobState.failed: ('failed', 'done', True)>, 'TO': <JobState.queued: ('queued', 'idle', False)>}
SUPPORTS_CANCEL = True
SUPPORTS_DEPENDENCY = True
SUPPORTS_HOLD_RELEASE = True
__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
configuration_fields = {'slurm_job_check_interval': (<type 'int'>, 30, 'The interval in which the job checker will startto check for stale jobs'), 'slurm_partition': (<type 'str'>, '', 'The slurm partition to use')}
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/slurmexecution.py'
job_status_check()[source]
module = <module 'slurmexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/slurmexecution.pyc'>
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.StrongrExecution(finished_callback=None, cancelled_callback=None, status_callback=None)

Bases: fastr.execution.executionpluginmanager.ExecutionPlugin

A execution plugin based on Redis Queue. Fastr will submit jobs to the redis queue and workers will peel the jobs from the queue and process them.

This system requires a running redis database and the database url has to be set in the fastr configuration.

Note

This execution plugin required the redis and rq packages to be installed before it can be loaded properly.

__abstractmethods__ = frozenset([])
__init__(finished_callback=None, cancelled_callback=None, status_callback=None)[source]

Setup the ExecutionPlugin

Parameters:
  • finished_callback – the callback to call after a job finished
  • cancelled_callback – the callback to call after a job cancelled
Returns:

newly created ExecutionPlugin

__module__ = 'fastr.plugins'
check_finished()[source]
cleanup()[source]

Method to call to clean up the ExecutionPlugin. This can be to clear temporary data, close connections, etc.

Parameters:force – force cleanup (e.g. kill instead of join a process)
configuration_fields = {}
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/strongrexecution.py'
module = <module 'strongrexecution' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/executionplugins/strongrexecution.pyc'>
classmethod test()[source]

Test the plugin, default behaviour is just to instantiate the plugin

class fastr.plugins.VirtualFileSystem

Bases: fastr.core.ioplugin.IOPlugin

The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the vfs:// url scheme.

A typical virtual filesystem url is formatted as vfs://mountpoint/relative/dir/from/mount.ext

Where the mountpoint is defined in the Config file. A list of the currently known mountpoints can be found in the fastr.config object

>>> fastr.config.mounts
{'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',
 'home': '/home/username/',
 'tmp': '/home/username/FastrTemp'}

This shows that a url with the mount home such as vfs://home/tempdir/testfile.txt would be translated into /home/username/tempdir/testfile.txt.

There are a few default mount points defined by Fastr (that can be changed via the config file).

mountpoint default location
home the users home directory (expanduser('~/'))
tmp the fastr temprorary dir, defaults to tempfile.gettempdir()
example_data the fastr example data directory, defaults $FASTRDIR/example/data
__abstractmethods__ = frozenset([])
__init__()[source]

Instantiate the VFS plugin

Returns:the VirtualFileSysten plugin
__module__ = 'fastr.plugins'
abstract = False
static copy_file_dir(inpath, outpath)[source]

Helper function, copies a file or directory not caring what the inpath actually is

Parameters:
  • inpath – path of the things to be copied
  • outpath – path of the destination
Returns:

the result of shutil.copy2 or shutil.copytree (depending on inpath pointing to a file or directory)

expand_url(url)[source]

Try to expand the url. For vfs with will return the original url.

Parameters:url – url to expand
Returns:the expanded url (same as url)
fetch_url(inurl, outpath)[source]

Fetch the files from the vfs.

Parameters:
  • inurl – url to the item in the data store, starts with vfs://
  • outpath – path where to store the fetch data locally
fetch_value(inurl)[source]

Fetch a value from an external vfs file.

Parameters:inurl – url of the value to read
Returns:the fetched value
path_to_url(path, mountpoint=None, scheme=None)[source]

Construct an url from a given mount point and a relative path to the mount point.

Parameters:path (str) – the path to find the url for
Mountpoint str:mountpoint the url should be under
Returns:url of the
put_url(inpath, outurl)[source]

Put the files to the external data store.

Parameters:
  • inpath – path of the local data
  • outurl – url to where to store the data, starts with vfs://
put_value(value, outurl)[source]

Put the value in the external data store.

Parameters:
  • value – value to store
  • outurl – url to where to store the data, starts with vfs://
scheme
setup()[source]

The plugin setup, does nothing but needs to be implemented

url_to_path(url, scheme=None)[source]

Get the path to a file from a vfs url

Parameters:

url (str) – url to get the path for

Returns:

the matching path

Return type:

str

Raises:

Example (the mountpoint tmp points to /tmp):

>>> fastr.vfs.url_to_path('vfs://tmp/file.ext')
'/tmp/file.ext'
class fastr.plugins.VirtualFileSystemRegularExpression

Bases: fastr.core.ioplugin.IOPlugin

The VirtualFileSystemValueList an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs.

A vfsregex:// URL is a vfs URL that can contain regular expressions on every level of the path. The regular expressions follow the re module definitions.

An example of a valid URLs would be:

vfsregex://tmp/network_dir/.*/.*/__fastr_result__.pickle.gz
vfsregex://tmp/network_dir/nodeX/(?P<id>.*)/__fastr_result__.pickle.gz

The first URL would result in all the __fastr_result__.pickle.gz in the working directory of a Network. The second URL would only result in the file for a specific node (nodeX), but by adding the named group id using (?P<id>.*) the sample id of the data is automatically set to that group (see Regular Expression Syntax under the special characters for more info on named groups in regular expression).

Concretely if we would have a directory vfs://mount/somedir containing:

image_1/Image.nii
image_2/image.nii
image_3/anotherimage.nii
image_5/inconsistentnamingftw.nii

we could match these files using vfsregex://mount/somedir/(?P<id>image_\d+)/.*\.nii which would result in the following source data after expanding the URL:

{'image_1': 'vfs://mount/somedir/image_1/Image.nii',
 'image_2': 'vfs://mount/somedir/image_2/image.nii',
 'image_3': 'vfs://mount/somedir/image_3/anotherimage.nii',
 'image_5': 'vfs://mount/somedir/image_5/inconsistentnamingftw.nii'}

Showing the power of this regular expression filtering. Also it shows how the ID group from the URL can be used to have sensible sample ids.

Warning

due to the nature of regexp on multiple levels, this method can be slow when having many matches on the lower level of the path (because the tree of potential matches grows) or when directories that are parts of the path are very large.

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/virtualfilesystemregularexpression.py'
module = <module 'virtualfilesystemregularexpression' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/virtualfilesystemregularexpression.pyc'>
scheme = 'vfsregex'
class fastr.plugins.VirtualFileSystemValueList

Bases: fastr.core.ioplugin.IOPlugin

The VirtualFileSystemValueList an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs. A vfslist:// URL basically is a url that points to a file using vfs. This file then contains a number lines each containing another URL.

If the contents of a file vfs://mount/some/path/contents would be:

vfs://mount/some/path/file1.txt
vfs://mount/some/path/file2.txt
vfs://mount/some/path/file3.txt
vfs://mount/some/path/file4.txt

Then using the URL vfslist://mount/some/path/contents as source data would result in the four files being pulled.

Note

The URLs in a vfslist file do not have to use the vfs scheme, but can use any scheme known to the Fastr system.

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/virtualfilesystemvaluelist.py'
module = <module 'virtualfilesystemvaluelist' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/virtualfilesystemvaluelist.pyc'>
scheme = 'vfslist'
class fastr.plugins.XNATStorage

Bases: fastr.core.ioplugin.IOPlugin

Warning

As this IOPlugin is under development, it has not been thoroughly tested.

The XNATStorage plugin is an IOPlugin that can download data from and upload data to an XNAT server. It uses its own xnat:// URL scheme. This is a scheme specific for this plugin and though it looks somewhat like the XNAT rest interface, a different type or URL.

Data resources can be access directly by a data url:

xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/experiment001/scans/T1/resources/DICOM
xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/*_BRAIN/scans/T1/resources/DICOM

In the second URL you can see a wildcard being used. This is possible at long as it resolves to exactly one item.

The id query element will change the field from the default experiment to subject and the label query element sets the use of the label as the fastr id (instead of the XNAT id) to True (the default is False)

To disable https transport and use http instead the query string can be modified to add insecure=true. This will make the plugin send requests over http:

xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/*_BRAIN/scans/T1/resources/DICOM?insecure=true

For sinks it is import to know where to save the data. Sometimes you want to save data in a new assessor/resource and it needs to be created. To allow the Fastr sink to create an object in XNAT, you have to supply the type as a query parameter:

xnat://xnat.bmia.nl/data/archive/projects/sandbox/subjects/S01/experiments/_BRAIN/assessors/test_assessor/resources/IMAGE/files/image.nii.gz?resource_type=xnat:resourceCatalog&assessor_type=xnat:qcAssessmentData

Valid options are: subject_type, experiment_type, assessor_type, scan_type, and resource_type.

If you want to do a search where multiple resources are returned, it is possible to use a search url:

xnat://xnat.example.com/search?projects=sandbox&subjects=subject[0-9][0-9][0-9]&experiments=*_BRAIN&scans=T1&resources=DICOM

This will return all DICOMs for the T1 scans for experiments that end with _BRAIN that belong to a subjectXXX where XXX is a 3 digit number. By default the ID for the samples will be the experiment XNAT ID (e.g. XNAT_E00123). The wildcards that can be the used are the same UNIX shell-style wildcards as provided by the module fnmatch.

It is possible to change the id to a different fields id or label. Valid fields are project, subject, experiment, scan, and resource:

xnat://xnat.example.com/search?projects=sandbox&subjects=subject[0-9][0-9][0-9]&experiments=*_BRAIN&scans=T1&resources=DICOM&id=subject&label=true

The following variables can be set in the search query:

variable default usage
projects * The project(s) to select, can contain wildcards (see fnmatch)
subjects * The subject(s) to select, can contain wildcards (see fnmatch)
experiments * The experiment(s) to select, can contain wildcards (see fnmatch)
scans * The scan(s) to select, can contain wildcards (see fnmatch)
resources * The resource(s) to select, can contain wildcards (see fnmatch)
id experiment What field to use a the id, can be: project, subject, experiment, scan, or resource
label false Indicate the XNAT label should be used as fastr id, options true or false
insecure false Change the url scheme to be used to http instead of https
verify true (Dis)able the verification of SSL certificates
regex false Change search to use regex re.match() instead of fnmatch for matching
overwrite false Tell XNAT to overwrite existing files if a file with the name is already present

For storing credentials the .netrc file can be used. This is a common way to store credentials on UNIX systems. It is required that the file is only accessible by the owner only or a NetrcParseError will be raised. A netrc file is really easy to create, as its entries look like:

machine xnat.example.com
        login username
        password secret123

See the netrc module or the GNU inet utils website for more information about the .netrc file.

Note

On windows the location of the netrc file is assumed to be os.path.expanduser('~/_netrc'). The leading underscore is because windows does not like filename starting with a dot.

Note

For scan the label will be the scan type (this is initially the same as the series description, but can be updated manually or the XNAT scan type cleanup).

Warning

labels in XNAT are not guaranteed to be unique, so be careful when using them as the sample ID.

For background on XNAT, see the XNAT API DIRECTORY for the REST API of XNAT.

__abstractmethods__ = frozenset([])
__init__()[source]

Initialization for the IOPlugin

Returns:newly created IOPlugin
__module__ = 'fastr.plugins'
cleanup()[source]

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

connect(server, path='', insecure=False, verify=True)[source]
expand_url(url)[source]

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand
Returns:the resulting url(s), a tuple if multiple, otherwise a str
Return type:str or tuple of str
fetch_url(inurl, outpath)[source]

Get the file(s) or values from XNAT.

Parameters:
  • inurl – url to the item in the data store
  • outpath – path where to store the fetch data locally
filename = '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/xnatstorage.py'
module = <module 'xnatstorage' from '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/2.1.1/local/lib/python2.7/site-packages/fastr-2.1.0-py2.7.egg/fastr/resources/plugins/ioplugins/xnatstorage.pyc'>
put_url(inpath, outurl)[source]

Upload the files to the XNAT storage

Parameters:
  • inpath – path to the local data
  • outurl – url to where to store the data in the external data store.
scheme = 'xnat'
server
static upload(resource, in_path, location, retries=3, overwrite=False)[source]
xnat
fastr.plugins.json

alias of fastr.plugins.JsonCollector

fastr.plugins.path

alias of fastr.plugins.PathCollector

fastr.plugins.stdout

alias of fastr.plugins.StdoutCollector

version Module

This module keeps track of the version of the currently used Fastr framework. It can check its version from mercurial or a saved file

fastr.version.clear_version()[source]

Remove the cached version info

fastr.version.get_base_version()[source]

Get the version from the top-level version file

Returns:the version
Rtype str:
fastr.version.get_hg_info()[source]

Read information about the current mercurial branch and revision

Returns:tuple containing head revision and branch
fastr.version.get_saved_version()[source]

Get cached version from file

Returns:tuple with version, head revision and branch
fastr.version.save_version(current_version, current_hg_head, current_hg_branch)[source]

Cache the version information (useful for when installing)

Parameters:
  • current_version (str) – version
  • current_hg_head (str) – mercurial head revision
  • current_hg_branch (str) – mercurial branch
Returns:

Subpackages