planning Package

planning Package

inputgroup Module

class fastr.planning.inputgroup.InputGroup(*args, **kwargs)[source]

Bases: collections.OrderedDict, fastr.core.dimension.HasDimensions

A class representing a group of inputs. Input groups allow the

__abstractmethods__ = frozenset()
__getitem__(key)[source]

x.__getitem__(y) <==> x[y]

__init__(*args, **kwargs)

Create a new InputGroup representation

Parameters:
  • parent (NodeRun) – the parent node
  • id (str) – the id of the input group
Raises:

FastrTypeError – if parent is not a NodeRun

Note

This is a wrapped version of fastr.planning.inputgroup.__init__ which triggers an update of the object after being called

__module__ = 'fastr.planning.inputgroup'
__setitem__(*args, **kwargs)

Assign an input to this input group.

Parameters:
  • key (str) – id of the input
  • value (Input) – the input to assign
Raises:

FastrTypeError – if value of valid type

Note

This is a wrapped version of fastr.planning.inputgroup.__setitem__ which triggers an update of the object after being called

__updatefunc__()[source]

Update the InputGroup. Triggers when a change is made to the content of the InputGroup. Automatically recalculates the size, primary Input etc.

__updatetriggers__ = ['__init__', '__setitem__', '__delitem__', 'clear', 'pop', 'popitem', 'setdefault', 'update']
dimensions

The dimensions of this InputGroup

empty

Bool indicating that this InputGroup is empty (has no data connected)

find_source_index(target_size, target_dimnames, source_size, source_dimnames, target_index)[source]
fullid
iterinputvalues

Iterate over the item in this InputGroup

Returns:iterator yielding SampleItems
parent

The parent node of this InputGroup

primary

The primary Input in this InputGroup. The primary Input is the Input that defines the size of this InputGroup. In case of ties it will be the first in the tool definition.

classmethod solve_broadcast(target_size, target_dimnames, source_size, source_dimnames, target_index, nodegroups=None)[source]

inputgroupcombiner Module

class fastr.planning.inputgroupcombiner.BaseInputGroupCombiner(parent)[source]

Bases: fastr.core.dimension.HasDimensions

An object that takes the different input groups and combines them in the correct way.

__abstractmethods__ = frozenset({'merge', 'iter_input_groups', 'unmerge'})
__init__(parent)[source]

Initialize self. See help(type(self)) for accurate signature.

__iter__()[source]
__module__ = 'fastr.planning.inputgroupcombiner'
dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
fullid

The full id of the InputGroupCombiner

input_groups
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine
Returns:combined list
merge_failed_annotations(list_of_failed_annotations)[source]
merge_payloads(sample_payloads)[source]
merge_sample_data(list_of_sample_data)[source]
merge_sample_id(list_of_sample_ids)[source]
merge_sample_index(list_of_sample_indexes)[source]
merge_sample_jobs(list_of_sample_jobs)[source]
unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters:item (list) – the item to unmerge
Returns:items
Return type:OrderedDict
update()[source]
class fastr.planning.inputgroupcombiner.DefaultInputGroupCombiner(parent)[source]

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The default input group combiner combines the input group in a cross product version, taking each combinations of samples between the input groups. So if there are two input groups with one with size N and the other with size M x P the result would be N x M x P samples, with all possible combinations of the samples in each input group.

__abstractmethods__ = frozenset()
__module__ = 'fastr.planning.inputgroupcombiner'
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine
Returns:combined list
unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters:item (list) – the item to unmerge
Returns:items
Return type:OrderedDict
class fastr.planning.inputgroupcombiner.MergingInputGroupCombiner(input_groups, merge_dimension)[source]

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The merging input group combiner takes a similar approach as the default combiner but merges dimensions that are the same. If input group A has N(3) x M(2) samples and B has M(2) x P(4) it wil not result in N(3) x M(2) x M(2) x P(4), but merge the dimensions M leading to N(3) x M(2) x P(4) in resulting size.

__abstractmethods__ = frozenset()
__init__(input_groups, merge_dimension)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.planning.inputgroupcombiner'
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine
Returns:combined list
unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters:item (list) – the item to unmerge
Returns:items
Return type:OrderedDict
update()[source]

inputoutput Module

Classes for arranging the input and output for nodes.

Exported classes:

Input – An input for a node (holding datatype). Output – The output of a node (holding datatype and value). ConstantOutput – The output of a node (holding datatype and value).

Warning

Don’t mess with the Link, Input and Output internals from other places. There will be a huge chances of breaking the network functionality!

class fastr.planning.inputoutput.AdvancedFlowOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

Output for nodes that have an advanced flow. This means that the output sample id and index is not the same as the input sample id and index. The AdvancedFlowOutput has one extra dimensions that is created by the Node.

__abstractmethods__ = frozenset()
__module__ = 'fastr.planning.inputoutput'
dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.BaseInput(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all inputs.

__abstractmethods__ = frozenset({'_update', 'itersubinputs', 'fullid', 'dimensions'})
__init__(node, description)[source]

Instantiate a BaseInput

Parameters:
  • node – the parent node the input/output belongs to.
  • description – the ParameterDescription describing the input/output.
Returns:

the created BaseInput

Raises:
__lshift__(other)[source]
__module__ = 'fastr.planning.inputoutput'
__rrshift__(other)[source]
check_cardinality(key=None, planning=False)[source]

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters:key – sample_index (tuple of int) or SampleId for desired sample
Returns:flag indicating that the cardinality is correct
Return type:bool
Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.
description_type

alias of fastr.core.interface.InputSpec

itersubinputs()[source]

Iterator over the SubInputs

Returns:iterator

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
class fastr.planning.inputoutput.BaseInputOutput(node, description)[source]

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

Base class for Input and Output classes. It mainly implements the properties to access the data from the underlying ParameterDescription.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'fullid'})
__getstate__()[source]

Retrieve the state of the BaseInputOutput

Returns:the state of the object
Rtype dict:
__init__(node, description)[source]

Instantiate a BaseInputOutput

Parameters:
  • node – the parent node the input/output belongs to.
  • description – the ParameterDescription describing the input/output.
Returns:

created BaseInputOutput

Raises:
__iter__()[source]

This function is blocked to avoid support for iteration using a lecacy __getitem__ method.

Returns:None
Raises:FastrNotImplementedError – always
__module__ = 'fastr.planning.inputoutput'
__ne__(other)[source]

Check two Node instances for inequality. This is the inverse of __eq__

Parameters:other (BaseInputOutput) – the other instances to compare to
Returns:True if unequal, False otherwise
__repr__()[source]

Get a string representation for the Input/Output

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the BaseInputOutput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
cardinality(key=None, job_data=None)[source]

Determine the cardinality of this Input/Output. Optionally a key can be given to determine for a sample.

Parameters:key – key for a specific sample
Returns:the cardinality
Return type:int, sympy.Symbol, or None
check_cardinality(key=None)[source]

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters:key – sample_index (tuple of int) or SampleId for desired sample
Returns:flag indicating that the cardinality is correct
Return type:bool
Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.
datatype

The datatype of this Input/Output

description

The description object of this input/output

description_type = None
fullid

The fullid of the Input/Output, the fullid should be unnique and makes the object retrievable by the network.

id

Id of the Input/Output

node

The NodeRun to which this Input/Output belongs

required

Flag indicating that the Input/Output is required

class fastr.planning.inputoutput.BaseOutput(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all outputs.

__abstractmethods__ = frozenset({'_update', 'fullid', 'dimensions'})
__init__(node, description)[source]

Instantiate a BaseOutput

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created BaseOutput

Raises:
__module__ = 'fastr.planning.inputoutput'
automatic

Flag indicating that the Output is generated automatically without being specified on the command line

blocking

Flag indicating that this Output will cause blocking in the execution

description_type

alias of fastr.core.interface.OutputSpec

class fastr.planning.inputoutput.Input(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInput

Class representing an input of a node. Such an input will be connected to the output of another node or the output of an constant node to provide the input value.

__abstractmethods__ = frozenset()
__eq__(other)[source]

Compare two Input instances with each other. This function ignores the parent node and update status, but tests rest of the dict for equality.

Parameters:other (Input) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getitem__(key)[source]

Retrieve an item from this Input.

Parameters:

key (Union[int, str]) – the key of the requested item

Return type:

SubInput

Returns:

The SubInput corresponding with the key will be returned.

Raises:
__getstate__()[source]

Retrieve the state of the Input

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(node, description)[source]

Instantiate an input.

Parameters:
  • node (NodeRun) – the parent node of this input.
  • description (ParameterDescription) – the ParameterDescription of the input.
Returns:

the created Input

__module__ = 'fastr.planning.inputoutput'
__setitem__(key, value)[source]

Create a link between a SubInput of this Inputs and an Output/Constant

Parameters:
  • key (int, str) – the key of the SubInput
  • value (BaseOutput, list, tuple, dict, OrderedDict) – the target to link, can be an output or a value to create a constant for
Raises:

FastrTypeError – if key is not of a valid type

__setstate__(state)[source]

Set the state of the Input by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Input

Returns:the string version
Return type:str
append(value)[source]

When you want to append a link to an Input, you can use the append property. This will automatically create a new SubInput to link to.

example:

>>> link = node2['input'].append(node1['output'])

will create a new SubInput in node2[‘input’] and link to that.

cardinality(key=None, job_data=None)[source]

Cardinality for an Input is the sum the cardinalities of the SubInputs, unless defined otherwise.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
clear()[source]
datatype

The datatype of this Input

dimensions

The list names of the dimensions in this Input. This will be a list of str.

fullid

The full defining ID for the Input

Return type:str
get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this Input

Returns:list of all connected Nodes
Return type:list
get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this Input

Returns:tuple of all connected Outputs
Return type:tuple
index(value)[source]

Find index of a SubInput

Parameters:value (SubInput) – the SubInput to find the index of
Returns:key
Return type:int, str
input_group

The id of the InputGroup this Input belongs to.

Return type:str
insert(index)[source]

Insert a new SubInput at index in the sources list

Parameters:key (int) – positive integer for position in _source list to insert to
Returns:newly inserted SubInput
Return type:SubInput
itersubinputs()[source]

Iterate over the SubInputs in this Input.

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
remove(value)[source]

Remove a SubInput from the SubInputs list based on the connected Link.

Parameters:value (SubInput, <fastr.planning.inputoutput.SubInput>`) – the SubInput or SubLink to removed from this Input
source

The mapping of SubInputs that are connected and have more than 0 elements.

class fastr.planning.inputoutput.MacroInput(node, description)[source]

Bases: fastr.planning.inputoutput.Input

__abstractmethods__ = frozenset()
__module__ = 'fastr.planning.inputoutput'
input_group

The id of the InputGroup this Input belongs to.

class fastr.planning.inputoutput.MacroOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

__abstractmethods__ = frozenset()
__module__ = 'fastr.planning.inputoutput'
dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.Output(node, description)[source]

Bases: fastr.planning.inputoutput.BaseOutput

Class representing an output of a node. It holds the output values of the tool ran. Output fields can be connected to inputs of other nodes.

__abstractmethods__ = frozenset()
__eq__(other)[source]

Compare two Output instances with each other. This function ignores the parent node, listeners and update status, but tests rest of the dict for equality.

Parameters:other (Output) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getitem__(key)[source]

Retrieve an item from this Output. The returned value depends on what type of key used:

  • Retrieving data using index tuple: [index_tuple]
  • Retrieving data sample_id str: [SampleId]
  • Retrieving a list of data using SampleId list: [sample_id1, …, sample_idN]
  • Retrieving a SubOutput using an int or slice: [n] or [n:m]
Parameters:key (Union[int, slice]) – the key of the requested suboutput, can be a numberor slice
Return type:SubOutput
Returns:the SubOutput for the corresponding index
Raises:FastrTypeError – if key is not of a valid type
__getstate__()[source]

Retrieve the state of the Output

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(node, description)[source]

Instantiate an Output

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created Output

Raises:
__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the Output by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Output

Returns:the string version
Return type:str
cardinality()[source]

Cardinality of this Output, may depend on the inputs of the parent Node.

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:
datatype

The datatype of this Output

dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

fullid

The full defining ID for the Output

listeners

The list of Links connected to this Output.

preferred_types

The list of preferred DataTypes for this Output.

resulting_datatype

The DataType that will the results of this Output will have.

valid

Check if the output is valid, i.e. has a valid cardinality

class fastr.planning.inputoutput.SourceOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

Output for a SourceNodeRun, this type of Output determines the cardinality in a different way than a normal NodeRun.

__abstractmethods__ = frozenset()
__init__(node, description)[source]

Instantiate a FlowOutput

Parameters:
  • node – the parent node the output belongs to.
  • description – the ParameterDescription describing the output.
Returns:

created FlowOutput

Raises:
__module__ = 'fastr.planning.inputoutput'
cardinality()[source]

Cardinality of this SourceOutput, may depend on the inputs of the parent NodeRun.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
linearized

A linearized version of the sample data, this is lazily cached linearized version of the underlying SampleCollection.

class fastr.planning.inputoutput.SubInput(input_)[source]

Bases: fastr.planning.inputoutput.BaseInput

This class is used by Input to allow for multiple links to an Input. The SubInput class can hold only a single Link to a (Sub)Output, but behaves very similar to an Input otherwise.

__abstractmethods__ = frozenset()
__eq__(other)[source]

Compare two SubInput instances with each other. This function ignores the parent, node, source and update status, but tests rest of the dict for equality.

Parameters:other (SubInput) – the other instances to compare to
Returns:True if equal, False otherwise
__getitem__(key)[source]

Retrieve an item from this SubInput.

Parameters:key (int) – the index of the requested item
Returns:the corresponding SubInput
Return type:SubInput
Raises:FastrTypeError – if key is not of a valid type

Note

As a SubInput has only one SubInput, only requesting int key 0 or -1 is allowed, and it will return self

__getstate__()[source]

Retrieve the state of the SubInput

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(input_)[source]

Instantiate an SubInput.

Parameters:input (Input) – the parent of this SubInput.
Returns:the created SubInput
__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the SubInput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the SubInput

Returns:the string version
Return type:str
cardinality(key=None, job_data=None)[source]

Get the cardinality for this SubInput. The cardinality for a SubInputs is defined by the incoming link.

Parameters:key (SampleIndex or SampleId) – key for a specific sample, can be sample index or id
Returns:the cardinality
Return type:int, sympy.Symbol, or None
description

The description object of this input/output

dimensions

List of dimension for this SubInput

fullid

The full defining ID for the SubInput

get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this SubInput

Returns:list of all connected Nodes
Return type:list
get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this SubInput

Returns:list of all connected Outputs
Return type:list
input_group

The id of the InputGroup this SubInputs parent belongs to.

iteritems()[source]

Iterate over the SampleItems that are in the SubInput.

Returns:iterator yielding SampleItem objects
itersubinputs()[source]

Iterate over SubInputs (for a SubInput it will yield self and stop iterating after that)

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
node

The Node to which this SubInputs parent belongs

remove(value)[source]

Remove a SubInput from parent Input.

Parameters:value (SubInput) – the SubInput to removed from this Input
source

A list with the source Link. The list is to be compatible with Input

source_output

The Output linked to this SubInput

class fastr.planning.inputoutput.SubOutput(output, index)[source]

Bases: fastr.planning.inputoutput.Output

The SubOutput is an Output that represents a slice of another Output.

__abstractmethods__ = frozenset()
__eq__(other)[source]

Compare two SubOutput instances with each other. This function ignores the parent, node and update status, but tests rest of the dict for equality. equality

Parameters:other (SubOutput) – the other instances to compare to
Returns:True if equal, False otherwise
Return type:bool
__getstate__()[source]

Retrieve the state of the SubOutput

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(output, index)[source]

Instantiate a SubOutput

Parameters:
  • output – the parent output the suboutput slices.
  • index (int or slice) – the way to slice the parent output
Returns:

created SubOutput

Raises:
__len__()[source]

Return the length of the Output.

Note

In a SubOutput this is always 1.

__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the SubOutput by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the SubOutput

Returns:the string version
Return type:str
cardinality()[source]

Cardinality of this SubOutput depends on the parent Output and self.index

Parameters:

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:
datatype

The datatype of this SubOutput

fullid

The full defining ID for the SubOutput

indexrep

Simple representation of the index.

listeners

The list of Links connected to this Output.

node

The NodeRun to which this SubOutput belongs

preferred_types

The list of preferred DataTypes for this SubOutput.

resulting_datatype

The DataType that will the results of this SubOutput will have.

samples

The SampleCollection for this SubOutput

network Module

Network module containing Network facilitators and analysers.

class fastr.planning.network.Network(id_='unnamed_network', version=None, filename=None)[source]

Bases: fastr.abc.serializable.Serializable

The NetworkRun contains the entire Run state for a Network execution. It has a working copy of the network, but also includes all temporary data required for the execution. These objects are meant to be single use.

NETWORK_DUMP_FILE_NAME = '__fastr_network__.json'
SINK_DUMP_FILE_NAME = '__sink_data__.json'
SOURCE_DUMP_FILE_NAME = '__source_data__.pickle.gz'
__dataschemafile__ = 'Network.schema.json'
__eq__(other)[source]

Compare two Networks and see if they are equal.

Parameters:other (Network) –
Returns:flag indicating that the Networks are the same
Return type:bool
__getitem__(item)[source]

Get an item by its fullid. The fullid can point to a link, node, input, output or even subinput/suboutput.

Parameters:item (str,unicode) – fullid of the item to retrieve
Returns:the requested item
__getstate__()[source]

Retrieve the state of the Network

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(id_='unnamed_network', version=None, filename=None)[source]

Create a new, empty Network

Parameters:name (str) – name of the Network
Returns:newly created Network
Raises:OSError – if the tmp mount in the config is not a writable directory
__module__ = 'fastr.planning.network'
__ne__(other)[source]

Tests for non-equality, this is the negated version __eq__

__repr__()[source]

Return repr(self).

__setstate__(state)[source]

Set the state of the Network by the given state. This completely overwrites the old state!

Parameters:state (dict) – The state to populate the object with
Returns:None

Add a Link to the Network. Make sure the link is in the link list and the link parent is set to this Network

Parameters:

link (Link) – link to add

Raises:
add_node(node)[source]

Add a Node to the Network. Make sure the node is in the node list and the node parent is set to this Network

Parameters:node (Node) – node to add
Raises:FastrTypeError – if node is incorrectly typed
add_stepid(stepid, node)[source]

Add a Node to a specific step id

Parameters:
  • stepid (str) – the stepid that the node will be added to
  • node (Node) – the node to add to the stepid
check_id(id_)[source]

Check if an id for an object is valid and unused in the Network. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:
create_constant(datatype, data, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a ConstantNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the constant node
  • data (datatype or list of datatype) – The data to hold in the constant node
  • id (str) – The id of the constant node to be created
  • stepid (str) – The stepid to add the created constant node to
  • resources – The resources required to run this node
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created constant node

Return type:

ConstantNode

Create a link between two Nodes and add it to the current Network.

Parameters:
  • source (BaseOutput) – the output that is the source of the link
  • target (BaseInput) – the input that is the target of the link
  • id (str) – the id of the link
Returns:

the created link

Type:

Link

create_macro(network, resources=None, id_=None)[source]
create_node(tool, tool_version, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a Node in this Network. The Node will be automatically added to the Network.

Parameters:
  • tool (Tool) – The Tool to base the Node on
  • id (str) – The id of the node to be created
  • stepid (str) – The stepid to add the created node to
  • resources – The resources required to run this node
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created node

Return type:

Node

create_reference(source_data, output_directory)[source]
create_sink(datatype, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a SinkNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the sink node
  • id (str) – The id of the sink node to be created
  • stepid (str) – The stepid to add the created sink node to
  • resources – The resources required to run this node
Returns:

the newly created sink node

Return type:

SinkNode

create_source(datatype, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a SourceNode in this Network. The Node will be automatically added to the Network.

Parameters:
  • datatype (BaseDataType) – The DataType of the source source_node
  • id (str) – The id of the source source_node to be created
  • stepid (str) – The stepid to add the created source source_node to
  • resources – The resources required to run this node
  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.
Returns:

the newly created source source_node

Return type:

SourceNode

draw(name=None, draw_dimensions=True, hide_unconnected=True, context=None, graph=None, expand_macro=False)[source]
draw_network(name='network_layout', img_format='svg', draw_dimension=True, hide_unconnected=True, expand_macro=False)[source]

Output a dot file and try to convert it to an image file.

Parameters:img_format (str) – extension of the image format to convert to
Returns:path of the image created or None if failed
Return type:str or None
execute(sourcedata, sinkdata, blocking=True, **kwargs)[source]
fullid

The fullid of the Network, within the network scope

global_id

The global id of the Network, this is different for networks used in macronodes, as they still have parents.

id

The id of the Network. This is a read only property.

is_valid()[source]
namespace = None

The namespace this network lives in, this will be set by the NetworkManager on load

nodegroups

Give an overview of the nodegroups in the network

ns_id

The namespace and id of the Tool

remove(value)[source]

Remove an item from the Network.

Parameters:value (Node or Link) – the item to remove
classmethod test(reference_data_dir, network=None, source_data=None, force_remove_temp=False, tmp_results_dir=None)[source]

Execute the network with the source data specified and test the results against the refence data. This effectively tests the network execution.

Parameters:
  • reference_data_dir (str) – The path or vfs url of reference data to compare with
  • source_data (dict) – The source data to use
  • force_remove_temp – Make sure the tmp results directory is cleaned at end of test
  • tmp_results_dir – Path to results directory

node Module

A module to maintain a network node.

Exported classes:

Node – A class encapsulating a tool. ConstantNode – A node encapsulating an Output to set scalar values. SourceNode – A class providing a handle to a file.

class fastr.planning.node.AdvancedFlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.FlowNode

__abstractmethods__ = frozenset()
__module__ = 'fastr.planning.node'
class fastr.planning.node.BaseNode[source]

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

NODE_TYPES = {'AdvancedFlowNode': <class 'fastr.planning.node.AdvancedFlowNode'>, 'ConstantNode': <class 'fastr.planning.node.ConstantNode'>, 'FlowNode': <class 'fastr.planning.node.FlowNode'>, 'MacroNode': <class 'fastr.planning.node.MacroNode'>, 'Node': <class 'fastr.planning.node.Node'>, 'SinkNode': <class 'fastr.planning.node.SinkNode'>, 'SourceNode': <class 'fastr.planning.node.SourceNode'>}
__abstractmethods__ = frozenset({'_update', 'dimensions'})
classmethod __init_subclass__(**kwargs)[source]

Register nodes in class for easly location

__module__ = 'fastr.planning.node'
class fastr.planning.node.ConstantNode(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.SourceNode

Class encapsulating one output for which a value can be set. For example used to set a scalar value to the input of a node.

__abstractmethods__ = frozenset()
__dataschemafile__ = 'ConstantNode.schema.json'
__getstate__()[source]

Retrieve the state of the ConstantNode

Returns:the state of the object
Rtype dict:
__init__(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the ConstantNode.

Parameters:
  • datatype – The datatype of the output.
  • data – the prefilled data to use.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a constant using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')

or alternatively create a constant node by assigning data to an item in an InputDict:

>>> node_a.inputs['in'] = ['some', 'data']

which automatically creates and links a ConstantNode to the specified Input

__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the ConstantNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
data

The data stored in this constant node

draw(context, graph, color=None)[source]
print_value
set_data(data=None, ids=None)[source]

Set the data of this constant node in the correct way. This is mainly for compatibility with the parent class SourceNode

Parameters:
  • data (dict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
class fastr.planning.node.FlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

A Flow Node is a special subclass of Nodes in which the amount of samples can vary per Output. This allows non-default data flows.

__abstractmethods__ = frozenset()
__init__(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiate a flow node.

Parameters:
  • tool (Tool) – The tool to base the node on
  • id (str) – the id of the node
  • parent (Network) – the parent network of the node
Returns:

the newly created FlowNode

__module__ = 'fastr.planning.node'
blocking

A FlowNode is (for the moment) always considered blocking.

Returns:True
dimensions

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

outputsize

Size of the outputs in this Node

class fastr.planning.node.InputDict[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Implements helper functions for the easy linking syntax.

__module__ = 'fastr.planning.node'
__setitem__(key, value)[source]

Set an item in the input dictionary. The behaviour depends on the type of the value. For a BaseInput, the input will simply be added to the list of inputs. For a BaseOutput, a link between the output and input will be created.

Parameters:
  • key (str) – id of the input to assign/link
  • value (BaseInput or BaseOutput) – either the input to add or the output to link
  • dict_setitem – the setitem function to use for the underlying OrderedDict insert
class fastr.planning.node.MacroNode(value, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

MacroNode encapsulates an entire network in a single node.

__abstractmethods__ = frozenset()
__eq__(other)[source]

Compare two MacroNode instances with each other. This function ignores the parent and update status, but tests rest of the dict for equality. equality

Parameters:other (MacroNode) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the MacroNode

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(value, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]
Parameters:network (Network) – network to create macronode for
__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
draw(context, graph, color=None)[source]
get_output_info(output)[source]

This functions maps the output dimensions based on the input dimensions of the macro. This is cached for speed as this can become rather costly otherwise

Parameters:output – output to get info for
Returns:tuple of Dimensions
network
class fastr.planning.node.Node(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.BaseNode

The class encapsulating a node in the network. The node is responsible for setting and checking inputs and outputs based on the description provided by a tool instance.

__abstractmethods__ = frozenset()
__dataschemafile__ = 'Node.schema.json'
__eq__(other)[source]

Check two Node instances for equality.

Parameters:other (Node) – the other instances to compare to
Returns:True if equal, False otherwise
__getstate__()[source]

Retrieve the state of the Node

Returns:the state of the object
Rtype dict:
__hash__ = None
__init__(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiate a node.

Parameters:
  • tool (Tool) – The tool to base the node on
  • id (str) – the id of the node
  • node_class (str) – The class of the NodeRun to create (e.g. SourceNodeRun, NodeRun)
  • parent (Network) – the parent network of the node
  • cores (int) – number of cores required for executing this Node
  • memory (str) – amount of memory required in the form d+[mMgG] where M is for megabyte and G for gigabyte
  • walltime (str) – amount of time required in second or in the form HOURS:MINUTES:SECOND
Returns:

the newly created Node

__module__ = 'fastr.planning.node'
__ne__(other)[source]

Check two Node instances for inequality. This is the inverse of __eq__

Parameters:other (Node) – the other instances to compare to
Returns:True if unequal, False otherwise
__repr__()[source]

Get a string representation for the Node

Returns:the string representation
Return type:str
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
__str__()[source]

Get a string version for the Node

Returns:the string version
Return type:str
blocking

Indicate that the results of this Node cannot be determined without first executing the Node, causing a blockage in the creation of jobs. A blocking Nodes causes the Chunk borders.

classmethod createobj(state, network=None)[source]

Create object function for generic objects

Parameters:
  • cls – The class to create
  • state – The state to use to create the Link
  • network – the parent Network
Returns:

newly created Link

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions
Return type:tuple
dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

draw(context, graph, color=None)[source]
draw_id(context)[source]
find_source_index(target_index, target, source)[source]
fullid

The full defining ID for the Node inside the network

get_sourced_nodes()[source]

A list of all Nodes connected as sources to this Node

Returns:list of all nodes that are connected to an input of this node
global_id

The global defining ID for the Node from the main network (goes out of macro nodes to root network)

id

The id of the Node

input_groups
A list of input groups for this Node. An input group is InputGroup
object filled according to the Node
inputs = None

A list of inputs of this Node

listeners

All the listeners requesting output of this node, this means the listeners of all Outputs and SubOutputs

merge_dimensions
name

Name of the Tool the Node was based on. In case a Toolless Node was used the class name is given.

nodegroup
outputs = None

A list of outputs of this Node

outputsize

The size of output of this SourceNode

parent

The parent is the Network this Node is part of

status
tool
update_input_groups()[source]

Update all input groups in this node

class fastr.planning.node.OutputDict[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Only checks if the inserted values are actually outputs.

__module__ = 'fastr.planning.node'
__setitem__(key, value)[source]

Set an output.

Parameters:
  • key (str) – the of the item to set
  • value (BaseOutput) – the output to set
  • dict_setitem – the setitem function to use for the underlying OrderedDict insert
class fastr.planning.node.SinkNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

Class which handles where the output goes. This can be any kind of file, e.g. image files, textfiles, config files, etc.

__abstractmethods__ = frozenset()
__dataschemafile__ = 'SinkNode.schema.json'
__getstate__()[source]

Retrieve the state of the Node

Returns:the state of the object
Rtype dict:
__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The datatype of the output.
  • id – the id of the node to create
Returns:

newly created sink node

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> sink = network.create_sink(datatype=types['ITKImageFile'], id_='SinkN')
__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
datatype

The datatype of the data this sink can store.

draw(context, graph, color=None)[source]
input

The default input of the sink Node

class fastr.planning.node.SourceNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.FlowNode

Class providing a connection to data resources. This can be any kind of file, stream, database, etc from which data can be received.

__abstractmethods__ = frozenset()
__dataschemafile__ = 'SourceNode.schema.json'
__getstate__()[source]

Retrieve the state of the SourceNode

Returns:the state of the object
Rtype dict:
__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters:
  • datatype – The (id of) the datatype of the output.
  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a source using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')
__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the SourceNode by the given state.

Parameters:state (dict) – The state to populate the object with
Returns:None
datatype

The datatype of the data this source supplies.

dimensions

The dimensions in the SourceNode output. These will be reflected in the SampleIdLists.

draw(context, graph, color=None)[source]
nodegroup
output

Shorthand for self.outputs['output']

set_data(data, ids=None)[source]

Set the data of this source node.

Parameters:
  • data (dict, OrderedDict or list of urls) – the data to use
  • ids – if data is a list, a list of accompanying ids
sourcegroup
valid

This does nothing. It only overloads the valid method of Node(). The original is intended to check if the inputs are connected to some output. Since this class does not implement inputs, it is skipped.