planning Package

planning Package

inputgroup Module

class fastr.planning.inputgroup.InputGroup(*args, **kwargs)[source]

Bases: collections.OrderedDict, fastr.core.dimension.HasDimensions

A class representing a group of inputs. Input groups allow the

__abstractmethods__ = frozenset({})
__getitem__(key)[source]

x.__getitem__(y) <==> x[y]

__init__(*args, **kwargs)

Create a new InputGroup representation

Parameters
  • parent (NodeRun) – the parent node

  • id (str) – the id of the input group

Raises

FastrTypeError – if parent is not a NodeRun

Note

This is a wrapped version of fastr.planning.inputgroup.__init__ which triggers an update of the object after being called

__module__ = 'fastr.planning.inputgroup'
__setitem__(*args, **kwargs)

Assign an input to this input group.

Parameters
  • key (str) – id of the input

  • value (Input) – the input to assign

Raises

FastrTypeError – if value of valid type

Note

This is a wrapped version of fastr.planning.inputgroup.__setitem__ which triggers an update of the object after being called

__updatefunc__()[source]

Update the InputGroup. Triggers when a change is made to the content of the InputGroup. Automatically recalculates the size, primary Input etc.

__updatetriggers__ = ['__init__', '__setitem__', '__delitem__', 'clear', 'pop', 'popitem', 'setdefault', 'update']
dimensions

The dimensions of this InputGroup

empty

Bool indicating that this InputGroup is empty (has no data connected)

find_source_index(target_size, target_dimnames, source_size, source_dimnames, target_index)[source]
fullid
iterinputvalues

Iterate over the item in this InputGroup

Returns

iterator yielding SampleItems

parent

The parent node of this InputGroup

primary

The primary Input in this InputGroup. The primary Input is the Input that defines the size of this InputGroup. In case of ties it will be the first in the tool definition.

classmethod solve_broadcast(target_size, target_dimnames, source_size, source_dimnames, target_index, nodegroups=None)[source]

inputgroupcombiner Module

class fastr.planning.inputgroupcombiner.BaseInputGroupCombiner(parent)[source]

Bases: fastr.core.dimension.HasDimensions

An object that takes the different input groups and combines them in the correct way.

__abstractmethods__ = frozenset({'iter_input_groups', 'merge', 'unmerge'})
__init__(parent)[source]

Initialize self. See help(type(self)) for accurate signature.

__iter__()[source]
__module__ = 'fastr.planning.inputgroupcombiner'
dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns

dimensions

Return type

tuple

fullid

The full id of the InputGroupCombiner

input_groups
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters

list_of_items (list) – items to combine

Returns

combined list

merge_failed_annotations(list_of_failed_annotations)[source]
merge_payloads(sample_payloads)[source]
merge_sample_data(list_of_sample_data)[source]
merge_sample_id(list_of_sample_ids)[source]
merge_sample_index(list_of_sample_indexes)[source]
merge_sample_jobs(list_of_sample_jobs)[source]
unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters

item (list) – the item to unmerge

Returns

items

Return type

OrderedDict

update()[source]
class fastr.planning.inputgroupcombiner.DefaultInputGroupCombiner(parent)[source]

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The default input group combiner combines the input group in a cross product version, taking each combinations of samples between the input groups. So if there are two input groups with one with size N and the other with size M x P the result would be N x M x P samples, with all possible combinations of the samples in each input group.

__abstractmethods__ = frozenset({})
__module__ = 'fastr.planning.inputgroupcombiner'
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters

list_of_items (list) – items to combine

Returns

combined list

unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters

item (list) – the item to unmerge

Returns

items

Return type

OrderedDict

class fastr.planning.inputgroupcombiner.MergingInputGroupCombiner(input_groups, merge_dimension)[source]

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The merging input group combiner takes a similar approach as the default combiner but merges dimensions that are the same. If input group A has N(3) x M(2) samples and B has M(2) x P(4) it wil not result in N(3) x M(2) x M(2) x P(4), but merge the dimensions M leading to N(3) x M(2) x P(4) in resulting size.

__abstractmethods__ = frozenset({})
__init__(input_groups, merge_dimension)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.planning.inputgroupcombiner'
iter_input_groups()[source]

Iterate over all the merged samples :return:

merge(list_of_items)[source]

Given a list of items for each input group, it returns the combined list of items.

Parameters

list_of_items (list) – items to combine

Returns

combined list

unmerge(item)[source]

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())
Parameters

item (list) – the item to unmerge

Returns

items

Return type

OrderedDict

update()[source]

inputoutput Module

Classes for arranging the input and output for nodes.

Exported classes:

Input – An input for a node (holding datatype). Output – The output of a node (holding datatype and value). ConstantOutput – The output of a node (holding datatype and value).

Warning

Don’t mess with the Link, Input and Output internals from other places. There will be a huge chances of breaking the network functionality!

class fastr.planning.inputoutput.AdvancedFlowOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

Output for nodes that have an advanced flow. This means that the output sample id and index is not the same as the input sample id and index. The AdvancedFlowOutput has one extra dimensions that is created by the Node.

__abstractmethods__ = frozenset({})
__module__ = 'fastr.planning.inputoutput'
dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.BaseInput(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all inputs.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'fullid', 'itersubinputs'})
__init__(node, description)[source]

Instantiate a BaseInput

Parameters
  • node – the parent node the input/output belongs to.

  • description – the ParameterDescription describing the input/output.

Returns

the created BaseInput

Raises
__lshift__(other)[source]
__module__ = 'fastr.planning.inputoutput'
__rrshift__(other)[source]
check_cardinality(key=None, planning=False)[source]

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters

key – sample_index (tuple of int) or SampleId for desired sample

Returns

flag indicating that the cardinality is correct

Return type

bool

Raises

FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

constant_id()[source]

The id that should be used for a constant created to serve this input.

Return type

str

description_type

alias of fastr.core.interface.InputSpec

item_index
itersubinputs()[source]

Iterator over the SubInputs

Returns

iterator

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
class fastr.planning.inputoutput.BaseInputOutput(node, description)[source]

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

Base class for Input and Output classes. It mainly implements the properties to access the data from the underlying ParameterDescription.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'fullid'})
__getstate__()[source]

Retrieve the state of the BaseInputOutput

Returns

the state of the object

Rtype dict

__init__(node, description)[source]

Instantiate a BaseInputOutput

Parameters
  • node – the parent node the input/output belongs to.

  • description – the ParameterDescription describing the input/output.

Returns

created BaseInputOutput

Raises
__iter__()[source]

This function is blocked to avoid support for iteration using a lecacy __getitem__ method.

Returns

None

Raises

FastrNotImplementedError – always

__module__ = 'fastr.planning.inputoutput'
__ne__(other)[source]

Check two Node instances for inequality. This is the inverse of __eq__

Parameters

other (BaseInputOutput) – the other instances to compare to

Returns

True if unequal, False otherwise

__repr__()[source]

Get a string representation for the Input/Output

Returns

the string representation

Return type

str

__setstate__(state)[source]

Set the state of the BaseInputOutput by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

cardinality(key=None, job_data=None)[source]

Determine the cardinality of this Input/Output. Optionally a key can be given to determine for a sample.

Parameters

key – key for a specific sample

Returns

the cardinality

Return type

int, sympy.Symbol, or None

check_cardinality(key=None)[source]

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters

key – sample_index (tuple of int) or SampleId for desired sample

Returns

flag indicating that the cardinality is correct

Return type

bool

Raises

FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

datatype

The datatype of this Input/Output

description

The description object of this input/output

description_type = None
fullid

The fullid of the Input/Output, the fullid should be unnique and makes the object retrievable by the network.

id

Id of the Input/Output

node

The NodeRun to which this Input/Output belongs

required

Flag indicating that the Input/Output is required

class fastr.planning.inputoutput.BaseOutput(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all outputs.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'fullid'})
__init__(node, description)[source]

Instantiate a BaseOutput

Parameters
  • node – the parent node the output belongs to.

  • description – the ParameterDescription describing the output.

Returns

created BaseOutput

Raises
__module__ = 'fastr.planning.inputoutput'
automatic

Flag indicating that the Output is generated automatically without being specified on the command line

blocking

Flag indicating that this Output will cause blocking in the execution

description_type

alias of fastr.core.interface.OutputSpec

class fastr.planning.inputoutput.Input(node, description)[source]

Bases: fastr.planning.inputoutput.BaseInput

Class representing an input of a node. Such an input will be connected to the output of another node or the output of an constant node to provide the input value.

__abstractmethods__ = frozenset({})
__eq__(other)[source]

Compare two Input instances with each other. This function ignores the parent node and update status, but tests rest of the dict for equality.

Parameters

other (Input) – the other instances to compare to

Returns

True if equal, False otherwise

Return type

bool

__getitem__(key)[source]

Retrieve an item from this Input.

Parameters

key (Union[int, str]) – the key of the requested item

Return type

Union[SubInput, NamedSubInput]

Returns

The SubInput corresponding with the key will be returned.

Raises
__getstate__()[source]

Retrieve the state of the Input

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(node, description)[source]

Instantiate an input.

Parameters
  • node (NodeRun) – the parent node of this input.

  • description (ParameterDescription) – the ParameterDescription of the input.

Returns

the created Input

__module__ = 'fastr.planning.inputoutput'
__setitem__(key, value)[source]

Create a link between a SubInput of this Inputs and an Output/Constant

Parameters
  • key (int, str) – the key of the SubInput

  • value (BaseOutput, list, tuple, dict, OrderedDict) – the target to link, can be an output or a value to create a constant for

Raises

FastrTypeError – if key is not of a valid type

__setstate__(state)[source]

Set the state of the Input by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

__str__()[source]

Get a string version for the Input

Returns

the string version

Return type

str

append(value)[source]

When you want to append a link to an Input, you can use the append property. This will automatically create a new SubInput to link to.

example:

>>> link = node2['input'].append(node1['output'])

will create a new SubInput in node2[‘input’] and link to that.

cardinality(key=None, job_data=None)[source]

Cardinality for an Input is the sum the cardinalities of the SubInputs, unless defined otherwise.

Parameters

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns

the cardinality

Return type

int, sympy.Symbol, or None

clear()[source]
constant_id

The id for a constant node that is attached to this input.

Return type

str

datatype

The datatype of this Input

dimensions

The list names of the dimensions in this Input. This will be a list of str.

fullid

The full defining ID for the Input

Return type

str

get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this Input

Returns

list of all connected Nodes

Return type

list

get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this Input

Returns

tuple of all connected Outputs

Return type

tuple

id

Id of the Input/Output

index(value)[source]

Find index of a SubInput

Parameters

value (SubInput) – the SubInput to find the index of

Returns

key

Return type

int, str

input_group

The id of the InputGroup this Input belongs to.

Return type

str

insert(index)[source]

Insert a new SubInput at index in the sources list

Parameters

key (int) – positive integer for position in _source list to insert to

Returns

newly inserted SubInput

Return type

SubInput

itersubinputs()[source]

Iterate over the SubInputs in this Input.

Returns

iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
remove(value)[source]

Remove a SubInput from the SubInputs list based on the connected Link.

Parameters

value (SubInput, <fastr.planning.inputoutput.SubInput>`) – the SubInput or Link to removed from this Input

source

The mapping of SubInputs that are connected and have more than 0 elements.

class fastr.planning.inputoutput.MacroInput(node, description)[source]

Bases: fastr.planning.inputoutput.Input

__abstractmethods__ = frozenset({})
__module__ = 'fastr.planning.inputoutput'
input_group

The id of the InputGroup this Input belongs to.

class fastr.planning.inputoutput.MacroOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

__abstractmethods__ = frozenset({})
__module__ = 'fastr.planning.inputoutput'
dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.NamedSubInput(parent)[source]

Bases: fastr.planning.inputoutput.Input

A named subinput for cases where the value of an input is mapping.

__abstractmethods__ = frozenset({})
__getitem__(key)[source]

Retrieve an item (a SubInput) from this NamedSubInput.

Parameters

key (int) – the key of the requested item

Return type

SubInput

Returns

The SubInput corresponding with the key will be returned.

Raises
__init__(parent)[source]

Instantiate an input.

Parameters
  • node (NodeRun) – the parent node of this input.

  • description (ParameterDescription) – the ParameterDescription of the input.

Returns

the created Input

__module__ = 'fastr.planning.inputoutput'
__str__()[source]

Get a string version for the NamedSubInput

Returns

the string version

Return type

str

constant_id

The id for a constant node that is attached to this input.

Return type

str

fullid

The full defining ID for the SubInput

item_index
class fastr.planning.inputoutput.Output(node, description)[source]

Bases: fastr.planning.inputoutput.BaseOutput

Class representing an output of a node. It holds the output values of the tool ran. Output fields can be connected to inputs of other nodes.

__abstractmethods__ = frozenset({})
__eq__(other)[source]

Compare two Output instances with each other. This function ignores the parent node, listeners and update status, but tests rest of the dict for equality.

Parameters

other (fastr.planning.inputoutput.Output) – the other instances to compare to

Returns

True if equal, False otherwise

Return type

bool

__getitem__(key)[source]

Retrieve an item from this Output. The returned value depends on what type of key used:

  • Retrieving data using index tuple: [index_tuple]

  • Retrieving data sample_id str: [SampleId]

  • Retrieving a list of data using SampleId list: [sample_id1, …, sample_idN]

  • Retrieving a SubOutput using an int or slice: [n] or [n:m]

Parameters

key (Union[int, slice]) – the key of the requested suboutput, can be a numberor slice

Return type

SubOutput

Returns

the SubOutput for the corresponding index

Raises

FastrTypeError – if key is not of a valid type

__getstate__()[source]

Retrieve the state of the Output

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(node, description)[source]

Instantiate an Output

Parameters
  • node – the parent node the output belongs to.

  • description – the ParameterDescription describing the output.

Returns

created Output

Raises
__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the Output by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

__str__()[source]

Get a string version for the Output

Returns

the string version

Return type

str

cardinality()[source]

Cardinality of this Output, may depend on the inputs of the parent Node.

Returns

the cardinality

Return type

int, sympy.Symbol, or None

Raises
datatype

The datatype of this Output

dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

fullid

The full defining ID for the Output

listeners

The list of Links connected to this Output.

preferred_types

The list of preferred DataTypes for this Output.

resulting_datatype

The DataType that will the results of this Output will have.

valid

Check if the output is valid, i.e. has a valid cardinality

class fastr.planning.inputoutput.SourceOutput(node, description)[source]

Bases: fastr.planning.inputoutput.Output

Output for a SourceNodeRun, this type of Output determines the cardinality in a different way than a normal NodeRun.

__abstractmethods__ = frozenset({})
__init__(node, description)[source]

Instantiate a FlowOutput

Parameters
  • node – the parent node the output belongs to.

  • description – the ParameterDescription describing the output.

Returns

created FlowOutput

Raises
__module__ = 'fastr.planning.inputoutput'
cardinality()[source]

Cardinality of this SourceOutput, may depend on the inputs of the parent NodeRun.

Parameters

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns

the cardinality

Return type

int, sympy.Symbol, or None

linearized

A linearized version of the sample data, this is lazily cached linearized version of the underlying SampleCollection.

class fastr.planning.inputoutput.SubInput(input_)[source]

Bases: fastr.planning.inputoutput.BaseInput

This class is used by Input to allow for multiple links to an Input. The SubInput class can hold only a single Link to a (Sub)Output, but behaves very similar to an Input otherwise.

__abstractmethods__ = frozenset({})
__eq__(other)[source]

Compare two SubInput instances with each other. This function ignores the parent, node, source and update status, but tests rest of the dict for equality.

Parameters

other (SubInput) – the other instances to compare to

Returns

True if equal, False otherwise

__getitem__(key)[source]

Retrieve an item from this SubInput.

Parameters

key (int) – the index of the requested item

Returns

the corresponding SubInput

Return type

SubInput

Raises

FastrTypeError – if key is not of a valid type

Note

As a SubInput has only one SubInput, only requesting int key 0 or -1 is allowed, and it will return self

__getstate__()[source]

Retrieve the state of the SubInput

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(input_)[source]

Instantiate an SubInput.

Parameters

input (Input) – the parent of this SubInput.

Returns

the created SubInput

__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the SubInput by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

__str__()[source]

Get a string version for the SubInput

Returns

the string version

Return type

str

cardinality(key=None, job_data=None)[source]

Get the cardinality for this SubInput. The cardinality for a SubInputs is defined by the incoming link.

Parameters

key (SampleIndex or SampleId) – key for a specific sample, can be sample index or id

Returns

the cardinality

Return type

int, sympy.Symbol, or None

constant_id

The id for a constant node that is attached to this input.

Return type

str

description

The description object of this input/output

dimensions

List of dimension for this SubInput

fullid

The full defining ID for the SubInput

get_sourced_nodes()[source]

Get a list of all Nodes connected as sources to this SubInput

Returns

list of all connected Nodes

Return type

list

get_sourced_outputs()[source]

Get a list of all Outputs connected as sources to this SubInput

Returns

list of all connected Outputs

Return type

list

input_group

The id of the InputGroup this SubInputs parent belongs to.

item_index
iteritems()[source]

Iterate over the SampleItems that are in the SubInput.

Returns

iterator yielding SampleItem objects

itersubinputs()[source]

Iterate over SubInputs (for a SubInput it will yield self and stop iterating after that)

Returns

iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput
node

The Node to which this SubInputs parent belongs

remove(value)[source]

Remove a SubInput from parent Input.

Parameters

value (SubInput) – the SubInput to removed from this Input

source

A list with the source Link. The list is to be compatible with Input

source_output

The Output linked to this SubInput

class fastr.planning.inputoutput.SubOutput(output, index)[source]

Bases: fastr.planning.inputoutput.Output

The SubOutput is an Output that represents a slice of another Output.

__abstractmethods__ = frozenset({})
__eq__(other)[source]

Compare two SubOutput instances with each other. This function ignores the parent, node and update status, but tests rest of the dict for equality. equality

Parameters

other (SubOutput) – the other instances to compare to

Returns

True if equal, False otherwise

Return type

bool

__getstate__()[source]

Retrieve the state of the SubOutput

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(output, index)[source]

Instantiate a SubOutput

Parameters
  • output – the parent output the suboutput slices.

  • index (int or slice) – the way to slice the parent output

Returns

created SubOutput

Raises
__len__()[source]

Return the length of the Output.

Note

In a SubOutput this is always 1.

__module__ = 'fastr.planning.inputoutput'
__setstate__(state)[source]

Set the state of the SubOutput by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

__str__()[source]

Get a string version for the SubOutput

Returns

the string version

Return type

str

cardinality()[source]

Cardinality of this SubOutput depends on the parent Output and self.index

Parameters

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns

the cardinality

Return type

int, sympy.Symbol, or None

Raises
datatype

The datatype of this SubOutput

fullid

The full defining ID for the SubOutput

indexrep

Simple representation of the index.

listeners

The list of Links connected to this Output.

node

The NodeRun to which this SubOutput belongs

preferred_types

The list of preferred DataTypes for this SubOutput.

resulting_datatype

The DataType that will the results of this SubOutput will have.

samples

The SampleCollection for this SubOutput

network Module

Network module containing Network facilitators and analysers.

class fastr.planning.network.Network(id_='unnamed_network', version=None, filename=None)[source]

Bases: fastr.abc.serializable.Serializable

The NetworkRun contains the entire Run state for a Network execution. It has a working copy of the network, but also includes all temporary data required for the execution. These objects are meant to be single use.

NETWORK_DUMP_FILE_NAME = '__fastr_network__.json'
SINK_DUMP_FILE_NAME = '__sink_data__.json'
SOURCE_DUMP_FILE_NAME = '__source_data__.pickle.gz'
__dataschemafile__ = 'Network.schema.json'
__eq__(other)[source]

Compare two Networks and see if they are equal.

Parameters

other (Network) –

Returns

flag indicating that the Networks are the same

Return type

bool

__getitem__(item)[source]

Get an item by its fullid. The fullid can point to a link, node, input, output or even subinput/suboutput.

Parameters

item (str,unicode) – fullid of the item to retrieve

Returns

the requested item

__getstate__()[source]

Retrieve the state of the Network

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(id_='unnamed_network', version=None, filename=None)[source]

Create a new, empty Network

Parameters

name (str) – name of the Network

Returns

newly created Network

Raises

OSError – if the tmp mount in the config is not a writable directory

__module__ = 'fastr.planning.network'
__ne__(other)[source]

Tests for non-equality, this is the negated version __eq__

__repr__()[source]

Return repr(self).

__setstate__(state)[source]

Set the state of the Network by the given state. This completely overwrites the old state!

Parameters

state (dict) – The state to populate the object with

Returns

None

Add a Link to the Network. Make sure the link is in the link list and the link parent is set to this Network

Parameters

link (Link) – link to add

Raises
add_node(node)[source]

Add a Node to the Network. Make sure the node is in the node list and the node parent is set to this Network

Parameters

node (Node) – node to add

Raises

FastrTypeError – if node is incorrectly typed

add_stepid(stepid, node)[source]

Add a Node to a specific step id

Parameters
  • stepid (str) – the stepid that the node will be added to

  • node (Node) – the node to add to the stepid

check_id(id_)[source]

Check if an id for an object is valid and unused in the Network. The method will always returns True if it does not raise an exception.

Parameters

id (str) – the id to check

Returns

True

Raises
create_constant(datatype, data, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a ConstantNode in this Network. The Node will be automatically added to the Network.

Parameters
  • datatype (BaseDataType) – The DataType of the constant node

  • data (datatype or list of datatype) – The data to hold in the constant node

  • id (str) – The id of the constant node to be created

  • stepid (str) – The stepid to add the created constant node to

  • resources – The resources required to run this node

  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns

the newly created constant node

Return type

ConstantNode

Create a link between two Nodes and add it to the current Network.

Parameters
  • source (BaseOutput) – the output that is the source of the link

  • target (BaseInput) – the input that is the target of the link

  • id (str) – the id of the link

Returns

the created link

Type

Link

create_macro(network, resources=None, id_=None)[source]
create_node(tool, tool_version, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a Node in this Network. The Node will be automatically added to the Network.

Parameters
  • tool (Tool) – The Tool to base the Node on

  • id (str) – The id of the node to be created

  • stepid (str) – The stepid to add the created node to

  • resources – The resources required to run this node

  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns

the newly created node

Return type

Node

create_reference(source_data, output_directory)[source]
create_sink(datatype, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a SinkNode in this Network. The Node will be automatically added to the Network.

Parameters
  • datatype (BaseDataType) – The DataType of the sink node

  • id (str) – The id of the sink node to be created

  • stepid (str) – The stepid to add the created sink node to

  • resources – The resources required to run this node

Returns

the newly created sink node

Return type

SinkNode

create_source(datatype, id_=None, stepid=None, resources=None, nodegroup=None)[source]

Create a SourceNode in this Network. The Node will be automatically added to the Network.

Parameters
  • datatype (BaseDataType) – The DataType of the source source_node

  • id (str) – The id of the source source_node to be created

  • stepid (str) – The stepid to add the created source source_node to

  • resources – The resources required to run this node

  • nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns

the newly created source source_node

Return type

SourceNode

draw(name=None, draw_dimensions=True, hide_unconnected=True, context=None, graph=None, expand_macro=False, font_size=14)[source]
draw_network(name='network_layout', img_format='svg', draw_dimension=True, hide_unconnected=True, expand_macro=False, font_size=14)[source]

Output a dot file and try to convert it to an image file.

Parameters

img_format (str) – extension of the image format to convert to

Returns

path of the image created or None if failed

Return type

str or None

execute(sourcedata, sinkdata, blocking=True, **kwargs)[source]
fullid

The fullid of the Network, within the network scope

global_id

The global id of the Network, this is different for networks used in macronodes, as they still have parents.

id

The id of the Network. This is a read only property.

is_valid()[source]
namespace = None

The namespace this network lives in, this will be set by the NetworkManager on load

nodegroups

Give an overview of the nodegroups in the network

ns_id

The namespace and id of the Tool

remove(value)[source]

Remove an item from the Network.

Parameters

value (Node or Link) – the item to remove

classmethod test(reference_data_dir, network=None, source_data=None, force_remove_temp=False, tmp_results_dir=None)[source]

Execute the network with the source data specified and test the results against the refence data. This effectively tests the network execution.

Parameters
  • reference_data_dir (str) – The path or vfs url of reference data to compare with

  • source_data (dict) – The source data to use

  • force_remove_temp – Make sure the tmp results directory is cleaned at end of test

  • tmp_results_dir – Path to results directory

node Module

A module to maintain a network node.

Exported classes:

Node – A class encapsulating a tool. ConstantNode – A node encapsulating an Output to set scalar values. SourceNode – A class providing a handle to a file.

class fastr.planning.node.AdvancedFlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.FlowNode

__abstractmethods__ = frozenset({})
__module__ = 'fastr.planning.node'
class fastr.planning.node.BaseNode[source]

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

NODE_TYPES = {'AdvancedFlowNode': <class 'fastr.planning.node.AdvancedFlowNode'>, 'ConstantNode': <class 'fastr.planning.node.ConstantNode'>, 'FlowNode': <class 'fastr.planning.node.FlowNode'>, 'MacroNode': <class 'fastr.planning.node.MacroNode'>, 'Node': <class 'fastr.planning.node.Node'>, 'SinkNode': <class 'fastr.planning.node.SinkNode'>, 'SourceNode': <class 'fastr.planning.node.SourceNode'>}
__abstractmethods__ = frozenset({'_update', 'dimensions'})
classmethod __init_subclass__(**kwargs)[source]

Register nodes in class for easly location

__module__ = 'fastr.planning.node'
class fastr.planning.node.ConstantNode(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.SourceNode

Class encapsulating one output for which a value can be set. For example used to set a scalar value to the input of a node.

__abstractmethods__ = frozenset({})
__dataschemafile__ = 'ConstantNode.schema.json'
__getstate__()[source]

Retrieve the state of the ConstantNode

Returns

the state of the object

Rtype dict

__init__(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the ConstantNode.

Parameters
  • datatype – The datatype of the output.

  • data – the prefilled data to use.

  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a constant using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')

or alternatively create a constant node by assigning data to an item in an InputDict:

>>> node_a.inputs['in'] = ['some', 'data']

which automatically creates and links a ConstantNode to the specified Input

__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the ConstantNode by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

data

The data stored in this constant node

draw(context, graph, color=None)[source]
print_value
set_data(data=None, ids=None)[source]

Set the data of this constant node in the correct way. This is mainly for compatibility with the parent class SourceNode

Parameters
  • data (dict or list of urls) – the data to use

  • ids – if data is a list, a list of accompanying ids

class fastr.planning.node.FlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

A Flow Node is a special subclass of Nodes in which the amount of samples can vary per Output. This allows non-default data flows.

__abstractmethods__ = frozenset({})
__init__(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiate a flow node.

Parameters
  • tool (Tool) – The tool to base the node on

  • id (str) – the id of the node

  • parent (Network) – the parent network of the node

Returns

the newly created FlowNode

__module__ = 'fastr.planning.node'
blocking

A FlowNode is (for the moment) always considered blocking.

Returns

True

dimensions

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

outputsize

Size of the outputs in this Node

class fastr.planning.node.InputDict[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Implements helper functions for the easy linking syntax.

__module__ = 'fastr.planning.node'
__setitem__(key, value)[source]

Set an item in the input dictionary. The behaviour depends on the type of the value. For a BaseInput, the input will simply be added to the list of inputs. For a BaseOutput, a link between the output and input will be created.

Parameters
  • key (str) – id of the input to assign/link

  • value (BaseInput or BaseOutput) – either the input to add or the output to link

  • dict_setitem – the setitem function to use for the underlying OrderedDict insert

class fastr.planning.node.MacroNode(value, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

MacroNode encapsulates an entire network in a single node.

__abstractmethods__ = frozenset({})
__eq__(other)[source]

Compare two MacroNode instances with each other. This function ignores the parent and update status, but tests rest of the dict for equality. equality

Parameters

other (MacroNode) – the other instances to compare to

Returns

True if equal, False otherwise

__getstate__()[source]

Retrieve the state of the MacroNode

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(value, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]
Parameters

network (fastr.planning.network.Network) – network to create macronode for

__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

draw(context, graph, color=None)[source]
get_output_info(output)[source]

This functions maps the output dimensions based on the input dimensions of the macro. This is cached for speed as this can become rather costly otherwise

Parameters

output – output to get info for

Returns

tuple of Dimensions

network
class fastr.planning.node.Node(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.BaseNode

The class encapsulating a node in the network. The node is responsible for setting and checking inputs and outputs based on the description provided by a tool instance.

__abstractmethods__ = frozenset({})
__dataschemafile__ = 'Node.schema.json'
__eq__(other)[source]

Check two Node instances for equality.

Parameters

other (fastr.planning.node.Node) – the other instances to compare to

Returns

True if equal, False otherwise

__getstate__()[source]

Retrieve the state of the Node

Returns

the state of the object

Rtype dict

__hash__ = None
__init__(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiate a node.

Parameters
  • tool (Tool) – The tool to base the node on

  • id (str) – the id of the node

  • node_class (str) – The class of the NodeRun to create (e.g. SourceNodeRun, NodeRun)

  • parent (Network) – the parent network of the node

  • cores (int) – number of cores required for executing this Node

  • memory (str) – amount of memory required in the form d+[mMgG] where M is for megabyte and G for gigabyte

  • walltime (str) – amount of time required in second or in the form HOURS:MINUTES:SECOND

Returns

the newly created Node

__module__ = 'fastr.planning.node'
__ne__(other)[source]

Check two Node instances for inequality. This is the inverse of __eq__

Parameters

other (fastr.planning.node.Node) – the other instances to compare to

Returns

True if unequal, False otherwise

__repr__()[source]

Get a string representation for the Node

Returns

the string representation

Return type

str

__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

__str__()[source]

Get a string version for the Node

Returns

the string version

Return type

str

blocking

Indicate that the results of this Node cannot be determined without first executing the Node, causing a blockage in the creation of jobs. A blocking Nodes causes the Chunk borders.

classmethod createobj(state, network=None)[source]

Create object function for generic objects

Parameters
  • cls – The class to create

  • state – The state to use to create the Link

  • network – the parent Network

Returns

newly created Link

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns

dimensions

Return type

tuple

dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

draw(context, graph, color=None)[source]
draw_id(context)[source]
find_source_index(target_index, target, source)[source]
fullid

The full defining ID for the Node inside the network

get_sourced_nodes()[source]

A list of all Nodes connected as sources to this Node

Returns

list of all nodes that are connected to an input of this node

global_id

The global defining ID for the Node from the main network (goes out of macro nodes to root network)

id

The id of the Node

input_groups
A list of input groups for this Node. An input group is InputGroup

object filled according to the Node

inputs = None

A list of inputs of this Node

listeners

All the listeners requesting output of this node, this means the listeners of all Outputs and SubOutputs

merge_dimensions
name

Name of the Tool the Node was based on. In case a Toolless Node was used the class name is given.

nodegroup
outputs = None

A list of outputs of this Node

outputsize

The size of output of this SourceNode

parent

The parent is the Network this Node is part of

status
tool
update_input_groups()[source]

Update all input groups in this node

class fastr.planning.node.OutputDict[source]

Bases: collections.OrderedDict

The container containing the Inputs of Node. Only checks if the inserted values are actually outputs.

__module__ = 'fastr.planning.node'
__setitem__(key, value)[source]

Set an output.

Parameters
  • key (str) – the of the item to set

  • value (BaseOutput) – the output to set

  • dict_setitem – the setitem function to use for the underlying OrderedDict insert

class fastr.planning.node.SinkNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.Node

Class which handles where the output goes. This can be any kind of file, e.g. image files, textfiles, config files, etc.

__abstractmethods__ = frozenset({})
__dataschemafile__ = 'SinkNode.schema.json'
__getstate__()[source]

Retrieve the state of the Node

Returns

the state of the object

Rtype dict

__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters
  • datatype – The datatype of the output.

  • id – the id of the node to create

Returns

newly created sink node

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> sink = network.create_sink(datatype=types['ITKImageFile'], id_='SinkN')
__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the Node by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

datatype

The datatype of the data this sink can store.

draw(context, graph, color=None)[source]
input

The default input of the sink Node

class fastr.planning.node.SourceNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Bases: fastr.planning.node.FlowNode

Class providing a connection to data resources. This can be any kind of file, stream, database, etc from which data can be received.

__abstractmethods__ = frozenset({})
__dataschemafile__ = 'SourceNode.schema.json'
__getstate__()[source]

Retrieve the state of the SourceNode

Returns

the state of the object

Rtype dict

__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)[source]

Instantiation of the SourceNode.

Parameters
  • datatype – The (id of) the datatype of the output.

  • id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a source using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')
__module__ = 'fastr.planning.node'
__setstate__(state)[source]

Set the state of the SourceNode by the given state.

Parameters

state (dict) – The state to populate the object with

Returns

None

datatype

The datatype of the data this source supplies.

dimensions

The dimensions in the SourceNode output. These will be reflected in the SampleIdLists.

draw(context, graph, color=None)[source]
nodegroup
output

Shorthand for self.outputs['output']

set_data(data, ids=None)[source]

Set the data of this source node.

Parameters
  • data (dict, OrderedDict or list of urls) – the data to use

  • ids – if data is a list, a list of accompanying ids

sourcegroup
valid

This does nothing. It only overloads the valid method of Node(). The original is intended to check if the inputs are connected to some output. Since this class does not implement inputs, it is skipped.