planning Package

planning Package

inputgroup Module

class fastr.planning.inputgroup.InputGroup(*args, **kwargs)

Bases: collections.OrderedDict, fastr.core.dimension.HasDimensions

A class representing a group of inputs. Input groups allow the

__abstractmethods__ = frozenset()

__getitem__(key)

x.__getitem__(y) <==> x[y]

__init__(*args, **kwargs)

Create a new InputGroup representation

Parameters:

• parent (NodeRun) – the parent node
• id (str) – the id of the input group

Raises:

FastrTypeError – if parent is not a NodeRun

Note

This is a wrapped version of fastr.planning.inputgroup.__init__ which triggers an update of the object after being called

__module__ = 'fastr.planning.inputgroup'

__setitem__(*args, **kwargs)

Assign an input to this input group.

Parameters:

• key (str) – id of the input
• value (Input) – the input to assign

Raises:

FastrTypeError – if value of valid type

Note

This is a wrapped version of fastr.planning.inputgroup.__setitem__ which triggers an update of the object after being called

__updatefunc__()

Update the InputGroup. Triggers when a change is made to the content of the InputGroup. Automatically recalculates the size, primary Input etc.

__updatetriggers__ = ['__init__', '__setitem__', '__delitem__', 'clear', 'pop', 'popitem', 'setdefault', 'update']

dimensions

The dimensions of this InputGroup

empty

Bool indicating that this InputGroup is empty (has no data connected)

find_source_index(target_size, target_dimnames, source_size, source_dimnames, target_index)

fullid

iterinputvalues

Iterate over the item in this InputGroup

Returns:iterator yielding SampleItems

parent

The parent node of this InputGroup

primary

The primary Input in this InputGroup. The primary Input is the Input that defines the size of this InputGroup. In case of ties it will be the first in the tool definition.

classmethod solve_broadcast(target_size, target_dimnames, source_size, source_dimnames, target_index, nodegroups=None)

inputgroupcombiner Module

class fastr.planning.inputgroupcombiner.BaseInputGroupCombiner(parent)

Bases: fastr.core.dimension.HasDimensions

An object that takes the different input groups and combines them in the correct way.

__abstractmethods__ = frozenset({'iter_input_groups', 'unmerge', 'merge'})

__init__(parent)

Initialize self. See help(type(self)) for accurate signature.

__iter__()

__module__ = 'fastr.planning.inputgroupcombiner'

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions Return type:tuple

fullid

The full id of the InputGroupCombiner

input_groups

iter_input_groups()

Iterate over all the merged samples :return:

merge(list_of_items)

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine Returns:combined list

merge_failed_annotations(list_of_failed_annotations)

merge_payloads(sample_payloads)

merge_sample_data(list_of_sample_data)

merge_sample_id(list_of_sample_ids)

merge_sample_index(list_of_sample_indexes)

merge_sample_jobs(list_of_sample_jobs)

unmerge(item)

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())

Parameters:item (list) – the item to unmerge Returns:items Return type:OrderedDict

update()

class fastr.planning.inputgroupcombiner.DefaultInputGroupCombiner(parent)

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The default input group combiner combines the input group in a cross product version, taking each combinations of samples between the input groups. So if there are two input groups with one with size N and the other with size M x P the result would be N x M x P samples, with all possible combinations of the samples in each input group.

__abstractmethods__ = frozenset()

__module__ = 'fastr.planning.inputgroupcombiner'

iter_input_groups()

Iterate over all the merged samples :return:

merge(list_of_items)

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine Returns:combined list

unmerge(item)

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())

Parameters:item (list) – the item to unmerge Returns:items Return type:OrderedDict

class fastr.planning.inputgroupcombiner.MergingInputGroupCombiner(input_groups, merge_dimension)

Bases: fastr.planning.inputgroupcombiner.BaseInputGroupCombiner

The merging input group combiner takes a similar approach as the default combiner but merges dimensions that are the same. If input group A has N(3) x M(2) samples and B has M(2) x P(4) it wil not result in N(3) x M(2) x M(2) x P(4), but merge the dimensions M leading to N(3) x M(2) x P(4) in resulting size.

__abstractmethods__ = frozenset()

__init__(input_groups, merge_dimension)

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.planning.inputgroupcombiner'

iter_input_groups()

Iterate over all the merged samples :return:

merge(list_of_items)

Given a list of items for each input group, it returns the combined list of items.

Parameters:list_of_items (list) – items to combine Returns:combined list

unmerge(item)

Given a item it will recreate the seperate items, basically this is the inverse operation of merge. However, this create an OrderedDict so that specific input groups can be easily retrieved. To get a round trip, the values of the OrderedDict should be taken:

>>> odict_of_items = combiner.unmerge(item)
>>> item = combiner.merge(odict_of_items.values())

Parameters:item (list) – the item to unmerge Returns:items Return type:OrderedDict

update()

inputoutput Module

Classes for arranging the input and output for nodes.

Exported classes:

Input – An input for a node (holding datatype). Output – The output of a node (holding datatype and value). ConstantOutput – The output of a node (holding datatype and value).

Warning

Don’t mess with the Link, Input and Output internals from other places. There will be a huge chances of breaking the network functionality!

class fastr.planning.inputoutput.AdvancedFlowOutput(node, description)

Bases: fastr.planning.inputoutput.Output

Output for nodes that have an advanced flow. This means that the output sample id and index is not the same as the input sample id and index. The AdvancedFlowOutput has one extra dimensions that is created by the Node.

__abstractmethods__ = frozenset()

__module__ = 'fastr.planning.inputoutput'

dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.BaseInput(node, description)

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all inputs.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'itersubinputs', 'fullid'})

__init__(node, description)

Instantiate a BaseInput

Parameters:

• node – the parent node the input/output belongs to.
• description – the ParameterDescription describing the input/output.

Returns:

the created BaseInput

Raises:

• FastrTypeError – if description is not of class ParameterDescription
• FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the types

__lshift__(other)

__module__ = 'fastr.planning.inputoutput'

__rrshift__(other)

check_cardinality(key=None, planning=False)

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters:key – sample_index (tuple of int) or SampleId for desired sample Returns:flag indicating that the cardinality is correct Return type:bool Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

create_link_from(value)

description_type

alias of fastr.core.interface.InputSpec

itersubinputs()

Iterator over the SubInputs

Returns:iterator

example:

>>> for subinput in input_a.itersubinputs():
        print subinput

class fastr.planning.inputoutput.BaseInputOutput(node, description)

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

Base class for Input and Output classes. It mainly implements the properties to access the data from the underlying ParameterDescription.

__abstractmethods__ = frozenset({'dimensions', '_update', 'fullid'})

__getstate__()

Retrieve the state of the BaseInputOutput

Returns:the state of the object Rtype dict:

__init__(node, description)

Instantiate a BaseInputOutput

Parameters:

• node – the parent node the input/output belongs to.
• description – the ParameterDescription describing the input/output.

Returns:

created BaseInputOutput

Raises:

• FastrTypeError – if description is not of class ParameterDescription
• FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the types

__iter__()

This function is blocked to avoid support for iteration using a lecacy __getitem__ method.

Returns:None Raises:FastrNotImplementedError – always

__module__ = 'fastr.planning.inputoutput'

__ne__(other)

Check two Node instances for inequality. This is the inverse of __eq__

Parameters:other (BaseInputOutput) – the other instances to compare to Returns:True if unequal, False otherwise

__repr__()

Get a string representation for the Input/Output

Returns:the string representation Return type:str

__setstate__(state)

Set the state of the BaseInputOutput by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

cardinality(key=None, job_data=None)

Determine the cardinality of this Input/Output. Optionally a key can be given to determine for a sample.

Parameters:key – key for a specific sample Returns:the cardinality Return type:int, sympy.Symbol, or None

check_cardinality(key=None)

Check if the actual cardinality matches the cardinality specified in the ParameterDescription. Optionally you can use a key to test for a specific sample.

Parameters:key – sample_index (tuple of int) or SampleId for desired sample Returns:flag indicating that the cardinality is correct Return type:bool Raises:FastrCardinalityError – if the Input/Output has an incorrect cardinality description.

datatype

The datatype of this Input/Output

description

The description object of this input/output

description_type = None

fullid

The fullid of the Input/Output, the fullid should be unnique and makes the object retrievable by the network.

id

Id of the Input/Output

node

The NodeRun to which this Input/Output belongs

required

Flag indicating that the Input/Output is required

class fastr.planning.inputoutput.BaseOutput(node, description)

Bases: fastr.planning.inputoutput.BaseInputOutput

Base class for all outputs.

__abstractmethods__ = frozenset({'_update', 'dimensions', 'fullid'})

__init__(node, description)

Instantiate a BaseOutput

Parameters:

• node – the parent node the output belongs to.
• description – the ParameterDescription describing the output.

Returns:

created BaseOutput

Raises:

• FastrTypeError – if description is not of class ParameterDescription
• FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the types

__module__ = 'fastr.planning.inputoutput'

automatic

Flag indicating that the Output is generated automatically without being specified on the command line

blocking

Flag indicating that this Output will cause blocking in the execution

description_type

alias of fastr.core.interface.OutputSpec

class fastr.planning.inputoutput.Input(node, description)

Bases: fastr.planning.inputoutput.BaseInput

Class representing an input of a node. Such an input will be connected to the output of another node or the output of an constant node to provide the input value.

__abstractmethods__ = frozenset()

__eq__(other)

Compare two Input instances with each other. This function ignores the parent node and update status, but tests rest of the dict for equality.

Parameters:other (Input) – the other instances to compare to Returns:True if equal, False otherwise Return type:bool

__getitem__(key)

Retrieve an item from this Input.

Parameters:

key (Union[int, str]) – the key of the requested item

Return type:

SubInput

Returns:

The SubInput corresponding with the key will be returned.

Raises:

• FastrTypeError – if key is not of a valid type
• FastrKeyError – if the key is not found

__getstate__()

Retrieve the state of the Input

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(node, description)

Instantiate an input.

Parameters:

• node (NodeRun) – the parent node of this input.
• description (ParameterDescription) – the ParameterDescription of the input.

Returns:

the created Input

__module__ = 'fastr.planning.inputoutput'

__setitem__(key, value)

Create a link between a SubInput of this Inputs and an Output/Constant

Parameters:

• key (int, str) – the key of the SubInput
• value (BaseOutput, list, tuple, dict, OrderedDict) – the target to link, can be an output or a value to create a constant for

Raises:

FastrTypeError – if key is not of a valid type

__setstate__(state)

Set the state of the Input by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

__str__()

Get a string version for the Input

Returns:the string version Return type:str

append(value)

When you want to append a link to an Input, you can use the append property. This will automatically create a new SubInput to link to.

example:

>>> link = node2['input'].append(node1['output'])

will create a new SubInput in node2[‘input’] and link to that.

cardinality(key=None, job_data=None)

Cardinality for an Input is the sum the cardinalities of the SubInputs, unless defined otherwise.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id Returns:the cardinality Return type:int, sympy.Symbol, or None

clear()

datatype

The datatype of this Input

dimensions

The list names of the dimensions in this Input. This will be a list of str.

fullid

The full defining ID for the Input

Return type:str

get_sourced_nodes()

Get a list of all Nodes connected as sources to this Input

Returns:list of all connected Nodes Return type:list

get_sourced_outputs()

Get a list of all Outputs connected as sources to this Input

Returns:tuple of all connected Outputs Return type:tuple

index(value)

Find index of a SubInput

Parameters:value (SubInput) – the SubInput to find the index of Returns:key Return type:int, str

input_group

The id of the InputGroup this Input belongs to.

Return type:str

insert(index)

Insert a new SubInput at index in the sources list

Parameters:key (int) – positive integer for position in _source list to insert to Returns:newly inserted SubInput Return type:SubInput

itersubinputs()

Iterate over the SubInputs in this Input.

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput

remove(value)

Remove a SubInput from the SubInputs list based on the connected Link.

Parameters:value (SubInput, <fastr.planning.inputoutput.SubInput>`) – the SubInput or SubLink to removed from this Input

source

The mapping of SubInputs that are connected and have more than 0 elements.

class fastr.planning.inputoutput.MacroInput(node, description)

Bases: fastr.planning.inputoutput.Input

__abstractmethods__ = frozenset()

__module__ = 'fastr.planning.inputoutput'

input_group

The id of the InputGroup this Input belongs to.

class fastr.planning.inputoutput.MacroOutput(node, description)

Bases: fastr.planning.inputoutput.Output

__abstractmethods__ = frozenset()

__module__ = 'fastr.planning.inputoutput'

dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

class fastr.planning.inputoutput.Output(node, description)

Bases: fastr.planning.inputoutput.BaseOutput

Class representing an output of a node. It holds the output values of the tool ran. Output fields can be connected to inputs of other nodes.

__abstractmethods__ = frozenset()

__eq__(other)

Compare two Output instances with each other. This function ignores the parent node, listeners and update status, but tests rest of the dict for equality.

Parameters:other (Output) – the other instances to compare to Returns:True if equal, False otherwise Return type:bool

__getitem__(key)

Retrieve an item from this Output. The returned value depends on what type of key used:

• Retrieving data using index tuple: [index_tuple]
• Retrieving data sample_id str: [SampleId]
• Retrieving a list of data using SampleId list: [sample_id1, …, sample_idN]
• Retrieving a SubOutput using an int or slice: [n] or [n:m]

Parameters:key (Union[int, slice]) – the key of the requested suboutput, can be a numberor slice Return type:SubOutput Returns:the SubOutput for the corresponding index Raises:FastrTypeError – if key is not of a valid type

__getstate__()

Retrieve the state of the Output

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(node, description)

Instantiate an Output

Parameters:

• node – the parent node the output belongs to.
• description – the ParameterDescription describing the output.

Returns:

created Output

Raises:

• FastrTypeError – if description is not of class ParameterDescription
• FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the types

__module__ = 'fastr.planning.inputoutput'

__setstate__(state)

Set the state of the Output by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

__str__()

Get a string version for the Output

Returns:the string version Return type:str

cardinality()

Cardinality of this Output, may depend on the inputs of the parent Node.

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:

• FastrCardinalityError – if cardinality references an invalid Input
• FastrTypeError – if the referenced cardinality values type cannot be case to int
• FastrValueError – if the referenced cardinality value cannot be case to int

datatype

The datatype of this Output

dimensions

The list of the dimensions in this Output. This will be a tuple of Dimension.

fullid

The full defining ID for the Output

listeners

The list of Links connected to this Output.

preferred_types

The list of preferred DataTypes for this Output.

resulting_datatype

The DataType that will the results of this Output will have.

valid

Check if the output is valid, i.e. has a valid cardinality

class fastr.planning.inputoutput.SourceOutput(node, description)

Bases: fastr.planning.inputoutput.Output

Output for a SourceNodeRun, this type of Output determines the cardinality in a different way than a normal NodeRun.

__abstractmethods__ = frozenset()

__init__(node, description)

Instantiate a FlowOutput

Parameters:

• node – the parent node the output belongs to.
• description – the ParameterDescription describing the output.

Returns:

created FlowOutput

Raises:

• FastrTypeError – if description is not of class ParameterDescription
• FastrDataTypeNotAvailableError – if the DataType requested cannot be found in the types

__module__ = 'fastr.planning.inputoutput'

cardinality()

Cardinality of this SourceOutput, may depend on the inputs of the parent NodeRun.

Parameters:key (tuple of int or SampleId) – key for a specific sample, can be sample index or id Returns:the cardinality Return type:int, sympy.Symbol, or None

linearized

A linearized version of the sample data, this is lazily cached linearized version of the underlying SampleCollection.

class fastr.planning.inputoutput.SubInput(input_)

Bases: fastr.planning.inputoutput.BaseInput

This class is used by Input to allow for multiple links to an Input. The SubInput class can hold only a single Link to a (Sub)Output, but behaves very similar to an Input otherwise.

__abstractmethods__ = frozenset()

__eq__(other)

Compare two SubInput instances with each other. This function ignores the parent, node, source and update status, but tests rest of the dict for equality.

Parameters:other (SubInput) – the other instances to compare to Returns:True if equal, False otherwise

__getitem__(key)

Retrieve an item from this SubInput.

Parameters:key (int) – the index of the requested item Returns:the corresponding SubInput Return type:SubInput Raises:FastrTypeError – if key is not of a valid type

Note

As a SubInput has only one SubInput, only requesting int key 0 or -1 is allowed, and it will return self

__getstate__()

Retrieve the state of the SubInput

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(input_)

Instantiate an SubInput.

Parameters:input (Input) – the parent of this SubInput. Returns:the created SubInput

__module__ = 'fastr.planning.inputoutput'

__setstate__(state)

Set the state of the SubInput by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

__str__()

Get a string version for the SubInput

Returns:the string version Return type:str

cardinality(key=None, job_data=None)

Get the cardinality for this SubInput. The cardinality for a SubInputs is defined by the incoming link.

Parameters:key (SampleIndex or SampleId) – key for a specific sample, can be sample index or id Returns:the cardinality Return type:int, sympy.Symbol, or None

description

The description object of this input/output

dimensions

List of dimension for this SubInput

fullid

The full defining ID for the SubInput

get_sourced_nodes()

Get a list of all Nodes connected as sources to this SubInput

Returns:list of all connected Nodes Return type:list

get_sourced_outputs()

Get a list of all Outputs connected as sources to this SubInput

Returns:list of all connected Outputs Return type:list

input_group

The id of the InputGroup this SubInputs parent belongs to.

iteritems()

Iterate over the SampleItems that are in the SubInput.

Returns:iterator yielding SampleItem objects

itersubinputs()

Iterate over SubInputs (for a SubInput it will yield self and stop iterating after that)

Returns:iterator yielding SubInput

example:

>>> for subinput in input_a.itersubinputs():
        print subinput

node

The Node to which this SubInputs parent belongs

remove(value)

Remove a SubInput from parent Input.

Parameters:value (SubInput) – the SubInput to removed from this Input

source

A list with the source Link. The list is to be compatible with Input

source_output

The Output linked to this SubInput

class fastr.planning.inputoutput.SubOutput(output, index)

Bases: fastr.planning.inputoutput.Output

The SubOutput is an Output that represents a slice of another Output.

__abstractmethods__ = frozenset()

__eq__(other)

Compare two SubOutput instances with each other. This function ignores the parent, node and update status, but tests rest of the dict for equality. equality

Parameters:other (SubOutput) – the other instances to compare to Returns:True if equal, False otherwise Return type:bool

__getstate__()

Retrieve the state of the SubOutput

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(output, index)

Instantiate a SubOutput

Parameters:

• output – the parent output the suboutput slices.
• index (int or slice) – the way to slice the parent output

Returns:

created SubOutput

Raises:

• FastrTypeError – if the output argument is not an instance of Output
• FastrTypeError – if the index argument is not an int or slice

__len__()

Return the length of the Output.

Note

In a SubOutput this is always 1.

__module__ = 'fastr.planning.inputoutput'

__setstate__(state)

Set the state of the SubOutput by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

__str__()

Get a string version for the SubOutput

Returns:the string version Return type:str

cardinality()

Cardinality of this SubOutput depends on the parent Output and self.index

Parameters:

key (tuple of int or SampleId) – key for a specific sample, can be sample index or id

Returns:

the cardinality

Return type:

int, sympy.Symbol, or None

Raises:

• FastrCardinalityError – if cardinality references an invalid Input
• FastrTypeError – if the referenced cardinality values type cannot be case to int
• FastrValueError – if the referenced cardinality value cannot be case to int

datatype

The datatype of this SubOutput

fullid

The full defining ID for the SubOutput

indexrep

Simple representation of the index.

listeners

The list of Links connected to this Output.

node

The NodeRun to which this SubOutput belongs

preferred_types

The list of preferred DataTypes for this SubOutput.

resulting_datatype

The DataType that will the results of this SubOutput will have.

samples

The SampleCollection for this SubOutput

link Module

The link module contain the Link class. This class represents the links in a network. These links lead from an output (BaseOutput) to an input (BaseInput) and indicate the desired data flow. Links are smart objects, in the sense that when you set their start or end point, they register themselves with the Input and Output. They do all the book keeping, so as long as you only set the source and target of the Link, the link should be valid.

Warning

Don’t mess with the Link, Input and Output internals from other places. There will be a huge chances of breaking the network functionality!

class fastr.planning.link.Link(source, target, parent, id_=None, collapse=None, expand=None)

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

Class for linking outputs (BaseOutput) to inputs (BaseInput)

Examples:

>>> import fastr
>>> network = fastr.create_network()
>>> link1 = network.create_link( n1.ouputs['out1'], n2.inputs['in2'] )

link2 = Link()
link2.source = n1.ouputs['out1']
link2.target = n2.inputs['in2']

__abstractmethods__ = frozenset()

__dataschemafile__ = 'Link.schema.json'

__eq__(other)

Test for equality between two Links

Parameters:other (Link) – object to test against Return type:bool Returns:True for equality, False otherwise

__getstate__()

Retrieve the state of the Link

Return type:dict Returns:the state of the object

__hash__ = None

__init__(source, target, parent, id_=None, collapse=None, expand=None)

Create a new Link in a Network.

Parameters:

• source (BaseOutput) – the source output
• target (BaseInput) – the target input
• parent (Network or None) – the parent network
• id (str or None) – the id of the link, if no id_ is given, the id will be in the form of “link_{:d}”
• collapse (int, str, or tuple of int/str) – the dimensions that the link has to collapse on
• expand (bool) – Does this link need to expand the cardinality into a new sample dimension

Returns:

newly created Link

Raises:

• FastrValueError – if parent is not given
• FastrValueError – if the source output is not in the same network as the Link
• FastrValueError – if the target input is not in the same network as the Link

__module__ = 'fastr.planning.link'

__repr__()

Get a string representation for the Link

Return type:str

__setstate__(state)

Set the state of the Link by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None Raises:FastrValueError – if the parent network is not set

cardinality(index=None)

Cardinality for a Link is given by source Output and the collapse/expand settings

Parameters:index (Optional[SampleIndex]) – index for a specific sample (can be only a sample index!) Return type:Union[int, Symbol] Returns:the cardinality Raises:FastrIndexError – if the index length does not match the number of dimension in the data

collapse

The converging dimensions of this link. Collapsing changes some dimensions of sample lists into cardinality, reshaping the data.

Collapse can be set to a tuple or an int/str, in which case it will be automatically wrapped in a tuple. The int will be seen as indices of the dimensions to collapse. The str will be seen as the name of the dimensions over which to collapse.

Raises:FastrTypeError – if assigning a collapse value of a wrong type

collapse_indexes

The converging dimensions of this link as integers. Dimension names are replaces with the corresponding int.

Collapsing changes some dimensions of sample lists into cardinality, reshaping the data

classmethod createobj(state, network=None)

Create object function for Link

Parameters:

• cls – The class to create
• state (dict) – The state to use to create the Link
• network – the parent Network

Returns:

newly created Link

destroy()

The destroy function of a link removes all default references to a link. This means the references in the network, input and output connected to this link. If there is no references in other places in the code, it will destroy the link (reference count dropping to zero).

This function is called when a source for an input is set to another value and the links becomes disconnected. This makes sure there is no dangling links.

dimensions

The dimensions of the data delivered by the link. This can be different from the source dimensions because the link can make data collapse or expand.

draw(context, graph)

expand

Flag indicating that the link will expand the cardininality into a new sample dimension to be created.

fullid

The full defining ID for the Input

parent

The Network to which this Link belongs.

source

The source BaseOutput of the Link. Setting the source will automatically register the Link with the source BaseOutput. Updating source will also make sure the Link is unregistered with the previous source.

Raises:FastrTypeError – if assigning a non BaseOutput

status

target

The target BaseInput of the Link. Setting the target will automatically register the Link with the target BaseInput. Updating target will also make sure the Link is unregistered with the previous target.

Raises:FastrTypeError – if assigning a non BaseInput

network Module

Network module containing Network facilitators and analysers.

class fastr.planning.network.Network(id_='unnamed_network', version=None, filename=None)

Bases: fastr.abc.serializable.Serializable

The NetworkRun contains the entire Run state for a Network execution. It has a working copy of the network, but also includes all temporary data required for the execution. These objects are meant to be single use.

NETWORK_DUMP_FILE_NAME = '__fastr_network__.json'

SINK_DUMP_FILE_NAME = '__sink_data__.json'

SOURCE_DUMP_FILE_NAME = '__source_data__.pickle.gz'

__dataschemafile__ = 'Network.schema.json'

__eq__(other)

Compare two Networks and see if they are equal.

Parameters:other (Network) – Returns:flag indicating that the Networks are the same Return type:bool

__getitem__(item)

Get an item by its fullid. The fullid can point to a link, node, input, output or even subinput/suboutput.

Parameters:item (str,unicode) – fullid of the item to retrieve Returns:the requested item

__getstate__()

Retrieve the state of the Network

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(id_='unnamed_network', version=None, filename=None)

Create a new, empty Network

Parameters:name (str) – name of the Network Returns:newly created Network Raises:OSError – if the tmp mount in the config is not a writable directory

__module__ = 'fastr.planning.network'

__ne__(other)

Tests for non-equality, this is the negated version __eq__

__repr__()

Return repr(self).

__setstate__(state)

Set the state of the Network by the given state. This completely overwrites the old state!

Parameters:state (dict) – The state to populate the object with Returns:None

add_link(link)

Add a Link to the Network. Make sure the link is in the link list and the link parent is set to this Network

Parameters:

link (Link) – link to add

Raises:

• FastrTypeError – if link is incorrectly typed
• FastrNetworkMismatchError – if the link already belongs to another Network

add_node(node)

Add a Node to the Network. Make sure the node is in the node list and the node parent is set to this Network

Parameters:node (Node) – node to add Raises:FastrTypeError – if node is incorrectly typed

add_stepid(stepid, node)

Add a Node to a specific step id

Parameters:

• stepid (str) – the stepid that the node will be added to
• node (Node) – the node to add to the stepid

check_id(id_)

Check if an id for an object is valid and unused in the Network. The method will always returns True if it does not raise an exception.

Parameters:

id (str) – the id to check

Returns:

True

Raises:

• FastrValueError – if the id is not correctly formatted
• FastrValueError – if the id is already in use

create_constant(datatype, data, id_=None, stepid=None, resources=None, nodegroup=None)

Create a ConstantNode in this Network. The Node will be automatically added to the Network.

Parameters:

• datatype (BaseDataType) – The DataType of the constant node
• data (datatype or list of datatype) – The data to hold in the constant node
• id (str) – The id of the constant node to be created
• stepid (str) – The stepid to add the created constant node to
• resources – The resources required to run this node
• nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns:

the newly created constant node

Return type:

ConstantNode

create_link(source, target, id_=None, collapse=None, expand=None)

Create a link between two Nodes and add it to the current Network.

Parameters:

• source (BaseOutput) – the output that is the source of the link
• target (BaseInput) – the input that is the target of the link
• id (str) – the id of the link

Returns:

the created link

Type:

Link

create_macro(network, resources=None, id_=None)

create_node(tool, tool_version, id_=None, stepid=None, resources=None, nodegroup=None)

Create a Node in this Network. The Node will be automatically added to the Network.

Parameters:

• tool (Tool) – The Tool to base the Node on
• id (str) – The id of the node to be created
• stepid (str) – The stepid to add the created node to
• resources – The resources required to run this node
• nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns:

the newly created node

Return type:

Node

create_reference(source_data, output_directory)

create_sink(datatype, id_=None, stepid=None, resources=None, nodegroup=None)

Create a SinkNode in this Network. The Node will be automatically added to the Network.

Parameters:

• datatype (BaseDataType) – The DataType of the sink node
• id (str) – The id of the sink node to be created
• stepid (str) – The stepid to add the created sink node to
• resources – The resources required to run this node

Returns:

the newly created sink node

Return type:

SinkNode

create_source(datatype, id_=None, stepid=None, resources=None, nodegroup=None)

Create a SourceNode in this Network. The Node will be automatically added to the Network.

Parameters:

• datatype (BaseDataType) – The DataType of the source source_node
• id (str) – The id of the source source_node to be created
• stepid (str) – The stepid to add the created source source_node to
• resources – The resources required to run this node
• nodegroup (str) – The group the node belongs to, this can be important for FlowNodes and such, as they will have matching dimension names.

Returns:

the newly created source source_node

Return type:

SourceNode

draw(name=None, draw_dimensions=True, hide_unconnected=True, context=None, graph=None, expand_macro=False)

draw_network(name='network_layout', img_format='svg', draw_dimension=True, hide_unconnected=True, expand_macro=False)

Output a dot file and try to convert it to an image file.

Parameters:img_format (str) – extension of the image format to convert to Returns:path of the image created or None if failed Return type:str or None

execute(sourcedata, sinkdata, blocking=True, **kwargs)

fullid

The fullid of the Network, within the network scope

global_id

The global id of the Network, this is different for networks used in macronodes, as they still have parents.

id

The id of the Network. This is a read only property.

is_valid()

namespace = None

The namespace this network lives in, this will be set by the NetworkManager on load

nodegroups

Give an overview of the nodegroups in the network

ns_id

The namespace and id of the Tool

remove(value)

Remove an item from the Network.

Parameters:value (Node or Link) – the item to remove

classmethod test(reference_data_dir, network=None, source_data=None, force_remove_temp=False, tmp_results_dir=None)

Execute the network with the source data specified and test the results against the refence data. This effectively tests the network execution.

Parameters:

• reference_data_dir (str) – The path or vfs url of reference data to compare with
• source_data (dict) – The source data to use
• force_remove_temp – Make sure the tmp results directory is cleaned at end of test
• tmp_results_dir – Path to results directory

node Module

A module to maintain a network node.

Exported classes:

Node – A class encapsulating a tool. ConstantNode – A node encapsulating an Output to set scalar values. SourceNode – A class providing a handle to a file.

class fastr.planning.node.AdvancedFlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.FlowNode

__abstractmethods__ = frozenset()

__module__ = 'fastr.planning.node'

class fastr.planning.node.BaseNode

Bases: fastr.core.dimension.HasDimensions, fastr.abc.updateable.Updateable, fastr.abc.serializable.Serializable

NODE_TYPES = {'AdvancedFlowNode': <class 'fastr.planning.node.AdvancedFlowNode'>, 'ConstantNode': <class 'fastr.planning.node.ConstantNode'>, 'FlowNode': <class 'fastr.planning.node.FlowNode'>, 'MacroNode': <class 'fastr.planning.node.MacroNode'>, 'Node': <class 'fastr.planning.node.Node'>, 'SinkNode': <class 'fastr.planning.node.SinkNode'>, 'SourceNode': <class 'fastr.planning.node.SourceNode'>}

__abstractmethods__ = frozenset({'_update', 'dimensions'})

classmethod __init_subclass__(**kwargs)

Register nodes in class for easly location

__module__ = 'fastr.planning.node'

class fastr.planning.node.ConstantNode(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.SourceNode

Class encapsulating one output for which a value can be set. For example used to set a scalar value to the input of a node.

__abstractmethods__ = frozenset()

__dataschemafile__ = 'ConstantNode.schema.json'

__getstate__()

Retrieve the state of the ConstantNode

Returns:the state of the object Rtype dict:

__init__(datatype, data, id_=None, parent=None, resource_limits=None, nodegroup=None)

Instantiation of the ConstantNode.

Parameters:

• datatype – The datatype of the output.
• data – the prefilled data to use.
• id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a constant using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')

or alternatively create a constant node by assigning data to an item in an InputDict:

>>> node_a.inputs['in'] = ['some', 'data']

which automatically creates and links a ConstantNode to the specified Input

__module__ = 'fastr.planning.node'

__setstate__(state)

Set the state of the ConstantNode by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

data

The data stored in this constant node

draw(context, graph, color=None)

print_value

set_data(data=None, ids=None)

Set the data of this constant node in the correct way. This is mainly for compatibility with the parent class SourceNode

Parameters:

• data (dict or list of urls) – the data to use
• ids – if data is a list, a list of accompanying ids

class fastr.planning.node.FlowNode(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.Node

A Flow Node is a special subclass of Nodes in which the amount of samples can vary per Output. This allows non-default data flows.

__abstractmethods__ = frozenset()

__init__(tool, id_=None, parent=None, resource_limits=None, nodegroup=None)

Instantiate a flow node.

Parameters:

• tool (Tool) – The tool to base the node on
• id (str) – the id of the node
• parent (Network) – the parent network of the node

Returns:

the newly created FlowNode

__module__ = 'fastr.planning.node'

blocking

A FlowNode is (for the moment) always considered blocking.

Returns:True

dimensions

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

outputsize

Size of the outputs in this Node

class fastr.planning.node.InputDict

Bases: collections.OrderedDict

The container containing the Inputs of Node. Implements helper functions for the easy linking syntax.

__module__ = 'fastr.planning.node'

__setitem__(key, value)

Set an item in the input dictionary. The behaviour depends on the type of the value. For a BaseInput, the input will simply be added to the list of inputs. For a BaseOutput, a link between the output and input will be created.

Parameters:

• key (str) – id of the input to assign/link
• value (BaseInput or BaseOutput) – either the input to add or the output to link
• dict_setitem – the setitem function to use for the underlying OrderedDict insert

class fastr.planning.node.MacroNode(value, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.Node

MacroNode encapsulates an entire network in a single node.

__abstractmethods__ = frozenset()

__eq__(other)

Compare two MacroNode instances with each other. This function ignores the parent and update status, but tests rest of the dict for equality. equality

Parameters:other (MacroNode) – the other instances to compare to Returns:True if equal, False otherwise

__getstate__()

Retrieve the state of the MacroNode

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(value, id_=None, parent=None, resource_limits=None, nodegroup=None)

Parameters:network (Network) – network to create macronode for

__module__ = 'fastr.planning.node'

__setstate__(state)

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

draw(context, graph, color=None)

draw_link_target(context, port_name, input=True)

get_output_info(output)

This functions maps the output dimensions based on the input dimensions of the macro. This is cached for speed as this can become rather costly otherwise

Parameters:output – output to get info for Returns:tuple of Dimensions

network

class fastr.planning.node.Node(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.BaseNode

The class encapsulating a node in the network. The node is responsible for setting and checking inputs and outputs based on the description provided by a tool instance.

__abstractmethods__ = frozenset()

__dataschemafile__ = 'Node.schema.json'

__eq__(other)

Check two Node instances for equality.

Parameters:other (Node) – the other instances to compare to Returns:True if equal, False otherwise

__getstate__()

Retrieve the state of the Node

Returns:the state of the object Rtype dict:

__hash__ = None

__init__(tool, id_=None, node_class=None, parent=None, resource_limits=None, nodegroup=None)

Instantiate a node.

Parameters:

• tool (Tool) – The tool to base the node on
• id (str) – the id of the node
• node_class (str) – The class of the NodeRun to create (e.g. SourceNodeRun, NodeRun)
• parent (Network) – the parent network of the node
• cores (int) – number of cores required for executing this Node
• memory (str) – amount of memory required in the form d+[mMgG] where M is for megabyte and G for gigabyte
• walltime (str) – amount of time required in second or in the form HOURS:MINUTES:SECOND

Returns:

the newly created Node

__module__ = 'fastr.planning.node'

__ne__(other)

Check two Node instances for inequality. This is the inverse of __eq__

Parameters:other (Node) – the other instances to compare to Returns:True if unequal, False otherwise

__repr__()

Get a string representation for the Node

Returns:the string representation Return type:str

__setstate__(state)

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

__str__()

Get a string version for the Node

Returns:the string version Return type:str

blocking

Indicate that the results of this Node cannot be determined without first executing the Node, causing a blockage in the creation of jobs. A blocking Nodes causes the Chunk borders.

classmethod createobj(state, network=None)

Create object function for generic objects

Parameters:

• cls – The class to create
• state – The state to use to create the Link
• network – the parent Network

Returns:

newly created Link

dimensions

The dimensions has to be implemented by any subclass. It has to provide a tuple of Dimensions.

Returns:dimensions Return type:tuple

dimnames

Names of the dimensions in the Node output. These will be reflected in the SampleIdList of this Node.

draw(context, graph, color=None)

draw_id(context)

draw_link_target(context, port_name, input=True)

find_source_index(target_index, target, source)

fullid

The full defining ID for the Node inside the network

get_sourced_nodes()

A list of all Nodes connected as sources to this Node

Returns:list of all nodes that are connected to an input of this node

global_id

The global defining ID for the Node from the main network (goes out of macro nodes to root network)

id

The id of the Node

input_groups

A list of input groups for this Node. An input group is InputGroup

object filled according to the Node

inputs = None

A list of inputs of this Node

listeners

All the listeners requesting output of this node, this means the listeners of all Outputs and SubOutputs

merge_dimensions

name

Name of the Tool the Node was based on. In case a Toolless Node was used the class name is given.

nodegroup

outputs = None

A list of outputs of this Node

outputsize

The size of output of this SourceNode

parent

The parent is the Network this Node is part of

status

tool

update_input_groups()

Update all input groups in this node

class fastr.planning.node.OutputDict

Bases: collections.OrderedDict

The container containing the Inputs of Node. Only checks if the inserted values are actually outputs.

__module__ = 'fastr.planning.node'

__setitem__(key, value)

Set an output.

Parameters:

• key (str) – the of the item to set
• value (BaseOutput) – the output to set
• dict_setitem – the setitem function to use for the underlying OrderedDict insert

class fastr.planning.node.SinkNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.Node

Class which handles where the output goes. This can be any kind of file, e.g. image files, textfiles, config files, etc.

__abstractmethods__ = frozenset()

__dataschemafile__ = 'SinkNode.schema.json'

__getstate__()

Retrieve the state of the Node

Returns:the state of the object Rtype dict:

__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)

Instantiation of the SourceNode.

Parameters:

• datatype – The datatype of the output.
• id – the id of the node to create

Returns:

newly created sink node

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> sink = network.create_sink(datatype=types['ITKImageFile'], id_='SinkN')

__module__ = 'fastr.planning.node'

__setstate__(state)

Set the state of the Node by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

datatype

The datatype of the data this sink can store.

draw(context, graph, color=None)

input

The default input of the sink Node

class fastr.planning.node.SourceNode(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)

Bases: fastr.planning.node.FlowNode

Class providing a connection to data resources. This can be any kind of file, stream, database, etc from which data can be received.

__abstractmethods__ = frozenset()

__dataschemafile__ = 'SourceNode.schema.json'

__getstate__()

Retrieve the state of the SourceNode

Returns:the state of the object Rtype dict:

__init__(datatype, id_=None, parent=None, resource_limits=None, nodegroup=None)

Instantiation of the SourceNode.

Parameters:

• datatype – The (id of) the datatype of the output.
• id – The url pattern.

This class should never be instantiated directly (unless you know what you are doing). Instead create a source using the network class like shown in the usage example below.

usage example:

>>> import fastr
>>> network = fastr.create_network()
>>> source = network.create_source(datatype=types['ITKImageFile'], id_='sourceN')

__module__ = 'fastr.planning.node'

__setstate__(state)

Set the state of the SourceNode by the given state.

Parameters:state (dict) – The state to populate the object with Returns:None

datatype

The datatype of the data this source supplies.

dimensions

The dimensions in the SourceNode output. These will be reflected in the SampleIdLists.

draw(context, graph, color=None)

nodegroup

output

Shorthand for self.outputs['output']

set_data(data, ids=None)

Set the data of this source node.

Parameters:

• data (dict, OrderedDict or list of urls) – the data to use
• ids – if data is a list, a list of accompanying ids

sourcegroup

valid

This does nothing. It only overloads the valid method of Node(). The original is intended to check if the inputs are connected to some output. Since this class does not implement inputs, it is skipped.

Subpackages

• test Package
  • test_network Module
  • test_node Module