User Manual¶

In this chapter we will discuss the parts of Fastr in more detail. We will give a more complete overview of the system and describe the more advanced features.

Tools¶

The Tools in Fastr are the building blocks of each workflow. A tool represents a program/script/binary that can be called by Fastr and can be seens as a template. A Node can be created based on a Tool. The Node will be one processing step in a workflow, and the tool defines what the step does.

On the import of Fastr, all available Tools will be loaded in a default ToolManager that can be accessed via fastr.toollist. To get an overview of the tools in the system, just print the repr() of the ToolManager:

>>> fastr.toollist
AddImages                v0.1    :  /home/hachterberg/dev/fastr/fastr/resources/tools/addimages/v1_0/addimages.xml
AddInt                   v0.1    :  /home/hachterberg/dev/fastr/fastr/resources/tools/addint/v1_0/addint.xml

As you can see it gives the tool id, version and the file from which it was loaded for each tool in the system. To view the layout of a tool, just print the repr() of the tool itself.

>>> fastr.toollist['AddInt']
Tool AddInt v0.1 (Add two integers)
       Inputs          |       Outputs
---------------------------------------------
left_hand  (Int)       |  result   (Int)
right_hand (Int)       |

To add a Tool to the system a file should be added to one of the path in fastr.config.tools_path. The structure of a tool file is described in Tool description

Create your own tool¶

There are 4 steps in creating a tool:

Create folders. We will call the tool ThrowDie. Create the folder throw_die in the folder fastr-tools. In this folder create another folder called bin.

Place executable in correct place. In this example we will use a snippet of executable python code:

#!/usr/bin/env python
import sys
import random
import json

if (len(sys.argv) > 1):
    sides = int(sys.argv[1])
else:
    sides = 6
result = [int(random.randint(1, sides ))]

print('RESULT={}'.format(json.dumps(result)))

Save this text in a file called throw_die.py

Place the executable python script in the folder throw_die/bin

Create and edit xml file for tool. See tool definition reference for all the fields that can be defined in a tool.

Put the following text in file called throw_die.xml.

<tool id="ThrowDie" description="Simulates a throw of a die. Number of sides of the die is provided by user"
      name="throw_die" version="1.0">
  <authors>
    <author name="John Doe" />
  </authors>
  <command version="1.0" >
    <authors>
      <author name="John Doe" url="http://a.b/c" />
    </authors>
    <targets>
      <target arch="*" bin="throw_die.py" interpreter="python" os="*" paths='bin/'/>
    </targets>
    <description>
       throw_die.py number_of_sides
       output = simulated die throw
    </description>
  </command>
  <interface>
    <inputs>
      <input cardinality="1" datatype="Int" description="Number of die sides" id="die_sides" name="die sides" nospace="False" order="0" required="True"/>
     </inputs>
    <outputs>
      <output id="output" name="output value" datatype="Int" automatic="True" cardinality="1" method="json" location="^RESULT=(.*)$" />
    </outputs>
  </interface>
</tool>

Put throw_die.xml in the folder example_tool. All Attributes in the example above are required. For a complete overview of the xml Attributes that can be used to define a tool, check the Tool description. The most important Attributes in this xml are:

id      : The id is used in in FASTR to create an instance of your tool, this name will appear in the toollist when you type fastr.toollist.
targets : This defines where the executables are located and on which platform they are available.
inputs  : This defines the inputs that you want to be used in FASTR, how FASTR should use them and what data is allowed to be put in there.

More xml examples can be found in the fastr-tools folder.

Edit configuration file. Append the line [PATH TO LOCATION OF FASTR-TOOLS]/fastr-tools/throw_die/ to the the config.py (located in ~/.fastr/ directory) to the tools_path. See Config file for more information on configuration.

You should now have a working tool. To test that everything is ok do the following in python:
```
>>> import fastr
>>> fastr.toollist
```

Now a list of available tools should be produced, including the tool throw_die

To test the tool create the script test_throwdie.py:

import fastr
network = fastr.Network()
source1 = network.create_source(fastr.typelist['Int'], id_='source1')
sink1 = network.create_sink(fastr.typelist['Int'], id_='sink1')
throwdie = network.create_node(fastr.toollist['ThrowDie'], id_='throwdie')
link1 = network.create_link(source1.output, throwdie.inputs['die_sides'])
link2 = network.create_link(throwdie.outputs['output'], sink1.inputs['input'])
source_data = {'source1': {'s1': 4, 's2': 5, 's3': 6, 's4': 7}}
sink_data = {'sink1': 'vfs://tmp/fastr_result_{sample_id}.txt'}
network.draw_network()
network.execute(source_data, sink_data)

Call the script from commandline by

$ python test_throwdie.py

An image of the network will be created in the current directory and result files will be put in the tmp directory. The result files are called fastr_result_s1.txt, fastr_result_s2.txt, fastr_result_s3.txt, and fastr_result_s4.txt

Note

If you have code which is operating system depend you will have to edit the xml file. The following gives and example of how the elastix tool does this:

<targets>
      <target os="windows" arch="*" bin="elastix.exe">
        <paths>
          <path type="bin" value="vfs://apps/elastix/4.7/install/" />
          <path type="lib" value="vfs://apps/elastix/4.7/install/lib" />
        </paths>
      </target>
      <target os="linux" arch="*" modules="elastix/4.7" bin="elastix">
        <paths>
          <path type="bin" value="vfs://apps/elastix/4.7/install/" />
          <path type="lib" value="vfs://apps/elastix/4.7/install/lib" />
        </paths>
      </target>
      <target os="darwin" arch="*" modules="elastix/4.7" bin="elastix">
        <paths>
          <path type="bin" value="vfs://apps/elastix/4.7/install/" />
          <path type="lib" value="vfs://apps/elastix/4.7/install/lib" />
        </paths>
      </target>
   </targets>

vfs is the virtual file system path, more information can be found at VirtualFileSystem.

Network¶

A Network represented an entire workflow. It hold all Nodes, Links and other information required to execute the workflow. Networks can be visualized as a number of building blocks (the Nodes) and links between them:

An empty network is easy to create, all you need is to name it:

>>> network = fastr.Network(id_="network_name")

The Network is the main interface to Fastr, from it you can create all elements to create a workflow. In the following sections the different elements of a Network will be described in more detail.

Node¶

Nodes are the point in the Network where the processing happens. A Node takes the input data and executes jobs as specified by the underlying Tool. A Nodes can be created in a two different ways:

>>> node1 = fastr.Node(tool, id_='node1', parent=network)
>>> node2 = network.create_node(tool, id_='node2', stepid='step1')

In the first way, we specifically create a Node object. We pass it an id and the parent network. If the parent is None the fastr.curent_network will be used. The Node constructor will automaticaly add the new node to the parent network.

Note

For a Node, the tool can be given both as the Tool class or the id of the

tool.

The second way, we tell the network to create a Node. The network will automatically assign itself as the parent. Optionally you can add define a stepid for the node which is a logical grouping of Nodes that is mostly used for visualization.

A Node contains Inputs and Outputs. To see the layout of the Node one can simply look at the repr().

>>> addint = fastr.Node(fastr.toollist['AddInt'], id_='addint')
>>> addint
Node addint (tool: AddInt v1.0)
       Inputs          |       Outputs
---------------------------------------------
left_hand  (Int)       |  result   (Int)
right_hand (Int)       |

The inputs and outputs are located in mappings with the same name:

>>> addint.inputs
InputDict([('left_hand', <Input: fastr:///networks/unnamed_network/nodelist/addint/inputs/left_hand>), ('right_hand', <Input: fastr:///networks/unnamed_network/nodelist/addint/inputs/right_hand>)])

>>> addint.outputs
OutputDict([('result', Output fastr:///networks/unnamed_network/nodelist/addint/outputs/result)])

The InputDict and OutputDict are classes that behave like mappings. The InputDict also facilitaties the linking shorthand. By assigning an Output to an existing key, the InputDict will create a Link between the InputDict and Output.

SourceNode¶

A SourceNode is a special kind of node that is the start of a workflow. The SourceNodes are given data at run-time that fetched via IOPlugins. On create, only the datatype of the data that the SourceNode supplied needs to be known. Creating a SourceNode is very similar to an ordinary node:

>>> source1 = fastr.SourceNode('Int', id_='source1')
>>> source2 = network.create_source(fastr.typelist['Int'], id_='source2', stepid='step1')

In both cases, the source is automatically automaticall assigned to a network. In the first case to the fastr.current_network and in the second case to the network used to call the method. A SourceNode only has a single output which has a short-cut access via source.output.

Note

For a source or constant node, the datatype can be given both as the BaseDataType class or the id of the datatype.

ConstantNode¶

A ConstantNode is another special node. It is a subclass of the SourceNode and has a similar function. However, instead of setting the data at run-time, the data of a constant is given at creation and saved in the object. Creating a ConstantNode is similar as creating a source, but with supplying data:

>>> constant1 = fastr.ConstantNode('Int', [42], id_='constant1')
>>> constant2 = network.create_constant('Int', [42], id_='constant2', stepid='step1')

Often, when a ConstantNode is created, it is created specifically for one input and will not be reused. In this case there is a shorthand to create and link a constant to an input:

>>> addint.inputs['value1'] = [42]

will create a constant node with the value 42 and create a link between the output and input addint.value1.

SinkNode¶

The SinkNode is the counter-part of the source node. Instead of get data into the workflow, it saves the data resulting from the workflow. For this a rule has to be given at run-time that determines where to store the data. The information about how to create such a rule is described at SinkNode.set_data. At creation time, only the datatype has to be specified:

>>> sink1 = fastr.Sink('Int', id_='sink1')
>>> sink2 = network.create_sink(fastr.typelist['Int'], id_='sink2', stepid='step1')

Link¶

Links indicate how the data flows between Nodes. Links can be created explicitly using on of the following:

>>> link = fastr.Link(node1.outputs['image'], node2.inputs['image'])
>>> link = network.create_link(node1.outputs['image'], node2.inputs['image'])

or can be create implicitly by assigning an Output to an Input in the InputDict.

# This style of assignment will create a Link similar to above
>>> node2.inputs['image'] = node1.outputs['image']

Note that a Link is also create automatically when using the short-hand for the ConstantNode

Data Flow¶

The data enters the Network via SourceNodes, flows via other Nodes and leaves the Network via SinkNodes. The flow between Nodes goes from an Output via a Link to an Input. In the following image it is simple to track the data from the SourceNodes at the left to the SinkNodes at right side:

Note that the data in Fastr is stored in the Output and the Link and Input just give access to it (possible while transforming the data).

Data flow inside a Node¶

In a Node all data from the Inputs will be combined and the jobs will be generated. There are strict rules to how this combination is performed. In the default case all inputs will be used pair-wise, and if there is only a single value for an input, it it will be considered as a constant.

To illustrate this we will consider the following Tool (note this is a simplified version of the real tool):

>>> fastr.toollist['Elastix']
Tool Elastix v4.8 (Elastix Registration)
                         Inputs                            |             Outputs
----------------------------------------------------------------------------------------------
fixed_image       (ITKImageFile)                           |  transform (ElastixTransformFile)
moving_image      (ITKImageFile)                           |
parameters        (ElastixParameterFile)                   |

Also it is important to know that for this tool (by definition) the cardinality of the transform Output will match the cardinality of the parameters Inputs

If we supply a Node based on this Tool with a single sample on each Input, there will be one single matching Output sample created:

If the cardinality of the parameters sample would be increased to 2, the resulting transform sample would also become 2:

../_images/flow_simple_one_sample_two_cardinality.svg

Now if the number of samples on fixed_image would be increased to 3, the moving_image and parameters will be considered constant and be repeated, resulting in 3 transform samples.

Then if the amount of samples for moving_image is also increased to 3, the moving_image and fixed_image will be used pairwise and the parameters will be constant.

../_images/flow_simple_three_sample_two_cardinality.svg

Advanced flows in a Node¶

Sometimes the default pairwise behaviour is not desirable. For example if you want to test all combinations of certain input samples. To achieve this we can change the input_group of Inputs to set them apart from the rest. By default all Inputs are assigned to the default input group. Now let us change that:

>>> node = network.create_node('Elastix', id_='elastix')
>>> node.inputs['moving_image'].input_group = 'moving'

This will result in moving_image to be put in a different input group. Now if we would supply fixed_image with 3 samples and moving_image with 4 samples, instead of an error we would get the following result:

Warning

TODO: Expand this section with the merging dimensions

Data flows in a Link¶

As mentioned before the data flows from an Output to an Input throuhg a Link. By default the Link passed the data as is, however there are two special directives that change the shape of the data:

Collapsing flow, this collapses certain dimensions from the sample array into the cardinality. As a user you have to specify the dimension or tuple of dimensions you want to collapse.

This is useful in situation where you want to use a tool that aggregates over a number of samples (e.g. take a mean or sum).

To achieve this you can set the collapse property of the Link as follows:
```
>>> link.collapse = 'dim1'
>>> link.collapse = ('dim1', 'dim2')  # In case you want to collapse multiple dimensions
```
Expanding flow, this turns the cardinality into a new dimension. The new dimension will be named after the Output from which the link originates. It will be in the form of {nodeid}__{outputid}

This flow directive is useful if you want to split a large sample in multiple smaller samples. This could be because processing the whole sample is not feasible because of resource constraints. An example would be splitting a 3D image into slices to process separately to avoid high memory use or to achieve parallelism.

To achieve this you can set the expand property of the Link to True:
```
>>> link.expand = True
```

Note

both collapsing and expanding can be used on the same link, it will executes similar to a expand-collapse sequence, but the newly created expand dimension is ignored in the collapse.

>>> link.collapse = 'dim1'
>>> link.expand = True

Data flows in an Input¶

If an Inputs has multiple Links attached to it, the data will be combined by concatenating the values for each corresponding sample in the cardinality.

Broadcasting (matching data of different dimensions)¶

Sometimes you might want to combine data that does not have the same number of dimensions. As long as all dimensions of the lower dimensional datasets match a dimension in the higher dimensional dataset, this can be achieved using broadcasting. The term broadcasting is borrowed from NumPy and described as:

“The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.”

—NumPy manual on broadcasting

In fastr it works similar, but to combined different Inputs in an InputGroup. To illustrate broadcasting it is best to use an example, the following network uses broadcasting in the transformix Node:

As you can see this visualization prints the dimensions for each Input and Output (e.g. the elastix.fixed_image Input has dimensions [N]). To explain what happens in more detail, we present an image illustrating the details for the samples in elastix and transformix:

In the figure the moving_image (and references to it) are identified with different colors, so they are easy to track across the different steps.

At the top the Inputs for the elastix Node are illustrated. Because the input groups a set differently, output samples are generated for all combinations of fixed_image and moving_image (see Advanced flows in a Node for details).

In the transformix Node, we want to combine a list of samples that is related to the moving_image (it has the same dimension name and sizes) with the resulting transform samples from the elastix Node. As you can see the sizes of the sample collections do not match ([N] vs [N x M]). This is where broadcasting comes into play, it allows the system to match these related sample collections. Because all the dimensions in [N] are known in [N x M], it is possible to match them uniquely. This is done automatically and the result is a new [N xM] sample collection. To create a matching sample collections, the samples in the transformix.image Input are reused as indicated by the colors.

Warning

Note that this might fail when there are data-blocks with non-unique dimension names, as it will be not be clear which of the dimensions with identical names should be matched!

DataTypes¶

In Fastr all data is contained in object of a specific type. The types in Fastr are represented by classes that subclass BaseDataType. There are a few different other classes under BaseDataType that are each a base class for a family of types:

DataType – The base class for all types that hold data
- ValueType – The base class for types that contain simple data (e.g. Int, String) that can be represented as a str
- EnumType – The base class for all types that are a choice from a set of options
- URLType – The base class for all types that have their data stored in files (which are referenced by URL)
TypeGroup – The base class for all types that actually represent a group of types

Fig. 2 The relation between the different DataType classes

The types are defined in xml files and created by the DataTypeManager. The DataTypeManager acts as a container containing all Fastr types. It is automatically instantiated as fastr.typelist. In fastr the created DataTypes classes are also automatically place in the fastr.datatypes module once created.

Resolving Datatypes¶

Outputs in fastr can have a TypeGroup or a number of DataTypes associated with them. The final DataType used will depend on the linked Inputs. The DataType resolving works as a two-step procedure.

All possible DataTypes are determined and considered as options.
The best possible DataType from options is selected for non-automatic Outputs

The options are defined as the intersection of the set of possible values for the Output and each separate Input connected to the Output. Given the resulting options there are three scenarios:

If there are no valid DataTypes (options is empty) the result will be None.
If there is a single valid DataType, then this is automatically the result (even if it is not a preferred DataType).
If there are multiple valid DataTypes, then the preferred DataTypes are used to resolve conflicts.

There are a number of places where the preferred DataTypes can be set, these are used in the order as given:

The preferred keyword argument to match_types
The preferred types specified in the fastr.config

Execution¶

Executing a Network is very simple:

>>> source_data = {'source_id1': ['val1', 'val2'],
                   'source_id2': {'id3': 'val3', 'id4': 'val4'}}
>>> sink_data = {'sink_id1': 'vfs://some_output_location/{sample_id}/file.txt'}
>>> network.execute(source_data, sink_data)

The Network.execute method takes a dict of source data and a dict sink data as arguments. The dictionaries should have a key for each SourceNode or SinkNode.

TODO: add .. figure:: images/execution_layers.*

The execution of a Network uses a layered model:

Network.execute will analyze the Network and call all Nodes.
Node.execute will create jobs and fill their payload
execute_job will execute the job on the execute machine and resolve any deferred values (val:// urls).
Tool.execute will find the correct target and call the interface and if required resolve vfs:// urls
Interface.execute will actually run the required command(s)

The ExecutionPlugin will call call the executionscript.py for each job, passing the job as a gzipped pickle file. The executionscript.py will resolve deferred values and then call Tool.execute which analyses the required target and executes the underlying Interface. The Interface actually executes the job and collect the results. The result is returned (via the Tool) to the executionscript.py. There we save the result, provenance and profiling in a new gzipped pickle file. The execution system will use a callback to load the data back into the Network.

The selection and settings of the ExecutionPlugin are defined in the fastr config.

Continuing a Network¶

Normally a random temporary directory is created for each run. To continue a previously stopped/crashed network, you should call the Network.execute method using the same temporary directory(tmp dir). You can set the temporary directory to a fixed value using the following code:

>>> tmpdir = '/tmp/example_network_rerun'
>>> network.execute(source_data, sink_data, tmpdir=tmpdir)

Warning

Be aware that at this moment, Fastr will rerun only the jobs where not all output files are present or if the job/tool parameters have been changed. It will not rerun if the input data of the node has changed or the actual tools have been adjusted. In these cases you should remove the output files of these nodes, to force a rerun.

IOPlugins¶

Sources and sink are used to get data in and out of a Network during execution. To make the data retrieval and storage easier, a plugin system was created that selects different plugins based on the URL scheme used. So for example, a url starting with vfs:// will be handles by the VirtualFileSystem plugin. A list of all the IOPlugins known by the system and their use can be found at IOPlugin Reference.

Secrets¶

Fastr uses a secrets system for storing and retrieving login credentials. Currently the following keyrings are supported:

Python keyring and keyrings.alt lib: - Mac OS X Keychain - Freedesktop Secret Service (requires secretstorage) - KWallet (requires dbus) - Windows Credential Vault - Gnome Keyring - Google Keyring (stores keyring on Google Docs) - Windows Crypto API (File-based keyring secured by Windows Crypto API) - Windows Registry Keyring (registry-based keyring secured by Windows Crypto API) - PyCrypto File Keyring - Plaintext File Keyring (not recommended)
Netrc (not recommended)

When a password is retrieved trough the fastr SecretService it loops trough all of the available SecretProviders (currently keyring and netrc) until a match is found.

The Python keyring library automatically picks the best available keyring backend. If you wish to choose your own python keyring backend it is possible to do so by make a keyring configuration file according to the keyring library documentation. The python keyring library connects to one keyring. Currently it cannot loop trough all available keyrings until a match is found.

Debugging¶

This section is about debugging Fastr tools wrappers, Fastr Networks (when building a Network) and Fastr Network Runs.

Debugging a Fastr tool¶

When wrapping a Tool in Fastr sometimes it will not work as expected or not load properly. Fastr is shipped with a command that helps checking Tools. The fastr verify command can try to load a Tool in steps to make it more easy to understand where the loading went wrong.

The fastr verify command will use the following steps:

Try to load the tool with and without compression
Try to find the correct serializer and make sure the format is correct
Try to validate the Tool content against the json_schema of a proper Tool
Try to create a Tool object
If available, execute the tool test

An example of the use of fastr verify:

$ fastr verify tool fastr/resources/tools/fastr/math/0.1/add.xml
[INFO]    verify:0020 >> Trying to read file with compression OFF
[INFO]    verify:0036 >> Read data from file successfully
[INFO]    verify:0040 >> Trying to load file using serializer "xml"
[INFO]    verify:0070 >> Validating data against Tool schema
[INFO]    verify:0080 >> Instantiating Tool object
[INFO]    verify:0088 >> Loaded tool <Tool: Add version: 1.0> successfully
[INFO]    verify:0090 >> Testing tool...

If your Tool is loading but not functioning as expected you might want to easily test your Tool without building an entire Network around it that can obscure errors. It is possible to run a tool from the Python prompt directly using tool.execute:

>>> tool.execute(left_hand=40, right_hand=2)
[INFO] localbinarytarget:0090 >> Changing ./bin
[INFO]      tool:0311 >> Target is <Plugin: LocalBinaryTarget>
[INFO]      tool:0318 >> Using payload: {'inputs': {'right_hand': (2,), 'left_hand': (40,)}, 'outputs': {}}
[INFO] localbinarytarget:0135 >> Adding extra PATH: ['/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/math/0.1/bin']
[INFO] fastrinterface:0393 >> Execution payload: {'inputs': {'right_hand': (2,), 'left_hand': (40,)}, 'outputs': {}}
[INFO] fastrinterface:0496 >> Adding (40,) to argument list based on <fastrinterface.InputParameterDescription object at 0x7fc950fa8850>
[INFO] fastrinterface:0496 >> Adding (2,) to argument list based on <fastrinterface.InputParameterDescription object at 0x7fc950fa87d0>
[INFO] localbinarytarget:0287 >> Options: ['/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/math/0.1/bin']
[INFO] localbinarytarget:0201 >> Calling command arguments: ['python', '/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/math/0.1/bin/addint.py', '--in1', '40', '--in2', '2']
[INFO] localbinarytarget:0205 >> Calling command: "'python' '/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/math/0.1/bin/addint.py' '--in1' '40' '--in2' '2'"
[INFO] fastrinterface:0400 >> Collecting results
[INFO] executionpluginmanager:0467 >> Callback processing thread ended!
[INFO] executionpluginmanager:0467 >> Callback processing thread ended!
[INFO] executionpluginmanager:0467 >> Callback processing thread ended!
[INFO] jsoncollector:0076 >> Setting data for result with [42]
<fastr.core.interface.InterfaceResult at 0x7fc9661ccfd0>

In this case an AddInt was ran from the python shell. As you can see it shows the payload it created based on the call, followed by the options for the directories that contain the binary. Then the command that is called is given both as a list and string (for easy copying to the prompt yourself). Finally the collected results is displayed.

Note

You can give input and outputs as keyword arguments for execute. If an input and output have the same name, you can disambiguate them by prefixing them with in_ or out_ (e.g. in_image and out_image)

Debugging an invalid Network¶

The simplest command to check if your Network is considered valid is to use the Network.is_valid method. It will simply check if the Network is valid:

>>> network.is_valid()
True

It will return a boolean that only indicates the validity of the Network, but it will print any errors it found to the console/log with the ERROR log level, for example when datatypes on a link do not match:

>>> invalid_network.is_valid()
[WARNING] datatypemanager:0388 >> No matching DataType available (args (<ValueType: Float class [Loaded]>, <ValueType: Int class [Loaded]>))
[WARNING]      link:0546 >> Cannot match datatypes <ValueType: Float class [Loaded]> and <ValueType: Int class [Loaded]> or not preferred datatype is set! Abort linking fastr:///networks/add_ints/0.0/nodelist/source/outputs/output to fastr:///networks/add_ints/0.0/nodelist/add/inputs/left_hand!
[WARNING] datatypemanager:0388 >> No matching DataType available (args (<ValueType: Float class [Loaded]>, <ValueType: Int class [Loaded]>))
[ERROR]   network:0571 >> [add] Input left_hand is not valid: SubInput fastr:///networks/add_ints/0.0/nodelist/add/inputs/left_hand/0 is not valid: SubInput source (link_0) is not valid
[ERROR]   network:0571 >> [add] Input left_hand is not valid: SubInput fastr:///networks/add_ints/0.0/nodelist/add/inputs/left_hand/0 is not valid: [link_0] source and target have non-matching datatypes: source Float and Int
[ERROR]   network:0571 >> [link_0] source and target have non-matching datatypes: source Float and Int
False

Because the messages might not always be enough to understand errors in the more complex Networks, we would advice you to create a plot of the network using the network.draw_network method:

>>> network.draw_network(network.id, draw_dimensions=True, expand_macro=True)
'add_ints.svg'

The value returned is the path of the output image generated (it will be placed in the current working directory. The draw_dimensions=True will make the drawing add indications about the sample dimensions in each Input and Output, whereas expand_macro=True causes the draw to expand MacroNodes and draw the content of them. If you have many nested MacroNodes, you can set expand_macro to an integer and that is the depth until which the MacroNodes will be draw in detail.

An example of a simple multi-atlas segmentation Network nicely shows the use of drawing the dimensions, the dimensions vary in certain Nodes due to the use of input_groups and a collapsing link (drawn in blue):

Debugging a Network run with errors¶

If a Network run did finish but there were errors detected, Fastr will report those at the end of the execution. We included an example of a Network that has failing samples in fastr/examples/failing_network.py which can be used to test debugging. An example of the output of a Network run with failures:

[INFO] networkrun:0604 >> ####################################
[INFO] networkrun:0605 >> #    network execution FINISHED    #
[INFO] networkrun:0606 >> ####################################
[INFO] networkrun:0618 >> ===== RESULTS =====
[INFO] networkrun:0627 >> sink_1: 2 success / 2 failed
[INFO] networkrun:0627 >> sink_2: 2 success / 2 failed
[INFO] networkrun:0627 >> sink_3: 1 success / 3 failed
[INFO] networkrun:0627 >> sink_4: 1 success / 3 failed
[INFO] networkrun:0627 >> sink_5: 1 success / 3 failed
[INFO] networkrun:0628 >> ===================
[WARNING] networkrun:0651 >> There were failed samples in the run, to start debugging you can run:

    fastr trace $RUNDIR/__sink_data__.json --sinks

see the debug section in the manual at https://fastr.readthedocs.io/en/default/static/user_manual.html#debugging for more information.

As you can see, there were failed samples in every sink. Also you already get the suggestion to use fastr trace. This command helps you inspect the staging directory of the Network run and pinpoint the errors.

The suggested command will print a similar summary as given by the network execution:

$ fastr trace $RUNDIR/__sink_data__.json --sinks
sink_1 -- 2 failed -- 2 succeeded
sink_2 -- 2 failed -- 2 succeeded
sink_3 -- 3 failed -- 1 succeeded
sink_4 -- 3 failed -- 1 succeeded
sink_5 -- 3 failed -- 1 succeeded

Since this is not given us new information we can add the -v flag for more output and limit the output to one sink, in this case sink_5:

$ fastr trace $RUNDIR/__sink_data__.json --sinks sink_5
sink_5 -- 3 failed -- 1 succeeded
  sample_1_1: Encountered error: [FastrOutputValidationError] Could not find result for output out_2 (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:970)
  sample_1_2: Encountered error: [FastrOutputValidationError] Could not find result for output out_1 (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:970)
  sample_1_3: Encountered error: [FastrOutputValidationError] Could not find result for output out_1 (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:970)
  sample_1_3: Encountered error: [FastrOutputValidationError] Could not find result for output out_2 (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:970)

Now we are given one error per sample, but this does not yet give us that much information. To get a very detailed report we have to specify one sink and one sample. This will make the fastr trace command print a complete error report for that sample:

$ fastr trace $RUNDIR/__sink_data__.json --sinks sink_5 --sample sample_1_1 -v
Tracing errors for sample sample_1_1 from sink sink_5
Located result pickle: /home/hachterberg/FastrTemp/fastr_failing_network_2017-09-04T10-44-58_uMWeMV/step_1/sample_1_1/__fastr_result__.pickle.gz


===== JOB failing_network___step_1___sample_1_1 =====
Network: failing_network
Run: failing_network_2017-09-04T10-44-58
Node: step_1
Sample index: (1)
Sample id: sample_1_1
Status: JobState.execution_failed
Timestamp: 2017-09-04 08:45:19.238192
Job file: /home/hachterberg/FastrTemp/fastr_failing_network_2017-09-04T10-44-58_uMWeMV/step_1/sample_1_1/__fastr_result__.pickle.gz

Command:
List representation: [u'python', u'/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/util/0.1/bin/fail.py', u'--in_1', u'1', u'--in_2', u'1', u'--fail_2']
String representation: 'python' '/home/hachterberg/dev/fastr-develop/fastr/fastr/resources/tools/fastr/util/0.1/bin/fail.py' '--in_1' '1' '--in_2' '1' '--fail_2'

Output data:
{'out_1': [<Int: 2>]}

Status history:
2017-09-04 08:45:19.238212: JobState.created
2017-09-04 08:45:21.537417: JobState.running
2017-09-04 08:45:31.578864: JobState.execution_failed

----- ERRORS -----
- FastrOutputValidationError: Could not find result for output out_2 (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:970)
- FastrValueError: [failing_network___step_1___sample_1_1] Output values are not valid! (/home/hachterberg/dev/fastr-develop/fastr/fastr/execution/job.py:747)
------------------

----- STDOUT -----
Namespace(fail_1=False, fail_2=True, in_1=1, in_2=1)
in 1  : 1
in 2  : 1
fail_1: False
fail_2: True
RESULT_1=[2]

------------------

----- STDERR -----

------------------

As shown above, it finds the result files of the failed job(s) and prints the most important information. The first paragraph shows the information about the Job that was involved. The second paragraph shows the command used both as a list (which is clearer and internally used in Python) and as a string (which you can copy/paste to the shell to test the command). Then there is the output data as determined by Fastr. The next section shows the status history of the Job which can give an indication about wait and run times. Then there are the errors that Fastr encounted during the execution of the Job. In this case it could not find the output for the Tool. Finally the stdout and stderr of the subprocess are printed. In this case we can see that RESULT_2=[…] was not in the stdout, and so the result could not be located.

Note

Sometimes there are no Job results in a directory, this usually means the process got killed before the Job could finished. On cluster environments, this often means that the process was killed due to memory constraints.

Asking for help with debugging¶

If you would like help with debugging, you can contact us via the fastr-users google group. To enable us to track the errors please include the following:

The entire log of the fastr run (can be copied from console or from the end of ~/.fastr/logs/info.log.
A dump of the network run, which can be created that by using the fastr dump command like:
```
$ fastr dump $RUNDIR fastr_run_dump.zip
```
This will create a zip file including all the job files, logs, etc but not the actual data files.

These should be enough information to trace most errors. In some cases we might need to ask for additional information (e.g. tool files, datatype files) or actions from your side.

Naming Convention¶

For the naming convention of the tools we tried to stay close to the Python PEP 8 coding style. In short, we defined toolnames as classes so they should be UpperCamelCased. The inputs and outputs of a tool we considered as functions or method arguments, these should we named lower_case_with_underscores.

An overview of the mapping of Fastr to PEP 8:

Fastr construct	Python PEP8 equivalent	Examples
Network.id	module	brain_tissue_segmentation
Tool.id	class	BrainExtractionTool, ThresholdImage
Node.id	variable name	brain_extraction, threshold_mask
Input/Output.id	method	image, number_of_classes, probability_image

Furthermore there are some small guidelines:

No input or output in the input or output names. This is already specified when setting or getting the data.

Add the type of the output that is named. i.e. enum, string, flag, image,

No File in the input/output name (Passing files around is what Fastr was developed for).

No type necessary where type is implied i.e. lower_threshold, number_of_levels, max_threads.

Where possible/useful use the fullname instead of an abbreviation.

Provenance¶

For every data derived data object, Fastr records the Provenance. The SinkNode write provenance records next to every data object it writes out. The records contain information on what operations were performed to obtain the resulting data object.

W3C Prov¶

The provenance is recorded using the W3C Prov Data Model (PROV-DM). Behind the scences we are using the python prov implementation.

The PROV-DM defines 3 Starting Point Classes and and their relating properties. See Fig. 3 for a graphic representation of the classes and the relations.

Fig. 3 The three Starting Point classes and the properties that relate them. The diagrams in this document depict Entities as yellow ovals, Activities as blue rectangles, and Agents as orange pentagons. The responsibility properties are shown in pink. [*]

Implementation¶

In the workflow document the provenance classes map to fastr concepts in the following way:

Agent:	Fastr, Networks, Tools, Nodes
Activity:	`Jobs`
Entities:	Data

Usage¶

The provenance is stored in ProvDocument objects in pickles. The convenience command line tool fastr prov can be used to extract the provenance in the PROV-N notation and can be serialized to PROV-JSON and PROV-XML. The provenance document can also be vizualized using the fastr prov command line tool.

Footnotes

[*]	This picture and caption is taken from http://www.w3.org/TR/prov-o/ . “Copyright © 2011-2013 World Wide Web Consortium, (MIT, ERCIM, Keio, Beihang). http://www.w3.org/Consortium/Legal/2015/doc-license”