Changelog#

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning

3.4.0 - 2024-05-23#

Added#

  • Adds configuration setting preferred_target to fix the Target type that is used.

  • Adds configuration flag singularity_enable_nv to enable the use of Nvidia GPUs from a SingularityTarget.

  • Adds logic to SingularityTarget to handle stored Singularity containers through FastrPI.

  • SLURM Plugin now supports a nodelist or node exclusion.

Fixed#

  • Fixed a bug where re-using old job results would compromise the provenance and extra_job_info files.

  • Fixed a bug with the FastrInterface where negative order was interpreted wrong.

  • Fixed a bug in Tool.target which blocked the use of multiple Targets in the same Tool.

Changed#

  • Removed heavy keyring dependency from requirements.txt, it is only needed for a plugin.

3.3.1 - 2022-10-13#

Added#

  • NetworkRun keeps track of the directory (scope) form which they were loaded

  • A network:// ioplugin that allows user to retrieve data relative to a network file

  • Added the examples and fastr_home default mounts to the configuration

  • Added support for GPUs and a memory multiplier in the ResourceLimit class. Limits and requirements set for the use of GPUs in ResourceLimit are now supported in the Slurm execution plugin.

Fixed#

  • Deffered without valid target could cause errors in checksum and validate

  • Python 3.10 compatability fix

  • Jobs are now submitted during creation and not after all jobs have been generated

3.3.0 - 2021-06-11#

Added#

  • Added concept op missing data. Using fastr.MISSING as source data will note a sample as missing. The network will run each job that has missing data will not be executed and they outputs will be noted as missing too. At the sink the result will be set to missing instead of failed, allowing for a partial execution of a network if it is know upfront some data is missing.

  • Added functionality for creating a reference result for Tool verification, in the form of fastr.utils.verify.create_tool_test().

  • tracking_id argument to a network run which gives the run a tracking id, all log messages from the network run will be tagged with the tracking id for filtering/combining the logs in a central log system

Fixes#

  • Fixes fastr verify for Tools by minor changes in Tool.test_tool().

  • Fixes bug in Slurm execution plugin

  • Fixes in ProcessPoolExecutor with the cleanup etc

  • Fixes bug with setting environment variable in LocalBinaryTarget

  • Fixes issue with case-sensitivity in vfs on Windows

3.2.3 - 2020-06-25#

Fixed#

  • Warning for non-production environment didn’t handle git tag correctly

3.2.2 - 2020-06-25#

Fixed#

  • Fixed a bug where ConstantNodes would not always set their data to use a DataType subclass.

  • Made version system scrape info from git instead of mercurial to reflect the change in versioning system.

3.2.1 - 2020-06-22#

Fixed#

  • Some bugs on windows due to use of Path in subprocess arguments

  • Added retry to serializable in case of small filesystem sync/timing errors

3.2.0 - 2020-06-19#

Changed#

  • Changed serialization in Fastr. Networks and Jobs have a better format and are serialized to yaml by default. This makes the job files human readable.

3.1.4 - 2020-06-10#

Added#

  • Added functionality to be able to use the cardinality of one of the items in an ordereddict input or output.

  • Added dependency list function to the Network API.

3.1.3 - 2019-11-28#

Added#

  • Support for FASTR_CONFIG_DIRS to add extra configuration directories (they will be loaded in order after the config.d directory has been loaded).

Improved#

  • The DRMAA execution plugin is more robust and less likely to encounter errors that will cause the execution to become stuck.

Fixed#

3.1.2 - 2019-06-18#

Improved#

  • Avoid execution plugins calling cleanup multiple times

  • Tools can now set an input to environment variables using the environ attribute. The parameter will NOT be put command-line anymore and instead be dispatched via an environment variable given by the environ argument value

Fixed#

  • Bug in XNATStorage plugin where files with a path within the resource could not be correctly located

  • Add timeout when waiting to send to PIM

  • Fix problem with non-requested outputs being able to invalidate a job execution

3.1.1 - 2019-05-02#

Fixed#

  • Packaging problem in release (old file left in build folder)

3.1.0 - 2019-05-02#

Added#

  • Added support for tools in YAML

  • fastr upgrade can also upgrade tools from XML to YAML

  • fastr report command to print an overview report of a job result

Fixed#

  • Re-added support for named sub-inputs

Improved#

  • Fixes in fastr upgrade to handle more exotic whitespace and arguments

  • Small documentation fixes (especially in configuration section)

  • Better windows support (tested by users)

Changed#

  • In ResourceLimits the default time of jobs is now None (no limit) instead of 1 hour.

  • By default do not log to files (we noticed fastr logs are not very often read by users and they could cause some issues with log rotation, by default logging to files is turned off, switching it back on can be done by setting log_to_file = True in the fastr.config

3.0.1 - 2019-03-28#

Fixed#

  • Improved implementation of fastr upgrade to handle newlines in the create_node function properly. Also can handle old-fashioned use of fastr.toollist[…] in create_node.

3.0.0 - 2019-03-05#

Changed#

  • Now ported to Python 3.6+ (Python 2 is no longer supported!)

  • New public API which is not fully compatible with fastr 2.x, the changes are small. The new API will be guaranteed in next minor version upgrades and is considered to be stable.

  • Clear way of defining resource limits for Nodes in a Network using the ResourceLimit class.

  • The datatype and cardinality of inputs of a tool are now checked before the tool is to be executed as an extra safety.

  • Dimensions are drawn by default in network.draw

  • The api now accepts types other than Output, list, tuple when creating a link. When a single value is given it is assumedly a constant from the network definition.

  • Drawing a network will not create temporary .dot files anymore

  • Sinkdata can be a string, it that case it will be the same string for all sink nodes so a {node} substitution should be used in the template

  • Make the xnat ioplugin use xnat+http:// and xnat+https:// url schemes in favour of xnat:// with ?insecure=… (old behaviour will also work for now)

  • Complete rewrite of PIM plugin (PIMReporter) making use of the new Reporter plugin infrastructure. It also caches all communication with PIM to be resilient against connection interruptions.

Added#

  • fastr upgrade command to automatically upgrade a network creation file from fastr 2.x to fastr 3.x API.

  • http(s) IOPlugin for downloading files via http(s)

  • network.draw now has a flag to hide the unconnected inputs and output of a node. The unconnected inputs/outputs are hidden by default.

  • Reporting plugins, Fastr now exposes a number of message hooks which can be listened to by Reporter plugins.

Fixed#

  • Fixed some bugs with drmaa communication (more safeties added)

  • Fixed a bug in the MacroNode update function which could cause networks with MacroNodes to be invalid

  • The margins and font size of the network.draw graph rendering are set a bit wider and smaller (resp.) to avoid excessive text overflow.

  • Fixed bug in provenance which did not properly chain the provenance of subsequent jobs.

2.1.2 - 2018-10-24#

Added#

  • Allow overriding the timestamp of the network execution

Changed#

  • Updated PIM publisher to support the new PIM API v2

  • Updated XNAT IOPlugin to not crash when creating a resource failed because another process already did that (race condition)

  • Make default resource limits for DRMAA configurable

  • Add stack trace to FastrExceptions

2.1.1 - 2018-06-29#

Fixed#

  • Fixed some issues with the type estimation of outputs of Jobs and update validation functions of NIFTI files

2.1.0 - 2018-04-13#

Added#

  • SLURM execution plugin based on sbatch, scancel, scontrol and squeue. The plugin supports job dependencies and cancellation.

  • Support for running tools in Docker containers using a DockerTarget

  • Support for running tools in Singularity containers using a SingularityTarget

  • Support for datatypes with multiple extensions (e.g. .tif and .tiff) by setting the extension to a tuple of options. The first extension is leading for deciding filenames in a sink.

Changed#

  • Source jobs now also validate the output (and do not only rely on the stderr of the tool)

  • Added preferred_types attribute to TypeGroups that gives the order of preference of members, alternatively the order of _members is used (this should be given as tuple or list to be meaningful)

  • In the config.py you can now access the USER_DIR and SYSTEM_DIR variables for use in setting other variables. These are only read and changing them will only change subsequent config reads but not the main config values.

  • checksum for nii.gz now takes the md5 checksum of the decompressed data

  • Serialization of MacroNodes now should function properly

Fixed#

  • BUG in XNAT plugin that made it impossible to download data from scans without an empty type string

  • BUG where the order of OrderedDict in a source was not preserved

  • BUG where newer Werkzeug version requires the web port to be an integer

2.0.1 - 2017-10-19#

  • Fix a bug in the validation of FilePrefix datatypes

2.0.0 - 2017-09-28#

Added#

  • The default python logger can now be configured from the fastr config file under key logging_config

  • Support for MacroNodes, a Network can be used as a Node inside of another Network. There is should be no limitation on the internal Network used, but currently the MacroNode ignores input_groups on its inputs.

  • A sync helper was added to assist in slow file synchronisation over NFS

  • Source and Sink can now handle S3 URL’s

  • FastrInterface can now forward errors from a subprocess if they are dumped to stdout or stderr in a json identified by __FASTR_ERRORS__ = [].

  • A specials.workdir field in the location field of automatic outputs that gives the current working directory (e.g. job directory)

  • Added support for Torque (using pbs-drmaa library) to DRMAAExecution

  • Added option to set a limit for number of jobs submitted at same time be the DRMAAExecution

  • Use of the ~/.fastr/config.d directory for adding additional config files. Any .py file in there will be parsed in alphabetical order.

  • XNATStorage IOPlugin now has a retry scheme for uploads, if an uploaded file could not be found on the server, it is retried up to 3 times.

  • Added fastr dump command to create a zip containing all important debugging information.

Changed#

  • FilePrefix type does not have an extension anymore (avoids ugly dot in middle of filename)

  • Allow expanding of link where samples have a non-uniform cardinality. This will not result in a sparse array.

  • The default for required for the automatic outputs is now False

  • Removed testtool commandline subcommand in favour of the test subcommand which can test both Tools and Networks

  • Moved nodegroup specification into the Node for speedup

Fixed#

  • Stop Jobs from failing when a non-required, non-requested output is invalid

  • Bug in boolean value parsing in the Boolean datatype

  • Bug in target that caused paths not to be expanded properly in some cases

  • Made sure failed sources also create a sample so the failure becomes visible and traceable.

  • Bug in XNAT IOPlugin that made download from XNAT seem to fail (while getting the correct data).

Removed#

  • fastr.current_network has been removed as it was deemed to “magical” and could change things out of the sight of the user.

1.2.2 - 2017-08-24#

Fixed#

  • Fixed a bug breaking the XNAT IOPlugin due to an xnatpy version update.

1.2.1 - 2017-04-04#

Added#

  • A FastrInterface can now specify a negate flag on an automatic output that also has a prefix, which will negate the flag. This is useful for flag the suppress the creation of an output (e.g. no_mask). An example is given in the Tool fastr.util.AutoPrefixNegateTest.

Changed#

  • The provenance and extra information of a Job now is not serialized in the Job, but exported to separate files next to the job file __fastr_prov__.json and __fastr_extra_job_info__.json which makes the information more accessible and reduces the memory footprint of the main process hugely as it will not read this information back anymore.

  • Most execution plugin will not overwrite the executionscript stdout and stderr but rather append it. This is only relevant when continuing a run in the an existing temporary directory, but avoids loss of information.

Fixed#

  • Bug that stopped the Link.append function from returning the newly created link

  • Bugs that caused some cardinality computations of the output to fail during execution

  • Bug in the job.tmpurl that caused double slashes somewhere. Some tools chocked on this when it was used for parameters.

1.2.0 - 2017-03-15#

Added#

  • Failed sample annotation: when a job fails, the result is annotated and forwarded until a SinkNode, where we can determine the status and possibly point of failure of the Sample.

  • Commandline tool fastr trace that can inspect a workflow run and help trace errors and print debug information

  • Supported for Lmod modules environment next to the old environmentmodules

  • BaseDataType descendants are now (un)picklable (including EnumTypes)

  • Option to use {extension} field in sink_data, which differs from {ext} in that it doesn’t include a leading dot.

  • Support for Docker targets. A Docker target will execute a command inside of a specified docker container, allowing Tools to use Docker for distribution

  • Using the right and left shift operator (<< and >>) for creating links to Inputs using input << output or output >> input.

  • In the FastrInterfaces, automatic outputs can have a prefix for a flag that should be set for the output to be actually generated.

  • Fastr is now able to limit the amount of SourceJobs that are allowed to run concurrently.

  • Ability to report progress to PIM (use the pim_host field in the config)

Changed#

  • Version can now also accept a format based on a date (e.g. 2017-02-17_bananas) which will be parsed the same way as 2017.02.17_bananas

  • Work on the ExecutionPlugin and the corresponding API. Has better fall-backs and a mechanism to advertise plugin capabilities.

  • The collector plugins have the input and input_parts fields merged, and the output and output_parts fields merged.

Fixed#

  • In some cases the log directory was not created properly, causing an handled exception

  • A bug making the handling of Booleans incorrect for the FastrInterface, when a Boolean was given a flag would also appear when it was False

  • Serialization of the namespace of a Network was not correct

  • Check version of Fastr that creates and executes a Job against each other

  • load_gpickle helper can handle data with Enums that use to cause an AttributeError

  • Output validation of Jobs did not work correctly for automatic outputs

1.1.2 - 2016-12-22#

Fixed#

  • The example network in resources/networks/add_ints.json was using an old serialization format making it non-functions. Replaced by a new network file.

1.1.1 - 2016-12-22#

Fixed#

  • Network runs called from an interpreter (and not file) caused a crash because the network tried to report the file used. Better handling of these situations.

1.1.0 - 2016-12-08#

Added#

  • Namespaces for resources (tools and networks)

  • Network manager located at fastr.networklist

  • RQExecution plugin. This plugin uses python-rq to manage a job queue.

  • LinearExecution plugin. This plugin uses a background thread for execution.

  • BlockingExecution plugin. This plugin executes jobs in a blocking fashion.

  • Automatic generation of documentation for all plugins, the configuration fields and all commandline tools.

Changed#

  • Provenance is updated with a network dump and used tool definitions.

  • New configuration system that uses python files

  • New plugin system that integrates with the new configuration system and enables automatic importing of plugins

  • The fastr command line tools now use an entrypoint which is located in fastr.utils.cmd. This code also dispatches the sub commands.

Removed#

  • fastr.config file. This is replaced by the config.py file. Go to the docs!

Fixed#

  • Adds explicit tool namespace and version to the provenance document.