Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning

2.1.1 - 2018-06-29

Fixed

  • Fixed some issues with the type estimation of outputs of Jobs and update validation functions of NIFTI files

2.1.0 - 2018-04-13

Added

  • SLURM execution plugin based on sbatch, scancel, scontrol and squeue. The plugin supports job dependencies and cancellation.
  • Support for running tools in Docker containers using a DockerTarget
  • Support for running tools in Singularity containers using a SingularityTarget
  • Support for datatypes with multiple extensions (e.g. .tif and .tiff) by setting the extension to a tuple of options. The first extension is leading for deciding filenames in a sink.

Changed

  • Source jobs now also validate the output (and do not only rely on the stderr of the tool)
  • Added preferred_types attribute to TypeGroups that gives the order of preference of members, alternatively the order of _members is used (this should be given as tuple or list to be meaningful)
  • In the config.py you can now access the USER_DIR and SYSTEM_DIR variables for use in setting other variables. These are only read and changing them will only change subsequent config reads but not the main config values.
  • checksum for nii.gz now takes the md5 checksum of the decompressed data
  • Serialization of MacroNodes now should function properly

Fixed

  • BUG in XNAT plugin that made it impossible to download data from scans without an empty type string
  • BUG where the order of OrderedDict in a source was not preserved
  • BUG where newer Werkzeug version requires the web port to be an integer

2.0.1 - 2017-10-19

  • Fix a bug in the validation of FilePrefix datatypes

2.0.0 - 2017-09-28

Added

  • The default python logger can now be configured from the fastr config file under key logging_config
  • Support for MacroNodes, a Network can be used as a Node inside of another Network. There is should be no limitation on the internal Network used, but currently the MacroNode ignores input_groups on its inputs.
  • A sync helper was added to assist in slow file synchronisation over NFS
  • Source and Sink can now handle S3 URL’s
  • FastrInterface can now forward errors from a subprocess if they are dumped to stdout or stderr in a json identified by __FASTR_ERRORS__ = [].
  • A specials.workdir field in the location field of automatic outputs that gives the current working directory (e.g. job directory)
  • Added support for Torque (using pbs-drmaa library) to DRMAAExecution
  • Added option to set a limit for number of jobs submitted at same time be the DRMAAExecution
  • Use of the ~/.fastr/config.d directory for adding additional config files. Any .py file in there will be parsed in alphabetical order.
  • XNATStorage IOPlugin now has a retry scheme for uploads, if an uploaded file could not be found on the server, it is retried up to 3 times.
  • Added fastr dump command to create a zip containing all important debugging information.

Changed

  • FilePrefix type does not have an extension anymore (avoids ugly dot in middle of filename)
  • Allow expanding of link where samples have a non-uniform cardinality. This will not result in a sparse array.
  • The default for required for the automatic outputs is now False
  • Removed testtool commandline subcommand in favour of the test subcommand which can test both Tools and Networks
  • Moved nodegroup specification into the Node for speedup

Fixed

  • Stop Jobs from failing when a non-required, non-requested output is invalid
  • Bug in boolean value parsing in the Boolean datatype
  • Bug in target that caused paths not to be expanded properly in some cases
  • Made sure failed sources also create a sample so the failure becomes visible and traceable.
  • Bug in XNAT IOPlugin that made download from XNAT seem to fail (while getting the correct data).

Removed

  • fastr.current_network has been removed as it was deemed to “magical” and could change things out of the sight of the user.

1.2.2 - 2017-08-24

Fixed

  • Fixed a bug breaking the XNAT IOPlugin due to an xnatpy version update.

1.2.1 - 2017-04-04

Added

  • A FastrInterface can now specify a negate flag on an automatic output that also has a prefix, which will negate the flag. This is useful for flag the suppress the creation of an output (e.g. no_mask). An example is given in the Tool fastr.util.AutoPrefixNegateTest.

Changed

  • The provenance and extra information of a Job now is not serialized in the Job, but exported to separate files next to the job file __fastr_prov__.json and __fastr_extra_job_info__.json which makes the information more accessible and reduces the memory footprint of the main process hugely as it will not read this information back anymore.
  • Most execution plugin will not overwrite the executionscript stdout and stderr but rather append it. This is only relevant when continuing a run in the an existing temporary directory, but avoids loss of information.

Fixed

  • Bug that stopped the Link.append function from returning the newly created link
  • Bugs that caused some cardinality computations of the output to fail during execution
  • Bug in the job.tmpurl that caused double slashes somewhere. Some tools chocked on this when it was used for parameters.

1.2.0 - 2017-03-15

Added

  • Failed sample annotation: when a job fails, the result is annotated and forwarded until a SinkNode, where we can determine the status and possibly point of failure of the Sample.
  • Commandline tool fastr trace that can inspect a workflow run and help trace errors and print debug information
  • Supported for Lmod modules environment next to the old environmentmodules
  • BaseDataType descendants are now (un)picklable (including EnumTypes)
  • Option to use {extension} field in sink_data, which differs from {ext} in that it doesn’t include a leading dot.
  • Support for Docker targets. A Docker target will execute a command inside of a specified docker container, allowing Tools to use Docker for distribution
  • Using the right and left shift operator (<< and >>) for creating links to Inputs using input << output or output >> input.
  • In the FastrInterfaces, automatic outputs can have a prefix for a flag that should be set for the output to be actually generated.
  • Fastr is now able to limit the amount of SourceJobs that are allowed to run concurrently.
  • Ability to report progress to PIM (use the pim_host field in the config)

Changed

  • Version can now also accept a format based on a date (e.g. 2017-02-17_bananas) which will be parsed the same way as 2017.02.17_bananas
  • Work on the ExecutionPlugin and the corresponding API. Has better fall-backs and a mechanism to advertise plugin capabilities.
  • The collector plugins have the input and input_parts fields merged, and the output and output_parts fields merged.

Fixed

  • In some cases the log directory was not created properly, causing an handled exception
  • A bug making the handling of Booleans incorrect for the FastrInterface, when a Boolean was given a flag would also appear when it was False
  • Serialization of the namespace of a Network was not correct
  • Check version of Fastr that creates and executes a Job against each other
  • load_gpickle helper can handle data with Enums that use to cause an AttributeError
  • Output validation of Jobs did not work correctly for automatic outputs

1.1.2 - 2016-12-22

Fixed

  • The example network in resources/networks/add_ints.json was using an old serialization format making it non-functions. Replaced by a new network file.

1.1.1 - 2016-12-22

Fixed

  • Network runs called from an interpreter (and not file) caused a crash because the network tried to report the file used. Better handling of these situations.

1.1.0 - 2016-12-08

Added

  • Namespaces for resources (tools and networks)
  • Network manager located at fastr.networklist
  • RQExecution plugin. This plugin uses python-rq to manage a job queue.
  • LinearExecution plugin. This plugin uses a background thread for execution.
  • BlockingExecution plugin. This plugin executes jobs in a blocking fashion.
  • Automatic generation of documentation for all plugins, the configuration fields and all commandline tools.

Changed

  • Provenance is updated with a network dump and used tool definitions.
  • New configuration system that uses python files
  • New plugin system that integrates with the new configuration system and enables automatic importing of plugins
  • The fastr command line tools now use an entrypoint which is located in fastr.utils.cmd. This code also dispatches the sub commands.

Removed

  • fastr.config file. This is replaced by the config.py file. Go to the docs!

Fixed

  • Adds explicit tool namespace and version to the provenance document.