Changelog¶
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning
3.1.0 - 2019-05-02¶
Added¶
Added support for tools in YAML
fastr upgrade
can also upgrade tools from XML to YAMLfastr report
command to print an overview report of a job result
Fixed¶
Re-added support for named sub-inputs
Improved¶
Fixes in
fastr upgrade
to handle more exotic whitespace and argumentsSmall documentation fixes (especially in configuration section)
Better windows support (tested by users)
Changed¶
In ResourceLimits the default time of jobs is now None (no limit) instead of 1 hour.
By default do not log to files (we noticed fastr logs are not very often read by users and they could cause some issues with log rotation, by default logging to files is turned off, switching it back on can be done by setting
log_to_file = True
in the fastr.config
3.0.1 - 2019-03-28¶
Fixed¶
Improved implementation of
fastr upgrade
to handle newlines in thecreate_node
function properly. Also can handle old-fashioned use of fastr.toollist[…] in create_node.
3.0.0 - 2019-03-05¶
Changed¶
Now ported to Python 3.6+ (Python 2 is no longer supported!)
New public API which is not fully compatible with fastr 2.x, the changes are small. The new API will be guaranteed in next minor version upgrades and is considered to be stable.
Clear way of defining resource limits for Nodes in a Network using the ResourceLimit class.
The datatype and cardinality of inputs of a tool are now checked before the tool is to be executed as an extra safety.
Dimensions are drawn by default in network.draw
The api now accepts types other than Output, list, tuple when creating a link. When a single value is given it is assumedly a constant from the network definition.
Drawing a network will not create temporary .dot files anymore
Sinkdata can be a string, it that case it will be the same string for all sink nodes so a {node} substitution should be used in the template
Make the xnat ioplugin use xnat+http:// and xnat+https:// url schemes in favour of xnat:// with ?insecure=… (old behaviour will also work for now)
Complete rewrite of PIM plugin (PIMReporter) making use of the new Reporter plugin infrastructure. It also caches all communication with PIM to be resilient against connection interruptions.
Added¶
fastr upgrade
command to automatically upgrade a network creation file from fastr 2.x to fastr 3.x API.http(s) IOPlugin for downloading files via http(s)
network.draw now has a flag to hide the unconnected inputs and output of a node. The unconnected inputs/outputs are hidden by default.
Reporting plugins, Fastr now exposes a number of message hooks which can be listened to by Reporter plugins.
Fixed¶
Fixed some bugs with drmaa communication (more safeties added)
Fixed a bug in the MacroNode update function which could cause networks with MacroNodes to be invalid
The margins and font size of the network.draw graph rendering are set a bit wider and smaller (resp.) to avoid excessive text overflow.
Fixed bug in provenance which did not properly chain the provenance of subsequent jobs.
2.1.2 - 2018-10-24¶
Added¶
Allow overriding the timestamp of the network execution
Changed¶
Updated PIM publisher to support the new PIM API v2
Updated XNAT IOPlugin to not crash when creating a resource failed because another process already did that (race condition)
Make default resource limits for DRMAA configurable
Add stack trace to FastrExceptions
2.1.1 - 2018-06-29¶
Fixed¶
Fixed some issues with the type estimation of outputs of Jobs and update validation functions of NIFTI files
2.1.0 - 2018-04-13¶
Added¶
SLURM execution plugin based on
sbatch
,scancel
,scontrol
andsqueue
. The plugin supports job dependencies and cancellation.Support for running tools in Docker containers using a DockerTarget
Support for running tools in Singularity containers using a SingularityTarget
Support for datatypes with multiple extensions (e.g. .tif and .tiff) by setting the extension to a tuple of options. The first extension is leading for deciding filenames in a sink.
Changed¶
Source jobs now also validate the output (and do not only rely on the stderr of the tool)
Added preferred_types attribute to TypeGroups that gives the order of preference of members, alternatively the order of _members is used (this should be given as tuple or list to be meaningful)
In the config.py you can now access the USER_DIR and SYSTEM_DIR variables for use in setting other variables. These are only read and changing them will only change subsequent config reads but not the main config values.
checksum for nii.gz now takes the md5 checksum of the decompressed data
Serialization of MacroNodes now should function properly
Fixed¶
BUG in XNAT plugin that made it impossible to download data from scans without an empty type string
BUG where the order of OrderedDict in a source was not preserved
BUG where newer Werkzeug version requires the web port to be an integer
2.0.1 - 2017-10-19¶
Fix a bug in the validation of FilePrefix datatypes
2.0.0 - 2017-09-28¶
Added¶
The default python logger can now be configured from the fastr config file under key logging_config
Support for MacroNodes, a Network can be used as a Node inside of another Network. There is should be no limitation on the internal Network used, but currently the MacroNode ignores input_groups on its inputs.
A sync helper was added to assist in slow file synchronisation over NFS
Source and Sink can now handle S3 URL’s
FastrInterface can now forward errors from a subprocess if they are dumped to stdout or stderr in a json identified by
__FASTR_ERRORS__ = []
.A
specials.workdir
field in the location field of automatic outputs that gives the current working directory (e.g. job directory)Added support for Torque (using pbs-drmaa library) to DRMAAExecution
Added option to set a limit for number of jobs submitted at same time be the DRMAAExecution
Use of the ~/.fastr/config.d directory for adding additional config files. Any
.py
file in there will be parsed in alphabetical order.XNATStorage IOPlugin now has a retry scheme for uploads, if an uploaded file could not be found on the server, it is retried up to 3 times.
Added
fastr dump
command to create a zip containing all important debugging information.
Changed¶
FilePrefix type does not have an extension anymore (avoids ugly dot in middle of filename)
Allow expanding of link where samples have a non-uniform cardinality. This will not result in a sparse array.
The default for
required
for the automatic outputs is nowFalse
Removed
testtool
commandline subcommand in favour of thetest
subcommand which can test both Tools and NetworksMoved nodegroup specification into the Node for speedup
Fixed¶
Stop Jobs from failing when a non-required, non-requested output is invalid
Bug in boolean value parsing in the Boolean datatype
Bug in target that caused paths not to be expanded properly in some cases
Made sure failed sources also create a sample so the failure becomes visible and traceable.
Bug in XNAT IOPlugin that made download from XNAT seem to fail (while getting the correct data).
Removed¶
fastr.current_network
has been removed as it was deemed to “magical” and could change things out of the sight of the user.
1.2.1 - 2017-04-04¶
Added¶
A FastrInterface can now specify a
negate
flag on an automatic output that also has a prefix, which will negate the flag. This is useful for flag the suppress the creation of an output (e.g. no_mask). An example is given in the Toolfastr.util.AutoPrefixNegateTest
.
Changed¶
The provenance and extra information of a Job now is not serialized in the Job, but exported to separate files next to the job file __fastr_prov__.json and __fastr_extra_job_info__.json which makes the information more accessible and reduces the memory footprint of the main process hugely as it will not read this information back anymore.
Most execution plugin will not overwrite the executionscript stdout and stderr but rather append it. This is only relevant when continuing a run in the an existing temporary directory, but avoids loss of information.
Fixed¶
Bug that stopped the
Link.append
function from returning the newly created linkBugs that caused some cardinality computations of the output to fail during execution
Bug in the job.tmpurl that caused double slashes somewhere. Some tools chocked on this when it was used for parameters.
1.2.0 - 2017-03-15¶
Added¶
Failed sample annotation: when a job fails, the result is annotated and forwarded until a SinkNode, where we can determine the status and possibly point of failure of the Sample.
Commandline tool
fastr trace
that can inspect a workflow run and help trace errors and print debug informationSupported for Lmod modules environment next to the old environmentmodules
BaseDataType descendants are now (un)picklable (including EnumTypes)
Option to use
{extension}
field in sink_data, which differs from{ext}
in that it doesn’t include a leading dot.Support for Docker targets. A Docker target will execute a command inside of a specified docker container, allowing Tools to use Docker for distribution
Using the right and left shift operator (<< and >>) for creating links to
Inputs
usinginput << output
oroutput >> input
.In the FastrInterfaces, automatic outputs can have a prefix for a flag that should be set for the output to be actually generated.
Fastr is now able to limit the amount of SourceJobs that are allowed to run concurrently.
Ability to report progress to PIM (use the pim_host field in the config)
Changed¶
Version can now also accept a format based on a date (e.g. 2017-02-17_bananas) which will be parsed the same way as 2017.02.17_bananas
Work on the ExecutionPlugin and the corresponding API. Has better fall-backs and a mechanism to advertise plugin capabilities.
The collector plugins have the
input
andinput_parts
fields merged, and theoutput
andoutput_parts
fields merged.
Fixed¶
In some cases the log directory was not created properly, causing an handled exception
A bug making the handling of Booleans incorrect for the FastrInterface, when a Boolean was given a flag would also appear when it was False
Serialization of the namespace of a Network was not correct
Check version of Fastr that creates and executes a Job against each other
load_gpickle helper can handle data with Enums that use to cause an AttributeError
Output validation of Jobs did not work correctly for automatic outputs
1.1.2 - 2016-12-22¶
Fixed¶
The example network in resources/networks/add_ints.json was using an old serialization format making it non-functions. Replaced by a new network file.
1.1.1 - 2016-12-22¶
Fixed¶
Network runs called from an interpreter (and not file) caused a crash because the network tried to report the file used. Better handling of these situations.
1.1.0 - 2016-12-08¶
Added¶
Namespaces for resources (tools and networks)
Network manager located at
fastr.networklist
RQExecution plugin. This plugin uses python-rq to manage a job queue.
LinearExecution plugin. This plugin uses a background thread for execution.
BlockingExecution plugin. This plugin executes jobs in a blocking fashion.
Automatic generation of documentation for all plugins, the configuration fields and all commandline tools.
Changed¶
Provenance is updated with a network dump and used tool definitions.
New configuration system that uses python files
New plugin system that integrates with the new configuration system and enables automatic importing of plugins
The fastr command line tools now use an entrypoint which is located in
fastr.utils.cmd
. This code also dispatches the sub commands.
Removed¶
fastr.config file. This is replaced by the config.py file. Go to the docs!
Fixed¶
Adds explicit tool namespace and version to the provenance document.