Changelog¶
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning
UNRELEASED¶
Added¶
- The default python logger can now be configured from the fastr config file under key logging_config
- Support for MacroNodes, a Network can be used as a Node inside of another Network. There is should be no limitation on the internal Network used, but currently the MacroNode ignores input_groups on its inputs.
- A sync helper was added to assist in slow file synchronisation over NFS
- Source and Sink can now handle S3 URL’s
- FastrInterface can now forward errors from a subprocess if they are dumped to
stdout or stderr in a json identified by
__FASTR_ERRORS__ = []
. - A
specials.workdir
field in the location field of automatic outputs that gives the current working directory (e.g. job directory) - Added support for Torque (using pbs-drmaa library) to DRMAAExecution
- Added option to set a limit for number of jobs submitted at same time be the DRMAAExecution
- Use of the ~/.fastr/config.d directory for adding additional config files.
Any
.py
file in there will be parsed in alphabetical order. - XNATStorage IOPlugin now has a retry scheme for uploads, if an uploaded file could not be found on the server, it is retried up to 3 times.
- Added
fastr dump
command to create a zip containing all important debugging information.
Changed¶
- FilePrefix type does not have an extension anymore (avoids ugly dot in middle of filename)
- Allow expanding of link where samples have a non-uniform cardinality. This will not result in a sparse array.
- The default for
required
for the automatic outputs is nowFalse
- Removed
testtool
commandline subcommand in favour of thetest
subcommand which can test both Tools and Networks - Moved nodegroup specification into the Node for speedup
Fixed¶
- Stop Jobs from failing when a non-required, non-requested output is invalid
- Bug in boolean value parsing in the Boolean datatype
- Bug in target that caused paths not to be expanded properly in some cases
- Made sure failed sources also create a sample so the failure becomes visible and traceable.
- Bug in XNAT IOPlugin that made download from XNAT seem to fail (while getting the correct data).
Removed¶
fastr.current_network
has been removed as it was deemed to “magical” and could change things out of the sight of the user.
1.2.1 - 2017-04-04¶
Added¶
- A FastrInterface can now specify a
negate
flag on an automatic output that also has a prefix, which will negate the flag. This is useful for flag the suppress the creation of an output (e.g. no_mask). An example is given in the Toolfastr.util.AutoPrefixNegateTest
.
Changed¶
- The provenance and extra information of a Job now is not serialized in the Job, but exported to separate files next to the job file __fastr_prov__.json and __fastr_extra_job_info__.json which makes the information more accessible and reduces the memory footprint of the main process hugely as it will not read this information back anymore.
- Most execution plugin will not overwrite the executionscript stdout and stderr but rather append it. This is only relevant when continuing a run in the an existing temporary directory, but avoids loss of information.
Fixed¶
- Bug that stopped the
Link.append
function from returning the newly created link - Bugs that caused some cardinality computations of the output to fail during execution
- Bug in the job.tmpurl that caused double slashes somewhere. Some tools chocked on this when it was used for parameters.
1.2.0 - 2017-03-15¶
Added¶
- Failed sample annotation: when a job fails, the result is annotated and forwarded until a SinkNode, where we can determine the status and possibly point of failure of the Sample.
- Commandline tool
fastr trace
that can inspect a workflow run and help trace errors and print debug information - Supported for Lmod modules environment next to the old environmentmodules
- BaseDataType descendants are now (un)picklable (including EnumTypes)
- Option to use
{extension}
field in sink_data, which differs from{ext}
in that it doesn’t include a leading dot. - Support for Docker targets. A Docker target will execute a command inside of a specified docker container, allowing Tools to use Docker for distribution
- Using the right and left shift operator (<< and >>) for creating links to
Inputs
usinginput << output
oroutput >> input
. - In the FastrInterfaces, automatic outputs can have a prefix for a flag that should be set for the output to be actually generated.
- Fastr is now able to limit the amount of SourceJobs that are allowed to run concurrently.
- Ability to report progress to PIM (use the pim_host field in the config)
Changed¶
- Version can now also accept a format based on a date (e.g. 2017-02-17_bananas) which will be parsed the same way as 2017.02.17_bananas
- Work on the ExecutionPlugin and the corresponding API. Has better fall-backs and a mechanism to advertise plugin capabilities.
- The collector plugins have the
input
andinput_parts
fields merged, and theoutput
andoutput_parts
fields merged.
Fixed¶
- In some cases the log directory was not created properly, causing an handled exception
- A bug making the handling of Booleans incorrect for the FastrInterface, when a Boolean was given a flag would also appear when it was False
- Serialization of the namespace of a Network was not correct
- Check version of Fastr that creates and executes a Job against each other
- load_gpickle helper can handle data with Enums that use to cause an AttributeError
- Output validation of Jobs did not work correctly for automatic outputs
1.1.2 - 2016-12-22¶
Fixed¶
- The example network in resources/networks/add_ints.json was using an old serialization format making it non-functions. Replaced by a new network file.
1.1.1 - 2016-12-22¶
Fixed¶
- Network runs called from an interpreter (and not file) caused a crash because the network tried to report the file used. Better handling of these situations.
1.1.0 - 2016-12-08¶
Added¶
- Namespaces for resources (tools and networks)
- Network manager located at
fastr.networklist
- RQExecution plugin. This plugin uses python-rq to manage a job queue.
- LinearExecution plugin. This plugin uses a background thread for execution.
- BlockingExecution plugin. This plugin executes jobs in a blocking fashion.
- Automatic generation of documentation for all plugins, the configuration fields and all commandline tools.
Changed¶
- Provenance is updated with a network dump and used tool definitions.
- New configuration system that uses python files
- New plugin system that integrates with the new configuration system and enables automatic importing of plugins
- The fastr command line tools now use an entrypoint which is located in
fastr.utils.cmd
. This code also dispatches the sub commands.
Removed¶
- fastr.config file. This is replaced by the config.py file. Go to the docs!
Fixed¶
- Adds explicit tool namespace and version to the provenance document.