ioplugins Package

ioplugins Package

commaseperatedvaluefile Module

This module contains the CommaSeperatedValueFile plugin for fastr

class fastr.resources.plugins.ioplugins.commaseperatedvaluefile.CommaSeperatedValueFile

Bases: fastr.core.ioplugin.IOPlugin

The CommaSeperatedValueFile an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs.

The csv:// URL is a vfs:// URL with a number of query variables available. The URL mount and path should point to a valid CSV file. The query variable then specify what column(s) of the file should be used.

The following variable can be set in the query:

variable	usage
value	the column containing the value of interest, can be int for index or string for key
id	the column containing the sample id (optional)
header	indicates if the first row is considered the header, can be true or false (optional)
delimiter	the delimiter used in the csv file (optional)
quote	the quote character used in the csv file (optional)
reformat	a reformatting string so that value = reformat.format(value) (used before relative_path)
relative_path	indicates the entries are relative paths (for files), can be true or false (optional)

The header is by default false if the neither the value and id are set as a string. If either of these are a string, the header is required to define the column names and it automatically is assumed true

The delimiter and quota characters of the file should be detected automatically using the Sniffer, but can be forced by setting them in the URL.

Example of valid csv URLs:

# Use the first column in the file (no header row assumed)
csv://mount/some/dir/file.csv?value=0

# Use the images column in the file (first row is assumed header row)
csv://mount/some/dir/file.csv?value=images

# Use the segmentations column in the file (first row is assumed header row)
# and use the id column as the sample id
csv://mount/some/dir/file.csv?value=segmentations&id=id

# Use the first column as the id and the second column as the value
# and skip the first row (considered the header)
csv://mount/some/dir/file.csv?value=1&id=0&header=true

# Use the first column and force the delimiter to be a comma
csv://mount/some/dir/file.csv?value=0&delimiter=,

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.commaseperatedvaluefile'

expand_url(url)

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand Returns:the resulting url(s), a tuple if multiple, otherwise a str Return type:str or tuple of str

scheme = 'csv'

filesystem Module

This module contains the FileSystem plugin for fastr

class fastr.resources.plugins.ioplugins.filesystem.FileSystem

Bases: fastr.core.ioplugin.IOPlugin

The FileSystem plugin is create to handle file:// type or URLs. This is generally not a good practice, as this is not portable over between machines. However, for test purposes it might be useful.

The URL scheme is rather simple: file://host/path (see wikipedia for details)

We do not make use of the host part and at the moment only support localhost (just leave the host empty) leading to file:/// URLs.

Warning

This plugin ignores the hostname in the URL and does only accept driver letters on Windows in the form c:/

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.filesystem'

fetch_url(inurl, outpath)

Fetch the files from the file.

Parameters:

• inurl – url to the item in the data store, starts with file://
• outpath – path where to store the fetch data locally

fetch_value(inurl)

Fetch a value from an external file file.

Parameters:inurl – url of the value to read Returns:the fetched value

path_to_url(path, mountpoint=None)

Construct an url from a given mount point and a relative path to the mount point.

put_url(inpath, outurl)

Put the files to the external data store.

Parameters:

• inpath – path of the local data
• outurl – url to where to store the data, starts with file://

put_value(value, outurl)

Put the value in the external data store.

Parameters:

• value – value to store
• outurl – url to where to store the data, starts with file://

scheme = 'file'

url_to_path(url)

Get the path to a file from a url. Currently supports the file:// scheme

Examples:

>>> 'file:///d:/data/project/file.ext'
'd:\data\project\file.ext'

Warning

file:// will not function cross platform and is mainly for testing

httpplugin Module

class fastr.resources.plugins.ioplugins.httpplugin.HTTPPlugin

Bases: fastr.core.ioplugin.IOPlugin

Warning

This Plugin is still under development and has not been tested at all. example url: https://server.io/path/to/resource

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.httpplugin'

fetch_url(inurl, outpath)

Download file from server.

Parameters:

• inurl – url to the file.
• outpath – path to store file

scheme = ('https', 'http')

null Module

This module contains the Null plugin for fastr

class fastr.resources.plugins.ioplugins.null.Null

Bases: fastr.core.ioplugin.IOPlugin

The Null plugin is create to handle null:// type or URLs. These URLs are indicating the sink should not do anything. The data is not written to anywhere. Besides the scheme, the rest of the URL is ignored.

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.null'

put_url(inpath, outurl)

Put the files to the external data store.

Parameters:

• inpath – path of the local data
• outurl – url to where to store the data, starts with file://

put_value(value, outurl)

Put the value in the external data store.

Parameters:

• value – value to store
• outurl – url to where to store the data, starts with file://

scheme = 'null'

reference Module

This module contains the Null plugin for fastr

class fastr.resources.plugins.ioplugins.reference.Reference

Bases: fastr.core.ioplugin.IOPlugin

The Reference plugin is create to handle ref:// type or URLs. These URLs are to make the sink just write a simple reference file to the data. The reference file contains the DataType and the value so the result can be reconstructed. It for files just leaves the data on disk by reference. This plugin is not useful for production, but is used for testing purposes.

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.reference'

push_sink_data(value, outurl, datatype=None)

Write out the sink data from the inpath to the outurl.

Parameters:

• value (str) – the path of the data to be pushed
• outurl (str) – the url to write the data to
• datatype (DataType) – the datatype of the data, used for determining the total contents of the transfer

Returns:

None

scheme = 'ref'

s3filesystem Module

This module contains the S3Storage plugin for fastr

class fastr.resources.plugins.ioplugins.s3filesystem.S3Filesystem

Bases: fastr.core.ioplugin.IOPlugin

Warning

As this IOPlugin is under development, it has not been thoroughly tested.

example url: s3://bucket.server/path/to/resource

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.s3filesystem'

cleanup()

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

expand_url(url)

Expand an S3 URL. This allows a source to collect multiple samples from a single url.

Parameters:url (str) – url to expand Returns:the resulting url(s), a tuple if multiple, otherwise a str Return type:str or tuple of str

fetch_url(inurl, outpath)

Get the file(s) or values from s3.

Parameters:

• inurl – url to the item in the data store
• outpath – path where to store the fetch data locally

fetch_value(inurl)

Fetch a value from S3

Parameters:inurl – url of the value to read Returns:the fetched value

put_url(inpath, outurl)

Upload the files to the S3 storage

Parameters:

• inpath – path to the local data
• outurl – url to where to store the data in the external data store.

put_value(value, outurl)

Put the value in S3

Parameters:

• value – value to store
• outurl – url to where to store the data, starts with file://

scheme = ('s3', 's3list')

classmethod test()

Test the plugin, default behaviour is just to instantiate the plugin

virtualfilesystem Module

This module contains the virtual file system code. This is internally used object as used as base class for the IOPlugin.

class fastr.resources.plugins.ioplugins.virtualfilesystem.VirtualFileSystem

Bases: fastr.core.vfs.VirtualFileSystem, fastr.core.ioplugin.IOPlugin

The virtual file system class. This is an IOPlugin, but also heavily used internally in fastr for working with directories. The VirtualFileSystem uses the vfs:// url scheme.

A typical virtual filesystem url is formatted as vfs://mountpoint/relative/dir/from/mount.ext

Where the mountpoint is defined in the Config file. A list of the currently known mountpoints can be found in the fastr.config object

>>> fastr.config.mounts
{'example_data': '/home/username/fastr-feature-documentation/fastr/fastr/examples/data',
 'home': '/home/username/',
 'tmp': '/home/username/FastrTemp'}

This shows that a url with the mount home such as vfs://home/tempdir/testfile.txt would be translated into /home/username/tempdir/testfile.txt.

There are a few default mount points defined by Fastr (that can be changed via the config file).

mountpoint	default location
home	the users home directory (expanduser('~/'))
tmp	the fastr temprorary dir, defaults to tempfile.gettempdir()
example_data	the fastr example data directory, defaults $FASTRDIR/example/data

__abstractmethods__ = frozenset()

__module__ = 'fastr.resources.plugins.ioplugins.virtualfilesystem'

scheme = 'vfs'

virtualfilesystemregularexpression Module

This module contains the VirtualFileSystemRegularExpression plugin for fastr

class fastr.resources.plugins.ioplugins.virtualfilesystemregularexpression.VirtualFileSystemRegularExpression

Bases: fastr.core.ioplugin.IOPlugin

The VirtualFileSystemValueList an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs.

A vfsregex:// URL is a vfs URL that can contain regular expressions on every level of the path. The regular expressions follow the re module definitions.

An example of a valid URLs would be:

vfsregex://tmp/network_dir/.*/.*/__fastr_result__.pickle.gz
vfsregex://tmp/network_dir/nodeX/(?P<id>.*)/__fastr_result__.pickle.gz

The first URL would result in all the __fastr_result__.pickle.gz in the working directory of a Network. The second URL would only result in the file for a specific node (nodeX), but by adding the named group id using (?P<id>.*) the sample id of the data is automatically set to that group (see Regular Expression Syntax under the special characters for more info on named groups in regular expression).

Concretely if we would have a directory vfs://mount/somedir containing:

image_1/Image.nii
image_2/image.nii
image_3/anotherimage.nii
image_5/inconsistentnamingftw.nii

we could match these files using vfsregex://mount/somedir/(?P<id>image_\d+)/.*\.nii which would result in the following source data after expanding the URL:

{'image_1': 'vfs://mount/somedir/image_1/Image.nii',
 'image_2': 'vfs://mount/somedir/image_2/image.nii',
 'image_3': 'vfs://mount/somedir/image_3/anotherimage.nii',
 'image_5': 'vfs://mount/somedir/image_5/inconsistentnamingftw.nii'}

Showing the power of this regular expression filtering. Also it shows how the ID group from the URL can be used to have sensible sample ids.

Warning

due to the nature of regexp on multiple levels, this method can be slow when having many matches on the lower level of the path (because the tree of potential matches grows) or when directories that are parts of the path are very large.

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.virtualfilesystemregularexpression'

expand_url(url)

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand Returns:the resulting url(s), a tuple if multiple, otherwise a str Return type:str or tuple of str

scheme = 'vfsregex'

virtualfilesystemvaluelist Module

This module contains the VirtualFileSystemValueList plugin for fastr

class fastr.resources.plugins.ioplugins.virtualfilesystemvaluelist.VirtualFileSystemValueList

Bases: fastr.core.ioplugin.IOPlugin

The VirtualFileSystemValueList an expand-only type of IOPlugin. No URLs can actually be fetched, but it can expand a single URL into a larger amount of URLs. A vfslist:// URL basically is a url that points to a file using vfs. This file then contains a number lines each containing another URL.

If the contents of a file vfs://mount/some/path/contents would be:

vfs://mount/some/path/file1.txt
vfs://mount/some/path/file2.txt
vfs://mount/some/path/file3.txt
vfs://mount/some/path/file4.txt

Then using the URL vfslist://mount/some/path/contents as source data would result in the four files being pulled.

Note

The URLs in a vfslist file do not have to use the vfs scheme, but can use any scheme known to the Fastr system.

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.virtualfilesystemvaluelist'

expand_url(url)

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand Returns:the resulting url(s), a tuple if multiple, otherwise a str Return type:str or tuple of str

scheme = 'vfslist'

xnatstorage Module

This module contains the XNATStorage plugin for fastr

class fastr.resources.plugins.ioplugins.xnatstorage.XNATStorage

Bases: fastr.core.ioplugin.IOPlugin

Warning

As this IOPlugin is under development, it has not been thoroughly tested.

The XNATStorage plugin is an IOPlugin that can download data from and upload data to an XNAT server. It uses its own xnat:// URL scheme. This is a scheme specific for this plugin and though it looks somewhat like the XNAT rest interface, a different type or URL.

Data resources can be access directly by a data url:

xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/experiment001/scans/T1/resources/DICOM
xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/*_BRAIN/scans/T1/resources/DICOM

In the second URL you can see a wildcard being used. This is possible at long as it resolves to exactly one item.

The id query element will change the field from the default experiment to subject and the label query element sets the use of the label as the fastr id (instead of the XNAT id) to True (the default is False)

To disable https transport and use http instead the query string can be modified to add insecure=true. This will make the plugin send requests over http:

xnat://xnat.example.com/data/archive/projects/sandbox/subjects/subject001/experiments/*_BRAIN/scans/T1/resources/DICOM?insecure=true

For sinks it is import to know where to save the data. Sometimes you want to save data in a new assessor/resource and it needs to be created. To allow the Fastr sink to create an object in XNAT, you have to supply the type as a query parameter:

xnat://xnat.bmia.nl/data/archive/projects/sandbox/subjects/S01/experiments/_BRAIN/assessors/test_assessor/resources/IMAGE/files/image.nii.gz?resource_type=xnat:resourceCatalog&assessor_type=xnat:qcAssessmentData

Valid options are: subject_type, experiment_type, assessor_type, scan_type, and resource_type.

If you want to do a search where multiple resources are returned, it is possible to use a search url:

xnat://xnat.example.com/search?projects=sandbox&subjects=subject[0-9][0-9][0-9]&experiments=*_BRAIN&scans=T1&resources=DICOM

This will return all DICOMs for the T1 scans for experiments that end with _BRAIN that belong to a subjectXXX where XXX is a 3 digit number. By default the ID for the samples will be the experiment XNAT ID (e.g. XNAT_E00123). The wildcards that can be the used are the same UNIX shell-style wildcards as provided by the module fnmatch.

It is possible to change the id to a different fields id or label. Valid fields are project, subject, experiment, scan, and resource:

xnat://xnat.example.com/search?projects=sandbox&subjects=subject[0-9][0-9][0-9]&experiments=*_BRAIN&scans=T1&resources=DICOM&id=subject&label=true

The following variables can be set in the search query:

variable	default	usage
projects	*	The project(s) to select, can contain wildcards (see fnmatch)
subjects	*	The subject(s) to select, can contain wildcards (see fnmatch)
experiments	*	The experiment(s) to select, can contain wildcards (see fnmatch)
scans	*	The scan(s) to select, can contain wildcards (see fnmatch)
resources	*	The resource(s) to select, can contain wildcards (see fnmatch)
id	experiment	What field to use a the id, can be: project, subject, experiment, scan, or resource
label	false	Indicate the XNAT label should be used as fastr id, options true or false
insecure	false	Change the url scheme to be used to http instead of https
verify	true	(Dis)able the verification of SSL certificates
regex	false	Change search to use regex re.match() instead of fnmatch for matching
overwrite	false	Tell XNAT to overwrite existing files if a file with the name is already present

For storing credentials the .netrc file can be used. This is a common way to store credentials on UNIX systems. It is required that the file is only accessible by the owner only or a NetrcParseError will be raised. A netrc file is really easy to create, as its entries look like:

machine xnat.example.com
        login username
        password secret123

See the netrc module or the GNU inet utils website for more information about the .netrc file.

Note

On windows the location of the netrc file is assumed to be os.path.expanduser('~/_netrc'). The leading underscore is because windows does not like filename starting with a dot.

Note

For scan the label will be the scan type (this is initially the same as the series description, but can be updated manually or the XNAT scan type cleanup).

Warning

labels in XNAT are not guaranteed to be unique, so be careful when using them as the sample ID.

For background on XNAT, see the XNAT API DIRECTORY for the REST API of XNAT.

__abstractmethods__ = frozenset()

__init__()

Initialization for the IOPlugin

Returns:newly created IOPlugin

__module__ = 'fastr.resources.plugins.ioplugins.xnatstorage'

cleanup()

(abstract) Clean up the IOPlugin. This is to do things like closing files or connections. Will be called when the plugin is no longer required.

connect(server, path='', insecure=False, verify=True)

expand_url(url)

(abstract) Expand an URL. This allows a source to collect multiple samples from a single url. The URL will have a wildcard or point to something with info and multiple urls will be returned.

Parameters:url (str) – url to expand Returns:the resulting url(s), a tuple if multiple, otherwise a str Return type:str or tuple of str

fetch_url(inurl, outpath)

Get the file(s) or values from XNAT.

Parameters:

• inurl – url to the item in the data store
• outpath – path where to store the fetch data locally

parse_uri(url)

put_url(inpath, outurl)

Upload the files to the XNAT storage

Parameters:

• inpath – path to the local data
• outurl – url to where to store the data in the external data store.

scheme = ('xnat', 'xnat+http', 'xnat+https')

server

static upload(resource, in_path, location, retries=3, overwrite=False)

xnat

Subpackages

• test Package
  • test_virtualfilesystem Module