helpers Package

helpers Package

fastr.helpers.config = # [bool] Flag to enable/disable debugging debug = False # [str] Directory containing the fastr examples examplesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples" # [str] The default execution plugin to use execution_plugin = "ProcessPoolExecution" # [str] Execution script location executionscript = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/execution/executionscript.py" # [str] Redis url e.g. redis://localhost:6379 filesynchelper_url = "" # [str] The level of cleanup required, options: all, no_cleanup, non_failed job_cleanup_level = "no_cleanup" # [str] Directory where the fastr logs will be placed logdir = "/home/docs/.fastr/logs" # [dict] Python logger config logging_config = {} # [int] The log level to use (as int), INFO is 20, WARNING is 30, etc loglevel = 20 # [str] Type of logging to use logtype = "default" # [dict] A dictionary containing all mount points in the VFS system mounts = { "tmp": "/tmp", "example_data": "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples/data", "home": "/home/docs/" } # [list] Directories to scan for networks networks_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/networks" ] # [list] Directories to scan for plugins plugins_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/plugins" ] # [list] A list indicating the order of the preferred types to use. First item is most preferred. preferred_types = [] # [list] A list of modules in the environmnet modules that are protected against unloading protected_modules = [] # [list] The reporting plugins to use, is a list of all plugins to be activated reporting_plugins = [ "SimpleReport" ] # [str] Directory containing the fastr system resources resourcesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources" # [str] Directory containing the fastr data schemas schemadir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/schemas" # [int] The number of source jobs allowed to run concurrently source_job_limit = 0 # [str] Fastr installation directory systemdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr" # [list] Directories to scan for tools tools_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/tools" ] # [list] Directories to scan for datatypes types_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/datatypes" ] # [str] Fastr user configuration directory userdir = "/home/docs/.fastr" # [bool] Warning users on import if this is not a production version of fastr warn_develop = True # [str] The hostname to expose the web app for web_hostname = "localhost" # [int] The port to expose the web app on web_port = 5000 # [str] The secret key to use for the flask web app web_secret_key = "VERYSECRETKEY!" # [int] Number of workers to use in a process pool process_pool_worker_number = 3 # [int] The interval in which the job checker will startto check for stale jobs slurm_job_check_interval = 30 # [str] The slurm partition to use slurm_partition = "" # [str] The PIM host to report to pim_host = "" # [float] The interval in which to send jobs to PIM pim_update_interval = 2.5 # [int] Maximum number of jobs that can be send to PIM in a single interval pim_batch_size = 100 # [bool] Setup PIM debug mode to send stdout stderr on job succes pim_debug = False

Configuration of the fastr system

checksum Module

This module contains a number of functions for checksumming files and objects

fastr.helpers.checksum.checksum(filepath, algorithm='md5', hasher=None, chunksize=32768)[source]

Generate the checksum of a file

Parameters:
  • filepath (str, list) – path of the file(s) to checksum
  • algorithm (str) – the algorithm to use
  • hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)
Returns:

the checksum

Return type:

str

fastr.helpers.checksum.checksum_directory(directory, algorithm='md5', hasher=None)[source]

Generate the checksum of an entire directory

Parameters:
  • directory (str) – path of the file(s) to checksum
  • algorithm (str) – the algorithm to use
  • hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)
Returns:

the checksum

Return type:

str

fastr.helpers.checksum.hashsum(objects, hasher=None)[source]

Generate the md5 checksum of (a) python object(s)

Parameters:
  • objects – the objects to hash
  • hasher – the hasher to use as a base
Returns:

the hash generated

Return type:

str

fastr.helpers.checksum.md5_checksum(filepath)[source]

Generate the md5 checksum of a file

Parameters:filepath (str, list) – path of the file(s) to checksum
Returns:the checksum
Return type:str
fastr.helpers.checksum.sha1_checksum(filepath)[source]

Generate the sha1 checksum of a file

Parameters:filepath (str, list) – path of the file(s) to checksum
Returns:the checksum
Return type:str

classproperty Module

Module containing the code to create class properties.

class fastr.helpers.classproperty.ClassPropertyDescriptor(fget)[source]

Bases: object

A descriptor that can act like a property for a class.

__dict__ = mappingproxy({'__module__': 'fastr.helpers.classproperty', '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function ClassPropertyDescriptor.__init__>, '__get__': <function ClassPropertyDescriptor.__get__>, '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>})
__get__(obj, cls=None)[source]
__init__(fget)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.classproperty'
__weakref__

list of weak references to the object (if defined)

fastr.helpers.classproperty.classproperty(func)[source]

Decorator to create a “class property”

Parameters:func – the function to wrap
Returns:a class property
Return type:ClassPropertyDescriptor

clear_pycs Module

A small tool to wipe all .pyc files from fastr

fastr.helpers.clear_pycs.dir_list(directory)[source]

Find all .pyc files

Parameters:directory (str) – directory to search
Returns:all .pyc files
Return type:list
fastr.helpers.clear_pycs.main()[source]

Main entry poitn

configmanager Module

This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.

class fastr.helpers.configmanager.Config(*configfiles)[source]

Bases: object

Class contain the fastr configuration

DEFAULT_FIELDS = {'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs/'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr', 'Fastr installation directory'), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'web_port': (<class 'int'>, 5000, 'The port to expose the web app on'), 'web_secret_key': (<class 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app')}
__dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': '\n Class contain the fastr configuration\n ', 'DEFAULT_FIELDS': {'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr', 'Fastr installation directory'), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/stable/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs/'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'web_port': (<class 'int'>, 5000, 'The port to expose the web app on'), 'web_secret_key': (<class 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379')}, '__init__': <function Config.__init__>, 'register_fields': <function Config.register_fields>, 'get_field': <function Config.get_field>, 'set_field': <function Config.set_field>, '_create_field_properties': <classmethod object>, '_field_property': <staticmethod object>, '__repr__': <function Config.__repr__>, 'read_config': <function Config.read_config>, 'read_config_string': <function Config.read_config_string>, 'web_url': <function Config.web_url>, '_update_logging': <function Config._update_logging>, '_deep_update': <function Config._deep_update>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'debug': <property object>, 'examplesdir': <property object>, 'execution_plugin': <property object>, 'executionscript': <property object>, 'filesynchelper_url': <property object>, 'job_cleanup_level': <property object>, 'logdir': <property object>, 'logging_config': <property object>, 'loglevel': <property object>, 'logtype': <property object>, 'mounts': <property object>, 'networks_path': <property object>, 'plugins_path': <property object>, 'preferred_types': <property object>, 'protected_modules': <property object>, 'reporting_plugins': <property object>, 'resourcesdir': <property object>, 'schemadir': <property object>, 'source_job_limit': <property object>, 'systemdir': <property object>, 'tools_path': <property object>, 'types_path': <property object>, 'userdir': <property object>, 'warn_develop': <property object>, 'web_hostname': <property object>, 'web_port': <property object>, 'web_secret_key': <property object>, 'process_pool_worker_number': <property object>, 'slurm_job_check_interval': <property object>, 'slurm_partition': <property object>, 'pim_host': <property object>, 'pim_update_interval': <property object>, 'pim_batch_size': <property object>, 'pim_debug': <property object>})
__init__(*configfiles)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.configmanager'
__repr__()[source]

Return repr(self).

__weakref__

list of weak references to the object (if defined)

debug
examplesdir
execution_plugin
executionscript
filesynchelper_url
get_field(item)[source]
job_cleanup_level
logdir
logging_config
loglevel
logtype
mounts
networks_path
pim_batch_size
pim_debug
pim_host
pim_update_interval
plugins_path
preferred_types
process_pool_worker_number
protected_modules
read_config(filename)[source]

Read a configuration and update the configuration object accordingly

Parameters:filename – the configuration file to read
read_config_files = None

Trace of the config files read by this object

read_config_string(value)[source]
register_fields(fields_spec)[source]

Register extra fields to the configuration manager.

reporting_plugins
resourcesdir
schemadir
set_field(item, value)[source]
slurm_job_check_interval
slurm_partition
source_job_limit
systemdir
tools_path
types_path
userdir
warn_develop
web_hostname
web_port
web_secret_key
web_url()[source]

Construct a fqdn from the web[‘hostname’] and web[‘port’] settings. :return: FQDN :rtype: str

class fastr.helpers.configmanager.EmptyDefault(data=None)[source]

Bases: object

Empty defaultdict.

__add__(right)[source]
__delitem__(key)[source]
__dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': ' Empty defaultdict. ', '__init__': <function EmptyDefault.__init__>, '__iadd__': <function EmptyDefault.__iadd__>, '__add__': <function EmptyDefault.__add__>, '__radd__': <function EmptyDefault.__radd__>, 'append': <function EmptyDefault.append>, 'prepend': <function EmptyDefault.prepend>, 'extend': <function EmptyDefault.extend>, 'update': <function EmptyDefault.update>, 'merge_default': <function EmptyDefault.merge_default>, '__getitem__': <function EmptyDefault.__getitem__>, '__setitem__': <function EmptyDefault.__setitem__>, '__delitem__': <function EmptyDefault.__delitem__>, 'aslist': <function EmptyDefault.aslist>, 'asdict': <function EmptyDefault.asdict>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>})
__getitem__(item)[source]
__iadd__(right)[source]
__init__(data=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.configmanager'
__radd__(other)[source]
__setitem__(key, value)[source]
__weakref__

list of weak references to the object (if defined)

append(value)[source]
asdict()[source]
aslist()[source]
extend(other)[source]
merge_default(field_spec)[source]

Merge the default into this EmptyDefault given the field spec :param field_spec: Field specification :return: Merged value

prepend(value)[source]
update(other)[source]

events Module

class fastr.helpers.events.EventType[source]

Bases: enum.Enum

An enumeration.

__module__ = 'fastr.helpers.events'
job_updated = 'job_updated'
log_record_emitted = 'log_record_emitted'
run_finished = 'run_finished'
run_started = 'run_started'
class fastr.helpers.events.FastrLogEventHandler(level=0)[source]

Bases: logging.Handler

Logging handler that sends the log records into the event system

__module__ = 'fastr.helpers.events'
emit(record)[source]

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

fastr.helpers.events.emit_event(event_type, data)[source]

Emit an event to all listeners :type event_type: EventType :param event_type: The type of event to emit :param data: The data object to send along

fastr.helpers.events.register_listener(event_type, function)[source]

Register a listeners to a specific event type

Parameters:
  • event_type (EventType) – The EventType to listen on
  • function (Callable[[object], None]) – The callable that will be called on each event
fastr.helpers.events.remove_listener(event_type, function)[source]

Remove a listeren from a type of event

Parameters:
  • event_type (EventType) – The event type to remove the listeners from
  • function (Callable[[object], None]) – The function to remove

filesynchelper Module

Some helper functions that aid with NFS file sync issues.

class fastr.helpers.filesynchelper.FileSyncHelper[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.helpers.filesynchelper', '_namespace': 'filesynchelper', '_redis': None, '__init__': <function FileSyncHelper.__init__>, 'job_finished': <function FileSyncHelper.job_finished>, 'wait_for_job': <function FileSyncHelper.wait_for_job>, 'wait_for_pickle': <function FileSyncHelper.wait_for_pickle>, 'store': <function FileSyncHelper.store>, 'load': <function FileSyncHelper.load>, '_generate_key_for_string': <function FileSyncHelper._generate_key_for_string>, '_generate_hash_from_string': <function FileSyncHelper._generate_hash_from_string>, 'make_file_promise': <function FileSyncHelper.make_file_promise>, 'has_file_promise': <function FileSyncHelper.has_file_promise>, 'wait_for_vfs_url': <function FileSyncHelper.wait_for_vfs_url>, 'wait_for_file': <function FileSyncHelper.wait_for_file>, '_get_suburl_hashes': <function FileSyncHelper._get_suburl_hashes>, '_glob_dir': <function FileSyncHelper._glob_dir>, '_wait_for_file_and_suburls': <function FileSyncHelper._wait_for_file_and_suburls>, '__dict__': <attribute '__dict__' of 'FileSyncHelper' objects>, '__weakref__': <attribute '__weakref__' of 'FileSyncHelper' objects>, '__doc__': None})
__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.filesynchelper'
__weakref__

list of weak references to the object (if defined)

has_file_promise(url)[source]
job_finished(jobfile)[source]
load(url)[source]
make_file_promise(url)[source]
store(url, data)[source]
wait_for_file(path, timeout=300)[source]
wait_for_job(jobfile)[source]
wait_for_pickle(url, timeout=300)[source]
wait_for_vfs_url(vfs_url, timeout=300)[source]
fastr.helpers.filesynchelper.filesynchelper_enabled()[source]

iohelpers Module

fastr.helpers.iohelpers.load_gpickle(path, retry_scheme=None)[source]
fastr.helpers.iohelpers.load_json(path)[source]
fastr.helpers.iohelpers.save_gpickle(path, data)[source]
fastr.helpers.iohelpers.save_json(path, data, indent=2)[source]

jsonschemaparser Module

The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.

class fastr.helpers.jsonschemaparser.FastrRefResolver(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]

Bases: jsonschema.validators.RefResolver

Adapted version of the RefResolver for handling inter-file references more to our liking

__init__(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]

Create a new FastrRefResolver

Parameters:
  • base_uri (str) – URI of the referring document
  • referrer – the actual referring document
  • store (dict) – a mapping from URIs to documents to cache
  • cache_remote (bool) – whether remote refs should be cached after first resolution
  • handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
__module__ = 'fastr.helpers.jsonschemaparser'
classmethod from_schema(schema, *args, **kwargs)[source]

Instantiate a RefResolver based on a schema

static readfastrschema(name)[source]

Open a json file based on a fastr:// url that points to a file in the fastr.schemadir

Parameters:name (str) – the url of the file to open
Returns:the resulting json schema data
static readfile(filename)[source]

Open a json file based on a simple filename

Parameters:filename (str) – the path of the file to read
Returns:the resulting json schema data
fastr.helpers.jsonschemaparser.any_of_draft4(validator, any_of, instance, schema)[source]

The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.

Parameters:
  • validator – the json schema validator
  • any_of (dict) – the current oneOf
  • instance – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.extend(validator_cls)[source]

Extend the given jsonschema.IValidator with the Seep layer.

fastr.helpers.jsonschemaparser.getblueprinter(uri, blueprint=None)[source]

Instantiate the given data using the blueprinter.

Parameters:blueprint – a blueprint (JSON Schema with Seep properties)
fastr.helpers.jsonschemaparser.items_prevalidate(validator, items, instance, schema)[source]

The pre-validation function for items

Parameters:
  • validator – the json schema validator
  • items (dict) – the current items
  • instance – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.not_draft4(validator, not_schema, instance, schema)[source]

The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema

Parameters:
  • validator – the json schema validator
  • not_schema (dict) – the current oneOf
  • instance – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.one_of_draft4(validator, one_of, instance, schema)[source]

The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.

Parameters:
  • validator – the json schema validator
  • one_of (dict) – the current one_of
  • instance – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.pattern_properties_prevalid(validator, pattern_properties, instance, schema)[source]

The pre-validation function for patternProperties

Parameters:
  • validator – the json schema validator
  • pattern_properties (dict) – the current patternProperties
  • instance (dict) – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.properties_postvalidate(validator, properties, instance, schema)[source]

# All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties

Parameters:
  • validator – the json schema validator
  • properties (dict) – the current properties
  • instance – the current object instance
  • schema (dict) – the current json schema
fastr.helpers.jsonschemaparser.properties_prevalidate(validator, properties, instance, schema)[source]

The pre-validation function for properties

Parameters:
  • validator – the json schema validator
  • properties (dict) – the current properties
  • instance – the current object instance
  • schema (dict) – the current json schema

lazy_module Module

This module contains the Manager class for Plugins in the fastr system

class fastr.helpers.lazy_module.LazyModule(name, parent, plugin_manager)[source]

Bases: module

A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.

__getattr__(item)[source]

The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.

Parameters:item (str) – attribute to retrieve
Returns:the requested attribute
__init__(name, parent, plugin_manager)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.lazy_module'
__repr__()[source]

Return repr(self).

lockfile Module

A module implenting a lock that ensures a directory is only being used by a single fastr run.

class fastr.helpers.lockfile.DirectoryLock(directory)[source]

Bases: object

A lock for a directory, it creates a directory to set the locked state and if successful writes the pid in a file inside that directory to claim the lock

__del__()[source]
__dict__ = mappingproxy({'__module__': 'fastr.helpers.lockfile', '__doc__': '\n A lock for a directory, it creates a directory to set the locked state and\n if successful writes the pid in a file inside that directory to claim the\n lock\n ', 'lock_dir_name': '.fastr.lock', 'pid_file_name': 'pid', '__init__': <function DirectoryLock.__init__>, 'lock_dir': <property object>, 'pid_file': <property object>, 'acquire': <function DirectoryLock.acquire>, 'release': <function DirectoryLock.release>, '__enter__': <function DirectoryLock.__enter__>, '__exit__': <function DirectoryLock.__exit__>, '__del__': <function DirectoryLock.__del__>, '__dict__': <attribute '__dict__' of 'DirectoryLock' objects>, '__weakref__': <attribute '__weakref__' of 'DirectoryLock' objects>})
__enter__()[source]
__exit__(type, value, traceback)[source]
__init__(directory)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.lockfile'
__weakref__

list of weak references to the object (if defined)

acquire()[source]
lock_dir
lock_dir_name = '.fastr.lock'
pid_file
pid_file_name = 'pid'
release()[source]

procutils Module

fastr.helpers.procutils.which(name)[source]
Find executable by name on the PATH, returns the executable that will be
found in case it is used for a Popen call

rest_generation Module

fastr.helpers.rest_generation.create_rest_table(data, headers)[source]

Create a ReST table from data. The data should be a list of columns and the headers should be a list of column names.

Parameters:
  • data (list) – List of lists/tuples representing the columns
  • headers (list) – List of strings for the column names
Returns:

a string representing the table in ReST

Return type:

str

schematotable Module

A module to generate reStructuredText tables from json schema files

class fastr.helpers.schematotable.SchemaPrinter(schema, skipfirst=False)[source]

Bases: object

Object that create a table in reStructuedText from a json schema

__dict__ = mappingproxy({'__module__': 'fastr.helpers.schematotable', '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function SchemaPrinter.__init__>, '__str__': <function SchemaPrinter.__str__>, 'descend': <function SchemaPrinter.descend>, 'parse': <function SchemaPrinter.parse>, 'printlines': <function SchemaPrinter.printlines>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>})
__init__(schema, skipfirst=False)[source]

Create the printer object

Parameters:
  • schema (dict) – the json schema to print
  • skipfirst (bool) – flag to indicate that the first line should not be printed
__module__ = 'fastr.helpers.schematotable'
__str__()[source]

String representation of json schema (that is the printed table)

__weakref__

list of weak references to the object (if defined)

descend(properties)[source]

Descend into a subschema

Parameters:properties (dict) – the properties in the subschema
parse(schema=None)[source]

Parse a schema

Parameters:schema (dict) – the schema to parse
printlines()[source]

Given a parsed schema (parsing happens when the object is constructed), print all the lines

Returns:the printed table
Return type:str

sysinfo Module

This module contains function to help gather system information use for the provenance of the Job execution.

fastr.helpers.sysinfo.get_cpu_usage()[source]

Get the current CPU usage

Returns:CPU usage info
Return type:dict
fastr.helpers.sysinfo.get_drmaa_info()[source]

Get information about the SGE cluster (if applicable)

Returns:cluster info
Return type:dict
fastr.helpers.sysinfo.get_hostinfo()[source]

Get all information about the current host machine

Returns:host info
Return type:dict
fastr.helpers.sysinfo.get_memory_usage()[source]

Get the current memory usage

Returns:memory usage info
Return type:dict
fastr.helpers.sysinfo.get_mounts()[source]

Get the current mounts known on the system

Returns:mount info
Return type:dict
fastr.helpers.sysinfo.get_os()[source]

Get information about the OS

Returns:OS information
Return type:dict
fastr.helpers.sysinfo.get_processes()[source]

Get a list of all currently running processes

Returns:process information
Return type:list
fastr.helpers.sysinfo.get_python()[source]

Get information about the currently used Python implementation

Returns:python info
Return type:dict
fastr.helpers.sysinfo.get_sysinfo()[source]

Get system information (cpu, memory, mounts and users)

Returns:system information
Return type:dict
fastr.helpers.sysinfo.get_users()[source]

Get current users on the system

Returns:user info
Return type:dict
fastr.helpers.sysinfo.namedtuple_to_dict(ntuple)[source]

Helper function to convert a named tuple into a dict

Parameters:ntuple (namedtuple) – the namedtuple to convert
Returns:named tuple as a dict
Return type:dict

xmltodict Module

This module contains tool for converting python dictionaries into XML object and vice-versa.

fastr.helpers.xmltodict.dump(data, filehandle)[source]

Write a dict to an XML file

Parameters:
  • data – data to write
  • filehandle – file handle to write to
fastr.helpers.xmltodict.dumps(data)[source]

Write a dict to an XML string

Parameters:data – data to write
Returns:the XML data
Return type:str
fastr.helpers.xmltodict.load(filehandle)[source]

Load an xml file and parse it to a dict

Parameters:filehandle – file handle to load
Returns:the parsed data
fastr.helpers.xmltodict.loads(data)[source]

Load an xml string and parse it to a dict

Parameters:data (str) – the xml data to load
Returns:the parsed data