helpers Package

helpers Package

fastr.helpers.config = # [bool] Flag to enable/disable debugging debug = False # [str] Directory containing the fastr examples examplesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples" # [str] The default execution plugin to use execution_plugin = "ProcessPoolExecution" # [str] Execution script location executionscript = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/execution/executionscript.py" # [list] Extra configuration directories to read extra_config_dirs = [ "" ] # [str] Redis url e.g. redis://localhost:6379 filesynchelper_url = "" # [str] The level of cleanup required, options: all, no_cleanup, non_failed job_cleanup_level = "no_cleanup" # [bool] Indicate if default logging settings should log to files or not log_to_file = False # [str] Directory where the fastr logs will be placed logdir = "/home/docs/.fastr/logs" # [dict] Python logger config logging_config = {} # [int] The log level to use (as int), INFO is 20, WARNING is 30, etc loglevel = 20 # [str] Type of logging to use logtype = "default" # [dict] A dictionary containing all mount points in the VFS system mounts = { "tmp": "/tmp", "example_data": "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples/data", "home": "/home/docs" } # [list] Directories to scan for networks networks_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/networks" ] # [list] Directories to scan for plugins plugins_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/plugins" ] # [list] A list indicating the order of the preferred types to use. First item is most preferred. preferred_types = [] # [list] A list of modules in the environmnet modules that are protected against unloading protected_modules = [] # [int] Interval in which to report the number of queued jobs (default is 0, no reporting) queue_report_interval = 0 # [list] The reporting plugins to use, is a list of all plugins to be activated reporting_plugins = [ "SimpleReport" ] # [str] Directory containing the fastr system resources resourcesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources" # [str] Directory containing the fastr data schemas schemadir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/schemas" # [int] The number of source jobs allowed to run concurrently source_job_limit = 0 # [str] Fastr installation directory systemdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr" # [list] Directories to scan for tools tools_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/tools" ] # [list] Directories to scan for datatypes types_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/datatypes" ] # [str] Fastr user configuration directory userdir = "/home/docs/.fastr" # [bool] Warning users on import if this is not a production version of fastr warn_develop = True # [str] The hostname to expose the web app for web_hostname = "localhost" # [str] The PIM host to report to pim_host = "" # [str] Username to send to PIM pim_username = "docs" # [float] The interval in which to send jobs to PIM pim_update_interval = 2.5 # [int] Maximum number of jobs that can be send to PIM in a single interval pim_batch_size = 100 # [bool] Setup PIM debug mode to send stdout stderr on job success pim_debug = False # [int] Maximum number of seconds after the network finished in which PIM tries to synchronize all remaining jobs pim_finished_timeout = 10 # [int] Number of workers to use in a process pool process_pool_worker_number = 1

Configuration of the fastr system

checksum Module

This module contains a number of functions for checksumming files and objects

fastr.helpers.checksum.checksum(filepath, algorithm='md5', hasher=None, chunksize=32768)[source]

Generate the checksum of a file

Parameters
  • filepath (str, list) – path of the file(s) to checksum

  • algorithm (str) – the algorithm to use

  • hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)

Returns

the checksum

Return type

str

fastr.helpers.checksum.checksum_directory(directory, algorithm='md5', hasher=None)[source]

Generate the checksum of an entire directory

Parameters
  • directory (str) – path of the file(s) to checksum

  • algorithm (str) – the algorithm to use

  • hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)

Returns

the checksum

Return type

str

fastr.helpers.checksum.hashsum(objects, hasher=None)[source]

Generate the md5 checksum of (a) python object(s)

Parameters
  • objects – the objects to hash

  • hasher – the hasher to use as a base

Returns

the hash generated

Return type

str

fastr.helpers.checksum.md5_checksum(filepath)[source]

Generate the md5 checksum of a file

Parameters

filepath (str, list) – path of the file(s) to checksum

Returns

the checksum

Return type

str

fastr.helpers.checksum.sha1_checksum(filepath)[source]

Generate the sha1 checksum of a file

Parameters

filepath (str, list) – path of the file(s) to checksum

Returns

the checksum

Return type

str

classproperty Module

Module containing the code to create class properties.

class fastr.helpers.classproperty.ClassPropertyDescriptor(fget)[source]

Bases: object

A descriptor that can act like a property for a class.

__dict__ = mappingproxy({'__module__': 'fastr.helpers.classproperty', '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function ClassPropertyDescriptor.__init__>, '__get__': <function ClassPropertyDescriptor.__get__>, '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>})
__get__(obj, cls=None)[source]
__init__(fget)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.classproperty'
__weakref__

list of weak references to the object (if defined)

fastr.helpers.classproperty.classproperty(func)[source]

Decorator to create a “class property”

Parameters

func – the function to wrap

Returns

a class property

Return type

ClassPropertyDescriptor

clear_pycs Module

A small tool to wipe all .pyc files from fastr

fastr.helpers.clear_pycs.dir_list(directory)[source]

Find all .pyc files

Parameters

directory (str) – directory to search

Returns

all .pyc files

Return type

list

fastr.helpers.clear_pycs.main()[source]

Main entry poitn

configmanager Module

This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.

class fastr.helpers.configmanager.Config(*configfiles)[source]

Bases: object

Class contain the fastr configuration

DEFAULT_FIELDS = {'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for')}
__dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': '\n Class contain the fastr configuration\n ', 'DEFAULT_FIELDS': {'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.1/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)')}, '__init__': <function Config.__init__>, 'register_fields': <function Config.register_fields>, 'get_field': <function Config.get_field>, 'set_field': <function Config.set_field>, '_create_field_properties': <classmethod object>, '_field_property': <staticmethod object>, '__repr__': <function Config.__repr__>, 'read_config': <function Config.read_config>, 'read_config_string': <function Config.read_config_string>, 'web_url': <function Config.web_url>, '_update_logging': <function Config._update_logging>, '_deep_update': <function Config._deep_update>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'debug': <property object>, 'examplesdir': <property object>, 'execution_plugin': <property object>, 'executionscript': <property object>, 'extra_config_dirs': <property object>, 'filesynchelper_url': <property object>, 'job_cleanup_level': <property object>, 'log_to_file': <property object>, 'logdir': <property object>, 'logging_config': <property object>, 'loglevel': <property object>, 'logtype': <property object>, 'mounts': <property object>, 'networks_path': <property object>, 'plugins_path': <property object>, 'preferred_types': <property object>, 'protected_modules': <property object>, 'queue_report_interval': <property object>, 'reporting_plugins': <property object>, 'resourcesdir': <property object>, 'schemadir': <property object>, 'source_job_limit': <property object>, 'systemdir': <property object>, 'tools_path': <property object>, 'types_path': <property object>, 'userdir': <property object>, 'warn_develop': <property object>, 'web_hostname': <property object>, 'pim_host': <property object>, 'pim_username': <property object>, 'pim_update_interval': <property object>, 'pim_batch_size': <property object>, 'pim_debug': <property object>, 'pim_finished_timeout': <property object>, 'process_pool_worker_number': <property object>})
__init__(*configfiles)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.configmanager'
__repr__()[source]

Return repr(self).

__weakref__

list of weak references to the object (if defined)

property debug
property examplesdir
property execution_plugin
property executionscript
property extra_config_dirs
property filesynchelper_url
get_field(item)[source]
property job_cleanup_level
property log_to_file
property logdir
property logging_config
property loglevel
property logtype
property mounts
property networks_path
property pim_batch_size
property pim_debug
property pim_finished_timeout
property pim_host
property pim_update_interval
property pim_username
property plugins_path
property preferred_types
property process_pool_worker_number
property protected_modules
property queue_report_interval
read_config(filename)[source]

Read a configuration and update the configuration object accordingly

Parameters

filename – the configuration file to read

read_config_files

Trace of the config files read by this object

read_config_string(value)[source]
register_fields(fields_spec)[source]

Register extra fields to the configuration manager.

property reporting_plugins
property resourcesdir
property schemadir
set_field(item, value)[source]
property source_job_limit
property systemdir
property tools_path
property types_path
property userdir
property warn_develop
property web_hostname
web_url()[source]

Construct a fqdn from the web[‘hostname’] and web[‘port’] settings. :return: FQDN :rtype: str

class fastr.helpers.configmanager.EmptyDefault(data=None)[source]

Bases: object

Empty defaultdict.

__add__(right)[source]
__delitem__(key)[source]
__dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': ' Empty defaultdict. ', '__init__': <function EmptyDefault.__init__>, '__iadd__': <function EmptyDefault.__iadd__>, '__add__': <function EmptyDefault.__add__>, '__radd__': <function EmptyDefault.__radd__>, 'append': <function EmptyDefault.append>, 'prepend': <function EmptyDefault.prepend>, 'extend': <function EmptyDefault.extend>, 'update': <function EmptyDefault.update>, 'merge_default': <function EmptyDefault.merge_default>, '__getitem__': <function EmptyDefault.__getitem__>, '__setitem__': <function EmptyDefault.__setitem__>, '__delitem__': <function EmptyDefault.__delitem__>, 'aslist': <function EmptyDefault.aslist>, 'asdict': <function EmptyDefault.asdict>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>})
__getitem__(item)[source]
__iadd__(right)[source]
__init__(data=None)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.configmanager'
__radd__(other)[source]
__setitem__(key, value)[source]
__weakref__

list of weak references to the object (if defined)

append(value)[source]
asdict()[source]
aslist()[source]
extend(other)[source]
merge_default(field_spec)[source]

Merge the default into this EmptyDefault given the field spec :param field_spec: Field specification :return: Merged value

prepend(value)[source]
update(other)[source]

events Module

class fastr.helpers.events.EventType(value)[source]

Bases: enum.Enum

An enumeration.

__module__ = 'fastr.helpers.events'
job_updated = 'job_updated'
log_record_emitted = 'log_record_emitted'
run_finished = 'run_finished'
run_started = 'run_started'
class fastr.helpers.events.FastrLogEventHandler(level=0)[source]

Bases: logging.Handler

Logging handler that sends the log records into the event system

__module__ = 'fastr.helpers.events'
emit(record)[source]

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

fastr.helpers.events.emit_event(event_type, data)[source]

Emit an event to all listeners :type event_type: EventType :param event_type: The type of event to emit :param data: The data object to send along

fastr.helpers.events.register_listener(event_type, function)[source]

Register a listeners to a specific event type

Parameters
  • event_type (EventType) – The EventType to listen on

  • function (Callable[[object], None]) – The callable that will be called on each event

fastr.helpers.events.remove_listener(event_type, function)[source]

Remove a listeren from a type of event

Parameters
  • event_type (EventType) – The event type to remove the listeners from

  • function (Callable[[object], None]) – The function to remove

filesynchelper Module

Some helper functions that aid with NFS file sync issues.

class fastr.helpers.filesynchelper.FileSyncHelper[source]

Bases: object

__dict__ = mappingproxy({'__module__': 'fastr.helpers.filesynchelper', '_namespace': 'filesynchelper', '_redis': None, '__init__': <function FileSyncHelper.__init__>, 'job_finished': <function FileSyncHelper.job_finished>, 'wait_for_job': <function FileSyncHelper.wait_for_job>, 'wait_for_pickle': <function FileSyncHelper.wait_for_pickle>, 'store': <function FileSyncHelper.store>, 'load': <function FileSyncHelper.load>, '_generate_key_for_string': <function FileSyncHelper._generate_key_for_string>, '_generate_hash_from_string': <function FileSyncHelper._generate_hash_from_string>, 'make_file_promise': <function FileSyncHelper.make_file_promise>, 'has_file_promise': <function FileSyncHelper.has_file_promise>, 'wait_for_vfs_url': <function FileSyncHelper.wait_for_vfs_url>, 'wait_for_file': <function FileSyncHelper.wait_for_file>, '_get_suburl_hashes': <function FileSyncHelper._get_suburl_hashes>, '_glob_dir': <function FileSyncHelper._glob_dir>, '_wait_for_file_and_suburls': <function FileSyncHelper._wait_for_file_and_suburls>, '__dict__': <attribute '__dict__' of 'FileSyncHelper' objects>, '__weakref__': <attribute '__weakref__' of 'FileSyncHelper' objects>, '__doc__': None})
__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.filesynchelper'
__weakref__

list of weak references to the object (if defined)

has_file_promise(url)[source]
job_finished(jobfile)[source]
load(url)[source]
make_file_promise(url)[source]
store(url, data)[source]
wait_for_file(path, timeout=300)[source]
wait_for_job(jobfile)[source]
wait_for_pickle(url, timeout=300)[source]
wait_for_vfs_url(vfs_url, timeout=300)[source]
fastr.helpers.filesynchelper.filesynchelper_enabled()[source]

iohelpers Module

fastr.helpers.iohelpers.load_gpickle(path, retry_scheme=None)[source]
fastr.helpers.iohelpers.load_json(path)[source]
fastr.helpers.iohelpers.save_gpickle(path, data)[source]
fastr.helpers.iohelpers.save_json(path, data, indent=2)[source]

jsonschemaparser Module

The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.

class fastr.helpers.jsonschemaparser.FastrRefResolver(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]

Bases: jsonschema.validators.RefResolver

Adapted version of the RefResolver for handling inter-file references more to our liking

__init__(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]

Create a new FastrRefResolver

Parameters
  • base_uri (str) – URI of the referring document

  • referrer – the actual referring document

  • store (dict) – a mapping from URIs to documents to cache

  • cache_remote (bool) – whether remote refs should be cached after first resolution

  • handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them

__module__ = 'fastr.helpers.jsonschemaparser'
classmethod from_schema(schema, *args, **kwargs)[source]

Instantiate a RefResolver based on a schema

static readfastrschema(name)[source]

Open a json file based on a fastr:// url that points to a file in the fastr.schemadir

Parameters

name (str) – the url of the file to open

Returns

the resulting json schema data

static readfile(filename)[source]

Open a json file based on a simple filename

Parameters

filename (str) – the path of the file to read

Returns

the resulting json schema data

fastr.helpers.jsonschemaparser.any_of_draft4(validator, any_of, instance, schema)[source]

The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.

Parameters
  • validator – the json schema validator

  • any_of (dict) – the current oneOf

  • instance – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.extend(validator_cls)[source]

Extend the given jsonschema.IValidator with the Seep layer.

fastr.helpers.jsonschemaparser.getblueprinter(uri, blueprint=None)[source]

Instantiate the given data using the blueprinter.

Parameters

blueprint – a blueprint (JSON Schema with Seep properties)

fastr.helpers.jsonschemaparser.items_prevalidate(validator, items, instance, schema)[source]

The pre-validation function for items

Parameters
  • validator – the json schema validator

  • items (dict) – the current items

  • instance – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.not_draft4(validator, not_schema, instance, schema)[source]

The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema

Parameters
  • validator – the json schema validator

  • not_schema (dict) – the current oneOf

  • instance – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.one_of_draft4(validator, one_of, instance, schema)[source]

The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.

Parameters
  • validator – the json schema validator

  • one_of (dict) – the current one_of

  • instance – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.pattern_properties_prevalid(validator, pattern_properties, instance, schema)[source]

The pre-validation function for patternProperties

Parameters
  • validator – the json schema validator

  • pattern_properties (dict) – the current patternProperties

  • instance (dict) – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.properties_postvalidate(validator, properties, instance, schema)[source]

# All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties

Parameters
  • validator – the json schema validator

  • properties (dict) – the current properties

  • instance – the current object instance

  • schema (dict) – the current json schema

fastr.helpers.jsonschemaparser.properties_prevalidate(validator, properties, instance, schema)[source]

The pre-validation function for properties

Parameters
  • validator – the json schema validator

  • properties (dict) – the current properties

  • instance – the current object instance

  • schema (dict) – the current json schema

lazy_module Module

This module contains the Manager class for Plugins in the fastr system

class fastr.helpers.lazy_module.LazyModule(name, parent, plugin_manager)[source]

Bases: module

A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.

__getattr__(item)[source]

The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.

Parameters

item (str) – attribute to retrieve

Returns

the requested attribute

__init__(name, parent, plugin_manager)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.lazy_module'
__repr__()[source]

Return repr(self).

lockfile Module

A module implenting a lock that ensures a directory is only being used by a single fastr run.

class fastr.helpers.lockfile.DirectoryLock(directory)[source]

Bases: object

A lock for a directory, it creates a directory to set the locked state and if successful writes the pid in a file inside that directory to claim the lock

__del__()[source]
__dict__ = mappingproxy({'__module__': 'fastr.helpers.lockfile', '__doc__': '\n A lock for a directory, it creates a directory to set the locked state and\n if successful writes the pid in a file inside that directory to claim the\n lock\n ', 'lock_dir_name': '.fastr.lock', 'pid_file_name': 'pid', '__init__': <function DirectoryLock.__init__>, 'lock_dir': <property object>, 'pid_file': <property object>, 'acquire': <function DirectoryLock.acquire>, 'release': <function DirectoryLock.release>, '__enter__': <function DirectoryLock.__enter__>, '__exit__': <function DirectoryLock.__exit__>, '__del__': <function DirectoryLock.__del__>, '__dict__': <attribute '__dict__' of 'DirectoryLock' objects>, '__weakref__': <attribute '__weakref__' of 'DirectoryLock' objects>})
__enter__()[source]
__exit__(type, value, traceback)[source]
__init__(directory)[source]

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'fastr.helpers.lockfile'
__weakref__

list of weak references to the object (if defined)

acquire()[source]
property lock_dir
lock_dir_name = '.fastr.lock'
property pid_file
pid_file_name = 'pid'
release()[source]

procutils Module

fastr.helpers.procutils.which(name)[source]
Find executable by name on the PATH, returns the executable that will be

found in case it is used for a Popen call

report Module

Some reporting functions, e.g. to print a report based on a job result

fastr.helpers.report.print_job_result(job_file, print_func=<built-in function print>, verbose=False)[source]

rest_generation Module

fastr.helpers.rest_generation.create_rest_table(data, headers)[source]

Create a ReST table from data. The data should be a list of columns and the headers should be a list of column names.

Parameters
  • data (list) – List of lists/tuples representing the columns

  • headers (list) – List of strings for the column names

Returns

a string representing the table in ReST

Return type

str

schematotable Module

A module to generate reStructuredText tables from json schema files

class fastr.helpers.schematotable.SchemaPrinter(schema, skipfirst=False)[source]

Bases: object

Object that create a table in reStructuedText from a json schema

__dict__ = mappingproxy({'__module__': 'fastr.helpers.schematotable', '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function SchemaPrinter.__init__>, '__str__': <function SchemaPrinter.__str__>, 'descend': <function SchemaPrinter.descend>, 'parse': <function SchemaPrinter.parse>, 'printlines': <function SchemaPrinter.printlines>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>})
__init__(schema, skipfirst=False)[source]

Create the printer object

Parameters
  • schema (dict) – the json schema to print

  • skipfirst (bool) – flag to indicate that the first line should not be printed

__module__ = 'fastr.helpers.schematotable'
__str__()[source]

String representation of json schema (that is the printed table)

__weakref__

list of weak references to the object (if defined)

descend(properties)[source]

Descend into a subschema

Parameters

properties (dict) – the properties in the subschema

parse(schema=None)[source]

Parse a schema

Parameters

schema (dict) – the schema to parse

printlines()[source]

Given a parsed schema (parsing happens when the object is constructed), print all the lines

Returns

the printed table

Return type

str

shellescape Module

Module with helper for shell escaping

fastr.helpers.shellescape.quote_argument(arg)[source]

Use shlex module to quote the argument properly :type arg: str :param arg: argument to quote :rtype: str :return: argument with quotes for safe use in a bash-like shell

sysinfo Module

This module contains function to help gather system information use for the provenance of the Job execution.

fastr.helpers.sysinfo.get_cpu_usage()[source]

Get the current CPU usage

Returns

CPU usage info

Return type

dict

fastr.helpers.sysinfo.get_drmaa_info()[source]

Get information about the SGE cluster (if applicable)

Returns

cluster info

Return type

dict

fastr.helpers.sysinfo.get_hostinfo()[source]

Get all information about the current host machine

Returns

host info

Return type

dict

fastr.helpers.sysinfo.get_memory_usage()[source]

Get the current memory usage

Returns

memory usage info

Return type

dict

fastr.helpers.sysinfo.get_mounts()[source]

Get the current mounts known on the system

Returns

mount info

Return type

dict

fastr.helpers.sysinfo.get_os()[source]

Get information about the OS

Returns

OS information

Return type

dict

fastr.helpers.sysinfo.get_processes()[source]

Get a list of all currently running processes

Returns

process information

Return type

list

fastr.helpers.sysinfo.get_python()[source]

Get information about the currently used Python implementation

Returns

python info

Return type

dict

fastr.helpers.sysinfo.get_sysinfo()[source]

Get system information (cpu, memory, mounts and users)

Returns

system information

Return type

dict

fastr.helpers.sysinfo.get_users()[source]

Get current users on the system

Returns

user info

Return type

dict

fastr.helpers.sysinfo.namedtuple_to_dict(ntuple)[source]

Helper function to convert a named tuple into a dict

Parameters

ntuple (namedtuple) – the namedtuple to convert

Returns

named tuple as a dict

Return type

dict

xmltodict Module

This module contains tool for converting python dictionaries into XML object and vice-versa.

fastr.helpers.xmltodict.dump(data, filehandle)[source]

Write a dict to an XML file

Parameters
  • data – data to write

  • filehandle – file handle to write to

fastr.helpers.xmltodict.dumps(data)[source]

Write a dict to an XML string

Parameters

data – data to write

Returns

the XML data

Return type

str

fastr.helpers.xmltodict.load(filehandle)[source]

Load an xml file and parse it to a dict

Parameters

filehandle – file handle to load

Returns

the parsed data

fastr.helpers.xmltodict.loads(data)[source]

Load an xml string and parse it to a dict

Parameters

data (str) – the xml data to load

Returns

the parsed data