helpers Package¶
helpers
Package¶
-
fastr.helpers.
config
= # [bool] Flag to enable/disable debugging debug = False # [str] Directory containing the fastr examples examplesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples" # [str] The default execution plugin to use execution_plugin = "ProcessPoolExecution" # [str] Execution script location executionscript = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/execution/executionscript.py" # [list] Extra configuration directories to read extra_config_dirs = [ "" ] # [str] Redis url e.g. redis://localhost:6379 filesynchelper_url = "" # [str] The level of cleanup required, options: all, no_cleanup, non_failed job_cleanup_level = "no_cleanup" # [bool] Indicate if default logging settings should log to files or not log_to_file = False # [str] Directory where the fastr logs will be placed logdir = "/home/docs/.fastr/logs" # [dict] Python logger config logging_config = {} # [int] The log level to use (as int), INFO is 20, WARNING is 30, etc loglevel = 20 # [str] Type of logging to use logtype = "default" # [dict] A dictionary containing all mount points in the VFS system mounts = { "tmp": "/tmp", "example_data": "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples/data", "home": "/home/docs" } # [list] Directories to scan for networks networks_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/networks" ] # [list] Directories to scan for plugins plugins_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/plugins" ] # [list] A list indicating the order of the preferred types to use. First item is most preferred. preferred_types = [] # [list] A list of modules in the environmnet modules that are protected against unloading protected_modules = [] # [int] Interval in which to report the number of queued jobs (default is 0, no reporting) queue_report_interval = 0 # [list] The reporting plugins to use, is a list of all plugins to be activated reporting_plugins = [ "SimpleReport" ] # [str] Directory containing the fastr system resources resourcesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources" # [str] Directory containing the fastr data schemas schemadir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/schemas" # [int] The number of source jobs allowed to run concurrently source_job_limit = 0 # [str] Fastr installation directory systemdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr" # [list] Directories to scan for tools tools_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/tools" ] # [list] Directories to scan for datatypes types_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/datatypes" ] # [str] Fastr user configuration directory userdir = "/home/docs/.fastr" # [bool] Warning users on import if this is not a production version of fastr warn_develop = True # [str] The hostname to expose the web app for web_hostname = "localhost" # [int] The interval in which the job checker will startto check for stale jobs slurm_job_check_interval = 30 # [str] The slurm partition to use slurm_partition = "" # [int] Number of workers to use in a process pool process_pool_worker_number = 1 # [str] The PIM host to report to pim_host = "" # [str] Username to send to PIM pim_username = "docs" # [float] The interval in which to send jobs to PIM pim_update_interval = 2.5 # [int] Maximum number of jobs that can be send to PIM in a single interval pim_batch_size = 100 # [bool] Setup PIM debug mode to send stdout stderr on job success pim_debug = False # [int] Maximum number of seconds after the network finished in which PIM tries to synchronize all remaining jobs pim_finished_timeout = 10 ¶ Configuration of the fastr system
checksum
Module¶
This module contains a number of functions for checksumming files and objects
-
fastr.helpers.checksum.
checksum
(filepath, algorithm='md5', hasher=None, chunksize=32768)[source]¶ Generate the checksum of a file
-
fastr.helpers.checksum.
checksum_directory
(directory, algorithm='md5', hasher=None)[source]¶ Generate the checksum of an entire directory
-
fastr.helpers.checksum.
hashsum
(objects, hasher=None)[source]¶ Generate the md5 checksum of (a) python object(s)
- Parameters
objects – the objects to hash
hasher – the hasher to use as a base
- Returns
the hash generated
- Return type
classproperty
Module¶
Module containing the code to create class properties.
-
class
fastr.helpers.classproperty.
ClassPropertyDescriptor
(fget)[source]¶ Bases:
object
A descriptor that can act like a property for a class.
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.classproperty', '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function ClassPropertyDescriptor.__init__>, '__get__': <function ClassPropertyDescriptor.__get__>, '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>})¶
-
__module__
= 'fastr.helpers.classproperty'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
clear_pycs
Module¶
A small tool to wipe all .pyc files from fastr
configmanager
Module¶
This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.
-
class
fastr.helpers.configmanager.
Config
(*configfiles)[source]¶ Bases:
object
Class contain the fastr configuration
-
DEFAULT_FIELDS
= {'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for')}¶
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': '\n Class contain the fastr configuration\n ', 'DEFAULT_FIELDS': {'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.2.3/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)')}, '__init__': <function Config.__init__>, 'register_fields': <function Config.register_fields>, 'get_field': <function Config.get_field>, 'set_field': <function Config.set_field>, '_create_field_properties': <classmethod object>, '_field_property': <staticmethod object>, '__repr__': <function Config.__repr__>, 'read_config': <function Config.read_config>, 'read_config_string': <function Config.read_config_string>, 'web_url': <function Config.web_url>, '_update_logging': <function Config._update_logging>, '_deep_update': <function Config._deep_update>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'debug': <property object>, 'examplesdir': <property object>, 'execution_plugin': <property object>, 'executionscript': <property object>, 'extra_config_dirs': <property object>, 'filesynchelper_url': <property object>, 'job_cleanup_level': <property object>, 'log_to_file': <property object>, 'logdir': <property object>, 'logging_config': <property object>, 'loglevel': <property object>, 'logtype': <property object>, 'mounts': <property object>, 'networks_path': <property object>, 'plugins_path': <property object>, 'preferred_types': <property object>, 'protected_modules': <property object>, 'queue_report_interval': <property object>, 'reporting_plugins': <property object>, 'resourcesdir': <property object>, 'schemadir': <property object>, 'source_job_limit': <property object>, 'systemdir': <property object>, 'tools_path': <property object>, 'types_path': <property object>, 'userdir': <property object>, 'warn_develop': <property object>, 'web_hostname': <property object>, 'slurm_job_check_interval': <property object>, 'slurm_partition': <property object>, 'process_pool_worker_number': <property object>, 'pim_host': <property object>, 'pim_username': <property object>, 'pim_update_interval': <property object>, 'pim_batch_size': <property object>, 'pim_debug': <property object>, 'pim_finished_timeout': <property object>})¶
-
__module__
= 'fastr.helpers.configmanager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
property
debug
¶
-
property
examplesdir
¶
-
property
execution_plugin
¶
-
property
executionscript
¶
-
property
extra_config_dirs
¶
-
property
filesynchelper_url
¶
-
property
job_cleanup_level
¶
-
property
log_to_file
¶
-
property
logdir
¶
-
property
logging_config
¶
-
property
loglevel
¶
-
property
logtype
¶
-
property
mounts
¶
-
property
networks_path
¶
-
property
pim_batch_size
¶
-
property
pim_debug
¶
-
property
pim_finished_timeout
¶
-
property
pim_host
¶
-
property
pim_update_interval
¶
-
property
pim_username
¶
-
property
plugins_path
¶
-
property
preferred_types
¶
-
property
process_pool_worker_number
¶
-
property
protected_modules
¶
-
property
queue_report_interval
¶
-
read_config
(filename)[source]¶ Read a configuration and update the configuration object accordingly
- Parameters
filename – the configuration file to read
-
read_config_files
¶ Trace of the config files read by this object
-
property
reporting_plugins
¶
-
property
resourcesdir
¶
-
property
schemadir
¶
-
property
slurm_job_check_interval
¶
-
property
slurm_partition
¶
-
property
source_job_limit
¶
-
property
systemdir
¶
-
property
tools_path
¶
-
property
types_path
¶
-
property
userdir
¶
-
property
warn_develop
¶
-
property
web_hostname
¶
-
-
class
fastr.helpers.configmanager.
EmptyDefault
(data=None)[source]¶ Bases:
object
Empty defaultdict.
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': ' Empty defaultdict. ', '__init__': <function EmptyDefault.__init__>, '__iadd__': <function EmptyDefault.__iadd__>, '__add__': <function EmptyDefault.__add__>, '__radd__': <function EmptyDefault.__radd__>, 'append': <function EmptyDefault.append>, 'prepend': <function EmptyDefault.prepend>, 'extend': <function EmptyDefault.extend>, 'update': <function EmptyDefault.update>, 'merge_default': <function EmptyDefault.merge_default>, '__getitem__': <function EmptyDefault.__getitem__>, '__setitem__': <function EmptyDefault.__setitem__>, '__delitem__': <function EmptyDefault.__delitem__>, 'aslist': <function EmptyDefault.aslist>, 'asdict': <function EmptyDefault.asdict>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>})¶
-
__module__
= 'fastr.helpers.configmanager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
events
Module¶
-
class
fastr.helpers.events.
EventType
(value)[source]¶ Bases:
enum.Enum
An enumeration.
-
__module__
= 'fastr.helpers.events'¶
-
job_updated
= 'job_updated'¶
-
log_record_emitted
= 'log_record_emitted'¶
-
run_finished
= 'run_finished'¶
-
run_started
= 'run_started'¶
-
-
class
fastr.helpers.events.
FastrLogEventHandler
(level=0)[source]¶ Bases:
logging.Handler
Logging handler that sends the log records into the event system
-
__module__
= 'fastr.helpers.events'¶
-
-
fastr.helpers.events.
emit_event
(event_type, data)[source]¶ Emit an event to all listeners :type event_type:
EventType
:param event_type: The type of event to emit :param data: The data object to send along
filesynchelper
Module¶
Some helper functions that aid with NFS file sync issues.
-
class
fastr.helpers.filesynchelper.
FileSyncHelper
[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.filesynchelper', '_namespace': 'filesynchelper', '_redis': None, '__init__': <function FileSyncHelper.__init__>, 'job_finished': <function FileSyncHelper.job_finished>, 'wait_for_job': <function FileSyncHelper.wait_for_job>, 'wait_for_pickle': <function FileSyncHelper.wait_for_pickle>, 'store': <function FileSyncHelper.store>, 'load': <function FileSyncHelper.load>, '_generate_key_for_string': <function FileSyncHelper._generate_key_for_string>, '_generate_hash_from_string': <function FileSyncHelper._generate_hash_from_string>, 'make_file_promise': <function FileSyncHelper.make_file_promise>, 'has_file_promise': <function FileSyncHelper.has_file_promise>, 'wait_for_vfs_url': <function FileSyncHelper.wait_for_vfs_url>, 'wait_for_file': <function FileSyncHelper.wait_for_file>, '_get_suburl_hashes': <function FileSyncHelper._get_suburl_hashes>, '_glob_dir': <function FileSyncHelper._glob_dir>, '_wait_for_file_and_suburls': <function FileSyncHelper._wait_for_file_and_suburls>, '__dict__': <attribute '__dict__' of 'FileSyncHelper' objects>, '__weakref__': <attribute '__weakref__' of 'FileSyncHelper' objects>, '__doc__': None})¶
-
__module__
= 'fastr.helpers.filesynchelper'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
iohelpers
Module¶
jsonschemaparser
Module¶
The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.
-
class
fastr.helpers.jsonschemaparser.
FastrRefResolver
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Bases:
jsonschema.validators.RefResolver
Adapted version of the RefResolver for handling inter-file references more to our liking
-
__init__
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Create a new FastrRefResolver
- Parameters
base_uri (str) – URI of the referring document
referrer – the actual referring document
store (dict) – a mapping from URIs to documents to cache
cache_remote (bool) – whether remote refs should be cached after first resolution
handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
-
__module__
= 'fastr.helpers.jsonschemaparser'¶
-
classmethod
from_schema
(schema, *args, **kwargs)[source]¶ Instantiate a RefResolver based on a schema
-
-
fastr.helpers.jsonschemaparser.
any_of_draft4
(validator, any_of, instance, schema)[source]¶ The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.
-
fastr.helpers.jsonschemaparser.
extend
(validator_cls)[source]¶ Extend the given
jsonschema.IValidator
with the Seep layer.
-
fastr.helpers.jsonschemaparser.
getblueprinter
(uri, blueprint=None)[source]¶ Instantiate the given data using the blueprinter.
- Parameters
blueprint – a blueprint (JSON Schema with Seep properties)
-
fastr.helpers.jsonschemaparser.
items_prevalidate
(validator, items, instance, schema)[source]¶ The pre-validation function for items
-
fastr.helpers.jsonschemaparser.
not_draft4
(validator, not_schema, instance, schema)[source]¶ The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema
-
fastr.helpers.jsonschemaparser.
one_of_draft4
(validator, one_of, instance, schema)[source]¶ The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.
-
fastr.helpers.jsonschemaparser.
pattern_properties_prevalid
(validator, pattern_properties, instance, schema)[source]¶ The pre-validation function for patternProperties
-
fastr.helpers.jsonschemaparser.
properties_postvalidate
(validator, properties, instance, schema)[source]¶ # All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties
lazy_module
Module¶
This module contains the Manager class for Plugins in the fastr system
-
class
fastr.helpers.lazy_module.
LazyModule
(name, parent, plugin_manager)[source]¶ Bases:
module
A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.
-
__getattr__
(item)[source]¶ The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.
- Parameters
item (str) – attribute to retrieve
- Returns
the requested attribute
-
__init__
(name, parent, plugin_manager)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
__module__
= 'fastr.helpers.lazy_module'¶
-
lockfile
Module¶
A module implenting a lock that ensures a directory is only being used by a single fastr run.
-
class
fastr.helpers.lockfile.
DirectoryLock
(directory)[source]¶ Bases:
object
A lock for a directory, it creates a directory to set the locked state and if successful writes the pid in a file inside that directory to claim the lock
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.lockfile', '__doc__': '\n A lock for a directory, it creates a directory to set the locked state and\n if successful writes the pid in a file inside that directory to claim the\n lock\n ', 'lock_dir_name': '.fastr.lock', 'pid_file_name': 'pid', '__init__': <function DirectoryLock.__init__>, 'lock_dir': <property object>, 'pid_file': <property object>, 'acquire': <function DirectoryLock.acquire>, 'release': <function DirectoryLock.release>, '__enter__': <function DirectoryLock.__enter__>, '__exit__': <function DirectoryLock.__exit__>, '__del__': <function DirectoryLock.__del__>, '__dict__': <attribute '__dict__' of 'DirectoryLock' objects>, '__weakref__': <attribute '__weakref__' of 'DirectoryLock' objects>})¶
-
__module__
= 'fastr.helpers.lockfile'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
property
lock_dir
¶
-
lock_dir_name
= '.fastr.lock'¶
-
property
pid_file
¶
-
pid_file_name
= 'pid'¶
-
procutils
Module¶
report
Module¶
Some reporting functions, e.g. to print a report based on a job result
rest_generation
Module¶
schematotable
Module¶
A module to generate reStructuredText tables from json schema files
-
class
fastr.helpers.schematotable.
SchemaPrinter
(schema, skipfirst=False)[source]¶ Bases:
object
Object that create a table in reStructuedText from a json schema
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.schematotable', '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function SchemaPrinter.__init__>, '__str__': <function SchemaPrinter.__str__>, 'descend': <function SchemaPrinter.descend>, 'parse': <function SchemaPrinter.parse>, 'printlines': <function SchemaPrinter.printlines>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>})¶
-
__module__
= 'fastr.helpers.schematotable'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
shellescape
Module¶
Module with helper for shell escaping
sysinfo
Module¶
This module contains function to help gather system information use for the provenance of the Job execution.
-
fastr.helpers.sysinfo.
get_cpu_usage
()[source]¶ Get the current CPU usage
- Returns
CPU usage info
- Return type
-
fastr.helpers.sysinfo.
get_drmaa_info
()[source]¶ Get information about the SGE cluster (if applicable)
- Returns
cluster info
- Return type
-
fastr.helpers.sysinfo.
get_hostinfo
()[source]¶ Get all information about the current host machine
- Returns
host info
- Return type
-
fastr.helpers.sysinfo.
get_memory_usage
()[source]¶ Get the current memory usage
- Returns
memory usage info
- Return type
-
fastr.helpers.sysinfo.
get_mounts
()[source]¶ Get the current mounts known on the system
- Returns
mount info
- Return type
-
fastr.helpers.sysinfo.
get_os
()[source]¶ Get information about the OS
- Returns
OS information
- Return type
-
fastr.helpers.sysinfo.
get_processes
()[source]¶ Get a list of all currently running processes
- Returns
process information
- Return type
-
fastr.helpers.sysinfo.
get_python
()[source]¶ Get information about the currently used Python implementation
- Returns
python info
- Return type
-
fastr.helpers.sysinfo.
get_sysinfo
()[source]¶ Get system information (cpu, memory, mounts and users)
- Returns
system information
- Return type
xmltodict
Module¶
This module contains tool for converting python dictionaries into XML object and vice-versa.
-
fastr.helpers.xmltodict.
dump
(data, filehandle)[source]¶ Write a dict to an XML file
- Parameters
data – data to write
filehandle – file handle to write to
-
fastr.helpers.xmltodict.
dumps
(data)[source]¶ Write a dict to an XML string
- Parameters
data – data to write
- Returns
the XML data
- Return type