helpers Package¶
helpers
Package¶
- fastr.helpers.config = # [bool] Flag to enable/disable debugging debug = False # [str] Directory containing the fastr examples examplesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples" # [str] The default execution plugin to use execution_plugin = "ProcessPoolExecution" # [str] Execution script location executionscript = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/execution/executionscript.py" # [list] Extra configuration directories to read extra_config_dirs = [ "" ] # [str] Redis url e.g. redis://localhost:6379 filesynchelper_url = "" # [str] The level of cleanup required, options: all, no_cleanup, non_failed job_cleanup_level = "no_cleanup" # [bool] Indicate if default logging settings should log to files or not log_to_file = False # [str] Directory where the fastr logs will be placed logdir = "/home/docs/.fastr/logs" # [dict] Python logger config logging_config = {} # [int] The log level to use (as int), INFO is 20, WARNING is 30, etc loglevel = 20 # [str] Type of logging to use logtype = "default" # [dict] A dictionary containing all mount points in the VFS system mounts = { "tmp": "/tmp", "example_data": "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples/data", "home": "/home/docs" } # [list] Directories to scan for networks networks_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/networks" ] # [list] Directories to scan for plugins plugins_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/plugins" ] # [list] A list indicating the order of the preferred types to use. First item is most preferred. preferred_types = [] # [list] A list of modules in the environmnet modules that are protected against unloading protected_modules = [] # [int] Interval in which to report the number of queued jobs (default is 0, no reporting) queue_report_interval = 0 # [list] The reporting plugins to use, is a list of all plugins to be activated reporting_plugins = [ "SimpleReport" ] # [str] Directory containing the fastr system resources resourcesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources" # [str] Directory containing the fastr data schemas schemadir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/schemas" # [int] The number of source jobs allowed to run concurrently source_job_limit = 0 # [str] Fastr installation directory systemdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr" # [list] Directories to scan for tools tools_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/tools" ] # [list] Directories to scan for datatypes types_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/datatypes" ] # [str] Fastr user configuration directory userdir = "/home/docs/.fastr" # [bool] Warning users on import if this is not a production version of fastr warn_develop = True # [str] The hostname to expose the web app for web_hostname = "localhost" # [int] The interval in which the job checker will startto check for stale jobs slurm_job_check_interval = 30 # [str] The slurm partition to use slurm_partition = "" # [int] Number of workers to use in a process pool process_pool_worker_number = 1 # [str] The PIM host to report to pim_host = "" # [str] Username to send to PIM pim_username = "docs" # [float] The interval in which to send jobs to PIM pim_update_interval = 2.5 # [int] Maximum number of jobs that can be send to PIM in a single interval pim_batch_size = 100 # [bool] Setup PIM debug mode to send stdout stderr on job success pim_debug = False # [int] Maximum number of seconds after the network finished in which PIM tries to synchronize all remaining jobs pim_finished_timeout = 10 ¶
Configuration of the fastr system
checksum
Module¶
This module contains a number of functions for checksumming files and objects
- fastr.helpers.checksum.checksum(filepath, algorithm='md5', hasher=None, chunksize=32768)[source]¶
Generate the checksum of a file
- fastr.helpers.checksum.checksum_directory(directory, algorithm='md5', hasher=None)[source]¶
Generate the checksum of an entire directory
- fastr.helpers.checksum.hashsum(objects, hasher=None)[source]¶
Generate the md5 checksum of (a) python object(s)
- Parameters
objects – the objects to hash
hasher – the hasher to use as a base
- Returns
the hash generated
- Return type
classproperty
Module¶
Module containing the code to create class properties.
- class fastr.helpers.classproperty.ClassPropertyDescriptor(fget)[source]¶
Bases:
object
A descriptor that can act like a property for a class.
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.classproperty', '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function ClassPropertyDescriptor.__init__>, '__get__': <function ClassPropertyDescriptor.__get__>, '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>, '__annotations__': {}})¶
- __module__ = 'fastr.helpers.classproperty'¶
- __weakref__¶
list of weak references to the object (if defined)
clear_pycs
Module¶
A small tool to wipe all .pyc files from fastr
configmanager
Module¶
This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.
- class fastr.helpers.configmanager.Config(*configfiles)[source]¶
Bases:
object
Class contain the fastr configuration
- DEFAULT_FIELDS = {'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for')}¶
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': '\n Class contain the fastr configuration\n ', 'DEFAULT_FIELDS': {'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'extra_config_dirs': (<class 'list'>, [''], 'Extra configuration directories to read'), 'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'log_to_file': (<class 'bool'>, False, 'Indicate if default logging settings should log to files or not'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr', 'Fastr installation directory', 'Directory of the top-level fastr package'), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.3.0/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'queue_report_interval': (<class 'int'>, 0, 'Interval in which to report the number of queued jobs (default is 0, no reporting)')}, '__init__': <function Config.__init__>, 'register_fields': <function Config.register_fields>, 'get_field': <function Config.get_field>, 'set_field': <function Config.set_field>, '_create_field_properties': <classmethod object>, '_field_property': <staticmethod object>, '__repr__': <function Config.__repr__>, 'read_config': <function Config.read_config>, 'read_config_string': <function Config.read_config_string>, 'web_url': <function Config.web_url>, '_update_logging': <function Config._update_logging>, '_deep_update': <function Config._deep_update>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'debug': <property object>, 'examplesdir': <property object>, 'execution_plugin': <property object>, 'executionscript': <property object>, 'extra_config_dirs': <property object>, 'filesynchelper_url': <property object>, 'job_cleanup_level': <property object>, 'log_to_file': <property object>, 'logdir': <property object>, 'logging_config': <property object>, 'loglevel': <property object>, 'logtype': <property object>, 'mounts': <property object>, 'networks_path': <property object>, 'plugins_path': <property object>, 'preferred_types': <property object>, 'protected_modules': <property object>, 'queue_report_interval': <property object>, 'reporting_plugins': <property object>, 'resourcesdir': <property object>, 'schemadir': <property object>, 'source_job_limit': <property object>, 'systemdir': <property object>, 'tools_path': <property object>, 'types_path': <property object>, 'userdir': <property object>, 'warn_develop': <property object>, 'web_hostname': <property object>, 'slurm_job_check_interval': <property object>, 'slurm_partition': <property object>, 'process_pool_worker_number': <property object>, 'pim_host': <property object>, 'pim_username': <property object>, 'pim_update_interval': <property object>, 'pim_batch_size': <property object>, 'pim_debug': <property object>, 'pim_finished_timeout': <property object>, '__annotations__': {}})¶
- __module__ = 'fastr.helpers.configmanager'¶
- __weakref__¶
list of weak references to the object (if defined)
- property debug¶
- property examplesdir¶
- property execution_plugin¶
- property executionscript¶
- property extra_config_dirs¶
- property filesynchelper_url¶
- property job_cleanup_level¶
- property log_to_file¶
- property logdir¶
- property logging_config¶
- property loglevel¶
- property logtype¶
- property mounts¶
- property networks_path¶
- property pim_batch_size¶
- property pim_debug¶
- property pim_finished_timeout¶
- property pim_host¶
- property pim_update_interval¶
- property pim_username¶
- property plugins_path¶
- property preferred_types¶
- property process_pool_worker_number¶
- property protected_modules¶
- property queue_report_interval¶
- read_config(filename)[source]¶
Read a configuration and update the configuration object accordingly
- Parameters
filename – the configuration file to read
- read_config_files¶
Trace of the config files read by this object
- property reporting_plugins¶
- property resourcesdir¶
- property schemadir¶
- property slurm_job_check_interval¶
- property slurm_partition¶
- property source_job_limit¶
- property systemdir¶
- property tools_path¶
- property types_path¶
- property userdir¶
- property warn_develop¶
- property web_hostname¶
- class fastr.helpers.configmanager.EmptyDefault(data=None)[source]¶
Bases:
object
Empty defaultdict.
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': ' Empty defaultdict. ', '__init__': <function EmptyDefault.__init__>, '__iadd__': <function EmptyDefault.__iadd__>, '__add__': <function EmptyDefault.__add__>, '__radd__': <function EmptyDefault.__radd__>, 'append': <function EmptyDefault.append>, 'prepend': <function EmptyDefault.prepend>, 'extend': <function EmptyDefault.extend>, 'update': <function EmptyDefault.update>, 'merge_default': <function EmptyDefault.merge_default>, '__getitem__': <function EmptyDefault.__getitem__>, '__setitem__': <function EmptyDefault.__setitem__>, '__delitem__': <function EmptyDefault.__delitem__>, 'aslist': <function EmptyDefault.aslist>, 'asdict': <function EmptyDefault.asdict>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>, '__annotations__': {}})¶
- __module__ = 'fastr.helpers.configmanager'¶
- __weakref__¶
list of weak references to the object (if defined)
- class fastr.helpers.configmanager.FastrLogRecordFilter(name='')[source]¶
Bases:
logging.Filter
- __module__ = 'fastr.helpers.configmanager'¶
events
Module¶
- class fastr.helpers.events.EventType(value)[source]¶
Bases:
enum.Enum
An enumeration.
- __module__ = 'fastr.helpers.events'¶
- job_updated = 'job_updated'¶
- log_record_emitted = 'log_record_emitted'¶
- run_finished = 'run_finished'¶
- run_started = 'run_started'¶
- class fastr.helpers.events.FastrLogEventHandler(level=0)[source]¶
Bases:
logging.Handler
Logging handler that sends the log records into the event system
- __module__ = 'fastr.helpers.events'¶
- fastr.helpers.events.emit_event(event_type, data)[source]¶
Emit an event to all listeners :type event_type:
EventType
:param event_type: The type of event to emit :param data: The data object to send along
filesynchelper
Module¶
Some helper functions that aid with NFS file sync issues.
- class fastr.helpers.filesynchelper.FileSyncHelper[source]¶
Bases:
object
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.filesynchelper', '_namespace': 'filesynchelper', '_redis': None, '__init__': <function FileSyncHelper.__init__>, 'job_finished': <function FileSyncHelper.job_finished>, 'wait_for_job': <function FileSyncHelper.wait_for_job>, 'wait_for_pickle': <function FileSyncHelper.wait_for_pickle>, 'store': <function FileSyncHelper.store>, 'load': <function FileSyncHelper.load>, '_generate_key_for_string': <function FileSyncHelper._generate_key_for_string>, '_generate_hash_from_string': <function FileSyncHelper._generate_hash_from_string>, 'make_file_promise': <function FileSyncHelper.make_file_promise>, 'has_file_promise': <function FileSyncHelper.has_file_promise>, 'wait_for_vfs_url': <function FileSyncHelper.wait_for_vfs_url>, 'wait_for_file': <function FileSyncHelper.wait_for_file>, '_get_suburl_hashes': <function FileSyncHelper._get_suburl_hashes>, '_glob_dir': <function FileSyncHelper._glob_dir>, '_wait_for_file_and_suburls': <function FileSyncHelper._wait_for_file_and_suburls>, '__dict__': <attribute '__dict__' of 'FileSyncHelper' objects>, '__weakref__': <attribute '__weakref__' of 'FileSyncHelper' objects>, '__doc__': None, '__annotations__': {}})¶
- __module__ = 'fastr.helpers.filesynchelper'¶
- __weakref__¶
list of weak references to the object (if defined)
iohelpers
Module¶
jsonschemaparser
Module¶
The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.
- class fastr.helpers.jsonschemaparser.FastrRefResolver(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶
Bases:
jsonschema.validators.RefResolver
Adapted version of the RefResolver for handling inter-file references more to our liking
- __init__(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶
Create a new FastrRefResolver
- Parameters
base_uri (str) – URI of the referring document
referrer – the actual referring document
store (dict) – a mapping from URIs to documents to cache
cache_remote (bool) – whether remote refs should be cached after first resolution
handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
- __module__ = 'fastr.helpers.jsonschemaparser'¶
- classmethod from_schema(schema, *args, **kwargs)[source]¶
Instantiate a RefResolver based on a schema
- fastr.helpers.jsonschemaparser.any_of_draft4(validator, any_of, instance, schema)[source]¶
The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.
- fastr.helpers.jsonschemaparser.extend(validator_cls)[source]¶
Extend the given
jsonschema.IValidator
with the Seep layer.
- fastr.helpers.jsonschemaparser.getblueprinter(uri, blueprint=None)[source]¶
Instantiate the given data using the blueprinter.
- Parameters
blueprint – a blueprint (JSON Schema with Seep properties)
- fastr.helpers.jsonschemaparser.items_prevalidate(validator, items, instance, schema)[source]¶
The pre-validation function for items
- fastr.helpers.jsonschemaparser.not_draft4(validator, not_schema, instance, schema)[source]¶
The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema
- fastr.helpers.jsonschemaparser.one_of_draft4(validator, one_of, instance, schema)[source]¶
The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.
- fastr.helpers.jsonschemaparser.pattern_properties_prevalid(validator, pattern_properties, instance, schema)[source]¶
The pre-validation function for patternProperties
- fastr.helpers.jsonschemaparser.properties_postvalidate(validator, properties, instance, schema)[source]¶
# All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties
lazy_module
Module¶
This module contains the Manager class for Plugins in the fastr system
- class fastr.helpers.lazy_module.LazyModule(name, parent, plugin_manager)[source]¶
Bases:
module
A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.
- __getattr__(item)[source]¶
The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.
- Parameters
item (str) – attribute to retrieve
- Returns
the requested attribute
- __init__(name, parent, plugin_manager)[source]¶
Initialize self. See help(type(self)) for accurate signature.
- __module__ = 'fastr.helpers.lazy_module'¶
lockfile
Module¶
A module implenting a lock that ensures a directory is only being used by a single fastr run.
- class fastr.helpers.lockfile.DirectoryLock(directory)[source]¶
Bases:
object
A lock for a directory, it creates a directory to set the locked state and if successful writes the pid in a file inside that directory to claim the lock
- __annotations__ = {'lock_dir_name': <class 'str'>, 'pid_file_name': <class 'str'>}¶
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.lockfile', '__annotations__': {'lock_dir_name': <class 'str'>, 'pid_file_name': <class 'str'>}, '__doc__': '\n A lock for a directory, it creates a directory to set the locked state and\n if successful writes the pid in a file inside that directory to claim the\n lock\n ', 'lock_dir_name': '.fastr.lock', 'pid_file_name': 'pid', '__init__': <function DirectoryLock.__init__>, 'lock_dir': <property object>, 'pid_file': <property object>, 'get_pid': <function DirectoryLock.get_pid>, '_checkpid': <staticmethod object>, 'acquire': <function DirectoryLock.acquire>, 'release': <function DirectoryLock.release>, '__enter__': <function DirectoryLock.__enter__>, '__exit__': <function DirectoryLock.__exit__>, '__del__': <function DirectoryLock.__del__>, '__dict__': <attribute '__dict__' of 'DirectoryLock' objects>, '__weakref__': <attribute '__weakref__' of 'DirectoryLock' objects>})¶
- __module__ = 'fastr.helpers.lockfile'¶
- __weakref__¶
list of weak references to the object (if defined)
procutils
Module¶
report
Module¶
Some reporting functions, e.g. to print a report based on a job result
rest_generation
Module¶
schematotable
Module¶
A module to generate reStructuredText tables from json schema files
- class fastr.helpers.schematotable.SchemaPrinter(schema, skipfirst=False)[source]¶
Bases:
object
Object that create a table in reStructuedText from a json schema
- __dict__ = mappingproxy({'__module__': 'fastr.helpers.schematotable', '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function SchemaPrinter.__init__>, '__str__': <function SchemaPrinter.__str__>, 'descend': <function SchemaPrinter.descend>, 'parse': <function SchemaPrinter.parse>, 'printlines': <function SchemaPrinter.printlines>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>, '__annotations__': {}})¶
- __module__ = 'fastr.helpers.schematotable'¶
- __weakref__¶
list of weak references to the object (if defined)
shellescape
Module¶
Module with helper for shell escaping
sysinfo
Module¶
This module contains function to help gather system information use for the provenance of the Job execution.
- fastr.helpers.sysinfo.get_cpu_usage()[source]¶
Get the current CPU usage
- Returns
CPU usage info
- Return type
- fastr.helpers.sysinfo.get_drmaa_info()[source]¶
Get information about the SGE cluster (if applicable)
- Returns
cluster info
- Return type
- fastr.helpers.sysinfo.get_hostinfo()[source]¶
Get all information about the current host machine
- Returns
host info
- Return type
- fastr.helpers.sysinfo.get_memory_usage()[source]¶
Get the current memory usage
- Returns
memory usage info
- Return type
- fastr.helpers.sysinfo.get_mounts()[source]¶
Get the current mounts known on the system
- Returns
mount info
- Return type
- fastr.helpers.sysinfo.get_os()[source]¶
Get information about the OS
- Returns
OS information
- Return type
- fastr.helpers.sysinfo.get_processes()[source]¶
Get a list of all currently running processes
- Returns
process information
- Return type
- fastr.helpers.sysinfo.get_python()[source]¶
Get information about the currently used Python implementation
- Returns
python info
- Return type
- fastr.helpers.sysinfo.get_sysinfo()[source]¶
Get system information (cpu, memory, mounts and users)
- Returns
system information
- Return type
xmltodict
Module¶
This module contains tool for converting python dictionaries into XML object and vice-versa.
- fastr.helpers.xmltodict.dump(data, filehandle)[source]¶
Write a dict to an XML file
- Parameters
data – data to write
filehandle – file handle to write to
- fastr.helpers.xmltodict.dumps(data)[source]¶
Write a dict to an XML string
- Parameters
data – data to write
- Returns
the XML data
- Return type