helpers Package¶
helpers
Package¶
-
fastr.helpers.
config
= # [bool] Flag to enable/disable debugging debug = False # [str] Directory containing the fastr examples examplesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples" # [str] The default execution plugin to use execution_plugin = "ProcessPoolExecution" # [str] Execution script location executionscript = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/execution/executionscript.py" # [str] Redis url e.g. redis://localhost:6379 filesynchelper_url = "" # [str] The level of cleanup required, options: all, no_cleanup, non_failed job_cleanup_level = "no_cleanup" # [str] Directory where the fastr logs will be placed logdir = "/home/docs/.fastr/logs" # [dict] Python logger config logging_config = {} # [int] The log level to use (as int), INFO is 20, WARNING is 30, etc loglevel = 20 # [str] Type of logging to use logtype = "default" # [dict] A dictionary containing all mount points in the VFS system mounts = { "tmp": "/tmp", "example_data": "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples/data", "home": "/home/docs/" } # [list] Directories to scan for networks networks_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/networks" ] # [list] Directories to scan for plugins plugins_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/plugins" ] # [list] A list indicating the order of the preferred types to use. First item is most preferred. preferred_types = [] # [list] A list of modules in the environmnet modules that are protected against unloading protected_modules = [] # [list] The reporting plugins to use, is a list of all plugins to be activated reporting_plugins = [ "SimpleReport" ] # [str] Directory containing the fastr system resources resourcesdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources" # [str] Directory containing the fastr data schemas schemadir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/schemas" # [int] The number of source jobs allowed to run concurrently source_job_limit = 0 # [str] Fastr installation directory systemdir = "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr" # [list] Directories to scan for tools tools_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/tools" ] # [list] Directories to scan for datatypes types_path = [ "/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/datatypes" ] # [str] Fastr user configuration directory userdir = "/home/docs/.fastr" # [bool] Warning users on import if this is not a production version of fastr warn_develop = True # [str] The hostname to expose the web app for web_hostname = "localhost" # [int] The port to expose the web app on web_port = 5000 # [str] The secret key to use for the flask web app web_secret_key = "VERYSECRETKEY!" # [int] Number of workers to use in a process pool process_pool_worker_number = 3 # [int] The interval in which the job checker will startto check for stale jobs slurm_job_check_interval = 30 # [str] The slurm partition to use slurm_partition = "" # [str] The PIM host to report to pim_host = "" # [float] The interval in which to send jobs to PIM pim_update_interval = 2.5 # [int] Maximum number of jobs that can be send to PIM in a single interval pim_batch_size = 100 # [bool] Setup PIM debug mode to send stdout stderr on job succes pim_debug = False ¶ Configuration of the fastr system
checksum
Module¶
This module contains a number of functions for checksumming files and objects
-
fastr.helpers.checksum.
checksum
(filepath, algorithm='md5', hasher=None, chunksize=32768)[source]¶ Generate the checksum of a file
Parameters: Returns: the checksum
Return type:
-
fastr.helpers.checksum.
checksum_directory
(directory, algorithm='md5', hasher=None)[source]¶ Generate the checksum of an entire directory
Parameters: Returns: the checksum
Return type:
-
fastr.helpers.checksum.
hashsum
(objects, hasher=None)[source]¶ Generate the md5 checksum of (a) python object(s)
Parameters: - objects – the objects to hash
- hasher – the hasher to use as a base
Returns: the hash generated
Return type:
classproperty
Module¶
Module containing the code to create class properties.
-
class
fastr.helpers.classproperty.
ClassPropertyDescriptor
(fget)[source]¶ Bases:
object
A descriptor that can act like a property for a class.
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.classproperty', '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function ClassPropertyDescriptor.__init__>, '__get__': <function ClassPropertyDescriptor.__get__>, '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>})¶
-
__module__
= 'fastr.helpers.classproperty'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
fastr.helpers.classproperty.
classproperty
(func)[source]¶ Decorator to create a “class property”
Parameters: func – the function to wrap Returns: a class property Return type: ClassPropertyDescriptor
clear_pycs
Module¶
A small tool to wipe all .pyc files from fastr
configmanager
Module¶
This module defines the Fastr Config class for managing the configuration of Fastr. The config object is stored directly in the fastr top-level module.
-
class
fastr.helpers.configmanager.
Config
(*configfiles)[source]¶ Bases:
object
Class contain the fastr configuration
-
DEFAULT_FIELDS
= {'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs/'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr', 'Fastr installation directory'), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'web_port': (<class 'int'>, 5000, 'The port to expose the web app on'), 'web_secret_key': (<class 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app')}¶
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': '\n Class contain the fastr configuration\n ', 'DEFAULT_FIELDS': {'logging_config': (<class 'dict'>, {}, 'Python logger config'), 'debug': (<class 'bool'>, False, 'Flag to enable/disable debugging'), 'logtype': (<class 'str'>, 'default', 'Type of logging to use'), 'loglevel': (<class 'int'>, 20, 'The log level to use (as int), INFO is 20, WARNING is 30, etc'), 'systemdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr', 'Fastr installation directory'), 'userdir': (<class 'str'>, '/home/docs/.fastr', 'Fastr user configuration directory', '$FASTRHOME or ~/.fastr'), 'logdir': (<class 'str'>, '/home/docs/.fastr/logs', 'Directory where the fastr logs will be placed', '$userdir/logs'), 'resourcesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources', 'Directory containing the fastr system resources', '$systemdir/resources'), 'examplesdir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples', 'Directory containing the fastr examples', '$systemdir/examples'), 'schemadir': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/schemas', 'Directory containing the fastr data schemas', '$systemdir/schemas'), 'executionscript': (<class 'str'>, '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/execution/executionscript.py', 'Execution script location', '$systemdir/execution/executionscript.py'), 'types_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/datatypes'], 'Directories to scan for datatypes', ['$userdir/datatypes', '$resourcedir/datatypes']), 'tools_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/tools'], 'Directories to scan for tools', ['$userdir/tools', '$resourcedir/tools']), 'networks_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/networks'], 'Directories to scan for networks', ['$userdir/networks', '$resourcedir/networks']), 'plugins_path': (<class 'list'>, ['/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/resources/plugins'], 'Directories to scan for plugins', ['$userdir/plugins', '$resourcedir/plugins']), 'mounts': (<class 'dict'>, {'tmp': '/tmp', 'example_data': '/home/docs/checkouts/readthedocs.org/user_builds/fastr/envs/3.0.1/lib/python3.6/site-packages/fastr/examples/data', 'home': '/home/docs/'}, 'A dictionary containing all mount points in the VFS system', {'tmp': '$TMPDIR', 'example_data': '$systemdir/examples/data', 'home': '~/'}), 'preferred_types': (<class 'list'>, [], 'A list indicating the order of the preferred types to use. First item is most preferred.'), 'protected_modules': (<class 'list'>, [], 'A list of modules in the environmnet modules that are protected against unloading'), 'execution_plugin': (<class 'str'>, 'ProcessPoolExecution', 'The default execution plugin to use'), 'reporting_plugins': (<class 'list'>, ['SimpleReport'], 'The reporting plugins to use, is a list of all plugins to be activated'), 'web_hostname': (<class 'str'>, 'localhost', 'The hostname to expose the web app for'), 'web_port': (<class 'int'>, 5000, 'The port to expose the web app on'), 'web_secret_key': (<class 'str'>, 'VERYSECRETKEY!', 'The secret key to use for the flask web app'), 'warn_develop': (<class 'bool'>, True, 'Warning users on import if this is not a production version of fastr'), 'source_job_limit': (<class 'int'>, 0, 'The number of source jobs allowed to run concurrently'), 'job_cleanup_level': (<class 'str'>, 'no_cleanup', 'The level of cleanup required, options: all, no_cleanup, non_failed', 'no_cleanup', <function Config.<lambda>>), 'filesynchelper_url': (<class 'str'>, '', 'Redis url e.g. redis://localhost:6379')}, '__init__': <function Config.__init__>, 'register_fields': <function Config.register_fields>, 'get_field': <function Config.get_field>, 'set_field': <function Config.set_field>, '_create_field_properties': <classmethod object>, '_field_property': <staticmethod object>, '__repr__': <function Config.__repr__>, 'read_config': <function Config.read_config>, 'read_config_string': <function Config.read_config_string>, 'web_url': <function Config.web_url>, '_update_logging': <function Config._update_logging>, '_deep_update': <function Config._deep_update>, '__dict__': <attribute '__dict__' of 'Config' objects>, '__weakref__': <attribute '__weakref__' of 'Config' objects>, 'debug': <property object>, 'examplesdir': <property object>, 'execution_plugin': <property object>, 'executionscript': <property object>, 'filesynchelper_url': <property object>, 'job_cleanup_level': <property object>, 'logdir': <property object>, 'logging_config': <property object>, 'loglevel': <property object>, 'logtype': <property object>, 'mounts': <property object>, 'networks_path': <property object>, 'plugins_path': <property object>, 'preferred_types': <property object>, 'protected_modules': <property object>, 'reporting_plugins': <property object>, 'resourcesdir': <property object>, 'schemadir': <property object>, 'source_job_limit': <property object>, 'systemdir': <property object>, 'tools_path': <property object>, 'types_path': <property object>, 'userdir': <property object>, 'warn_develop': <property object>, 'web_hostname': <property object>, 'web_port': <property object>, 'web_secret_key': <property object>, 'process_pool_worker_number': <property object>, 'slurm_job_check_interval': <property object>, 'slurm_partition': <property object>, 'pim_host': <property object>, 'pim_update_interval': <property object>, 'pim_batch_size': <property object>, 'pim_debug': <property object>})¶
-
__module__
= 'fastr.helpers.configmanager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
debug
¶
-
examplesdir
¶
-
execution_plugin
¶
-
executionscript
¶
-
filesynchelper_url
¶
-
job_cleanup_level
¶
-
logdir
¶
-
logging_config
¶
-
loglevel
¶
-
logtype
¶
-
mounts
¶
-
networks_path
¶
-
pim_batch_size
¶
-
pim_debug
¶
-
pim_host
¶
-
pim_update_interval
¶
-
plugins_path
¶
-
preferred_types
¶
-
process_pool_worker_number
¶
-
protected_modules
¶
-
read_config
(filename)[source]¶ Read a configuration and update the configuration object accordingly
Parameters: filename – the configuration file to read
-
read_config_files
= None¶ Trace of the config files read by this object
-
reporting_plugins
¶
-
resourcesdir
¶
-
schemadir
¶
-
slurm_job_check_interval
¶
-
slurm_partition
¶
-
source_job_limit
¶
-
systemdir
¶
-
tools_path
¶
-
types_path
¶
-
userdir
¶
-
warn_develop
¶
-
web_hostname
¶
-
web_port
¶
-
web_secret_key
¶
-
-
class
fastr.helpers.configmanager.
EmptyDefault
(data=None)[source]¶ Bases:
object
Empty defaultdict.
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.configmanager', '__doc__': ' Empty defaultdict. ', '__init__': <function EmptyDefault.__init__>, '__iadd__': <function EmptyDefault.__iadd__>, '__add__': <function EmptyDefault.__add__>, '__radd__': <function EmptyDefault.__radd__>, 'append': <function EmptyDefault.append>, 'prepend': <function EmptyDefault.prepend>, 'extend': <function EmptyDefault.extend>, 'update': <function EmptyDefault.update>, 'merge_default': <function EmptyDefault.merge_default>, '__getitem__': <function EmptyDefault.__getitem__>, '__setitem__': <function EmptyDefault.__setitem__>, '__delitem__': <function EmptyDefault.__delitem__>, 'aslist': <function EmptyDefault.aslist>, 'asdict': <function EmptyDefault.asdict>, '__dict__': <attribute '__dict__' of 'EmptyDefault' objects>, '__weakref__': <attribute '__weakref__' of 'EmptyDefault' objects>})¶
-
__module__
= 'fastr.helpers.configmanager'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
events
Module¶
-
class
fastr.helpers.events.
EventType
[source]¶ Bases:
enum.Enum
An enumeration.
-
__module__
= 'fastr.helpers.events'¶
-
job_updated
= 'job_updated'¶
-
log_record_emitted
= 'log_record_emitted'¶
-
run_finished
= 'run_finished'¶
-
run_started
= 'run_started'¶
-
-
class
fastr.helpers.events.
FastrLogEventHandler
(level=0)[source]¶ Bases:
logging.Handler
Logging handler that sends the log records into the event system
-
__module__
= 'fastr.helpers.events'¶
-
-
fastr.helpers.events.
emit_event
(event_type, data)[source]¶ Emit an event to all listeners :type event_type:
EventType
:param event_type: The type of event to emit :param data: The data object to send along
filesynchelper
Module¶
Some helper functions that aid with NFS file sync issues.
-
class
fastr.helpers.filesynchelper.
FileSyncHelper
[source]¶ Bases:
object
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.filesynchelper', '_namespace': 'filesynchelper', '_redis': None, '__init__': <function FileSyncHelper.__init__>, 'job_finished': <function FileSyncHelper.job_finished>, 'wait_for_job': <function FileSyncHelper.wait_for_job>, 'wait_for_pickle': <function FileSyncHelper.wait_for_pickle>, 'store': <function FileSyncHelper.store>, 'load': <function FileSyncHelper.load>, '_generate_key_for_string': <function FileSyncHelper._generate_key_for_string>, '_generate_hash_from_string': <function FileSyncHelper._generate_hash_from_string>, 'make_file_promise': <function FileSyncHelper.make_file_promise>, 'has_file_promise': <function FileSyncHelper.has_file_promise>, 'wait_for_vfs_url': <function FileSyncHelper.wait_for_vfs_url>, 'wait_for_file': <function FileSyncHelper.wait_for_file>, '_get_suburl_hashes': <function FileSyncHelper._get_suburl_hashes>, '_glob_dir': <function FileSyncHelper._glob_dir>, '_wait_for_file_and_suburls': <function FileSyncHelper._wait_for_file_and_suburls>, '__dict__': <attribute '__dict__' of 'FileSyncHelper' objects>, '__weakref__': <attribute '__weakref__' of 'FileSyncHelper' objects>, '__doc__': None})¶
-
__module__
= 'fastr.helpers.filesynchelper'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
iohelpers
Module¶
jsonschemaparser
Module¶
The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.
-
class
fastr.helpers.jsonschemaparser.
FastrRefResolver
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Bases:
jsonschema.validators.RefResolver
Adapted version of the RefResolver for handling inter-file references more to our liking
-
__init__
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Create a new FastrRefResolver
Parameters: - base_uri (str) – URI of the referring document
- referrer – the actual referring document
- store (dict) – a mapping from URIs to documents to cache
- cache_remote (bool) – whether remote refs should be cached after first resolution
- handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
-
__module__
= 'fastr.helpers.jsonschemaparser'¶
-
classmethod
from_schema
(schema, *args, **kwargs)[source]¶ Instantiate a RefResolver based on a schema
-
-
fastr.helpers.jsonschemaparser.
any_of_draft4
(validator, any_of, instance, schema)[source]¶ The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.
Parameters:
-
fastr.helpers.jsonschemaparser.
extend
(validator_cls)[source]¶ Extend the given
jsonschema.IValidator
with the Seep layer.
-
fastr.helpers.jsonschemaparser.
getblueprinter
(uri, blueprint=None)[source]¶ Instantiate the given data using the blueprinter.
Parameters: blueprint – a blueprint (JSON Schema with Seep properties)
-
fastr.helpers.jsonschemaparser.
items_prevalidate
(validator, items, instance, schema)[source]¶ The pre-validation function for items
Parameters:
-
fastr.helpers.jsonschemaparser.
not_draft4
(validator, not_schema, instance, schema)[source]¶ The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema
Parameters:
-
fastr.helpers.jsonschemaparser.
one_of_draft4
(validator, one_of, instance, schema)[source]¶ The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.
Parameters:
-
fastr.helpers.jsonschemaparser.
pattern_properties_prevalid
(validator, pattern_properties, instance, schema)[source]¶ The pre-validation function for patternProperties
Parameters:
-
fastr.helpers.jsonschemaparser.
properties_postvalidate
(validator, properties, instance, schema)[source]¶ # All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties
Parameters:
lazy_module
Module¶
This module contains the Manager class for Plugins in the fastr system
-
class
fastr.helpers.lazy_module.
LazyModule
(name, parent, plugin_manager)[source]¶ Bases:
module
A module that allows content to be loaded lazily from plugins. It generally is (almost) empty and gets (partially) populated when an attribute cannot be found. This allows lazy loading and plugins depending on other plugins.
-
__getattr__
(item)[source]¶ The getattr is called when getattribute does not return a value and is used as a fallback. In this case we try to find the value normally and will trigger the plugin manager if it cannot be found.
Parameters: item (str) – attribute to retrieve Returns: the requested attribute
-
__init__
(name, parent, plugin_manager)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
__module__
= 'fastr.helpers.lazy_module'¶
-
lockfile
Module¶
A module implenting a lock that ensures a directory is only being used by a single fastr run.
-
class
fastr.helpers.lockfile.
DirectoryLock
(directory)[source]¶ Bases:
object
A lock for a directory, it creates a directory to set the locked state and if successful writes the pid in a file inside that directory to claim the lock
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.lockfile', '__doc__': '\n A lock for a directory, it creates a directory to set the locked state and\n if successful writes the pid in a file inside that directory to claim the\n lock\n ', 'lock_dir_name': '.fastr.lock', 'pid_file_name': 'pid', '__init__': <function DirectoryLock.__init__>, 'lock_dir': <property object>, 'pid_file': <property object>, 'acquire': <function DirectoryLock.acquire>, 'release': <function DirectoryLock.release>, '__enter__': <function DirectoryLock.__enter__>, '__exit__': <function DirectoryLock.__exit__>, '__del__': <function DirectoryLock.__del__>, '__dict__': <attribute '__dict__' of 'DirectoryLock' objects>, '__weakref__': <attribute '__weakref__' of 'DirectoryLock' objects>})¶
-
__module__
= 'fastr.helpers.lockfile'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
lock_dir
¶
-
lock_dir_name
= '.fastr.lock'¶
-
pid_file
¶
-
pid_file_name
= 'pid'¶
-
procutils
Module¶
rest_generation
Module¶
schematotable
Module¶
A module to generate reStructuredText tables from json schema files
-
class
fastr.helpers.schematotable.
SchemaPrinter
(schema, skipfirst=False)[source]¶ Bases:
object
Object that create a table in reStructuedText from a json schema
-
__dict__
= mappingproxy({'__module__': 'fastr.helpers.schematotable', '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function SchemaPrinter.__init__>, '__str__': <function SchemaPrinter.__str__>, 'descend': <function SchemaPrinter.descend>, 'parse': <function SchemaPrinter.parse>, 'printlines': <function SchemaPrinter.printlines>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>})¶
-
__module__
= 'fastr.helpers.schematotable'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
sysinfo
Module¶
This module contains function to help gather system information use for the provenance of the Job execution.
-
fastr.helpers.sysinfo.
get_cpu_usage
()[source]¶ Get the current CPU usage
Returns: CPU usage info Return type: dict
-
fastr.helpers.sysinfo.
get_drmaa_info
()[source]¶ Get information about the SGE cluster (if applicable)
Returns: cluster info Return type: dict
-
fastr.helpers.sysinfo.
get_hostinfo
()[source]¶ Get all information about the current host machine
Returns: host info Return type: dict
-
fastr.helpers.sysinfo.
get_memory_usage
()[source]¶ Get the current memory usage
Returns: memory usage info Return type: dict
-
fastr.helpers.sysinfo.
get_mounts
()[source]¶ Get the current mounts known on the system
Returns: mount info Return type: dict
-
fastr.helpers.sysinfo.
get_os
()[source]¶ Get information about the OS
Returns: OS information Return type: dict
-
fastr.helpers.sysinfo.
get_processes
()[source]¶ Get a list of all currently running processes
Returns: process information Return type: list
-
fastr.helpers.sysinfo.
get_python
()[source]¶ Get information about the currently used Python implementation
Returns: python info Return type: dict
-
fastr.helpers.sysinfo.
get_sysinfo
()[source]¶ Get system information (cpu, memory, mounts and users)
Returns: system information Return type: dict
xmltodict
Module¶
This module contains tool for converting python dictionaries into XML object and vice-versa.
-
fastr.helpers.xmltodict.
dump
(data, filehandle)[source]¶ Write a dict to an XML file
Parameters: - data – data to write
- filehandle – file handle to write to
-
fastr.helpers.xmltodict.
dumps
(data)[source]¶ Write a dict to an XML string
Parameters: data – data to write Returns: the XML data Return type: str