utils Package¶
utils
Package¶
A collections of utils for fastr (command line tools or non-core functionality)
checksum
Module¶
This module contains a number of functions for checksumming files and objects
-
fastr.utils.checksum.
checksum
(filepath, algorithm='md5', hasher=None)[source]¶ Generate the checksum of a file
Parameters: Returns: the checksum
Return type:
-
fastr.utils.checksum.
checksum_directory
(directory, algorithm='md5', hasher=None)[source]¶ Generate the checksum of an entire directory
Parameters: Returns: the checksum
Return type:
-
fastr.utils.checksum.
hashsum
(objects, hasher=None)[source]¶ Generate the md5 checksum of (a) python object(s)
Parameters: - objects – the objects to hash
- hasher – the hasher to use as a base
Returns: the hash generated
Return type:
classproperty
Module¶
Module containing the code to create class properties.
-
class
fastr.utils.classproperty.
ClassPropertyDescriptor
(fget)[source]¶ Bases:
object
A descriptor that can act like a property for a class.
-
__dict__
= dict_proxy({'__module__': 'fastr.utils.classproperty', '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>, '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function __init__>, '__get__': <function __get__>})¶
-
__module__
= 'fastr.utils.classproperty'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
fastr.utils.classproperty.
classproperty
(func)[source]¶ Decorator to create a “class property”
Parameters: func – the function to wrap Returns: a class property Return type: ClassPropertyDescriptor
clear_pycs
Module¶
A small tool to wipe all .pyc files from fastr
compare
Module¶
Module to compare various fastr specific things such as a execution directory or a reference directory.
dicteq
Module¶
Some helper function to compare dictionaries and find the parts of the dict that are different. This is mostly to help in debugging.
-
fastr.utils.dicteq.
dicteq
(self, other)[source]¶ Compare two dicts for equality
Parameters: - self – the first object to compare
- other – the oth
Returns:
-
fastr.utils.dicteq.
diffdict
(self, other, path=None, visited=None)[source]¶ Find the differences in two dictionaries.
Parameters: Returns: list of messages indicating the differences
Return type:
-
fastr.utils.dicteq.
diffobj
(self, other, path=None, visited=None)[source]¶ Compare two objects by comparing their __dict__ entries
Parameters: - self – the first object to compare
- other – other objects to compare
- path (list) – the path for nested dicts (too keep track of recursion)
Returns: list of messages
Return type:
-
fastr.utils.dicteq.
diffobj_str
(self, other)[source]¶ Compare two objects by comparing their __dict__ entries, but returns the differences in a single string ready for logging.
Parameters: - self – the first object to compare
- other – other object to compare to
Returns: the description of the differences
Return type:
iohelpers
Module¶
jsonschemaparser
Module¶
The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.
-
class
fastr.utils.jsonschemaparser.
FastrRefResolver
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Bases:
jsonschema.validators.RefResolver
Adapted version of the RefResolver for handling inter-file references more to our liking
-
__init__
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Create a new FastrRefResolver
Parameters: - base_uri (str) – URI of the referring document
- referrer – the actual referring document
- store (dict) – a mapping from URIs to documents to cache
- cache_remote (bool) – whether remote refs should be cached after first resolution
- handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
-
__module__
= 'fastr.utils.jsonschemaparser'¶
-
classmethod
from_schema
(schema, *args, **kwargs)[source]¶ Instantiate a RefResolver based on a schema
-
-
fastr.utils.jsonschemaparser.
any_of_draft4
(validator, any_of, instance, schema)[source]¶ The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.
Parameters:
-
fastr.utils.jsonschemaparser.
extend
(validator_cls)[source]¶ Extend the given
jsonschema.IValidator
with the Seep layer.
-
fastr.utils.jsonschemaparser.
getblueprinter
(uri, blueprint=None)[source]¶ Instantiate the given data using the blueprinter.
Parameters: blueprint – a blueprint (JSON Schema with Seep properties)
-
fastr.utils.jsonschemaparser.
items_prevalidate
(validator, items, instance, schema)[source]¶ The pre-validation function for items
Parameters:
-
fastr.utils.jsonschemaparser.
not_draft4
(validator, not_schema, instance, schema)[source]¶ The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema
Parameters:
-
fastr.utils.jsonschemaparser.
one_of_draft4
(validator, one_of, instance, schema)[source]¶ The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.
Parameters:
-
fastr.utils.jsonschemaparser.
pattern_properties_prevalid
(validator, pattern_properties, instance, schema)[source]¶ The pre-validation function for patternProperties
Parameters:
-
fastr.utils.jsonschemaparser.
properties_postvalidate
(validator, properties, instance, schema)[source]¶ # All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties
Parameters:
multiprocesswrapper
Module¶
pim_publisher
Module¶
-
class
fastr.utils.pim_publisher.
PimPublisher
(uri=None)[source]¶ Bases:
object
Class to publish to PIM
-
PIM_STATUS_MAPPING
= {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}¶
-
__dict__
= dict_proxy({'__dict__': <attribute '__dict__' of 'PimPublisher' objects>, '__module__': 'fastr.utils.pim_publisher', '__init__': <function __init__>, 'PIM_STATUS_MAPPING': {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}, 'pim_register_run': <function pim_register_run>, 'pim_serialize_network': <staticmethod object>, '__weakref__': <attribute '__weakref__' of 'PimPublisher' objects>, '__doc__': '\n Class to publish to PIM\n ', 'pim_update_status': <function pim_update_status>})¶
-
__module__
= 'fastr.utils.pim_publisher'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
procutils
Module¶
rest_generation
Module¶
schematotable
Module¶
A module to generate reStructuredText tables from json schema files
-
class
fastr.utils.schematotable.
SchemaPrinter
(schema, skipfirst=False)[source]¶ Bases:
object
Object that create a table in reStructuedText from a json schema
-
__dict__
= dict_proxy({'parse': <function parse>, '__module__': 'fastr.utils.schematotable', 'descend': <function descend>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>, '__str__': <function __str__>, 'printlines': <function printlines>, '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function __init__>})¶
-
__module__
= 'fastr.utils.schematotable'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
sysinfo
Module¶
This module contains function to help gather system information use for the provenance of the Job execution.
-
fastr.utils.sysinfo.
get_cpu_usage
()[source]¶ Get the current CPU usage
Returns: CPU usage info Return type: dict
-
fastr.utils.sysinfo.
get_drmaa_info
()[source]¶ Get information about the SGE cluster (if applicable)
Returns: cluster info Return type: dict
-
fastr.utils.sysinfo.
get_hostinfo
()[source]¶ Get all information about the current host machine
Returns: host info Return type: dict
-
fastr.utils.sysinfo.
get_memory_usage
()[source]¶ Get the current memory usage
Returns: memory usage info Return type: dict
-
fastr.utils.sysinfo.
get_mounts
()[source]¶ Get the current mounts known on the system
Returns: mount info Return type: dict
-
fastr.utils.sysinfo.
get_os
()[source]¶ Get information about the OS
Returns: OS information Return type: dict
-
fastr.utils.sysinfo.
get_processes
()[source]¶ Get a list of all currently running processes
Returns: process information Return type: list
-
fastr.utils.sysinfo.
get_python
()[source]¶ Get information about the currently used Python implementation
Returns: python info Return type: dict
-
fastr.utils.sysinfo.
get_sysinfo
()[source]¶ Get system information (cpu, memory, mounts and users)
Returns: system information Return type: dict
verify
Module¶
xmltodict
Module¶
This module contains tool for converting python dictionaries into XML object and vice-versa.
-
fastr.utils.xmltodict.
dump
(data, filehandle)[source]¶ Write a dict to an XML file
Parameters: - data – data to write
- filehandle – file handle to write to
-
fastr.utils.xmltodict.
dumps
(data)[source]¶ Write a dict to an XML string
Parameters: data – data to write Returns: the XML data Return type: str