utils Package¶
utils
Package¶
A collections of utils for fastr (command line tools or non-core functionality)
checksum
Module¶
This module contains a number of functions for checksumming files and objects
-
fastr.utils.checksum.
checksum
(filepath, algorithm='md5', hasher=None)[source]¶ Generate the checksum of a file
Parameters: Returns: the checksum
Return type:
-
fastr.utils.checksum.
checksum_directory
(directory, algorithm='md5', hasher=None)[source]¶ Generate the checksum of an entire directory
Parameters: Returns: the checksum
Return type:
-
fastr.utils.checksum.
hashsum
(objects, hasher=None)[source]¶ Generate the md5 checksum of (a) python object(s)
Parameters: - objects – the objects to hash
- hasher – the hasher to use as a base
Returns: the hash generated
Return type:
classproperty
Module¶
Module containing the code to create class properties.
-
class
fastr.utils.classproperty.
ClassPropertyDescriptor
(fget)[source]¶ Bases:
object
A descriptor that can act like a property for a class.
-
__dict__
= dict_proxy({'__module__': 'fastr.utils.classproperty', '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>, '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function __init__>, '__get__': <function __get__>})¶
-
__module__
= 'fastr.utils.classproperty'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
fastr.utils.classproperty.
classproperty
(func)[source]¶ Decorator to create a “class property”
Parameters: func – the function to wrap Returns: a class property Return type: ClassPropertyDescriptor
clear_pycs
Module¶
A small tool to wipe all .pyc files from fastr
compare
Module¶
Module to compare various fastr specific things such as a execution directory or a reference directory.
-
fastr.utils.compare.
compare_set
(set1, set2, path, sub_compare_func, f_args=None, f_kwargs=None)[source]¶ Compare two sets and dispatch each item to a sub comparison function
Parameters: - set1 (Iterable) – first set of items
- set2 (Iterable) – second set of items
- path (str) – identifier of the data location
- sub_compare_func – function to apply to items
- f_args – args to pass to sub_compare_func
- f_kwargs – kwargs to pass to sub_compare_func
Returns: generator that iterates over the differences
Return type: generator
dicteq
Module¶
Some helper function to compare dictionaries and find the parts of the dict that are different. This is mostly to help in debugging.
-
fastr.utils.dicteq.
dicteq
(self, other)[source]¶ Compare two dicts for equality
Parameters: - self – the first object to compare
- other – the oth
Returns:
-
fastr.utils.dicteq.
diffdict
(self, other, path=None, visited=None)[source]¶ Find the differences in two dictionaries.
Parameters: - self – the first object to compare
- other (dict) – other dictionary
- path (list) – the path for nested dicts (too keep track of recursion)
Returns: list of messages indicating the differences
Return type: list
-
fastr.utils.dicteq.
diffobj
(self, other, path=None, visited=None)[source]¶ Compare two objects by comparing their __dict__ entries
Parameters: - self – the first object to compare
- other – other objects to compare
- path (list) – the path for nested dicts (too keep track of recursion)
Returns: list of messages
Return type: list
-
fastr.utils.dicteq.
diffobj_str
(self, other)[source]¶ Compare two objects by comparing their __dict__ entries, but returns the differences in a single string ready for logging.
Parameters: - self – the first object to compare
- other – other object to compare to
Returns: the description of the differences
Return type:
filesynchelper
Module¶
Some helper functions that aid with NFS file sync issues.
iohelpers
Module¶
jsonschemaparser
Module¶
The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.
-
class
fastr.utils.jsonschemaparser.
FastrRefResolver
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Bases:
jsonschema.validators.RefResolver
Adapted version of the RefResolver for handling inter-file references more to our liking
-
__init__
(base_uri, referrer, store=(), cache_remote=True, handlers=())[source]¶ Create a new FastrRefResolver
Parameters: - base_uri (str) – URI of the referring document
- referrer – the actual referring document
- store (dict) – a mapping from URIs to documents to cache
- cache_remote (bool) – whether remote refs should be cached after first resolution
- handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them
-
__module__
= 'fastr.utils.jsonschemaparser'¶
-
classmethod
from_schema
(schema, *args, **kwargs)[source]¶ Instantiate a RefResolver based on a schema
-
-
fastr.utils.jsonschemaparser.
any_of_draft4
(validator, any_of, instance, schema)[source]¶ The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.
Parameters:
-
fastr.utils.jsonschemaparser.
extend
(validator_cls)[source]¶ Extend the given
jsonschema.IValidator
with the Seep layer.
-
fastr.utils.jsonschemaparser.
getblueprinter
(uri, blueprint=None)[source]¶ Instantiate the given data using the blueprinter.
Parameters: blueprint – a blueprint (JSON Schema with Seep properties)
-
fastr.utils.jsonschemaparser.
items_prevalidate
(validator, items, instance, schema)[source]¶ The pre-validation function for items
Parameters:
-
fastr.utils.jsonschemaparser.
not_draft4
(validator, not_schema, instance, schema)[source]¶ The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema
Parameters:
-
fastr.utils.jsonschemaparser.
one_of_draft4
(validator, one_of, instance, schema)[source]¶ The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.
Parameters:
-
fastr.utils.jsonschemaparser.
pattern_properties_prevalid
(validator, pattern_properties, instance, schema)[source]¶ The pre-validation function for patternProperties
Parameters:
-
fastr.utils.jsonschemaparser.
properties_postvalidate
(validator, properties, instance, schema)[source]¶ # All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties
Parameters:
multiprocesswrapper
Module¶
pim_publisher
Module¶
-
class
fastr.utils.pim_publisher.
BasePimAPI
[source]¶ Bases:
object
-
__abstractmethods__
= frozenset(['pim_register_run', 'pim_update_status'])¶
-
__dict__
= dict_proxy({'_abc_cache': <_weakrefset.WeakSet object>, '__module__': 'fastr.utils.pim_publisher', '__metaclass__': <class 'abc.ABCMeta'>, 'pim_register_run': <function pim_register_run>, '_abc_registry': <_weakrefset.WeakSet object>, '__abstractmethods__': frozenset(['pim_register_run', 'pim_update_status']), 'pim_update_status': <function pim_update_status>, '_abc_negative_cache_version': 36, '_abc_negative_cache': <_weakrefset.WeakSet object>, '__dict__': <attribute '__dict__' of 'BasePimAPI' objects>, '__weakref__': <attribute '__weakref__' of 'BasePimAPI' objects>, '__doc__': None})¶
-
__metaclass__
¶ alias of
abc.ABCMeta
-
__module__
= 'fastr.utils.pim_publisher'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
pim_register_run
(network)[source]¶ Send the basic Network layout to PIM and register the run.
Parameters: network (NetworkRun) – The network run to register to PIM
-
pim_update_status
(network, job)[source]¶ Update the status of a job
Parameters: - network (NetworkRun) – The network run the job belongs to
- job (Job) – The job which to update
-
-
class
fastr.utils.pim_publisher.
PimAPI_v1
(uri=None)[source]¶ Bases:
object
Class to publish to PIM
-
PIM_STATUS_MAPPING
= {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}¶
-
__dict__
= dict_proxy({'__dict__': <attribute '__dict__' of 'PimAPI_v1' objects>, '__module__': 'fastr.utils.pim_publisher', '__init__': <function __init__>, 'PIM_STATUS_MAPPING': {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}, 'pim_register_run': <function pim_register_run>, 'pim_serialize_network': <function pim_serialize_network>, '__weakref__': <attribute '__weakref__' of 'PimAPI_v1' objects>, '__doc__': '\n Class to publish to PIM\n ', 'pim_update_status': <function pim_update_status>})¶
-
__module__
= 'fastr.utils.pim_publisher'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
fastr.utils.pim_publisher.
PimAPI_v2
(uri=None)[source]¶ Bases:
object
Class to publish to PIM
-
NODE_CLASSES
= {'ConstantNodeRun': 'constant', 'NodeRun': 'node', 'SinkNodeRun': 'sink', 'SourceNodeRun': 'source'}¶
-
PIM_STATUS_MAPPING
= {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 1, <JobState.nonexistent: ('nonexistent', 'idle', False)>: 5, <JobState.running: ('running', 'in_progress', False)>: 1, <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 1, <JobState.hold: ('hold', 'idle', False)>: 0, <JobState.created: ('created', 'idle', False)>: 0, <JobState.finished: ('finished', 'done', False)>: 2, <JobState.failed: ('failed', 'done', True)>: 3, <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 1, <JobState.cancelled: ('cancelled', 'done', True)>: 4, <JobState.queued: ('queued', 'idle', False)>: 0}¶
-
STATUS_TYPES
= [{'color': '#aaccff', 'description': 'Jobs that are waiting for input', 'title': 'idle'}, {'color': '#daa520', 'description': 'Jobs that are running', 'title': 'running'}, {'color': '#23b22f', 'description': 'Jobs that finished successfully', 'title': 'success'}, {'color': '#dd3311', 'description': 'Jobs that have failed', 'title': 'failed'}, {'color': '#334477', 'description': 'Jobs which were cancelled', 'title': 'cancelled'}, {'color': '#ccaa99', 'description': 'Jobs with an undefined state', 'title': 'undefined'}]¶
-
__dict__
= dict_proxy({'STATUS_TYPES': [{'color': '#aaccff', 'description': 'Jobs that are waiting for input', 'title': 'idle'}, {'color': '#daa520', 'description': 'Jobs that are running', 'title': 'running'}, {'color': '#23b22f', 'description': 'Jobs that finished successfully', 'title': 'success'}, {'color': '#dd3311', 'description': 'Jobs that have failed', 'title': 'failed'}, {'color': '#334477', 'description': 'Jobs which were cancelled', 'title': 'cancelled'}, {'color': '#ccaa99', 'description': 'Jobs with an undefined state', 'title': 'undefined'}], '__module__': 'fastr.utils.pim_publisher', 'pim_serialize_macro': <function pim_serialize_macro>, 'pim_register_run': <function pim_register_run>, 'pim_serialize_network': <function pim_serialize_network>, 'NODE_CLASSES': {'ConstantNodeRun': 'constant', 'SourceNodeRun': 'source', 'NodeRun': 'node', 'SinkNodeRun': 'sink'}, 'pim_update_status': <function pim_update_status>, 'pim_serialize_node': <function pim_serialize_node>, 'pim_serialize_link': <function pim_serialize_link>, 'PIM_STATUS_MAPPING': {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 1, <JobState.nonexistent: ('nonexistent', 'idle', False)>: 5, <JobState.running: ('running', 'in_progress', False)>: 1, <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 1, <JobState.hold: ('hold', 'idle', False)>: 0, <JobState.created: ('created', 'idle', False)>: 0, <JobState.finished: ('finished', 'done', False)>: 2, <JobState.failed: ('failed', 'done', True)>: 3, <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 1, <JobState.cancelled: ('cancelled', 'done', True)>: 4, <JobState.queued: ('queued', 'idle', False)>: 0}, '__dict__': <attribute '__dict__' of 'PimAPI_v2' objects>, '__weakref__': <attribute '__weakref__' of 'PimAPI_v2' objects>, '__doc__': '\n Class to publish to PIM\n ', '__init__': <function __init__>})¶
-
__module__
= 'fastr.utils.pim_publisher'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
fastr.utils.pim_publisher.
PimPublisher
(uri=None)[source]¶ Bases:
object
-
SUPPORTED_APIS
= {1: <class 'fastr.utils.pim_publisher.PimAPI_v1'>, 2: <class 'fastr.utils.pim_publisher.PimAPI_v2'>}¶
-
__dict__
= dict_proxy({'__module__': 'fastr.utils.pim_publisher', 'pim_register_run': <function pim_register_run>, 'pim_update_status': <function pim_update_status>, 'SUPPORTED_APIS': {1: <class 'fastr.utils.pim_publisher.PimAPI_v1'>, 2: <class 'fastr.utils.pim_publisher.PimAPI_v2'>}, '__dict__': <attribute '__dict__' of 'PimPublisher' objects>, '__weakref__': <attribute '__weakref__' of 'PimPublisher' objects>, '__doc__': None, '__init__': <function __init__>})¶
-
__module__
= 'fastr.utils.pim_publisher'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
procutils
Module¶
rest_generation
Module¶
-
fastr.utils.rest_generation.
create_rest_table
(data, headers)[source]¶ Create a ReST table from data. The data should be a list of columns and the headers should be a list of column names.
Parameters: - data (list) – List of lists/tuples representing the columns
- headers (list) – List of strings for the column names
Returns: a string representing the table in ReST
Return type:
schematotable
Module¶
A module to generate reStructuredText tables from json schema files
-
class
fastr.utils.schematotable.
SchemaPrinter
(schema, skipfirst=False)[source]¶ Bases:
object
Object that create a table in reStructuedText from a json schema
-
__dict__
= dict_proxy({'parse': <function parse>, '__module__': 'fastr.utils.schematotable', 'descend': <function descend>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>, '__str__': <function __str__>, 'printlines': <function printlines>, '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function __init__>})¶
-
__module__
= 'fastr.utils.schematotable'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
sysinfo
Module¶
This module contains function to help gather system information use for the provenance of the Job execution.
-
fastr.utils.sysinfo.
get_cpu_usage
()[source]¶ Get the current CPU usage
Returns: CPU usage info Return type: dict
-
fastr.utils.sysinfo.
get_drmaa_info
()[source]¶ Get information about the SGE cluster (if applicable)
Returns: cluster info Return type: dict
-
fastr.utils.sysinfo.
get_hostinfo
()[source]¶ Get all information about the current host machine
Returns: host info Return type: dict
-
fastr.utils.sysinfo.
get_memory_usage
()[source]¶ Get the current memory usage
Returns: memory usage info Return type: dict
-
fastr.utils.sysinfo.
get_mounts
()[source]¶ Get the current mounts known on the system
Returns: mount info Return type: dict
-
fastr.utils.sysinfo.
get_os
()[source]¶ Get information about the OS
Returns: OS information Return type: dict
-
fastr.utils.sysinfo.
get_processes
()[source]¶ Get a list of all currently running processes
Returns: process information Return type: list
-
fastr.utils.sysinfo.
get_python
()[source]¶ Get information about the currently used Python implementation
Returns: python info Return type: dict
-
fastr.utils.sysinfo.
get_sysinfo
()[source]¶ Get system information (cpu, memory, mounts and users)
Returns: system information Return type: dict
verify
Module¶
xmltodict
Module¶
This module contains tool for converting python dictionaries into XML object and vice-versa.
-
fastr.utils.xmltodict.
dump
(data, filehandle)[source]¶ Write a dict to an XML file
Parameters: - data – data to write
- filehandle – file handle to write to
-
fastr.utils.xmltodict.
dumps
(data)[source]¶ Write a dict to an XML string
Parameters: data – data to write Returns: the XML data Return type: str