utils Package

utils Package

A collections of utils for fastr (command line tools or non-core functionality)

checksum Module

This module contains a number of functions for checksumming files and objects

fastr.utils.checksum.checksum(filepath, algorithm='md5', hasher=None)

Generate the checksum of a file

Parameters:

• filepath (str, list) – path of the file(s) to checksum
• algorithm (str) – the algorithm to use
• hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)

Returns:

the checksum

Return type:

str

fastr.utils.checksum.checksum_directory(directory, algorithm='md5', hasher=None)

Generate the checksum of an entire directory

Parameters:

• directory (str) – path of the file(s) to checksum
• algorithm (str) – the algorithm to use
• hasher (_hashlib.HASH) – a hasher to continue updating (rather then creating a new one)

Returns:

the checksum

Return type:

str

fastr.utils.checksum.hashsum(objects, hasher=None)

Generate the md5 checksum of (a) python object(s)

Parameters:

• objects – the objects to hash
• hasher – the hasher to use as a base

Returns:

the hash generated

Return type:

str

fastr.utils.checksum.md5_checksum(filepath)

Generate the md5 checksum of a file

Parameters:filepath (str, list) – path of the file(s) to checksum Returns:the checksum Return type:str

fastr.utils.checksum.sha1_checksum(filepath)

Generate the sha1 checksum of a file

Parameters:filepath (str, list) – path of the file(s) to checksum Returns:the checksum Return type:str

classproperty Module

Module containing the code to create class properties.

class fastr.utils.classproperty.ClassPropertyDescriptor(fget)

Bases: object

A descriptor that can act like a property for a class.

__dict__ = dict_proxy({'__module__': 'fastr.utils.classproperty', '__dict__': <attribute '__dict__' of 'ClassPropertyDescriptor' objects>, '__weakref__': <attribute '__weakref__' of 'ClassPropertyDescriptor' objects>, '__doc__': '\n A descriptor that can act like a property for a class.\n ', '__init__': <function __init__>, '__get__': <function __get__>})

__get__(obj, cls=None)

__init__(fget)

__module__ = 'fastr.utils.classproperty'

__weakref__

list of weak references to the object (if defined)

fastr.utils.classproperty.classproperty(func)

Decorator to create a “class property”

Parameters:func – the function to wrap Returns:a class property Return type:ClassPropertyDescriptor

clear_pycs Module

A small tool to wipe all .pyc files from fastr

fastr.utils.clear_pycs.dir_list(directory)

Find all .pyc files

Parameters:directory (str) – directory to search Returns:all .pyc files Return type:list

fastr.utils.clear_pycs.main()

Main entry poitn

compare Module

Module to compare various fastr specific things such as a execution directory or a reference directory.

fastr.utils.compare.compare_execution_dir(path1, path2)

dicteq Module

Some helper function to compare dictionaries and find the parts of the dict that are different. This is mostly to help in debugging.

fastr.utils.dicteq.dicteq(self, other)

Compare two dicts for equality

Parameters:

• self – the first object to compare
• other – the oth

Returns:

fastr.utils.dicteq.diffdict(self, other, path=None, visited=None)

Find the differences in two dictionaries.

Parameters:

• self – the first object to compare
• other (dict) – other dictionary
• path (list) – the path for nested dicts (too keep track of recursion)

Returns:

list of messages indicating the differences

Return type:

list

fastr.utils.dicteq.diffobj(self, other, path=None, visited=None)

Compare two objects by comparing their __dict__ entries

Parameters:

• self – the first object to compare
• other – other objects to compare
• path (list) – the path for nested dicts (too keep track of recursion)

Returns:

list of messages

Return type:

list

fastr.utils.dicteq.diffobj_str(self, other)

Compare two objects by comparing their __dict__ entries, but returns the differences in a single string ready for logging.

Parameters:

• self – the first object to compare
• other – other object to compare to

Returns:

the description of the differences

Return type:

str

gettools Module

fastr.utils.gettools.main()

iohelpers Module

fastr.utils.iohelpers.load_gpickle(path, retry_scheme=None)

fastr.utils.iohelpers.load_json(path)

fastr.utils.iohelpers.save_gpickle(path, data)

fastr.utils.iohelpers.save_json(path, data, indent=2)

jsonschemaparser Module

The JSON schema parser validates a json data structure and if possible casts data to the correct type and fills out default values. The result in a valid document that can be used to construct objects.

class fastr.utils.jsonschemaparser.FastrRefResolver(base_uri, referrer, store=(), cache_remote=True, handlers=())

Bases: jsonschema.validators.RefResolver

Adapted version of the RefResolver for handling inter-file references more to our liking

__init__(base_uri, referrer, store=(), cache_remote=True, handlers=())

Create a new FastrRefResolver

Parameters:

• base_uri (str) – URI of the referring document
• referrer – the actual referring document
• store (dict) – a mapping from URIs to documents to cache
• cache_remote (bool) – whether remote refs should be cached after first resolution
• handlers (dict) – a mapping from URI schemes to functions that should be used to retrieve them

__module__ = 'fastr.utils.jsonschemaparser'

classmethod from_schema(schema, *args, **kwargs)

Instantiate a RefResolver based on a schema

static readfastrschema(name)

Open a json file based on a fastr:// url that points to a file in the fastr.schemadir

Parameters:name (str) – the url of the file to open Returns:the resulting json schema data

static readfile(filename)

Open a json file based on a simple filename

Parameters:filename (str) – the path of the file to read Returns:the resulting json schema data

fastr.utils.jsonschemaparser.any_of_draft4(validator, any_of, instance, schema)

The oneOf directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Then for all the schemas that are valid, we perform the validation on the actual data so that only the valid subschemas will effect the data.

Parameters:

• validator – the json schema validator
• any_of (dict) – the current oneOf
• instance – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.extend(validator_cls)

Extend the given jsonschema.IValidator with the Seep layer.

fastr.utils.jsonschemaparser.getblueprinter(uri, blueprint=None)

Instantiate the given data using the blueprinter.

Parameters:blueprint – a blueprint (JSON Schema with Seep properties)

fastr.utils.jsonschemaparser.items_prevalidate(validator, items, instance, schema)

The pre-validation function for items

Parameters:

• validator – the json schema validator
• items (dict) – the current items
• instance – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.not_draft4(validator, not_schema, instance, schema)

The not needs to use a temporary copy of the instance, not to change the instance with the invalid schema

Parameters:

• validator – the json schema validator
• not_schema (dict) – the current oneOf
• instance – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.one_of_draft4(validator, one_of, instance, schema)

The one_of directory needs to be done stepwise, because a validation even if it fails will try to change types / set defaults etc. Therefore we first create a copy of the data per subschema and test if they match. Once we found a proper match, we only validate that branch on the real data so that only the valid piece of schema will effect the data.

Parameters:

• validator – the json schema validator
• one_of (dict) – the current one_of
• instance – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.pattern_properties_prevalid(validator, pattern_properties, instance, schema)

The pre-validation function for patternProperties

Parameters:

• validator – the json schema validator
• pattern_properties (dict) – the current patternProperties
• instance (dict) – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.properties_postvalidate(validator, properties, instance, schema)

# All arguments must be used because this function is called like this # pylint: disable=unused-argument The post-validation function for properties

Parameters:

• validator – the json schema validator
• properties (dict) – the current properties
• instance – the current object instance
• schema (dict) – the current json schema

fastr.utils.jsonschemaparser.properties_prevalidate(validator, properties, instance, schema)

The pre-validation function for properties

Parameters:

• validator – the json schema validator
• properties (dict) – the current properties
• instance – the current object instance
• schema (dict) – the current json schema

multiprocesswrapper Module

fastr.utils.multiprocesswrapper.function_wrapper(filepath, fnc_name, *args, **kwargs)

pim_publisher Module

class fastr.utils.pim_publisher.PimPublisher(uri=None)

Bases: object

Class to publish to PIM

PIM_STATUS_MAPPING = {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}

__dict__ = dict_proxy({'__dict__': <attribute '__dict__' of 'PimPublisher' objects>, '__module__': 'fastr.utils.pim_publisher', '__init__': <function __init__>, 'PIM_STATUS_MAPPING': {<JobState.execution_done: ('execution_done', 'in_progress', False)>: 'running', <JobState.nonexistent: ('nonexistent', 'idle', False)>: 'unknown', <JobState.running: ('running', 'in_progress', False)>: 'running', <JobState.processing_callback: ('processing_callback', 'in_progress', False)>: 'running', <JobState.hold: ('hold', 'idle', False)>: 'idle', <JobState.created: ('created', 'idle', False)>: 'idle', <JobState.finished: ('finished', 'done', False)>: 'success', <JobState.failed: ('failed', 'done', True)>: 'failed', <JobState.execution_failed: ('execution_failed', 'in_progress', True)>: 'running', <JobState.cancelled: ('cancelled', 'done', True)>: 'failed', <JobState.queued: ('queued', 'idle', False)>: 'idle'}, 'pim_register_run': <function pim_register_run>, 'pim_serialize_network': <staticmethod object>, '__weakref__': <attribute '__weakref__' of 'PimPublisher' objects>, '__doc__': '\n Class to publish to PIM\n ', 'pim_update_status': <function pim_update_status>})

__init__(uri=None)

__module__ = 'fastr.utils.pim_publisher'

__weakref__

list of weak references to the object (if defined)

pim_register_run(network)

static pim_serialize_network(network)

Serialize Network in the correct for to use with PIM.

Returns:json data for PIM

pim_update_status(job)

procutils Module

fastr.utils.procutils.which(name)

Find executable by name on the PATH, returns the executable that will be

found in case it is used for a Popen call

rest_generation Module

fastr.utils.rest_generation.create_rest_table(data, headers)

Create a ReST table from data. The data should be a list of columns and the headers should be a list of column names.

Parameters:

• data (list) – List of lists/tuples representing the columns
• headers (list) – List of strings for the column names

Returns:

a string representing the table in ReST

Return type:

str

schematotable Module

A module to generate reStructuredText tables from json schema files

class fastr.utils.schematotable.SchemaPrinter(schema, skipfirst=False)

Bases: object

Object that create a table in reStructuedText from a json schema

__dict__ = dict_proxy({'parse': <function parse>, '__module__': 'fastr.utils.schematotable', 'descend': <function descend>, '__dict__': <attribute '__dict__' of 'SchemaPrinter' objects>, '__weakref__': <attribute '__weakref__' of 'SchemaPrinter' objects>, '__str__': <function __str__>, 'printlines': <function printlines>, '__doc__': '\n Object that create a table in reStructuedText from a json schema\n ', '__init__': <function __init__>})

__init__(schema, skipfirst=False)

Create the printer object

Parameters:

• schema (dict) – the json schema to print
• skipfirst (bool) – flag to indicate that the first line should not be printed

__module__ = 'fastr.utils.schematotable'

__str__()

String representation of json schema (that is the printed table)

__weakref__

list of weak references to the object (if defined)

descend(properties)

Descend into a subschema

Parameters:properties (dict) – the properties in the subschema

parse(schema=None)

Parse a schema

Parameters:schema (dict) – the schema to parse

printlines()

Given a parsed schema (parsing happens when the object is constructed), print all the lines

Returns:the printed table Return type:str

sysinfo Module

This module contains function to help gather system information use for the provenance of the Job execution.

fastr.utils.sysinfo.get_cpu_usage()

Get the current CPU usage

Returns:CPU usage info Return type:dict

fastr.utils.sysinfo.get_drmaa_info()

Get information about the SGE cluster (if applicable)

Returns:cluster info Return type:dict

fastr.utils.sysinfo.get_hostinfo()

Get all information about the current host machine

Returns:host info Return type:dict

fastr.utils.sysinfo.get_memory_usage()

Get the current memory usage

Returns:memory usage info Return type:dict

fastr.utils.sysinfo.get_mounts()

Get the current mounts known on the system

Returns:mount info Return type:dict

fastr.utils.sysinfo.get_os()

Get information about the OS

Returns:OS information Return type:dict

fastr.utils.sysinfo.get_processes()

Get a list of all currently running processes

Returns:process information Return type:list

fastr.utils.sysinfo.get_python()

Get information about the currently used Python implementation

Returns:python info Return type:dict

fastr.utils.sysinfo.get_sysinfo()

Get system information (cpu, memory, mounts and users)

Returns:system information Return type:dict

fastr.utils.sysinfo.get_users()

Get current users on the system

Returns:user info Return type:dict

fastr.utils.sysinfo.namedtuple_to_dict(ntuple)

Helper function to convert a named tuple into a dict

Parameters:ntuple (namedtuple) – the namedtuple to convert Returns:named tuple as a dict Return type:dict

verify Module

fastr.utils.verify.verify_resource_loading(filename)

fastr.utils.verify.verify_tool(filename)

Verify that a file

xmltodict Module

This module contains tool for converting python dictionaries into XML object and vice-versa.

fastr.utils.xmltodict.dump(data, filehandle)

Write a dict to an XML file

Parameters:

• data – data to write
• filehandle – file handle to write to

fastr.utils.xmltodict.dumps(data)

Write a dict to an XML string

Parameters:data – data to write Returns:the XML data Return type:str

fastr.utils.xmltodict.load(filehandle)

Load an xml file and parse it to a dict

Parameters:filehandle – file handle to load Returns:the parsed data

fastr.utils.xmltodict.loads(data)

Load an xml string and parse it to a dict

Parameters:data (str) – the xml data to load Returns:the parsed data

Subpackages

• cmd Package
  • cmd Package
  • cat Module
  • execute Module
  • extract_argparse Module
  • prov Module
  • run Module
  • testtool Module
  • trace Module
  • verify Module
  • webapp Module