Table Of Contents

This Page

nepi.util package

Submodules

nepi.util.environ module

nepi.util.environ.execute(cmd)[source]
nepi.util.environ.backticks(cmd)[source]

nepi.util.execfuncs module

nepi.util.execfuncs.lcopy(source, dest, recursive=False)[source]

Copies from/to localy.

nepi.util.execfuncs.lexec(command, user=None, sudo=False, env=None)[source]

Executes a local command, returns ((stdout,stderr),process)

nepi.util.execfuncs.lgetpid(pidfile)[source]

Check the pidfile of a process spawned with remote_spawn.

Parameters:
pidfile: the pidfile passed to remote_span

Returns:

A (pid, ppid) tuple useful for calling remote_status and remote_kill, or None if the pidfile isn’t valid yet (maybe the process is still starting).
nepi.util.execfuncs.lkill(pid, ppid, sudo=False)[source]

Kill a process spawned with lspawn.

First tries a SIGTERM, and if the process does not end in 10 seconds, it sends a SIGKILL.

Parameters:

pid/ppid: pid and parent-pid of the spawned process. See remote_check_pid

sudo: whether the command was run with sudo - careful killing like this.

Returns:

Nothing, should have killed the process
nepi.util.execfuncs.lspawn(command, pidfile, stdout='/dev/null', stderr=<class nepi.util.sshfuncs.STDOUT at 0x7f3058d754c8>, stdin='/dev/null', home=None, create_home=False, sudo=False, user=None)[source]

Spawn a local command such that it will continue working asynchronously.

Parameters:

command: the command to run - it should be a single line.

pidfile: path of a (ideally unique to this task) pidfile for tracking the process.

stdout: path of a file to redirect standard output to - must be a string.
Defaults to /dev/null
stderr: path of a file to redirect standard error to - string or the special STDOUT value
to redirect to the same file stdout was redirected to. Defaults to STDOUT.

stdin: path of a file with input to be piped into the command’s standard input

home: path of a folder to use as working directory - should exist, unless you specify create_home

create_home: if True, the home folder will be created first with mkdir -p

sudo: whether the command needs to be executed as root

Returns:

(stdout, stderr), process

Of the spawning process, which only captures errors at spawning time. Usually only useful for diagnostics.

nepi.util.execfuncs.lstatus(pid, ppid)[source]

Check the status of a process spawned with remote_spawn.

Parameters:
pid/ppid: pid and parent-pid of the spawned process. See remote_check_pid

Returns:

One of NOT_STARTED, RUNNING, FINISHED

nepi.util.guid module

class nepi.util.guid.GuidGenerator[source]

Bases: object

next(guid=None)[source]

nepi.util.logger module

class nepi.util.logger.Logger(logger_component)[source]

Bases: object

debug(msg, out=None, err=None)[source]
error(msg, out=None, err=None)[source]
info(msg, out=None, err=None)[source]
log(msg, level, out=None, err=None)[source]
log_message(msg)[source]
logger[source]
warning(msg, out=None, err=None)[source]

nepi.util.manifoldapi module

class nepi.util.manifoldapi.MANIFOLDAPI(username, password, hostname, urlpattern)[source]

Bases: object

API to query different data platforms as SFA, TopHat, OML Central Server, using Manifold Framework, the backend of MySlice.

add_resource_to_slice(slicename, resource_urn)[source]

Add resource to user’s slice. The query needs to specify the new resource plus the previous resources already in the slice.

api[source]
get_resource_info(filters=None, fields=None)[source]

Create and execute the Manifold API Query to get the resources according fields and filters. :param filters: resource’s constraints for the experiment :type filters: dict :param fields: desire fields in the result of the query :type fields: list

get_resource_urn(filters=None)[source]

Retrieves the resources urn of the resources matching filters.

get_session_key()[source]

Retrieves the session key, in order to use the same session for queries.

get_slice_resources(slicename)[source]

Retrieves resources attached to user’s slice. return value: list of resources’ urn

remove_resource_from_slice(slicename, resource_urn)[source]

Remove resource from user’s slice. The query needs to specify the list of previous resources in the slice without the one to be remove.

class nepi.util.manifoldapi.MANIFOLDAPIFactory[source]

Bases: object

API Factory to manage a map of MANIFOLDAPI instances as key-value pairs, it instanciate a single instance per key. The key represents the same SFA, MF (ManiFold) credentials.

classmethod get_api(username, password, hostname='test.myslice.info', urlpattern='http://%(hostname)s:7080')[source]
Parameters:
  • username (str) – Manifold user (also used for MySlice web login)
  • password (str) – Manifold password (also used for MySlice web login)
  • hostname (str) – Hostname of the Manifold API to query SFA, TopHat, etc
  • urlpattern (str) – Url of the Manifold API to query SFA, TopHat, etc
classmethod make_key(*args)[source]

nepi.util.netgraph module

class nepi.util.netgraph.NetGraph(**kwargs)[source]

Bases: object

NetGraph represents a network topology. Network graphs are internally using the networkx library.

add_edge(nid1, nid2)[source]
add_node(nid)[source]
annotate_edge(nid1, nid2, name, value)[source]
annotate_edge_net(nid1, nid2, ip1, ip2, mask, network, prefixlen)[source]
annotate_node(nid, name, value)[source]
annotate_node_ip(nid, ip)[source]
assign_p2p_ips(network='10.0.0.0', prefix=8, version=4)[source]

Assign IP addresses to each end of each edge of the network graph, computing all the point to point subnets and addresses in the network representation.

param network:Base network address used for subnetting.
type network:str
param prefix:Prefix for the base network address used for subnetting.
type prefixt:int
param version:IP version (either 4 or 6).
type version:int
del_edge_annotation(nid1, nid2, name)[source]
del_node_annotation(nid, name)[source]
edge_annotation(nid1, nid2, name)[source]
edge_annotations(nid1, nid2)[source]
edge_net_annotation(nid1, nid2)[source]
edges()[source]
generate_topology(topo_type, node_count, branches=None)[source]
get_p2p_info(nid1, nid2)[source]
is_source(nid)[source]
is_target(nid)[source]
node_annotation(nid, name)[source]
node_annotations(nid)[source]
node_ip_annotations(nid)[source]
nodes()[source]
order[source]
select_random_source(**kwargs)[source]

Mark a random node as source.

select_target_zero()[source]

Mark the node 0 as target

set_source(nid)[source]
set_target(nid)[source]
sources()[source]

Returns the nodes that are sources

targets()[source]

Returns the nodes that are targets

topo_type[source]
topology[source]
class nepi.util.netgraph.TopologyType[source]
ADHOC = 'adhoc'
LADDER = 'ladder'
LINEAR = 'linear'
MESH = 'mesh'
STAR = 'star'
TREE = 'tree'

nepi.util.parallel module

class nepi.util.parallel.ParallelRun(maxthreads=None, maxqueue=None, results=True)[source]

Bases: object

destroy()[source]
empty()[source]
initialize_workers()[source]
join()[source]
put(callable, *args, **kwargs)[source]
put_nowait(callable, *args, **kwargs)[source]
start()[source]
sync()[source]
class nepi.util.parallel.WorkerThread(group=None, target=None, name=None, args=(), kwargs=None, verbose=None)[source]

Bases: threading.Thread

class QUIT[source]
WorkerThread.attach(queue, rvqueue, delayed_exceptions)[source]
WorkerThread.quit()[source]
WorkerThread.run()[source]

nepi.util.plotter module

class nepi.util.plotter.ECPlotter[source]

Bases: object

plot(ec, dirpath=None, format='figure', show=False)[source]
class nepi.util.plotter.PFormats[source]
DOT = 'dot'
FIGURE = 'figure'

nepi.util.rmatcher module

nepi.util.rmatcher.find_boxes(box, all_tags=None, exact_tags=None, max_depth=1)[source]

Look for the connected boxes with the required tags, doing breath-first search, until max_depth ( max_depth = None will traverse the entire graph ).

nepi.util.rmatcher.match_tags(box, all_tags, exact_tags)[source]

returns True if box has required tags

nepi.util.serializer module

class nepi.util.serializer.ECSerializer[source]

Bases: object

load(filepath, format='xml')[source]
save(ec, dirpath, format='xml')[source]
serialize(ec, format='xml')[source]
class nepi.util.serializer.SFormats[source]
XML = 'xml'

nepi.util.sfaapi module

class nepi.util.sfaapi.SFAAPI(sfi_user, sfi_auth, sfi_registry, sfi_sm, private_key, ec, batch, rtype, timeout)[source]

Bases: object

API for quering the SFA service. It uses Sfi class from the tool sfi client.

add_resource_to_slice(slicename, resource_hrn, leases=None)[source]

Get the list of resources’ urn, build the rspec string and call the allocate and provision method.

add_resource_to_slice_batch(slicename, resource_hrn, properties=None, leases=None)[source]

Method to add all resources together to the slice. Previous deletion of slivers. Specially used for wilabt that doesn’t allow to add more resources to the slice after some resources are added. Every sliver have to be deleted and the batch has to be added at once.

blacklist_resource(resource_hrn)[source]

Adding resource_hrn to blacklist, and taking the resource from the reserved list.

blacklisted(resource_hrn)[source]

Check if the resource is in the blacklist.

get_resources_hrn(resources=None)[source]

Get list of resources hrn, without the resource info.

get_resources_info()[source]

Get all resources and its attributes from aggregate.

get_slice_resources(slicename)[source]

Get resources and info from slice.

release()[source]

Remove hosts from the reserved and blacklist lists, and in case the persist attribute is set, it saves the blacklisted hosts in the blacklist file.

remove_all_from_slice(slicename)[source]

De-allocate and de-provision all slivers of the named slice. Currently sfi doesn’t support removing particular slivers, so this method works only for removing every sliver. Setting the resource_hrn parameter is not necessary.

remove_resource_from_slice(slicename, resource_hrn, leases=None)[source]

Remove slivers from slice. Currently sfi doesn’t support removing particular slivers.

reserve_resource(resource_hrn)[source]

Add resource to the reserved list.

reserved(resource_hrn)[source]

Check that the resource in not reserved.

class nepi.util.sfaapi.SFAAPIFactory[source]

Bases: object

API Factory to manage a map of SFAAPI instances as key-value pairs, it instanciate a single instance per key. The key represents the same SFA, credentials.

classmethod get_api(sfi_user, sfi_auth, sfi_registry, sfi_sm, private_key, ec, batch=False, rtype=None, timeout=None)[source]
classmethod make_key(*args)[source]

nepi.util.sfarspec_proc module

class nepi.util.sfarspec_proc.SfaRSpecProcessing(config=None)[source]

Bases: object

Class to process SFA RSpecs, parse the RSpec replies such as Advertisement RSpecs, and build in the case of Request RSpecs.

build_sfa_rspec(slice_id, resources, properties, leases)[source]

Build the XML RSpec from list of resources’ urns. eg. resources = [“urn:publicid:IDN+ple:modenaple+node+planetlab-1.ing.unimo.it“]

make_dict_rec(obj)[source]
parse_sfa_rspec(rspec_string)[source]

Parse the RSpec XML as a string.

nepi.util.sshfuncs module

class nepi.util.sshfuncs.ProcStatus[source]

Codes for status of remote spawned process

FINISHED = 2
NOT_STARTED = 3
RUNNING = 1
class nepi.util.sshfuncs.STDOUT[source]

Special value that when given to rspawn in stderr causes stderr to redirect to whatever stdout was redirected to.

nepi.util.sshfuncs.eintr_retry(func)[source]

Retries a function invocation when a EINTR occurs

nepi.util.sshfuncs.gethostbyname(host)[source]
nepi.util.sshfuncs.log(msg, level, out=None, err=None)[source]
nepi.util.sshfuncs.make_control_path(agent, forward_x11)[source]
nepi.util.sshfuncs.make_server_key_args(server_key, host, port)[source]

Returns a reference to a temporary known_hosts file, to which the server key has been added.

Make sure to hold onto the temp file reference until the process is done with it

Parameters:
  • server_key (str) – the server public key
  • host (str) – the hostname
  • port (str) – the ssh port
nepi.util.sshfuncs.openssh_has_persist()[source]

The ssh_config options ControlMaster and ControlPersist allow to reuse a same network connection for multiple ssh sessions. In this way limitations on number of open ssh connections can be bypassed. However, older versions of openSSH do not support this feature. This function is used to determine if ssh connection persist features can be used.

nepi.util.sshfuncs.rcopy(source, dest, port=None, gwuser=None, gw=None, recursive=False, identity=None, server_key=None, retry=3, strict_host_checking=True)[source]

Copies from/to remote sites.

Source and destination should have the user and host encoded as per scp specs.

Source can be a list of files to copy to a single destination, (in which case it is advised that the destination be a folder), or a single file in a string.

nepi.util.sshfuncs.resolve_hostname(host)[source]
nepi.util.sshfuncs.rexec(command, host, user, port=None, gwuser=None, gw=None, agent=True, sudo=False, identity=None, server_key=None, env=None, tty=False, connect_timeout=30, retry=3, persistent=True, forward_x11=False, blocking=True, strict_host_checking=True)[source]

Executes a remote command, returns ((stdout,stderr),process)

nepi.util.sshfuncs.rgetpid(*p, **kw)[source]

Returns the pid and ppid of a process from a remote file where the information was stored.

param home:Path to directory where the pidfile is located
type home:str
param pidfile:Name of file containing the pid information
type pidfile:str
rtype:int

A (pid, ppid) tuple useful for calling rstatus and rkill, or None if the pidfile isn’t valid yet (can happen when process is staring up)

nepi.util.sshfuncs.rkill(*p, **kw)[source]

Sends a kill signal to a remote process.

First tries a SIGTERM, and if the process does not end in 10 seconds, it sends a SIGKILL.

param pid:Process id of process to be killed
type pid:int
param ppid:Parent process id of process to be killed
type ppid:int
param sudo:Flag indicating if sudo should be used to kill the process
type sudo:bool
nepi.util.sshfuncs.rspawn(command, pidfile, stdout='/dev/null', stderr=<class nepi.util.sshfuncs.STDOUT at 0x7f3058d754c8>, stdin='/dev/null', home=None, create_home=False, sudo=False, host=None, port=None, user=None, gwuser=None, gw=None, agent=None, identity=None, server_key=None, tty=False, strict_host_checking=True)[source]

Spawn a remote command such that it will continue working asynchronously in background.

param command:The command to run, it should be a single line.
type command:str
param pidfile:Path to a file where to store the pid and ppid of the spawned process
type pidfile:str
param stdout:Path to file to redirect standard output. The default value is /dev/null
type stdout:str
param stderr:Path to file to redirect standard error. If the special STDOUT value is used, stderr will be redirected to the same file as stdout
type stderr:str
param stdin:Path to a file with input to be piped into the command’s standard input
type stdin:str
param home:Path to working directory folder. It is assumed to exist unless the create_home flag is set.
type home:str
param create_home:
 Flag to force creation of the home folder before running the command
type create_home:
 bool
param sudo:Flag forcing execution with sudo user
type sudo:bool
rtype:tuple

(stdout, stderr), process

Of the spawning process, which only captures errors at spawning time. Usually only useful for diagnostics.

nepi.util.sshfuncs.rstatus(*p, **kw)[source]

Returns a code representing the the status of a remote process

param pid:Process id of the process
type pid:int
param ppid:Parent process id of process
type ppid:int
rtype:int (One of NOT_STARTED, RUNNING, FINISHED)
nepi.util.sshfuncs.shell_escape(s)[source]

Escapes strings so that they are safe to use as command-line arguments

nepi.util.statfuncs module

nepi.util.statfuncs.compute_mean(sample)[source]

nepi.util.timefuncs module

nepi.util.timefuncs.compute_delay_ms(timestamp2, timestamp1)[source]
nepi.util.timefuncs.stabsformat(sdate, dbase=None)[source]

Constructs a datetime object from a string date. The string date can be expressed as an absolute date ( i.e. format YYYYMMddHHMMSSffff ) or as a relative time ( e.g. format ‘5m’ or ‘10s’). If the date is a relative time and the dbase parameter is given (dbase must be datetime object), the returned date will be dbase + sdate. If dbase is None, current time will be used instead as base time.

:param date : string date :type date : date

nepi.util.timefuncs.stformat(sdate)[source]

Constructs a datetime object from a string date with format YYYYMMddHHMMSSffff

nepi.util.timefuncs.tdiff(date1, date2)[source]

Returns difference ( date1 - date2 ) as a datetime object, where date1 and date 2 are datetime objects

nepi.util.timefuncs.tdiffsec(date1, date2)[source]

Returns the date difference ( date1 - date2 ) in seconds, where date1 and date 2 are datetime objects

nepi.util.timefuncs.tnow()[source]

Returns datetime object with the current time

nepi.util.timefuncs.tsformat(date=None)[source]

Formats a datetime object to a string with format YYYYMMddHHMMSSffff. If no date is given, the current date is used.

Module contents