%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\begin{itemize}
- \item EC API
- \begin{itemize}
- \item register resource
- \item RM States (State transition DIAGRAM)
- \item RM Actions
- \item Attributes
- \item The critical attribute
- \item Traces (how to collect traces to the local repo, talk about the Collector RM)
- \item deploy (interactive deployment)
- \item workflow - register condition
- \end{itemize}
- \item Resource Factory
- \begin{itemize}
- \item Populate factory (happens automatically)
- \item How to discover available rm types and their resources
- \end{itemize}
-\end{itemize}
+The ExperimentController (EC) is the entity in charge of turning the
+experiment description into a running experiment.
+In order to do this the EC needs to know which resources are to be
+used, how they should be configured and how resources relate to one another.
+To this purpose the EC exposes methods to register resources, specify their
+configuration, and register dependencies between. These methods are part of
+the EC design API.
+Likewise, in order to deploy and control resources, and collect data,
+the EC exposes another set of methods, which form the execution API.
+These two APIs are described in detail in the rest of this chapter.
+
+
+\section{The experiment script}
+
+NEPI is a Python-based language and all classes and functions can
+be used by importing the \emph{nepi} module from a Python script.
+
+In particular, the ExperimentController class can be imported as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.ec import ExperimentController
+
+\end{lstlisting}
+
+Once this is done, an ExperimentController must be instantiated for
+the experiment. The ExperimentController constructor receives
+the optional argument \emph{exp\_id}. This argument is important because
+it defines the experiment identity and allows to distinguish among different
+experiments. If an experiment id is not explicitly given, NEPI will automatically
+generate a unique id for the experiment.
+
+\begin{lstlisting}[language=Python]
+
+ec = ExperimentController(exp_id = "my-exp-id")
+
+\end{lstlisting}
+
+The experiment id can always be retrieved as follows
+
+\begin{lstlisting}[language=Python]
+
+exp_id = ec.exp_id
+
+\end{lstlisting}
+
+Since a same experiment can be ran more than one time, and this is
+often desirable to obtain statistical data, the EC identifies
+different runs of an experiment with a same \emph{exp\_id} with
+another attribute, the \emph{run\_id}. The \emph{run\_id} is
+a timestamp string value, and in combination with the \emph{exp\_id},
+it allows to uniquely identify an experiment instance.
+
+\begin{lstlisting}[language=Python]
+
+run_id = ec.run_id
+
+\end{lstlisting}
+
+
+
+\section{The design API}
+
+Once an ExperimentController has been instantiated, it is possible to start
+describing the experiment. The design API is the set of methods which
+allow to do so.
+
+
+\subsection{Registering resources}
+
+Every resource supported by NEPI is controlled by a specific ResourceManager
+(RM). The RM instances are automatically created by the EC, and the user does
+not need to interact with them directly.
+
+Each type of RM is associated with a \emph{type\_id} which uniquely identifies
+a concrete kind of resource (e.g PlanetLab node, application that runs in
+a Linux machine, etc).
+The \emph{type\_ids} are string identifiers, and they are required
+to register a resource with the EC.
+
+To discover all the available RMs and their \emph{type\_ids} we
+can make use of the ResourceFactory class.
+This class is a \emph{Singleton} that holds the templates and information
+of all the RMs supported by NEPI. We can retrieve this information as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceFactory
+
+for type_id in ResourceFactory.resource_types():
+ rm_type = ResourceFactory.get_resource_type(type_id)
+ print type_id, ":", rm_type.get_help()
+
+\end{lstlisting}
+
+Once the \emph{type\_id} of the resource is known, the registration of a
+new resource with the EC is simple:
+
+\begin{lstlisting}[language=Python]
+
+type_id = "SomeRMType"
+guid = ec.register_resources(type_id)
+
+\end{lstlisting}
+
+When a resource is registered, the EC instantiates a RM of the
+requested \emph{type\_id} and assigns a global unique identifier
+(guid) to it. The guid is an incremental integer number and it
+is the value returned by the \emph{register\_resource} method.
+The EC keeps internal references to all RMs, which the user can
+reference using the corresponding guid value.
+
+
+\subsection{Attributes}
+
+ResourceManagers expose the configurable parameters of resources
+through a list of attributes. An attribute can be seen as a
+\emph{{name:value}} pair, that represents a certain aspect of
+the resource (whether information or configuration information).
+
+It is possible to discover the list of attributes exposed by an
+RM type as follows:
+
+\begin{lstlisting}[language=Python]
+from nepi.execution.resource import ResourceFactory
+
+type_id = "SomeRMType"
+rm_type = ResourceFactory.get_resource_type(type_id)
+
+for attr in rm_type.get_attributes():
+ print " ", attr.name, ":", attr.help
+
+\end{lstlisting}
+
+To configure or retrieve the value of a certain attribute of
+an registered resource we can use the \emph{get} and \emph{set}
+methods of the EC.
+
+\begin{lstlisting}[language=Python]
+
+old_value = ec.get(guid, "attr_name")
+ec.set(guid, "attr_name", new_value)
+new_value = ec.get(guid, "attr_name")
+
+\end{lstlisting}
+
+Since each RM type exposes the characteristics of a particular type
+of resource, it is to be expected that different RMs will have different
+attributes. However, there a particular attribute that is common to all RMs.
+This is the \emph{critical} attribute, and it is meant to indicate to the EC
+how it should behave when a failure occurs during the experiment.
+The \emph{critical} attribute has a default value of \emph{True}, since
+all resources are considered critical by default.
+When this attribute is set to \emph{False} the EC will ignore failures on that
+resource and carry on with the experiment. Otherwise, the EC will immediately
+interrupt the experiment.
+
+
+\subsection{Traces}
+
+A Trace represent a stream of data collected during the experiment and associated
+to a single resource. ResourceManagers expose a list of traces, which are identified
+by a name. Particular traces might or might not need activation, since some traces
+are enabled by default.
+
+It is possible to discover the list of traces exposed by an
+RM type as follows:
+
+\begin{lstlisting}[language=Python]
+from nepi.execution.resource import ResourceFactory
+
+type_id = "SomeRMType"
+rm_type = ResourceFactory.get_resource_type(type_id)
+
+for trace in rm_type.get_traces():
+ print " ", trace.name, ":", trace.enabled
+
+\end{lstlisting}
+
+The \emph{enable\_trace} method allows to enable a specific trace for a
+RM instance
+
+\begin{lstlisting}[language=Python]
+
+ec.enable_trace(guid, "trace-name")
+
+print ec.trace_enabled(guid, "trace-name")
+
+\end{lstlisting}
+
+
+\subsection{Registering connections}
+
+In order to describe the experiment set-up, a resources need to be
+associated at least to one another. Through the process of connecting resources
+the \emph{topology graph} is constructed. A certain application might
+need to be configured and executed on a certain node, and this
+must be indicated to the EC by connecting the application RM to the node
+RM.
+
+Connections are registered using the \emph{register\_connection} method,
+which receives the guids of the two RM.
+
+\begin{lstlisting}[language=Python]
+
+ec.register_connection(node_guid, app_guid)
+
+\end{lstlisting}
+
+The order in which the guids are given is not important, since the
+\emph{topology\_graph} is not directed, and the corresponding
+RMs \emph{`know'} internally how to interpret the connection
+relationship.
+
+
+\subsection{Registering conditions}
+
+All ResourceMangers must go through the same sequence of state transitions.
+Associated to those states are the actions that trigger the transitions.
+As an example, a RM will initially be in the state NEW. When the DEPLOY action
+is invoked, it will transition to the DISCOVERED, then PROVISIONED, then READY
+states. Likewise, the action START will make a RM pass from state READY to
+STARTED, and the action STOP will change a RM from state STARTED to STOPPED.
+
+Using these states and actions, it is possible to specify workflow dependencies
+between resources. For instance, it would be possible to indicate that
+one application should start after another application by registering a
+condition with the EC.
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceActions
+
+ec.register_condition(app1_guid, ResourceAction.START, app2_guid, ResourceState.STARTED)
+
+\end{lstlisting}
+
+The above invocation should be read "Application 1 should START after application 2
+has STARTED". It is also possible to indicate a relative time from the moment a state
+change occurs to the moment the action should be taken as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceActions
+
+ec.register_condition(app1_guid, ResourceAction.START, app2_guid, ResourceState.STARTED, time = "5s")
+
+\end{lstlisting}
+
+This line should be read "Application 1 should START at least 5 seconds after
+application 2 has STARTED". \\
+
+Allowed actions are: DEPLOY, START and STOP. \\
+
+Existing states are: NEW, DISCOVERED, PROVISIONED, READY, STARTED, STOPPED,
+FAILED and RELEASED. \\
+
+
+
+\section{The execution API}
+
+After registering all the resources and connections and setting attributes and
+traces, once the experiment we want to conduct has been described, we can
+proceed to run it. To this purpose we make use of the \emph{execution} methods
+exposed by the EC.
+
+
+\subsection{Deploying an experiment}
+
+Deploying an experiment is very easy, it only requires to invoke the
+\emph{deploy} method of the EC.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy()
+
+\end{lstlisting}
+
+Given the experiment description provided earlier, the EC will take care
+of automatically performing all necessary actions to discover, provision,
+configure and start all resources registered in the experiment.
+
+Furthermore, NEPI does not restrict deployment to only one time, it allows
+to continue to register, connect and configure resources and deploy them
+at any moment. We call this feature \emph{interactive} or \emph{dynamic}
+deployment.
+
+The \emph{deploy} method can receive other optional arguments to customize
+deployment. By default, the EC will deploy all registered RMs that are in
+state NEW. However, it is possible to specify a subset of resources to be
+deployed using the \emph{guids} argument.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(guids=[guid1, guid2, guid3])
+
+\end{lstlisting}
+
+Another useful argument of the \emph{deploy} method is \emph{wait\_all\_ready}.
+This argument has a default value of \emph{True}, and it is used as a barrier
+to force the START action to be invoked on all RMs being deploy only after
+they have all reached the state READY.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(wait_all_ready=False)
+
+\end{lstlisting}
+
+
+\subsection{Getting attributes}
+
+Attribute values can be retrieved at any moment during the experiment run,
+using the \emph{get} method.
+However, not all attributes can be modified after a resource has
+been deployed. The possibility of changing the value of a certain attribute
+depends strongly on the RM and on the attribute itself.
+As an example, once a \emph{hostname} has been specified for a certain Node
+RM, it might not be possible to change it after deployment.
+
+\begin{lstlisting}[language=Python]
+
+attr_value = ec.get(guid, "attr-name")
+
+\end{lstlisting}
+
+Attributes have flags that indicate whether their values can be changed
+and when it is possible to change them (e.g. before or after deployment,
+or both). These flags are \emph{NoFlags} (the attribute value can be
+modified always), \emph{ReadOnly} (the attribute value can never be
+modified), \emph{ExecReadOnly} (the attribute value can only be modified
+before deployment). The flags of a certain attribute can be validated
+as shown in the example below, and the value of the attribute can be
+changed using the \emph{set} method.
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.attribute import Flags
+
+attr = ec.get_attribute(guid, "attr-name")
+
+if not attr.has_flag(Flags.ReadOnly):
+ ec.set(guid, "attr-name", attr_value)
+
+\end{lstlisting}
+
+\subsection{Quering the state}
+
+It is possible to query the state of any resource at any moment.
+The state of a resource is requested using the \emph{state} method.
+This method receives the optional parameter \emph{hr} to output the
+state in a \emph{human readable} string format instead of an integer
+state code.
+
+\begin{lstlisting}[language=Python]
+
+state_id = ec.state(guid)
+
+# Human readable state
+state = ec.state(guid, hr = True)
+
+\end{lstlisting}
+
+\subsection{Getting traces}
+
+After a ResourceManager has been deployed it is possible to get information
+about the active traces and the trace streams of the generated data using
+the \emph{trace} method.
+
+Most traces are collected to a file in the host where they are generated,
+the total trace size and the file path in the (remote) host can be
+retrieved as follows.
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.trace import TraceAttr
+
+path = ec.trace(guid, "trace-name", TraceAttr.PATH)
+size = ec.trace(guid, "trace-name", TraceAttr.SIZE)
+
+\end{lstlisting}
+
+The trace content can be retrieved in a stream, block by block.
+
+\begin{lstlisting}[language=Python]
+
+trace_block = ec.trace(guid, "trace-name", TraceAttr.STREAM, block=1, offset=0)
+
+\end{lstlisting}
+
+It is also possible to directly retrieve the complete trace content.
+
+\begin{lstlisting}[language=Python]
+
+trace_stream = ec.trace(guid, "trace-name")
+
+\end{lstlisting}
+
+Using the \emph{trace} method it is easy to collect all traces
+to the local user machine.
+
+\begin{lstlisting}[language=Python]
+
+for trace in ec.get_traces(guid):
+ trace_stream = ec.trace(guid, "trace-name")
+ f = open("trace-name", "w")
+ f.write(trace_stream)
+ f.close()
+
+\end{lstlisting}
+
+
+% TODO: how to retrieve an application trace when the Node failed? (critical attribute)
+
+
+% \subsection{The collector RM}
+
+%%%%%%%%%%
+%% TODO
+%%%%%%%%%%%
+
+\subsection{API reference}
+
+Further information about classes and method signatures
+can be found using the Python \emph{help} method.
+For this inspection work, we recommend to instantiate an
+ExperimentController from an IPython console. This is an
+interactive console that allows to dynamically send input
+to the python interpreter.
+
+If NEPI is not installed in the system, you will need to add the
+NEPI sources path to the PYTHONPATH environmental variable
+before invoking \emph{ipython}.
+
+\begin{lstlisting}[language=Python]
+
+$ PYTHONPATH=$PYTHONPATH:src ipython
+Python 2.7.3 (default, Jan 2 2013, 13:56:14)
+Type "copyright", "credits" or "license" for more information.
+
+
+IPython 0.13.1 -- An enhanced Interactive Python.
+? -> Introduction and overview of IPython's features.
+%quickref -> Quick reference.
+help -> Python's own help system.
+object? -> Details about 'object', use 'object??' for extra details.
+
+In [1]: from nepi.execution.ec import ExperimentController
+
+In [2]: ec = ExperimentController(exp_id = "test-tap")
+
+In [3]: help(ec.set)
+
+\end{lstlisting}
+
+The example above will show the following information related to the
+\emph{set} method of the EC API.
+
+\begin{lstlisting}[language=Python]
+
+Help on method set in module nepi.execution.ec:
+
+set(self, guid, name, value) method of nepi.execution.ec.ExperimentController instance
+ Modifies the value of the attribute with name 'name' on the RM with guid 'guid'.
+
+ :param guid: Guid of the RM
+ :type guid: int
+
+ :param name: Name of the attribute
+ :type name: str
+
+ :param value: Value of the attribute
+
+\end{lstlisting}
+
\begin{itemize}
\item the RMs dictionary
+ \item The scheduling API
\item The scheduler queue, the tasks dictionary, the schedule method
\item the processing thread and the \_process method, the inmediate execution queueu and the ParallelRunner
\item the \_execute method
- \item The deploy method, deployment groups
+ \item The deploy method (implementation), deployment groups
\item The FailManager and what happens upon release (critical attribute)
\end{itemize}
So, 'One ring to rule them all', sorry I meant, 'One tool to
control them all'... or something like that.
-We though it was a good ide to abstract platform details
+We though it was a good idea to abstract platform details
behind a common resource management interface, and let
NEPI deal with the details and give you back the results.
ssh_key = # Path to SSH public key file to access host
node = ec.register_resource("LinuxNode")
-ec.set(node, "hostname", host)
-ec.set(node, "username", user)
+ec.set(node, "hostname", hostname)
+ec.set(node, "username", username)
ec.set(node, "identity", ssh_key)
app = ec.register_resource("LinuxApplication")
A first prototype was implemented in 2010.
Versions 1.0 and 2.0 were released in 2011 and 2012, respectively.
The current version is 3.0, and it was completely redesigned and
-rewritten to broaden the scope, and to include several cool
+rewritten to broaden the scope, and to include several
new features, which will be described in detail in this document.
The following people has contributed to the project:
\section{Is it free?}
-Yes, NEPI is free software. It is free to use, free to to modify, free to share.
+Yes, NEPI is free software. It is free to use, free to modify, free to share.
NEPI v3.0 is licensed under GPL v3, so you can do whatever you want with it,
as long as you keep the same license.
There are many ways you can contribute to the project.
The first one is using it and reporting bugs.
-You can report bugs on the NEPI bugzilla page
-(\textbf{not yet set-up, so for now send the bugs by email to the users list}): \\
+You can report bugs on the NEPI bugzilla page at:
-\url{https://nepi.inria.fr/bugzilla/} \\
+\url{http://nepi.inria.fr/bugzilla} \\
You can also become a part of the NEPI community and join our mailing lists:
\section{How can I report a bug ?}
-To report a bug take a look at the NEPI bugzilla page at:
+To report a bug take a look at the NEPI bugzilla page at :
-\url{https://nepi.inria.fr/bugzilla/} \\
+\url{http://nepi.inria.fr/bugzilla} \\
\section{Where can I get more information ?}
-For more information visit NEPI web site at
+For more information visit NEPI web site at :
-\url{https://nepi.inria.fr} \\
+\url{http://nepi.inria.fr} \\
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-NEPI is written in Python, so you will need to install Python before
-before being able to run experiments with NEPI.
+NEPI is written in Python, so you will need to install Python before
+being able to run experiments with NEPI.
NEPI is known to work on Linux (Fedore, Debian, Ubuntu) and Mac (OS X).
\section{Dependencies}
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
- $ PYTHONATH=$PYTHONPATH:<path-to-nepi>/src python experiment.py
+ $ PYTHON
PATH=$PYTHONPATH:<path-to-nepi>/src python experiment.py
\end{verbatim}
\endgroup
\endgroup
-\subsection{Run interactively}
+\subsection{Run NEPI interactively}
-The Python interpreter can be used as an interactive console to execute
-Python instructions.
-We can take advantage of this feature, to interactively run experiments
-with NEPI. We use the \textit{ipython} console for the example below,
-you can install it with \textit{sudo yum/aptitude install ipython} on
-Fedora/Debian.
+The IPython console can be used as an interactive interpreter to
+execute Python instructions. We can take advantage of this feature,
+to interactively run NEPI experiments.
+We will use the IPython console for the example below.
+
+You can easily install IPython on Debian, Ubuntu, Fedora or Mac as follows:\\
+
+\textbf{Debian/Ubuntu}
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+
+$ sudo apt-get install ipython
+
+\end{verbatim}
+
+\endgroup
+
+\textbf{Fedora}\\
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+
+$ sudo yum install ipython
+
+\end{verbatim}
+
+\endgroup
+
+\textbf{Mac}\\
\begingroup
\fontsize{10pt}{12pt}\selectfont
+\begin{verbatim}
+
+$ pip install ipython
+
+\end{verbatim}
+
+\endgroup
+
+Before starting, make sure to add Python and IPython source directory
+path to the PYTHONPATH environment variable
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+
+$ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python:/usr/local/share/python/ipython
+
+\end{verbatim}
+
+\endgroup
+
+Then you can start IPython as follows:
+
\begin{verbatim}
$ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
$ ipython
\end{verbatim}
-\endgroup
-
-With ipython, if you want to paste many lines at once you will need to type
-\emph{\%cpaste} and finish the paste block with \emph{\-\-}.
+If you want to paste many lines at once in IPython, you will need
+to type \emph{\%cpaste} and finish the paste block with \emph{\-\-}.
-The first thing we have to do is to import the NEPI classes
-we will use.
-In particular we need to import the ExperimentController class.
+The first thing we need to do to describe an experiment with NEPI
+is to import the NEPI Python modules.
+In particular we need to import the ExperimentController class.
+To do this type the following in the Python console:
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Then we need to create an ExperimentController instance.
-The \textit{exp-id} argument serves as a human readable identifier
-for the experiment to be ran.
+After importing the ExperimentController class, it is possible to
+create a new instance of an the ExperimentController (EC) for
+your experiment.
+The <exp-id> argument is the name you want to give the experiment
+to identify it and distinguish it from other experiments.
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Next we will define two Python functions, one to register \emph{LinuxNode}
-resources and the other to register \emph{LinuxApplication} resources.
-A \emph{LinuxNode} resource (or ResourceManager) will serve as an abstraction
-to a Linux host resource, that can be accessed using SSH key authentication.
-A \emph{LinuxApplication} resource represents anything that can be executed
-on a Linux host as a BASH command.
+Next we will define two Python functions: \emph{add\_node} and \emph{add\_app}.
+The first one to register \textit{LinuxNodes} resources and the second one to
+register LinuxApplications resources.
\begin{lstlisting}[language=Python]
%cpaste
-def add_node(ec, host, user, ssh_key):
+def add_node(ec, hostname, username, ssh_key):
node = ec.register_resource("LinuxNode")
- ec.set(node, "hostname", host)
- ec.set(node, "username", user)
+ ec.set(node, "hostname", hostname)
+ ec.set(node, "username", username)
ec.set(node, "identity", ssh_key)
ec.set(node, "cleanHome", True)
ec.set(node, "cleanProcesses", True)
\end{lstlisting}
-The method \textit{register\_resource} declares a resource instance to the
-Experiment Controller. The method \textit{register\_connection} indicates
-that two resource will interact during the experiment.
+The method \textit{register\_resource} registers a resource instance with the
+ExperimentController. The method \textit{register\_connection} indicates
+that two resources will interact during the experiment.
Note that invoking \textit{add\_node} or \textit{add\_app} has no effect other
than informing the EC about the resources that will be used during the experiment.
The actual deployment of the experiment requires the method \textit{deploy} to
be invoked.
-The Resource Managers (RM) that abstract the concrete resources expose
-configuration attributes. In the LinuxNode RM we set the \emph{hostname}
-and \emph{username} as attributes, while in the LinuxApplication RM
-we set the \emph{command} attribute.
-
-Apart from teh \emph{command} attribute, the \emph{LinuxApplication}
-ResourceManager exposed several other attributes
-that permit to upload, compile and install arbitrary sources,
-to run any application might be needed to run an experiment.
-More details will be given in the following sections of this document.
-
-Lets now use these functions to describe the experiment we will run.
+The \textit{LinuxNode} resource exposes the hostname, username and identity
+attributes. This attributes provide information about the SSH credentials
+needed to log in to the Linux host.
+The \textit{hostname} is the one that identifies the physical host you want
+to access during the experiment. The \textit{username} must correspond to a
+valid account on that host, and the \textit{identity} attribute is the
+'absolute' path to the SSH private key in your local computer that allows you
+to log in to the host.
+
+The \textit{command} attribute of the \textit{LinuxApplication} resource
+expects a BASH command line string to be executed in the remote host.
+Apart from the \emph{command} attribute, the \emph{LinuxApplication}
+resource exposes several other attributes that allow to upload,
+compile and install arbitrary sources.
+The add\_app function registers a connection between a \textit{LinuxNode} and a
+\textit{LinuxApplication}.
+
+Lets now use these functions to describe a simple experiment.
Choose a host where you have an account, and can access using SSH
-key authentication. Define string variables with the right
-values for the \emph{hostname}, \emph{username} and path to the
-SSH public key in as \emph{ssh\_key}, and then type the following lines.
+key authentication.
\begin{lstlisting}[language=Python]
+hostname = "<the-hostname>"
+username = "<my-username>"
+identity = "</home/myuser/.ssh/id_rsa>"
+
node = add_node(ec, hostname, username, ssh_key)
app = add_app(ec, "ping -c3 nepi.inria.fr", node)
\end{lstlisting}
-Now that we have described our simple PING experiment, we can deploy
-it. Invoking the \emph{deploy} command will not only configure the
+The values returned by the functions add\_node and add\_app are global
+unique identifiers (guid) of the resources that were registered with the EC.
+The guid is used to reference the ResourceManager associated to a registered
+resource (for instance to retrieve results or change attribute values).
+
+Now that we have registered some resources, we can ask the ExperimentController
+(EC) to deploy them.
+Invoking the \emph{deploy} command will not only configure the
resource but also automatically launch the applications.
\begin{lstlisting}[language=Python]
\end{lstlisting}
-After some seconds, we should see some log messages informing about
-the progress of the deployment.
-If you now open another terminal and connect to the host through SSH,
-you should find some directories created by NEPI.
-You should see a directory named \emph{nepi-exp}, and under that directory
-you will find another with the identifier you gave when you created the
-experiment controller (the <exp-id>).
-Under the experiment directory, you will find directories for each of the
-resources deployed (e.g. \emph{node-1}, \emph{app-2}).
-The resource directories are named with a short string that identifies the
-type of resource (e.g. 'node', 'app', etc), followed by a unique number that
-uniquely identifies a given resource within the experiment,
-the global unique identifier (guid).
-
-From the ipython console, we can check the deployment status of each resource
-by querying the EC with the method \emph{state}.
-The argument \emph{hr} stand for `human readable`, and will return a string
-state instead of a state number.
+After some seconds, we should see some output messages informing us about the
+progress in the host deployment.
+If you now open another terminal and you connect to the host using
+SSH (as indicated below), you should see that a directory for your experiment
+has been created in the host. In the remote host you will see that two NEPI
+directories were created in the \$HOME directory: \emph{nepi-src} and \emph{nepi-exp}.
+The first one is where NEPI will store files that might be re used by many
+experiments (e.g. source code, input files) . The second directory \emph{nepi-exp},
+is where experiment specific files (e.g. results, deployment scripts) will be stored.
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+
+$ ssh -i identity username@hostname
+
+\end{verbatim}
+
+\endgroup
+
+Inside the \emph{nepi-exp} directory, you will find another directory with
+the <exp-id> assigned to your EC, and inside that directory you should find
+one directory named node-1 which will contain the files (e.g. result traces)
+associated to the LinuxNode reosurce you just deployed.
+In fact for every resource deployed associated to that host (e.g. each
+LinuxApplication), NEPI will create a directory to place files related to it.
+The name of the directory identifies the type of resources (e.g. 'node',
+'app', etc) and it is followed by the global unique identifier (guid).
+
+We can see if a resource finished deploying by querying its state through the EC
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Once the LinuxApplication is STARTED, we can retrieve the PING output using
-stored as a \emph{trace} file on the host. For this we use use \emph{trace}
-method, and specify the resource and the trace name (i.e. stdout).
+Once a \textit{LinuxApplication} has reached the state 'STARTED',
+we can retrieve the 'stdout' trace, which should contain the output
+of the PING command.
\begin{lstlisting}[language=Python]
\end{lstlisting}
+\subsection{Define a workflow}
+
+Now that we have introduced to the basics of NEPI, we will register
+two more applications and define a workflow where one application
+will start after the other one has finished executing.
+For this we will use the EC \textit{register\_condition} method described below:
+
+\begin{lstlisting}[language=Python]
+
+register_condition(self, guids1, action, guids2, state, time=None):
+ Registers an action START, STOP or DEPLOY for all RM on list
+ guids1 to occur at time 'time' after all elements in list guids2
+ have reached state 'state'.
+
+ :param guids1: List of guids of RMs subjected to action
+ :type guids1: list
+
+ :param action: Action to perform (either START, STOP or DEPLOY)
+ :type action: ResourceAction
+
+ :param guids2: List of guids of RMs to we waited for
+ :type guids2: list
+
+ :param state: State to wait for on RMs of list guids2 (STARTED,
+ STOPPED, etc)
+ :type state: ResourceState
+
+ :param time: Time to wait after guids2 has reached status
+ :type time: string
+
+\end{lstlisting}
+
+To use the \textit{register\_condition} method we will need to import the
+ResourceState and the ResourceAction classes
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceAction
+
+\end{lstlisting}
+
+Then, we register the two applications. The first application will
+wait for 5 seconds and the create a file in the host called "greetings"
+with the content "HELLO WORLD".
+The second application will read the content of the file and output it
+to standard output. If the file doesn't exist il will instead output the
+string "FAILED".
+
+\begin{lstlisting}[language=Python]
+
+app1 = add_app(ec, "sleep 5; echo 'HELLO WORLD!' > ~/greetings", node)
+app2 = add_app(ec, "cat ~/greetings || echo 'FAILED'", node)
+
+\end{lstlisting}
+
+In order to guarantee that the second application is successful, we need to
+make sure that the first application is executed first. For this we register
+a condition:
+
+\begin{lstlisting}[language=Python]
+
+ec.register_condition (app2, ResourceAction.START, app1, ResourceState.STOPPED)
+
+\end{lstlisting}
+
+We then deploy the two application:
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(guids=[app1,app2])
+
+\end{lstlisting}
+
+Finally, we retrieve the standard output of the second application,
+which should return the string "HELLO WORLD!".
+
+\begin{lstlisting}[language=Python]
+
+ec.trace(app2, "stdout")
+\end{lstlisting}
another. Indeed, different platforms provide their own mechanisms to
access resources and different tools to conduct experiments.
These tools vary widely, for instance, to run a ns-3 simulation it is
-necessary to write a C++ program, while to conduct and experiment using
+necessary to write a C++ program, while to conduct an experiment using
PlanetLab nodes, one must first provision resources through a special web
service, and then connect to the nodes using SSH to launch any applications
involved in the experiment.
can be a time consuming and difficult task, specially if distributed
resources need to be synchronised to perform the right action at the
right time. Detecting and handling possible errors during experiment
-execution also posses a challenge, even more so when dealing with large size
+execution also posses a challenge, even more when dealing with large size
experiments. Additionally, difficulties related to instrumenting the
experiment and gathering the results must also be considered.
% Challenge
In this context, the challenges that NEPI addresses are manifold.
-Firstly, simplifying the complexity of running network experiments.
-Secondly, simplifying the use of different experimentation platforms,
+Firstly, to simplify the complexity of running network experiments.
+Secondly, to simplify the use of different experimentation platforms,
allowing to easily switch from one to another.
-Thirdly, simplifying the
+Thirdly, to simplify the
use of resources from different platforms at the same time in
a single experiment.
connections. The attributes define the configuration of the resource,
the traces represent the results that can be collected for that resource
during the experiment and the connections represent how a resource relates
-to other resources in th experiment.
+to other resources in the experiment.
\begin{figure}[h]
\centering
\label{fig:intro_resources}
\end{figure}
-Examples of attributes are a linux host host name, an IP address to be
+Examples of attributes are a linux hostname, an IP address to be
assigned to a network interface, a command to run as a remote application.
Examples of traces are the standard output or standard error of a
running application, a tcpdump on a network interface, etc.
an \emph{interactive deployment} mode, where new resources can de
declared and deployed on the fly, according to the experiment needs.
This interactive mode can be useful to run experiments with the
-purpose of exploring a new technology, or to use NEPI as a adaptive
+purpose of exploring a new technology, or to use NEPI as an adaptive
experimentation tool, that could change an experiment according to
-changing external conditions or measurements.
+external conditions or measurements.
\section{Resource Management: The EC \& The RMs}
For each resource that the user registers in the \emph{topology graph}, the EC
will instantiate a RM of a corresponding type. A RM is a resource specific
-controller and different types of resources require a different type of
-RM, specifically adapted to manage them.
+controller and different types of resources require different type of
+RMs, specifically adapted to manage them.
The EC communicates with the RMs through a well defined API that exposes
the necessary methods (actions) to achieve all the state transitions defined by the
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Releases in NEPI do not occur in strictly regular periods. Usually a new
+release will be done every 4 or 5 months.
+
+\section{The development branch}
+
+The main development branch for NEPI 3 is \emph{nepi-3-dev}.
+There might be other branches to develop new features, but they will
+eventually end up being merged into the \emph{nepi-3-dev} branch.
+
+\section{Versioning}
+
+Releases are named following the \emph{major.minor.revision} convention.
+The \emph{major} number reflects a major change in functionality or architecture.
+It is to be expected that this number will remind in 3 for a long period.
+
+The \emph{minor} number reflects the incorporation of new features into NEPI.
+This number is expected to be increased on each release.
+
+The \emph{revision} number is incremented when a considerable number of
+bugs have been fixed. No release will be done when only the \emph{revision}
+number is incremented.
+
+\section{The release process}
+
+The creation of a new NEPI release will always follow the same sequence of steps.
+
+\begin{enumerate}
+ \item A new \emph{nepi-3.<minor>-pre-release} branch will be created from the \emph{nepi-3-dev} branch
+ \item During two to three weeks intensive work on testing will be carried out on the new branch. No new functionality will be added on this branch and only changes that fix bugs will be accepted.
+ \item At the end of this period, the pre-release branch will be branched again into the release branch, named \emph{nepi-3.<minor>-release}
+ \item A tag will be added to this new branch including the revision number (i.e. \emph{release-3.<minor>.<revision>})
+ \item Finally, the pre-release branch will be merged into the development branch, nepi-3-dev, to incorporate to it all the bug fixes.
+\end{enumerate}
+
%%TODO: Explain how to create patch
-TODO
\section{Linux resources}
\begin{itemize}
+ \item Linux Node (Clean home, etc)
\item SSH
\item The directory structure
- \item Linux Node (Clean home, etc)
- \item The api (run, exceute, x11, etc)
\item Traces and collection
\item Linux Application
\item LinuxPing, LinuxTraceroute, etc
title=\lstname % show the filename of files included with \lstinputlisting; also try caption instead of title
}
-
\title{NEPI v3.0 User Manual}
\date{}
\author{}
+\renewcommand{\maketitlehookd}{%
+ \begin{center}
+ \includegraphics[width=6cm]{nepi_logo.png}
+\end{center}}
\begin{document}
%%\label{unit_testing}
%%\input{unit_testing.tex}
-%%\chapter{Release Cycle}
-%%\label{release_cycle}
-%%\input{release_cycle.tex}
+\chapter{Release Cycle}
+\label{release_cycle}
+\input{release_cycle.tex}
%%\chapter{Coding Style}
%%\label{coding_style}
--- /dev/null
+#!/usr/bin/env python
+#
+# NEPI, a framework to manage network experiments
+# Copyright (C) 2013 INRIA
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Author: Alina Quereilhac <alina.quereilhac@inria.fr>
+
+from nepi.execution.ec import ExperimentController
+
+ec = ExperimentController(exp_id = "ping-exp")
+
+node = ec.register_resource("LinuxNode")
+ec.set(node, "hostname", host)
+ec.set(node, "username", username)
+ec.set(node, "cleanHome", True)
+ec.set(node, "cleanProcesses", True)
+
+app = ec.register_resource("LinuxApplication")
+ec.set(app, "command", "ping -c3 www.google.com")
+ec.register_connection(app, node)
+
+ec.deploy()
+
+ec.wait_finished(app)
+
+print ec.trace(app, "stdout")
+
+
+ec.shutdown()
return app
# Create the EC
-ec = ExperimentController(exp_id = "one")
+ec = ExperimentController(exp_id = "test")
switch1 = "planetlab2.virtues.fi"
switch2 = "planetlab2.upc.es"
host1 = "planetlab2.ionio.gr"
-host2 = "planetlab2.diku.dk"
+host2 = "iraplab2.iralab.uni-karlsruhe.de"
slicename = "inria_nepi"
pl_user = os.environ.get("PL_USER")
-pl_password = os.environ.get("PL_PASS")
+pl_password = os.environ.get("PL_PASS")
s1_node = add_node(ec, switch1, slicename, pl_user, pl_password)
s2_node = add_node(ec, switch2, slicename, pl_user, pl_password)
import weakref
class FailureLevel(object):
- """ Describes the system failure state
- """
+ """ Describes the system failure state """
OK = 1
RM_FAILURE = 2
EC_FAILURE = 3
class FailureManager(object):
- """ The FailureManager is responsible for handling errors,
- and deciding whether an experiment should be aborted
+ """ The FailureManager is responsible for handling errors
+ and deciding whether an experiment should be aborted or not
+
"""
def __init__(self, ec):
@property
def ec(self):
- """ Returns the Experiment Controller """
+ """ Returns the ExperimentController associated to this FailureManager
+
+ """
+
return self._ec()
@property
class ECState(object):
- """ State of the Experiment Controller
+ """ Possible states for an ExperimentController
"""
RUNNING = 1
.. note::
- An experiment, or scenario, is defined by a concrete set of resources,
- behavior, configuration and interconnection of those resources.
- The Experiment Description (ED) is a detailed representation of a
- single experiment. It contains all the necessary information to
- allow repeating the experiment. NEPI allows to describe
- experiments by registering components (resources), configuring them
- and interconnecting them.
-
- A same experiment (scenario) can be executed many times, generating
- different results. We call an experiment execution (instance) a 'run'.
-
- The ExperimentController (EC), is the entity responsible of
- managing an experiment run. The same scenario can be
- recreated (and re-run) by instantiating an EC and recreating
- the same experiment description.
-
- In NEPI, an experiment is represented as a graph of interconnected
- resources. A resource is a generic concept in the sense that any
- component taking part of an experiment, whether physical of
- virtual, is considered a resource. A resources could be a host,
- a virtual machine, an application, a simulator, a IP address.
-
- A ResourceManager (RM), is the entity responsible for managing a
- single resource. ResourceManagers are specific to a resource
- type (i.e. An RM to control a Linux application will not be
- the same as the RM used to control a ns-3 simulation).
- To support a new type of resource in NEPI, a new RM must be
- implemented. NEPI already provides a variety of
- RMs to control basic resources, and new can be extended from
- the existing ones.
-
- Through the EC interface the user can create ResourceManagers (RMs),
- configure them and interconnect them, to describe an experiment.
- Describing an experiment through the EC does not run the experiment.
- Only when the 'deploy()' method is invoked on the EC, the EC will take
- actions to transform the 'described' experiment into a 'running' experiment.
-
- While the experiment is running, it is possible to continue to
- create/configure/connect RMs, and to deploy them to involve new
- resources in the experiment (this is known as 'interactive' deployment).
-
- An experiments in NEPI is identified by a string id,
- which is either given by the user, or automatically generated by NEPI.
- The purpose of this identifier is to separate files and results that
- belong to different experiment scenarios.
- However, since a same 'experiment' can be run many times, the experiment
- id is not enough to identify an experiment instance (run).
- For this reason, the ExperimentController has two identifier, the
- exp_id, which can be re-used in different ExperimentController,
- and the run_id, which is unique to one ExperimentController instance, and
- is automatically generated by NEPI.
+ An experiment, or scenario, is defined by a concrete set of resources,
+ behavior, configuration and interconnection of those resources.
+ The Experiment Description (ED) is a detailed representation of a
+ single experiment. It contains all the necessary information to
+ allow repeating the experiment. NEPI allows to describe
+ experiments by registering components (resources), configuring them
+ and interconnecting them.
+
+ A same experiment (scenario) can be executed many times, generating
+ different results. We call an experiment execution (instance) a 'run'.
+
+ The ExperimentController (EC), is the entity responsible of
+ managing an experiment run. The same scenario can be
+ recreated (and re-run) by instantiating an EC and recreating
+ the same experiment description.
+
+ In NEPI, an experiment is represented as a graph of interconnected
+ resources. A resource is a generic concept in the sense that any
+ component taking part of an experiment, whether physical of
+ virtual, is considered a resource. A resources could be a host,
+ a virtual machine, an application, a simulator, a IP address.
+
+ A ResourceManager (RM), is the entity responsible for managing a
+ single resource. ResourceManagers are specific to a resource
+ type (i.e. An RM to control a Linux application will not be
+ the same as the RM used to control a ns-3 simulation).
+ To support a new type of resource in NEPI, a new RM must be
+ implemented. NEPI already provides a variety of
+ RMs to control basic resources, and new can be extended from
+ the existing ones.
+
+ Through the EC interface the user can create ResourceManagers (RMs),
+ configure them and interconnect them, to describe an experiment.
+ Describing an experiment through the EC does not run the experiment.
+ Only when the 'deploy()' method is invoked on the EC, the EC will take
+ actions to transform the 'described' experiment into a 'running' experiment.
+
+ While the experiment is running, it is possible to continue to
+ create/configure/connect RMs, and to deploy them to involve new
+ resources in the experiment (this is known as 'interactive' deployment).
+
+ An experiments in NEPI is identified by a string id,
+ which is either given by the user, or automatically generated by NEPI.
+ The purpose of this identifier is to separate files and results that
+ belong to different experiment scenarios.
+ However, since a same 'experiment' can be run many times, the experiment
+ id is not enough to identify an experiment instance (run).
+ For this reason, the ExperimentController has two identifier, the
+ exp_id, which can be re-used in different ExperimentController,
+ and the run_id, which is unique to one ExperimentController instance, and
+ is automatically generated by NEPI.
"""
def __init__(self, exp_id = None):
super(ExperimentController, self).__init__()
+
# Logging
self._logger = logging.getLogger("ExperimentController")
@property
def logger(self):
- """ Return the logger of the Experiment Controller
+ """ Returns the logger instance of the Experiment Controller
"""
return self._logger
@property
def ecstate(self):
- """ Return the state of the Experiment Controller
+ """ Returns the state of the Experiment Controller
"""
return self._state
@property
def exp_id(self):
- """ Return the experiment id assigned by the user
+ """ Returns the experiment id assigned by the user
"""
return self._exp_id
@property
def run_id(self):
- """ Return the experiment instance (run) identifier
+ """ Returns the experiment instance (run) identifier (automatically
+ generated)
"""
return self._run_id
@property
def abort(self):
+ """ Returns True if the experiment has failed and should be interrupted,
+ False otherwise.
+
+ """
return self._fm.abort
def wait_finished(self, guids):
- """ Blocking method that wait until all RMs in the 'guid' list
- reach a state >= STOPPED (i.e. FINISHED, STOPPED, FAILED or
- RELEASED ) or until a System Failure occurs (e.g. Task Failure)
-
- :param guids: List of guids
- :type guids: list
+ """ Blocking method that waits until all RMs in the 'guids' list
+ have reached a state >= STOPPED (i.e. STOPPED, FAILED or
+ RELEASED ), or until a failure in the experiment occurs
+ (i.e. abort == True)
+
+ :param guids: List of guids
+ :type guids: list
"""
quit = quit)
def wait_started(self, guids):
- """ Blocking method that wait until all RMs in the 'guid' list
- reach a state >= STARTED or until a System Failure occurs
- (e.g. Task Failure)
+ """ Blocking method that waits until all RMs in the 'guids' list
+ have reached a state >= STARTED, or until a failure in the
+ experiment occurs (i.e. abort == True)
+
+ :param guids: List of guids
+ :type guids: list
- :param guids: List of guids
- :type guids: list
"""
def quit():
quit = quit)
def wait_released(self, guids):
- """ Blocking method that wait until all RMs in the 'guid' list
- reach a state = RELEASED or until the EC fails
+ """ Blocking method that waits until all RMs in the 'guids' list
+ have reached a state == RELEASED, or until the EC fails
+
+ :param guids: List of guids
+ :type guids: list
- :param guids: List of guids
- :type guids: list
"""
def quit():
quit = quit)
def wait_deployed(self, guids):
- """ Blocking method that wait until all RMs in the 'guid' list
- reach a state >= READY or until a System Failure occurs
- (e.g. Task Failure)
+ """ Blocking method that waits until all RMs in the 'guids' list
+ have reached a state >= READY, or until a failure in the
+ experiment occurs (i.e. abort == True)
+
+ :param guids: List of guids
+ :type guids: list
- :param guids: List of guids
- :type guids: list
"""
def quit():
quit = quit)
def wait(self, guids, state, quit):
- """ Blocking method that wait until all RMs in the 'guid' list
- reach a state >= 'state' or until quit yileds True
+ """ Blocking method that waits until all RMs in the 'guids' list
+ have reached a state >= 'state', or until the 'quit' callback
+ yields True
- :param guids: List of guids
- :type guids: list
+ :param guids: List of guids
+ :type guids: list
+
"""
+
if isinstance(guids, int):
guids = [guids]
guids.remove(guid)
rm = self.get_resource(guid)
self.logger.debug(" %s guid %d DONE - state is %s, required is >= %s " % (
- rm.rtype(), guid, hrrstate, hrstate))
+ rm.get_rtype(), guid, hrrstate, hrstate))
else:
# Debug...
self.logger.debug(" WAITING FOR guid %d - state is %s, required is >= %s " % (
time.sleep(0.5)
def get_task(self, tid):
- """ Get a specific task
+ """ Returns a task by its id
- :param tid: Id of the task
- :type tid: int
- :rtype: Task
+ :param tid: Id of the task
+ :type tid: int
+
+ :rtype: Task
+
"""
return self._tasks.get(tid)
def get_resource(self, guid):
- """ Get a specific Resource Manager
+ """ Returns a registered ResourceManager by its guid
- :param guid: Id of the task
- :type guid: int
- :rtype: ResourceManager
+ :param guid: Id of the task
+ :type guid: int
+
+ :rtype: ResourceManager
+
"""
return self._resources.get(guid)
@property
def resources(self):
- """ Returns the list of all the Resource Manager Id
+ """ Returns the set() of guids of all the ResourceManager
- :rtype: set
+ :return: Set of all RM guids
+ :rtype: set
"""
return self._resources.keys()
def register_resource(self, rtype, guid = None):
- """ Register a Resource Manager. It creates a new 'guid', if it is not specified,
- for the RM of type 'rtype' and add it to the list of Resources.
+ """ Registers a new ResourceManager of type 'rtype' in the experiment
+
+ This method will assign a new 'guid' for the RM, if no guid
+ is specified.
+
+ :param rtype: Type of the RM
+ :type rtype: str
- :param rtype: Type of the RM
- :type rtype: str
- :return: Id of the RM
- :rtype: int
+ :return: Guid of the RM
+ :rtype: int
+
"""
# Get next available guid
guid = self._guid_generator.next(guid)
return guid
def get_attributes(self, guid):
- """ Return all the attibutes of a specific RM
+ """ Returns all the attributes of the RM with guid 'guid'
+
+ :param guid: Guid of the RM
+ :type guid: int
+
+ :return: List of attributes
+ :rtype: list
- :param guid: Guid of the RM
- :type guid: int
- :return: List of attributes
- :rtype: list
"""
rm = self.get_resource(guid)
return rm.get_attributes()
+ def get_attribute(self, guid, name):
+ """ Returns the attribute 'name' of the RM with guid 'guid'
+
+ :param guid: Guid of the RM
+ :type guid: int
+
+ :param name: Name of the attribute
+ :type name: str
+
+ :return: The attribute with name 'name'
+ :rtype: Attribute
+
+ """
+ rm = self.get_resource(guid)
+ return rm.get_attribute(name)
+
def register_connection(self, guid1, guid2):
- """ Registers a guid1 with a guid2.
- The declaration order is not important
+ """ Registers a connection between a RM with guid 'guid1'
+ and another RM with guid 'guid2'.
+
+ The order of the in which the two guids are provided is not
+ important, since the connection relationship is symmetric.
:param guid1: First guid to connect
:type guid1: ResourceManager
:param guid2: Second guid to connect
:type guid: ResourceManager
+
"""
rm1 = self.get_resource(guid1)
rm2 = self.get_resource(guid2)
def register_condition(self, guids1, action, guids2, state,
time = None):
- """ Registers an action START or STOP for all RM on guids1 to occur
- time 'time' after all elements in guids2 reached state 'state'.
+ """ Registers an action START, STOP or DEPLOY for all RM on list
+ guids1 to occur at time 'time' after all elements in list guids2
+ have reached state 'state'.
:param guids1: List of guids of RMs subjected to action
:type guids1: list
- :param action: Action to register (either START or STOP)
+ :param action: Action to perform (either START, STOP or DEPLOY)
:type action: ResourceAction
:param guids2: List of guids of RMs to we waited for
:type guids2: list
- :param state: State to wait for on RMs (STARTED, STOPPED, etc)
+ :param state: State to wait for on RMs of list guids2 (STARTED,
+ STOPPED, etc)
:type state: ResourceState
:param time: Time to wait after guids2 has reached status
rm.register_condition(action, guids2, state, time)
def enable_trace(self, guid, name):
- """ Enable trace
+ """ Enables a trace to be collected during the experiment run
+
+ :param name: Name of the trace
+ :type name: str
- :param name: Name of the trace
- :type name: str
"""
rm = self.get_resource(guid)
rm.enable_trace(name)
def trace_enabled(self, guid, name):
- """ Returns True if trace is enabled
+ """ Returns True if the trace of name 'name' is enabled
+
+ :param name: Name of the trace
+ :type name: str
- :param name: Name of the trace
- :type name: str
"""
rm = self.get_resource(guid)
return rm.trace_enabled(name)
def trace(self, guid, name, attr = TraceAttr.ALL, block = 512, offset = 0):
- """ Get information on collected trace
+ """ Returns information on a collected trace, the trace stream or
+ blocks (chunks) of the trace stream
- :param name: Name of the trace
- :type name: str
+ :param name: Name of the trace
+ :type name: str
- :param attr: Can be one of:
+ :param attr: Can be one of:
- TraceAttr.ALL (complete trace content),
- - TraceAttr.STREAM (block in bytes to read starting at offset),
+ - TraceAttr.STREAM (block in bytes to read starting
+ at offset),
- TraceAttr.PATH (full path to the trace file),
- TraceAttr.SIZE (size of trace file).
- :type attr: str
+ :type attr: str
- :param block: Number of bytes to retrieve from trace, when attr is TraceAttr.STREAM
- :type name: int
+ :param block: Number of bytes to retrieve from trace, when attr is
+ TraceAttr.STREAM
+ :type name: int
- :param offset: Number of 'blocks' to skip, when attr is TraceAttr.STREAM
- :type name: int
+ :param offset: Number of 'blocks' to skip, when attr is TraceAttr.STREAM
+ :type name: int
+
+ :rtype: str
- :rtype: str
"""
rm = self.get_resource(guid)
return rm.trace(name, attr, block, offset)
+ def get_traces(self, guid):
+ """ Returns the list of the trace names of the RM with guid 'guid'
+
+ :param guid: Guid of the RM
+ :type guid: int
+
+ :return: List of trace names
+ :rtype: list
+
+ """
+ rm = self.get_resource(guid)
+ return rm.get_traces()
+
+
def discover(self, guid):
- """ Discover a specific RM defined by its 'guid'
+ """ Discovers an available resource matching the criteria defined
+ by the RM with guid 'guid', and associates that resource to the RM
+
+ Not all RM types require (or are capable of) performing resource
+ discovery. For the RM types which are not capable of doing so,
+ invoking this method does not have any consequences.
:param guid: Guid of the RM
:type guid: int
return rm.discover()
def provision(self, guid):
- """ Provision a specific RM defined by its 'guid'
+ """ Provisions the resource associated to the RM with guid 'guid'.
+
+ Provisioning means making a resource 'accessible' to the user.
+ Not all RM types require (or are capable of) performing resource
+ provisioning. For the RM types which are not capable of doing so,
+ invoking this method does not have any consequences.
:param guid: Guid of the RM
:type guid: int
return rm.provision()
def get(self, guid, name):
- """ Get a specific attribute 'name' from the RM 'guid'
+ """ Returns the value of the attribute with name 'name' on the
+ RM with guid 'guid'
:param guid: Guid of the RM
:type guid: int
- :param name: attribute's name
+ :param name: Name of the attribute
:type name: str
+ :return: The value of the attribute with name 'name'
+
"""
rm = self.get_resource(guid)
return rm.get(name)
def set(self, guid, name, value):
- """ Set a specific attribute 'name' from the RM 'guid'
- with the value 'value'
+ """ Modifies the value of the attribute with name 'name' on the
+ RM with guid 'guid'.
:param guid: Guid of the RM
:type guid: int
- :param name: attribute's name
+ :param name: Name of the attribute
:type name: str
- :param value: attribute's value
+ :param value: Value of the attribute
"""
rm = self.get_resource(guid)
return state
def stop(self, guid):
- """ Stop a specific RM defined by its 'guid'
+ """ Stops the RM with guid 'guid'
+
+ Stopping a RM means that the resource it controls will
+ no longer take part of the experiment.
:param guid: Guid of the RM
:type guid: int
return rm.stop()
def start(self, guid):
- """ Start a specific RM defined by its 'guid'
+ """ Starts the RM with guid 'guid'
+
+ Starting a RM means that the resource it controls will
+ begin taking part of the experiment.
:param guid: Guid of the RM
:type guid: int
def set_with_conditions(self, name, value, guids1, guids2, state,
time = None):
- """ Set value 'value' on attribute with name 'name' on all RMs of
- guids1 when 'time' has elapsed since all elements in guids2
- have reached state 'state'.
+ """ Modifies the value of attribute with name 'name' on all RMs
+ on the guids1 list when time 'time' has elapsed since all
+ elements in guids2 list have reached state 'state'.
:param name: Name of attribute to set in RM
:type name: string
rm.set_with_conditions(name, value, guids2, state, time)
def deploy(self, guids = None, wait_all_ready = True, group = None):
- """ Deploy all resource manager in guids list
+ """ Deploys all ResourceManagers in the guids list.
+
+ If the argument 'guids' is not given, all RMs with state NEW
+ are deployed.
- :param guids: List of guids of RMs to deploy
- :type guids: list
+ :param guids: List of guids of RMs to deploy
+ :type guids: list
- :param wait_all_ready: Wait until all RMs are ready in
- order to start the RMs
- :type guid: int
+ :param wait_all_ready: Wait until all RMs are ready in
+ order to start the RMs
+ :type guid: int
- :param group: Id of deployment group in which to deploy RMs
- :type group: int
+ :param group: Id of deployment group in which to deploy RMs
+ :type group: int
"""
self.logger.debug(" ------- DEPLOY START ------ ")
self.schedule("0s", rm.stop_with_conditions)
def release(self, guids = None):
- """ Release al RMs on the guids list or
- all the resources if no list is specified
+ """ Releases all ResourceManagers in the guids list.
+
+ If the argument 'guids' is not given, all RMs registered
+ in the experiment are released.
:param guids: List of RM guids
:type guids: list
self.wait_released(guids)
def shutdown(self):
- """ Shutdown the Experiment Controller.
- Releases all the resources and stops task processing thread
+ """ Releases all resources and stops the ExperimentController
"""
# If there was a major failure we can't exit gracefully
self._thread.join()
def schedule(self, date, callback, track = False):
- """ Schedule a callback to be executed at time date.
+ """ Schedules a callback to be executed at time 'date'.
:param date: string containing execution time for the task.
It can be expressed as an absolute time, using
Python function, and receives args and kwargs
as arguments.
- :param track: if set to True, the task will be retrivable with
+ :param track: if set to True, the task will be retrievable with
the get_task() method
:return : The Id of the task
+ :rtype: int
+
"""
timestamp = stabsformat(date)
task = Task(timestamp, callback)
""" Process scheduled tasks.
.. note::
+
+ Tasks are scheduled by invoking the schedule method with a target
+ callback and an execution time.
+ The schedule method creates a new Task object with that callback
+ and execution time, and pushes it into the '_scheduler' queue.
+ The execution time and the order of arrival of tasks are used
+ to order the tasks in the queue.
+
+ The _process method is executed in an independent thread held by
+ the ExperimentController for as long as the experiment is running.
+ This method takes tasks from the '_scheduler' queue in a loop
+ and processes them in parallel using multithreading.
+ The environmental variable NEPI_NTHREADS can be used to control
+ the number of threads used to process tasks. The default value is
+ 50.
- The _process method is executed in an independent thread held by the
- ExperimentController for as long as the experiment is running.
+ To execute tasks in parallel, a ParallelRunner (PR) object is used.
+ This object keeps a pool of threads (workers), and a queue of tasks
+ scheduled for 'immediate' execution.
- Tasks are scheduled by invoking the schedule method with a target callback.
- The schedule method is given a execution time which controls the
- order in which tasks are processed.
+ On each iteration, the '_process' loop will take the next task that
+ is scheduled for 'future' execution from the '_scheduler' queue,
+ and if the execution time of that task is >= to the current time,
+ it will push that task into the PR for 'immediate execution'.
+ As soon as a worker is free, the PR will assign the next task to
+ that worker.
- Tasks are processed in parallel using multithreading.
- The environmental variable NEPI_NTHREADS can be used to control
- the number of threads used to process tasks. The default value is 50.
-
- Exception handling:
-
- To execute tasks in parallel, an ParallelRunner (PR) object, holding
- a pool of threads (workers), is used.
- For each available thread in the PR, the next task popped from
- the scheduler queue is 'put' in the PR.
- Upon receiving a task to execute, each PR worker (thread) invokes the
- _execute method of the EC, passing the task as argument.
- This method, calls task.callback inside a try/except block. If an
- exception is raised by the tasks.callback, it will be trapped by the
- try block, logged to standard error (usually the console), and the EC
- state will be set to ECState.FAILED.
- The invocation of _notify immediately after, forces the processing
- loop in the _process method, to wake up if it was blocked waiting for new
- tasks to arrived, and to check the EC state.
- As the EC is in FAILED state, the processing loop exits and the
- 'finally' block is invoked. In the 'finally' block, the 'sync' method
- of the PR is invoked, which forces the PR to raise any unchecked errors
- that might have been raised by the workers.
+ Upon receiving a task to execute, each PR worker (thread) will
+ invoke the _execute method of the EC, passing the task as
+ argument.
+ The _execute method will then invoke task.callback inside a
+ try/except block. If an exception is raised by the tasks.callback,
+ it will be trapped by the try block, logged to standard error
+ (usually the console), and the task will be marked as failed.
"""
:param task: Object containing the callback to execute
:type task: Task
- .. note::
-
- If the invokation of the task callback raises an
- exception, the processing thread of the ExperimentController
- will be stopped and the experiment will be aborted.
-
"""
# Invoke callback
task.status = TaskStatus.DONE
self.logger.error("Error occurred while executing task: %s" % err)
def _notify(self):
- """ Awakes the processing thread in case it is blocked waiting
- for a new task to be scheduled.
+ """ Awakes the processing thread if it is blocked waiting
+ for new tasks to arrive
+
"""
self._cond.acquire()
self._cond.notify()
READY = 3
STARTED = 4
STOPPED = 5
- FINISHED = 6
- FAILED = 7
- RELEASED = 8
+ FAILED = 6
+ RELEASED = 7
ResourceState2str = dict({
ResourceState.NEW : "NEW",
ResourceState.READY : "READY",
ResourceState.STARTED : "STARTED",
ResourceState.STOPPED : "STOPPED",
- ResourceState.FINISHED : "FINISHED",
ResourceState.FAILED : "FAILED",
ResourceState.RELEASED : "RELEASED",
})
def failtrap(func):
""" Decorator function for instance methods that should set the
RM state to FAILED when an error is raised. The methods that must be
- decorated are: discover, provision, deploy, start, stop and finish.
+ decorated are: discover, provision, deploy, start, stop.
"""
def wrapped(self, *args, **kwargs):
cls._register_traces()
@classmethod
- def rtype(cls):
+ def get_rtype(cls):
""" Returns the type of the Resource Manager
"""
"""
return copy.deepcopy(cls._attributes.values())
+ @classmethod
+ def get_attribute(cls, name):
+ """ Returns a copy of the attribute with name 'name'
+
+ """
+ return copy.deepcopy(cls._attributes[name])
+
+
@classmethod
def get_traces(cls):
""" Returns a copy of the traces
return cls._backend
def __init__(self, ec, guid):
- super(ResourceManager, self).__init__(self.rtype())
+ super(ResourceManager, self).__init__(self.get_rtype())
self._guid = guid
self._ec = weakref.ref(ec)
self._provision_time = None
self._ready_time = None
self._release_time = None
- self._finish_time = None
self._failed_time = None
self._state = ResourceState.NEW
""" Returns the release time of the RM as a timestamp """
return self._release_time
- @property
- def finish_time(self):
- """ Returns the finalization time of the RM as a timestamp """
- return self._finish_time
-
@property
def failed_time(self):
""" Returns the time failure occured for the RM as a timestamp """
self.set_released()
self.debug("----- RELEASED ---- ")
- @failtrap
- def finish(self):
- """ Sets the RM to state FINISHED.
-
- The FINISHED state is different from STOPPED state in that it
- should not be directly invoked by the user.
- STOPPED indicates that the user interrupted the RM, FINISHED means
- that the RM concluded normally the actions it was supposed to perform.
-
- This method should not be overriden directly. Specific functionality
- should be added in the do_finish method.
-
- """
- with self._release_lock:
- if self._state != ResourceState.RELEASED:
- self.do_finish()
-
def fail(self):
""" Sets the RM to state FAILED.
t = rm.start_time
elif state == ResourceState.STOPPED:
t = rm.stop_time
- elif state == ResourceState.FINISHED:
- t = rm.finish_time
elif state == ResourceState.RELEASED:
t = rm.release_time
else:
def do_release(self):
pass
- def do_finish(self):
- # In case the RM passed from STARTED directly to FINISHED,
- # we set the stop_time for consistency
- if self.stop_time == None:
- self.set_stopped()
-
- self.set_finished()
-
def do_fail(self):
self.set_failed()
""" Mark ResourceManager as REALEASED """
self.set_state(ResourceState.RELEASED, "_release_time")
- def set_finished(self):
- """ Mark ResourceManager as FINISHED """
- self.set_state(ResourceState.FINISHED, "_finish_time")
-
def set_failed(self):
""" Mark ResourceManager as FAILED """
self.set_state(ResourceState.FAILED, "_failed_time")
@classmethod
def register_type(cls, rclass):
"""Register a new Ressource Manager"""
- cls._resource_types[rclass.rtype()] = rclass
+ cls._resource_types[rclass.get_rtype()] = rclass
@classmethod
def create(cls, rtype, ec, guid):
"""
- def __init__(self, name, help):
+ def __init__(self, name, help, enabled = False):
self._name = name
self._help = help
- self.enabled = False
+ self.enabled = enabled
@property
def name(self):
.. note::
- A LinuxApplication RM represents a process that can be executed in
- a remote Linux host using SSH.
+ A LinuxApplication RM represents a process that can be executed in
+ a remote Linux host using SSH.
- The LinuxApplication RM takes care of uploadin sources and any files
- needed to run the experiment, to the remote host.
- It also allows to provide source compilation (build) and installation
- instructions, and takes care of automating the sources build and
- installation tasks for the user.
+ The LinuxApplication RM takes care of uploadin sources and any files
+ needed to run the experiment, to the remote host.
+ It also allows to provide source compilation (build) and installation
+ instructions, and takes care of automating the sources build and
+ installation tasks for the user.
- It is important to note that files uploaded to the remote host have
- two possible scopes: single-experiment or multi-experiment.
- Single experiment files are those that will not be re-used by other
- experiments. Multi-experiment files are those that will.
- Sources and shared files are always made available to all experiments.
+ It is important to note that files uploaded to the remote host have
+ two possible scopes: single-experiment or multi-experiment.
+ Single experiment files are those that will not be re-used by other
+ experiments. Multi-experiment files are those that will.
+ Sources and shared files are always made available to all experiments.
- Directory structure:
+ Directory structure:
- The directory structure used by LinuxApplication RM at the Linux
- host is the following:
+ The directory structure used by LinuxApplication RM at the Linux
+ host is the following:
${HOME}/nepi-usr --> Base directory for multi-experiment files
|
@classmethod
def _register_traces(cls):
- stdout = Trace("stdout", "Standard output stream")
- stderr = Trace("stderr", "Standard error stream")
+ stdout = Trace("stdout", "Standard output stream", enabled = True)
+ stderr = Trace("stderr", "Standard error stream", enabled = True)
cls._register_trace(stdout)
cls._register_trace(stderr)
@property
def node(self):
- node = self.get_connected(LinuxNode.rtype())
+ node = self.get_connected(LinuxNode.get_rtype())
if node: return node[0]
return None
super(LinuxApplication, self).do_provision()
- def upload_start_command(self):
+ def upload_start_command(self, overwrite = False):
# Upload command to remote bash script
# - only if command can be executed in background and detached
command = self.get("command")
self.node.upload_command(command,
shfile = shfile,
env = env,
- overwrite = False)
+ overwrite = overwrite)
def execute_deploy_command(self, command):
if command:
if not command:
# If no command was given (i.e. Application was used for dependency
- # installation), then the application is directly marked as FINISHED
- super(LinuxApplication, self).do_finish()
+ # installation), then the application is directly marked as STOPPED
+ super(LinuxApplication, self).set_stopped()
else:
if self.in_foreground:
self._run_in_foreground()
self.do_fail()
elif retcode == 0:
- self.do_finish()
+ self.set_stopped()
else:
# We need to query the status of the command we launched in
# background. In order to avoid overwhelming the remote host and
self.error(msg, out, err)
self.do_fail()
else:
- self.do_finish()
+ self.set_stopped()
self._last_state_check = tnow()
@property
def ccnd(self):
- ccnd = self.get_connected(LinuxCCND.rtype())
+ ccnd = self.get_connected(LinuxCCND.get_rtype())
if ccnd: return ccnd[0]
return None
@property
def ccnr(self):
- ccnr = self.get_connected(LinuxCCNR.rtype())
+ ccnr = self.get_connected(LinuxCCNR.get_rtype())
if ccnr: return ccnr[0]
return None
if retcode == 1 and err.find("No such file or directory") > -1:
# ccnd is not running (socket not found)
- self.set_finished()
+ self.set_stopped()
elif retcode:
# other errors ...
msg = " Failed to execute command '%s'" % self.get("command")
@property
def ccnpingserver(self):
- ccnpingserver = self.get_connected(LinuxCCNPingServer.rtype())
+ ccnpingserver = self.get_connected(LinuxCCNPingServer.get_rtype())
if ccnpingserver: return ccnpingserver[0]
return None
@property
def ccnd(self):
- ccnd = self.get_connected(LinuxCCND.rtype())
+ ccnd = self.get_connected(LinuxCCND.get_rtype())
if ccnd: return ccnd[0]
return None
@property
def ccnd(self):
- ccnd = self.get_connected(LinuxCCND.rtype())
+ ccnd = self.get_connected(LinuxCCND.get_rtype())
if ccnd: return ccnd[0]
return None
@property
def node(self):
- node = self.get_connected(LinuxNode.rtype())
+ node = self.get_connected(LinuxNode.get_rtype())
if node: return node[0]
return None
@property
def channel(self):
- chan = self.get_connected(LinuxChannel.rtype())
+ chan = self.get_connected(LinuxChannel.get_rtype())
if chan: return chan[0]
return None
# To work arround this, repeat the operation N times or
# until the result is not empty string
out = ""
- retrydelay = 1.0
- for i in xrange(2):
- try:
- (out, err), proc = self.execute("cat /etc/issue",
- retry = 5,
- with_lock = True,
- blocking = True)
-
- if out.strip() != "":
- return out
- except:
- trace = traceback.format_exc()
- msg = "Error detecting OS: %s " % trace
- self.error(msg, out, err)
- return False
-
- time.sleep(min(30.0, retrydelay))
- retrydelay *= 1.5
+ try:
+ (out, err), proc = self.execute("cat /etc/issue",
+ with_lock = True,
+ blocking = True)
+ except:
+ trace = traceback.format_exc()
+ msg = "Error detecting OS: %s " % trace
+ self.error(msg, out, err)
+
+ return out
@property
def use_deb(self):
# Node needs to wait until all associated interfaces are
# ready before it can finalize deployment
from nepi.resources.linux.interface import LinuxInterface
- ifaces = self.get_connected(LinuxInterface.rtype())
+ ifaces = self.get_connected(LinuxInterface.get_rtype())
for iface in ifaces:
if iface.state < ResourceState.READY:
self.ec.schedule(reschedule_delay, self.deploy)
# TODO: Validate!
return True
- def clean_processes(self, killer = False):
+ def clean_processes(self):
self.info("Cleaning up processes")
- if killer:
- # Hardcore kill
- cmd = ("sudo -S killall python tcpdump || /bin/true ; " +
- "sudo -S killall python tcpdump || /bin/true ; " +
- "sudo -S kill $(ps -N -T -o pid --no-heading | grep -v $PPID | sort) || /bin/true ; " +
- "sudo -S killall -u root || /bin/true ; " +
- "sudo -S killall -u root || /bin/true ; ")
- else:
- # Be gentler...
- cmd = ("sudo -S killall tcpdump || /bin/true ; " +
- "sudo -S killall tcpdump || /bin/true ; " +
- "sudo -S killall -u %s || /bin/true ; " % self.get("username") +
- "sudo -S killall -u %s || /bin/true ; " % self.get("username"))
+ cmd = ("sudo -S killall tcpdump || /bin/true ; " +
+ "sudo -S kill $(ps aux | grep '[n]epi' | awk '{print $2}') || /bin/true ; " +
+ "sudo -S killall -u %s || /bin/true ; " % self.get("username"))
out = err = ""
(out, err), proc = self.execute(cmd, retry = 1, with_lock = True)
if self.localhost:
(out, err), proc = execfuncs.lexec(command,
- user = user,
+ user = self.get("username"), # still problem with localhost
sudo = sudo,
stdin = stdin,
env = env)
return True
out = err = ""
+ msg = "Unresponsive host. Wrong answer. "
+
# The underlying SSH layer will sometimes return an empty
# output (even if the command was executed without errors).
# To work arround this, repeat the operation N times or
# until the result is not empty string
- retrydelay = 1.0
- for i in xrange(2):
- try:
- (out, err), proc = self.execute("echo 'ALIVE'",
- retry = 5,
- blocking = True,
- with_lock = True)
-
- if out.find("ALIVE") > -1:
- return True
- except:
- trace = traceback.format_exc()
- msg = "Unresponsive host. Error reaching host: %s " % trace
- self.error(msg, out, err)
- return False
-
- time.sleep(min(30.0, retrydelay))
- retrydelay *= 1.5
+ try:
+ (out, err), proc = self.execute("echo 'ALIVE'",
+ blocking = True,
+ with_lock = True)
+
+ if out.find("ALIVE") > -1:
+ return True
+ except:
+ trace = traceback.format_exc()
+ msg = "Unresponsive host. Error reaching host: %s " % trace
- if out.find("ALIVE") > -1:
- return True
- else:
- msg = "Unresponsive host. Wrong answer. "
- self.error(msg, out, err)
- return False
+ self.error(msg, out, err)
+ return False
def find_home(self):
""" Retrieves host home directory
# output (even if the command was executed without errors).
# To work arround this, repeat the operation N times or
# until the result is not empty string
- retrydelay = 1.0
- for i in xrange(2):
- try:
- (out, err), proc = self.execute("echo ${HOME}",
- retry = 5,
- blocking = True,
- with_lock = True)
-
- if out.strip() != "":
- self._home_dir = out.strip()
- break
- except:
- trace = traceback.format_exc()
- msg = "Impossible to retrieve HOME directory" % trace
- self.error(msg, out, err)
- return False
-
- time.sleep(min(30.0, retrydelay))
- retrydelay *= 1.5
+ msg = "Impossible to retrieve HOME directory"
+ try:
+ (out, err), proc = self.execute("echo ${HOME}",
+ blocking = True,
+ with_lock = True)
+
+ if out.strip() != "":
+ self._home_dir = out.strip()
+ except:
+ trace = traceback.format_exc()
+ msg = "Impossible to retrieve HOME directory %s" % trace
if not self._home_dir:
- msg = "Impossible to retrieve HOME directory"
- self.error(msg, out, err)
+ self.error(msg)
raise RuntimeError, msg
def filter_existing_files(self, src, dst):
self.error(msg, err1, err2)
self.fail()
else:
- self.set_finished()
+ self.set_stopped()
self._last_state_check = tnow()
@property
def node(self):
- rm_list = self.get_connected(OMFNode.rtype())
+ rm_list = self.get_connected(OMFNode.get_rtype())
if rm_list: return rm_list[0]
return None
"""
rm = self.ec.get_resource(guid)
- if rm.rtype() not in self._authorized_connections:
+ if rm.get_rtype() not in self._authorized_connections:
msg = ("Connection between %s %s and %s %s refused: "
"An Application can be connected only to a Node" ) % \
- (self.rtype(), self._guid, rm.rtype(), guid)
+ (self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return False
elif len(self.connections) != 0 :
msg = ("Connection between %s %s and %s %s refused: "
"This Application is already connected" ) % \
- (self.rtype(), self._guid, rm.rtype(), guid)
+ (self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return False
else :
msg = "Connection between %s %s and %s %s accepted" % (
- self.rtype(), self._guid, rm.rtype(), guid)
+ self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return True
self.set('env', " ")
# Some information to check the information in parameter
- msg = " " + self.rtype() + " ( Guid : " + str(self._guid) +") : " + \
+ msg = " " + self.get_rtype() + " ( Guid : " + str(self._guid) +") : " + \
self.get('appid') + " : " + self.get('path') + " : " + \
self.get('args') + " : " + self.get('env')
self.info(msg)
"""
rm = self.ec.get_resource(guid)
- if rm.rtype() in self._authorized_connections:
+ if rm.get_rtype() in self._authorized_connections:
msg = "Connection between %s %s and %s %s accepted" % (
- self.rtype(), self._guid, rm.rtype(), guid)
+ self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return True
msg = "Connection between %s %s and %s %s refused" % (
- self.rtype(), self._guid, rm.rtype(), guid)
+ self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return False
rm_iface = self.ec.get_resource(elt)
for conn in rm_iface.connections:
rm_node = self.ec.get_resource(conn)
- if rm_node.rtype() == "OMFNode" and rm_node.get('hostname'):
+ if rm_node.get_rtype() == "OMFNode" and rm_node.get('hostname'):
if rm_iface.state < ResourceState.PROVISIONED or \
rm_node.state < ResourceState.READY:
return "reschedule"
"""
rm = self.ec.get_resource(guid)
- if rm.rtype() in self._authorized_connections:
+ if rm.get_rtype() in self._authorized_connections:
msg = "Connection between %s %s and %s %s accepted" % \
- (self.rtype(), self._guid, rm.rtype(), guid)
+ (self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return True
msg = "Connection between %s %s and %s %s refused" % \
- (self.rtype(), self._guid, rm.rtype(), guid)
+ (self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return False
@property
def node(self):
- rm_list = self.get_connected(OMFNode.rtype())
+ rm_list = self.get_connected(OMFNode.get_rtype())
if rm_list: return rm_list[0]
return None
@property
def channel(self):
- rm_list = self.get_connected(OMFChannel.rtype())
+ rm_list = self.get_connected(OMFChannel.get_rtype())
if rm_list: return rm_list[0]
return None
raise RuntimeError, msg
# Just for information
- self.debug(" " + self.rtype() + " ( Guid : " + str(self._guid) +") : " + \
+ self.debug(" " + self.get_rtype() + " ( Guid : " + str(self._guid) +") : " + \
self.get('mode') + " : " + self.get('type') + " : " + \
self.get('essid') + " : " + self.get('ip'))
"""
rm = self.ec.get_resource(guid)
- if rm.rtype() in self._authorized_connections:
+ if rm.get_rtype() in self._authorized_connections:
msg = "Connection between %s %s and %s %s accepted" % (
- self.rtype(), self._guid, rm.rtype(), guid)
+ self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return True
msg = "Connection between %s %s and %s %s refused" % (
- self.rtype(), self._guid, rm.rtype(), guid)
+ self.get_rtype(), self._guid, rm.get_rtype(), guid)
self.debug(msg)
return False
from nepi.util.logger import Logger
-import sleekxmpp
-from sleekxmpp.exceptions import IqError, IqTimeout
+
+try:
+ import sleekxmpp
+ from sleekxmpp.exceptions import IqError, IqTimeout
+ class BaseOMFClient(sleekxmpp.ClientXMPP):
+ pass
+except ImportError:
+ print "SleekXMPP is not installed. Without this library, \n" + \
+ " You will be not able to use OMF Resources \n"+ \
+ " If you want to install SleekXmpp : \n"+ \
+ " git clone -b develop git://github.com/fritzy/SleekXMPP.git \n"+ \
+ " cd SleekXMPP \n"+ \
+ "sudo python setup.py install\n"
+ class BaseOMFClient(object):
+ pass
+
import traceback
import xml.etree.ElementTree as ET
self._plapi = None
self._node_to_provision = None
self._slicenode = False
+
+ def _skip_provision(self):
+ pl_user = self.get("pluser")
+ pl_pass = self.get("plpassword")
+ if not pl_user and not pl_pass:
+ return True
+ else: return False
@property
def plapi(self):
pl_ptn = self.get("plcApiPattern")
self._plapi = PLCAPIFactory.get_api(pl_user, pl_pass, pl_url,
- pl_ptn)
+ pl_ptn)
if not self._plapi:
self.fail_plapi()
Based on the attributes defined by the user, discover the suitable
nodes for provision.
"""
+ if self._skip_provision():
+ super(PlanetlabNode, self).do_discover()
+ return
+
hostname = self._get_hostname()
if hostname:
# the user specified one particular node to be provisioned
# more nodes can match these constraints
nodes = self._filter_based_on_attributes()
nodes_alive = self._query_if_alive(nodes)
-
+
# nodes that are already part of user's slice have the priority to
# provisioned
nodes_inslice = self._check_if_in_slice(nodes_alive)
Add node to user's slice after verifing that the node is functioning
correctly
"""
+ if self._skip_provision():
+ super(PlanetlabNode, self).do_provision()
+ return
+
provision_ok = False
ssh_ok = False
proc_ok = False
range, by the user
"""
node_tags = self.plapi.get_node_tags(filters)
- if node_tags is not None:
+ if node_tags:
if len(nodes_id) == 0:
# first attribute being matched
alive_nodes_id = self._get_nodes_id(filters)
if len(alive_nodes_id) == 0:
- self.fail_node_not_alive(self, hostname)
+ self.fail_node_not_alive(hostname)
else:
nodes_id = list()
for node_id in alive_nodes_id:
ip = self._get_ip(node_id)
if not ip: return ping_ok
- command = "ping -c2 %s" % ip
+ command = "ping -c4 %s" % ip
(out, err) = lexec(command)
- if not out.find("2 received") < 0:
+ if not out.find("2 received") or not out.find("3 received") or not \
+ out.find("4 received") < 0:
ping_ok = True
return ping_ok
raise RuntimeError, msg
def fail_plapi(self):
- msg = "Failing while trying to instanciate the PLC API"
+ msg = "Failing while trying to instanciate the PLC API.\nSet the" + \
+ " attributes pluser and plpassword."
raise RuntimeError, msg
def valid_connection(self, guid):
@property
def node(self):
- node = self.get_connected(PlanetlabNode.rtype())
+ node = self.get_connected(PlanetlabNode.get_rtype())
if node: return node[0]
return None
# """
# rm = self.ec.get_resource(guid)
-# if rm.rtype() in self._authorized_connections:
+# if rm.get_rtype() in self._authorized_connections:
# msg = "Connection between %s %s and %s %s accepted" % \
-# (self.rtype(), self._guid, rm.rtype(), guid)
+# (self.get_rtype(), self._guid, rm.get_rtype(), guid)
# self.debug(msg)
# return True
# msg = "Connection between %s %s and %s %s refused" % \
-# (self.rtype(), self._guid, rm.rtype(), guid)
+# (self.get_rtype(), self._guid, rm.get_rtype(), guid)
# self.debug(msg)
# return False
(out, err), proc = self.node.check_output(self.ovs_checks, 'start_srv_exitcode')
if out != "0\n":
- self.debug("Servers have not started")
+ self.error("Servers have not started")
raise RuntimeError, msg
cmd = "ps -A | grep ovsdb-server"
(out, err), proc = self.node.check_output(self.ovs_checks, 'status_srv_exitcode')
if out != "0\n":
- self.debug("Servers are not running")
+ self.error("Servers are not running")
raise RuntimeError, msg
self.info("Servers started")
# Node needs to wait until all associated RMs are released
# to be released
from nepi.resources.planetlab.openvswitch.ovsport import OVSPort
- rm = self.get_connected(OVSPort.rtype())
+ rm = self.get_connected(OVSPort.get_rtype())
- if rm[0].state < ResourceState.FINISHED:
+ if rm[0].state < ResourceState.STOPPED:
self.ec.schedule(reschedule_delay, self.release)
return
@property
def node(self):
- rm_list = self.get_connected(OVSWitch.rtype())
+ rm_list = self.get_connected(OVSWitch.get_rtype())
if rm_list:
for elt in rm_list:
- node = elt.get_connected(PlanetlabNode.rtype())
+ node = elt.get_connected(PlanetlabNode.get_rtype())
if node: return node[0]
return node[0]
@property
def ovswitch(self):
- ovswitch = self.get_connected(OVSWitch.rtype())
+ ovswitch = self.get_connected(OVSWitch.get_rtype())
if ovswitch: return ovswitch[0]
return None
# """
# rm = self.ec.get_resource(guid)
-# if rm.rtype() in self._authorized_connections:
-# msg = "Connection between %s %s and %s %s accepted" % (self.rtype(), self._guid, rm.rtype(), guid)
+# if rm.get_rtype() in self._authorized_connections:
+# msg = "Connection between %s %s and %s %s accepted" % (self.get_rtype(), self._guid, rm.get_rtype(), guid)
# self.debug(msg)
# return True
-# msg = "Connection between %s %s and %s %s refused" % (self.rtype(), self._guid, rm.rtype(), guid)
+# msg = "Connection between %s %s and %s %s refused" % (self.get_rtype(), self._guid, rm.get_rtype(), guid)
# self.debug(msg)
def get_host_ip(self):
# OVS needs to wait until all associated RMs are released
# to be released
from nepi.resources.planetlab.openvswitch.tunnel import OVSTunnel
- rm = self.get_connected(OVSTunnel.rtype())
+ rm = self.get_connected(OVSTunnel.get_rtype())
- if rm and rm[0].state < ResourceState.FINISHED:
+ if rm and rm[0].state < ResourceState.STOPPED:
self.ec.schedule(reschedule_delay, self.release)
return
# Get connected to the nodes
res = []
if hasattr(endpoint, "create_port"):
- rm_list = endpoint.get_connected(OVSWitch.rtype())
+ rm_list = endpoint.get_connected(OVSWitch.get_rtype())
if rm_list:
- rm = rm_list[0].get_connected(PlanetlabNode.rtype())
+ rm = rm_list[0].get_connected(PlanetlabNode.get_rtype())
else:
- rm = endpoint.get_connected(PlanetlabNode.rtype())
+ rm = endpoint.get_connected(PlanetlabNode.get_rtype())
if rm :
res.append(rm[0])
@property
def node(self):
- node = self.get_connected(PlanetlabNode.rtype())
+ node = self.get_connected(PlanetlabNode.get_rtype())
if node: return node[0]
return None
stop_command = self.replace_paths(self._stop_command)
self.node.upload(stop_command,
os.path.join(self.app_home, "stop.sh"),
- text = True,
- overwrite = False)
+ text = True,
+ # Overwrite file every time.
+ # The stop.sh has the path to the socket, wich should change
+ # on every experiment run.
+ overwrite = True)
def upload_start_command(self):
- super(PlanetlabTap, self).upload_start_command()
+ # Overwrite file every time.
+ # The stop.sh has the path to the socket, wich should change
+ # on every experiment run.
+ super(PlanetlabTap, self).upload_start_command(overwrite = True)
# We want to make sure the device is up and running
# before the deploy finishes (so things will be ready
if out.strip().find(self.get("deviceName")) == -1:
# tap is not running is not running (socket not found)
- self.finish()
+ self.set_stopped()
self._last_state_check = tnow()
# Node needs to wait until all associated RMs are released
# to be released
from nepi.resources.linux.udptunnel import UdpTunnel
- rms = self.get_connected(UdpTunnel.rtype())
+ rms = self.get_connected(UdpTunnel.get_rtype())
for rm in rms:
if rm.state < ResourceState.STOPPED:
self.ec.schedule(reschedule_delay, self.release)
super(Interface, self).__init__(ec, guid)
def do_deploy(self):
- node = self.get_connected(Node.rtype())[0]
- chan = self.get_connected(Channel.rtype())[0]
+ node = self.get_connected(Node.get_rtype())[0]
+ chan = self.get_connected(Channel.get_rtype())[0]
if node.state < ResourceState.PROVISIONED:
self.ec.schedule("0.5s", self.deploy)
self.logger.debug(" -------- PROVISIONED ------- ")
self.ec.schedule("1s", self.deploy)
elif self.state == ResourceState.PROVISIONED:
- ifaces = self.get_connected(Interface.rtype())
+ ifaces = self.get_connected(Interface.get_rtype())
for rm in ifaces:
if rm.state < ResourceState.READY:
self.ec.schedule("0.5s", self.deploy)
super(Application, self).__init__(ec, guid)
def do_deploy(self):
- node = self.get_connected(Node.rtype())[0]
+ node = self.get_connected(Node.get_rtype())[0]
if node.state < ResourceState.READY:
self.ec.schedule("0.5s", self.deploy)
else:
super(ErrorApplication, self).__init__(ec, guid)
def do_deploy(self):
- node = self.get_connected(Node.rtype())[0]
+ node = self.get_connected(Node.get_rtype())[0]
if node.state < ResourceState.READY:
self.ec.schedule("0.5s", self.deploy)
else:
ResourceFactory.register_type(MyResource)
ResourceFactory.register_type(AnotherResource)
- self.assertEquals(MyResource.rtype(), "MyResource")
+ self.assertEquals(MyResource.get_rtype(), "MyResource")
self.assertEquals(len(MyResource._attributes), 2)
- self.assertEquals(ResourceManager.rtype(), "Resource")
+ self.assertEquals(ResourceManager.get_rtype(), "Resource")
self.assertEquals(len(ResourceManager._attributes), 1)
- self.assertEquals(AnotherResource.rtype(), "AnotherResource")
+ self.assertEquals(AnotherResource.get_rtype(), "AnotherResource")
self.assertEquals(len(AnotherResource._attributes), 1)
self.assertEquals(len(ResourceFactory.resource_types()), 2)
return wrapped
+def skipIfNotPLCredentials(func):
+ name = func.__name__
+ def wrapped(*args, **kwargs):
+ pl_user = os.environ.get("PL_USER")
+ pl_pass = os.environ.get("PL_PASS")
+ pl_slice = os.environ.get("PL_SLICE")
+ if not (pl_user and pl_pass and pl_slice):
+ print "*** WARNING: Skipping test %s: Planetlab user, password and slicename not defined\n" % name
+ return
+
+ return func(*args, **kwargs)
+
+ return wrapped
ec.set(node, "cleanHome", True)
ec.set(node, "cleanProcesses", True)
- sources = "http://nepi.inria.fr/code/nef/archive/tip.tar.gz " \
- " http://nepi.inria.fr/code/nef/raw-file/8ace577d4079/src/nef/images/menu/connect.png"
+ sources = "http://yans.pl.sophia.inria.fr/code/nef/archive/tip.tar.gz " \
+ " http://yans.pl.sophia.inria.fr/code/nef/raw-file/8ace577d4079/src/nef/images/menu/connect.png"
app = ec.register_resource("LinuxApplication")
ec.set(app, "sources", sources)
class OMFResourceFactoryTestCase(unittest.TestCase):
def test_creation_phase(self):
- self.assertEquals(OMFNode.rtype(), "OMFNode")
+ self.assertEquals(OMFNode.get_rtype(), "OMFNode")
self.assertEquals(len(OMFNode._attributes), 6)
- self.assertEquals(OMFWifiInterface.rtype(), "OMFWifiInterface")
+ self.assertEquals(OMFWifiInterface.get_rtype(), "OMFWifiInterface")
self.assertEquals(len(OMFWifiInterface._attributes), 10)
- self.assertEquals(OMFChannel.rtype(), "OMFChannel")
+ self.assertEquals(OMFChannel.get_rtype(), "OMFChannel")
self.assertEquals(len(OMFChannel._attributes), 6)
- self.assertEquals(OMFApplication.rtype(), "OMFApplication")
+ self.assertEquals(OMFApplication.get_rtype(), "OMFApplication")
self.assertEquals(len(OMFApplication._attributes), 13)
class OMFEachTestCase(unittest.TestCase):
from nepi.resources.planetlab.node import PlanetlabNode
from nepi.resources.planetlab.plcapi import PLCAPI, PLCAPIFactory
+from test_utils import skipIfNotPLCredentials
+
import os
import time
import unittest
import multiprocessing
-def create_node(ec, username, pl_user, pl_password, hostname = None,
- country = None, operatingSystem = None, minBandwidth = None,
- minCpu = None):
+class DummyEC(ExperimentController):
+ pass
+
+def create_node(ec, username, pl_user=None, pl_password=None, pl_url=None,
+ hostname=None, country=None, operatingSystem=None,
+ minBandwidth=None, minCpu=None,
+ architecture=None, city=None):
node = ec.register_resource("PlanetlabNode")
ec.set(node, "pluser", pl_user)
if pl_password:
ec.set(node, "plpassword", pl_password)
-
+ if pl_url:
+ ec.set(node, "plcApiUrl", pl_url)
+
if hostname:
ec.set(node, "hostname", hostname)
if country:
iec.set(node, "minBandwidth", minBandwidth)
if minCpu:
ec.set(node, "minCpu", minCpu)
+ if architecture:
+ ec.set(node, "architecture", architecture)
+ if city:
+ ec.set(node, "city", city)
ec.set(node, "cleanHome", True)
ec.set(node, "cleanProcesses", True)
return node
class PLNodeFactoryTestCase(unittest.TestCase):
- def test_creation_phase(self):
- self.assertEquals(PlanetlabNode.rtype(), "PlanetlabNode")
- self.assertEquals(len(PlanetlabNode._attributes), 29)
+ def test_creation_phase(self):
+ self.assertEquals(PlanetlabNode._rtype, "PlanetlabNode")
+ self.assertEquals(len(PlanetlabNode._attributes), 30)
class PLNodeTestCase(unittest.TestCase):
"""
- This tests use inria_sfatest slice, and planetlab2.utt.fr already added to
+ This tests use inria_nepi slice, and already added to
the slice, and ONLY this one in order for the test not to fail.
"""
def setUp(self):
- self.ec = ExperimentController()
- self.username = "inria_sfatest"
+ self.ec = DummyEC()
+ self.username = 'inria_nepi'
self.pl_user = os.environ.get("PL_USER")
self.pl_password = os.environ.get("PL_PASS")
- self.pl_url = "www.planet-lab.eu"
- self.pl_ptn = "https://%(hostname)s:443/PLCAPI/"
+ self.pl_url = "nepiplc.pl.sophia.inria.fr"
+ @skipIfNotPLCredentials
def test_plapi(self):
"""
Check that the api to discover and reserve resources is well
instanciated, and is an instance of PLCAPI. Ignore error while
executing the ec.shutdown method, the error is due to the name
of the host not being defined yet for this test.
- """
- node1 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France")
+ """
+ node1 = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url,
+ architecture="x86_64")
plnode_rm1 = self.ec.get_resource(node1)
hostname = plnode_rm1.get("hostname")
self.assertEquals(len(api1.reserved()), 0)
self.assertEquals(len(api1.blacklisted()), 0)
- node2 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France")
+ node2 = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url,
+ architecture="x86_64")
plnode_rm2 = self.ec.get_resource(node2)
api2 = plnode_rm2.plapi
self.assertEquals(api1, api2)
-
+
# Set hostname attribute in order for the shutdown method not to fail
- plnode_rm1._set_hostname_attr(7057)
- plnode_rm2._set_hostname_attr(7057)
+ plnode_rm1._set_hostname_attr(3)
+ plnode_rm2._set_hostname_attr(3)
+
+ def test_no_plcredentials(self):
+ """
+ Check that if no PL credentials are set, the PlanetlabNode skip the
+ discover and provision for the node.
+ """
+ node = create_node(self.ec, self.username, hostname='nepi2.pl.sophia.inria.fr')
+ plnode_rm = self.ec.get_resource(node)
+
+ plnode_rm.do_discover()
+ plnode_rm.do_provision()
+
+ self.assertIsNone(plnode_rm._node_to_provision)
+ @skipIfNotPLCredentials
def test_discover_inslice(self):
"""
- This test uses the fact that the node planetlab2.utt.fr is already in
- the slice and match the constraints OS Fedora12 and country France.
- Check planetlab2.utt.fr is alive if the test fails.
+ This test uses the fact that the node nepi2.pl.sophia.inria.fr is already in
+ the slice and match the constraints OS Fedora12 and arch x86_64.
+ Check nepi2.pl.sophia.inria.fr is alive if the test fails.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f12")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url, architecture="x86_64",
+ operatingSystem="f12")
plnode_rm = self.ec.get_resource(node)
-
+
hostname = plnode_rm.get("hostname")
self.assertIsNone(hostname)
- plnode_rm.discover()
- self.assertEquals(plnode_rm._node_to_provision, 7057)
+ api = plnode_rm.plapi
+ api.add_slice_nodes(self.username, ['nepi2.pl.sophia.inria.fr'])
+
+ plnode_rm.do_discover()
+ self.assertEquals(plnode_rm._node_to_provision, 3)
- # Set hostname attribute in order for the shutdown method not to fail
+ # Set hostname attribute in order for the shutdown method not to fail
plnode_rm._set_hostname_attr(plnode_rm._node_to_provision)
+ @skipIfNotPLCredentials
def test_discover_not_inslice(self):
"""
This test checks that if the node is not in the slice, anyway the
discover method picks one that match constraints outside from the
slice.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url, country="France",
+ operatingSystem="f12")
plnode_rm = self.ec.get_resource(node)
- plnode_rm.discover()
+ plnode_rm.do_discover()
- result = [14281, 1034, 7035] # nodes matching f14 and France
+ result = [4]
self.assertIn(plnode_rm._node_to_provision, result)
self.assertIsNot(plnode_rm.plapi.reserved(), set())
# Set hostname attribute in order for the shutdown method not to fail
plnode_rm._set_hostname_attr(plnode_rm._node_to_provision)
+ @skipIfNotPLCredentials
def test_discover_hostname(self):
"""
This test checks that if the user specify the hostname, only that node
is discovered.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, hostname="planetlab1.sics.se")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url,
+ hostname="nepi2.pl.sophia.inria.fr")
plnode_rm = self.ec.get_resource(node)
- plnode_rm.discover()
+ plnode_rm.do_discover()
- self.assertEquals(plnode_rm._node_to_provision, 14871)
- self.assertEquals(plnode_rm.plapi.reserved(), set([14871]))
+ self.assertEquals(plnode_rm._node_to_provision, 3)
+ self.assertEquals(plnode_rm.plapi.reserved(), set([3]))
+ @skipIfNotPLCredentials
def test_discover_with_ranges(self):
"""
Checks that defining max or min attributes, the discover method works.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, minCpu=50) #minBandwidth=500)
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url, minCpu=50)
plnode_rm = self.ec.get_resource(node)
- plnode_rm.discover()
+ plnode_rm.do_discover()
- #result = [15815, 15814, 425, 417, 1054, 1102, 1107, 505, 1031]
- result = [425, 15815, 15814, 14842, 427, 41, 14466]
+ result = [6]
self.assertIn(plnode_rm._node_to_provision, result)
self.assertIsNot(plnode_rm.plapi.reserved(), set())
# Set hostname attribute in order for the shutdown method not to fail
plnode_rm._set_hostname_attr(plnode_rm._node_to_provision)
-
+
+ @skipIfNotPLCredentials
def test_blacklist_nodes(self):
"""
Test that if the node is malfunctioning it gets blacklisted, the node
- planetlab-1a.ics.uci.edu is used, if the test fails, check that the
+ nepi1.pl.sophia.inria.fr is used, if the test fails, check that the
result of the plcapi query is actually empty.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, hostname="planetlab-1a.ics.uci.edu")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url,
+ hostname="nepi1.pl.sophia.inria.fr")
plnode_rm = self.ec.get_resource(node)
self.assertEquals(plnode_rm.plapi.blacklisted(), set())
# check that the node is actually malfunctioning
api = plnode_rm.plapi
- filters = {'boot_state': 'boot', '>last_contact': 1378299413,
- 'node_type': 'regular', 'hostname': 'planetlab-1a.ics.uci.edu',
- 'run_level': 'boot'}
+ filters = {'boot_state': 'boot', 'node_type': 'regular',
+ 'hostname': 'nepi1.pl.sophia.inria.fr', 'run_level': 'boot',
+ '>last_contact': int(time.time()) - 2*3600}
node_id = api.get_nodes(filters, fields=['node_id'])
if not node_id:
with self.assertRaises(RuntimeError):
- plnode_rm.discover()
- self.assertEquals(plnode_rm.plapi.blacklisted(), set([14871]))
+ plnode_rm.do_discover()
+ self.assertEquals(plnode_rm.plapi.blacklisted(), set([1]))
+ @skipIfNotPLCredentials
def test_provision_node_inslice(self):
"""
- Check provision of the node planetlab2.utt.fr.
+ Check provision of the node nepi2.pl.sophia.inria.fr.
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f12")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url,
+ architecture="x86_64", operatingSystem="f12")
plnode_rm = self.ec.get_resource(node)
self.assertEquals(len(plnode_rm.plapi.blacklisted()), 0)
self.assertEquals(len(plnode_rm.plapi.reserved()), 0)
- plnode_rm.discover()
- plnode_rm.provision()
+ api = plnode_rm.plapi
+ api.add_slice_nodes(self.username, ['nepi2.pl.sophia.inria.fr'])
+
+ plnode_rm.do_discover()
+ plnode_rm.do_provision()
ip = plnode_rm.get("ip")
- self.assertEquals(ip, "194.254.215.12")
+ self.assertEquals(ip, "138.96.116.32")
self.assertEquals(len(plnode_rm.plapi.reserved()), 1)
+ @skipIfNotPLCredentials
def test_provision_node_not_inslice(self):
"""
- Check provision of one of the nodes f14 France, nodes:
- node1pl.planet-lab.telecom-lille1.eu
- ple5.ipv6.lip6.fr
- node2pl.planet-lab.telecom-lille1.eu
+ Check provision of one of the nodes f12, nodes:
+ 'nepi3.pl.sophia.inria.fr'
+ 'nepi5.pl.sophia.inria.fr'
"""
- node = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
+ node = create_node(self.ec, self.username, pl_user=self.pl_user,
+ pl_password=self.pl_password, pl_url=self.pl_url, operatingSystem="f12",
+ city='Paris')
plnode_rm = self.ec.get_resource(node)
self.assertEquals(plnode_rm.plapi.blacklisted(), set())
self.assertEquals(plnode_rm.plapi.reserved(), set())
- plnode_rm.discover()
- plnode_rm.provision()
+ api = plnode_rm.plapi
+ api.add_slice_nodes(self.username, ['nepi2.pl.sophia.inria.fr'])
+
+ plnode_rm.do_discover()
+ plnode_rm.do_provision()
ip = plnode_rm.get("ip")
- result = ["194.167.254.18","132.227.62.123","194.167.254.19"]
+ result = ["138.96.116.33","138.96.116.35"]
self.assertIn(ip, result)
-
- def test_provision_more_than_available(self):
- """
- Check that if the user wants to provision 4 nodes with fedora 14, he
- gets RuntimeError, there are only 3 nodes f14.
- """
- node1 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- plnode_rm1 = self.ec.get_resource(node1)
- plnode_rm1.discover()
- plnode_rm1.provision()
-
- node2 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- plnode_rm2 = self.ec.get_resource(node2)
- plnode_rm2.discover()
- plnode_rm2.provision()
-
- node3 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- plnode_rm3 = self.ec.get_resource(node3)
- with self.assertRaises(RuntimeError):
- plnode_rm3.discover()
- with self.assertRaises(RuntimeError):
- plnode_rm3.provision()
-
- node4 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- plnode_rm4 = self.ec.get_resource(node4)
- with self.assertRaises(RuntimeError):
- plnode_rm4.discover()
-
- host1 = plnode_rm1.get('hostname')
-
- plnode_rm3._set_hostname_attr(host1)
- plnode_rm4._set_hostname_attr(host1)
-
- def test_concurrence(self):
- """
- Test with the nodes being discover and provision at the same time.
- """
- node1 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- node2 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- node3 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- node4 = create_node(self.ec, self.username, self.pl_user,
- self.pl_password, country="France", operatingSystem="f14")
-
- self.ec.deploy()
- self.ec.wait_finished([node1, node2, node3, node4])
- state = self.ec.ecstate
- self.assertEquals(state, 2)
+
+# @skipIfNotPLCredentials
+# def test_provision_more_than_available(self):
+# """
+# Check that if the user wants to provision 3 nodes in Paris, he
+# gets RuntimeError, there are only 2 nodes with city=Paris.
+# """
+# node1 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# city="Paris", operatingSystem="f12")
+#
+# plnode_rm1 = self.ec.get_resource(node1)
+#
+# api = plnode_rm.plapi
+# api.add_slice_nodes(self.username, ['nepi2.pl.sophia.inria.fr'])
+#
+# plnode_rm1.do_discover()
+# plnode_rm1.do_provision()
+#
+# node2 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# city="Paris", operatingSystem="f12")
+#
+# plnode_rm2 = self.ec.get_resource(node2)
+# plnode_rm2.do_discover()
+# plnode_rm2.do_provision()
+#
+# node3 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# city="Paris", operatingSystem="f12")
+#
+# plnode_rm3 = self.ec.get_resource(node3)
+# with self.assertRaises(RuntimeError):
+# plnode_rm3.do_discover()
+# with self.assertRaises(RuntimeError):
+# plnode_rm3.do_provision()
+#
+# host1 = plnode_rm1.get('hostname')
+#
+# plnode_rm3._set_hostname_attr(host1)
+#
+# @skipIfNotPLCredentials
+# def test_concurrence(self):
+# """
+# Test with the nodes being discover and provision at the same time.
+# It should fail as the test before.
+# """
+# node1 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# architecture="x86_64", operatingSystem="f12")
+#
+# node2 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# architecture="x86_64", operatingSystem="f12")
+#
+# node3 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# architecture="x86_64", operatingSystem="f12")
+#
+# node4 = create_node(self.ec, self.username, pl_user=self.pl_user,
+# pl_password=self.pl_password, pl_url=self.pl_url,
+# architecture="x86_64", operatingSystem="f12")
+#
+# self.ec.deploy()
+# self.ec.wait_finished([node1, node2, node3, node4])
+# state = self.ec.state(node1, hr=True)
+# self.assertIn(state, ['READY', 'FAILED'])
+# state = self.ec.state(node2, hr=True)
+# self.assertIn(state, ['READY', 'FAILED'])
+# state = self.ec.state(node3, hr=True)
+# self.assertIn(state, ['READY', 'FAILED'])
+# state = self.ec.state(node4, hr=True)
+# self.assertIn(state, ['READY', 'FAILED'])
def tearDown(self):
commonapi=PLCAPIFactory.get_api(self.pl_user, self.pl_password,
- self.pl_url, self.pl_ptn)
+ self.pl_url)
+ #commonapi.add_slice_nodes(self.username, ['nepi2.pl.sophia.inria.fr'])
commonapi._reserved = set()
commonapi._blacklist = set()
+
self.ec.shutdown()
git://git.onelab.eu / nepi.git / commitdiff