-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% NEPI, a framework to manage network experiments
% Copyright (C) 2013 INRIA
%
% This program is free software: you can redistribute it and/or modify
-% it under the terms of the GNU General Public License as published by
-% the Free Software Foundation, either version 3 of the License, or
-% (at your option) any later version.
+% it under the terms of the GNU General Public License version 2 as
+% published by the Free Software Foundation;
%
% This program is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-NEPI is written in Python, so you will need to install Python before
-before being able to run experiments with NEPI.
+NEPI is written in Python, so you will need to install Python before
+being able to run experiments with NEPI.
NEPI is known to work on Linux (Fedore, Debian, Ubuntu) and Mac (OS X).
\section{Dependencies}
Mandatory dependencies:
\begin{itemize}
\item Python 2.6+
- \item Mercurial
+ \item Mercurial
+ \item python-ipaddr
+ \item python-networkx
+ \item python-pygraphviz
+ \item python-matplotlib
\end{itemize}
Optional dependencies:
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
- $ sudo aptitude install -y python mercurial
+ $ sudo apt-get -y install python mercurial python-ipaddr python-networkx python-pygraphviz python-matplotlib
\end{verbatim}
\endgroup
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
- $ sudo yum install -y python mercurial
+ $ sudo yum -y install python mercurial python-ipaddr python-networkx graphviz-python python-matplotlib
\end{verbatim}
\endgroup
\subsection{Install dependencies on Mac}
First install homebrew (\url{http://mxcl.github.io/homebrew/}),
-then install Python.
+then you can install Python and the rest of the dependencies as follows:
\begingroup
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
- $ brew install python
+ $ brew install python
+ $ sudo port install mercurial
+ $ sudo easy_install pip
+ $ sudo pip install ipaddr
+ $ sudo pip install networkx
+ $ sudo pip install pygraphviz
+ $ sudo pip install matplotlib
+\end{verbatim}
+
+\endgroup
+
+To use Python you will need to set the PATH environmen variable as:
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+ $ export PATH=$PATH:/usr/local/share/python
\end{verbatim}
\endgroup
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
-$ hg clone http://nepi.inria.fr/code/nepi -r nepi-3.0-release
+$ hg clone http://nepi.inria.fr/code/nepi -r nepi-3.2.0
\end{verbatim}
\endgroup
\fontsize{10pt}{12pt}\selectfont
\begin{verbatim}
- $ PYTHONATH=$PYTHONPATH:<path-to-nepi>/src python experiment.py
+ $ export PYTHONPATH=$PYTHONPATH:<path-to-nepi>/src
\end{verbatim}
\endgroup
Writing a simple NEPI expeiment script is easy.
Take a look at the example in the FAQ section \ref{faq:ping_example}.
Once you have written down the script, you can run it using
-Python. Note that since NEPI is not yet installed in your system,
+Python. If NEPI is not installed in your system,
you will need to export the path to NEPI's source code to
the PYTHONPATH environment variable, so that Python can find
NEPI's libraries.
\begin{verbatim}
$ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
- $ python first-experiment.py
+ $ cd <path-to-nepi>
+ $ python examples/linux/ping.py -a localhost
\end{verbatim}
\endgroup
-\subsection{Run interactively}
+\subsection{Run NEPI interactively}
-The Python interpreter can be used as an interactive console to execute
-Python instructions.
-We can take advantage of this feature, to interactively run experiments
-with NEPI. We use the \textit{ipython} console for the example below,
-you can install it with \textit{sudo yum/aptitude install ipython} on
-Fedora/Debian.
+The IPython console can be used as an interactive interpreter to
+execute Python instructions. We can take advantage of this feature,
+to interactively run NEPI experiments.
+We will use the IPython console for the example below.
+
+You can easily install IPython on Debian, Ubuntu, Fedora or Mac as follows:\\
+
+\textbf{Debian/Ubuntu}
\begingroup
\fontsize{10pt}{12pt}\selectfont
+\begin{verbatim}
+ $ sudo apt-get install ipython
+\end{verbatim}
+
+\endgroup
+
+\textbf{Fedora}\\
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+ $ sudo yum install ipython
+\end{verbatim}
+
+\endgroup
+
+\textbf{Mac}\\
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+ $ pip install ipython
+\end{verbatim}
+
+\endgroup
+
+Before starting, make sure to add Python and IPython source directory
+path to the PYTHONPATH environment variable
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+ $ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python:/usr/local/share/python/ipython
+\end{verbatim}
+
+\endgroup
+
+Then you can start IPython as follows:
+
\begin{verbatim}
$ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
$ ipython
\end{verbatim}
-\endgroup
-
-With ipython, if you want to paste many lines at once you will need to type
-\emph{\%cpaste} and finish the paste block with \emph{\-\-}.
+If you want to paste many lines at once in IPython, you will need
+to type \emph{\%cpaste} and finish the paste block with \emph{\-\-}.
-The first thing we have to do is to import the NEPI classes
-we will use.
-In particular we need to import the ExperimentController class.
+The first thing we need to do to describe an experiment with NEPI
+is to import the NEPI Python modules.
+In particular we need to import the ExperimentController class.
+To do this type the following in the Python console:
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Then we need to create an ExperimentController instance.
-The \textit{exp-id} argument serves as a human readable identifier
-for the experiment to be ran.
+After importing the ExperimentController class, it is possible to
+create a new instance of an the ExperimentController (EC) for
+your experiment.
+The <exp-id> argument is the name you want to give the experiment
+to identify it and distinguish it from other experiments.
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Next we will define two Python functions, one to register \emph{LinuxNode}
-resources and the other to register \emph{LinuxApplication} resources.
-A \emph{LinuxNode} resource (or ResourceManager) will serve as an abstraction
-to a Linux host resource, that can be accessed using SSH key authentication.
-A \emph{LinuxApplication} resource represents anything that can be executed
-on a Linux host as a BASH command.
+Next we will define two Python functions: \emph{add\_node} and \emph{add\_app}.
+The first one to register linux::Node resources and the second one to
+register linux::Application resources.
\begin{lstlisting}[language=Python]
%cpaste
-def add_node(ec, host, user, ssh_key):
- node = ec.register_resource("LinuxNode")
- ec.set(node, "hostname", host)
- ec.set(node, "username", user)
+def add_node(ec, hostname, username, ssh_key):
+ node = ec.register_resource("linux::Node")
+ ec.set(node, "hostname", hostname)
+ ec.set(node, "username", username)
ec.set(node, "identity", ssh_key)
- ec.set(node, "cleanHome", True)
+ ec.set(node, "cleanExperiment", True)
ec.set(node, "cleanProcesses", True)
return node
def add_app(ec, command, node):
- app = ec.register_resource("LinuxApplication")
+ app = ec.register_resource("linux::Application")
ec.set(app, "command", command)
ec.register_connection(app, node)
return app
\end{lstlisting}
-The method \textit{register\_resource} declares a resource instance to the
-Experiment Controller. The method \textit{register\_connection} indicates
-that two resource will interact during the experiment.
+The method \textit{register\_resource} registers a resource instance with the
+ExperimentController. The method \textit{register\_connection} indicates
+that two resources will interact during the experiment.
Note that invoking \textit{add\_node} or \textit{add\_app} has no effect other
than informing the EC about the resources that will be used during the experiment.
The actual deployment of the experiment requires the method \textit{deploy} to
be invoked.
-The Resource Managers (RM) that abstract the concrete resources expose
-configuration attributes. In the LinuxNode RM we set the \emph{hostname}
-and \emph{username} as attributes, while in the LinuxApplication RM
-we set the \emph{command} attribute.
-
-Apart from teh \emph{command} attribute, the \emph{LinuxApplication}
-ResourceManager exposed several other attributes
-that permit to upload, compile and install arbitrary sources,
-to run any application might be needed to run an experiment.
-More details will be given in the following sections of this document.
-
-Lets now use these functions to describe the experiment we will run.
+The linux::Node resource exposes the hostname, username and identity
+attributes. This attributes provide information about the SSH credentials
+needed to log in to the Linux host.
+The \textit{hostname} is the one that identifies the physical host you want
+to access during the experiment. The \textit{username} must correspond to a
+valid account on that host, and the \textit{identity} attribute is the
+'absolute' path to the SSH private key in your local computer that allows you
+to log in to the host.
+
+The \textit{command} attribute of the linux::Application resource
+expects a BASH command line string to be executed in the remote host.
+Apart from the \emph{command} attribute, the linux::Application
+resource exposes several other attributes that allow to upload,
+compile and install arbitrary sources.
+The add\_app function registers a connection between a linux::Node and a
+linux::Application.
+
+Lets now use these functions to describe a simple experiment.
Choose a host where you have an account, and can access using SSH
-key authentication. Define string variables with the right
-values for the \emph{hostname}, \emph{username} and path to the
-SSH public key in as \emph{ssh\_key}, and then type the following lines.
+key authentication.
\begin{lstlisting}[language=Python]
+hostname = <the-hostname>
+username = <my-username>
+identity = </home/myuser/.ssh/id_rsa>
+
node = add_node(ec, hostname, username, ssh_key)
app = add_app(ec, "ping -c3 nepi.inria.fr", node)
\end{lstlisting}
-Now that we have described our simple PING experiment, we can deploy
-it. Invoking the \emph{deploy} command will not only configure the
+The values returned by the functions add\_node and add\_app are global
+unique identifiers (guid) of the resources that were registered with the EC.
+The guid is used to reference the ResourceManager associated to a registered
+resource (for instance to retrieve results or change attribute values).
+
+Now that we have registered some resources, we can ask the ExperimentController
+(EC) to deploy them.
+Invoking the \emph{deploy} command will not only configure the
resource but also automatically launch the applications.
\begin{lstlisting}[language=Python]
\end{lstlisting}
-After some seconds, we should see some log messages informing about
-the progress of the deployment.
-If you now open another terminal and connect to the host through SSH,
-you should find some directories created by NEPI.
-You should see a directory named \emph{nepi-exp}, and under that directory
-you will find another with the identifier you gave when you created the
-experiment controller (the <exp-id>).
-Under the experiment directory, you will find directories for each of the
-resources deployed (e.g. \emph{node-1}, \emph{app-2}).
-The resource directories are named with a short string that identifies the
-type of resource (e.g. 'node', 'app', etc), followed by a unique number that
-uniquely identifies a given resource within the experiment,
-the global unique identifier (guid).
-
-From the ipython console, we can check the deployment status of each resource
-by querying the EC with the method \emph{state}.
-The argument \emph{hr} stand for `human readable`, and will return a string
-state instead of a state number.
+After some seconds, we should see some output messages informing us about the
+progress in the host deployment.
+If you now open another terminal and you connect to the host using
+SSH (as indicated below), you should see that a directory for your experiment
+has been created in the host. In the remote host you will see that two NEPI
+directories were created in the \$HOME directory: \emph{nepi-src} and \emph{nepi-exp}.
+The first one is where NEPI will store files that might be re used by many
+experiments (e.g. source code, input files) . The second directory \emph{nepi-exp},
+is where experiment specific files (e.g. results, deployment scripts) will be stored.
+
+\begingroup
+ \fontsize{10pt}{12pt}\selectfont
+
+\begin{verbatim}
+
+$ ssh -i identity username@hostname
+
+\end{verbatim}
+
+\endgroup
+
+Inside the \emph{nepi-exp} directory, you will find another directory with
+the <exp-id> assigned to your EC, and inside that directory you should find
+one directory named node-1 which will contain the files (e.g. result traces)
+associated to the LinuxNode reosurce you just deployed.
+In fact for every resource deployed associated to that host (e.g. each
+linux::Application), NEPI will create a directory to place files related to it.
+The name of the directory identifies the type of resources (e.g. 'node',
+'app', etc) and it is followed by the global unique identifier (guid).
+
+We can see if a resource finished deploying by querying its state through the EC
\begin{lstlisting}[language=Python]
\end{lstlisting}
-Once the LinuxApplication is STARTED, we can retrieve the PING output using
-stored as a \emph{trace} file on the host. For this we use use \emph{trace}
-method, and specify the resource and the trace name (i.e. stdout).
+Once a linux::Application has reached the state 'STARTED',
+we can retrieve the 'stdout' trace, which should contain the output
+of the PING command.
\begin{lstlisting}[language=Python]
\end{lstlisting}
+\subsection{Define a workflow}
+
+Now that we have introduced to the basics of NEPI, we will register
+two more applications and define a workflow where one application
+will start after the other one has finished executing.
+For this we will use the EC \textit{register\_condition} method described below:
+
+\begin{lstlisting}[language=Python]
+
+register_condition(self, guids1, action, guids2, state, time=None):
+ Registers an action START, STOP or DEPLOY for all RM on list
+ guids1 to occur at time 'time' after all elements in list guids2
+ have reached state 'state'.
+
+ :param guids1: List of guids of RMs subjected to action
+ :type guids1: list
+
+ :param action: Action to perform (either START, STOP or DEPLOY)
+ :type action: ResourceAction
+
+ :param guids2: List of guids of RMs to we waited for
+ :type guids2: list
+
+ :param state: State to wait for on RMs of list guids2 (STARTED,
+ STOPPED, etc)
+ :type state: ResourceState
+
+ :param time: Time to wait after guids2 has reached status
+ :type time: string
+
+\end{lstlisting}
+
+To use the \textit{register\_condition} method we will need to import the
+ResourceState and the ResourceAction classes
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceAction
+\end{lstlisting}
+
+Then, we register the two applications. The first application will
+wait for 5 seconds and the create a file in the host called "greetings"
+with the content "HELLO WORLD".
+The second application will read the content of the file and output it
+to standard output. If the file doesn't exist il will instead output the
+string "FAILED".
+
+\begin{lstlisting}[language=Python]
+
+app1 = add_app(ec, "sleep 5; echo 'HELLO WORLD!' > ~/greetings", node)
+app2 = add_app(ec, "cat ~/greetings || echo 'FAILED'", node)
+
+\end{lstlisting}
+
+In order to guarantee that the second application is successful, we need to
+make sure that the first application is executed first. For this we register
+a condition:
+
+\begin{lstlisting}[language=Python]
+
+ec.register_condition (app2, ResourceAction.START, app1, ResourceState.STOPPED)
+
+\end{lstlisting}
+
+We then deploy the two application:
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(guids=[app1,app2])
+
+\end{lstlisting}
+
+Finally, we retrieve the standard output of the second application,
+which should return the string "HELLO WORLD!".
+
+\begin{lstlisting}[language=Python]
+
+ec.trace(app2, "stdout")
+
+\end{lstlisting}
git://git.onelab.eu / nepi.git / blobdiff