1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
3 % NEPI, a framework to manage network experiments
4 % Copyright (C) 2013 INRIA
6 % This program is free software: you can redistribute it and/or modify
7 % it under the terms of the GNU General Public License as published by
8 % the Free Software Foundation, either version 3 of the License, or
9 % (at your option) any later version.
11 % This program is distributed in the hope that it will be useful,
12 % but WITHOUT ANY WARRANTY; without even the implied warranty of
13 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 % GNU General Public License for more details.
16 % You should have received a copy of the GNU General Public License
17 % along with this program. If not, see <http://www.gnu.org/licenses/>.
19 % Author: Alina Quereilhac <alina.quereilhac@inria.fr>
21 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
23 NEPI is written in Python, so you will need to install Python before
24 being able to run experiments with NEPI.
25 NEPI is known to work on Linux (Fedore, Debian, Ubuntu) and Mac (OS X).
27 \section{Dependencies}
29 Dependencies for NEPI vary according to the features you want to enable.
30 Make sure the following dependencies are correctly installed in your system
33 Mandatory dependencies:
39 Optional dependencies:
41 \item SleekXMPP - Required to run experiments on OMF testbeds
44 \subsection{Install dependencies on Debian/Ubuntu}
47 \fontsize{10pt}{12pt}\selectfont
50 $ sudo aptitude install -y python mercurial
55 \subsection{Install dependencies on Fedora}
58 \fontsize{10pt}{12pt}\selectfont
61 $ sudo yum install -y python mercurial
66 \subsection{Install dependencies on Mac}
68 First install homebrew (\url{http://mxcl.github.io/homebrew/}),
72 \fontsize{10pt}{12pt}\selectfont
80 \subsection{Install SleekXMPP}
82 You will need \textit{git} to get the SleekXMPP sources.
85 \fontsize{10pt}{12pt}\selectfont
88 $ git clone -b develop git://github.com/fritzy/SleekXMPP.git
90 $ sudo python setup.py install
95 \section{The source code}
97 To get NEPI's source code you will need Mercurial version
98 control system. The Mercurial NEPI repo can also be browsed online at: \\
100 \url{http://nepi.inria.fr/code/nepi/}
102 \subsection{Clone the repo}
105 \fontsize{10pt}{12pt}\selectfont
108 $ hg clone http://nepi.inria.fr/code/nepi -r nepi-3.0-release
113 \section{Install NEPI in your system}
115 You don't need to install NEPI in your system to be able to run
116 experiments. However this might be convenient if you don't
117 plan to modify or extend the sources.
119 To install NEPI, just run \emph{make install} in the NEPI source
123 \fontsize{10pt}{12pt}\selectfont
132 If you are developing your own NEPI extensions, the installed
133 NEPI version might interfere with your work.
134 In this case it is probably more convenient to tell
135 Python where to find the NEPI sources, using the PYTHONPATH
136 environmental variable, when you run a NEPI script.
139 \fontsize{10pt}{12pt}\selectfont
142 $ PYTHONPATH=$PYTHONPATH:<path-to-nepi>/src python experiment.py
147 \section{Run experiments}
149 There are two ways you can use NEPI to run your experiments.
150 The first one is writing a Python script, which will import
151 NEPI libraries, and run it.
152 The second one is in interactive mode by using Python console.
154 \subsection{Run from script}
156 Writing a simple NEPI expeiment script is easy.
157 Take a look at the example in the FAQ section \ref{faq:ping_example}.
158 Once you have written down the script, you can run it using
159 Python. Note that since NEPI is not yet installed in your system,
160 you will need to export the path to NEPI's source code to
161 the PYTHONPATH environment variable, so that Python can find
165 \fontsize{10pt}{12pt}\selectfont
168 $ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
169 $ python first-experiment.py
174 \subsection{Run NEPI interactively}
176 The IPython console can be used as an interactive interpreter to
177 execute Python instructions. We can take advantage of this feature,
178 to interactively run NEPI experiments.
179 We will use the IPython console for the example below.
181 You can easily install IPython on Debian, Ubuntu, Fedora or Mac as follows:\\
183 \textbf{Debian/Ubuntu}
186 \fontsize{10pt}{12pt}\selectfont
190 $ sudo apt-get install ipython
199 \fontsize{10pt}{12pt}\selectfont
203 $ sudo yum install ipython
212 \fontsize{10pt}{12pt}\selectfont
216 $ pip install ipython
222 Before starting, make sure to add Python and IPython source directory
223 path to the PYTHONPATH environment variable
226 \fontsize{10pt}{12pt}\selectfont
230 $ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python:/usr/local/share/python/ipython
236 Then you can start IPython as follows:
239 $ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
241 Python 2.7.3 (default, Jan 2 2013, 13:56:14)
242 Type "copyright", "credits" or "license" for more information.
244 IPython 0.13.1 -- An enhanced Interactive Python.
245 ? -> Introduction and overview of IPython's features.
246 %quickref -> Quick reference.
247 help -> Python's own help system.
248 object? -> Details about 'object', use 'object??' for extra details.
252 If you want to paste many lines at once in IPython, you will need
253 to type \emph{\%cpaste} and finish the paste block with \emph{\-\-}.
255 The first thing we need to do to describe an experiment with NEPI
256 is to import the NEPI Python modules.
257 In particular we need to import the ExperimentController class.
258 To do this type the following in the Python console:
260 \begin{lstlisting}[language=Python]
262 from nepi.execution.ec import ExperimentController
266 After importing the ExperimentController class, it is possible to
267 create a new instance of an the ExperimentController (EC) for
269 The <exp-id> argument is the name you want to give the experiment
270 to identify it and distinguish it from other experiments.
272 \begin{lstlisting}[language=Python]
274 ec = ExperimentController(exp_id = "<your-exp-id>")
278 Next we will define two Python functions: \emph{add\_node} and \emph{add\_app}.
279 The first one to register \textit{LinuxNodes} resources and the second one to
280 register LinuxApplications resources.
282 \begin{lstlisting}[language=Python]
285 def add_node(ec, hostname, username, ssh_key):
286 node = ec.register_resource("LinuxNode")
287 ec.set(node, "hostname", hostname)
288 ec.set(node, "username", username)
289 ec.set(node, "identity", ssh_key)
290 ec.set(node, "cleanHome", True)
291 ec.set(node, "cleanProcesses", True)
294 def add_app(ec, command, node):
295 app = ec.register_resource("LinuxApplication")
296 ec.set(app, "command", command)
297 ec.register_connection(app, node)
303 The method \textit{register\_resource} registers a resource instance with the
304 ExperimentController. The method \textit{register\_connection} indicates
305 that two resources will interact during the experiment.
306 Note that invoking \textit{add\_node} or \textit{add\_app} has no effect other
307 than informing the EC about the resources that will be used during the experiment.
308 The actual deployment of the experiment requires the method \textit{deploy} to
311 The \textit{LinuxNode} resource exposes the hostname, username and identity
312 attributes. This attributes provide information about the SSH credentials
313 needed to log in to the Linux host.
314 The \textit{hostname} is the one that identifies the physical host you want
315 to access during the experiment. The \textit{username} must correspond to a
316 valid account on that host, and the \textit{identity} attribute is the
317 'absolute' path to the SSH private key in your local computer that allows you
318 to log in to the host.
320 The \textit{command} attribute of the \textit{LinuxApplication} resource
321 expects a BASH command line string to be executed in the remote host.
322 Apart from the \emph{command} attribute, the \emph{LinuxApplication}
323 resource exposes several other attributes that allow to upload,
324 compile and install arbitrary sources.
325 The add\_app function registers a connection between a \textit{LinuxNode} and a
326 \textit{LinuxApplication}.
328 Lets now use these functions to describe a simple experiment.
329 Choose a host where you have an account, and can access using SSH
332 \begin{lstlisting}[language=Python]
334 hostname = "<the-hostname>"
335 username = "<my-username>"
336 identity = "</home/myuser/.ssh/id_rsa>"
338 node = add_node(ec, hostname, username, ssh_key)
339 app = add_app(ec, "ping -c3 nepi.inria.fr", node)
343 The values returned by the functions add\_node and add\_app are global
344 unique identifiers (guid) of the resources that were registered with the EC.
345 The guid is used to reference the ResourceManager associated to a registered
346 resource (for instance to retrieve results or change attribute values).
348 Now that we have registered some resources, we can ask the ExperimentController
350 Invoking the \emph{deploy} command will not only configure the
351 resource but also automatically launch the applications.
353 \begin{lstlisting}[language=Python]
359 After some seconds, we should see some output messages informing us about the
360 progress in the host deployment.
361 If you now open another terminal and you connect to the host using
362 SSH (as indicated below), you should see that a directory for your experiment
363 has been created in the host. In the remote host you will see that two NEPI
364 directories were created in the \$HOME directory: \emph{nepi-src} and \emph{nepi-exp}.
365 The first one is where NEPI will store files that might be re used by many
366 experiments (e.g. source code, input files) . The second directory \emph{nepi-exp},
367 is where experiment specific files (e.g. results, deployment scripts) will be stored.
370 \fontsize{10pt}{12pt}\selectfont
374 $ ssh -i identity username@hostname
380 Inside the \emph{nepi-exp} directory, you will find another directory with
381 the <exp-id> assigned to your EC, and inside that directory you should find
382 one directory named node-1 which will contain the files (e.g. result traces)
383 associated to the LinuxNode reosurce you just deployed.
384 In fact for every resource deployed associated to that host (e.g. each
385 LinuxApplication), NEPI will create a directory to place files related to it.
386 The name of the directory identifies the type of resources (e.g. 'node',
387 'app', etc) and it is followed by the global unique identifier (guid).
389 We can see if a resource finished deploying by querying its state through the EC
391 \begin{lstlisting}[language=Python]
393 ec.state(app, hr=True)
397 Once a \textit{LinuxApplication} has reached the state 'STARTED',
398 we can retrieve the 'stdout' trace, which should contain the output
401 \begin{lstlisting}[language=Python]
403 ec.trace(app, "stdout")
407 That is it. We can terminate the experiment by invoking the method \emph{shutdown}.
409 \begin{lstlisting}[language=Python]
415 \subsection{Define a workflow}
417 Now that we have introduced to the basics of NEPI, we will register
418 two more applications and define a workflow where one application
419 will start after the other one has finished executing.
420 For this we will use the EC \textit{register\_condition} method described below:
422 \begin{lstlisting}[language=Python]
424 register_condition(self, guids1, action, guids2, state, time=None):
425 Registers an action START, STOP or DEPLOY for all RM on list
426 guids1 to occur at time 'time' after all elements in list guids2
427 have reached state 'state'.
429 :param guids1: List of guids of RMs subjected to action
432 :param action: Action to perform (either START, STOP or DEPLOY)
433 :type action: ResourceAction
435 :param guids2: List of guids of RMs to we waited for
438 :param state: State to wait for on RMs of list guids2 (STARTED,
440 :type state: ResourceState
442 :param time: Time to wait after guids2 has reached status
447 To use the \textit{register\_condition} method we will need to import the
448 ResourceState and the ResourceAction classes
450 \begin{lstlisting}[language=Python]
452 from nepi.execution.resource import ResourceState, ResourceAction
456 Then, we register the two applications. The first application will
457 wait for 5 seconds and the create a file in the host called "greetings"
458 with the content "HELLO WORLD".
459 The second application will read the content of the file and output it
460 to standard output. If the file doesn't exist il will instead output the
463 \begin{lstlisting}[language=Python]
465 app1 = add_app(ec, "sleep 5; echo 'HELLO WORLD!' > ~/greetings", node)
466 app2 = add_app(ec, "cat ~/greetings || echo 'FAILED'", node)
470 In order to guarantee that the second application is successful, we need to
471 make sure that the first application is executed first. For this we register
474 \begin{lstlisting}[language=Python]
476 ec.register_condition (app2, ResourceAction.START, app1, ResourceState.STOPPED)
480 We then deploy the two application:
482 \begin{lstlisting}[language=Python]
484 ec.deploy(guids=[app1,app2])
488 Finally, we retrieve the standard output of the second application,
489 which should return the string "HELLO WORLD!".
491 \begin{lstlisting}[language=Python]
493 ec.trace(app2, "stdout")