1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
3 % NEPI, a framework to manage network experiments
4 % Copyright (C) 2013 INRIA
6 % This program is free software: you can redistribute it and/or modify
7 % it under the terms of the GNU General Public License as published by
8 % the Free Software Foundation, either version 3 of the License, or
9 % (at your option) any later version.
11 % This program is distributed in the hope that it will be useful,
12 % but WITHOUT ANY WARRANTY; without even the implied warranty of
13 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 % GNU General Public License for more details.
16 % You should have received a copy of the GNU General Public License
17 % along with this program. If not, see <http://www.gnu.org/licenses/>.
19 % Author: Alina Quereilhac <alina.quereilhac@inria.fr>
21 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
23 NEPI is written in Python, so you will need to install Python before
24 being able to run experiments with NEPI.
25 NEPI is known to work on Linux (Fedore, Debian, Ubuntu) and Mac (OS X).
27 \section{Dependencies}
29 Dependencies for NEPI vary according to the features you want to enable.
30 Make sure the following dependencies are correctly installed in your system
33 Mandatory dependencies:
39 \item python-pygraphviz
40 \item python-matplotlib
43 Optional dependencies:
45 \item SleekXMPP - Required to run experiments on OMF testbeds
48 \subsection{Install dependencies on Debian/Ubuntu}
51 \fontsize{10pt}{12pt}\selectfont
54 $ sudo apt-get -y install python mercurial python-ipaddr python-networkx python-pygraphviz python-matplotlib
59 \subsection{Install dependencies on Fedora}
62 \fontsize{10pt}{12pt}\selectfont
65 $ sudo yum -y install python mercurial python-ipaddr python-networkx graphviz-python python-matplotlib
70 \subsection{Install dependencies on Mac}
72 First install homebrew (\url{http://mxcl.github.io/homebrew/}),
73 then you can install Python and the rest of the dependencies as follows:
76 \fontsize{10pt}{12pt}\selectfont
80 $ sudo port install mercurial
81 $ sudo easy_install pip
82 $ sudo pip install ipaddr
83 $ sudo pip install networkx
84 $ sudo pip install pygraphviz
85 $ sudo pip install matplotlib
90 To use Python you will need to set the PATH environmental variable as:
93 \fontsize{10pt}{12pt}\selectfont
96 $ export PATH=$PATH:/usr/local/share/python
101 \subsection{Install SleekXMPP}
103 You will need \textit{git} to get the SleekXMPP sources.
106 \fontsize{10pt}{12pt}\selectfont
109 $ git clone -b develop git://github.com/fritzy/SleekXMPP.git
111 $ sudo python setup.py install
116 \section{The source code}
118 To get NEPI's source code you will need Mercurial version
119 control system. The Mercurial NEPI repo can also be browsed online at: \\
121 \url{http://nepi.inria.fr/code/nepi/}
123 \subsection{Clone the repo}
126 \fontsize{10pt}{12pt}\selectfont
129 $ hg clone http://nepi.inria.fr/code/nepi -r nepi-3-dev
134 \section{Install NEPI in your system}
136 You don't need to install NEPI in your system to be able to run
137 experiments. However this might be convenient if you don't
138 plan to modify or extend the sources.
140 To install NEPI, just run \emph{make install} in the NEPI source
144 \fontsize{10pt}{12pt}\selectfont
153 If you are developing your own NEPI extensions, the installed
154 NEPI version might interfere with your work.
155 In this case it is probably more convenient to tell
156 Python where to find the NEPI sources, using the PYTHONPATH
157 environmental variable, when you run a NEPI script.
160 \fontsize{10pt}{12pt}\selectfont
163 $ export PYTHONPATH=$PYTHONPATH:<path-to-nepi>/src
168 \section{Run experiments}
170 There are two ways you can use NEPI to run your experiments.
171 The first one is writing a Python script, which will import
172 NEPI libraries, and run it.
173 The second one is in interactive mode by using Python console.
175 \subsection{Run from script}
177 Writing a simple NEPI expeiment script is easy.
178 Take a look at the example in the FAQ section \ref{faq:ping_example}.
179 Once you have written down the script, you can run it using
180 Python. If NEPI is not installed in your system,
181 you will need to export the path to NEPI's source code to
182 the PYTHONPATH environment variable, so that Python can find
186 \fontsize{10pt}{12pt}\selectfont
189 $ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
191 $ python examples/linux/ping.py -a localhost
196 \subsection{Run NEPI interactively}
198 The IPython console can be used as an interactive interpreter to
199 execute Python instructions. We can take advantage of this feature,
200 to interactively run NEPI experiments.
201 We will use the IPython console for the example below.
203 You can easily install IPython on Debian, Ubuntu, Fedora or Mac as follows:\\
205 \textbf{Debian/Ubuntu}
208 \fontsize{10pt}{12pt}\selectfont
211 $ sudo apt-get install ipython
219 \fontsize{10pt}{12pt}\selectfont
222 $ sudo yum install ipython
230 \fontsize{10pt}{12pt}\selectfont
233 $ pip install ipython
238 Before starting, make sure to add Python and IPython source directory
239 path to the PYTHONPATH environment variable
242 \fontsize{10pt}{12pt}\selectfont
245 $ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python:/usr/local/share/python/ipython
250 Then you can start IPython as follows:
253 $ export PYTHONPATH=<path-to-nepi>/src:$PYTHONPATH
255 Python 2.7.3 (default, Jan 2 2013, 13:56:14)
256 Type "copyright", "credits" or "license" for more information.
258 IPython 0.13.1 -- An enhanced Interactive Python.
259 ? -> Introduction and overview of IPython's features.
260 %quickref -> Quick reference.
261 help -> Python's own help system.
262 object? -> Details about 'object', use 'object??' for extra details.
266 If you want to paste many lines at once in IPython, you will need
267 to type \emph{\%cpaste} and finish the paste block with \emph{\-\-}.
269 The first thing we need to do to describe an experiment with NEPI
270 is to import the NEPI Python modules.
271 In particular we need to import the ExperimentController class.
272 To do this type the following in the Python console:
274 \begin{lstlisting}[language=Python]
276 from nepi.execution.ec import ExperimentController
280 After importing the ExperimentController class, it is possible to
281 create a new instance of an the ExperimentController (EC) for
283 The <exp-id> argument is the name you want to give the experiment
284 to identify it and distinguish it from other experiments.
286 \begin{lstlisting}[language=Python]
288 ec = ExperimentController(exp_id = "<your-exp-id>")
292 Next we will define two Python functions: \emph{add\_node} and \emph{add\_app}.
293 The first one to register \textit{LinuxNodes} resources and the second one to
294 register LinuxApplications resources.
296 \begin{lstlisting}[language=Python]
299 def add_node(ec, hostname, username, ssh_key):
300 node = ec.register_resource("LinuxNode")
301 ec.set(node, "hostname", hostname)
302 ec.set(node, "username", username)
303 ec.set(node, "identity", ssh_key)
304 ec.set(node, "cleanHome", True)
305 ec.set(node, "cleanProcesses", True)
308 def add_app(ec, command, node):
309 app = ec.register_resource("LinuxApplication")
310 ec.set(app, "command", command)
311 ec.register_connection(app, node)
317 The method \textit{register\_resource} registers a resource instance with the
318 ExperimentController. The method \textit{register\_connection} indicates
319 that two resources will interact during the experiment.
320 Note that invoking \textit{add\_node} or \textit{add\_app} has no effect other
321 than informing the EC about the resources that will be used during the experiment.
322 The actual deployment of the experiment requires the method \textit{deploy} to
325 The \textit{LinuxNode} resource exposes the hostname, username and identity
326 attributes. This attributes provide information about the SSH credentials
327 needed to log in to the Linux host.
328 The \textit{hostname} is the one that identifies the physical host you want
329 to access during the experiment. The \textit{username} must correspond to a
330 valid account on that host, and the \textit{identity} attribute is the
331 'absolute' path to the SSH private key in your local computer that allows you
332 to log in to the host.
334 The \textit{command} attribute of the \textit{LinuxApplication} resource
335 expects a BASH command line string to be executed in the remote host.
336 Apart from the \emph{command} attribute, the \emph{LinuxApplication}
337 resource exposes several other attributes that allow to upload,
338 compile and install arbitrary sources.
339 The add\_app function registers a connection between a \textit{LinuxNode} and a
340 \textit{LinuxApplication}.
342 Lets now use these functions to describe a simple experiment.
343 Choose a host where you have an account, and can access using SSH
346 \begin{lstlisting}[language=Python]
348 hostname = "<the-hostname>"
349 username = "<my-username>"
350 identity = "</home/myuser/.ssh/id_rsa>"
352 node = add_node(ec, hostname, username, ssh_key)
353 app = add_app(ec, "ping -c3 nepi.inria.fr", node)
357 The values returned by the functions add\_node and add\_app are global
358 unique identifiers (guid) of the resources that were registered with the EC.
359 The guid is used to reference the ResourceManager associated to a registered
360 resource (for instance to retrieve results or change attribute values).
362 Now that we have registered some resources, we can ask the ExperimentController
364 Invoking the \emph{deploy} command will not only configure the
365 resource but also automatically launch the applications.
367 \begin{lstlisting}[language=Python]
373 After some seconds, we should see some output messages informing us about the
374 progress in the host deployment.
375 If you now open another terminal and you connect to the host using
376 SSH (as indicated below), you should see that a directory for your experiment
377 has been created in the host. In the remote host you will see that two NEPI
378 directories were created in the \$HOME directory: \emph{nepi-src} and \emph{nepi-exp}.
379 The first one is where NEPI will store files that might be re used by many
380 experiments (e.g. source code, input files) . The second directory \emph{nepi-exp},
381 is where experiment specific files (e.g. results, deployment scripts) will be stored.
384 \fontsize{10pt}{12pt}\selectfont
388 $ ssh -i identity username@hostname
394 Inside the \emph{nepi-exp} directory, you will find another directory with
395 the <exp-id> assigned to your EC, and inside that directory you should find
396 one directory named node-1 which will contain the files (e.g. result traces)
397 associated to the LinuxNode reosurce you just deployed.
398 In fact for every resource deployed associated to that host (e.g. each
399 LinuxApplication), NEPI will create a directory to place files related to it.
400 The name of the directory identifies the type of resources (e.g. 'node',
401 'app', etc) and it is followed by the global unique identifier (guid).
403 We can see if a resource finished deploying by querying its state through the EC
405 \begin{lstlisting}[language=Python]
407 ec.state(app, hr=True)
411 Once a \textit{LinuxApplication} has reached the state 'STARTED',
412 we can retrieve the 'stdout' trace, which should contain the output
415 \begin{lstlisting}[language=Python]
417 ec.trace(app, "stdout")
421 That is it. We can terminate the experiment by invoking the method \emph{shutdown}.
423 \begin{lstlisting}[language=Python]
429 \subsection{Define a workflow}
431 Now that we have introduced to the basics of NEPI, we will register
432 two more applications and define a workflow where one application
433 will start after the other one has finished executing.
434 For this we will use the EC \textit{register\_condition} method described below:
436 \begin{lstlisting}[language=Python]
438 register_condition(self, guids1, action, guids2, state, time=None):
439 Registers an action START, STOP or DEPLOY for all RM on list
440 guids1 to occur at time 'time' after all elements in list guids2
441 have reached state 'state'.
443 :param guids1: List of guids of RMs subjected to action
446 :param action: Action to perform (either START, STOP or DEPLOY)
447 :type action: ResourceAction
449 :param guids2: List of guids of RMs to we waited for
452 :param state: State to wait for on RMs of list guids2 (STARTED,
454 :type state: ResourceState
456 :param time: Time to wait after guids2 has reached status
461 To use the \textit{register\_condition} method we will need to import the
462 ResourceState and the ResourceAction classes
464 \begin{lstlisting}[language=Python]
466 from nepi.execution.resource import ResourceState, ResourceAction
470 Then, we register the two applications. The first application will
471 wait for 5 seconds and the create a file in the host called "greetings"
472 with the content "HELLO WORLD".
473 The second application will read the content of the file and output it
474 to standard output. If the file doesn't exist il will instead output the
477 \begin{lstlisting}[language=Python]
479 app1 = add_app(ec, "sleep 5; echo 'HELLO WORLD!' > ~/greetings", node)
480 app2 = add_app(ec, "cat ~/greetings || echo 'FAILED'", node)
484 In order to guarantee that the second application is successful, we need to
485 make sure that the first application is executed first. For this we register
488 \begin{lstlisting}[language=Python]
490 ec.register_condition (app2, ResourceAction.START, app1, ResourceState.STOPPED)
494 We then deploy the two application:
496 \begin{lstlisting}[language=Python]
498 ec.deploy(guids=[app1,app2])
502 Finally, we retrieve the standard output of the second application,
503 which should return the string "HELLO WORLD!".
505 \begin{lstlisting}[language=Python]
507 ec.trace(app2, "stdout")