X-Git-Url: http://git.onelab.eu/?a=blobdiff_plain;f=doc%2Fuser_manual%2Fec_api.tex;h=9cc59dd8bc7ef5a2fea1dc36fb5a51ff526dfc0a;hb=018f5406c4662904b5edfbe0eeeb493a471be4d8;hp=68599dd38773e5e49e7b81abd38848a3d2b4c50e;hpb=16b00f15bf8a2364cd7a3bb7b62035064059df16;p=nepi.git

diff --git a/doc/user_manual/ec_api.tex b/doc/user_manual/ec_api.tex
index 68599dd3..9cc59dd8 100644
--- a/doc/user_manual/ec_api.tex
+++ b/doc/user_manual/ec_api.tex
@@ -21,22 +21,479 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 
-\begin{itemize}
-  \item EC API
-    \begin{itemize}
-        \item register resource
-        \item RM States (State transition DIAGRAM)
-        \item RM Actions
-        \item Attributes
-        \item The critical attribute
-        \item Traces (how to collect traces to the local repo, talk about the Collector RM)
-        \item deploy (interactive deployment)
-        \item workflow - register condition
-    \end{itemize}
-  \item Resource Factory
-    \begin{itemize}
-        \item Populate factory (happens automatically)
-        \item How to discover available rm types and their resources
-    \end{itemize}
-\end{itemize}
+The ExperimentController (EC) is the entity in charge of turning the 
+experiment description into a running experiment. 
+In order to do this the EC needs to know which resources are to be 
+used, how they should be configured and how resources relate to one another.
+To this purpose the EC exposes methods to register resources, specify their 
+configuration, and register dependencies between. These methods are part of
+the EC design API.
+Likewise, in order to deploy and control resources, and collect data, 
+the EC exposes another set of methods, which form the execution API. 
+These two APIs are described in detail in the rest of this chapter.
+
+
+\section{The experiment script}
+
+NEPI is a Python-based language and all classes and functions can
+be used by importing the \emph{nepi} module from a Python script.
+
+In particular, the ExperimentController class can be imported as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.ec import ExperimentController
+
+\end{lstlisting}
+
+Once this is done, an ExperimentController must be instantiated for
+the experiment. The ExperimentController constructor receives
+the optional argument \emph{exp\_id}. This argument is important because
+it defines the experiment identity and allows to distinguish among different
+experiments. If an experiment id is not explicitly given, NEPI will automatically
+generate a unique id for the experiment. 
+
+\begin{lstlisting}[language=Python]
+
+ec = ExperimentController(exp_id = "my-exp-id")
+
+\end{lstlisting}
+
+The experiment id can always be retrieved as follows
+
+\begin{lstlisting}[language=Python]
+
+exp_id = ec.exp_id 
+
+\end{lstlisting}
+
+Since a same experiment can be ran more than one time, and this is 
+often desirable to obtain statistical data, the EC identifies 
+different runs of an experiment with a same \emph{exp\_id} with  
+another attribute, the \emph{run\_id}. The \emph{run\_id} is
+a timestamp string value, and in combination with the \emph{exp\_id},
+it allows to uniquely identify an experiment instance.
+
+\begin{lstlisting}[language=Python]
+
+run_id = ec.run_id 
+
+\end{lstlisting}
+
+
+
+\section{The design API}
+
+Once an ExperimentController has been instantiated, it is possible to start
+describing the experiment. The design API is the set of methods which
+allow to do so.
+
+
+\subsection{Registering resources}
+
+Every resource supported by NEPI is controlled by a specific ResourceManager 
+(RM). The RM instances are automatically created by the EC, and the user does 
+not need to interact with them directly. 
+
+Each type of RM is associated with a \emph{type\_id} which uniquely identifies 
+a concrete kind of resource (e.g PlanetLab node, application that runs in
+a Linux machine, etc).
+The \emph{type\_ids} are string identifiers, and they are required  
+to register a resource with the EC.
+
+To discover all the available RMs and their \emph{type\_ids} we
+can make use of the ResourceFactory class.
+This class is a \emph{Singleton} that holds the templates and information 
+of all the RMs supported by NEPI. We can retrieve this information as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceFactory
+
+for type_id in ResourceFactory.resource_types():
+    rm_type = ResourceFactory.get_resource_type(type_id)
+    print type_id, ":", rm_type.get_help()
+
+\end{lstlisting}
+
+Once the \emph{type\_id} of the resource is known, the registration of a
+new resource with the EC is simple:
+
+\begin{lstlisting}[language=Python]
+
+type_id = "SomeRMType"
+guid = ec.register_resources(type_id)
+
+\end{lstlisting}
+
+When a resource is registered, the EC instantiates a RM of the 
+requested \emph{type\_id} and assigns a global unique identifier 
+(guid) to it. The guid is an incremental integer number and it 
+is the value returned by the \emph{register\_resource} method.
+The EC keeps internal references to all RMs, which the user can
+reference using the corresponding guid value.
+
+
+\subsection{Attributes}
+
+ResourceManagers expose the configurable parameters of resources
+through a list of attributes. An attribute can be seen as a
+\emph{{name:value}} pair, that represents a certain aspect of
+the resource (whether information or configuration information).
+
+It is possible to discover the list of attributes exposed by an 
+RM type as follows:
+
+\begin{lstlisting}[language=Python]
+from nepi.execution.resource import ResourceFactory
+
+type_id = "SomeRMType"
+rm_type = ResourceFactory.get_resource_type(type_id)
+
+for attr in rm_type.get_attributes():
+    print "       ",  attr.name, ":", attr.help
+    
+\end{lstlisting}
+
+To configure or retrieve the value of a certain attribute of
+an registered resource we can use the \emph{get} and \emph{set}
+methods of the EC.
+
+\begin{lstlisting}[language=Python]
+
+old_value = ec.get(guid, "attr_name")
+ec.set(guid, "attr_name", new_value)
+new_value = ec.get(guid, "attr_name")
+
+\end{lstlisting}
+
+Since each RM type exposes the characteristics of a particular type
+of resource, it is to be expected that different RMs will have different
+attributes. However, there a particular attribute that is common to all RMs.
+This is the \emph{critical} attribute, and it is meant to indicate to the EC
+how it should behave when a failure occurs during the experiment. 
+The \emph{critical} attribute has a default value of \emph{True}, since
+all resources are considered critical by default. 
+When this attribute is set to \emph{False} the EC will ignore failures on that 
+resource and carry on with the experiment. Otherwise, the EC will immediately 
+interrupt the experiment.
+
+
+\subsection{Traces}
+
+A Trace represent a stream of data collected during the experiment and associated
+to a single resource. ResourceManagers expose a list of traces, which are identified
+by a name. Particular traces might or might not need activation, since some traces
+are enabled by default.
+
+It is possible to discover the list of traces exposed by an 
+RM type as follows:
+
+\begin{lstlisting}[language=Python]
+from nepi.execution.resource import ResourceFactory
+
+type_id = "SomeRMType"
+rm_type = ResourceFactory.get_resource_type(type_id)
+
+for trace in rm_type.get_traces():
+    print "       ",  trace.name, ":", trace.enabled
+    
+\end{lstlisting}
+
+The \emph{enable\_trace} method allows to enable a specific trace for a 
+RM instance
+
+\begin{lstlisting}[language=Python]
+
+ec.enable_trace(guid, "trace-name")
+
+print ec.trace_enabled(guid, "trace-name")
+
+\end{lstlisting}
+
+
+\subsection{Registering connections}
+
+In order to describe the experiment set-up, a resources need to be 
+associated at least to one another. Through the process of connecting resources
+the \emph{topology graph} is constructed. A certain application might
+need to be configured and executed on a certain node, and this
+must be indicated to the EC by connecting the application RM to the node
+RM.
+
+Connections are registered using the \emph{register\_connection} method,
+which receives the guids of the two RM.
+
+\begin{lstlisting}[language=Python]
+
+ec.register_connection(node_guid, app_guid)
+
+\end{lstlisting}
+
+The order in which the guids are given is not important, since the
+\emph{topology\_graph} is not directed, and the corresponding 
+RMs \emph{`know'} internally how to interpret the connection 
+relationship.
+
+
+\subsection{Registering conditions}
+
+All ResourceMangers must go through the same sequence of state transitions.
+Associated to those states are the actions that trigger the transitions.
+As an example, a RM will initially be in the state NEW. When the DEPLOY action
+is invoked, it will transition to the DISCOVERED, then PROVISIONED, then READY
+states. Likewise, the action START will make a RM pass from state READY to 
+STARTED, and the action STOP will change a RM from state STARTED to STOPPED.
+
+Using these states and actions, it is possible to specify workflow dependencies 
+between resources. For instance, it would be possible to indicate that
+one application should start after another application by registering a 
+condition with the EC.
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceActions
+
+ec.register_condition(app1_guid, ResourceAction.START, app2_guid, ResourceState.STARTED)
+
+\end{lstlisting}
+
+The above invocation should be read "Application 1 should START after application 2 
+has STARTED". It is also possible to indicate a relative time from the moment a state
+change occurs to the moment the action should be taken as follows:
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.resource import ResourceState, ResourceActions
+
+ec.register_condition(app1_guid, ResourceAction.START, app2_guid, ResourceState.STARTED, time = "5s")
+
+\end{lstlisting}
+
+This line should be read "Application 1 should START at least 5 seconds after 
+application 2 has STARTED". \\
+
+Allowed actions are: DEPLOY, START and STOP. \\
+
+Existing states are: NEW, DISCOVERED, PROVISIONED, READY, STARTED, STOPPED, 
+FAILED and RELEASED. \\
+
+
+
+\section{The execution API}
+
+After registering all the resources and connections and setting attributes and
+traces, once the experiment we want to conduct has been described, we can
+proceed to run it. To this purpose we make use of the \emph{execution} methods
+exposed by the EC.
+
+
+\subsection{Deploying an experiment}
+
+Deploying an experiment is very easy, it only requires to invoke the 
+\emph{deploy} method of the EC.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy()
+
+\end{lstlisting}
+
+Given the experiment description provided earlier, the EC will take care
+of automatically performing all necessary actions to discover, provision,
+configure and start all resources registered in the experiment. 
+
+Furthermore, NEPI does not restrict deployment to only one time, it allows
+to continue to register, connect and configure resources and deploy them
+at any moment. We call this feature \emph{interactive} or \emph{dynamic}
+deployment. 
+
+The \emph{deploy} method can receive other optional arguments to customize
+deployment. By default, the EC will deploy all registered RMs that are in
+state NEW. However, it is possible to specify a subset of resources to be
+deployed using the \emph{guids} argument.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(guids=[guid1, guid2, guid3])
+
+\end{lstlisting}
+
+Another useful argument of the \emph{deploy} method is \emph{wait\_all\_ready}.
+This argument has a default value of \emph{True}, and it is used as a barrier
+to force the START action to be invoked on all RMs being deploy only after
+they have all reached the state READY.
+
+\begin{lstlisting}[language=Python]
+
+ec.deploy(wait_all_ready=False)
+
+\end{lstlisting}
+
+
+\subsection{Getting attributes}
+
+Attribute values can be retrieved at any moment during the experiment run, 
+using the \emph{get} method. 
+However, not all attributes can be modified after a resource has
+been deployed. The possibility of changing the value of a certain attribute 
+depends strongly on the RM and on the attribute itself. 
+As an example, once a \emph{hostname} has been specified for a certain Node 
+RM, it might not be possible to change it after deployment.
+
+\begin{lstlisting}[language=Python]
+
+attr_value = ec.get(guid, "attr-name")
+
+\end{lstlisting}
+
+Attributes have flags that indicate whether their values can be changed
+and when it is possible to change them (e.g. before or after deployment, 
+or both). These flags are \emph{NoFlags} (the attribute value can be 
+modified always), \emph{ReadOnly} (the attribute value can never be
+modified), \emph{ExecReadOnly} (the attribute value can only be modified
+before deployment). The flags of a certain attribute can be validated 
+as shown in the example below, and the value of the attribute can be
+changed using the \emph{set} method.  
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.attribute import Flags
+
+attr = ec.get_attribute(guid, "attr-name")
+
+if not attr.has_flag(Flags.ReadOnly):
+    ec.set(guid, "attr-name", attr_value)
+
+\end{lstlisting}
+
+\subsection{Quering the state}
+
+It is possible to query the state of any resource at any moment.
+The state of a resource is requested using the \emph{state} method.
+This method receives the optional parameter \emph{hr} to output the
+state in a \emph{human readable} string format instead of an integer
+state code.
+
+\begin{lstlisting}[language=Python]
+
+state_id = ec.state(guid)
+
+# Human readable state
+state = ec.state(guid, hr = True)
+
+\end{lstlisting}
+
+\subsection{Getting traces}
+
+After a ResourceManager has been deployed it is possible to get information
+about the active traces and the trace streams of the generated data using
+the \emph{trace} method.
+
+Most traces are collected to a file in the host where they are generated, 
+the total trace size and the file path in the (remote) host can be 
+retrieved as follows.
+
+\begin{lstlisting}[language=Python]
+
+from nepi.execution.trace import TraceAttr
+
+path = ec.trace(guid, "trace-name", TraceAttr.PATH)
+size = ec.trace(guid, "trace-name", TraceAttr.SIZE)
+
+\end{lstlisting}
+
+The trace content can be retrieved in a stream, block by block.
+
+\begin{lstlisting}[language=Python]
+
+trace_block = ec.trace(guid, "trace-name", TraceAttr.STREAM, block=1, offset=0)
+
+\end{lstlisting}
+
+It is also possible to directly retrieve the complete trace content.
+
+\begin{lstlisting}[language=Python]
+
+trace_stream = ec.trace(guid, "trace-name")
+
+\end{lstlisting}
+
+Using the \emph{trace} method it is easy to collect all traces 
+to the local user machine. 
+
+\begin{lstlisting}[language=Python]
+
+for trace in ec.get_traces(guid):
+    trace_stream = ec.trace(guid, "trace-name")
+    f = open("trace-name", "w")
+    f.write(trace_stream)
+    f.close()
+
+\end{lstlisting}
+
+
+% TODO: how to retrieve an application trace when the Node failed? (critical attribute)
+
+
+% \subsection{The collector RM}
+
+%%%%%%%%%%
+%% TODO
+%%%%%%%%%%%
+
+\subsection{API reference}
+
+Further information about classes and method signatures
+can be found using the Python \emph{help} method.
+For this inspection work, we recommend to instantiate an
+ExperimentController from an IPython console. This is an
+interactive console that allows to dynamically send input
+to the python interpreter. 
+
+If NEPI is not installed in the system, you will need to add the
+NEPI sources path to the PYTHONPATH environmental variable 
+before invoking \emph{ipython}.
+
+\begin{lstlisting}[language=Python]
+
+$ PYTHONPATH=$PYTHONPATH:src ipython
+Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
+Type "copyright", "credits" or "license" for more information.
+
+
+IPython 0.13.1 -- An enhanced Interactive Python.
+?         -> Introduction and overview of IPython's features.
+%quickref -> Quick reference.
+help      -> Python's own help system.
+object?   -> Details about 'object', use 'object??' for extra details.
+
+In [1]: from nepi.execution.ec import ExperimentController
+
+In [2]: ec = ExperimentController(exp_id = "test-tap")
+
+In [3]: help(ec.set)
+
+\end{lstlisting}
+
+The example above will show the following information related to the
+\emph{set} method of the EC API.
+
+\begin{lstlisting}[language=Python]
+
+Help on method set in module nepi.execution.ec:
+
+set(self, guid, name, value) method of nepi.execution.ec.ExperimentController instance
+    Modifies the value of the attribute with name 'name' on the RM with guid 'guid'.
+    
+    :param guid: Guid of the RM
+    :type guid: int
+    
+    :param name: Name of the attribute
+    :type name: str
+    
+    :param value: Value of the attribute
+
+\end{lstlisting}
+