+++ /dev/null
-THE NEW NODE MANAGER
-====================
-
-This is a very preliminary version of the new node manager. Currently
-it is set up to download slices.xml; however, not all of the
-implemented functionality is accessible via slices.xml.
-
-FILES
-=====
-
-accounts.py - Account management functionality generic between
-delegate accounts and VServers.
-
-api.py - XMLRPC interface to Node Manager functionality. Runs on port
-812, supports a Help() call with more information.
-
-bwcap.py - Sets the bandwidth cap via the bwlimit module. The bwlimit
-calls are commented out because they've been giving me a bunch of
-errors.
-
-config.py - Configuration parameters. You'll probably want to change
-SA_HOSTNAME to the PLC address.
-
-database.py - The dreaded NM database. The main class defined is a
-dict subclass, which both indexes and stores various records. These
-records include the sliver/delegate records, as well as the timestamp,
-node bw cap, and any other crap PLC wants to put there.
-
-delegate.py - Create and delete delegate accounts. These accounts
-have low space overhead (unlike a VServer) and serve to authenticate
-remote NM users.
-
-forward_api_calls.c - The forward_api_calls program proxies stdin to
-the Unix domain socket /tmp/node_mgr_api, letting Node Manager take
-advantage of ssh authentication. It is intended for use as a shell on
-a special delegate account.
-
-logger.py - This is a very basic logger.
-
-Makefile - For compiling forward_api_calls.
-
-nm.py - The main program.
-
-plc.py - Downloads and parses slices.xml, reads the node id file.
-
-README.txt - Duh.
-
-sliver.py - Handles all VServer functionality.
-
-ticket.py - Not used at the moment; contains a demonstration of
-xmlsec1.
-
-tools.py - Various convenience functions for functionality provided by
-Linux.
-
-RUNNING
-=======
-
-Change SA_HOSTNAME in config.py and run nm.py. No bootstrapping
-required.
-
-INTERNALS
-=========
-
-At the moment, the main thread loops forever, fetching slices.xml and
-updating the database. Other threads handle incoming API connections
-(each connection is handled by a separate thread) and the database
-dumper. There is also one thread per account, which supervises
-creation/deletion/resource initialization for that account. The other
-threads request operations by means of a queue.
-
-Other than the queues, the threads synchronize by acquiring a global
-database lock before reading/writing the database. The database
-itself is a collection of records, which are just Python dicts with
-certain required fields. The most important of these fields are
-'timestamp', 'expiry', and 'record_key'. 'record_key' serves to
-uniquely identify a particular record; the only naming conventions
-followed are that account records have record_key <account
-type>_<account name>; thus sliver princeton_sirius has record_key
-'sliver_princeton_sirius'.
-
-The two main features that will not be familiar from the old node
-manager are delegates and loans. Delegates, as described above, are
-lightweight accounts whose sole purpose is to proxy NM API calls from
-outside. The current code makes a delegate account 'del_snoop' that's
-allowed to spy on everyone's RSpec; you'll need to change the key in
-plc.py order to use it. Loans are resource transfers from one sliver
-to another; the format for loans is a list of triples: recipient
-sliver, resource type, amount. Thus for princeton_sirius to give 20%
-guaranteed CPU to princeton_eisentest, it would call
-
-api.SetLoans(['princeton_eisentest', 'nm_cpu_guaranteed_share', 200])
-
-provided, of course, that it has 200 guaranteed shares :)
-
-POSTSCRIPT
-==========
-
-The log file will come in a great deal of use when attempting to
-use/debug node manager; it lives at /var/log/pl_node_mgr.log. If you
-break the DB, you should kill the pickled copy, which lives at
-<config.py:DB_FILE>.
-
-I have been refactoring the code constantly in an attempt to keep the
-amount of glue to a minimum; unfortunately comments quickly grow stale
-in such an environment, and I have not yet made any attempt to comment
-reasonably. Until such time as I do, I'm on the hook for limited
-support of this thing. Please feel free to contact me at
-deisenst@cs.princeton.edu.
"""Functionality common to all account classes.
-Each subclass of Account must provide five methods: create(),
-destroy(), configure(), start(), and stop(). In addition, it must
+Each subclass of Account must provide five methods: create() and
+destroy(), which are static; configure(), start(), and stop(), which
+are not. configure(), which takes a record as its only argument, does
+things like set up ssh keys. In addition, an Account subclass must
provide static member variables SHELL, which contains the unique shell
-that it uses; and TYPE, which contains a description of the type that
-it uses. TYPE is divided hierarchically by periods; at the moment the
-only convention is that all sliver accounts have type that begins with
-sliver.
+that it uses; and TYPE, a string that is used by the account creation
+code. For no particular reason, TYPE is divided hierarchically by
+periods; at the moment the only convention is that all sliver accounts
+have type that begins with sliver.
There are any number of race conditions that may result from the fact
that account names are not unique over time. Moreover, it's a bad
logger.log('%s: installing ssh keys' % self.name)
tools.fork_as(self.name, do_installation)
- def start(self): pass
+ def start(self, delay=0): pass
def stop(self): pass
def ensure_destroyed(self): self._q.put((self._ensure_destroyed,))
def _ensure_destroyed(self): self._destroy(self._get_class())
- def start(self): self._q.put((self._start,))
- def _start(self): self._acct.start()
+ def start(self, delay=0): self._q.put((self._start, delay))
+ def _start(self, d): self._acct.start(delay=d)
def stop(self): self._q.put((self._stop,))
def _stop(self): self._acct.stop()
return shell_acct_class[shell]
def _run(self):
+ """Repeatedly pull commands off the queue and execute. If memory usage becomes an issue, it might be wise to terminate after a while."""
while True:
try:
cmd = self._q.get()
+"""Sliver manager API.
+
+This module exposes an XMLRPC interface that allows PlanetLab users to
+create/destroy slivers with delegated instantiation, start and stop
+slivers, make resource loans, and examine resource allocations. The
+XMLRPC is provided on a localhost-only TCP port as well as via a Unix
+domain socket that is accessible by ssh-ing into a delegate account
+with the forward_api_calls shell.
+"""
+
import SimpleXMLRPCServer
import SocketServer
-import cPickle
import errno
import os
import pwd
API_SERVER_PORT = 812
-UNIX_ADDR = '/tmp/node_mgr.api'
+UNIX_ADDR = '/tmp/sliver_mgr.api'
api_method_dict = {}
return ''.join([method.__doc__ + '\n' for method in api_method_dict.itervalues()])
@export_to_api(1)
-def CreateSliver(rec):
- """CreateSliver(sliver_name): create a non-PLC-instantiated sliver"""
+def Create(rec):
+ """Create(sliver_name): create a non-PLC-instantiated sliver"""
if rec['instantiation'] == 'delegated': accounts.get(rec['name']).ensure_created(rec)
@export_to_api(1)
-def DestroySliver(rec):
- """DestroySliver(sliver_name): destroy a non-PLC-instantiated sliver"""
+def Destroy(rec):
+ """Destroy(sliver_name): destroy a non-PLC-instantiated sliver"""
if rec['instantiation'] == 'delegated': accounts.get(rec['name']).ensure_destroyed()
@export_to_api(1)
def validate_loans(obj):
"""Check that <obj> is a valid loan specification."""
- def validate_loan(obj): return (type(obj)==list or type(obj)==tuple) and len(obj)==3 and type(obj[0])==str and type(obj[1])==str and obj[1] in database.LOANABLE_RESOURCES and type(obj[2])==int and obj[2]>0
+ def validate_loan(obj): return (type(obj)==list or type(obj)==tuple) and len(obj)==3 and type(obj[0])==str and type(obj[1])==str and obj[1] in database.LOANABLE_RESOURCES and type(obj[2])==int and obj[2]>=0
return type(obj)==list and False not in map(validate_loan, obj)
@export_to_api(2)
+++ /dev/null
-import bwlimit
-
-import logger
-import tools
-
-
-_old_rec = {}
-
-def update(rec):
- global _old_rec
- if rec != _old_rec:
- if rec['cap'] != _old_rec.get('cap'):
- logger.log('setting node bw cap to %d' % rec['cap'])
-# bwlimit.init('eth0', rec['cap'])
- if rec['exempt_ips'] != _old_rec.get('exempt_ips'):
- logger.log('initializing exempt ips to %s' % rec['exempt_ips'])
-# bwlimit.exempt_init('Internet2', rec['exempt_ips'])
- _old_rec = tools.deepcopy(rec)
+"""The database houses information on slivers. This information
+reaches the sliver manager in two different ways: one, through the
+GetSlivers() call made periodically; two, by users delivering tickets.
+The sync() method of the Database class turns this data into reality.
+
+The database format is a dictionary that maps account names to records
+(also known as dictionaries). Inside each record, data supplied or
+computed locally is stored under keys that begin with an underscore,
+and data from PLC is stored under keys that don't.
+
+In order to maintain service when the node reboots during a network
+partition, the database is constantly being dumped to disk.
+"""
+
import cPickle
import threading
import time
-try: from bwlimit import bwmin, bwmax
-except ImportError: bwmin, bwmax = 8, 1000000000
import accounts
import logger
import tools
-DB_FILE = '/root/node_mgr_db.pickle'
+# We enforce minimum allocations to keep the clueless from hosing their slivers.
+# Disallow disk loans because there's currently no way to punish slivers over quota.
+MINIMUM_ALLOCATION = {'cpu_min': 0, 'cpu_share': 16, 'net_min': 0, 'net_max': 8, 'net2_min': 0, 'net2_max': 8, 'net_share': 1}
+LOANABLE_RESOURCES = MINIMUM_ALLOCATION.keys()
-LOANABLE_RESOURCES = ['cpu_min', 'cpu_share', 'net_min', 'net_max', 'net2_min', 'net2_max', 'net_share', 'disk_max']
-
-DEFAULT_ALLOCATIONS = {'enabled': 1, 'cpu_min': 0, 'cpu_share': 32, 'net_min': bwmin, 'net_max': bwmax, 'net2_min': bwmin, 'net2_max': bwmax, 'net_share': 1, 'disk_max': 5000000}
+DB_FILE = '/root/sliver_mgr_db.pickle'
# database object and associated lock
dump_requested = False
# decorator that acquires and releases the database lock before and after the decorated operation
+# XXX - replace with "with" statements once we switch to 2.5
def synchronized(fn):
def sync_fn(*args, **kw_args):
db_lock.acquire()
eff_rspec = rec['_rspec']
resid_rspec = rec['rspec'].copy()
for target, resname, amt in rec.get('_loans', []):
- if target in slivers and amt < resid_rspec[resname]:
+ if target in slivers and amt <= resid_rspec[resname] - MINIMUM_ALLOCATION[resname]:
eff_rspec[resname] -= amt
resid_rspec[resname] -= amt
slivers[target]['_rspec'][resname] += amt
elif rec['timestamp'] >= self._min_timestamp: self[name] = rec
def set_min_timestamp(self, ts):
+ """The ._min_timestamp member is the timestamp on the last comprehensive update. We use it to determine if a record is stale. This method should be called whenever new GetSlivers() data comes in."""
self._min_timestamp = ts
for name, rec in self.items():
if rec['timestamp'] < ts: del self[name]
def sync(self):
+ """Synchronize reality with the database contents. This method does a lot of things, and it's currently called after every single batch of database changes (a GetSlivers(), a loan, a record). It may be necessary in the future to do something smarter."""
+
# delete expired records
now = time.time()
for name, rec in self.items():
logger.log_exc()
db = Database()
tools.as_daemon_thread(run)
-
-@synchronized
-def GetSlivers_callback(data):
- for d in data:
- for sliver in d['slivers']:
- rec = sliver.copy()
- rec.setdefault('timestamp', d['timestamp'])
- rec.setdefault('type', 'sliver.VServer')
-
- # convert attributes field to a proper dict
- attr_dict = {}
- for attr in rec.pop('attributes'): attr_dict[attr['name']] = attr['value']
-
- # squash keys
- keys = rec.pop('keys')
- rec.setdefault('keys', '\n'.join([key_struct['key'] for key_struct in keys]))
-
- rec.setdefault('initscript', attr_dict.get('initscript'))
- rec.setdefault('delegations', []) # XXX - delegation not yet supported
-
- # extract the implied rspec
- rspec = {}
- rec['rspec'] = rspec
- for resname, default_amt in DEFAULT_ALLOCATIONS.iteritems():
- try: amt = int(attr_dict[resname])
- except (KeyError, ValueError): amt = default_amt
- rspec[resname] = amt
- db.deliver_record(rec)
- db.set_min_timestamp(d['timestamp'])
- db.sync()
* Doesn't handle Unicode properly. UTF-8 is probably OK.
*
* Change History:
+ * 2006/10/30: [deisenst] Changed location of Unix socket.
* 2006/09/14: [deisenst] Switched to PF_UNIX sockets so that SO_PEERCRED works
* 2006/09/08: [deisenst] First version.
*/
static const int TIMEOUT_SECS = 30;
-const char *API_addr = "/tmp/node_mgr.api";
+const char *API_addr = "/tmp/sliver_mgr.api";
static const char *Header =
"POST / HTTP/1.0\r\n"
import time
import xmlrpclib
-import accounts
-import api
-import database
-import delegate
import logger
-import sliver_vs
+import sm
import tools
for mod in modules: mod.GetSlivers_callback(data)
def start_and_register_callback(mod):
- mod.start()
+ mod.start(options)
modules.append(mod)
try:
if options.daemon: tools.daemon()
- accounts.register_class(sliver_vs.Sliver_VS)
- accounts.register_class(delegate.Delegate)
other_pid = tools.pid_file()
if other_pid != None:
print """There might be another instance of the node manager running as pid %d. If this is not the case, please remove the pid file %s""" % (other_pid, tools.PID_FILE)
return
- start_and_register_callback(database)
- api.start()
+ start_and_register_callback(sm)
while True:
try: GetSlivers()
except: logger.log_exc()
import errno
import os
+import time
import vserver
import accounts
class Sliver_VS(accounts.Account, vserver.VServer):
- """This class wraps vserver.VServer to make its interface closer to what we need for the Node Manager."""
+ """This class wraps vserver.VServer to make its interface closer to what we need."""
SHELL = '/bin/vsh'
TYPE = 'sliver.VServer'
accounts.Account.configure(self, rec) # install ssh keys
- def start(self):
+ def start(self, delay=0):
if self.rspec['enabled']:
- logger.log('%s: starting' % self.name)
+ logger.log('%s: starting in %d seconds' % (self.name, delay))
child_pid = os.fork()
if child_pid == 0:
# VServer.start calls fork() internally, so just close the nonstandard fds and fork once to avoid creating zombies
tools.close_nonstandard_fds()
+ time.sleep(delay)
vserver.VServer.start(self, True)
os._exit(0)
else: os.waitpid(child_pid, 0)
--- /dev/null
+"""Sliver manager.
+
+The sliver manager has several functions. It is responsible for
+creating, resource limiting, starting, stopping, and destroying
+slivers. It provides an API for users to access these functions and
+also to make inter-sliver resource loans. The sliver manager is also
+responsible for handling delegation accounts.
+"""
+
+try: from bwlimit import bwmin, bwmax
+except ImportError: bwmin, bwmax = 8, 1000*1000*1000
+import accounts
+import api
+import database
+import delegate
+import sliver_vs
+
+
+DEFAULT_ALLOCATION = {'enabled': 1, 'cpu_min': 0, 'cpu_share': 32, 'net_min': bwmin, 'net_max': bwmax, 'net2_min': bwmin, 'net2_max': bwmax, 'net_share': 1, 'disk_max': 5000000}
+
+start_requested = False # set to True in order to request that all slivers be started
+
+
+@database.synchronized
+def GetSlivers_callback(data):
+ """This function has two purposes. One, convert GetSlivers() data into a more convenient format. Two, even if no updates are coming in, use the GetSlivers() heartbeat as a cue to scan for expired slivers."""
+ for d in data:
+ for sliver in d['slivers']:
+ rec = sliver.copy()
+ rec.setdefault('timestamp', d['timestamp'])
+ rec.setdefault('type', 'sliver.VServer')
+
+ # convert attributes field to a proper dict
+ attr_dict = {}
+ for attr in rec.pop('attributes'): attr_dict[attr['name']] = attr['value']
+
+ # squash keys
+ keys = rec.pop('keys')
+ rec.setdefault('keys', '\n'.join([key_struct['key'] for key_struct in keys]))
+
+ rec.setdefault('initscript', attr_dict.get('initscript', ''))
+ rec.setdefault('delegations', []) # XXX - delegation not yet supported
+
+ # extract the implied rspec
+ rspec = {}
+ rec['rspec'] = rspec
+ for resname, default_amt in DEFAULT_ALLOCATION.iteritems():
+ try: amt = int(attr_dict[resname])
+ except (KeyError, ValueError): amt = default_amt
+ rspec[resname] = amt
+ database.db.deliver_record(rec)
+ database.db.set_min_timestamp(d['timestamp'])
+ database.db.sync()
+
+ # handle requested startup
+ global start_requested
+ if start_requested:
+ start_requested = False
+ cumulative_delay = 0
+ for name in database.db.iterkeys():
+ accounts.get(name).start(delay=cumulative_delay)
+ cumulative_delay += 3
+
+
+def start(options):
+ accounts.register_class(sliver_vs.Sliver_VS)
+ accounts.register_class(delegate.Delegate)
+ global start_requested
+ start_requested = options.startup
+ database.start()
+ api.start()
+"""A few things that didn't seem to fit anywhere else."""
+
import cPickle
import errno
import os
git://git.onelab.eu / nodemanager.git / commitdiff