From: David E. Eisenstat Date: Mon, 30 Oct 2006 16:13:54 +0000 (+0000) Subject: Support for -s. More comments. Removed out of date documentation and bwcap. X-Git-Tag: planetlab-4_0-rc1~77 X-Git-Url: http://git.onelab.eu/?p=nodemanager.git;a=commitdiff_plain;h=2bea6afe154924341f57f7b8633b9ca87b53b82f Support for -s. More comments. Removed out of date documentation and bwcap. --- diff --git a/README.txt b/README.txt deleted file mode 100644 index f532a85..0000000 --- a/README.txt +++ /dev/null @@ -1,109 +0,0 @@ -THE NEW NODE MANAGER -==================== - -This is a very preliminary version of the new node manager. Currently -it is set up to download slices.xml; however, not all of the -implemented functionality is accessible via slices.xml. - -FILES -===== - -accounts.py - Account management functionality generic between -delegate accounts and VServers. - -api.py - XMLRPC interface to Node Manager functionality. Runs on port -812, supports a Help() call with more information. - -bwcap.py - Sets the bandwidth cap via the bwlimit module. The bwlimit -calls are commented out because they've been giving me a bunch of -errors. - -config.py - Configuration parameters. You'll probably want to change -SA_HOSTNAME to the PLC address. - -database.py - The dreaded NM database. The main class defined is a -dict subclass, which both indexes and stores various records. These -records include the sliver/delegate records, as well as the timestamp, -node bw cap, and any other crap PLC wants to put there. - -delegate.py - Create and delete delegate accounts. These accounts -have low space overhead (unlike a VServer) and serve to authenticate -remote NM users. - -forward_api_calls.c - The forward_api_calls program proxies stdin to -the Unix domain socket /tmp/node_mgr_api, letting Node Manager take -advantage of ssh authentication. It is intended for use as a shell on -a special delegate account. - -logger.py - This is a very basic logger. - -Makefile - For compiling forward_api_calls. - -nm.py - The main program. - -plc.py - Downloads and parses slices.xml, reads the node id file. - -README.txt - Duh. - -sliver.py - Handles all VServer functionality. - -ticket.py - Not used at the moment; contains a demonstration of -xmlsec1. - -tools.py - Various convenience functions for functionality provided by -Linux. - -RUNNING -======= - -Change SA_HOSTNAME in config.py and run nm.py. No bootstrapping -required. - -INTERNALS -========= - -At the moment, the main thread loops forever, fetching slices.xml and -updating the database. Other threads handle incoming API connections -(each connection is handled by a separate thread) and the database -dumper. There is also one thread per account, which supervises -creation/deletion/resource initialization for that account. The other -threads request operations by means of a queue. - -Other than the queues, the threads synchronize by acquiring a global -database lock before reading/writing the database. The database -itself is a collection of records, which are just Python dicts with -certain required fields. The most important of these fields are -'timestamp', 'expiry', and 'record_key'. 'record_key' serves to -uniquely identify a particular record; the only naming conventions -followed are that account records have record_key _; thus sliver princeton_sirius has record_key -'sliver_princeton_sirius'. - -The two main features that will not be familiar from the old node -manager are delegates and loans. Delegates, as described above, are -lightweight accounts whose sole purpose is to proxy NM API calls from -outside. The current code makes a delegate account 'del_snoop' that's -allowed to spy on everyone's RSpec; you'll need to change the key in -plc.py order to use it. Loans are resource transfers from one sliver -to another; the format for loans is a list of triples: recipient -sliver, resource type, amount. Thus for princeton_sirius to give 20% -guaranteed CPU to princeton_eisentest, it would call - -api.SetLoans(['princeton_eisentest', 'nm_cpu_guaranteed_share', 200]) - -provided, of course, that it has 200 guaranteed shares :) - -POSTSCRIPT -========== - -The log file will come in a great deal of use when attempting to -use/debug node manager; it lives at /var/log/pl_node_mgr.log. If you -break the DB, you should kill the pickled copy, which lives at -. - -I have been refactoring the code constantly in an attempt to keep the -amount of glue to a minimum; unfortunately comments quickly grow stale -in such an environment, and I have not yet made any attempt to comment -reasonably. Until such time as I do, I'm on the hook for limited -support of this thing. Please feel free to contact me at -deisenst@cs.princeton.edu. diff --git a/accounts.py b/accounts.py index 8363e7f..da88049 100644 --- a/accounts.py +++ b/accounts.py @@ -1,12 +1,14 @@ """Functionality common to all account classes. -Each subclass of Account must provide five methods: create(), -destroy(), configure(), start(), and stop(). In addition, it must +Each subclass of Account must provide five methods: create() and +destroy(), which are static; configure(), start(), and stop(), which +are not. configure(), which takes a record as its only argument, does +things like set up ssh keys. In addition, an Account subclass must provide static member variables SHELL, which contains the unique shell -that it uses; and TYPE, which contains a description of the type that -it uses. TYPE is divided hierarchically by periods; at the moment the -only convention is that all sliver accounts have type that begins with -sliver. +that it uses; and TYPE, a string that is used by the account creation +code. For no particular reason, TYPE is divided hierarchically by +periods; at the moment the only convention is that all sliver accounts +have type that begins with sliver. There are any number of race conditions that may result from the fact that account names are not unique over time. Moreover, it's a bad @@ -78,7 +80,7 @@ class Account: logger.log('%s: installing ssh keys' % self.name) tools.fork_as(self.name, do_installation) - def start(self): pass + def start(self, delay=0): pass def stop(self): pass @@ -115,8 +117,8 @@ class Worker: def ensure_destroyed(self): self._q.put((self._ensure_destroyed,)) def _ensure_destroyed(self): self._destroy(self._get_class()) - def start(self): self._q.put((self._start,)) - def _start(self): self._acct.start() + def start(self, delay=0): self._q.put((self._start, delay)) + def _start(self, d): self._acct.start(delay=d) def stop(self): self._q.put((self._stop,)) def _stop(self): self._acct.stop() @@ -134,6 +136,7 @@ class Worker: return shell_acct_class[shell] def _run(self): + """Repeatedly pull commands off the queue and execute. If memory usage becomes an issue, it might be wise to terminate after a while.""" while True: try: cmd = self._q.get() diff --git a/api.py b/api.py index c4dec57..0dd7115 100644 --- a/api.py +++ b/api.py @@ -1,6 +1,15 @@ +"""Sliver manager API. + +This module exposes an XMLRPC interface that allows PlanetLab users to +create/destroy slivers with delegated instantiation, start and stop +slivers, make resource loans, and examine resource allocations. The +XMLRPC is provided on a localhost-only TCP port as well as via a Unix +domain socket that is accessible by ssh-ing into a delegate account +with the forward_api_calls shell. +""" + import SimpleXMLRPCServer import SocketServer -import cPickle import errno import os import pwd @@ -16,7 +25,7 @@ import tools API_SERVER_PORT = 812 -UNIX_ADDR = '/tmp/node_mgr.api' +UNIX_ADDR = '/tmp/sliver_mgr.api' api_method_dict = {} @@ -36,13 +45,13 @@ def Help(): return ''.join([method.__doc__ + '\n' for method in api_method_dict.itervalues()]) @export_to_api(1) -def CreateSliver(rec): - """CreateSliver(sliver_name): create a non-PLC-instantiated sliver""" +def Create(rec): + """Create(sliver_name): create a non-PLC-instantiated sliver""" if rec['instantiation'] == 'delegated': accounts.get(rec['name']).ensure_created(rec) @export_to_api(1) -def DestroySliver(rec): - """DestroySliver(sliver_name): destroy a non-PLC-instantiated sliver""" +def Destroy(rec): + """Destroy(sliver_name): destroy a non-PLC-instantiated sliver""" if rec['instantiation'] == 'delegated': accounts.get(rec['name']).ensure_destroyed() @export_to_api(1) @@ -72,7 +81,7 @@ def GetLoans(rec): def validate_loans(obj): """Check that is a valid loan specification.""" - def validate_loan(obj): return (type(obj)==list or type(obj)==tuple) and len(obj)==3 and type(obj[0])==str and type(obj[1])==str and obj[1] in database.LOANABLE_RESOURCES and type(obj[2])==int and obj[2]>0 + def validate_loan(obj): return (type(obj)==list or type(obj)==tuple) and len(obj)==3 and type(obj[0])==str and type(obj[1])==str and obj[1] in database.LOANABLE_RESOURCES and type(obj[2])==int and obj[2]>=0 return type(obj)==list and False not in map(validate_loan, obj) @export_to_api(2) diff --git a/bwcap.py b/bwcap.py deleted file mode 100644 index 3d95db0..0000000 --- a/bwcap.py +++ /dev/null @@ -1,18 +0,0 @@ -import bwlimit - -import logger -import tools - - -_old_rec = {} - -def update(rec): - global _old_rec - if rec != _old_rec: - if rec['cap'] != _old_rec.get('cap'): - logger.log('setting node bw cap to %d' % rec['cap']) -# bwlimit.init('eth0', rec['cap']) - if rec['exempt_ips'] != _old_rec.get('exempt_ips'): - logger.log('initializing exempt ips to %s' % rec['exempt_ips']) -# bwlimit.exempt_init('Internet2', rec['exempt_ips']) - _old_rec = tools.deepcopy(rec) diff --git a/database.py b/database.py index c762638..d7ddf04 100644 --- a/database.py +++ b/database.py @@ -1,19 +1,32 @@ +"""The database houses information on slivers. This information +reaches the sliver manager in two different ways: one, through the +GetSlivers() call made periodically; two, by users delivering tickets. +The sync() method of the Database class turns this data into reality. + +The database format is a dictionary that maps account names to records +(also known as dictionaries). Inside each record, data supplied or +computed locally is stored under keys that begin with an underscore, +and data from PLC is stored under keys that don't. + +In order to maintain service when the node reboots during a network +partition, the database is constantly being dumped to disk. +""" + import cPickle import threading import time -try: from bwlimit import bwmin, bwmax -except ImportError: bwmin, bwmax = 8, 1000000000 import accounts import logger import tools -DB_FILE = '/root/node_mgr_db.pickle' +# We enforce minimum allocations to keep the clueless from hosing their slivers. +# Disallow disk loans because there's currently no way to punish slivers over quota. +MINIMUM_ALLOCATION = {'cpu_min': 0, 'cpu_share': 16, 'net_min': 0, 'net_max': 8, 'net2_min': 0, 'net2_max': 8, 'net_share': 1} +LOANABLE_RESOURCES = MINIMUM_ALLOCATION.keys() -LOANABLE_RESOURCES = ['cpu_min', 'cpu_share', 'net_min', 'net_max', 'net2_min', 'net2_max', 'net_share', 'disk_max'] - -DEFAULT_ALLOCATIONS = {'enabled': 1, 'cpu_min': 0, 'cpu_share': 32, 'net_min': bwmin, 'net_max': bwmax, 'net2_min': bwmin, 'net2_max': bwmax, 'net_share': 1, 'disk_max': 5000000} +DB_FILE = '/root/sliver_mgr_db.pickle' # database object and associated lock @@ -25,6 +38,7 @@ db_cond = threading.Condition(db_lock) dump_requested = False # decorator that acquires and releases the database lock before and after the decorated operation +# XXX - replace with "with" statements once we switch to 2.5 def synchronized(fn): def sync_fn(*args, **kw_args): db_lock.acquire() @@ -50,7 +64,7 @@ class Database(dict): eff_rspec = rec['_rspec'] resid_rspec = rec['rspec'].copy() for target, resname, amt in rec.get('_loans', []): - if target in slivers and amt < resid_rspec[resname]: + if target in slivers and amt <= resid_rspec[resname] - MINIMUM_ALLOCATION[resname]: eff_rspec[resname] -= amt resid_rspec[resname] -= amt slivers[target]['_rspec'][resname] += amt @@ -66,11 +80,14 @@ class Database(dict): elif rec['timestamp'] >= self._min_timestamp: self[name] = rec def set_min_timestamp(self, ts): + """The ._min_timestamp member is the timestamp on the last comprehensive update. We use it to determine if a record is stale. This method should be called whenever new GetSlivers() data comes in.""" self._min_timestamp = ts for name, rec in self.items(): if rec['timestamp'] < ts: del self[name] def sync(self): + """Synchronize reality with the database contents. This method does a lot of things, and it's currently called after every single batch of database changes (a GetSlivers(), a loan, a record). It may be necessary in the future to do something smarter.""" + # delete expired records now = time.time() for name, rec in self.items(): @@ -112,33 +129,3 @@ def start(): logger.log_exc() db = Database() tools.as_daemon_thread(run) - -@synchronized -def GetSlivers_callback(data): - for d in data: - for sliver in d['slivers']: - rec = sliver.copy() - rec.setdefault('timestamp', d['timestamp']) - rec.setdefault('type', 'sliver.VServer') - - # convert attributes field to a proper dict - attr_dict = {} - for attr in rec.pop('attributes'): attr_dict[attr['name']] = attr['value'] - - # squash keys - keys = rec.pop('keys') - rec.setdefault('keys', '\n'.join([key_struct['key'] for key_struct in keys])) - - rec.setdefault('initscript', attr_dict.get('initscript')) - rec.setdefault('delegations', []) # XXX - delegation not yet supported - - # extract the implied rspec - rspec = {} - rec['rspec'] = rspec - for resname, default_amt in DEFAULT_ALLOCATIONS.iteritems(): - try: amt = int(attr_dict[resname]) - except (KeyError, ValueError): amt = default_amt - rspec[resname] = amt - db.deliver_record(rec) - db.set_min_timestamp(d['timestamp']) - db.sync() diff --git a/forward_api_calls.c b/forward_api_calls.c index 52d7024..fa7741f 100644 --- a/forward_api_calls.c +++ b/forward_api_calls.c @@ -6,12 +6,13 @@ * Doesn't handle Unicode properly. UTF-8 is probably OK. * * Change History: + * 2006/10/30: [deisenst] Changed location of Unix socket. * 2006/09/14: [deisenst] Switched to PF_UNIX sockets so that SO_PEERCRED works * 2006/09/08: [deisenst] First version. */ static const int TIMEOUT_SECS = 30; -const char *API_addr = "/tmp/node_mgr.api"; +const char *API_addr = "/tmp/sliver_mgr.api"; static const char *Header = "POST / HTTP/1.0\r\n" diff --git a/nm.py b/nm.py index 1e5559d..3f52609 100644 --- a/nm.py +++ b/nm.py @@ -4,12 +4,8 @@ import optparse import time import xmlrpclib -import accounts -import api -import database -import delegate import logger -import sliver_vs +import sm import tools @@ -26,7 +22,7 @@ def GetSlivers(): for mod in modules: mod.GetSlivers_callback(data) def start_and_register_callback(mod): - mod.start() + mod.start(options) modules.append(mod) @@ -34,16 +30,13 @@ def run(): try: if options.daemon: tools.daemon() - accounts.register_class(sliver_vs.Sliver_VS) - accounts.register_class(delegate.Delegate) other_pid = tools.pid_file() if other_pid != None: print """There might be another instance of the node manager running as pid %d. If this is not the case, please remove the pid file %s""" % (other_pid, tools.PID_FILE) return - start_and_register_callback(database) - api.start() + start_and_register_callback(sm) while True: try: GetSlivers() except: logger.log_exc() diff --git a/sliver_vs.py b/sliver_vs.py index 1ada697..58798c5 100644 --- a/sliver_vs.py +++ b/sliver_vs.py @@ -18,6 +18,7 @@ don't have to guess if there is a running process or not. import errno import os +import time import vserver import accounts @@ -26,7 +27,7 @@ import tools class Sliver_VS(accounts.Account, vserver.VServer): - """This class wraps vserver.VServer to make its interface closer to what we need for the Node Manager.""" + """This class wraps vserver.VServer to make its interface closer to what we need.""" SHELL = '/bin/vsh' TYPE = 'sliver.VServer' @@ -65,13 +66,14 @@ class Sliver_VS(accounts.Account, vserver.VServer): accounts.Account.configure(self, rec) # install ssh keys - def start(self): + def start(self, delay=0): if self.rspec['enabled']: - logger.log('%s: starting' % self.name) + logger.log('%s: starting in %d seconds' % (self.name, delay)) child_pid = os.fork() if child_pid == 0: # VServer.start calls fork() internally, so just close the nonstandard fds and fork once to avoid creating zombies tools.close_nonstandard_fds() + time.sleep(delay) vserver.VServer.start(self, True) os._exit(0) else: os.waitpid(child_pid, 0) diff --git a/sm.py b/sm.py new file mode 100644 index 0000000..79403ca --- /dev/null +++ b/sm.py @@ -0,0 +1,71 @@ +"""Sliver manager. + +The sliver manager has several functions. It is responsible for +creating, resource limiting, starting, stopping, and destroying +slivers. It provides an API for users to access these functions and +also to make inter-sliver resource loans. The sliver manager is also +responsible for handling delegation accounts. +""" + +try: from bwlimit import bwmin, bwmax +except ImportError: bwmin, bwmax = 8, 1000*1000*1000 +import accounts +import api +import database +import delegate +import sliver_vs + + +DEFAULT_ALLOCATION = {'enabled': 1, 'cpu_min': 0, 'cpu_share': 32, 'net_min': bwmin, 'net_max': bwmax, 'net2_min': bwmin, 'net2_max': bwmax, 'net_share': 1, 'disk_max': 5000000} + +start_requested = False # set to True in order to request that all slivers be started + + +@database.synchronized +def GetSlivers_callback(data): + """This function has two purposes. One, convert GetSlivers() data into a more convenient format. Two, even if no updates are coming in, use the GetSlivers() heartbeat as a cue to scan for expired slivers.""" + for d in data: + for sliver in d['slivers']: + rec = sliver.copy() + rec.setdefault('timestamp', d['timestamp']) + rec.setdefault('type', 'sliver.VServer') + + # convert attributes field to a proper dict + attr_dict = {} + for attr in rec.pop('attributes'): attr_dict[attr['name']] = attr['value'] + + # squash keys + keys = rec.pop('keys') + rec.setdefault('keys', '\n'.join([key_struct['key'] for key_struct in keys])) + + rec.setdefault('initscript', attr_dict.get('initscript', '')) + rec.setdefault('delegations', []) # XXX - delegation not yet supported + + # extract the implied rspec + rspec = {} + rec['rspec'] = rspec + for resname, default_amt in DEFAULT_ALLOCATION.iteritems(): + try: amt = int(attr_dict[resname]) + except (KeyError, ValueError): amt = default_amt + rspec[resname] = amt + database.db.deliver_record(rec) + database.db.set_min_timestamp(d['timestamp']) + database.db.sync() + + # handle requested startup + global start_requested + if start_requested: + start_requested = False + cumulative_delay = 0 + for name in database.db.iterkeys(): + accounts.get(name).start(delay=cumulative_delay) + cumulative_delay += 3 + + +def start(options): + accounts.register_class(sliver_vs.Sliver_VS) + accounts.register_class(delegate.Delegate) + global start_requested + start_requested = options.startup + database.start() + api.start() diff --git a/tools.py b/tools.py index 51527cc..83f0327 100644 --- a/tools.py +++ b/tools.py @@ -1,3 +1,5 @@ +"""A few things that didn't seem to fit anywhere else.""" + import cPickle import errno import os