Tony Mack [Fri, 1 Dec 2006 16:37:15 +0000 (16:37 +0000)]
- modified insert_new_item to support multiple value insert in single sql statement
Thierry Parmentelat [Fri, 1 Dec 2006 14:52:10 +0000 (14:52 +0000)]
oops-need to revert last move
Thierry Parmentelat [Fri, 1 Dec 2006 14:32:39 +0000 (14:32 +0000)]
cosmetic
Thierry Parmentelat [Fri, 1 Dec 2006 13:16:54 +0000 (13:16 +0000)]
returns ellapsed time
Thierry Parmentelat [Fri, 1 Dec 2006 08:22:59 +0000 (08:22 +0000)]
defines persons_per_slice to avoid quadratic complexity - can run with DB dumps
Tony Mack [Thu, 30 Nov 2006 22:13:56 +0000 (22:13 +0000)]
- populate sites.max_slices correctly
Thierry Parmentelat [Thu, 30 Nov 2006 10:55:32 +0000 (10:55 +0000)]
foreign_xrefs should be a list and not a dict anymore
Thierry Parmentelat [Thu, 30 Nov 2006 10:53:01 +0000 (10:53 +0000)]
GetPeerData returns separate items for local and peer data
Thierry Parmentelat [Thu, 30 Nov 2006 10:23:25 +0000 (10:23 +0000)]
new naming scheme for outputs and reference outputs
Thierry Parmentelat [Thu, 30 Nov 2006 10:14:06 +0000 (10:14 +0000)]
new limit on nodes_per_slice to avoid quadratic total time on big tests
Thierry Parmentelat [Thu, 30 Nov 2006 10:12:01 +0000 (10:12 +0000)]
(*) implements validate_ methods for all timestamp objects
(*) now allows multiple sync() calls on the same object by
reverting human-readable form to unix timestamp if needed
(*) removes extra reload formerly needed in Cache
Tony Mack [Wed, 29 Nov 2006 22:14:32 +0000 (22:14 +0000)]
- order queried events by event_id
Tony Mack [Wed, 29 Nov 2006 19:46:58 +0000 (19:46 +0000)]
- added support for wildcards in dictionary filters
Mark Huang [Wed, 29 Nov 2006 19:43:17 +0000 (19:43 +0000)]
- only add optional arguments if specified
Tony Mack [Wed, 29 Nov 2006 19:05:37 +0000 (19:05 +0000)]
- log affected object_ids
Tony Mack [Wed, 29 Nov 2006 18:30:10 +0000 (18:30 +0000)]
- - removed anything having to do with event_type/event_object
Tony Mack [Wed, 29 Nov 2006 17:57:27 +0000 (17:57 +0000)]
- removed anything having to with event_type/event_object
Mark Huang [Wed, 29 Nov 2006 03:44:36 +0000 (03:44 +0000)]
- fix probable checkin of debug code
Tony Mack [Tue, 28 Nov 2006 22:02:33 +0000 (22:02 +0000)]
- added logging vars
Tony Mack [Tue, 28 Nov 2006 22:00:14 +0000 (22:00 +0000)]
- insert Sliver into object_types table
Tony Mack [Tue, 28 Nov 2006 21:48:08 +0000 (21:48 +0000)]
- insert peers to object_types table
Mark Huang [Tue, 28 Nov 2006 21:34:48 +0000 (21:34 +0000)]
- fix check for foreign node
- fix case when multiple slice attributes of the same type have been set
(e.g. proper_op)
Thierry Parmentelat [Tue, 28 Nov 2006 16:23:56 +0000 (16:23 +0000)]
creates a fixed number of keys, huge test would issue 200x200 keys through xmlrpc otherwise
Thierry Parmentelat [Tue, 28 Nov 2006 16:23:45 +0000 (16:23 +0000)]
updated with dump
Thierry Parmentelat [Tue, 28 Nov 2006 14:55:00 +0000 (14:55 +0000)]
(*) direct cross refs redefined as NOT NULL in the database
(e.g. Node:site_id, Person:site_id, Slice:creator_person_id)
NOTE. I screwed up, this was already in former commit of planetlab4.sql,
(*) cross-ref mechanism reviewed
could not handle several xrefs with the same type in the same table
(*) undid ad-hoc trick for selecting local/foreign nodes (was named scope)
had no added-value and was only used in TestPeers anyway
Thierry Parmentelat [Tue, 28 Nov 2006 10:25:03 +0000 (10:25 +0000)]
adopted as reference
Tony Mack [Tue, 28 Nov 2006 01:02:15 +0000 (01:02 +0000)]
- fixed how node['slice_ids'] are filterd
Tony Mack [Mon, 27 Nov 2006 18:33:13 +0000 (18:33 +0000)]
- use all_slice_ids when checking slice_ids in node['slice_ids']
Thierry Parmentelat [Mon, 27 Nov 2006 16:43:32 +0000 (16:43 +0000)]
(*) full support for database caching, including SliceAttributes
(*) looks OK but further tests are required
(*) system slices to be checked with latest implementation from Mark
(I haven't got anything realting to pl_conf and such in GetSlivers
since Mark made that change)
(*) TestPeers to dump both status verbatim
Tony Mack [Mon, 27 Nov 2006 16:09:56 +0000 (16:09 +0000)]
- remove invalid (i.e. expired) slice_ids from from node['slice_ids']
Thierry Parmentelat [Mon, 27 Nov 2006 12:28:08 +0000 (12:28 +0000)]
ooops
Thierry Parmentelat [Mon, 27 Nov 2006 12:18:13 +0000 (12:18 +0000)]
(*) slice atttribute types get cached
(*) TestPeers to create custom slice attribute types and slice attributes
(*) new method GetPeerData to allow RefreshPeer to perform
a single xmlrpc request (overall improvement : x 3 on small deployments)
(*) Cache created with a peer_id rather than with a peer,
so as to remove a useless dependency (arg check done by RefreshPeer)
(*) caching slice attributes will require another method,
equality cannot be based on a single column as for other types
of cached data; ongoing
Thierry Parmentelat [Mon, 27 Nov 2006 12:13:16 +0000 (12:13 +0000)]
removes time-dependent data before diffing out and ref
Thierry Parmentelat [Mon, 27 Nov 2006 08:23:48 +0000 (08:23 +0000)]
more precise timing
Thierry Parmentelat [Sat, 25 Nov 2006 09:41:14 +0000 (09:41 +0000)]
more detailed info passed when raising an exception
Thierry Parmentelat [Sat, 25 Nov 2006 09:38:34 +0000 (09:38 +0000)]
cleanup
Thierry Parmentelat [Sat, 25 Nov 2006 09:35:36 +0000 (09:35 +0000)]
caching sites, connected to persons, and nodes
Cache had a bug in handling simple (belongs_to-like) xrefs (e.g. site_id in persons)
Thierry Parmentelat [Sat, 25 Nov 2006 09:33:53 +0000 (09:33 +0000)]
new targets
Thierry Parmentelat [Sat, 25 Nov 2006 09:33:25 +0000 (09:33 +0000)]
more consistent slice-naming scheme
Thierry Parmentelat [Fri, 24 Nov 2006 20:18:40 +0000 (20:18 +0000)]
new ref output for TestPeers.py
Thierry Parmentelat [Fri, 24 Nov 2006 20:17:49 +0000 (20:17 +0000)]
create multiple sites, attached to persons and slices
Thierry Parmentelat [Fri, 24 Nov 2006 14:25:25 +0000 (14:25 +0000)]
persons get connected to slices. GetSlivers reports the right keys
the test framework provides timing information for evaluating performance
also the number of xmlrpc requests is kept to 4, for keys, persons, nodes and slices
Thierry Parmentelat [Fri, 24 Nov 2006 12:06:00 +0000 (12:06 +0000)]
Persons gets cached and connected to Keys (not tested until GetSlivers yet)
reference output of TestPeers under cvs
yet todo:
check GetSlivers
Optimize the set of calls each peer sends other peers, the current
scheme tends to be slow because of too many small requests being issued
consider writing a dedicated Method at some point
Thierry Parmentelat [Fri, 24 Nov 2006 11:57:15 +0000 (11:57 +0000)]
frun to run fast (1x1x1x1) test
Thierry Parmentelat [Thu, 23 Nov 2006 19:43:34 +0000 (19:43 +0000)]
add system slices on local hosts only -
though I'm not sure I understand what sense it makes for a plc
to answer GetSlivers for a non-local node
Thierry Parmentelat [Thu, 23 Nov 2006 19:36:07 +0000 (19:36 +0000)]
testing keys
Thierry Parmentelat [Thu, 23 Nov 2006 19:35:38 +0000 (19:35 +0000)]
cached Keys
the class_id field in cached classes is deprecated, it was redundant with primary_key,
Thierry Parmentelat [Thu, 23 Nov 2006 15:25:16 +0000 (15:25 +0000)]
accessories objects (transcoder and xreftable) instantiated once per xref_class
Thierry Parmentelat [Thu, 23 Nov 2006 11:55:24 +0000 (11:55 +0000)]
this version uses the first release of Cache.py
Thierry Parmentelat [Thu, 23 Nov 2006 11:51:13 +0000 (11:51 +0000)]
Generic cache management algorithm
(*) Transcoder class implements translation between alien ids (ids referring to another peer's DB) into the local DB
(*) XrefTable class implements DB access to bilateral n-to-n relationsships like slice_node
(*) Cache basically gets object lists from the peer, and calls update_table sequentially
A lot of stuff was removed from Peer and RefreshPeer, as well as Slices and was rewritten here
The classes subject to this now need to define extra attributes:
(*) class_id : e.g. 'node_id'
(*) class_key : e.g. 'hostname'
(*) foreign_fields : the list of stuff we copy verbatim from foreign objects
(*) foreign_xrefs : specify what cross-references should be solved and how, see Slices for an example
yet to do:
(*) improve intermediate objects allocation (transcoders and xreftables are created on the fly)
(*) use for importing keys, persons and slice attributes.
Thierry Parmentelat [Thu, 23 Nov 2006 11:37:40 +0000 (11:37 +0000)]
miscell
Tony Mack [Wed, 22 Nov 2006 19:51:01 +0000 (19:51 +0000)]
- fix print statements
Tony Mack [Tue, 21 Nov 2006 20:07:06 +0000 (20:07 +0000)]
- comments
Tony Mack [Tue, 21 Nov 2006 20:01:25 +0000 (20:01 +0000)]
- fix db archive
Mark Huang [Tue, 21 Nov 2006 20:00:53 +0000 (20:00 +0000)]
- cast default values correctly
Tony Mack [Tue, 21 Nov 2006 16:52:51 +0000 (16:52 +0000)]
- fix db encoding check
Tony Mack [Tue, 21 Nov 2006 15:25:30 +0000 (15:25 +0000)]
- support joins and wheres in config file
Thierry Parmentelat [Tue, 21 Nov 2006 10:59:38 +0000 (10:59 +0000)]
regenerate
Thierry Parmentelat [Tue, 21 Nov 2006 10:57:00 +0000 (10:57 +0000)]
iteration 4 & last:
- ForeignNode(s) and ForeignSlice(s) deprecated
- Node and Slice both have a peer_id field that can be used
for testing locality (peer_is is None)
- associations peer x node and peer x slice not managed in
a separate table anymore (were peer_node and peer_slice)
peer_id is defined as a native column instead
- local nodes can be retrieved with
GetNodes (auth, {'peer_id':None})
GetNodes (auth, {'hostname':[name1,name2], 'peer_id':None})
or
GetNodes (auth, None, None, 'local')
GetNodes (auth, ['name1','name2'] , None, 'local')
- foreign nodes can be retrieved similarly with
GetNodes (auth, {'hostname':[name1,name2], '~peer_id':None})
or
GetNodes (auth, ['name1','name2'] , None, 'foreign')
- the 4th argument to GetNodes is named 'scope'.
It is only experimental by now, Slices do not have it as of now,
will decide later whether we provide it for all cached entities
- WARNING :
Having Nodes and Slices return ALL entities might have impacts on
other methods; most of which making sense on LOCAL entities only.
Might need a review.
Thierry Parmentelat [Tue, 21 Nov 2006 10:22:05 +0000 (10:22 +0000)]
support for negation with fields starting with ~
Thierry Parmentelat [Tue, 21 Nov 2006 10:06:40 +0000 (10:06 +0000)]
updated
Tony Mack [Mon, 20 Nov 2006 16:17:26 +0000 (16:17 +0000)]
- changes to addresses, site_address, address_address_type table config (person_addresses with type 10001 now considered site_addresses)
Thierry Parmentelat [Mon, 20 Nov 2006 10:06:36 +0000 (10:06 +0000)]
reverting deleted->is_deleted
Tony Mack [Fri, 17 Nov 2006 19:48:35 +0000 (19:48 +0000)]
- update sequences values after repopulating tables
Tony Mack [Fri, 17 Nov 2006 19:32:24 +0000 (19:32 +0000)]
- updated to reflect recent db schema changes
Tony Mack [Fri, 17 Nov 2006 19:31:08 +0000 (19:31 +0000)]
- change all occurrences of slices.deleted to slices.is_deleted
Thierry Parmentelat [Fri, 17 Nov 2006 11:55:28 +0000 (11:55 +0000)]
adapted to cope with a myplc that doesnt know about 'system' slices
Thierry Parmentelat [Fri, 17 Nov 2006 11:15:30 +0000 (11:15 +0000)]
updated
Thierry Parmentelat [Fri, 17 Nov 2006 10:43:17 +0000 (10:43 +0000)]
added in view site_slices
WHERE deleted is false
so that deleted slices dont get reported in Site,
and more importantly dont get counted when checking for max_slices at slice creation
Thierry Parmentelat [Fri, 17 Nov 2006 10:01:00 +0000 (10:01 +0000)]
provides tests for slice deletion
scales to n nodes and s slices (currently 5 and 3)
kind of sslloooowww due to numerous xmlrpc requests
could use a generic mapper function in the API for optimizing that
Thierry Parmentelat [Fri, 17 Nov 2006 09:57:28 +0000 (09:57 +0000)]
oops, probably broke the build - my mistake
this change was need togother with another one I made yesterday on
AddSliceToNodes.py DeleteSliceFromNodes.py
remove ugly hack - just assert node isinstance of Node OR ForeignNode
Mark Huang [Thu, 16 Nov 2006 18:44:27 +0000 (18:44 +0000)]
- consider slices with slice attribute system=1 to be system slices
- don't add the same sliver twice
Mark Huang [Thu, 16 Nov 2006 17:03:36 +0000 (17:03 +0000)]
add subject field to messages table
Thierry Parmentelat [Thu, 16 Nov 2006 11:54:58 +0000 (11:54 +0000)]
reviewed for (more) idempotence - parts are more independant
Thierry Parmentelat [Thu, 16 Nov 2006 11:53:57 +0000 (11:53 +0000)]
remove ugly hack - just assert node isinstance of Node OR ForeignNode
Thierry Parmentelat [Thu, 16 Nov 2006 10:20:32 +0000 (10:20 +0000)]
checks for node removal in slices
Thierry Parmentelat [Thu, 16 Nov 2006 10:19:26 +0000 (10:19 +0000)]
DeleteSliceFromNodes needs the same cooking as AddSliceToNodes
this is not the right way to do it but it works for now
Mark Huang [Wed, 15 Nov 2006 22:18:17 +0000 (22:18 +0000)]
- don't quit if no message template exists
Tony Mack [Wed, 15 Nov 2006 21:36:04 +0000 (21:36 +0000)]
- define/modify logging variables
Tony Mack [Wed, 15 Nov 2006 21:34:48 +0000 (21:34 +0000)]
- add SliceInstantiation as an object_type in object_types table
Tony Mack [Wed, 15 Nov 2006 21:07:30 +0000 (21:07 +0000)]
- add message as an object_type in object_types table
Tony Mack [Wed, 15 Nov 2006 20:19:55 +0000 (20:19 +0000)]
- dump the db as postgres instead of pgsqluser
Thierry Parmentelat [Wed, 15 Nov 2006 17:25:06 +0000 (17:25 +0000)]
too numerous changes to detail
can run in shell mode from empty databases
still need to check for slice deletions
Tony Mack [Wed, 15 Nov 2006 17:23:01 +0000 (17:23 +0000)]
- remove object_ids as class variable declaration
Thierry Parmentelat [Wed, 15 Nov 2006 16:55:28 +0000 (16:55 +0000)]
still works in unfederated context, with a bug fix:
slice_ids needs to be initialized from a copy of default_slice_ids
also starts working on a federating peer. in this case
- and right now the keys/attributes that come from another peer
- i.e. through ForiegnSlices - are filled with dummy data
- we expect GetSlivers to be called mainly from a node to its canonical(MA) plc
Tony Mack [Wed, 15 Nov 2006 16:53:07 +0000 (16:53 +0000)]
- remove some print statements
Thierry Parmentelat [Wed, 15 Nov 2006 16:50:58 +0000 (16:50 +0000)]
now returns a dict rather than a tuple
Tony Mack [Wed, 15 Nov 2006 16:46:50 +0000 (16:46 +0000)]
- keep up to date with recent changes in planetlab4.sql schema
Tony Mack [Wed, 15 Nov 2006 16:42:12 +0000 (16:42 +0000)]
- fix peer_slice_slice_id_idx
Tony Mack [Wed, 15 Nov 2006 16:04:42 +0000 (16:04 +0000)]
- fix random bugs
Tony Mack [Wed, 15 Nov 2006 15:23:27 +0000 (15:23 +0000)]
- fix bug in archive_db()
Thierry Parmentelat [Wed, 15 Nov 2006 10:59:55 +0000 (10:59 +0000)]
- provides ability to cache foreign slices
only LOCAL nodes are known to foreign slices, with their local id as expected
- RefreshPeer logic currently is
*) invoke GetNodes and cache into ForeignNodes
*) invoke ForeignNodes, not cached for now, used in next step
*) invoke GetSlices, then caches into ForeignSlices,
only local nodes are considered, with their alien node_ids transcoded
to local node_ids
- gets rid of the foreign_id columns in peer_node and peer_slice,
that turned out useless
=== a few pending issues
- GetSlivers not yet updated accordingly
it still works on non-federated plcs
- site_id and creator_person_id in the slices table
are not declared 'NOT NULL' anymore.
as far as GetSlices is concerned these entries are still non null,
as per the WHERE stuff added on view_slices
- refreshing the db to update the slice x node association is done
in a clear-all/add-all fashion that clearly sucks.
- the caching logic is currently in Peers.py which is clearly wrong
there's a need for refactoring these 2 types of cached objects
to provide more genericity
Thierry Parmentelat [Wed, 15 Nov 2006 10:18:36 +0000 (10:18 +0000)]
old commented stuff for indexing (now that we use Table.dict) really deleted
Tony Mack [Tue, 14 Nov 2006 22:08:31 +0000 (22:08 +0000)]
- reconnect to db after utf8 encoding
Tony Mack [Tue, 14 Nov 2006 21:52:00 +0000 (21:52 +0000)]
- initial checking of db upgrade scripts
Thierry Parmentelat [Tue, 14 Nov 2006 10:04:07 +0000 (10:04 +0000)]
rename column is_deleted in slices as deleted for consistency with all other tables
Thierry Parmentelat [Tue, 14 Nov 2006 09:46:48 +0000 (09:46 +0000)]
pkills instances of Shell.py prior to dropping database
Thierry Parmentelat [Tue, 14 Nov 2006 09:44:40 +0000 (09:44 +0000)]
rename column is_deleted in slices as deleted for consistency with all other tables
Thierry Parmentelat [Tue, 14 Nov 2006 09:43:28 +0000 (09:43 +0000)]
take advantage of the new Table::dict method
Mark Huang [Mon, 13 Nov 2006 18:47:34 +0000 (18:47 +0000)]
- add default system slices (those beginning with PLC_SLICE_PREFIX) to
each local node
- use Table.dict()