[distributedratelimiting.git] / drl / README

2-20-08

Use this to capture the rates from the individual slices.  

 while true; do /sbin/tc -s class show dev eth0 | grep -A 3 11f8 | grep -A
 1 "class htb" | grep Sent | cut -d " " -f 3; sleep 1; done > GRD_pl_drl_34





2-15-08

Damn sysnet33 db won't come up.  says /tmp is full in the chroot
myplc/plc/root/var/log/pglog, but it's not full.  
So I'm trying to replace the loopback filesystem with something clean from
the RPM. 
using cpio --no-absolute-filenames
And 
rpm2cpio <rpm> | cpio -diu 
but u is don't prompt. so I'll issue
rpm2cpio <rpm> | cpio -di --no-absolute_filenames

OK, did this, then I copied over root.img to where I have it installed. 
And it all worked!  Sweet.  The image is mostly independent of the data! 






ADDING A PL NODE
1.) turn off crond for restarting ulogd
in pl_netflow:/etc/crond.d/netflow
edited the line that calls init.d/netflow restart
because I don't want it to keep restarting ulogd as I test it. 
2.) Allow pl_netflow to run tc
I am also changing the pl_netflow.conf in /etc/vservers/pl_netflow.conf
So that S_CAPS="CAP_NET_ADMIN"
Then, from the root slice, vserver pl_netflow restart. 
3.) copy over the ./start script from the pl_netflow /root directory.
4.) put bwlimit_drl.py into /usr/share/util-vserver/bwlimit2.py on the root
slice
put bwlimit_drl.py into /usr/share/util-vserver/bwlimit_drl.py on the
pl_netflow slice.
Future, make the names consistent
5.) pull the sch_netem.ko over and insmod it. 

cd /lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/
[root@sysnet32 sched]# scp -i ~/.ssh/root_ssh_key.rsa root@sysnet34:/lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/sch_netem_plab.ko .
insmod sch_netem_plab.ko

6.) [root@vserver:pl_netflow ~]/etc/init.d/ulogd stop
7.) vserver pl_netflow enter
8.)     copy over ulogd.  run ./start
9.) Edit the ulogd configuration script.   


2-12-08

bwlimit.py cannot look up the slice names in the pl_netflow slice b/c there
are not entries in the /etc/passwd file for the slices.  Of course there
aren't.  Even ulogd_NETFLOW.c replaces /etc/passwd with /dev/null.   

so in the short term I will copy over the roots /etc/passwd and place it
where the new bwlimit_drl exists.   /usr/share/util-vserver/passwd_drl
I can have the root periodically copy this file into the pl_netflow slice
in a cron job, or whenever a new slice is created.   

This is a hack.  There are probably better ways, but it's a moving target.  
Now hack bwlimit_drl to use that file to look up slice contexts. 



HTB DOCS http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

2-10-08

We probably have to put the netem at the bottom of the tbf.  If we put it
in the middle, say off of the 1:10 class, then we need to direct all
traffic that would have been directed to a lower class in the htb to the
1:10 class.  Then the traffic has to be directed to the lower classes and
be accounted for directly.   This screws with how traffic classification is
done; I don't even know if it will obey the tb appropriately either, since
the best you can do is put another htb qdisc below the netem class.  And
there is no way to directly have that qdisc borrow from the "parent" one
that's just been terminated by the netem qdisc.  Got that? 






2-7-08

Screwing around with all the kernel configs has forced the build process to
rebuild all the kernels.   blech.   


BUILDING NETEM for plab machines: 
Kevin made an ubunto chroot install on his linux machine and built 3.3 gcc
in it, and then copied over the source linux tree that had the .config
stolen from sysnet34 /proc/config.gz.  We also had to change the magic
string so that it believed it was bulit to fc 4.5, not 4.7. 
Use insmod
/lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/sch_netem_plab.ko


see sysnet35:/root/myplc/build/SOURCES/CRAP/linux-2.6.12/.config


We added a node at the root qdisc and it succesfully appears to delay
packets.  So now let's figure out how to add it to the bwlimit.py code so
that it is always present and can be tweaked (replaced) when running
ulogd. 

----
tc qdisc del dev eth0 root handle 1:
tc qdisc add dev eth0 parent 1: handle 2: htb default 1fff
And everything then falls out of 2:

But this will netem everything out of hte host, leaving nothing for
"exempt" slices like the root.  Not what we want. 

We place a netem qdisc at the leaf of every non-exempt slice traffic class
in the HTB. 

2-5-08

in pl_netflow vserver
/etc/cron.d/netflow  keeps restarting netflow init.d script which keeps
restarting ulogd if started by hand.   Fucking buggy. 

in root vserver
see pl_mom cron job
which runs bwmon and pl_mop. pl_mop makes sure pl_netflow is running. 

!!!!!<CHANGE TO SYSNET34>!!!!! 
in sysnet34 pl_netflow:/etc/crond.d/netflow
edited the line that calls init.d/netflow restart
because I don't want it to keep restarting ulogd as I test it. 
!!!!!
I am also changing the pl_netflow.conf in /etc/vservers/pl_netflow.conf
So that S_CAPS="CAP_NET_ADMIN"
Then, from the root slice, vserver pl_netflow restart. 

<changing kernel config files in build/linux-x-x-x/configs directory
i686-smp-plab
i686-onelab
i686-planetlab
all added CONFIG_NET_SCH_NETEM=m


2-1-08

GRD dropping.  Use netem, which is accessible also through tc.   
I believe what we want to do is put the netem on the interface for all
traffic?  OK, that's not going to work for setting up for traffic that goes
to "blessed" destinations.   

We will 1.) set up a qdisc for netem on the device, and have it affect all
traffic in the htb first. 
Then we will try to set up a netem only for certain traffic in the htb. 


Can you list all of the destinations or iproute2 rules used to classify
traffic to tc classes?

so we have a couple of options.  We can look at the tc source in iproute2
code I downloaded on sysnet35.  However, this is pretty low-level code; we
have to communicate over a netlink socket in order to feed commands to TC.   
The alternative is to fork/exec from ulogd when we need to issue new tc
commands.   In the short term, let's do this.  



12-31-08

so the hack is to change the ceilings on all of the child slice classes.
That's one thing to do for all outbound traffic.  Theoretically we should
be able to just set the rule for 1:10 but that doesn't seem to be working.   


1.) /usr/share/util-vserver/bwlimit.py -v init
2.) /usr/share/util-vserver/bwlimit.py -v on pl_drl  1 8bit 1gbit 8bit 1gbit
3.) /usr/share/util-vserver/bwlimit.py -v on pl_drl_two  1 8bit 1gbit 8bit 1gbit

You can change the 1:10 class, and then you can off/on the slice limits.
The script makes sure to set the ceilings of the children classes to the
ceiling of the parents.  

Fine.   That's how they want it, that's how they'll get it.   What we need
to make sure is that the individual slices still share the outgoing
bandwidth appropriately.  

I'm not too sure what the traffic control does for you when they share
everything identically.   Let's assume that they want it.  

For FPS the limiter is a token bucket.  We want to simply set the rate to
something with this token bucket.  We should be able to do this exactly
with the right TC command.   No problem.  

For GRD the limiter is a packet dropper.  Drop X % of packets.   Let me
look at the outgoing schedulers.   It doesn't look like we will be able to
do it with TC.   We may be able to do it with a netfilter module.  Probably
a custom module.  They probably use netfilter commands to assign packets to
tc classes anyhow.   so we need to add similar rules.  

So for now let's focus on FPS.   In this case we need to be able to
interact with TC from the ulogd program, either through a call or
directly.  





12-29-08

check this out.  The example looks familiar to how tc is used in plab.
http://edseek.com/~jasonb/articles/traffic_shaping/class.html


qdiscs are added at the very bottom for each class whose minor number is
the xid.   There is one parent qdisc which allows you to add classes.  That
is the htb classfull qdisc.   The bottom qdisc's are pfifo's which I
believe are classless.  pfifo is the scheduler that all leaf nodes of
classful qdiscs are assigned.  The only thing you can do is to determine
the size the queue.  This is scheduling, not shaping. 

You can add other schedulers: sfq, which tries to be flow fair.  
tbf:  token bucket filter.  tbf is now a classful qdisc. 


You had better make sure that your rates of your child classes do not
exceed the rate of the parent class.   those are the minimums.   What about
the ceilings?   That says how much you can borrow.   Classes borrow in
proportion to the minimum rates that they are assigned.  Individual
ceilings should not exceed the parents rate.  Because it says how much you
can borrow when no one else is using anything.  

So do I have to write all new rules for all slices that are active when we
want to constrain them all?  

What happens when the parents rate/ceiling is less than the maximum that
they can borrow?  That apparently doesn't do any damm thing. But this is
probably why the script makes sure that the ceilings don't go above the
1:10 or 1:1.  It never checks for 1:20, which is probably b/c they always
expect it to have a ceiling equal to bandwidth max. 

OK.   If we set the ceiling of the child class to the ceiling of the
parent, tokens are borrowed appropriately.  If the ceiling is greater of
the child, then it looks like cburst has a lot to do with it.   So I have
to see what is happening with cburst. 


Setting 1:1 to rate 5mb no ceiling
setting 1:10 to rate 8 ceil 5mb
setting 1:1myslice rate 8 ceil 1gbit

This configuration does not work. 

when 1myslice ceil is > then parent ceil, looks like no borrowing occurs
unless the cburst is very low.  if you set cburst to 0, then tc sets cburst
to something that allows the class to reach its ceiling. 



HOWTO: Set a ceiling on a particular slice.  
/usr/share/util-vaserver/bwlimit.py on pl_drl 1 8bit 500kbit 8bit 1gib
The first arg is share.   Works in the way you expect.
The next is min guaranteed.  The next is max or ceiling.  This is the one
to set.  The next two are min/ceiling for special destinations.

HOWTO:  Set a ceiling on all slices leaving a node. 
1.) We can change the ceiling for the 1:10 subclass. 
this should limit all traffic for all slices to non-exempted destinations

 /sbin/tc class replace dev eth0 parent 1:1 classid 1:10 htb rate 8bit ceil 500kbit
This does not work. 

2.) We can change the ceiling for the 1:1 root class. 

 /sbin/tc class replace dev eth0 parent 1: classid 1:1 htb rate 500kbit
modified this to 
 /sbin/tc class replace dev eth0 parent 1: classid 1:1 htb rate 8bit ceil 500kbit
With no effect. 

Mucking with classid 1:1 seems to make the machine very unhappy.  


bwlimit seems to set the bottom classes ceilings to the ceiling of the 1:10
block.  Then why have the 1:10 block? 



Do 1&2, and neither have any effect.

It looks like only the bottom most class has any affect.  If I re-init the
bwlimit.py script, then a rule parent 1:10 leaf 1:1fff seems to get all the
traffic.  And it is the limit that seems to matter. 

I see.  1fff is the deafult_xid.  So when something has not been caught
earlier this is where it goes to be limited and caught. 


tcp netperf tests for rate limiting from a slice. 
Experiment with 34->35.   Logged in to the pl_drl slice to issue the
netperf.

vserver pl_drl status
vserver pl_drl_two status

vservers aren't running until you log in to them. so the above command
won't return something useful until that point. 



12-18-08

/etc/init.d/ncsd restart or reload (caches ldap information) this is for
the sysnet machines. 


The files that you care about are in the release.   they are
/usr/share/util-vaserver/bwlimit.py
This is what makes the calls to TC to do the traffic control.  



OK:  Logging in to the drl slice, and setting up the netperf
server/client. 
ssh -i ~/.ssh/myplc_admin.rsa -l pl_drl sysnet34.ucsd.edu

So I'm sending netperf outbound flows from the pl_drl slice
and it is getting recorded in the class htb 1:11f8 (which is context 504).
When you do a vserver list, though, it says there's only a 503 (though when
you log in to 503, it says the "security context" is 504). 


 1111  /usr/share/util-vserver/bwlimit.py -d eth0 -v init
 1115  /usr/share/util-vserver/bwlimit.py -d eth0 on pl_drl 1 1kbit 10kbit 1kbit 10kbit  


11-14-07


planetlab website login
kyocum@cs.ucsd.edu
ate smurfs as well
sysnet35 kyocum - big Dog - and killing name root


symlink cvs source code to a ulogd/drl directory. 

for file  in '*.h'; do echo $file; rm /root/myplc/ulogd/drl/$file ; done
for file  in '*.c'; do echo $file;  ln -s /root/ratelimiting/myplc/$file /root/myplc/ulogd/drl/. ; done


11-11-07

Brought back 32 and 33 from power offs.  32 is mounting everything read
only.  34 is working fine, but it was never powered off.  



10-29-07
Back from NSDI, SOSP, Texas, and firestorm.  Will modify ulogd to perform
DRL.   Will first use GRD, as we can drop based on simple probabilities.
Though we should be able to see all the packets that are coming in to the
machine using ulogd. 




9-16-07

vserver pl_netflow enter
scp root@sysnet35:~/myplc/build/RPMS/i386/ulogd-1.02-11.i386.rpm .

Where does the output go?   


9-14-07 #2

Now let's try to install the ulog srpm that was created on one of the myplc
nodes. 

sysnet32,34: myplc nodes
sysnet35: build box
sysnet33: myplc controller

On 34, entering vserver for ulogd, downloading srpm, and trying to
install. 

/etc/rc.d/init.d/ulogd stop
rpm -e --nodeps ulogd
rpm -Uvh ulogd-1.02-11.i386.rpm
/etc/rc.d/init.d/ulogd start


9-14-07

Have installed FC4 on sysnet35.  Am following Marc's instructions in his email.Check out rc1 build and rc1 ulogd.   
OK, I'm trying to build things using marc's instructions and not the entire
myplc build envi
ronment that is in the GUIDE.  I'll try that next.  
Ok, I've set it all up.  I'm using
mirrors.kernel.org 
and fedora/core/4/os 
--
Ok, now I'm about to cd into build and make ulogd . . .
q

xPWD=$PWD
cvs -d :pserver:anonymous@cvs.planet-lab.org:/cvs co -r planetlab-4_1-rc1
build
cvs -d :pserver:anonymous@cvs.planet-lab.org:/cvs co -r planetlab-4_1-rc1
ulogd
cd build
mkdir SPEC SOURCES
cd SPEC
# note that you will need these symlinks to use the full paths --- NOT
relative paths
# which is why I am using $xPWD
ln -fs $xPWD/ulogd/ulogd.spec ulogd.spec
cd ../SOURCES
ln -fs $xPWD/ulogd ulogd
cd ..
make ulogd

Assuming the dependencies are set up right, this should first suck down the
kernel and mysql modules via CVS, build them (which may take some time),
and then build ulogd.

Success!!!!!

Now, check out ratelimiting but add root to cvs group number 30008
groupadd -g 30008 cvs (add group def)


8-17-07

Made two slices.  pl_drl_one and pl_drl_two
to log in to the slices I uploaded a key, myplc_admin.{rsa,pub}

sake:~/.ssh grant$ ssh -i myplc_admin.rsa -l pl_drl_one sysnet34


We want to modify ULOGD

can't run it.  root@sysnet34 sbin]# ./ulogd --help
./ulogd: error while loading shared libraries: libproper.so.0: cannot open
shared object file: No such file or directory

trying chroot to the netflow slice first. 
That seems to do the trick.  

bash-3.00# ./ulogd 
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `raw'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `oob'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `ip'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `tcp'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `icmp'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `udp'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `ahesp'
Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `gre'
Fri Aug 17 20:05:08 2007 <5> ulogd.c:355 registering output `netflow'
ERROR: Unable to create netlink socket: Bad file descriptor
8-21-07

get yum to work on sysnet34.
in /etc/yum.conf, changed reposdir=/etc/yum.repos.d/
http://onelab-build.inria.fr/websvn/filedetails.php?repname=OneLab&path=%2Fnew_plc_www%2Ftrunk%2FPlanetLabConf%2Fyum.conf.php
has a crazy yum.conf file. 
yum install cvs
yum install gcc

[root@sysnet34 2.6.12-1.1398_FC4.5.planetlab]# cd /usr/src/
[root@sysnet34 src]# mkdir kernels

Install the kernel source the lib/modules directory symlinks to
/usr/src/kernels/ 
where we place 2.6.12-1.1398_FC4.5.planetlab

Checkout kernel:
cvs co -r planetlab-4_1-rc1 linux-2.6
placed in /usr/src/kernels/

Building PLAB kernel: 

fuck yum, no ncurses devel, no yum supository. 
wget http://coblitz.planet-lab.org/pub/fedora/linux/core/updates/4/x86_64/ncurses-devel-5.4-19.fc4.i386.rpm
rpm -i ncurses----

OK, now the stupid make menuconfig worked, but we're going to copy a kernel
config from /proc/config.gz 
zcat config.gz > /root/kernels/linux-2.6/.config
make O=/root/kernels/linux-2.6/build/ oldconfig 

So none of that worked, copy config to .config in top level directory.
then run menuconfig, then save config, then compile. that works. 

Yay!

./build/include/linux/version.h
so config that shit. 

Configuring ulogd
./configure --with-kernel=/root/kernels/linux-2.6/build/
config success!

download RPM and install from location above. 
mysql-devel-4.1.20-1.FC4.1.x86_64.rpm
rpm -i <>

Ok, had to re-enable yum.conf to use yum.repos.d
had to modify yum.repos.d/fedora-devel so that it wasn't used. 
then I could yum install openssl-devel and mysql-devel

The linker is bitching about mysql objects not being present. 
trying yum install mysql.i386
already installed trying
yum install ulogd-pgsql.i386 <no help>
yum install ulogd-mysql.i386 <no help>
yum install mysqlclient10-devel.i386

Ok, all that is shit.   Marc sent me a yumrepos.tgz that is on my
desktop. I put all the files in the right place on 34.  He then sent me a
yum install line that is this. 

yum -y install beecrypt-devel bzip2 coreutils cpio createrepo curl
curl-devel cvs db4-devel dev diffutils dnsmasq docbook-utils-pdf dosfstools
doxygen expect gcc-c++ gd glibc glibc-common gnupg gperf gzip httpd install
iptables less libpcap libpcap-devel libtool linuxdoc-tools mailx make
metadata mkisofs mod_python mod_ssl mysql mysql-devel mysql-server nasm
ncurses-devel openssh openssl php php-devel php-gd php-pgsql postgresql
postgresql-devel postgresql-python postgresql-server python python-devel
PyXML readline-devel redhat-rpm-config rpm rpm-build rpm-devel rsync
sendmail sendmail-cf sharutils sudo tar tetex-latex time vconfig vixie-cron
wget xmlsec1 xmlsec1-openssl yum

which should make the box a build box. 

Of course, what I should really be doing is making a fresh FC4 node as my
build environment, but whatever.  


------

Ok, trying to figure out if there is a way to "start" the pl_netflow
vserver.  Don't know if it's currently running.  Perhaps it is.
vserver-stat shows 4 vservers, besides the root context, 0. 
#503 is pl_netflow.  The other two are the two drl slices. 
Each vserver has a configuration file shown here: 
[root@sysnet34 sbin]# more /etc/vservers/pl_netflow.conf 

To enter a vserver context you can say
vserver <name> enter

To run ulogd

vserver pl_netflow enter
cd /usr/sbin
./ulogd

The configuration file is in 
/etc/ulogd.conf
/var/log/ulogd.log




8-07
Installing myPLC.  Trying to build rate limiting infrastructure on top of
it. 


sysnet33.   has only mounted /dev/xen_vg/dom0 on /
that is only 2GB.  There is another LVM called /dev/xen_vg/lvol0.
It has 10GB.  We will try to mount that. 
Display with lvmdisplay

The underlying disk partition is /dev/sda3 (physical volume)
display pv's with pvdisplay

adding line in /etc/fstab to mount lvol0 on /lvol0
This partition has plenty of space.  Everything looks good. 

use linux authconfig to see if it's using ldap for user authentication
to disable selinux
http://www.crypt.gen.nz/selinux/disable_selinux.html
It was already disabled, see /etc/selinux/config 
INSTALLED in /lvol0
I edited /lvol0/myplc/etc/sysconfig/plc
I symlinked /etc/init.d/plc --> installed
I symlinked /etc/sysconfig/plc --> installed

I edited etc/sysconfig/plc to point to my installation in /lvol0/myplc

root@localhost.localdomain
made a new account as well.
kyocum@cs.ucsd.edu
8smurf2

---

Trying to install the images for the new nodes.  They are in /lvol0/myplc/plc/root/data/var/www/html/download

on sysnet33,
yum install mkisofs
./bootcustom.sh DRL.iso plnode34.txt 


---

Machines come up in debug mode.  They have insufficient resources.
Probably disk space.   Need to modify the boot script that is sent over
from myplc (beckerr), or we can add a "/minhw" to the node type in the
controller. The boot program is at: 

-bash-3.00$ pwd
/lvol0/myplc/plc/data/var/www/html/boot/bootmanager.sh
nope just contains binary of bootmanager.
/lvol0/myplc/plc/root/etc/plc.d/bootmanager is where it's at. nope.
qq/usr/share/bootmanager/


Generally, running BootManager by hand is the standard procedure. Set the 
machine to "Boot" state via your PLC API or web server. When the machine
sfails 
to boot, hit Ctrl-C, login as root/root, cd /tmp/source, and run 
"./BootManager.py". Figure out where it dies, then start adding prints in
the 
appropriate places to figure out why.