-2-20-08
-
-Use this to capture the rates from the individual slices.
-
- while true; do /sbin/tc -s class show dev eth0 | grep -A 3 11f8 | grep -A
- 1 "class htb" | grep Sent | cut -d " " -f 3; sleep 1; done > GRD_pl_drl_34
-
-
-
-
-
-2-15-08
-
-Damn sysnet33 db won't come up. says /tmp is full in the chroot
-myplc/plc/root/var/log/pglog, but it's not full.
-So I'm trying to replace the loopback filesystem with something clean from
-the RPM.
-using cpio --no-absolute-filenames
-And
-rpm2cpio <rpm> | cpio -diu
-but u is don't prompt. so I'll issue
-rpm2cpio <rpm> | cpio -di --no-absolute_filenames
-
-OK, did this, then I copied over root.img to where I have it installed.
-And it all worked! Sweet. The image is mostly independent of the data!
-
-
-
-
-
-
-ADDING A PL NODE
-1.) turn off crond for restarting ulogd
-in pl_netflow:/etc/crond.d/netflow
-edited the line that calls init.d/netflow restart
-because I don't want it to keep restarting ulogd as I test it.
-2.) Allow pl_netflow to run tc
-I am also changing the pl_netflow.conf in /etc/vservers/pl_netflow.conf
-So that S_CAPS="CAP_NET_ADMIN"
-Then, from the root slice, vserver pl_netflow restart.
-3.) copy over the ./start script from the pl_netflow /root directory.
-4.) put bwlimit_drl.py into /usr/share/util-vserver/bwlimit2.py on the root
-slice
-put bwlimit_drl.py into /usr/share/util-vserver/bwlimit_drl.py on the
-pl_netflow slice.
-Future, make the names consistent
-5.) pull the sch_netem.ko over and insmod it.
-
-cd /lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/
-[root@sysnet32 sched]# scp -i ~/.ssh/root_ssh_key.rsa root@sysnet34:/lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/sch_netem_plab.ko .
-insmod sch_netem_plab.ko
-
-6.) [root@vserver:pl_netflow ~]/etc/init.d/ulogd stop
-7.) vserver pl_netflow enter
-8.) copy over ulogd. run ./start
-9.) Edit the ulogd configuration script.
-
-
-2-12-08
-
-bwlimit.py cannot look up the slice names in the pl_netflow slice b/c there
-are not entries in the /etc/passwd file for the slices. Of course there
-aren't. Even ulogd_NETFLOW.c replaces /etc/passwd with /dev/null.
-
-so in the short term I will copy over the roots /etc/passwd and place it
-where the new bwlimit_drl exists. /usr/share/util-vserver/passwd_drl
-I can have the root periodically copy this file into the pl_netflow slice
-in a cron job, or whenever a new slice is created.
-
-This is a hack. There are probably better ways, but it's a moving target.
-Now hack bwlimit_drl to use that file to look up slice contexts.
-
-
-
-HTB DOCS http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
-
-2-10-08
-
-We probably have to put the netem at the bottom of the tbf. If we put it
-in the middle, say off of the 1:10 class, then we need to direct all
-traffic that would have been directed to a lower class in the htb to the
-1:10 class. Then the traffic has to be directed to the lower classes and
-be accounted for directly. This screws with how traffic classification is
-done; I don't even know if it will obey the tb appropriately either, since
-the best you can do is put another htb qdisc below the netem class. And
-there is no way to directly have that qdisc borrow from the "parent" one
-that's just been terminated by the netem qdisc. Got that?
-
-
-
-
-
-
-2-7-08
-
-Screwing around with all the kernel configs has forced the build process to
-rebuild all the kernels. blech.
-
-
-BUILDING NETEM for plab machines:
-Kevin made an ubunto chroot install on his linux machine and built 3.3 gcc
-in it, and then copied over the source linux tree that had the .config
-stolen from sysnet34 /proc/config.gz. We also had to change the magic
-string so that it believed it was bulit to fc 4.5, not 4.7.
-Use insmod
-/lib/modules/2.6.12-1.1398_FC4.5.planetlab/kernel/net/sched/sch_netem_plab.ko
-
-
-see sysnet35:/root/myplc/build/SOURCES/CRAP/linux-2.6.12/.config
-
-
-We added a node at the root qdisc and it succesfully appears to delay
-packets. So now let's figure out how to add it to the bwlimit.py code so
-that it is always present and can be tweaked (replaced) when running
-ulogd.
-
-----
-tc qdisc del dev eth0 root handle 1:
-tc qdisc add dev eth0 parent 1: handle 2: htb default 1fff
-And everything then falls out of 2:
-
-But this will netem everything out of hte host, leaving nothing for
-"exempt" slices like the root. Not what we want.
-
-We place a netem qdisc at the leaf of every non-exempt slice traffic class
-in the HTB.
-
-2-5-08
-
-in pl_netflow vserver
-/etc/cron.d/netflow keeps restarting netflow init.d script which keeps
-restarting ulogd if started by hand. Fucking buggy.
-
-in root vserver
-see pl_mom cron job
-which runs bwmon and pl_mop. pl_mop makes sure pl_netflow is running.
-
-!!!!!<CHANGE TO SYSNET34>!!!!!
-in sysnet34 pl_netflow:/etc/crond.d/netflow
-edited the line that calls init.d/netflow restart
-because I don't want it to keep restarting ulogd as I test it.
-!!!!!
-I am also changing the pl_netflow.conf in /etc/vservers/pl_netflow.conf
-So that S_CAPS="CAP_NET_ADMIN"
-Then, from the root slice, vserver pl_netflow restart.
-
-<changing kernel config files in build/linux-x-x-x/configs directory
-i686-smp-plab
-i686-onelab
-i686-planetlab
-all added CONFIG_NET_SCH_NETEM=m
-
-
-2-1-08
-
-GRD dropping. Use netem, which is accessible also through tc.
-I believe what we want to do is put the netem on the interface for all
-traffic? OK, that's not going to work for setting up for traffic that goes
-to "blessed" destinations.
-
-We will 1.) set up a qdisc for netem on the device, and have it affect all
-traffic in the htb first.
-Then we will try to set up a netem only for certain traffic in the htb.
-
-
-Can you list all of the destinations or iproute2 rules used to classify
-traffic to tc classes?
-
-so we have a couple of options. We can look at the tc source in iproute2
-code I downloaded on sysnet35. However, this is pretty low-level code; we
-have to communicate over a netlink socket in order to feed commands to TC.
-The alternative is to fork/exec from ulogd when we need to issue new tc
-commands. In the short term, let's do this.
-
-
-
-12-31-08
-
-so the hack is to change the ceilings on all of the child slice classes.
-That's one thing to do for all outbound traffic. Theoretically we should
-be able to just set the rule for 1:10 but that doesn't seem to be working.
-
-
-1.) /usr/share/util-vserver/bwlimit.py -v init
-2.) /usr/share/util-vserver/bwlimit.py -v on pl_drl 1 8bit 1gbit 8bit 1gbit
-3.) /usr/share/util-vserver/bwlimit.py -v on pl_drl_two 1 8bit 1gbit 8bit 1gbit
-
-You can change the 1:10 class, and then you can off/on the slice limits.
-The script makes sure to set the ceilings of the children classes to the
-ceiling of the parents.
-
-Fine. That's how they want it, that's how they'll get it. What we need
-to make sure is that the individual slices still share the outgoing
-bandwidth appropriately.
-
-I'm not too sure what the traffic control does for you when they share
-everything identically. Let's assume that they want it.
-
-For FPS the limiter is a token bucket. We want to simply set the rate to
-something with this token bucket. We should be able to do this exactly
-with the right TC command. No problem.
-
-For GRD the limiter is a packet dropper. Drop X % of packets. Let me
-look at the outgoing schedulers. It doesn't look like we will be able to
-do it with TC. We may be able to do it with a netfilter module. Probably
-a custom module. They probably use netfilter commands to assign packets to
-tc classes anyhow. so we need to add similar rules.
-
-So for now let's focus on FPS. In this case we need to be able to
-interact with TC from the ulogd program, either through a call or
-directly.
-
-
-
-
-
-12-29-08
-
-check this out. The example looks familiar to how tc is used in plab.
-http://edseek.com/~jasonb/articles/traffic_shaping/class.html
-
-
-qdiscs are added at the very bottom for each class whose minor number is
-the xid. There is one parent qdisc which allows you to add classes. That
-is the htb classfull qdisc. The bottom qdisc's are pfifo's which I
-believe are classless. pfifo is the scheduler that all leaf nodes of
-classful qdiscs are assigned. The only thing you can do is to determine
-the size the queue. This is scheduling, not shaping.
-
-You can add other schedulers: sfq, which tries to be flow fair.
-tbf: token bucket filter. tbf is now a classful qdisc.
-
-
-You had better make sure that your rates of your child classes do not
-exceed the rate of the parent class. those are the minimums. What about
-the ceilings? That says how much you can borrow. Classes borrow in
-proportion to the minimum rates that they are assigned. Individual
-ceilings should not exceed the parents rate. Because it says how much you
-can borrow when no one else is using anything.
-
-So do I have to write all new rules for all slices that are active when we
-want to constrain them all?
-
-What happens when the parents rate/ceiling is less than the maximum that
-they can borrow? That apparently doesn't do any damm thing. But this is
-probably why the script makes sure that the ceilings don't go above the
-1:10 or 1:1. It never checks for 1:20, which is probably b/c they always
-expect it to have a ceiling equal to bandwidth max.
-
-OK. If we set the ceiling of the child class to the ceiling of the
-parent, tokens are borrowed appropriately. If the ceiling is greater of
-the child, then it looks like cburst has a lot to do with it. So I have
-to see what is happening with cburst.
-
-
-Setting 1:1 to rate 5mb no ceiling
-setting 1:10 to rate 8 ceil 5mb
-setting 1:1myslice rate 8 ceil 1gbit
-
-This configuration does not work.
-
-when 1myslice ceil is > then parent ceil, looks like no borrowing occurs
-unless the cburst is very low. if you set cburst to 0, then tc sets cburst
-to something that allows the class to reach its ceiling.
-
-
-
-HOWTO: Set a ceiling on a particular slice.
-/usr/share/util-vaserver/bwlimit.py on pl_drl 1 8bit 500kbit 8bit 1gib
-The first arg is share. Works in the way you expect.
-The next is min guaranteed. The next is max or ceiling. This is the one
-to set. The next two are min/ceiling for special destinations.
-
-HOWTO: Set a ceiling on all slices leaving a node.
-1.) We can change the ceiling for the 1:10 subclass.
-this should limit all traffic for all slices to non-exempted destinations
-
- /sbin/tc class replace dev eth0 parent 1:1 classid 1:10 htb rate 8bit ceil 500kbit
-This does not work.
-
-2.) We can change the ceiling for the 1:1 root class.
-
- /sbin/tc class replace dev eth0 parent 1: classid 1:1 htb rate 500kbit
-modified this to
- /sbin/tc class replace dev eth0 parent 1: classid 1:1 htb rate 8bit ceil 500kbit
-With no effect.
-
-Mucking with classid 1:1 seems to make the machine very unhappy.
-
-
-bwlimit seems to set the bottom classes ceilings to the ceiling of the 1:10
-block. Then why have the 1:10 block?
-
-
-
-Do 1&2, and neither have any effect.
-
-It looks like only the bottom most class has any affect. If I re-init the
-bwlimit.py script, then a rule parent 1:10 leaf 1:1fff seems to get all the
-traffic. And it is the limit that seems to matter.
-
-I see. 1fff is the deafult_xid. So when something has not been caught
-earlier this is where it goes to be limited and caught.
-
-
-tcp netperf tests for rate limiting from a slice.
-Experiment with 34->35. Logged in to the pl_drl slice to issue the
-netperf.
-
-vserver pl_drl status
-vserver pl_drl_two status
-
-vservers aren't running until you log in to them. so the above command
-won't return something useful until that point.
-
-
-
-12-18-08
-
-/etc/init.d/ncsd restart or reload (caches ldap information) this is for
-the sysnet machines.
-
-
-The files that you care about are in the release. they are
-/usr/share/util-vaserver/bwlimit.py
-This is what makes the calls to TC to do the traffic control.
-
-
-
-OK: Logging in to the drl slice, and setting up the netperf
-server/client.
-ssh -i ~/.ssh/myplc_admin.rsa -l pl_drl sysnet34.ucsd.edu
-
-So I'm sending netperf outbound flows from the pl_drl slice
-and it is getting recorded in the class htb 1:11f8 (which is context 504).
-When you do a vserver list, though, it says there's only a 503 (though when
-you log in to 503, it says the "security context" is 504).
-
-
- 1111 /usr/share/util-vserver/bwlimit.py -d eth0 -v init
- 1115 /usr/share/util-vserver/bwlimit.py -d eth0 on pl_drl 1 1kbit 10kbit 1kbit 10kbit
-
-
-11-14-07
-
-
-planetlab website login
-kyocum@cs.ucsd.edu
-ate smurfs as well
-sysnet35 kyocum - big Dog - and killing name root
-
-
-symlink cvs source code to a ulogd/drl directory.
-
-for file in '*.h'; do echo $file; rm /root/myplc/ulogd/drl/$file ; done
-for file in '*.c'; do echo $file; ln -s /root/ratelimiting/myplc/$file /root/myplc/ulogd/drl/. ; done
-
-
-11-11-07
-
-Brought back 32 and 33 from power offs. 32 is mounting everything read
-only. 34 is working fine, but it was never powered off.
-
-
-
-10-29-07
-Back from NSDI, SOSP, Texas, and firestorm. Will modify ulogd to perform
-DRL. Will first use GRD, as we can drop based on simple probabilities.
-Though we should be able to see all the packets that are coming in to the
-machine using ulogd.
-
-
-
-
-9-16-07
-
-vserver pl_netflow enter
-scp root@sysnet35:~/myplc/build/RPMS/i386/ulogd-1.02-11.i386.rpm .
-
-Where does the output go?
-
-
-9-14-07 #2
-
-Now let's try to install the ulog srpm that was created on one of the myplc
-nodes.
-
-sysnet32,34: myplc nodes
-sysnet35: build box
-sysnet33: myplc controller
-
-On 34, entering vserver for ulogd, downloading srpm, and trying to
-install.
-
-/etc/rc.d/init.d/ulogd stop
-rpm -e --nodeps ulogd
-rpm -Uvh ulogd-1.02-11.i386.rpm
-/etc/rc.d/init.d/ulogd start
-
-
-9-14-07
-
-Have installed FC4 on sysnet35. Am following Marc's instructions in his email.Check out rc1 build and rc1 ulogd.
-OK, I'm trying to build things using marc's instructions and not the entire
-myplc build envi
-ronment that is in the GUIDE. I'll try that next.
-Ok, I've set it all up. I'm using
-mirrors.kernel.org
-and fedora/core/4/os
---
-Ok, now I'm about to cd into build and make ulogd . . .
-q
-
-xPWD=$PWD
-cvs -d :pserver:anonymous@cvs.planet-lab.org:/cvs co -r planetlab-4_1-rc1
-build
-cvs -d :pserver:anonymous@cvs.planet-lab.org:/cvs co -r planetlab-4_1-rc1
-ulogd
-cd build
-mkdir SPEC SOURCES
-cd SPEC
-# note that you will need these symlinks to use the full paths --- NOT
-relative paths
-# which is why I am using $xPWD
-ln -fs $xPWD/ulogd/ulogd.spec ulogd.spec
-cd ../SOURCES
-ln -fs $xPWD/ulogd ulogd
-cd ..
-make ulogd
-
-Assuming the dependencies are set up right, this should first suck down the
-kernel and mysql modules via CVS, build them (which may take some time),
-and then build ulogd.
-
-Success!!!!!
-
-Now, check out ratelimiting but add root to cvs group number 30008
-groupadd -g 30008 cvs (add group def)
-
-
-8-17-07
-
-Made two slices. pl_drl_one and pl_drl_two
-to log in to the slices I uploaded a key, myplc_admin.{rsa,pub}
-
-sake:~/.ssh grant$ ssh -i myplc_admin.rsa -l pl_drl_one sysnet34
-
-
-We want to modify ULOGD
-
-can't run it. root@sysnet34 sbin]# ./ulogd --help
-./ulogd: error while loading shared libraries: libproper.so.0: cannot open
-shared object file: No such file or directory
-
-trying chroot to the netflow slice first.
-That seems to do the trick.
-
-bash-3.00# ./ulogd
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `raw'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `oob'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `ip'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `tcp'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `icmp'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `udp'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `ahesp'
-Fri Aug 17 20:05:08 2007 <3> ulogd.c:300 registering interpreter `gre'
-Fri Aug 17 20:05:08 2007 <5> ulogd.c:355 registering output `netflow'
-ERROR: Unable to create netlink socket: Bad file descriptor
-8-21-07
-
-get yum to work on sysnet34.
-in /etc/yum.conf, changed reposdir=/etc/yum.repos.d/
-http://onelab-build.inria.fr/websvn/filedetails.php?repname=OneLab&path=%2Fnew_plc_www%2Ftrunk%2FPlanetLabConf%2Fyum.conf.php
-has a crazy yum.conf file.
-yum install cvs
-yum install gcc
-
-[root@sysnet34 2.6.12-1.1398_FC4.5.planetlab]# cd /usr/src/
-[root@sysnet34 src]# mkdir kernels
-
-Install the kernel source the lib/modules directory symlinks to
-/usr/src/kernels/
-where we place 2.6.12-1.1398_FC4.5.planetlab
-
-Checkout kernel:
-cvs co -r planetlab-4_1-rc1 linux-2.6
-placed in /usr/src/kernels/
-
-Building PLAB kernel:
-
-fuck yum, no ncurses devel, no yum supository.
-wget http://coblitz.planet-lab.org/pub/fedora/linux/core/updates/4/x86_64/ncurses-devel-5.4-19.fc4.i386.rpm
-rpm -i ncurses----
-
-OK, now the stupid make menuconfig worked, but we're going to copy a kernel
-config from /proc/config.gz
-zcat config.gz > /root/kernels/linux-2.6/.config
-make O=/root/kernels/linux-2.6/build/ oldconfig
-
-So none of that worked, copy config to .config in top level directory.
-then run menuconfig, then save config, then compile. that works.
-
-Yay!
-
-./build/include/linux/version.h
-so config that shit.
-
-Configuring ulogd
-./configure --with-kernel=/root/kernels/linux-2.6/build/
-config success!
-
-download RPM and install from location above.
-mysql-devel-4.1.20-1.FC4.1.x86_64.rpm
-rpm -i <>
-
-Ok, had to re-enable yum.conf to use yum.repos.d
-had to modify yum.repos.d/fedora-devel so that it wasn't used.
-then I could yum install openssl-devel and mysql-devel
-
-The linker is bitching about mysql objects not being present.
-trying yum install mysql.i386
-already installed trying
-yum install ulogd-pgsql.i386 <no help>
-yum install ulogd-mysql.i386 <no help>
-yum install mysqlclient10-devel.i386
-
-Ok, all that is shit. Marc sent me a yumrepos.tgz that is on my
-desktop. I put all the files in the right place on 34. He then sent me a
-yum install line that is this.
-
-yum -y install beecrypt-devel bzip2 coreutils cpio createrepo curl
-curl-devel cvs db4-devel dev diffutils dnsmasq docbook-utils-pdf dosfstools
-doxygen expect gcc-c++ gd glibc glibc-common gnupg gperf gzip httpd install
-iptables less libpcap libpcap-devel libtool linuxdoc-tools mailx make
-metadata mkisofs mod_python mod_ssl mysql mysql-devel mysql-server nasm
-ncurses-devel openssh openssl php php-devel php-gd php-pgsql postgresql
-postgresql-devel postgresql-python postgresql-server python python-devel
-PyXML readline-devel redhat-rpm-config rpm rpm-build rpm-devel rsync
-sendmail sendmail-cf sharutils sudo tar tetex-latex time vconfig vixie-cron
-wget xmlsec1 xmlsec1-openssl yum
-
-which should make the box a build box.
-
-Of course, what I should really be doing is making a fresh FC4 node as my
-build environment, but whatever.
-
-
-------
-
-Ok, trying to figure out if there is a way to "start" the pl_netflow
-vserver. Don't know if it's currently running. Perhaps it is.
-vserver-stat shows 4 vservers, besides the root context, 0.
-#503 is pl_netflow. The other two are the two drl slices.
-Each vserver has a configuration file shown here:
-[root@sysnet34 sbin]# more /etc/vservers/pl_netflow.conf
-
-To enter a vserver context you can say
-vserver <name> enter
-
-To run ulogd
-
-vserver pl_netflow enter
-cd /usr/sbin
-./ulogd
-
-The configuration file is in
-/etc/ulogd.conf
-/var/log/ulogd.log
-
-
-
-
-8-07
-Installing myPLC. Trying to build rate limiting infrastructure on top of
-it.
-
-
-sysnet33. has only mounted /dev/xen_vg/dom0 on /
-that is only 2GB. There is another LVM called /dev/xen_vg/lvol0.
-It has 10GB. We will try to mount that.
-Display with lvmdisplay
-
-The underlying disk partition is /dev/sda3 (physical volume)
-display pv's with pvdisplay
-
-adding line in /etc/fstab to mount lvol0 on /lvol0
-This partition has plenty of space. Everything looks good.
-
-use linux authconfig to see if it's using ldap for user authentication
-to disable selinux
-http://www.crypt.gen.nz/selinux/disable_selinux.html
-It was already disabled, see /etc/selinux/config
-INSTALLED in /lvol0
-I edited /lvol0/myplc/etc/sysconfig/plc
-I symlinked /etc/init.d/plc --> installed
-I symlinked /etc/sysconfig/plc --> installed
-
-I edited etc/sysconfig/plc to point to my installation in /lvol0/myplc
-
-root@localhost.localdomain
-made a new account as well.
-kyocum@cs.ucsd.edu
-8smurf2
-
----
-
-Trying to install the images for the new nodes. They are in /lvol0/myplc/plc/root/data/var/www/html/download
-
-on sysnet33,
-yum install mkisofs
-./bootcustom.sh DRL.iso plnode34.txt
-
-
----
-
-Machines come up in debug mode. They have insufficient resources.
-Probably disk space. Need to modify the boot script that is sent over
-from myplc (beckerr), or we can add a "/minhw" to the node type in the
-controller. The boot program is at:
-
--bash-3.00$ pwd
-/lvol0/myplc/plc/data/var/www/html/boot/bootmanager.sh
-nope just contains binary of bootmanager.
-/lvol0/myplc/plc/root/etc/plc.d/bootmanager is where it's at. nope.
-qq/usr/share/bootmanager/
-
-
-Generally, running BootManager by hand is the standard procedure. Set the
-machine to "Boot" state via your PLC API or web server. When the machine
-sfails
-to boot, hit Ctrl-C, login as root/root, cd /tmp/source, and run
-"./BootManager.py". Figure out where it dies, then start adding prints in
-the
-appropriate places to figure out why.
-
-
-
git://git.onelab.eu / distributedratelimiting.git / commitdiff