From e3c7ed29100937d6bd6247342c96bc90b7ff9c1c Mon Sep 17 00:00:00 2001 From: Ben Pfaff Date: Fri, 14 Feb 2014 10:34:58 -0800 Subject: [PATCH] tests: Add support for automatically running Ryu tests against OVS. The Ryu controller comes with an extensive library of OpenFlow tests, but it doesn't seem so easy to me to run all of them against a development version of Open vSwitch. This commit introduces a Makefile target so that one can run all the Ryu tests with a simple "make check-ryu". This commit adds documentation for the new target to INSTALL. It also moves the documentation for the "check-oftest" target and the --enable-coverage configure option into INSTALL. Signed-off-by: Ben Pfaff Reviewed-by: Simon Horman --- INSTALL | 110 +++++++++++++++++++++++++++++++++---- Makefile.am | 2 - NEWS | 2 + README-OFTest | 77 -------------------------- README-gcov | 18 ------- tests/automake.mk | 5 ++ tests/run-ryu | 134 ++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 242 insertions(+), 106 deletions(-) delete mode 100644 README-OFTest delete mode 100644 README-gcov create mode 100755 tests/run-ryu diff --git a/INSTALL b/INSTALL index 001d3cbea..9f9491ff0 100644 --- a/INSTALL +++ b/INSTALL @@ -185,6 +185,11 @@ Prerequisites section, follow the procedure below to build. command line, turning warnings into errors. That makes it impossible to miss warnings generated by the build. + To build with gcov code coverage support, add --enable-coverage, + e.g.: + + % ./configure --enable-coverage + The configure script accepts a number of other options and honors additional environment variables. For a full list, invoke configure with the --help option. @@ -379,20 +384,27 @@ above, but also replaces the old kernel module with the new one. Open vSwitch startup scripts for Debian, XenServer and RHEL use ovs-ctl's functions and it is recommended that these functions be used for other software platforms too. -Running the Testsuite -===================== +Testsuites +========== + +This section describe Open vSwitch's built-in support for various test +suites. You must configure and build Open vSwitch (steps 1 through 3 +in "Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD" +above) before you run the tests described here. You do not need to +install Open vSwitch or to build or load the kernel module to run +these test suites. You do not need supervisor privilege to run these +test suites. -Open vSwitch includes a testsuite. Before you submit patches +Self-Tests +---------- + +Open vSwitch includes a suite of self-tests. Before you submit patches upstream, we advise that you run the tests and ensure that they pass. If you add new features to Open vSwitch, then adding tests for those features will ensure your features don't break as developers modify other areas of Open vSwitch. -You must configure and build Open vSwitch (steps 1 through 3 in -"Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD" above) -before you run the testsuite. You do not need to install Open vSwitch -or to build or load the kernel module to run the testsuite. You do -not need supervisor privilege to run the testsuite. +Refer to "Testsuites" above for prerequisites. To run all the unit tests in Open vSwitch, one at a time: make check @@ -432,7 +444,87 @@ usually a bug in the testsuite, not a bug in Open vSwitch itself. If you find that a test fails intermittently, please report it, since the developers may not have noticed. +OFTest +------ + +OFTest is an OpenFlow protocol testing suite. Open vSwitch includes a +Makefile target to run OFTest with Open vSwitch in "dummy mode". In +this mode of testing, no packets travel across physical or virtual +networks. Instead, Unix domain sockets stand in as simulated +networks. This simulation is imperfect, but it is much easier to set +up, does not require extra physical or virtual hardware, and does not +require supervisor privileges. + +To run OFTest with Open vSwitch, first read and follow the +instructions under "Testsuites" above. Second, obtain a copy of +OFTest and install its prerequisites. You need a copy of OFTest that +includes commit 406614846c5 (make ovs-dummy platform work again). +This commit was merged into the OFTest repository on Feb 1, 2013, so +any copy of OFTest more recent than that should work. Testing OVS in +dummy mode does not require root privilege, so you may ignore that +requirement. + +Optionally, add the top-level OFTest directory (containing the "oft" +program) to your $PATH. This slightly simplifies running OFTest later. + +To run OFTest in dummy mode, run the following command from your Open +vSwitch build directory: + make check-oftest OFT= +where is the absolute path to the "oft" program in +OFTest. + +If you added "oft" to your $PATH, you may omit the OFT variable +assignment: + make check-oftest +By default, "check-oftest" passes "oft" just enough options to enable +dummy mode. You can use OFTFLAGS to pass additional options. For +example, to run just the basic.Echo test instead of all tests (the +default) and enable verbose logging: + make check-oftest OFT= OFTFLAGS='--verbose -T basic.Echo' + +If you use OFTest that does not include commit 4d1f3eb2c792 (oft: +change default port to 6653), merged into the OFTest repository in +October 2013, then you need to add an option to use the IETF-assigned +controller port: + make check-oftest OFT= OFTFLAGS='--port=6653' + +Please interpret OFTest results cautiously. Open vSwitch can fail a +given test in OFTest for many reasons, including bugs in Open vSwitch, +bugs in OFTest, bugs in the "dummy mode" integration, and differing +interpretations of the OpenFlow standard and other standards. + +Open vSwitch has not been validated against OFTest. Please do report +test failures that you believe to represent bugs in Open vSwitch. +Include the precise versions of Open vSwitch and OFTest in your bug +report, plus any other information needed to reproduce the problem. + +Ryu +--- + +Ryu is an OpenFlow controller written in Python that includes an +extensive OpenFlow testsuite. Open vSwitch includes a Makefile target +to run Ryu in "dummy mode". See "OFTest" above for an explanation of +dummy mode. + +To run Ryu tests with Open vSwitch, first read and follow the +instructions under "Testsuites" above. Second, obtain a copy of Ryu, +install its prerequisites, and build it. You do not need to install +Ryu (some of the tests do not get installed, so it does not help). + +To run Ryu tests, run the following command from your Open vSwitch +build directory: + make check-ryu RYUDIR= +where is the absolute path to the root of the Ryu +source distribution. The default is $srcdir/../ryu +where $srcdir is your Open vSwitch source directory, so if this +default is correct then you make simply run "make check-ryu". + +Open vSwitch has not been validated against Ryu. Please do report +test failures that you believe to represent bugs in Open vSwitch. +Include the precise versions of Open vSwitch and Ryu in your bug +report, plus any other information needed to reproduce the problem. + Bug Reporting -------------- +============= Please report problems to bugs@openvswitch.org. diff --git a/Makefile.am b/Makefile.am index 340b12dbe..6a2aca240 100644 --- a/Makefile.am +++ b/Makefile.am @@ -73,8 +73,6 @@ EXTRA_DIST = \ NOTICE \ OPENFLOW-1.1+ \ PORTING \ - README-OFTest \ - README-gcov \ README-lisp \ REPORTING-BUGS \ SubmittingPatches \ diff --git a/NEWS b/NEWS index fe4b8cf2e..34dde6b9f 100644 --- a/NEWS +++ b/NEWS @@ -2,6 +2,8 @@ Post-v2.1.0 --------------------- - The "ovsdbmonitor" graphical tool has been removed, because it was poorly maintained and not widely used. + - New "check-ryu" Makefile target for running Ryu tests for OpenFlow + controllers against Open vSwitch. See INSTALL for details. v2.1.0 - xx xxx xxxx --------------------- diff --git a/README-OFTest b/README-OFTest deleted file mode 100644 index cdca06d37..000000000 --- a/README-OFTest +++ /dev/null @@ -1,77 +0,0 @@ - How to Use OFTest With Open vSwitch - =================================== - -This document describes how to use the OFTest OpenFlow protocol -testing suite with Open vSwitch in "dummy mode". In this mode of -testing, no packets travel across physical or virtual networks. -Instead, Unix domain sockets stand in as simulated networks. This -simulation is imperfect, but it is much easier to set up, does not -require extra physical or virtual hardware, and does not require -supervisor privileges. - -Prerequisites -------------- - -First, build Open vSwitch according to the instructions in INSTALL. -You need not install it. - -Second, obtain a copy of OFTest and install its prerequisites. You -need a copy of OFTest that includes commit 406614846c5 (make ovs-dummy -platform work again). This commit was merged into the OFTest -repository on Feb 1, 2013, so any copy of OFTest more recent than that -should work. - -Testing OVS in dummy mode does not require root privilege, so you may -ignore that requirement. - -Optionally, add the top-level OFTest directory (containing the "oft" -program) to your $PATH. This slightly simplifies running OFTest later. - -Running OFTest --------------- - -To run OFTest in dummy mode, run the following command from your Open -vSwitch build directory: - - make check-oftest OFT= - -where is the absolute path to the "oft" program in -OFTest. - -If you added "oft" to your $PATH, you may omit the OFT variable -assignment: - - make check-oftest - -By default, "check-oftest" passes "oft" just enough options to enable -dummy mode. You can use OFTFLAGS to pass additional options. For -example, to run just the basic.Echo test instead of all tests (the -default) and enable verbose logging: - - make check-oftest OFT= OFTFLAGS='--verbose -T basic.Echo' - -If you use OFTest that does not include commit 4d1f3eb2c792 (oft: -change default port to 6653), merged into the OFTest repository in -October 2013, then you need to add an option to use the IETF-assigned -controller port: - - make check-oftest OFT= OFTFLAGS='--port=6653' - -Interpreting OFTest Results ---------------------------- - -Please interpret OFTest results cautiously. Open vSwitch can fail a -given test in OFTest for many reasons, including bugs in Open vSwitch, -bugs in OFTest, bugs in the "dummy mode" integration, and differing -interpretations of the OpenFlow standard and other standards. - -Open vSwitch has not been validated against OFTest. Please do report -test failures that you believe to represent bugs in Open vSwitch. -Include the precise versions of Open vSwitch and OFTest in your bug -report, plus any other information needed to reproduce the problem. - -Contact -------- - -bugs@openvswitch.org -http://openvswitch.org/ diff --git a/README-gcov b/README-gcov deleted file mode 100644 index 2fe9f3a7c..000000000 --- a/README-gcov +++ /dev/null @@ -1,18 +0,0 @@ -Building with gcov support -========================== - -The Open vSwitch "configure" script supports the following -code-coverage related options: - - --disable-coverage - --enable-coverage=no - - Do not build with gcov code coverage support. - - This is the default if no coverage option is passed to - "configure". - - --enable-coverage - --enable-coverage=yes - - Build with gcov code coverage support. diff --git a/tests/automake.mk b/tests/automake.mk index f78547bdb..0cf45a4b8 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -154,6 +154,11 @@ check-valgrind: all tests/atconfig tests/atlocal $(TESTSUITE) \ check-oftest: all srcdir='$(srcdir)' $(SHELL) $(srcdir)/tests/run-oftest EXTRA_DIST += tests/run-oftest + +# Ryu support. +check-ryu: all + srcdir='$(srcdir)' $(SHELL) $(srcdir)/tests/run-ryu +EXTRA_DIST += tests/run-ryu clean-local: test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean diff --git a/tests/run-ryu b/tests/run-ryu new file mode 100755 index 000000000..678980f73 --- /dev/null +++ b/tests/run-ryu @@ -0,0 +1,134 @@ +#! /bin/sh + +run () { + echo "$@" + "$@" || exit 1 +} + +# Put built tools early in $PATH. +builddir=`pwd` +if test ! -e vswitchd/ovs-vswitchd; then + echo >&2 'not in build directory, please change directory or run via \"make check-ryu' + exit 1 +fi +PATH=$builddir/ovsdb:$builddir/vswitchd:$builddir/utilities:$PATH; export PATH + +# Find srcdir. +case $srcdir in + '') srcdir=$builddir ;; + /*) ;; + *) srcdir=`pwd`/$srcdir ;; +esac +if test ! -e "$srcdir"/WHY-OVS; then + echo >&2 'source directory not found, please set $srcdir or run via \"make check-ryu' + exit 1 +fi + +# Make sure ryu is available. +if test X"$RYUDIR" = X; then + RYUDIR=$srcdir/../ryu +fi +PYTHONPATH=$RYUDIR:$PYTHONPATH; export PYTHONPATH +PATH=$RYUDIR/bin:$PATH; export PATH +if (ryu-manager --version) >/dev/null 2>&1; then + : +else + echo >&2 '"ryu-manager" binary not found or cannot be run, please set $RYUDIR' + exit 1 +fi + +# Create sandbox. +rm -rf sandbox +mkdir sandbox +cd sandbox +sandbox=`pwd` + +# Set up environment for OVS programs to sandbox themselves. +OVS_RUNDIR=$sandbox; export OVS_RUNDIR +OVS_LOGDIR=$sandbox; export OVS_LOGDIR +OVS_DBDIR=$sandbox; export OVS_DBDIR +OVS_SYSCONFDIR=$sandbox; export OVS_SYSCONFDIR + +for signal in 0 1 2 3 13 14 15; do + trap 'kill `cat $sandbox/*.pid`; trap - $signal; kill -$signal $$' $signal +done + +# Create database and start ovsdb-server. +touch .conf.db.~lock~ +rm -f conf.db +run ovsdb-tool create conf.db "$srcdir"/vswitchd/vswitch.ovsschema +run ovsdb-server --detach --no-chdir --pidfile -vconsole:off --log-file \ + --remote=punix:"$sandbox"/db.sock + +# Start ovs-vswitchd. +run ovs-vswitchd --detach --no-chdir --pidfile -vconsole:off --log-file \ + --enable-dummy --disable-system -vvconn -vnetdev_dummy + +# Add bridges for Ryu to use, and configure them to connect to Ryu. +for config in \ + 'br0 0000000000000001 a c b d' \ + 'br1 0000000000000002 c a d b' +do + set $config + bridge=$1 dpid=$2 port1=$3 peer1=$4 port2=$5 peer2=$6 + run ovs-vsctl --no-wait \ + -- add-br $bridge \ + -- set bridge $bridge \ + datapath-type=dummy fail-mode=secure \ + protocols='[OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13]' \ + other-config:datapath-id=$dpid \ + -- set-controller $bridge tcp:127.0.0.1:6633 \ + -- set controller $bridge connection-mode=out-of-band \ + max-backoff=1000 \ + -- add-port $bridge $port1 \ + -- set interface $port1 ofport_request=1 type=patch options:peer=$peer1 \ + -- add-port $bridge $port2 \ + -- set interface $port2 ofport_request=2 type=patch options:peer=$peer2 +done + +# Run Ryu. +cd $RYUDIR +logs= +for app in \ + ryu/tests/switch/tester.py \ + ryu/tests/integrated/test_add_flow_v10.py \ + ryu/tests/integrated/test_request_reply_v12.py \ + ryu/tests/integrated/test_add_flow_v12_actions.py \ + ryu/tests/integrated/test_add_flow_v12_matches.py +do + cat < "$sandbox/ryu.pid" + i=0 + while sleep 1; do + if grep -q -E 'TEST_FINISHED|Test end|uncaught exception' "$logfile" \ + >/dev/null + then + break + fi + + i=`expr $i + 1` + if test $i -ge 600; then + echo "--- TIMEOUT after $i seconds" + break + fi + done + kill $pid + wait +done + +cat <