+Q: I have a sophisticated network setup involving Open vSwitch, VMs or
+ multiple hosts, and other components. The behavior isn't what I
+ expect. Help!
+
+A: To debug network behavior problems, trace the path of a packet,
+ hop-by-hop, from its origin in one host to a remote host. If
+ that's correct, then trace the path of the response packet back to
+ the origin.
+
+ Usually a simple ICMP echo request and reply ("ping") packet is
+ good enough. Start by initiating an ongoing "ping" from the origin
+ host to a remote host. If you are tracking down a connectivity
+ problem, the "ping" will not display any successful output, but
+ packets are still being sent. (In this case the packets being sent
+ are likely ARP rather than ICMP.)
+
+ Tools available for tracing include the following:
+
+ - "tcpdump" and "wireshark" for observing hops across network
+ devices, such as Open vSwitch internal devices and physical
+ wires.
+
+ - "ovs-appctl dpif/dump-flows <br>" in Open vSwitch 1.10 and
+ later or "ovs-dpctl dump-flows <br>" in earlier versions.
+ These tools allow one to observe the actions being taken on
+ packets in ongoing flows.
+
+ See ovs-vswitchd(8) for "ovs-appctl dpif/dump-flows"
+ documentation, ovs-dpctl(8) for "ovs-dpctl dump-flows"
+ documentation, and "Why are there so many different ways to
+ dump flows?" above for some background.
+
+ - "ovs-appctl ofproto/trace" to observe the logic behind how
+ ovs-vswitchd treats packets. See ovs-vswitchd(8) for
+ documentation. You can out more details about a given flow
+ that "ovs-dpctl dump-flows" displays, by cutting and pasting
+ a flow from the output into an "ovs-appctl ofproto/trace"
+ command.
+
+ - SPAN, RSPAN, and ERSPAN features of physical switches, to
+ observe what goes on at these physical hops.
+
+ Starting at the origin of a given packet, observe the packet at
+ each hop in turn. For example, in one plausible scenario, you
+ might:
+
+ 1. "tcpdump" the "eth" interface through which an ARP egresses
+ a VM, from inside the VM.
+
+ 2. "tcpdump" the "vif" or "tap" interface through which the ARP
+ ingresses the host machine.
+
+ 3. Use "ovs-dpctl dump-flows" to spot the ARP flow and observe
+ the host interface through which the ARP egresses the
+ physical machine. You may need to use "ovs-dpctl show" to
+ interpret the port numbers. If the output seems surprising,
+ you can use "ovs-appctl ofproto/trace" to observe details of
+ how ovs-vswitchd determined the actions in the "ovs-dpctl
+ dump-flows" output.
+
+ 4. "tcpdump" the "eth" interface through which the ARP egresses
+ the physical machine.
+
+ 5. "tcpdump" the "eth" interface through which the ARP
+ ingresses the physical machine, at the remote host that
+ receives the ARP.
+
+ 6. Use "ovs-dpctl dump-flows" to spot the ARP flow on the
+ remote host that receives the ARP and observe the VM "vif"
+ or "tap" interface to which the flow is directed. Again,
+ "ovs-dpctl show" and "ovs-appctl ofproto/trace" might help.
+
+ 7. "tcpdump" the "vif" or "tap" interface to which the ARP is
+ directed.
+
+ 8. "tcpdump" the "eth" interface through which the ARP
+ ingresses a VM, from inside the VM.
+
+ It is likely that during one of these steps you will figure out the
+ problem. If not, then follow the ARP reply back to the origin, in
+ reverse.
+
+Q: How do I make a flow drop packets?
+
+A: To drop a packet is to receive it without forwarding it. OpenFlow
+ explicitly specifies forwarding actions. Thus, a flow with an
+ empty set of actions does not forward packets anywhere, causing
+ them to be dropped. You can specify an empty set of actions with
+ "actions=" on the ovs-ofctl command line. For example:
+
+ ovs-ofctl add-flow br0 priority=65535,actions=
+
+ would cause every packet entering switch br0 to be dropped.
+
+ You can write "drop" explicitly if you like. The effect is the
+ same. Thus, the following command also causes every packet
+ entering switch br0 to be dropped:
+
+ ovs-ofctl add-flow br0 priority=65535,actions=drop
+
+ "drop" is not an action, either in OpenFlow or Open vSwitch.
+ Rather, it is only a way to say that there are no actions.
+
+Q: I added a flow to send packets out the ingress port, like this:
+
+ ovs-ofctl add-flow br0 in_port=2,actions=2
+
+ but OVS drops the packets instead.
+
+A: Yes, OpenFlow requires a switch to ignore attempts to send a packet
+ out its ingress port. The rationale is that dropping these packets
+ makes it harder to loop the network. Sometimes this behavior can
+ even be convenient, e.g. it is often the desired behavior in a flow
+ that forwards a packet to several ports ("floods" the packet).
+
+ Sometimes one really needs to send a packet out its ingress port.
+ In this case, output to OFPP_IN_PORT, which in ovs-ofctl syntax is
+ expressed as just "in_port", e.g.:
+
+ ovs-ofctl add-flow br0 in_port=2,actions=in_port
+
+ This also works in some circumstances where the flow doesn't match
+ on the input port. For example, if you know that your switch has
+ five ports numbered 2 through 6, then the following will send every
+ received packet out every port, even its ingress port:
+
+ ovs-ofctl add-flow br0 actions=2,3,4,5,6,in_port
+
+ or, equivalently:
+
+ ovs-ofctl add-flow br0 actions=all,in_port
+
+ Sometimes, in complicated flow tables with multiple levels of
+ "resubmit" actions, a flow needs to output to a particular port
+ that may or may not be the ingress port. It's difficult to take
+ advantage of OFPP_IN_PORT in this situation. To help, Open vSwitch
+ provides, as an OpenFlow extension, the ability to modify the
+ in_port field. Whatever value is currently in the in_port field is
+ the port to which outputs will be dropped, as well as the
+ destination for OFPP_IN_PORT. This means that the following will
+ reliably output to port 2 or to ports 2 through 6, respectively:
+
+ ovs-ofctl add-flow br0 in_port=2,actions=load:0->NXM_OF_IN_PORT[],2
+ ovs-ofctl add-flow br0 actions=load:0->NXM_OF_IN_PORT[],2,3,4,5,6
+
+ If the input port is important, then one may save and restore it on
+ the stack:
+
+ ovs-ofctl add-flow br0 actions=push:NXM_OF_IN_PORT[],\
+ load:0->NXM_OF_IN_PORT[],\
+ 2,3,4,5,6,\
+ pop:NXM_OF_IN_PORT[]
+
+Q: My bridge br0 has host 192.168.0.1 on port 1 and host 192.168.0.2
+ on port 2. I set up flows to forward only traffic destined to the
+ other host and drop other traffic, like this:
+
+ priority=5,in_port=1,ip,nw_dst=192.168.0.2,actions=2
+ priority=5,in_port=2,ip,nw_dst=192.168.0.1,actions=1
+ priority=0,actions=drop
+
+ But it doesn't work--I don't get any connectivity when I do this.
+ Why?
+
+A: These flows drop the ARP packets that IP hosts use to establish IP
+ connectivity over Ethernet. To solve the problem, add flows to
+ allow ARP to pass between the hosts:
+
+ priority=5,in_port=1,arp,actions=2
+ priority=5,in_port=2,arp,actions=1
+
+ This issue can manifest other ways, too. The following flows that
+ match on Ethernet addresses instead of IP addresses will also drop
+ ARP packets, because ARP requests are broadcast instead of being
+ directed to a specific host:
+
+ priority=5,in_port=1,dl_dst=54:00:00:00:00:02,actions=2
+ priority=5,in_port=2,dl_dst=54:00:00:00:00:01,actions=1
+ priority=0,actions=drop
+
+ The solution already described above will also work in this case.
+ It may be better to add flows to allow all multicast and broadcast
+ traffic:
+
+ priority=5,in_port=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=2
+ priority=5,in_port=2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=1
+