--- /dev/null
+\documentstyle[12pt,twoside]{article}
+\def\TITLE{Tunnels over IP}
+\input preamble
+\begin{center}
+\Large\bf Tunnels over IP in Linux-2.2
+\end{center}
+
+
+\begin{center}
+{ \large Alexey~N.~Kuznetsov } \\
+\em Institute for Nuclear Research, Moscow \\
+\verb|kuznet@ms2.inr.ac.ru| \\
+\rm March 17, 1999
+\end{center}
+
+\vspace{5mm}
+
+\tableofcontents
+
+
+\section{Instead of introduction: micro-FAQ.}
+
+\begin{itemize}
+
+\item
+Q: In linux-2.0.36 I used:
+\begin{verbatim}
+ ifconfig tunl1 10.0.0.1 pointopoint 193.233.7.65
+\end{verbatim}
+to create tunnel. It does not work in 2.2.0!
+
+A: You are right, it does not work. The command written above is split to two commands.
+\begin{verbatim}
+ ip tunnel add MY-TUNNEL mode ipip remote 193.233.7.65
+\end{verbatim}
+will create tunnel device with name \verb|MY-TUNNEL|. Now you may configure
+it with:
+\begin{verbatim}
+ ifconfig MY-TUNNEL 10.0.0.1
+\end{verbatim}
+Certainly, if you prefer name \verb|tunl1| to \verb|MY-TUNNEL|,
+you still may use it.
+
+\item
+Q: In linux-2.0.36 I used:
+\begin{verbatim}
+ ifconfig tunl0 10.0.0.1
+ route add -net 10.0.0.0 gw 193.233.7.65 dev tunl0
+\end{verbatim}
+to tunnel net 10.0.0.0 via router 193.233.7.65. It does not
+work in 2.2.0! Moreover, \verb|route| prints a funny error sort of
+``network unreachable'' and after this I found a strange direct route
+to 10.0.0.0 via \verb|tunl0| in routing table.
+
+A: Yes, in 2.2 the rule that {\em normal} gateway must reside on directly
+connected network has not any exceptions. You may tell kernel, that
+this particular route is {\em abnormal}:
+\begin{verbatim}
+ ifconfig tunl0 10.0.0.1 netmask 255.255.255.255
+ ip route add 10.0.0.0/8 via 193.233.7.65 dev tunl0 onlink
+\end{verbatim}
+Note keyword \verb|onlink|, it is the magic key that orders kernel
+not to check for consistency of gateway address.
+Probably, after this explanation you have already guessed another method
+to cheat kernel:
+\begin{verbatim}
+ ifconfig tunl0 10.0.0.1 netmask 255.255.255.255
+ route add -host 193.233.7.65 dev tunl0
+ route add -net 10.0.0.0 netmask 255.0.0.0 gw 193.233.7.65
+ route del -host 193.233.7.65 dev tunl0
+\end{verbatim}
+Well, if you like such tricks, nobody may prohibit you to use them.
+Only do not forget
+that between \verb|route add| and \verb|route del| host 193.233.7.65 is
+unreachable.
+
+\item
+Q: In 2.0.36 I used to load \verb|tunnel| device module and \verb|ipip| module.
+I cannot find any \verb|tunnel| in 2.2!
+
+A: Linux-2.2 has single module \verb|ipip| for both directions of tunneling
+and for all IPIP tunnel devices.
+
+\item
+Q: \verb|traceroute| does not work over tunnel! Well, stop... It works,
+ only skips some number of hops.
+
+A: Yes. By default tunnel driver copies \verb|ttl| value from
+inner packet to outer one. It means that path traversed by tunneled
+packets to another endpoint is not hidden. If you dislike this, or if you
+are going to use some routing protocol expecting that packets
+with ttl 1 will reach peering host (f.e.\ RIP, OSPF or EBGP)
+and you are not afraid of
+tunnel loops, you may append option \verb|ttl 64|, when creating tunnel
+with \verb|ip tunnel add|.
+
+\item
+Q: ... Well, list of things, which 2.0 was able to do finishes.
+
+\end{itemize}
+
+\paragraph{Summary of differences between 2.2 and 2.0.}
+
+\begin{itemize}
+
+\item {\bf In 2.0} you could compile tunnel device into kernel
+ and got set of 4 devices \verb|tunl0| ... \verb|tunl3| or,
+ alternatively, compile it as module and load new module
+ for each new tunnel. Also, module \verb|ipip| was necessary
+ to receive tunneled packets.
+
+ {\bf 2.2} has {\em one\/} module \verb|ipip|. Loading it you get base
+ tunnel device \verb|tunl0| and another tunnels may be created with command
+ \verb|ip tunnel add|. These new devices may have arbitrary names.
+
+
+\item {\bf In 2.0} you set remote tunnel endpoint address with
+ the command \verb|ifconfig| ... \verb|pointopoint A|.
+
+ {\bf In 2.2} this command has the same semantics on all
+ the interfaces, namely it sets not tunnel endpoint,
+ but address of peering host, which is directly reachable
+ via this tunnel,
+ rather than via Internet. Actual tunnel endpoint address \verb|A|
+ should be set with \verb|ip tunnel add ... remote A|.
+
+\item {\bf In 2.0} you create tunnel routes with the command:
+\begin{verbatim}
+ route add -net 10.0.0.0 gw A dev tunl0
+\end{verbatim}
+
+ {\bf 2.2} interprets this command equally for all device
+ kinds and gateway is required to be directly reachable via this tunnel,
+ rather than via Internet. You still may use \verb|ip route add ... onlink|
+ to override this behaviour.
+
+\end{itemize}
+
+
+\section{Tunnel setup: basics}
+
+Standard Linux-2.2 kernel supports three flavor of tunnels,
+listed in the following table:
+\vspace{2mm}
+
+\begin{tabular}{lll}
+\vrule depth 0.8ex width 0pt\relax
+Mode & Description & Base device \\
+ipip & IP over IP & tunl0 \\
+sit & IPv6 over IP & sit0 \\
+gre & ANY over GRE over IP & gre0
+\end{tabular}
+
+\vspace{2mm}
+
+\noindent All the kinds of tunnels are created with one command:
+\begin{verbatim}
+ ip tunnel add <NAME> mode <MODE> [ local <S> ] [ remote <D> ]
+\end{verbatim}
+
+This command creates new tunnel device with name \verb|<NAME>|.
+The \verb|<NAME>| is an arbitrary string. Particularly,
+it may be even \verb|eth0|. The rest of parameters set
+different tunnel characteristics.
+
+\begin{itemize}
+
+\item
+\verb|mode <MODE>| sets tunnel mode. Three modes are available now
+ \verb|ipip|, \verb|sit| and \verb|gre|.
+
+\item
+\verb|remote <D>| sets remote endpoint of the tunnel to IP
+ address \verb|<D>|.
+\item
+\verb|local <S>| sets fixed local address for tunneled
+ packets. It must be an address on another interface of this host.
+
+\end{itemize}
+
+\let\thefootnote\oldthefootnote
+
+Both \verb|remote| and \verb|local| may be omitted. In this case we
+say that they are zero or wildcard. Two tunnels of one mode cannot
+have the same \verb|remote| and \verb|local|. Particularly it means
+that base device or fallback tunnel cannot be replicated.\footnote{
+This restriction is relaxed for keyed GRE tunnels.}
+
+Tunnels are divided to two classes: {\bf pointopoint} tunnels, which
+have some not wildcard \verb|remote| address and deliver all the packets
+to this destination, and {\bf NBMA} (i.e. Non-Broadcast Multi-Access) tunnels,
+which have no \verb|remote|. Particularly, base devices (f.e.\ \verb|tunl0|)
+are NBMA, because they have neither \verb|remote| nor
+\verb|local| addresses.
+
+
+After tunnel device is created you should configure it as you did
+it with another devices. Certainly, the configuration of tunnels has
+some features related to the fact that they work over existing Internet
+routing infrastructure and simultaneously create new virtual links,
+which changes this infrastructure. The danger that not enough careful
+tunnel setup will result in formation of tunnel loops,
+collapse of routing or flooding network with exponentially
+growing number of tunneled fragments is very real.
+
+
+Protocol setup on pointopoint tunnels does not differ of configuration
+of another devices. You should set a protocol address with \verb|ifconfig|
+and add routes with \verb|route| utility.
+
+NBMA tunnels are different. To route something via NBMA tunnel
+you have to explain to driver, where it should deliver packets to.
+The only way to make it is to create special routes with gateway
+address pointing to desired endpoint. F.e.\
+\begin{verbatim}
+ ip route add 10.0.0.0/24 via <A> dev tunl0 onlink
+\end{verbatim}
+It is important to use option \verb|onlink|, otherwise
+kernel will refuse request to create route via gateway not directly
+reachable over device \verb|tunl0|. With IPv6 the situation is much simpler:
+when you start device \verb|sit0|, it automatically configures itself
+with all IPv4 addresses mapped to IPv6 space, so that all IPv4
+Internet is {\em really reachable} via \verb|sit0|! Excellent, the command
+\begin{verbatim}
+ ip route add 3FFE::/16 via ::193.233.7.65 dev sit0
+\end{verbatim}
+will route \verb|3FFE::/16| via \verb|sit0|, sending all the packets
+destined to this prefix to 193.233.7.65.
+
+\section{Tunnel setup: options}
+
+Command \verb|ip tunnel add| has several additional options.
+\begin{itemize}
+
+\item \verb|ttl N| --- set fixed TTL \verb|N| on tunneled packets.
+ \verb|N| is number in the range 1--255. 0 is special value,
+ meaning that packets inherit TTL value.
+ Default value is: \verb|inherit|.
+
+\item \verb|tos T| --- set fixed tos \verb|T| on tunneled packets.
+ Default value is: \verb|inherit|.
+
+\item \verb|dev DEV| --- bind tunnel to device \verb|DEV|, so that
+ tunneled packets will be routed only via this device and will
+ not be able to escape to another device, when route to endpoint changes.
+
+\item \verb|nopmtudisc| --- disable Path MTU Discovery on this tunnel.
+ It is enabled by default. Note that fixed ttl is incompatible
+ with this option: tunnels with fixed ttl always make pmtu discovery.
+
+\end{itemize}
+
+\verb|ipip| and \verb|sit| tunnels have no more options. \verb|gre|
+tunnels are more complicated:
+
+\begin{itemize}
+
+\item \verb|key K| --- use keyed GRE with key \verb|K|. \verb|K| is
+ either number or IP address-like dotted quad.
+
+\item \verb|csum| --- checksum tunneled packets.
+
+\item \verb|seq| --- serialize packets.
+\begin{NB}
+ I think this option does not
+ work. At least, I did not test it, did not debug it and
+ even do not understand, how it is supposed to work and for what
+ purpose Cisco planned to use it.
+\end{NB}
+
+\end{itemize}
+
+
+Actually, these GRE options can be set separately for input and
+output directions by prefixing corresponding keywords with letter
+\verb|i| or \verb|o|. F.e.\ \verb|icsum| orders to accept only
+packets with correct checksum and \verb|ocsum| means, that
+our host will calculate and send checksum.
+
+Command \verb|ip tunnel add| is not the only operation,
+which can be made with tunnels. Certainly, you may get short help page
+with:
+\begin{verbatim}
+ ip tunnel help
+\end{verbatim}
+
+Besides that, you may view list of installed tunnels with the help of command:
+\begin{verbatim}
+ ip tunnel ls
+\end{verbatim}
+Also you may look at statistics:
+\begin{verbatim}
+ ip -s tunnel ls Cisco
+\end{verbatim}
+where \verb|Cisco| is name of tunnel device. Command
+\begin{verbatim}
+ ip tunnel del Cisco
+\end{verbatim}
+destroys tunnel \verb|Cisco|. And, finally,
+\begin{verbatim}
+ ip tunnel change Cisco mode sit local ME remote HE ttl 32
+\end{verbatim}
+changes its parameters.
+
+\section{Differences 2.2 and 2.0 tunnels revisited.}
+
+Now we can discuss more subtle differences between tunneling in 2.0
+and 2.2.
+
+\begin{itemize}
+
+\item In 2.0 all tunneled packets were received promiscuously
+as soon as you loaded module \verb|ipip|. 2.2 tries to select the best
+tunnel device and packet looks as received on this. F.e.\ if host
+received \verb|ipip| packet from host \verb|D| destined to our
+local address \verb|S|, kernel searches for matching tunnels
+in order:
+
+\begin{tabular}{ll}
+1 & \verb|remote| is \verb|D| and \verb|local| is \verb|S| \\
+2 & \verb|remote| is \verb|D| and \verb|local| is wildcard \\
+3 & \verb|remote| is wildcard and \verb|local| is \verb|S| \\
+4 & \verb|tunl0|
+\end{tabular}
+
+If tunnel exists, but it is not in \verb|UP| state, the tunnel is ignored.
+Note, that if \verb|tunl0| is \verb|UP| it receives all the IPIP packets,
+not acknowledged by more specific tunnels.
+Be careful, it means that without carefully installed firewall rules
+anyone on the Internet may inject to your network any packets with
+source addresses indistinguishable from local ones. It is not so bad idea
+to design tunnels in the way enforcing maximal route symmetry
+and to enable reversed path filter (\verb|rp_filter| sysctl option) on
+tunnel devices.
+
+\item In 2.2 you can monitor and debug tunnels with \verb|tcpdump|.
+F.e.\ \verb|tcpdump| \verb|-i Cisco| \verb|-nvv| will dump packets,
+which kernel output, via tunnel \verb|Cisco| and the packets received on it
+from kernel viewpoint.
+
+\end{itemize}
+
+
+\section{Linux and Cisco IOS tunnels.}
+
+Among another tunnels Cisco IOS supports IPIP and GRE.
+Essentially, Cisco setup is subset of options, available for Linux.
+Let us consider the simplest example:
+
+\begin{verbatim}
+interface Tunnel0
+ tunnel mode gre ip
+ tunnel source 10.10.14.1
+ tunnel destination 10.10.13.2
+\end{verbatim}
+
+
+This command set translates to:
+
+\begin{verbatim}
+ ip tunnel add Tunnel0 \
+ mode gre \
+ local 10.10.14.1 \
+ remote 10.10.13.2
+\end{verbatim}
+
+Any questions? No questions.
+
+\section{Interaction IPIP tunnels and DVMRP.}
+
+DVMRP exploits IPIP tunnels to route multicasts via Internet.
+\verb|mrouted| creates
+IPIP tunnels listed in its configuration file automatically.
+From kernel and user viewpoints there are no differences between
+tunnels, created in this way, and tunnels created by \verb|ip tunnel|.
+I.e.\ if \verb|mrouted| created some tunnel, it may be used to
+route unicast packets, provided appropriate routes are added.
+And vice versa, if administrator has already created a tunnel,
+it will be reused by \verb|mrouted|, if it requests DVMRP
+tunnel with the same local and remote addresses.
+
+Do not wonder, if your manually configured tunnel is
+destroyed, when mrouted exits.
+
+
+\section{Broadcast GRE ``tunnels''.}
+
+It is possible to set \verb|remote| for GRE tunnel to a multicast
+address. Such tunnel becomes {\bf broadcast} tunnel (though word
+tunnel is not quite appropriate in this case, it is rather virtual network).
+\begin{verbatim}
+ ip tunnel add Universe local 193.233.7.65 \
+ remote 224.66.66.66 ttl 16
+ ip addr add 10.0.0.1/16 dev Universe
+ ip link set Universe up
+\end{verbatim}
+This tunnel is true broadcast network and broadcast packets are
+sent to multicast group 224.66.66.66. By default such tunnel starts
+to resolve both IP and IPv6 addresses via ARP/NDISC, so that
+if multicast routing is supported in surrounding network, all GRE nodes
+will find one another automatically and will form virtual Ethernet-like
+broadcast network. If multicast routing does not work, it is unpleasant
+but not fatal flaw. The tunnel becomes NBMA rather than broadcast network.
+You may disable dynamic ARPing by:
+\begin{verbatim}
+ echo 0 > /proc/sys/net/ipv4/neigh/Universe/mcast_solicit
+\end{verbatim}
+and to add required information to ARP tables manually:
+\begin{verbatim}
+ ip neigh add 10.0.0.2 lladdr 128.6.190.2 dev Universe nud permanent
+\end{verbatim}
+In this case packets sent to 10.0.0.2 will be encapsulated in GRE
+and sent to 128.6.190.2. It is possible to facilitate address resolution
+using methods typical for another NBMA networks f.e.\ to start user
+level \verb|arpd| daemon, which will maintain database of hosts attached
+to GRE virtual network or ask for information
+dedicated ARP or NHRP server.
+
+
+Actually, such setup is the most natural for tunneling,
+it is really flexible, scalable and easily managable, so that
+it is strongly recommended to be used with GRE tunnels instead of ugly
+hack with NBMA mode and \verb|onlink| modifier. Unfortunately,
+by historical reasons broadcast mode is not supported by IPIP tunnels,
+but this probably will change in future.
+
+
+
+\section{Traffic control issues.}
+
+Tunnels are devices, hence all the power of Linux traffic control
+applies to them. The simplest (and the most useful in practice)
+example is limiting tunnel bandwidth. The following command:
+\begin{verbatim}
+ tc qdisc add dev tunl0 root tbf \
+ rate 128Kbit burst 4K limit 10K
+\end{verbatim}
+will limit tunneled traffic to 128Kbit with maximal burst size of 4K
+and queuing not more than 10K.
+
+However, you should remember, that tunnels are {\em virtual} devices
+implemented in software and true queue management is impossible for them
+just because they have no queues. Instead, it is better to create classes
+on real physical interfaces and to map tunneled packets to them.
+In general case of dynamic routing you should create such classes
+on all outgoing interfaces, or, alternatively,
+to use option \verb|dev DEV| to bind tunnel to a fixed physical device.
+In the last case packets will be routed only via specified device
+and you need to setup corresponding classes only on it.
+Though you have to pay for this convenience,
+if routing will change, your tunnel will fail.
+
+Suppose that CBQ class \verb|1:ABC| has been created on device \verb|eth0|
+specially for tunnel \verb|Cisco| with endpoints \verb|S| and \verb|D|.
+Now you can select IPIP packets with addresses \verb|S| and \verb|D|
+with some classifier and map them to class \verb|1:ABC|. F.e.\
+it is easy to make with \verb|rsvp| classifier:
+\begin{verbatim}
+ tc filter add dev eth0 pref 100 proto ip rsvp \
+ session D ipproto ipip filter S \
+ classid 1:ABC
+\end{verbatim}
+
+If you want to make more detailed classification of sub-flows
+transmitted via tunnel, you can build CBQ subtree,
+rooted at \verb|1:ABC| and attach to subroot set of rules parsing
+IPIP packets more deeply.
+
+\end{document}
git://git.onelab.eu / iproute2.git / blobdiff