- Linux Ethernet Bonding Driver mini-howto
+ Linux Ethernet Bonding Driver HOWTO
+
+ Latest update: 24 April 2006
Initial release : Thomas Davis <tadavis at lbl.gov>
Corrections, HA extensions : 2000/10/03-15 :
- Janice Girouard <girouard at us dot ibm dot com>
- Jay Vosburgh <fubar at us dot ibm dot com>
-Note :
-------
-The bonding driver originally came from Donald Becker's beowulf patches for
-kernel 2.0. It has changed quite a bit since, and the original tools from
-extreme-linux and beowulf sites will not work with this version of the driver.
+Reorganized and updated Feb 2005 by Jay Vosburgh
+Added Sysfs information: 2006/04/24
+ - Mitch Williams <mitch.a.williams at intel.com>
+
+Introduction
+============
-For new versions of the driver, patches for older kernels and the updated
-userspace tools, please follow the links at the end of this file.
+ The Linux bonding driver provides a method for aggregating
+multiple network interfaces into a single logical "bonded" interface.
+The behavior of the bonded interfaces depends upon the mode; generally
+speaking, modes provide either hot standby or load balancing services.
+Additionally, link integrity monitoring may be performed.
+
+ The bonding driver originally came from Donald Becker's
+beowulf patches for kernel 2.0. It has changed quite a bit since, and
+the original tools from extreme-linux and beowulf sites will not work
+with this version of the driver.
+ For new versions of the driver, updated userspace tools, and
+who to ask for help, please follow the links at the end of this file.
Table of Contents
=================
-Installation
-Bond Configuration
-Module Parameters
-Configuring Multiple Bonds
-Switch Configuration
-Verifying Bond Configuration
-Frequently Asked Questions
-High Availability
-Promiscuous Sniffing notes
-8021q VLAN support
-Limitations
-Resources and Links
-
-
-Installation
-============
+1. Bonding Driver Installation
-1) Build kernel with the bonding driver
----------------------------------------
-For the latest version of the bonding driver, use kernel 2.4.12 or above
-(otherwise you will need to apply a patch).
+2. Bonding Driver Options
-Configure kernel with `make menuconfig/xconfig/config', and select "Bonding
-driver support" in the "Network device support" section. It is recommended
-to configure the driver as module since it is currently the only way to
-pass parameters to the driver and configure more than one bonding device.
+3. Configuring Bonding Devices
+3.1 Configuration with Sysconfig Support
+3.1.1 Using DHCP with Sysconfig
+3.1.2 Configuring Multiple Bonds with Sysconfig
+3.2 Configuration with Initscripts Support
+3.2.1 Using DHCP with Initscripts
+3.2.2 Configuring Multiple Bonds with Initscripts
+3.3 Configuring Bonding Manually with Ifenslave
+3.3.1 Configuring Multiple Bonds Manually
+3.4 Configuring Bonding Manually via Sysfs
-Build and install the new kernel and modules.
+4. Querying Bonding Configuration
+4.1 Bonding Configuration
+4.2 Network Configuration
-2) Get and install the userspace tools
---------------------------------------
-This version of the bonding driver requires updated ifenslave program. The
-original one from extreme-linux and beowulf will not work. Kernels 2.4.12
-and above include the updated version of ifenslave.c in
-Documentation/networking directory. For older kernels, please follow the
-links at the end of this file.
+5. Switch Configuration
-IMPORTANT!!! If you are running on Redhat 7.1 or greater, you need
-to be careful because /usr/include/linux is no longer a symbolic link
-to /usr/src/linux/include/linux. If you build ifenslave while this is
-true, ifenslave will appear to succeed but your bond won't work. The purpose
-of the -I option on the ifenslave compile line is to make sure it uses
-/usr/src/linux/include/linux/if_bonding.h instead of the version from
-/usr/include/linux.
+6. 802.1q VLAN Support
-To install ifenslave.c, do:
- # gcc -Wall -Wstrict-prototypes -O -I/usr/src/linux/include ifenslave.c -o ifenslave
- # cp ifenslave /sbin/ifenslave
+7. Link Monitoring
+7.1 ARP Monitor Operation
+7.2 Configuring Multiple ARP Targets
+7.3 MII Monitor Operation
+8. Potential Trouble Sources
+8.1 Adventures in Routing
+8.2 Ethernet Device Renaming
+8.3 Painfully Slow Or No Failed Link Detection By Miimon
-Bond Configuration
-==================
+9. SNMP agents
-You will need to add at least the following line to /etc/modprobe.conf
-so the bonding driver will automatically load when the bond0 interface is
-configured. Refer to the modprobe.conf manual page for specific modprobe.conf
-syntax details. The Module Parameters section of this document describes each
-bonding driver parameter.
+10. Promiscuous mode
- alias bond0 bonding
+11. Configuring Bonding for High Availability
+11.1 High Availability in a Single Switch Topology
+11.2 High Availability in a Multiple Switch Topology
+11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+11.2.2 HA Link Monitoring for Multiple Switch Topology
-Use standard distribution techniques to define the bond0 network interface. For
-example, on modern Red Hat distributions, create an ifcfg-bond0 file in
-the /etc/sysconfig/network-scripts directory that resembles the following:
+12. Configuring Bonding for Maximum Throughput
+12.1 Maximum Throughput in a Single Switch Topology
+12.1.1 MT Bonding Mode Selection for Single Switch Topology
+12.1.2 MT Link Monitoring for Single Switch Topology
+12.2 Maximum Throughput in a Multiple Switch Topology
+12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
+12.2.2 MT Link Monitoring for Multiple Switch Topology
+
+13. Switch Behavior Issues
+13.1 Link Establishment and Failover Delays
+13.2 Duplicated Incoming Packets
-DEVICE=bond0
-IPADDR=192.168.1.1
-NETMASK=255.255.255.0
-NETWORK=192.168.1.0
-BROADCAST=192.168.1.255
-ONBOOT=yes
-BOOTPROTO=none
-USERCTL=no
+14. Hardware Specific Considerations
+14.1 IBM BladeCenter
-(use appropriate values for your network above)
+15. Frequently Asked Questions
-All interfaces that are part of a bond should have SLAVE and MASTER
-definitions. For example, in the case of Red Hat, if you wish to make eth0 and
-eth1 a part of the bonding interface bond0, their config files (ifcfg-eth0 and
-ifcfg-eth1) should resemble the following:
+16. Resources and Links
-DEVICE=eth0
-USERCTL=no
-ONBOOT=yes
-MASTER=bond0
-SLAVE=yes
-BOOTPROTO=none
-Use DEVICE=eth1 in the ifcfg-eth1 config file. If you configure a second
-bonding interface (bond1), use MASTER=bond1 in the config file to make the
-network interface be a slave of bond1.
+1. Bonding Driver Installation
+==============================
-Restart the networking subsystem or just bring up the bonding device if your
-administration tools allow it. Otherwise, reboot. On Red Hat distros you can
-issue `ifup bond0' or `/etc/rc.d/init.d/network restart'.
+ Most popular distro kernels ship with the bonding driver
+already available as a module and the ifenslave user level control
+program installed and ready for use. If your distro does not, or you
+have need to compile bonding from source (e.g., configuring and
+installing a mainline kernel from kernel.org), you'll need to perform
+the following steps:
-If the administration tools of your distribution do not support
-master/slave notation in configuring network interfaces, you will need to
-manually configure the bonding device with the following commands:
+1.1 Configure and build the kernel with bonding
+-----------------------------------------------
- # /sbin/ifconfig bond0 192.168.1.1 netmask 255.255.255.0 \
- broadcast 192.168.1.255 up
+ The current version of the bonding driver is available in the
+drivers/net/bonding subdirectory of the most recent kernel source
+(which is available on http://kernel.org). Most users "rolling their
+own" will want to use the most recent kernel from kernel.org.
- # /sbin/ifenslave bond0 eth0
- # /sbin/ifenslave bond0 eth1
+ Configure kernel with "make menuconfig" (or "make xconfig" or
+"make config"), then select "Bonding driver support" in the "Network
+device support" section. It is recommended that you configure the
+driver as module since it is currently the only way to pass parameters
+to the driver or configure more than one bonding device.
-(use appropriate values for your network above)
+ Build and install the new kernel and modules, then continue
+below to install ifenslave.
-You can then create a script containing these commands and place it in the
-appropriate rc directory.
+1.2 Install ifenslave Control Utility
+-------------------------------------
-If you specifically need all network drivers loaded before the bonding driver,
-adding the following line to modprobe.conf will cause the network driver for
-eth0 and eth1 to be loaded before the bonding driver.
+ The ifenslave user level control program is included in the
+kernel source tree, in the file Documentation/networking/ifenslave.c.
+It is generally recommended that you use the ifenslave that
+corresponds to the kernel that you are using (either from the same
+source tree or supplied with the distro), however, ifenslave
+executables from older kernels should function (but features newer
+than the ifenslave release are not supported). Running an ifenslave
+that is newer than the kernel is not supported, and may or may not
+work.
+
+ To install ifenslave, do the following:
+
+# gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave
+# cp ifenslave /sbin/ifenslave
-install bond0 /sbin/modprobe -a eth0 eth1 && /sbin/modprobe bonding
+ If your kernel source is not in "/usr/src/linux," then replace
+"/usr/src/linux/include" in the above with the location of your kernel
+source include directory.
-Be careful not to reference bond0 itself at the end of the line, or modprobe
-will die in an endless recursive loop.
+ You may wish to back up any existing /sbin/ifenslave, or, for
+testing or informal use, tag the ifenslave to the kernel version
+(e.g., name the ifenslave executable /sbin/ifenslave-2.6.10).
-If running SNMP agents, the bonding driver should be loaded before any network
-drivers participating in a bond. This requirement is due to the the interface
-index (ipAdEntIfIndex) being associated to the first interface found with a
-given IP address. That is, there is only one ipAdEntIfIndex for each IP
-address. For example, if eth0 and eth1 are slaves of bond0 and the driver for
-eth0 is loaded before the bonding driver, the interface for the IP address
-will be associated with the eth0 interface. This configuration is shown below,
-the IP address 192.168.1.1 has an interface index of 2 which indexes to eth0
-in the ifDescr table (ifDescr.2).
+IMPORTANT NOTE:
- interfaces.ifTable.ifEntry.ifDescr.1 = lo
- interfaces.ifTable.ifEntry.ifDescr.2 = eth0
- interfaces.ifTable.ifEntry.ifDescr.3 = eth1
- interfaces.ifTable.ifEntry.ifDescr.4 = eth2
- interfaces.ifTable.ifEntry.ifDescr.5 = eth3
- interfaces.ifTable.ifEntry.ifDescr.6 = bond0
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
-
-This problem is avoided by loading the bonding driver before any network
-drivers participating in a bond. Below is an example of loading the bonding
-driver first, the IP address 192.168.1.1 is correctly associated with
-ifDescr.2.
+ If you omit the "-I" or specify an incorrect directory, you
+may end up with an ifenslave that is incompatible with the kernel
+you're trying to build it for. Some distros (e.g., Red Hat from 7.1
+onwards) do not have /usr/include/linux symbolically linked to the
+default kernel source include directory.
- interfaces.ifTable.ifEntry.ifDescr.1 = lo
- interfaces.ifTable.ifEntry.ifDescr.2 = bond0
- interfaces.ifTable.ifEntry.ifDescr.3 = eth0
- interfaces.ifTable.ifEntry.ifDescr.4 = eth1
- interfaces.ifTable.ifEntry.ifDescr.5 = eth2
- interfaces.ifTable.ifEntry.ifDescr.6 = eth3
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
- ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
+SECOND IMPORTANT NOTE:
+ If you plan to configure bonding using sysfs, you do not need
+to use ifenslave.
-While some distributions may not report the interface name in ifDescr,
-the association between the IP address and IfIndex remains and SNMP
-functions such as Interface_Scan_Next will report that association.
+2. Bonding Driver Options
+=========================
+ Options for the bonding driver are supplied as parameters to
+the bonding module at load time. They may be given as command line
+arguments to the insmod or modprobe command, but are usually specified
+in either the /etc/modules.conf or /etc/modprobe.conf configuration
+file, or in a distro-specific configuration file (some of which are
+detailed in the next section).
+
+ The available bonding driver parameters are listed below. If a
+parameter is not specified the default value is used. When initially
+configuring a bond, it is recommended "tail -f /var/log/messages" be
+run in a separate window to watch for bonding driver error messages.
-Module Parameters
-=================
+ It is critical that either the miimon or arp_interval and
+arp_ip_target parameters be specified, otherwise serious network
+degradation will occur during link failures. Very few devices do not
+support at least miimon, so there is really no reason not to use it.
-Optional parameters for the bonding driver can be supplied as command line
-arguments to the insmod command. Typically, these parameters are specified in
-the file /etc/modprobe.conf (see the manual page for modprobe.conf). The
-available bonding driver parameters are listed below. If a parameter is not
-specified the default value is used. When initially configuring a bond, it
-is recommended "tail -f /var/log/messages" be run in a separate window to
-watch for bonding driver error messages.
+ Options with textual values will accept either the text name
+or, for backwards compatibility, the option value. E.g.,
+"mode=802.3ad" and "mode=4" set the same mode.
-It is critical that either the miimon or arp_interval and arp_ip_target
-parameters be specified, otherwise serious network degradation will occur
-during link failures.
+ The parameters are as follows:
arp_interval
- Specifies the ARP monitoring frequency in milli-seconds.
- If ARP monitoring is used in a load-balancing mode (mode 0 or 2), the
- switch should be configured in a mode that evenly distributes packets
- across all links - such as round-robin. If the switch is configured to
- distribute the packets in an XOR fashion, all replies from the ARP
- targets will be received on the same link which could cause the other
- team members to fail. ARP monitoring should not be used in conjunction
- with miimon. A value of 0 disables ARP monitoring. The default value
- is 0.
+ Specifies the ARP link monitoring frequency in milliseconds.
+
+ The ARP monitor works by periodically checking the slave
+ devices to determine whether they have sent or received
+ traffic recently (the precise criteria depends upon the
+ bonding mode, and the state of the slave). Regular traffic is
+ generated via ARP probes issued for the addresses specified by
+ the arp_ip_target option.
+
+ This behavior can be modified by the arp_validate option,
+ below.
+
+ If ARP monitoring is used in an etherchannel compatible mode
+ (modes 0 and 2), the switch should be configured in a mode
+ that evenly distributes packets across all links. If the
+ switch is configured to distribute the packets in an XOR
+ fashion, all replies from the ARP targets will be received on
+ the same link which could cause the other team members to
+ fail. ARP monitoring should not be used in conjunction with
+ miimon. A value of 0 disables ARP monitoring. The default
+ value is 0.
arp_ip_target
- Specifies the ip addresses to use when arp_interval is > 0. These
- are the targets of the ARP request sent to determine the health of
- the link to the targets. Specify these values in ddd.ddd.ddd.ddd
- format. Multiple ip adresses must be seperated by a comma. At least
- one ip address needs to be given for ARP monitoring to work. The
- maximum number of targets that can be specified is set at 16.
+ Specifies the IP addresses to use as ARP monitoring peers when
+ arp_interval is > 0. These are the targets of the ARP request
+ sent to determine the health of the link to the targets.
+ Specify these values in ddd.ddd.ddd.ddd format. Multiple IP
+ addresses must be separated by a comma. At least one IP
+ address must be given for ARP monitoring to function. The
+ maximum number of targets that can be specified is 16. The
+ default value is no IP addresses.
+
+arp_validate
+
+ Specifies whether or not ARP probes and replies should be
+ validated in the active-backup mode. This causes the ARP
+ monitor to examine the incoming ARP requests and replies, and
+ only consider a slave to be up if it is receiving the
+ appropriate ARP traffic.
+
+ Possible values are:
+
+ none or 0
+
+ No validation is performed. This is the default.
+
+ active or 1
+
+ Validation is performed only for the active slave.
+
+ backup or 2
+
+ Validation is performed only for backup slaves.
+
+ all or 3
+
+ Validation is performed for all slaves.
+
+ For the active slave, the validation checks ARP replies to
+ confirm that they were generated by an arp_ip_target. Since
+ backup slaves do not typically receive these replies, the
+ validation performed for backup slaves is on the ARP request
+ sent out via the active slave. It is possible that some
+ switch or network configurations may result in situations
+ wherein the backup slaves do not receive the ARP requests; in
+ such a situation, validation of backup slaves must be
+ disabled.
+
+ This option is useful in network configurations in which
+ multiple bonding hosts are concurrently issuing ARPs to one or
+ more targets beyond a common switch. Should the link between
+ the switch and target fail (but not the switch itself), the
+ probe traffic generated by the multiple bonding instances will
+ fool the standard ARP monitor into considering the links as
+ still up. Use of the arp_validate option can resolve this, as
+ the ARP monitor will only consider ARP requests and replies
+ associated with its own instance of bonding.
+
+ This option was added in bonding version 3.1.0.
downdelay
- Specifies the delay time in milli-seconds to disable a link after a
- link failure has been detected. This should be a multiple of miimon
- value, otherwise the value will be rounded. The default value is 0.
+ Specifies the time, in milliseconds, to wait before disabling
+ a slave after a link failure has been detected. This option
+ is only valid for the miimon link monitor. The downdelay
+ value should be a multiple of the miimon value; if not, it
+ will be rounded down to the nearest multiple. The default
+ value is 0.
lacp_rate
- Option specifying the rate in which we'll ask our link partner to
- transmit LACPDU packets in 802.3ad mode. Possible values are:
+ Option specifying the rate in which we'll ask our link partner
+ to transmit LACPDU packets in 802.3ad mode. Possible values
+ are:
slow or 0
- Request partner to transmit LACPDUs every 30 seconds (default)
+ Request partner to transmit LACPDUs every 30 seconds
fast or 1
Request partner to transmit LACPDUs every 1 second
+ The default is slow.
+
max_bonds
Specifies the number of bonding devices to create for this
miimon
- Specifies the frequency in milli-seconds that MII link monitoring
- will occur. A value of zero disables MII link monitoring. A value
- of 100 is a good starting point. See High Availability section for
- additional information. The default value is 0.
+ Specifies the MII link monitoring frequency in milliseconds.
+ This determines how often the link state of each slave is
+ inspected for link failures. A value of zero disables MII
+ link monitoring. A value of 100 is a good starting point.
+ The use_carrier option, below, affects how the link state is
+ determined. See the High Availability section for additional
+ information. The default value is 0.
mode
Specifies one of the bonding policies. The default is
- round-robin (balance-rr). Possible values are (you can use
- either the text or numeric option):
+ balance-rr (round robin). Possible values are:
balance-rr or 0
- Round-robin policy: Transmit in a sequential order
- from the first available slave through the last. This
- mode provides load balancing and fault tolerance.
+ Round-robin policy: Transmit packets in sequential
+ order from the first available slave through the
+ last. This mode provides load balancing and fault
+ tolerance.
active-backup or 1
Active-backup policy: Only one slave in the bond is
- active. A different slave becomes active if, and only
- if, the active slave fails. The bond's MAC address is
+ active. A different slave becomes active if, and only
+ if, the active slave fails. The bond's MAC address is
externally visible on only one port (network adapter)
- to avoid confusing the switch. This mode provides
- fault tolerance.
+ to avoid confusing the switch.
+
+ In bonding version 2.6.2 or later, when a failover
+ occurs in active-backup mode, bonding will issue one
+ or more gratuitous ARPs on the newly active slave.
+ One gratuitous ARP is issued for the bonding master
+ interface and each VLAN interfaces configured above
+ it, provided that the interface has at least one IP
+ address configured. Gratuitous ARPs issued for VLAN
+ interfaces are tagged with the appropriate VLAN id.
+
+ This mode provides fault tolerance. The primary
+ option, documented below, affects the behavior of this
+ mode.
balance-xor or 2
- XOR policy: Transmit based on [(source MAC address
- XOR'd with destination MAC address) modula slave
- count]. This selects the same slave for each
- destination MAC address. This mode provides load
- balancing and fault tolerance.
+ XOR policy: Transmit based on the selected transmit
+ hash policy. The default policy is a simple [(source
+ MAC address XOR'd with destination MAC address) modulo
+ slave count]. Alternate transmit policies may be
+ selected via the xmit_hash_policy option, described
+ below.
+
+ This mode provides load balancing and fault tolerance.
broadcast or 3
Broadcast policy: transmits everything on all slave
- interfaces. This mode provides fault tolerance.
+ interfaces. This mode provides fault tolerance.
802.3ad or 4
- IEEE 802.3ad Dynamic link aggregation. Creates aggregation
- groups that share the same speed and duplex settings.
- Transmits and receives on all slaves in the active
- aggregator.
+ IEEE 802.3ad Dynamic link aggregation. Creates
+ aggregation groups that share the same speed and
+ duplex settings. Utilizes all slaves in the active
+ aggregator according to the 802.3ad specification.
+
+ Slave selection for outgoing traffic is done according
+ to the transmit hash policy, which may be changed from
+ the default simple XOR policy via the xmit_hash_policy
+ option, documented below. Note that not all transmit
+ policies may be 802.3ad compliant, particularly in
+ regards to the packet mis-ordering requirements of
+ section 43.2.4 of the 802.3ad standard. Differing
+ peer implementations will have varying tolerances for
+ noncompliance.
- Pre-requisites:
+ Prerequisites:
- 1. Ethtool support in the base drivers for retrieving the
- speed and duplex of each slave.
+ 1. Ethtool support in the base drivers for retrieving
+ the speed and duplex of each slave.
2. A switch that supports IEEE 802.3ad Dynamic link
aggregation.
+ Most switches will require some type of configuration
+ to enable 802.3ad mode.
+
balance-tlb or 5
- Adaptive transmit load balancing: channel bonding that does
- not require any special switch support. The outgoing
- traffic is distributed according to the current load
- (computed relative to the speed) on each slave. Incoming
- traffic is received by the current slave. If the receiving
- slave fails, another slave takes over the MAC address of
- the failed receiving slave.
+ Adaptive transmit load balancing: channel bonding that
+ does not require any special switch support. The
+ outgoing traffic is distributed according to the
+ current load (computed relative to the speed) on each
+ slave. Incoming traffic is received by the current
+ slave. If the receiving slave fails, another slave
+ takes over the MAC address of the failed receiving
+ slave.
Prerequisite:
balance-alb or 6
- Adaptive load balancing: includes balance-tlb + receive
- load balancing (rlb) for IPV4 traffic and does not require
- any special switch support. The receive load balancing is
- achieved by ARP negotiation. The bonding driver intercepts
- the ARP Replies sent by the server on their way out and
- overwrites the src hw address with the unique hw address of
- one of the slaves in the bond such that different clients
- use different hw addresses for the server.
-
- Receive traffic from connections created by the server is
- also balanced. When the server sends an ARP Request the
- bonding driver copies and saves the client's IP information
- from the ARP. When the ARP Reply arrives from the client,
- its hw address is retrieved and the bonding driver
- initiates an ARP reply to this client assigning it to one
- of the slaves in the bond. A problematic outcome of using
- ARP negotiation for balancing is that each time that an ARP
- request is broadcasted it uses the hw address of the
- bond. Hence, clients learn the hw address of the bond and
- the balancing of receive traffic collapses to the current
- salve. This is handled by sending updates (ARP Replies) to
- all the clients with their assigned hw address such that
- the traffic is redistributed. Receive traffic is also
- redistributed when a new slave is added to the bond and
- when an inactive slave is re-activated. The receive load is
- distributed sequentially (round robin) among the group of
- highest speed slaves in the bond.
-
- When a link is reconnected or a new slave joins the bond
- the receive traffic is redistributed among all active
- slaves in the bond by intiating ARP Replies with the
- selected mac address to each of the clients. The updelay
- modeprobe parameter must be set to a value equal or greater
- than the switch's forwarding delay so that the ARP Replies
- sent to the clients will not be blocked by the switch.
+ Adaptive load balancing: includes balance-tlb plus
+ receive load balancing (rlb) for IPV4 traffic, and
+ does not require any special switch support. The
+ receive load balancing is achieved by ARP negotiation.
+ The bonding driver intercepts the ARP Replies sent by
+ the local system on their way out and overwrites the
+ source hardware address with the unique hardware
+ address of one of the slaves in the bond such that
+ different peers use different hardware addresses for
+ the server.
+
+ Receive traffic from connections created by the server
+ is also balanced. When the local system sends an ARP
+ Request the bonding driver copies and saves the peer's
+ IP information from the ARP packet. When the ARP
+ Reply arrives from the peer, its hardware address is
+ retrieved and the bonding driver initiates an ARP
+ reply to this peer assigning it to one of the slaves
+ in the bond. A problematic outcome of using ARP
+ negotiation for balancing is that each time that an
+ ARP request is broadcast it uses the hardware address
+ of the bond. Hence, peers learn the hardware address
+ of the bond and the balancing of receive traffic
+ collapses to the current slave. This is handled by
+ sending updates (ARP Replies) to all the peers with
+ their individually assigned hardware address such that
+ the traffic is redistributed. Receive traffic is also
+ redistributed when a new slave is added to the bond
+ and when an inactive slave is re-activated. The
+ receive load is distributed sequentially (round robin)
+ among the group of highest speed slaves in the bond.
+
+ When a link is reconnected or a new slave joins the
+ bond the receive traffic is redistributed among all
+ active slaves in the bond by initiating ARP Replies
+ with the selected MAC address to each of the
+ clients. The updelay parameter (detailed below) must
+ be set to a value equal or greater than the switch's
+ forwarding delay so that the ARP Replies sent to the
+ peers will not be blocked by the switch.
Prerequisites:
- 1. Ethtool support in the base drivers for retrieving the
- speed of each slave.
+ 1. Ethtool support in the base drivers for retrieving
+ the speed of each slave.
- 2. Base driver support for setting the hw address of a
- device also when it is open. This is required so that there
- will always be one slave in the team using the bond hw
- address (the curr_active_slave) while having a unique hw
- address for each slave in the bond. If the curr_active_slave
- fails it's hw address is swapped with the new curr_active_slave
- that was chosen.
+ 2. Base driver support for setting the hardware
+ address of a device while it is open. This is
+ required so that there will always be one slave in the
+ team using the bond hardware address (the
+ curr_active_slave) while having a unique hardware
+ address for each slave in the bond. If the
+ curr_active_slave fails its hardware address is
+ swapped with the new curr_active_slave that was
+ chosen.
primary
- A string (eth0, eth2, etc) to equate to a primary device. If this
- value is entered, and the device is on-line, it will be used first
- as the output media. Only when this device is off-line, will
- alternate devices be used. Otherwise, once a failover is detected
- and a new default output is chosen, it will remain the output media
- until it too fails. This is useful when one slave was preferred
- over another, i.e. when one slave is 1000Mbps and another is
- 100Mbps. If the 1000Mbps slave fails and is later restored, it may
- be preferred the faster slave gracefully become the active slave -
- without deliberately failing the 100Mbps slave. Specifying a
- primary is only valid in active-backup mode.
+ A string (eth0, eth2, etc) specifying which slave is the
+ primary device. The specified device will always be the
+ active slave while it is available. Only when the primary is
+ off-line will alternate devices be used. This is useful when
+ one slave is preferred over another, e.g., when one slave has
+ higher throughput than another.
+
+ The primary option is only valid for active-backup mode.
updelay
- Specifies the delay time in milli-seconds to enable a link after a
- link up status has been detected. This should be a multiple of miimon
- value, otherwise the value will be rounded. The default value is 0.
+ Specifies the time, in milliseconds, to wait before enabling a
+ slave after a link recovery has been detected. This option is
+ only valid for the miimon link monitor. The updelay value
+ should be a multiple of the miimon value; if not, it will be
+ rounded down to the nearest multiple. The default value is 0.
use_carrier
- Specifies whether or not miimon should use MII or ETHTOOL
- ioctls vs. netif_carrier_ok() to determine the link status.
- The MII or ETHTOOL ioctls are less efficient and utilize a
- deprecated calling sequence within the kernel. The
- netif_carrier_ok() relies on the device driver to maintain its
- state with netif_carrier_on/off; at this writing, most, but
- not all, device drivers support this facility.
+ Specifies whether or not miimon should use MII or ETHTOOL
+ ioctls vs. netif_carrier_ok() to determine the link
+ status. The MII or ETHTOOL ioctls are less efficient and
+ utilize a deprecated calling sequence within the kernel. The
+ netif_carrier_ok() relies on the device driver to maintain its
+ state with netif_carrier_on/off; at this writing, most, but
+ not all, device drivers support this facility.
- If bonding insists that the link is up when it should not be,
- it may be that your network device driver does not support
- netif_carrier_on/off. This is because the default state for
- netif_carrier is "carrier on." In this case, disabling
- use_carrier will cause bonding to revert to the MII / ETHTOOL
- ioctl method to determine the link state.
+ If bonding insists that the link is up when it should not be,
+ it may be that your network device driver does not support
+ netif_carrier_on/off. The default state for netif_carrier is
+ "carrier on," so if a driver does not support netif_carrier,
+ it will appear as if the link is always up. In this case,
+ setting use_carrier to 0 will cause bonding to revert to the
+ MII / ETHTOOL ioctl method to determine the link state.
- A value of 1 enables the use of netif_carrier_ok(), a value of
- 0 will use the deprecated MII / ETHTOOL ioctls. The default
- value is 1.
+ A value of 1 enables the use of netif_carrier_ok(), a value of
+ 0 will use the deprecated MII / ETHTOOL ioctls. The default
+ value is 1.
+xmit_hash_policy
-Configuring Multiple Bonds
-==========================
+ Selects the transmit hash policy to use for slave selection in
+ balance-xor and 802.3ad modes. Possible values are:
-If several bonding interfaces are required, either specify the max_bonds
-parameter (described above), or load the driver multiple times. Using
-the max_bonds parameter is less complicated, but has the limitation that
-all bonding instances created will have the same options. Loading the
-driver multiple times allows each instance of the driver to have differing
-options.
+ layer2
-For example, to configure two bonding interfaces, one with mii link
-monitoring performed every 100 milliseconds, and one with ARP link
-monitoring performed every 200 milliseconds, the /etc/conf.modules should
-resemble the following:
+ Uses XOR of hardware MAC addresses to generate the
+ hash. The formula is
-alias bond0 bonding
-alias bond1 bonding
+ (source MAC XOR destination MAC) modulo slave count
-options bond0 miimon=100
-options bond1 -o bonding1 arp_interval=200 arp_ip_target=10.0.0.1
+ This algorithm will place all traffic to a particular
+ network peer on the same slave.
-Configuring Multiple ARP Targets
-================================
+ This algorithm is 802.3ad compliant.
-While ARP monitoring can be done with just one target, it can be useful
-in a High Availability setup to have several targets to monitor. In the
-case of just one target, the target itself may go down or have a problem
-making it unresponsive to ARP requests. Having an additional target (or
-several) increases the reliability of the ARP monitoring.
+ layer3+4
-Multiple ARP targets must be seperated by commas as follows:
+ This policy uses upper layer protocol information,
+ when available, to generate the hash. This allows for
+ traffic to a particular network peer to span multiple
+ slaves, although a single connection will not span
+ multiple slaves.
-# example options for ARP monitoring with three targets
-alias bond0 bonding
-options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
+ The formula for unfragmented TCP and UDP packets is
-For just a single target the options would resemble:
+ ((source port XOR dest port) XOR
+ ((source IP XOR dest IP) AND 0xffff)
+ modulo slave count
-# example options for ARP monitoring with one target
-alias bond0 bonding
-options bond0 arp_interval=60 arp_ip_target=192.168.0.100
+ For fragmented TCP or UDP packets and all other IP
+ protocol traffic, the source and destination port
+ information is omitted. For non-IP traffic, the
+ formula is the same as for the layer2 transmit hash
+ policy.
-Potential Problems When Using ARP Monitor
-=========================================
+ This policy is intended to mimic the behavior of
+ certain switches, notably Cisco switches with PFC2 as
+ well as some Foundry and IBM products.
-1. Driver support
+ This algorithm is not fully 802.3ad compliant. A
+ single TCP or UDP conversation containing both
+ fragmented and unfragmented packets will see packets
+ striped across two interfaces. This may result in out
+ of order delivery. Most traffic types will not meet
+ this criteria, as TCP rarely fragments traffic, and
+ most UDP traffic is not involved in extended
+ conversations. Other implementations of 802.3ad may
+ or may not tolerate this noncompliance.
-The ARP monitor relies on the network device driver to maintain two
-statistics: the last receive time (dev->last_rx), and the last
-transmit time (dev->trans_start). If the network device driver does
-not update one or both of these, then the typical result will be that,
-upon startup, all links in the bond will immediately be declared down,
-and remain that way. A network monitoring tool (tcpdump, e.g.) will
-show ARP requests and replies being sent and received on the bonding
-device.
+ The default value is layer2. This option was added in bonding
+version 2.6.3. In earlier versions of bonding, this parameter does
+not exist, and the layer2 policy is the only policy.
+
+
+3. Configuring Bonding Devices
+==============================
+
+ You can configure bonding using either your distro's network
+initialization scripts, or manually using either ifenslave or the
+sysfs interface. Distros generally use one of two packages for the
+network initialization scripts: initscripts or sysconfig. Recent
+versions of these packages have support for bonding, while older
+versions do not.
-The possible resolutions for this are to (a) fix the device driver, or
-(b) discontinue the ARP monitor (using miimon as an alternative, for
-example).
+ We will first describe the options for configuring bonding for
+distros using versions of initscripts and sysconfig with full or
+partial support for bonding, then provide information on enabling
+bonding without support from the network initialization scripts (i.e.,
+older versions of initscripts or sysconfig).
-2. Adventures in Routing
+ If you're unsure whether your distro uses sysconfig or
+initscripts, or don't know if it's new enough, have no fear.
+Determining this is fairly straightforward.
-When bonding is set up with the ARP monitor, it is important that the
-slave devices not have routes that supercede routes of the master (or,
-generally, not have routes at all). For example, suppose the bonding
-device bond0 has two slaves, eth0 and eth1, and the routing table is
-as follows:
+ First, issue the command:
-Kernel IP routing table
-Destination Gateway Genmask Flags MSS Window irtt Iface
-10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
-10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
-10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
-127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
+$ rpm -qf /sbin/ifup
-In this case, the ARP monitor (and ARP itself) may become confused,
-because ARP requests will be sent on one interface (bond0), but the
-corresponding reply will arrive on a different interface (eth0). This
-reply looks to ARP as an unsolicited ARP reply (because ARP matches
-replies on an interface basis), and is discarded. This will likely
-still update the receive/transmit times in the driver, but will lose
-packets.
-
-The resolution here is simply to insure that slaves do not have routes
-of their own, and if for some reason they must, those routes do not
-supercede routes of their master. This should generally be the case,
-but unusual configurations or errant manual or automatic static route
-additions may cause trouble.
-
-Switch Configuration
-====================
+ It will respond with a line of text starting with either
+"initscripts" or "sysconfig," followed by some numbers. This is the
+package that provides your network initialization scripts.
-While the switch does not need to be configured when the active-backup,
-balance-tlb or balance-alb policies (mode=1,5,6) are used, it does need to
-be configured for the round-robin, XOR, broadcast, or 802.3ad policies
-(mode=0,2,3,4).
+ Next, to determine if your installation supports bonding,
+issue the command:
+$ grep ifenslave /sbin/ifup
-Verifying Bond Configuration
-============================
+ If this returns any matches, then your initscripts or
+sysconfig has support for bonding.
-1) Bonding information files
-----------------------------
-The bonding driver information files reside in the /proc/net/bonding directory.
+3.1 Configuration with Sysconfig Support
+----------------------------------------
+
+ This section applies to distros using a version of sysconfig
+with bonding support, for example, SuSE Linux Enterprise Server 9.
+
+ SuSE SLES 9's networking configuration system does support
+bonding, however, at this writing, the YaST system configuration
+front end does not provide any means to work with bonding devices.
+Bonding devices can be managed by hand, however, as follows.
+
+ First, if they have not already been configured, configure the
+slave devices. On SLES 9, this is most easily done by running the
+yast2 sysconfig configuration utility. The goal is for to create an
+ifcfg-id file for each slave device. The simplest way to accomplish
+this is to configure the devices for DHCP (this is only to get the
+file ifcfg-id file created; see below for some issues with DHCP). The
+name of the configuration file for each device will be of the form:
+
+ifcfg-id-xx:xx:xx:xx:xx:xx
+
+ Where the "xx" portion will be replaced with the digits from
+the device's permanent MAC address.
+
+ Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
+created, it is necessary to edit the configuration files for the slave
+devices (the MAC addresses correspond to those of the slave devices).
+Before editing, the file will contain multiple lines, and will look
+something like this:
+
+BOOTPROTO='dhcp'
+STARTMODE='on'
+USERCTL='no'
+UNIQUE='XNzu.WeZGOGF+4wE'
+_nm_name='bus-pci-0001:61:01.0'
+
+ Change the BOOTPROTO and STARTMODE lines to the following:
+
+BOOTPROTO='none'
+STARTMODE='off'
+
+ Do not alter the UNIQUE or _nm_name lines. Remove any other
+lines (USERCTL, etc).
+
+ Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
+it's time to create the configuration file for the bonding device
+itself. This file is named ifcfg-bondX, where X is the number of the
+bonding device to create, starting at 0. The first such file is
+ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig
+network configuration system will correctly start multiple instances
+of bonding.
+
+ The contents of the ifcfg-bondX file is as follows:
+
+BOOTPROTO="static"
+BROADCAST="10.0.2.255"
+IPADDR="10.0.2.10"
+NETMASK="255.255.0.0"
+NETWORK="10.0.2.0"
+REMOTE_IPADDR=""
+STARTMODE="onboot"
+BONDING_MASTER="yes"
+BONDING_MODULE_OPTS="mode=active-backup miimon=100"
+BONDING_SLAVE0="eth0"
+BONDING_SLAVE1="bus-pci-0000:06:08.1"
+
+ Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
+values with the appropriate values for your network.
+
+ The STARTMODE specifies when the device is brought online.
+The possible values are:
+
+ onboot: The device is started at boot time. If you're not
+ sure, this is probably what you want.
+
+ manual: The device is started only when ifup is called
+ manually. Bonding devices may be configured this
+ way if you do not wish them to start automatically
+ at boot for some reason.
+
+ hotplug: The device is started by a hotplug event. This is not
+ a valid choice for a bonding device.
+
+ off or ignore: The device configuration is ignored.
+
+ The line BONDING_MASTER='yes' indicates that the device is a
+bonding master device. The only useful value is "yes."
+
+ The contents of BONDING_MODULE_OPTS are supplied to the
+instance of the bonding module for this device. Specify the options
+for the bonding mode, link monitoring, and so on here. Do not include
+the max_bonds bonding parameter; this will confuse the configuration
+system if you have multiple bonding devices.
+
+ Finally, supply one BONDING_SLAVEn="slave device" for each
+slave. where "n" is an increasing value, one for each slave. The
+"slave device" is either an interface name, e.g., "eth0", or a device
+specifier for the network device. The interface name is easier to
+find, but the ethN names are subject to change at boot time if, e.g.,
+a device early in the sequence has failed. The device specifiers
+(bus-pci-0000:06:08.1 in the example above) specify the physical
+network device, and will not change unless the device's bus location
+changes (for example, it is moved from one PCI slot to another). The
+example above uses one of each type for demonstration purposes; most
+configurations will choose one or the other for all slave devices.
+
+ When all configuration files have been modified or created,
+networking must be restarted for the configuration changes to take
+effect. This can be accomplished via the following:
+
+# /etc/init.d/network restart
+
+ Note that the network control script (/sbin/ifdown) will
+remove the bonding module as part of the network shutdown processing,
+so it is not necessary to remove the module by hand if, e.g., the
+module parameters have changed.
+
+ Also, at this writing, YaST/YaST2 will not manage bonding
+devices (they do not show bonding interfaces on its list of network
+devices). It is necessary to edit the configuration file by hand to
+change the bonding configuration.
+
+ Additional general options and details of the ifcfg file
+format can be found in an example ifcfg template file:
+
+/etc/sysconfig/network/ifcfg.template
+
+ Note that the template does not document the various BONDING_
+settings described above, but does describe many of the other options.
+
+3.1.1 Using DHCP with Sysconfig
+-------------------------------
+
+ Under sysconfig, configuring a device with BOOTPROTO='dhcp'
+will cause it to query DHCP for its IP address information. At this
+writing, this does not function for bonding devices; the scripts
+attempt to obtain the device address from DHCP prior to adding any of
+the slave devices. Without active slaves, the DHCP requests are not
+sent to the network.
+
+3.1.2 Configuring Multiple Bonds with Sysconfig
+-----------------------------------------------
+
+ The sysconfig network initialization system is capable of
+handling multiple bonding devices. All that is necessary is for each
+bonding instance to have an appropriately configured ifcfg-bondX file
+(as described above). Do not specify the "max_bonds" parameter to any
+instance of bonding, as this will confuse sysconfig. If you require
+multiple bonding devices with identical parameters, create multiple
+ifcfg-bondX files.
+
+ Because the sysconfig scripts supply the bonding module
+options in the ifcfg-bondX file, it is not necessary to add them to
+the system /etc/modules.conf or /etc/modprobe.conf configuration file.
+
+3.2 Configuration with Initscripts Support
+------------------------------------------
+
+ This section applies to distros using a version of initscripts
+with bonding support, for example, Red Hat Linux 9 or Red Hat
+Enterprise Linux version 3 or 4. On these systems, the network
+initialization scripts have some knowledge of bonding, and can be
+configured to control bonding devices.
+
+ These distros will not automatically load the network adapter
+driver unless the ethX device is configured with an IP address.
+Because of this constraint, users must manually configure a
+network-script file for all physical adapters that will be members of
+a bondX link. Network script files are located in the directory:
+
+/etc/sysconfig/network-scripts
+
+ The file name must be prefixed with "ifcfg-eth" and suffixed
+with the adapter's physical adapter number. For example, the script
+for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
+Place the following text in the file:
+
+DEVICE=eth0
+USERCTL=no
+ONBOOT=yes
+MASTER=bond0
+SLAVE=yes
+BOOTPROTO=none
+
+ The DEVICE= line will be different for every ethX device and
+must correspond with the name of the file, i.e., ifcfg-eth1 must have
+a device line of DEVICE=eth1. The setting of the MASTER= line will
+also depend on the final bonding interface name chosen for your bond.
+As with other network devices, these typically start at 0, and go up
+one for each device, i.e., the first bonding instance is bond0, the
+second is bond1, and so on.
+
+ Next, create a bond network script. The file name for this
+script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
+the number of the bond. For bond0 the file is named "ifcfg-bond0",
+for bond1 it is named "ifcfg-bond1", and so on. Within that file,
+place the following text:
+
+DEVICE=bond0
+IPADDR=192.168.1.1
+NETMASK=255.255.255.0
+NETWORK=192.168.1.0
+BROADCAST=192.168.1.255
+ONBOOT=yes
+BOOTPROTO=none
+USERCTL=no
+
+ Be sure to change the networking specific lines (IPADDR,
+NETMASK, NETWORK and BROADCAST) to match your network configuration.
+
+ Finally, it is necessary to edit /etc/modules.conf (or
+/etc/modprobe.conf, depending upon your distro) to load the bonding
+module with your desired options when the bond0 interface is brought
+up. The following lines in /etc/modules.conf (or modprobe.conf) will
+load the bonding module, and select its options:
+
+alias bond0 bonding
+options bond0 mode=balance-alb miimon=100
+
+ Replace the sample parameters with the appropriate set of
+options for your configuration.
+
+ Finally run "/etc/rc.d/init.d/network restart" as root. This
+will restart the networking subsystem and your bond link should be now
+up and running.
-Sample contents of /proc/net/bonding/bond0 after the driver is loaded with
-parameters of mode=0 and miimon=1000 is shown below.
+3.2.1 Using DHCP with Initscripts
+---------------------------------
+
+ Recent versions of initscripts (the version supplied with
+Fedora Core 3 and Red Hat Enterprise Linux 4 is reported to work) do
+have support for assigning IP information to bonding devices via DHCP.
+
+ To configure bonding for DHCP, configure it as described
+above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
+and add a line consisting of "TYPE=Bonding". Note that the TYPE value
+is case sensitive.
+
+3.2.2 Configuring Multiple Bonds with Initscripts
+-------------------------------------------------
+
+ At this writing, the initscripts package does not directly
+support loading the bonding driver multiple times, so the process for
+doing so is the same as described in the "Configuring Multiple Bonds
+Manually" section, below.
+
+ NOTE: It has been observed that some Red Hat supplied kernels
+are apparently unable to rename modules at load time (the "-o bond1"
+part). Attempts to pass that option to modprobe will produce an
+"Operation not permitted" error. This has been reported on some
+Fedora Core kernels, and has been seen on RHEL 4 as well. On kernels
+exhibiting this problem, it will be impossible to configure multiple
+bonds with differing parameters.
+
+3.3 Configuring Bonding Manually with Ifenslave
+-----------------------------------------------
+
+ This section applies to distros whose network initialization
+scripts (the sysconfig or initscripts package) do not have specific
+knowledge of bonding. One such distro is SuSE Linux Enterprise Server
+version 8.
+
+ The general method for these systems is to place the bonding
+module parameters into /etc/modules.conf or /etc/modprobe.conf (as
+appropriate for the installed distro), then add modprobe and/or
+ifenslave commands to the system's global init script. The name of
+the global init script differs; for sysconfig, it is
+/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
+
+ For example, if you wanted to make a simple bond of two e100
+devices (presumed to be eth0 and eth1), and have it persist across
+reboots, edit the appropriate file (/etc/init.d/boot.local or
+/etc/rc.d/rc.local), and add the following:
+
+modprobe bonding mode=balance-alb miimon=100
+modprobe e100
+ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
+ifenslave bond0 eth0
+ifenslave bond0 eth1
+
+ Replace the example bonding module parameters and bond0
+network configuration (IP address, netmask, etc) with the appropriate
+values for your configuration.
+
+ Unfortunately, this method will not provide support for the
+ifup and ifdown scripts on the bond devices. To reload the bonding
+configuration, it is necessary to run the initialization script, e.g.,
+
+# /etc/init.d/boot.local
+
+ or
+
+# /etc/rc.d/rc.local
+
+ It may be desirable in such a case to create a separate script
+which only initializes the bonding configuration, then call that
+separate script from within boot.local. This allows for bonding to be
+enabled without re-running the entire global init script.
+
+ To shut down the bonding devices, it is necessary to first
+mark the bonding device itself as being down, then remove the
+appropriate device driver modules. For our example above, you can do
+the following:
+
+# ifconfig bond0 down
+# rmmod bonding
+# rmmod e100
+
+ Again, for convenience, it may be desirable to create a script
+with these commands.
+
+
+3.3.1 Configuring Multiple Bonds Manually
+-----------------------------------------
+
+ This section contains information on configuring multiple
+bonding devices with differing options for those systems whose network
+initialization scripts lack support for configuring multiple bonds.
+
+ If you require multiple bonding devices, but all with the same
+options, you may wish to use the "max_bonds" module parameter,
+documented above.
+
+ To create multiple bonding devices with differing options, it
+is necessary to load the bonding driver multiple times. Note that
+current versions of the sysconfig network initialization scripts
+handle this automatically; if your distro uses these scripts, no
+special action is needed. See the section Configuring Bonding
+Devices, above, if you're not sure about your network initialization
+scripts.
+
+ To load multiple instances of the module, it is necessary to
+specify a different name for each instance (the module loading system
+requires that every loaded module, even multiple instances of the same
+module, have a unique name). This is accomplished by supplying
+multiple sets of bonding options in /etc/modprobe.conf, for example:
+
+alias bond0 bonding
+options bond0 -o bond0 mode=balance-rr miimon=100
+
+alias bond1 bonding
+options bond1 -o bond1 mode=balance-alb miimon=50
+
+ will load the bonding module two times. The first instance is
+named "bond0" and creates the bond0 device in balance-rr mode with an
+miimon of 100. The second instance is named "bond1" and creates the
+bond1 device in balance-alb mode with an miimon of 50.
+ In some circumstances (typically with older distributions),
+the above does not work, and the second bonding instance never sees
+its options. In that case, the second options line can be substituted
+as follows:
+
+install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
+ mode=balance-alb miimon=50
+
+ This may be repeated any number of times, specifying a new and
+unique name in place of bond1 for each subsequent instance.
+
+3.4 Configuring Bonding Manually via Sysfs
+------------------------------------------
+
+ Starting with version 3.0, Channel Bonding may be configured
+via the sysfs interface. This interface allows dynamic configuration
+of all bonds in the system without unloading the module. It also
+allows for adding and removing bonds at runtime. Ifenslave is no
+longer required, though it is still supported.
+
+ Use of the sysfs interface allows you to use multiple bonds
+with different configurations without having to reload the module.
+It also allows you to use multiple, differently configured bonds when
+bonding is compiled into the kernel.
+
+ You must have the sysfs filesystem mounted to configure
+bonding this way. The examples in this document assume that you
+are using the standard mount point for sysfs, e.g. /sys. If your
+sysfs filesystem is mounted elsewhere, you will need to adjust the
+example paths accordingly.
+
+Creating and Destroying Bonds
+-----------------------------
+To add a new bond foo:
+# echo +foo > /sys/class/net/bonding_masters
+
+To remove an existing bond bar:
+# echo -bar > /sys/class/net/bonding_masters
+
+To show all existing bonds:
+# cat /sys/class/net/bonding_masters
+
+NOTE: due to 4K size limitation of sysfs files, this list may be
+truncated if you have more than a few hundred bonds. This is unlikely
+to occur under normal operating conditions.
+
+Adding and Removing Slaves
+--------------------------
+ Interfaces may be enslaved to a bond using the file
+/sys/class/net/<bond>/bonding/slaves. The semantics for this file
+are the same as for the bonding_masters file.
+
+To enslave interface eth0 to bond bond0:
+# ifconfig bond0 up
+# echo +eth0 > /sys/class/net/bond0/bonding/slaves
+
+To free slave eth0 from bond bond0:
+# echo -eth0 > /sys/class/net/bond0/bonding/slaves
+
+ NOTE: The bond must be up before slaves can be added. All
+slaves are freed when the interface is brought down.
+
+ When an interface is enslaved to a bond, symlinks between the
+two are created in the sysfs filesystem. In this case, you would get
+/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
+/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
+
+ This means that you can tell quickly whether or not an
+interface is enslaved by looking for the master symlink. Thus:
+# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
+will free eth0 from whatever bond it is enslaved to, regardless of
+the name of the bond interface.
+
+Changing a Bond's Configuration
+-------------------------------
+ Each bond may be configured individually by manipulating the
+files located in /sys/class/net/<bond name>/bonding
+
+ The names of these files correspond directly with the command-
+line parameters described elsewhere in this file, and, with the
+exception of arp_ip_target, they accept the same values. To see the
+current setting, simply cat the appropriate file.
+
+ A few examples will be given here; for specific usage
+guidelines for each parameter, see the appropriate section in this
+document.
+
+To configure bond0 for balance-alb mode:
+# ifconfig bond0 down
+# echo 6 > /sys/class/net/bond0/bonding/mode
+ - or -
+# echo balance-alb > /sys/class/net/bond0/bonding/mode
+ NOTE: The bond interface must be down before the mode can be
+changed.
+
+To enable MII monitoring on bond0 with a 1 second interval:
+# echo 1000 > /sys/class/net/bond0/bonding/miimon
+ NOTE: If ARP monitoring is enabled, it will disabled when MII
+monitoring is enabled, and vice-versa.
+
+To add ARP targets:
+# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
+# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
+ NOTE: up to 10 target addresses may be specified.
+
+To remove an ARP target:
+# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
+
+Example Configuration
+---------------------
+ We begin with the same example that is shown in section 3.3,
+executed with sysfs, and without using ifenslave.
+
+ To make a simple bond of two e100 devices (presumed to be eth0
+and eth1), and have it persist across reboots, edit the appropriate
+file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
+following:
+
+modprobe bonding
+modprobe e100
+echo balance-alb > /sys/class/net/bond0/bonding/mode
+ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
+echo 100 > /sys/class/net/bond0/bonding/miimon
+echo +eth0 > /sys/class/net/bond0/bonding/slaves
+echo +eth1 > /sys/class/net/bond0/bonding/slaves
+
+ To add a second bond, with two e1000 interfaces in
+active-backup mode, using ARP monitoring, add the following lines to
+your init script:
+
+modprobe e1000
+echo +bond1 > /sys/class/net/bonding_masters
+echo active-backup > /sys/class/net/bond1/bonding/mode
+ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
+echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
+echo 2000 > /sys/class/net/bond1/bonding/arp_interval
+echo +eth2 > /sys/class/net/bond1/bonding/slaves
+echo +eth3 > /sys/class/net/bond1/bonding/slaves
+
+
+4. Querying Bonding Configuration
+=================================
+
+4.1 Bonding Configuration
+-------------------------
+
+ Each bonding device has a read-only file residing in the
+/proc/net/bonding directory. The file contents include information
+about the bonding configuration, options and state of each slave.
+
+ For example, the contents of /proc/net/bonding/bond0 after the
+driver is loaded with parameters of mode=0 and miimon=1000 is
+generally as follows:
+
+ Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
Bonding Mode: load balancing (round-robin)
Currently Active Slave: eth0
MII Status: up
MII Status: up
Link Failure Count: 1
-2) Network verification
------------------------
-The network configuration can be verified using the ifconfig command. In
-the example below, the bond0 interface is the master (MASTER) while eth0 and
-eth1 are slaves (SLAVE). Notice all slaves of bond0 have the same MAC address
-(HWaddr) as bond0 for all modes except TLB and ALB that require a unique MAC
-address for each slave.
+ The precise format and contents will change depending upon the
+bonding configuration, state, and version of the bonding driver.
+
+4.2 Network configuration
+-------------------------
+
+ The network configuration can be inspected using the ifconfig
+command. Bonding devices will have the MASTER flag set; Bonding slave
+devices will have the SLAVE flag set. The ifconfig output does not
+contain information on which slaves are associated with which masters.
-[root]# /sbin/ifconfig
+ In the example below, the bond0 interface is the master
+(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
+bond0 have the same MAC address (HWaddr) as bond0 for all modes except
+TLB and ALB that require a unique MAC address for each slave.
+
+# /sbin/ifconfig
bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
collisions:0 txqueuelen:0
eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
- inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
Interrupt:10 Base address:0x1080
eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
- inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
Interrupt:9 Base address:0x1400
+5. Switch Configuration
+=======================
+
+ For this section, "switch" refers to whatever system the
+bonded devices are directly connected to (i.e., where the other end of
+the cable plugs into). This may be an actual dedicated switch device,
+or it may be another regular system (e.g., another computer running
+Linux),
+
+ The active-backup, balance-tlb and balance-alb modes do not
+require any specific configuration of the switch.
+
+ The 802.3ad mode requires that the switch have the appropriate
+ports configured as an 802.3ad aggregation. The precise method used
+to configure this varies from switch to switch, but, for example, a
+Cisco 3550 series switch requires that the appropriate ports first be
+grouped together in a single etherchannel instance, then that
+etherchannel is set to mode "lacp" to enable 802.3ad (instead of
+standard EtherChannel).
+
+ The balance-rr, balance-xor and broadcast modes generally
+require that the switch have the appropriate ports grouped together.
+The nomenclature for such a group differs between switches, it may be
+called an "etherchannel" (as in the Cisco example, above), a "trunk
+group" or some other similar variation. For these modes, each switch
+will also have its own configuration options for the switch's transmit
+policy to the bond. Typical choices include XOR of either the MAC or
+IP addresses. The transmit policy of the two peers does not need to
+match. For these three modes, the bonding mode really selects a
+transmit policy for an EtherChannel group; all three will interoperate
+with another EtherChannel group.
+
+
+6. 802.1q VLAN Support
+======================
+
+ It is possible to configure VLAN devices over a bond interface
+using the 8021q driver. However, only packets coming from the 8021q
+driver and passing through bonding will be tagged by default. Self
+generated packets, for example, bonding's learning packets or ARP
+packets generated by either ALB mode or the ARP monitor mechanism, are
+tagged internally by bonding itself. As a result, bonding must
+"learn" the VLAN IDs configured above it, and use those IDs to tag
+self generated packets.
+
+ For reasons of simplicity, and to support the use of adapters
+that can do VLAN hardware acceleration offloading, the bonding
+interface declares itself as fully hardware offloading capable, it gets
+the add_vid/kill_vid notifications to gather the necessary
+information, and it propagates those actions to the slaves. In case
+of mixed adapter types, hardware accelerated tagged packets that
+should go through an adapter that is not offloading capable are
+"un-accelerated" by the bonding driver so the VLAN tag sits in the
+regular location.
+
+ VLAN interfaces *must* be added on top of a bonding interface
+only after enslaving at least one slave. The bonding interface has a
+hardware address of 00:00:00:00:00:00 until the first slave is added.
+If the VLAN interface is created prior to the first enslavement, it
+would pick up the all-zeroes hardware address. Once the first slave
+is attached to the bond, the bond device itself will pick up the
+slave's hardware address, which is then available for the VLAN device.
+
+ Also, be aware that a similar problem can occur if all slaves
+are released from a bond that still has one or more VLAN interfaces on
+top of it. When a new slave is added, the bonding interface will
+obtain its hardware address from the first slave, which might not
+match the hardware address of the VLAN interfaces (which was
+ultimately copied from an earlier slave).
+
+ There are two methods to insure that the VLAN device operates
+with the correct hardware address if all slaves are removed from a
+bond interface:
+
+ 1. Remove all VLAN interfaces then recreate them
+
+ 2. Set the bonding interface's hardware address so that it
+matches the hardware address of the VLAN interfaces.
+
+ Note that changing a VLAN interface's HW address would set the
+underlying device -- i.e. the bonding interface -- to promiscuous
+mode, which might not be what you want.
+
+
+7. Link Monitoring
+==================
-Frequently Asked Questions
-==========================
-
-1. Is it SMP safe?
-
- Yes. The old 2.0.xx channel bonding patch was not SMP safe.
- The new driver was designed to be SMP safe from the start.
-
-2. What type of cards will work with it?
-
- Any Ethernet type cards (you can even mix cards - a Intel
- EtherExpress PRO/100 and a 3com 3c905b, for example).
- You can even bond together Gigabit Ethernet cards!
-
-3. How many bonding devices can I have?
-
- There is no limit.
-
-4. How many slaves can a bonding device have?
-
- Limited by the number of network interfaces Linux supports and/or the
- number of network cards you can place in your system.
-
-5. What happens when a slave link dies?
-
- If your ethernet cards support MII or ETHTOOL link status monitoring
- and the MII monitoring has been enabled in the driver (see description
- of module parameters), there will be no adverse consequences. This
- release of the bonding driver knows how to get the MII information and
- enables or disables its slaves according to their link status.
- See section on High Availability for additional information.
-
- For ethernet cards not supporting MII status, the arp_interval and
- arp_ip_target parameters must be specified for bonding to work
- correctly. If packets have not been sent or received during the
- specified arp_interval duration, an ARP request is sent to the
- targets to generate send and receive traffic. If after this
- interval, either the successful send and/or receive count has not
- incremented, the next slave in the sequence will become the active
- slave.
-
- If neither mii_monitor and arp_interval is configured, the bonding
- driver will not handle this situation very well. The driver will
- continue to send packets but some packets will be lost. Retransmits
- will cause serious degradation of performance (in the case when one
- of two slave links fails, 50% packets will be lost, which is a serious
- problem for both TCP and UDP).
-
-6. Can bonding be used for High Availability?
-
- Yes, if you use MII monitoring and ALL your cards support MII link
- status reporting. See section on High Availability for more
- information.
-
-7. Which switches/systems does it work with?
-
- In round-robin and XOR mode, it works with systems that support
- trunking:
-
- * Many Cisco switches and routers (look for EtherChannel support).
- * SunTrunking software.
- * Alteon AceDirector switches / WebOS (use Trunks).
- * BayStack Switches (trunks must be explicitly configured). Stackable
- models (450) can define trunks between ports on different physical
- units.
- * Linux bonding, of course !
+ The bonding driver at present supports two schemes for
+monitoring a slave device's link state: the ARP monitor and the MII
+monitor.
+
+ At the present time, due to implementation restrictions in the
+bonding driver itself, it is not possible to enable both ARP and MII
+monitoring simultaneously.
+
+7.1 ARP Monitor Operation
+-------------------------
+
+ The ARP monitor operates as its name suggests: it sends ARP
+queries to one or more designated peer systems on the network, and
+uses the response as an indication that the link is operating. This
+gives some assurance that traffic is actually flowing to and from one
+or more peers on the local network.
+
+ The ARP monitor relies on the device driver itself to verify
+that traffic is flowing. In particular, the driver must keep up to
+date the last receive time, dev->last_rx, and transmit start time,
+dev->trans_start. If these are not updated by the driver, then the
+ARP monitor will immediately fail any slaves using that driver, and
+those slaves will stay down. If networking monitoring (tcpdump, etc)
+shows the ARP requests and replies on the network, then it may be that
+your device driver is not updating last_rx and trans_start.
+
+7.2 Configuring Multiple ARP Targets
+------------------------------------
- In 802.3ad mode, it works with with systems that support IEEE 802.3ad
- Dynamic Link Aggregation:
+ While ARP monitoring can be done with just one target, it can
+be useful in a High Availability setup to have several targets to
+monitor. In the case of just one target, the target itself may go
+down or have a problem making it unresponsive to ARP requests. Having
+an additional target (or several) increases the reliability of the ARP
+monitoring.
- * Extreme networks Summit 7i (look for link-aggregation).
- * Many Cisco switches and routers (look for LACP support; this may
- require an upgrade to your IOS software; LACP support was added
- by Cisco in late 2002).
- * Foundry Big Iron 4000
+ Multiple ARP targets must be separated by commas as follows:
- In active-backup, balance-tlb and balance-alb modes, it should work
- with any Layer-II switch.
+# example options for ARP monitoring with three targets
+alias bond0 bonding
+options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
+ For just a single target the options would resemble:
-8. Where does a bonding device get its MAC address from?
+# example options for ARP monitoring with one target
+alias bond0 bonding
+options bond0 arp_interval=60 arp_ip_target=192.168.0.100
- If not explicitly configured with ifconfig, the MAC address of the
- bonding device is taken from its first slave device. This MAC address
- is then passed to all following slaves and remains persistent (even if
- the the first slave is removed) until the bonding device is brought
- down or reconfigured.
- If you wish to change the MAC address, you can set it with ifconfig:
+7.3 MII Monitor Operation
+-------------------------
+
+ The MII monitor monitors only the carrier state of the local
+network interface. It accomplishes this in one of three ways: by
+depending upon the device driver to maintain its carrier state, by
+querying the device's MII registers, or by making an ethtool query to
+the device.
+
+ If the use_carrier module parameter is 1 (the default value),
+then the MII monitor will rely on the driver for carrier state
+information (via the netif_carrier subsystem). As explained in the
+use_carrier parameter information, above, if the MII monitor fails to
+detect carrier loss on the device (e.g., when the cable is physically
+disconnected), it may be that the driver does not support
+netif_carrier.
+
+ If use_carrier is 0, then the MII monitor will first query the
+device's (via ioctl) MII registers and check the link state. If that
+request fails (not just that it returns carrier down), then the MII
+monitor will make an ethtool ETHOOL_GLINK request to attempt to obtain
+the same information. If both methods fail (i.e., the driver either
+does not support or had some error in processing both the MII register
+and ethtool requests), then the MII monitor will assume the link is
+up.
+
+8. Potential Sources of Trouble
+===============================
+
+8.1 Adventures in Routing
+-------------------------
+
+ When bonding is configured, it is important that the slave
+devices not have routes that supersede routes of the master (or,
+generally, not have routes at all). For example, suppose the bonding
+device bond0 has two slaves, eth0 and eth1, and the routing table is
+as follows:
- # ifconfig bond0 hw ether 00:11:22:33:44:55
+Kernel IP routing table
+Destination Gateway Genmask Flags MSS Window irtt Iface
+10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
+10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
+10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
+127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
- The MAC address can be also changed by bringing down/up the device
- and then changing its slaves (or their order):
+ This routing configuration will likely still update the
+receive/transmit times in the driver (needed by the ARP monitor), but
+may bypass the bonding driver (because outgoing traffic to, in this
+case, another host on network 10 would use eth0 or eth1 before bond0).
+
+ The ARP monitor (and ARP itself) may become confused by this
+configuration, because ARP requests (generated by the ARP monitor)
+will be sent on one interface (bond0), but the corresponding reply
+will arrive on a different interface (eth0). This reply looks to ARP
+as an unsolicited ARP reply (because ARP matches replies on an
+interface basis), and is discarded. The MII monitor is not affected
+by the state of the routing table.
+
+ The solution here is simply to insure that slaves do not have
+routes of their own, and if for some reason they must, those routes do
+not supersede routes of their master. This should generally be the
+case, but unusual configurations or errant manual or automatic static
+route additions may cause trouble.
+
+8.2 Ethernet Device Renaming
+----------------------------
- # ifconfig bond0 down ; modprobe -r bonding
- # ifconfig bond0 .... up
- # ifenslave bond0 eth...
+ On systems with network configuration scripts that do not
+associate physical devices directly with network interface names (so
+that the same physical device always has the same "ethX" name), it may
+be necessary to add some special logic to either /etc/modules.conf or
+/etc/modprobe.conf (depending upon which is installed on the system).
- This method will automatically take the address from the next slave
- that will be added.
+ For example, given a modules.conf containing the following:
- To restore your slaves' MAC addresses, you need to detach them
- from the bond (`ifenslave -d bond0 eth0'). The bonding driver will then
- restore the MAC addresses that the slaves had before they were enslaved.
+alias bond0 bonding
+options bond0 mode=some-mode miimon=50
+alias eth0 tg3
+alias eth1 tg3
+alias eth2 e1000
+alias eth3 e1000
+
+ If neither eth0 and eth1 are slaves to bond0, then when the
+bond0 interface comes up, the devices may end up reordered. This
+happens because bonding is loaded first, then its slave device's
+drivers are loaded next. Since no other drivers have been loaded,
+when the e1000 driver loads, it will receive eth0 and eth1 for its
+devices, but the bonding configuration tries to enslave eth2 and eth3
+(which may later be assigned to the tg3 devices).
+
+ Adding the following:
+
+add above bonding e1000 tg3
+
+ causes modprobe to load e1000 then tg3, in that order, when
+bonding is loaded. This command is fully documented in the
+modules.conf manual page.
+
+ On systems utilizing modprobe.conf (or modprobe.conf.local),
+an equivalent problem can occur. In this case, the following can be
+added to modprobe.conf (or modprobe.conf.local, as appropriate), as
+follows (all on one line; it has been split here for clarity):
+
+install bonding /sbin/modprobe tg3; /sbin/modprobe e1000;
+ /sbin/modprobe --ignore-install bonding
+
+ This will, when loading the bonding module, rather than
+performing the normal action, instead execute the provided command.
+This command loads the device drivers in the order needed, then calls
+modprobe with --ignore-install to cause the normal action to then take
+place. Full documentation on this can be found in the modprobe.conf
+and modprobe manual pages.
+
+8.3. Painfully Slow Or No Failed Link Detection By Miimon
+---------------------------------------------------------
+
+ By default, bonding enables the use_carrier option, which
+instructs bonding to trust the driver to maintain carrier state.
+
+ As discussed in the options section, above, some drivers do
+not support the netif_carrier_on/_off link state tracking system.
+With use_carrier enabled, bonding will always see these links as up,
+regardless of their actual state.
+
+ Additionally, other drivers do support netif_carrier, but do
+not maintain it in real time, e.g., only polling the link state at
+some fixed interval. In this case, miimon will detect failures, but
+only after some long period of time has expired. If it appears that
+miimon is very slow in detecting link failures, try specifying
+use_carrier=0 to see if that improves the failure detection time. If
+it does, then it may be that the driver checks the carrier state at a
+fixed interval, but does not cache the MII register values (so the
+use_carrier=0 method of querying the registers directly works). If
+use_carrier=0 does not improve the failover, then the driver may cache
+the registers, or the problem may be elsewhere.
+
+ Also, remember that miimon only checks for the device's
+carrier state. It has no way to determine the state of devices on or
+beyond other ports of a switch, or if a switch is refusing to pass
+traffic while still maintaining carrier on.
+
+9. SNMP agents
+===============
+
+ If running SNMP agents, the bonding driver should be loaded
+before any network drivers participating in a bond. This requirement
+is due to the interface index (ipAdEntIfIndex) being associated to
+the first interface found with a given IP address. That is, there is
+only one ipAdEntIfIndex for each IP address. For example, if eth0 and
+eth1 are slaves of bond0 and the driver for eth0 is loaded before the
+bonding driver, the interface for the IP address will be associated
+with the eth0 interface. This configuration is shown below, the IP
+address 192.168.1.1 has an interface index of 2 which indexes to eth0
+in the ifDescr table (ifDescr.2).
-9. Which transmit polices can be used?
+ interfaces.ifTable.ifEntry.ifDescr.1 = lo
+ interfaces.ifTable.ifEntry.ifDescr.2 = eth0
+ interfaces.ifTable.ifEntry.ifDescr.3 = eth1
+ interfaces.ifTable.ifEntry.ifDescr.4 = eth2
+ interfaces.ifTable.ifEntry.ifDescr.5 = eth3
+ interfaces.ifTable.ifEntry.ifDescr.6 = bond0
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
- Round-robin, based on the order of enslaving, the output device
- is selected base on the next available slave. Regardless of
- the source and/or destination of the packet.
+ This problem is avoided by loading the bonding driver before
+any network drivers participating in a bond. Below is an example of
+loading the bonding driver first, the IP address 192.168.1.1 is
+correctly associated with ifDescr.2.
- Active-backup policy that ensures that one and only one device will
- transmit at any given moment. Active-backup policy is useful for
- implementing high availability solutions using two hubs (see
- section on High Availability).
+ interfaces.ifTable.ifEntry.ifDescr.1 = lo
+ interfaces.ifTable.ifEntry.ifDescr.2 = bond0
+ interfaces.ifTable.ifEntry.ifDescr.3 = eth0
+ interfaces.ifTable.ifEntry.ifDescr.4 = eth1
+ interfaces.ifTable.ifEntry.ifDescr.5 = eth2
+ interfaces.ifTable.ifEntry.ifDescr.6 = eth3
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
+ ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
- XOR, based on (src hw addr XOR dst hw addr) % slave count. This
- policy selects the same slave for each destination hw address.
+ While some distributions may not report the interface name in
+ifDescr, the association between the IP address and IfIndex remains
+and SNMP functions such as Interface_Scan_Next will report that
+association.
- Broadcast policy transmits everything on all slave interfaces.
+10. Promiscuous mode
+====================
- 802.3ad, based on XOR but distributes traffic among all interfaces
- in the active aggregator.
+ When running network monitoring tools, e.g., tcpdump, it is
+common to enable promiscuous mode on the device, so that all traffic
+is seen (instead of seeing only traffic destined for the local host).
+The bonding driver handles promiscuous mode changes to the bonding
+master device (e.g., bond0), and propagates the setting to the slave
+devices.
- Transmit load balancing (balance-tlb) balances the traffic
- according to the current load on each slave. The balancing is
- clients based and the least loaded slave is selected for each new
- client. The load of each slave is calculated relative to its speed
- and enables load balancing in mixed speed teams.
+ For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
+the promiscuous mode setting is propagated to all slaves.
- Adaptive load balancing (balance-alb) uses the Transmit load
- balancing for the transmit load. The receive load is balanced only
- among the group of highest speed active slaves in the bond. The
- load is distributed with round-robin i.e. next available slave in
- the high speed group of active slaves.
+ For the active-backup, balance-tlb and balance-alb modes, the
+promiscuous mode setting is propagated only to the active slave.
-High Availability
-=================
+ For balance-tlb mode, the active slave is the slave currently
+receiving inbound traffic.
-To implement high availability using the bonding driver, the driver needs to be
-compiled as a module, because currently it is the only way to pass parameters
-to the driver. This may change in the future.
+ For balance-alb mode, the active slave is the slave used as a
+"primary." This slave is used for mode-specific control traffic, for
+sending to peers that are unassigned or if the load is unbalanced.
-High availability is achieved by using MII or ETHTOOL status reporting. You
-need to verify that all your interfaces support MII or ETHTOOL link status
-reporting. On Linux kernel 2.2.17, all the 100 Mbps capable drivers and
-yellowfin gigabit driver support MII. To determine if ETHTOOL link reporting
-is available for interface eth0, type "ethtool eth0" and the "Link detected:"
-line should contain the correct link status. If your system has an interface
-that does not support MII or ETHTOOL status reporting, a failure of its link
-will not be detected! A message indicating MII and ETHTOOL is not supported by
-a network driver is logged when the bonding driver is loaded with a non-zero
-miimon value.
+ For the active-backup, balance-tlb and balance-alb modes, when
+the active slave changes (e.g., due to a link failure), the
+promiscuous setting will be propagated to the new active slave.
-The bonding driver can regularly check all its slaves links using the ETHTOOL
-IOCTL (ETHTOOL_GLINK command) or by checking the MII status registers. The
-check interval is specified by the module argument "miimon" (MII monitoring).
-It takes an integer that represents the checking time in milliseconds. It
-should not come to close to (1000/HZ) (10 milli-seconds on i386) because it
-may then reduce the system interactivity. A value of 100 seems to be a good
-starting point. It means that a dead link will be detected at most 100
-milli-seconds after it goes down.
+11. Configuring Bonding for High Availability
+=============================================
-Example:
+ High Availability refers to configurations that provide
+maximum network availability by having redundant or backup devices,
+links or switches between the host and the rest of the world. The
+goal is to provide the maximum availability of network connectivity
+(i.e., the network always works), even though other configurations
+could provide higher throughput.
- # modprobe bonding miimon=100
+11.1 High Availability in a Single Switch Topology
+--------------------------------------------------
-Or, put the following line in /etc/modprobe.conf:
+ If two hosts (or a host and a single switch) are directly
+connected via multiple physical links, then there is no availability
+penalty to optimizing for maximum bandwidth. In this case, there is
+only one switch (or peer), so if it fails, there is no alternative
+access to fail over to. Additionally, the bonding load balance modes
+support link monitoring of their members, so if individual links fail,
+the load will be rebalanced across the remaining devices.
- options bond0 miimon=100
+ See Section 13, "Configuring Bonding for Maximum Throughput"
+for information on configuring bonding with one peer device.
-There are currently two policies for high availability. They are dependent on
-whether:
+11.2 High Availability in a Multiple Switch Topology
+----------------------------------------------------
- a) hosts are connected to a single host or switch that support trunking
+ With multiple switches, the configuration of bonding and the
+network changes dramatically. In multiple switch topologies, there is
+a trade off between network availability and usable bandwidth.
- b) hosts are connected to several different switches or a single switch that
- does not support trunking
+ Below is a sample network, configured to maximize the
+availability of the network:
+ | |
+ |port3 port3|
+ +-----+----+ +-----+----+
+ | |port2 ISL port2| |
+ | switch A +--------------------------+ switch B |
+ | | | |
+ +-----+----+ +-----++---+
+ |port1 port1|
+ | +-------+ |
+ +-------------+ host1 +---------------+
+ eth0 +-------+ eth1
-1) High Availability on a single switch or host - load balancing
+ In this configuration, there is a link between the two
+switches (ISL, or inter switch link), and multiple ports connecting to
+the outside world ("port3" on each switch). There is no technical
+reason that this could not be extended to a third switch.
+
+11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+-------------------------------------------------------------
+
+ In a topology such as the example above, the active-backup and
+broadcast modes are the only useful bonding modes when optimizing for
+availability; the other modes require all links to terminate on the
+same peer for them to behave rationally.
+
+active-backup: This is generally the preferred mode, particularly if
+ the switches have an ISL and play together well. If the
+ network configuration is such that one switch is specifically
+ a backup switch (e.g., has lower capacity, higher cost, etc),
+ then the primary option can be used to insure that the
+ preferred link is always used when it is available.
+
+broadcast: This mode is really a special purpose mode, and is suitable
+ only for very specific needs. For example, if the two
+ switches are not connected (no ISL), and the networks beyond
+ them are totally independent. In this case, if it is
+ necessary for some specific one-way traffic to reach both
+ independent networks, then the broadcast mode may be suitable.
+
+11.2.2 HA Link Monitoring Selection for Multiple Switch Topology
----------------------------------------------------------------
-It is the easiest to set up and to understand. Simply configure the
-remote equipment (host or switch) to aggregate traffic over several
-ports (Trunk, EtherChannel, etc.) and configure the bonding interfaces.
-If the module has been loaded with the proper MII option, it will work
-automatically. You can then try to remove and restore different links
-and see in your logs what the driver detects. When testing, you may
-encounter problems on some buggy switches that disable the trunk for a
-long time if all ports in a trunk go down. This is not Linux, but really
-the switch (reboot it to ensure).
-
-Example 1 : host to host at twice the speed
-
- +----------+ +----------+
- | |eth0 eth0| |
- | Host A +--------------------------+ Host B |
- | +--------------------------+ |
- | |eth1 eth1| |
- +----------+ +----------+
-
- On each host :
- # modprobe bonding miimon=100
- # ifconfig bond0 addr
- # ifenslave bond0 eth0 eth1
-Example 2 : host to switch at twice the speed
+ The choice of link monitoring ultimately depends upon your
+switch. If the switch can reliably fail ports in response to other
+failures, then either the MII or ARP monitors should work. For
+example, in the above example, if the "port3" link fails at the remote
+end, the MII monitor has no direct means to detect this. The ARP
+monitor could be configured with a target at the remote end of port3,
+thus detecting that failure without switch support.
+
+ In general, however, in a multiple switch topology, the ARP
+monitor can provide a higher level of reliability in detecting end to
+end connectivity failures (which may be caused by the failure of any
+individual component to pass traffic for any reason). Additionally,
+the ARP monitor should be configured with multiple targets (at least
+one for each switch in the network). This will insure that,
+regardless of which switch is active, the ARP monitor has a suitable
+target to query.
+
+
+12. Configuring Bonding for Maximum Throughput
+==============================================
+
+12.1 Maximizing Throughput in a Single Switch Topology
+------------------------------------------------------
+
+ In a single switch configuration, the best method to maximize
+throughput depends upon the application and network environment. The
+various load balancing modes each have strengths and weaknesses in
+different environments, as detailed below.
+
+ For this discussion, we will break down the topologies into
+two categories. Depending upon the destination of most traffic, we
+categorize them into either "gatewayed" or "local" configurations.
+
+ In a gatewayed configuration, the "switch" is acting primarily
+as a router, and the majority of traffic passes through this router to
+other networks. An example would be the following:
+
+
+ +----------+ +----------+
+ | |eth0 port1| | to other networks
+ | Host A +---------------------+ router +------------------->
+ | +---------------------+ | Hosts B and C are out
+ | |eth1 port2| | here somewhere
+ +----------+ +----------+
+
+ The router may be a dedicated router device, or another host
+acting as a gateway. For our discussion, the important point is that
+the majority of traffic from Host A will pass through the router to
+some other network before reaching its final destination.
+
+ In a gatewayed network configuration, although Host A may
+communicate with many other systems, all of its traffic will be sent
+and received via one other peer on the local network, the router.
+
+ Note that the case of two systems connected directly via
+multiple physical links is, for purposes of configuring bonding, the
+same as a gatewayed configuration. In that case, it happens that all
+traffic is destined for the "gateway" itself, not some other network
+beyond the gateway.
+
+ In a local configuration, the "switch" is acting primarily as
+a switch, and the majority of traffic passes through this switch to
+reach other stations on the same network. An example would be the
+following:
+
+ +----------+ +----------+ +--------+
+ | |eth0 port1| +-------+ Host B |
+ | Host A +------------+ switch |port3 +--------+
+ | +------------+ | +--------+
+ | |eth1 port2| +------------------+ Host C |
+ +----------+ +----------+port4 +--------+
+
+
+ Again, the switch may be a dedicated switch device, or another
+host acting as a gateway. For our discussion, the important point is
+that the majority of traffic from Host A is destined for other hosts
+on the same local network (Hosts B and C in the above example).
+
+ In summary, in a gatewayed configuration, traffic to and from
+the bonded device will be to the same MAC level peer on the network
+(the gateway itself, i.e., the router), regardless of its final
+destination. In a local configuration, traffic flows directly to and
+from the final destinations, thus, each destination (Host B, Host C)
+will be addressed directly by their individual MAC addresses.
+
+ This distinction between a gatewayed and a local network
+configuration is important because many of the load balancing modes
+available use the MAC addresses of the local network source and
+destination to make load balancing decisions. The behavior of each
+mode is described below.
+
+
+12.1.1 MT Bonding Mode Selection for Single Switch Topology
+-----------------------------------------------------------
+
+ This configuration is the easiest to set up and to understand,
+although you will have to decide which bonding mode best suits your
+needs. The trade offs for each mode are detailed below:
+
+balance-rr: This mode is the only mode that will permit a single
+ TCP/IP connection to stripe traffic across multiple
+ interfaces. It is therefore the only mode that will allow a
+ single TCP/IP stream to utilize more than one interface's
+ worth of throughput. This comes at a cost, however: the
+ striping often results in peer systems receiving packets out
+ of order, causing TCP/IP's congestion control system to kick
+ in, often by retransmitting segments.
+
+ It is possible to adjust TCP/IP's congestion limits by
+ altering the net.ipv4.tcp_reordering sysctl parameter. The
+ usual default value is 3, and the maximum useful value is 127.
+ For a four interface balance-rr bond, expect that a single
+ TCP/IP stream will utilize no more than approximately 2.3
+ interface's worth of throughput, even after adjusting
+ tcp_reordering.
+
+ Note that this out of order delivery occurs when both the
+ sending and receiving systems are utilizing a multiple
+ interface bond. Consider a configuration in which a
+ balance-rr bond feeds into a single higher capacity network
+ channel (e.g., multiple 100Mb/sec ethernets feeding a single
+ gigabit ethernet via an etherchannel capable switch). In this
+ configuration, traffic sent from the multiple 100Mb devices to
+ a destination connected to the gigabit device will not see
+ packets out of order. However, traffic sent from the gigabit
+ device to the multiple 100Mb devices may or may not see
+ traffic out of order, depending upon the balance policy of the
+ switch. Many switches do not support any modes that stripe
+ traffic (instead choosing a port based upon IP or MAC level
+ addresses); for those devices, traffic flowing from the
+ gigabit device to the many 100Mb devices will only utilize one
+ interface.
+
+ If you are utilizing protocols other than TCP/IP, UDP for
+ example, and your application can tolerate out of order
+ delivery, then this mode can allow for single stream datagram
+ performance that scales near linearly as interfaces are added
+ to the bond.
+
+ This mode requires the switch to have the appropriate ports
+ configured for "etherchannel" or "trunking."
+
+active-backup: There is not much advantage in this network topology to
+ the active-backup mode, as the inactive backup devices are all
+ connected to the same peer as the primary. In this case, a
+ load balancing mode (with link monitoring) will provide the
+ same level of network availability, but with increased
+ available bandwidth. On the plus side, active-backup mode
+ does not require any configuration of the switch, so it may
+ have value if the hardware available does not support any of
+ the load balance modes.
+
+balance-xor: This mode will limit traffic such that packets destined
+ for specific peers will always be sent over the same
+ interface. Since the destination is determined by the MAC
+ addresses involved, this mode works best in a "local" network
+ configuration (as described above), with destinations all on
+ the same local network. This mode is likely to be suboptimal
+ if all your traffic is passed through a single router (i.e., a
+ "gatewayed" network configuration, as described above).
+
+ As with balance-rr, the switch ports need to be configured for
+ "etherchannel" or "trunking."
+
+broadcast: Like active-backup, there is not much advantage to this
+ mode in this type of network topology.
+
+802.3ad: This mode can be a good choice for this type of network
+ topology. The 802.3ad mode is an IEEE standard, so all peers
+ that implement 802.3ad should interoperate well. The 802.3ad
+ protocol includes automatic configuration of the aggregates,
+ so minimal manual configuration of the switch is needed
+ (typically only to designate that some set of devices is
+ available for 802.3ad). The 802.3ad standard also mandates
+ that frames be delivered in order (within certain limits), so
+ in general single connections will not see misordering of
+ packets. The 802.3ad mode does have some drawbacks: the
+ standard mandates that all devices in the aggregate operate at
+ the same speed and duplex. Also, as with all bonding load
+ balance modes other than balance-rr, no single connection will
+ be able to utilize more than a single interface's worth of
+ bandwidth.
+
+ Additionally, the linux bonding 802.3ad implementation
+ distributes traffic by peer (using an XOR of MAC addresses),
+ so in a "gatewayed" configuration, all outgoing traffic will
+ generally use the same device. Incoming traffic may also end
+ up on a single device, but that is dependent upon the
+ balancing policy of the peer's 8023.ad implementation. In a
+ "local" configuration, traffic will be distributed across the
+ devices in the bond.
+
+ Finally, the 802.3ad mode mandates the use of the MII monitor,
+ therefore, the ARP monitor is not available in this mode.
+
+balance-tlb: The balance-tlb mode balances outgoing traffic by peer.
+ Since the balancing is done according to MAC address, in a
+ "gatewayed" configuration (as described above), this mode will
+ send all traffic across a single device. However, in a
+ "local" network configuration, this mode balances multiple
+ local network peers across devices in a vaguely intelligent
+ manner (not a simple XOR as in balance-xor or 802.3ad mode),
+ so that mathematically unlucky MAC addresses (i.e., ones that
+ XOR to the same value) will not all "bunch up" on a single
+ interface.
+
+ Unlike 802.3ad, interfaces may be of differing speeds, and no
+ special switch configuration is required. On the down side,
+ in this mode all incoming traffic arrives over a single
+ interface, this mode requires certain ethtool support in the
+ network device driver of the slave interfaces, and the ARP
+ monitor is not available.
+
+balance-alb: This mode is everything that balance-tlb is, and more.
+ It has all of the features (and restrictions) of balance-tlb,
+ and will also balance incoming traffic from local network
+ peers (as described in the Bonding Module Options section,
+ above).
+
+ The only additional down side to this mode is that the network
+ device driver must support changing the hardware address while
+ the device is open.
+
+12.1.2 MT Link Monitoring for Single Switch Topology
+----------------------------------------------------
+
+ The choice of link monitoring may largely depend upon which
+mode you choose to use. The more advanced load balancing modes do not
+support the use of the ARP monitor, and are thus restricted to using
+the MII monitor (which does not provide as high a level of end to end
+assurance as the ARP monitor).
+
+12.2 Maximum Throughput in a Multiple Switch Topology
+-----------------------------------------------------
+
+ Multiple switches may be utilized to optimize for throughput
+when they are configured in parallel as part of an isolated network
+between two or more systems, for example:
+
+ +-----------+
+ | Host A |
+ +-+---+---+-+
+ | | |
+ +--------+ | +---------+
+ | | |
+ +------+---+ +-----+----+ +-----+----+
+ | Switch A | | Switch B | | Switch C |
+ +------+---+ +-----+----+ +-----+----+
+ | | |
+ +--------+ | +---------+
+ | | |
+ +-+---+---+-+
+ | Host B |
+ +-----------+
+
+ In this configuration, the switches are isolated from one
+another. One reason to employ a topology such as this is for an
+isolated network with many hosts (a cluster configured for high
+performance, for example), using multiple smaller switches can be more
+cost effective than a single larger switch, e.g., on a network with 24
+hosts, three 24 port switches can be significantly less expensive than
+a single 72 port switch.
+
+ If access beyond the network is required, an individual host
+can be equipped with an additional network device connected to an
+external network; this host then additionally acts as a gateway.
+
+12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
+-------------------------------------------------------------
+
+ In actual practice, the bonding mode typically employed in
+configurations of this type is balance-rr. Historically, in this
+network configuration, the usual caveats about out of order packet
+delivery are mitigated by the use of network adapters that do not do
+any kind of packet coalescing (via the use of NAPI, or because the
+device itself does not generate interrupts until some number of
+packets has arrived). When employed in this fashion, the balance-rr
+mode allows individual connections between two hosts to effectively
+utilize greater than one interface's bandwidth.
+
+12.2.2 MT Link Monitoring for Multiple Switch Topology
+------------------------------------------------------
+
+ Again, in actual practice, the MII monitor is most often used
+in this configuration, as performance is given preference over
+availability. The ARP monitor will function in this topology, but its
+advantages over the MII monitor are mitigated by the volume of probes
+needed as the number of systems involved grows (remember that each
+host in the network is configured with bonding).
+
+13. Switch Behavior Issues
+==========================
- +----------+ +----------+
- | |eth0 port1| |
- | Host A +--------------------------+ switch |
- | +--------------------------+ |
- | |eth1 port2| |
- +----------+ +----------+
+13.1 Link Establishment and Failover Delays
+-------------------------------------------
+
+ Some switches exhibit undesirable behavior with regard to the
+timing of link up and down reporting by the switch.
+
+ First, when a link comes up, some switches may indicate that
+the link is up (carrier available), but not pass traffic over the
+interface for some period of time. This delay is typically due to
+some type of autonegotiation or routing protocol, but may also occur
+during switch initialization (e.g., during recovery after a switch
+failure). If you find this to be a problem, specify an appropriate
+value to the updelay bonding module option to delay the use of the
+relevant interface(s).
+
+ Second, some switches may "bounce" the link state one or more
+times while a link is changing state. This occurs most commonly while
+the switch is initializing. Again, an appropriate updelay value may
+help.
+
+ Note that when a bonding interface has no active links, the
+driver will immediately reuse the first link that goes up, even if the
+updelay parameter has been specified (the updelay is ignored in this
+case). If there are slave interfaces waiting for the updelay timeout
+to expire, the interface that first went into that state will be
+immediately reused. This reduces down time of the network if the
+value of updelay has been overestimated, and since this occurs only in
+cases with no connectivity, there is no additional penalty for
+ignoring the updelay.
+
+ In addition to the concerns about switch timings, if your
+switches take a long time to go into backup mode, it may be desirable
+to not activate a backup interface immediately after a link goes down.
+Failover may be delayed via the downdelay bonding module option.
+
+13.2 Duplicated Incoming Packets
+--------------------------------
+
+ It is not uncommon to observe a short burst of duplicated
+traffic when the bonding device is first used, or after it has been
+idle for some period of time. This is most easily observed by issuing
+a "ping" to some other host on the network, and noticing that the
+output from ping flags duplicates (typically one per slave).
+
+ For example, on a bond in active-backup mode with five slaves
+all connected to one switch, the output may appear as follows:
+
+# ping -n 10.0.4.2
+PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data.
+64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms
+64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
+64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
+64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
+64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
+64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms
+64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms
+64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms
+
+ This is not due to an error in the bonding driver, rather, it
+is a side effect of how many switches update their MAC forwarding
+tables. Initially, the switch does not associate the MAC address in
+the packet with a particular switch port, and so it may send the
+traffic to all ports until its MAC forwarding table is updated. Since
+the interfaces attached to the bond may occupy multiple ports on a
+single switch, when the switch (temporarily) floods the traffic to all
+ports, the bond device receives multiple copies of the same packet
+(one per slave device).
+
+ The duplicated packet behavior is switch dependent, some
+switches exhibit this, and some do not. On switches that display this
+behavior, it can be induced by clearing the MAC forwarding table (on
+most Cisco switches, the privileged command "clear mac address-table
+dynamic" will accomplish this).
+
+14. Hardware Specific Considerations
+====================================
+
+ This section contains additional information for configuring
+bonding on specific hardware platforms, or for interfacing bonding
+with particular switches or other devices.
+
+14.1 IBM BladeCenter
+--------------------
+
+ This applies to the JS20 and similar systems.
+
+ On the JS20 blades, the bonding driver supports only
+balance-rr, active-backup, balance-tlb and balance-alb modes. This is
+largely due to the network topology inside the BladeCenter, detailed
+below.
+
+JS20 network adapter information
+--------------------------------
+
+ All JS20s come with two Broadcom Gigabit Ethernet ports
+integrated on the planar (that's "motherboard" in IBM-speak). In the
+BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
+I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
+An add-on Broadcom daughter card can be installed on a JS20 to provide
+two more Gigabit Ethernet ports. These ports, eth2 and eth3, are
+wired to I/O Modules 3 and 4, respectively.
+
+ Each I/O Module may contain either a switch or a passthrough
+module (which allows ports to be directly connected to an external
+switch). Some bonding modes require a specific BladeCenter internal
+network topology in order to function; these are detailed below.
+
+ Additional BladeCenter-specific networking information can be
+found in two IBM Redbooks (www.ibm.com/redbooks):
+
+"IBM eServer BladeCenter Networking Options"
+"IBM eServer BladeCenter Layer 2-7 Network Switching"
+
+BladeCenter networking configuration
+------------------------------------
- On host A : On the switch :
- # modprobe bonding miimon=100 # set up a trunk on port1
- # ifconfig bond0 addr and port2
- # ifenslave bond0 eth0 eth1
+ Because a BladeCenter can be configured in a very large number
+of ways, this discussion will be confined to describing basic
+configurations.
+
+ Normally, Ethernet Switch Modules (ESMs) are used in I/O
+modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
+JS20 will be connected to different internal switches (in the
+respective I/O modules).
+
+ A passthrough module (OPM or CPM, optical or copper,
+passthrough module) connects the I/O module directly to an external
+switch. By using PMs in I/O module #1 and #2, the eth0 and eth1
+interfaces of a JS20 can be redirected to the outside world and
+connected to a common external switch.
+
+ Depending upon the mix of ESMs and PMs, the network will
+appear to bonding as either a single switch topology (all PMs) or as a
+multiple switch topology (one or more ESMs, zero or more PMs). It is
+also possible to connect ESMs together, resulting in a configuration
+much like the example in "High Availability in a Multiple Switch
+Topology," above.
+
+Requirements for specific modes
+-------------------------------
+
+ The balance-rr mode requires the use of passthrough modules
+for devices in the bond, all connected to an common external switch.
+That switch must be configured for "etherchannel" or "trunking" on the
+appropriate ports, as is usual for balance-rr.
+
+ The balance-alb and balance-tlb modes will function with
+either switch modules or passthrough modules (or a mix). The only
+specific requirement for these modes is that all network interfaces
+must be able to reach all destinations for traffic sent over the
+bonding device (i.e., the network must converge at some point outside
+the BladeCenter).
+
+ The active-backup mode has no additional requirements.
+
+Link monitoring issues
+----------------------
+
+ When an Ethernet Switch Module is in place, only the ARP
+monitor will reliably detect link loss to an external switch. This is
+nothing unusual, but examination of the BladeCenter cabinet would
+suggest that the "external" network ports are the ethernet ports for
+the system, when it fact there is a switch between these "external"
+ports and the devices on the JS20 system itself. The MII monitor is
+only able to detect link failures between the ESM and the JS20 system.
+
+ When a passthrough module is in place, the MII monitor does
+detect failures to the "external" port, which is then directly
+connected to the JS20 system.
+
+Other concerns
+--------------
+
+ The Serial Over LAN (SoL) link is established over the primary
+ethernet (eth0) only, therefore, any loss of link to eth0 will result
+in losing your SoL connection. It will not fail over with other
+network traffic, as the SoL system is beyond the control of the
+bonding driver.
+
+ It may be desirable to disable spanning tree on the switch
+(either the internal Ethernet Switch Module, or an external switch) to
+avoid fail-over delay issues when using bonding.
+
+
+15. Frequently Asked Questions
+==============================
+1. Is it SMP safe?
-2) High Availability on two or more switches (or a single switch without
- trunking support)
----------------------------------------------------------------------------
-This mode is more problematic because it relies on the fact that there
-are multiple ports and the host's MAC address should be visible on one
-port only to avoid confusing the switches.
+ Yes. The old 2.0.xx channel bonding patch was not SMP safe.
+The new driver was designed to be SMP safe from the start.
-If you need to know which interface is the active one, and which ones are
-backup, use ifconfig. All backup interfaces have the NOARP flag set.
+2. What type of cards will work with it?
-To use this mode, pass "mode=1" to the module at load time :
+ Any Ethernet type cards (you can even mix cards - a Intel
+EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
+devices need not be of the same speed.
- # modprobe bonding miimon=100 mode=active-backup
+3. How many bonding devices can I have?
- or:
+ There is no limit.
- # modprobe bonding miimon=100 mode=1
+4. How many slaves can a bonding device have?
-Or, put in your /etc/modprobe.conf :
+ This is limited only by the number of network interfaces Linux
+supports and/or the number of network cards you can place in your
+system.
- options bond0 miimon=100 mode=active-backup
+5. What happens when a slave link dies?
-Example 1: Using multiple host and multiple switches to build a "no single
-point of failure" solution.
+ If link monitoring is enabled, then the failing device will be
+disabled. The active-backup mode will fail over to a backup link, and
+other modes will ignore the failed link. The link will continue to be
+monitored, and should it recover, it will rejoin the bond (in whatever
+manner is appropriate for the mode). See the sections on High
+Availability and the documentation for each mode for additional
+information.
+
+ Link monitoring can be enabled via either the miimon or
+arp_interval parameters (described in the module parameters section,
+above). In general, miimon monitors the carrier state as sensed by
+the underlying network device, and the arp monitor (arp_interval)
+monitors connectivity to another host on the local network.
+
+ If no link monitoring is configured, the bonding driver will
+be unable to detect link failures, and will assume that all links are
+always available. This will likely result in lost packets, and a
+resulting degradation of performance. The precise performance loss
+depends upon the bonding mode and network configuration.
+6. Can bonding be used for High Availability?
- | |
- |port3 port3|
- +-----+----+ +-----+----+
- | |port7 ISL port7| |
- | switch A +--------------------------+ switch B |
- | +--------------------------+ |
- | |port8 port8| |
- +----++----+ +-----++---+
- port2||port1 port1||port2
- || +-------+ ||
- |+-------------+ host1 +---------------+|
- | eth0 +-------+ eth1 |
- | |
- | +-------+ |
- +--------------+ host2 +----------------+
- eth0 +-------+ eth1
+ Yes. See the section on High Availability for details.
-In this configuration, there is an ISL - Inter Switch Link (could be a trunk),
-several servers (host1, host2 ...) attached to both switches each, and one or
-more ports to the outside world (port3...). One and only one slave on each host
-is active at a time, while all links are still monitored (the system can
-detect a failure of active and backup links).
+7. Which switches/systems does it work with?
-Each time a host changes its active interface, it sticks to the new one until
-it goes down. In this example, the hosts are negligibly affected by the
-expiration time of the switches' forwarding tables.
+ The full answer to this depends upon the desired mode.
-If host1 and host2 have the same functionality and are used in load balancing
-by another external mechanism, it is good to have host1's active interface
-connected to one switch and host2's to the other. Such system will survive
-a failure of a single host, cable, or switch. The worst thing that may happen
-in the case of a switch failure is that half of the hosts will be temporarily
-unreachable until the other switch expires its tables.
+ In the basic balance modes (balance-rr and balance-xor), it
+works with any system that supports etherchannel (also called
+trunking). Most managed switches currently available have such
+support, and many unmanaged switches as well.
-Example 2: Using multiple ethernet cards connected to a switch to configure
- NIC failover (switch is not required to support trunking).
+ The advanced balance modes (balance-tlb and balance-alb) do
+not have special switch requirements, but do need device drivers that
+support specific features (described in the appropriate section under
+module parameters, above).
+ In 802.3ad mode, it works with systems that support IEEE
+802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
+switches currently available support 802.3ad.
- +----------+ +----------+
- | |eth0 port1| |
- | Host A +--------------------------+ switch |
- | +--------------------------+ |
- | |eth1 port2| |
- +----------+ +----------+
+ The active-backup mode should work with any Layer-II switch.
- On host A : On the switch :
- # modprobe bonding miimon=100 mode=1 # (optional) minimize the time
- # ifconfig bond0 addr # for table expiration
- # ifenslave bond0 eth0 eth1
+8. Where does a bonding device get its MAC address from?
-Each time the host changes its active interface, it sticks to the new one until
-it goes down. In this example, the host is strongly affected by the expiration
-time of the switch forwarding table.
+ If not explicitly configured (with ifconfig or ip link), the
+MAC address of the bonding device is taken from its first slave
+device. This MAC address is then passed to all following slaves and
+remains persistent (even if the first slave is removed) until the
+bonding device is brought down or reconfigured.
+ If you wish to change the MAC address, you can set it with
+ifconfig or ip link:
-3) Adapting to your switches' timing
-------------------------------------
-If your switches take a long time to go into backup mode, it may be
-desirable not to activate a backup interface immediately after a link goes
-down. It is possible to delay the moment at which a link will be
-completely disabled by passing the module parameter "downdelay" (in
-milliseconds, must be a multiple of miimon).
+# ifconfig bond0 hw ether 00:11:22:33:44:55
-When a switch reboots, it is possible that its ports report "link up" status
-before they become usable. This could fool a bond device by causing it to
-use some ports that are not ready yet. It is possible to delay the moment at
-which an active link will be reused by passing the module parameter "updelay"
-(in milliseconds, must be a multiple of miimon).
+# ip link set bond0 address 66:77:88:99:aa:bb
-A similar situation can occur when a host re-negotiates a lost link with the
-switch (a case of cable replacement).
+ The MAC address can be also changed by bringing down/up the
+device and then changing its slaves (or their order):
-A special case is when a bonding interface has lost all slave links. Then the
-driver will immediately reuse the first link that goes up, even if updelay
-parameter was specified. (If there are slave interfaces in the "updelay" state,
-the interface that first went into that state will be immediately reused.) This
-allows to reduce down-time if the value of updelay has been overestimated.
+# ifconfig bond0 down ; modprobe -r bonding
+# ifconfig bond0 .... up
+# ifenslave bond0 eth...
-Examples :
+ This method will automatically take the address from the next
+slave that is added.
- # modprobe bonding miimon=100 mode=1 downdelay=2000 updelay=5000
- # modprobe bonding miimon=100 mode=balance-rr downdelay=0 updelay=5000
+ To restore your slaves' MAC addresses, you need to detach them
+from the bond (`ifenslave -d bond0 eth0'). The bonding driver will
+then restore the MAC addresses that the slaves had before they were
+enslaved.
+16. Resources and Links
+=======================
-Promiscuous Sniffing notes
-==========================
+The latest version of the bonding driver can be found in the latest
+version of the linux kernel, found on http://kernel.org
-If you wish to bond channels together for a network sniffing
-application --- you wish to run tcpdump, or ethereal, or an IDS like
-snort, with its input aggregated from multiple interfaces using the
-bonding driver --- then you need to handle the Promiscuous interface
-setting by hand. Specifically, when you "ifconfing bond0 up" you
-must add the promisc flag there; it will be propagated down to the
-slave interfaces at ifenslave time; a full example might look like:
+The latest version of this document can be found in either the latest
+kernel source (named Documentation/networking/bonding.txt), or on the
+bonding sourceforge site:
- ifconfig bond0 promisc up
- for if in eth1 eth2 ...;do
- ifconfig $if up
- ifenslave bond0 $if
- done
- snort ... -i bond0 ...
+http://www.sourceforge.net/projects/bonding
-Ifenslave also wants to propagate addresses from interface to
-interface, appropriately for its design functions in HA and channel
-capacity aggregating; but it works fine for unnumbered interfaces;
-just ignore all the warnings it emits.
+Discussions regarding the bonding driver take place primarily on the
+bonding-devel mailing list, hosted at sourceforge.net. If you have
+questions or problems, post them to the list. The list address is:
+bonding-devel@lists.sourceforge.net
-8021q VLAN support
-==================
+ The administrative interface (to subscribe or unsubscribe) can
+be found at:
-It is possible to configure VLAN devices over a bond interface using the 8021q
-driver. However, only packets coming from the 8021q driver and passing through
-bonding will be tagged by default. Self generated packets, like bonding's
-learning packets or ARP packets generated by either ALB mode or the ARP
-monitor mechanism, are tagged internally by bonding itself. As a result,
-bonding has to "learn" what VLAN IDs are configured on top of it, and it uses
-those IDs to tag self generated packets.
-
-For simplicity reasons, and to support the use of adapters that can do VLAN
-hardware acceleration offloding, the bonding interface declares itself as
-fully hardware offloaing capable, it gets the add_vid/kill_vid notifications
-to gather the necessary information, and it propagates those actions to the
-slaves.
-In case of mixed adapter types, hardware accelerated tagged packets that should
-go through an adapter that is not offloading capable are "un-accelerated" by the
-bonding driver so the VLAN tag sits in the regular location.
-
-VLAN interfaces *must* be added on top of a bonding interface only after
-enslaving at least one slave. This is because until the first slave is added the
-bonding interface has a HW address of 00:00:00:00:00:00, which will be copied by
-the VLAN interface when it is created.
-
-Notice that a problem would occur if all slaves are released from a bond that
-still has VLAN interfaces on top of it. When later coming to add new slaves, the
-bonding interface would get a HW address from the first slave, which might not
-match that of the VLAN interfaces. It is recommended that either all VLANs are
-removed and then re-added, or to manually set the bonding interface's HW
-address so it matches the VLAN's. (Note: changing a VLAN interface's HW address
-would set the underlying device -- i.e. the bonding interface -- to promiscouos
-mode, which might not be what you want).
-
-
-Limitations
-===========
-The main limitations are :
- - only the link status is monitored. If the switch on the other side is
- partially down (e.g. doesn't forward anymore, but the link is OK), the link
- won't be disabled. Another way to check for a dead link could be to count
- incoming frames on a heavily loaded host. This is not applicable to small
- servers, but may be useful when the front switches send multicast
- information on their links (e.g. VRRP), or even health-check the servers.
- Use the arp_interval/arp_ip_target parameters to count incoming/outgoing
- frames.
-
-
-
-Resources and Links
-===================
-
-Current development on this driver is posted to:
- - http://www.sourceforge.net/projects/bonding/
+https://lists.sourceforge.net/lists/listinfo/bonding-devel
Donald Becker's Ethernet Drivers and diag programs may be found at :
- http://www.scyld.com/network/
-You will also find a lot of information regarding Ethernet, NWay, MII, etc. at
-www.scyld.com.
-
-Patches for 2.2 kernels are at Willy Tarreau's site :
- - http://wtarreau.free.fr/pub/bonding/
- - http://www-miaif.lip6.fr/~tarreau/pub/bonding/
-
-To get latest informations about Linux Kernel development, please consult
-the Linux Kernel Mailing List Archives at :
- http://www.ussg.iu.edu/hypermail/linux/kernel/
+You will also find a lot of information regarding Ethernet, NWay, MII,
+etc. at www.scyld.com.
-- END --
git://git.onelab.eu / linux-2.6.git / blobdiff