The Education of a SysAdmin: Bonded, Vlan'ed, NICs with Multiple Assigned IPs on a RHEL 6.4 Server with LACP Protocol Enabled

This document describes how to set up multiple NICs on a server, into a bonded NIC that attaches to multiple VLANs with multiple IPs per VLAN.

Much of this process was intuitive from a high-level point of view, but the devil was in the details. Getting the correct key=value pairs in each file proved to be more difficult than intuition alone could solve. There were many examples on the Internet for setting up bonded interfaces, attaching a bonded interface to a vlan, assiging more than one IP to an interface or a bonded interface. I was not able to find a document that described setting up all three at once. The initial problem was to decide which comes first, vlans or IPs, when giving a value to DEVICE in the ifcfg files.

So, here is the solution I worked out piecing together various documents from the web and getting a bit of help from NetworkManager. BTW: the solution that NetworkManager came up with did not work either, but it did get me closer to my goal.

To facilitate the description I will use an example network setup.

1. Large corporate network cloud.
2. Two of the many vlans in the cloud we need to connect to: vlan 236 & vland 723
3. VLAN 236 is set up for network 136.159.57.0/24 - 136.159.57.113 & 114 255.255.255.0
4. VLAN 723 is set up for network 21.152.120.0/21 - 21.152.125.78 & 79 255.255.248.0
5. A server with multiple Ethernet and Infiniband NICs.

This is our desired configuration:
* ethX + ethY = bond0
* bond0 has 4 IPs. 2 IPs attach to one vlan. the other 2 IPs attach to the second vlan.
* bond0 also has to send LACP packets to the switch to help avoid looping.
* vlan 236, ip 136.159.57.113
* vlan 236, ip 136.159.57.114
* vlan 723, ip 21.152.125.78
* vlan 723, ip 21.152.125.79

Ethernet 5 & 7 are on physically separate NICs and will be used to create the bonded interface.

1. Create the bondX configuration file replacing X with a appropriate number. For this example we have no other bonded interfaces, so the file will be named ifcfg-bond0. You can name this file pretty much whatever you wanted to: ifcfg-bondX, ifcfg-bond-foobar, ifcfg-bond007, etc.

cd /etc/sysconfig/network-scripts
vi ifcfg-bond0, and input the following contents:

DEVICE=bond0
NAME=bond0
TYPE=Bond
ONBOOT=yes
BONDING_MASTER=yes
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NM_CONTROLLED=no
BONDING_OPTS="mode=4 updelay=200 miimon=100 downdelay=200"
PEERDNS=yes
PEERROUTES=yes

We are using mode=4 here to have the interface send LACP packets to the switch.

This is a typical bond interface minus specifics for an IP address, but as I mentioned before, the devil is in the details, Here are the important lines to pay particular attention to: TYPE, BONDING_MASTER, DEFROUTE, BONDING, PEERDNS, and PEERROUTES.

2. Set up the config file for NIC 5 & 7 to be slaves to bond0

vi ifcfg-eth5, and edit it to have the following entries

DEVICE=eth5
HWADDR="38:EA:A7:90:30:79"
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
USERCTL=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

vi ifcfg-eth7, and edit it to have the following entries

DEVICE=eth7
HWADDR="38:EA:A7:90:32:65"
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPVINIT=no
USERCTL=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

So far this is an almost normal setup for bonding two NICs together. In fact the actual device ifcfg files look exactly like other examples found in manuals and on the Internet in regard to bonding.

We did not assign an IP address to bond0 because we need 4 total IPs setup on this one bond interface. The IPs will be configure next along with the vlan assignments. Normally this is done on the individual NIC files. For example ifcfg-eth5 might be setup for vlan 236 by have a DEVICE= line that
indicated eth5.236, so that it would attach to vlan 236. In this case we want to set up the bond0 interface to attach to a couple of vlans.

Starting with vlan 236 and IP address 136.159.57.113, this is how the ifcfg file is setup:

vi ifcfg-vlan236-ip113

VLAN=yes
TYPE=Vlan
DEVICE=bond0.236
PHYSDEV=bond0
VLAN_ID=236
REORDER_HDR=0
BOOTPROTO=none
IPADDR=136.159.57.113
PREFIX=24
GATEWAY=136.159.57.1
NETWORK=136.159.57.0
BROADCAST=136.159.57.255
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=vlan236-ip113
ONBOOT=yes
NM_CONTROLLED=no

Notice that the filename and the DEVICE line do not need to match. This is a typically held myth by many SysAdmins including myself up to this point.

Notice the following lines: VLAN, TYPE, DEVICE, PHYSDEV, VLAN_ID, DEFROUTE, & NAME.

Next I added the config file for the second vlan with the first of the pair of IPs to be assigned to it. Even without some of the important key=value pairs defined, I was able to get this configuration to work several times based on intuition and a couple examples from the Internet. However, for clarity, I am going to keep list the files by vlan, not in the order I figured things out.

So, the difficult part comes next: adding that second IP to the vlan. Do I define the IP first or reference the vlan first? What other key=value pairs need to be in the file, and what values do they get? Look at the config for the second IP to be assigned, and attached to vlan 236.

cat ifcfg-vlan236-ip114

VLAN=yes
TYPE=Vlan
DEVICE=bond0.236:1
PHYSDEV=bond0
VLAN_ID=236
REORDER_HDR=0
BOOTPROTO=none
IPADDR=136.159.57.114
PREFIX=24
GATEWAY=136.159.57.1
NETWORK=136.159.57.0
BROADCAST=136.159.57.255
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=vlan236-ip114
UUID=abfb74e6-9b4a-46df-b98a-756740699f71
ONBOOT=yes
NM_CONTROLLED=no

Again, take note of the following lines and how they did or did not change from ifcfg-vlan236-ip113: VLAN, TYPE, DEVICE, PHYSDEV, VLAN_ID, DEFROUTE, & NAME.

Now to heap more difficulty we add the second vlan and second pair of IPs. Again, I was able to get the initial vlan/IP config working, but it was the second IP that was troublesome. Here are the ifcfg files for the next two IPs on vlan 723.

cat ifcfg-vlan723-ip78

VLAN=yes
TYPE=Vlan
DEVICE=bond0.723
PHYSDEV=bond0
VLAN_ID=723
REORDER_HDR=0
BOOTPROTO=none
IPADDR=21.152.125.78
PREFIX=21
GATEWAY=21.152.120.1
NETWORK=21.152.120.0
BROADCAST=21.152.127.255
DEFROUTE=no

IPV4_FAILURE_FATAL=no

IPV6INIT=no

NAME=vlan723-ip78

ONBOOT=yes

NM_CONTROLLED=no

cat ifcfg-vlan723-ip79

VLAN=yes

TYPE=Vlan

DEVICE=bond0.723:1

PHYSDEV=bond0

VLAN_ID=723

REORDER_HDR=0

BOOTPROTO=none

IPADDR=21.152.125.79

PREFIX=21

GATEWAY=21.152.120.1

NETWORK=21.152.120.0

BROADCAST=21.152.127.255

DEFROUTE=no

IPV4_FAILURE_FATAL=no

IPV6INIT=no

NAME=vlan723-ip79

ONBOOT=yes

NM_CONTROLLED=no

Below is what the devices look like when up and running:

ip link show

7: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master

bond0 state UP qlen 1000