In one of my last posts, I was writing about my CCIE home rack which has one server that runs Ubuntu + Ethernet Quad cards + Cisco switches. Before connecting the cables between server and Cisco switches, I had one problem that was driving me crazy.
I have three Ethernet Quad cards connected in 3 PCI slots in my server. The issue is that sometimes (quite often) the port numbers were changing during the reboot of my client. To give you and example, an ethernet port that was eth1 during one server boot could change to eth2 next time. For twelve ports you can imagine what mess this creates after almost each server reboot. When this was happening, the ports were not matched correctly in Dynamips and from it would result in a lot of connectivity problems.
I will try to explain in a few word why this is happening. With Udev and modular network drivers, the network interface numbering is not persistent across reboots by default, because the drivers are loaded in parallel and, thus, in random order. For example, on a computer having two network cards made by Intel and Realtek, the network card manufactured by Intel may become eth0 and the Realtek card becomes eth1. In some cases, after a reboot the cards get renumbered the other way around. To avoid this, create Udev rules that assign stable names to network cards based on their MAC addresses or bus positions.
My solution is based on MAC addresses. I’m not saying that’s the best, but it works for me and I hope it will help you as well. Before editing the file for persistent Udev rules, we have to gather some information.
First I needed to check what cards I have in my server:
[email protected]:/etc/udev/rules.d# lspci | grep Ethernet
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:01.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:02.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:03.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
05:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
So, I have my onboard Realtek card, then one Intel quad card and then two D-Link DFE-570TX quad cards. The Intel card I got it from a friend and the D-Link I bought over eBay. I must say that Ethernet quad cards tend to be a little bit too expensive, but I’ve found 2 brands over eBay that were cheaper, around 25 Eur / piece. One brand is the D-Link and the other one is from Sun. There are some arguments to choose D-Link over Sun on the INE Online Community discussion. You should check that discussion as it has a lot of good tips and tricks. All cards are supported natively in Ubuntu.
Next, I had to collect the MAC addresses of all Ethernet cards.
[email protected]:/etc/udev/rules.d# grep -H . /sys/class/net/*/address | grep eth
/sys/class/net/eth0/address:00:25:22:53:57:40
/sys/class/net/eth10/address:00:80:c8:ca:d8:7e
/sys/class/net/eth11/address:00:80:c8:ca:d8:7f
/sys/class/net/eth12/address:00:80:c8:ca:d8:80
/sys/class/net/eth1/address:00:e0:b6:06:a6:3b
/sys/class/net/eth2/address:00:e0:b6:06:a6:3a
/sys/class/net/eth3/address:00:e0:b6:06:a6:39
/sys/class/net/eth4/address:00:e0:b6:06:a6:38
/sys/class/net/eth5/address:00:80:c8:ca:bb:59
/sys/class/net/eth6/address:00:80:c8:ca:bb:5a
/sys/class/net/eth7/address:00:80:c8:ca:bb:5b
/sys/class/net/eth8/address:00:80:c8:ca:bb:5c
/sys/class/net/eth9/address:00:80:c8:ca:d8:7d
On the last step I had to edit the file in charge for Udev persistent rules for network cards. In Ubuntu, this file is under:
/etc/udev/rules.d/70-persistent-net.rules
I already had a sample file, so I just modified it so match my own rules. My file looks like this:
# This file maintains persistent names for network interfaces.
# See udev(7) for syntax.
#
# Entries are automatically added by the 75-persistent-net-generator.rules
# file; however you are also free to add your own entries.
#
### File generated by cc ####
### Count start from top card left side with eth1 ####
### # eth4 # eth3 # eth2 # eth1 ###
### # eth8 # eth7 # eth6 # eth5 ###
### # eth12 # eth11 # eth10 # eth9 ###
# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:22:53:57:40", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:80", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth12"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7f", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth11"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth10"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7d", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth9"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth8"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth7"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:59", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:38", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:39", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:3a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:3b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
I did some remarks for myself on the top file, just to know in future how the ports are arranged. Save this file and you’re good to go.
Please remember, don’t just copy / paste the output from this file. I mean you can do it, but change at least the ATTR{address} and NAME. I don’t know what other attributes (ATTR{dev_id}==”0x0″, ATTR{type}==”1″) are doing. They were in the original file and I just copy/paste them. Everything is working and for me that is enough for me.
Good luck and let me know if you have any problems implementing the above solution.