Tag: home rack

CCIE home rack – Ubuntu persistent net rules

In one of my last posts, I was writing about my CCIE home rack which has one server that runs Ubuntu + Ethernet Quad cards + Cisco switches. Before connecting the cables between server and Cisco switches, I had one problem that was driving me crazy.

I have three Ethernet Quad cards connected in 3 PCI slots in my server. The issue is that sometimes (quite often) the port numbers were changing during the reboot of my client. To give you and example, an ethernet port that was  eth1 during one server boot could change to eth2 next time. For twelve ports you can imagine what mess this creates after almost each server reboot. When this was happening, the ports were not matched correctly in Dynamips and from it would result in a lot of connectivity problems.

I will try to explain in a few word why this is happening. With Udev and modular network drivers, the network interface numbering is not persistent across reboots by default, because the drivers are loaded in parallel and, thus, in random order. For example, on a computer having two network cards made by Intel and Realtek, the network card manufactured by Intel may become eth0 and the Realtek card becomes eth1. In some cases, after a reboot the cards get renumbered the other way around. To avoid this, create Udev rules that assign stable names to network cards based on their MAC addresses or bus positions.

My solution is based on MAC addresses. I’m not saying that’s the best, but it works for me and I hope it will help you as well. Before editing the file for persistent Udev rules, we have to gather some information.

First I needed to check what cards I have in my server:

root@athos:/etc/udev/rules.d# lspci | grep Ethernet
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:01.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:02.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
04:03.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 05)
05:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
05:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
06:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)

So, I have my onboard Realtek card, then one Intel quad card and then two D-Link DFE-570TX quad cards. The Intel card I got it from a friend and the D-Link I bought over eBay. I must say that Ethernet quad cards tend to be a little bit too expensive, but I’ve found 2 brands over eBay that were cheaper, around 25 Eur / piece. One brand is the D-Link and the other one is from Sun. There are some arguments to choose D-Link over Sun on the INE Online Community discussion. You should check that discussion as it has a lot of good tips and tricks. All cards are supported natively in Ubuntu.

Next, I had to collect the MAC addresses of all Ethernet cards.

root@athos:/etc/udev/rules.d# grep -H . /sys/class/net/*/address | grep eth
/sys/class/net/eth0/address:00:25:22:53:57:40
/sys/class/net/eth10/address:00:80:c8:ca:d8:7e
/sys/class/net/eth11/address:00:80:c8:ca:d8:7f
/sys/class/net/eth12/address:00:80:c8:ca:d8:80
/sys/class/net/eth1/address:00:e0:b6:06:a6:3b
/sys/class/net/eth2/address:00:e0:b6:06:a6:3a
/sys/class/net/eth3/address:00:e0:b6:06:a6:39
/sys/class/net/eth4/address:00:e0:b6:06:a6:38
/sys/class/net/eth5/address:00:80:c8:ca:bb:59
/sys/class/net/eth6/address:00:80:c8:ca:bb:5a
/sys/class/net/eth7/address:00:80:c8:ca:bb:5b
/sys/class/net/eth8/address:00:80:c8:ca:bb:5c
/sys/class/net/eth9/address:00:80:c8:ca:d8:7d

On the last step I had to edit the file in charge for Udev persistent rules for network cards. In Ubuntu, this file is under:

/etc/udev/rules.d/70-persistent-net.rules

I already had a sample file, so I just modified it so match my own rules. My file looks like this:

# This file maintains persistent names for network interfaces.
# See udev(7) for syntax.
#
# Entries are automatically added by the 75-persistent-net-generator.rules
# file; however you are also free to add your own entries.
#
### File generated by cc ####
### Count start from top card left side with eth1 ####
### # eth4  # eth3  # eth2  # eth1 ###
### # eth8  # eth7  # eth6  # eth5 ###
### # eth12 # eth11 # eth10 # eth9 ###
 
# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:22:53:57:40", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:80", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth12"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7f", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth11"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth10"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:d8:7d", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth9"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth8"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth7"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:5a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"
 
# PCI device 0x1011:0x0019 (tulip)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:80:c8:ca:bb:59", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"
 
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:38", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"
 
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:39", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
 
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:3a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
 
# PCI device 0x8086:0x1229 (e100)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:b6:06:a6:3b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

I did some remarks for myself on the top file, just to know in future how the ports are arranged. Save this file and you’re good to go.

Please remember, don’t just copy / paste the output from this file. I mean you can do it, but change at least the ATTR{address} and NAME. I don’t know what other attributes (ATTR{dev_id}==”0x0″, ATTR{type}==”1″) are doing. They were in the original file and I just copy/paste them. Everything is working and for me that is enough for me.

Good luck and let me know if you have any problems implementing the above solution.

Home lab for CCIE exam training

Before I started with my preparation I was in front of a dilemma. I knew that I will need a rack to practice for CCIE training and I had two options: remote rack rental or home rack.

I put together side by side all pluses and minuses about the two options, but somehow I couldn’t decide which solution to choose. Both options have good and bad parts. You’ll have to figure it out which solution you can afford and works best in your case. My words below are just to be used as guidelines.

In first phase I bought some time on online racks. The prices were acceptable but I had big issues to find a time window that suits for me. As you probably know, the “good” windows are already booked in advanced. Next, most of the racks are out of Europe (where I’m located), so there is a time difference. Then I had problems with latency. It was acceptable but I could feel that the connection cannot keep up with my typing speed. All this together made me think to find a new solution.

Phase two: Dynamips. I believe that everybody who’s preparing for an Cisco exam heard of this software and used and least once. I put together a strong machine to support Dynamips and then I look around for a training provider that would offer their labs in this format. Back then, INE was my choice and it went pretty well. Almost perfect, just that the routers were stopping to respond from time to time, lack of connectivity in the emulated topology and of course I couldn’t test everything on switches.

I had some months of practice -> Brussels -> exam … and failed. Yes, failure is part of the game. Judging after my exam results (the percent part) I realized that I’m not totally out of topics, but I need to polish my knowledge a bit more to pass.

After this episode I had to stop for a while my preparation due to lack of time. A while ago, I decided to give another exam try, but before I need  to start polishing my knowledge.

Phase three: mixed home rack. I have now a mixed rack with Dynamips for routers and switches in hardware. I opted for this solution because of multiple problems like lack of space for a full rack, costs, power consumption, noise and so on.

To put the things together for this mixed topology, I had to buy some things on eBay. Here is the list:

3 x Quad Ethernet cards

2 x Cisco 3560

2 x Cisco 3550

3 x Cisco 2600 (for BB routers)

There is another solution, that don’t require quad cards, but an extra switch that support QinQ. You can read more about that solution here.

The there quad cards will give me 12 ethernet ports enough for 6 routers (R1..R6) in common topologies. The serial connections will be emulated in Dynamips and the Ethernet will be something like:

R1 F0/0 -> Eth1 (Linux box) -> 3560 Fa0/1

and so on…

The three BB routers are not really necessary if you use the topology from INE , as one of the BB has a Serial connection to topology routers, and I cannot have that connection from virtual to physical environment.

If you use MicronicsTrainig, well-know as Narbik topology, then you will need the physical routers or an additional quad card. In this topology the BB routers have Ethernet connection to the switches and I don’t have enough ports with my three quad cards. Anyway the price was not a problem for the 2600.

With IPexpert training books, it’s a bit different. I didn’t check too much into their topology, because I understand that you need more resources as it include more routers (about nine). I will have a look in future, as I would like to see if my home rack can be used with any provider.

Here are some pictures of my home lab:

In my next post I will describe how I build a “rack” from an IKEA table, cable connection, console configuration and so more technical details for those who would like to follow this solution.