Cisco: How to use CEF for load-balancing

According to Cisco’s website, Cisco Express Forwarding (CEF) is advanced, Layer 3 IP switching technology. CEF optimizes network performance and scalability for networks with large and dynamic traffic patterns, such as the Internet, on networks characterized by intensive Web-based applications, or interactive sessions.

The term Load balancing describes a functionality in a router that distributes packets across multiple links based on layer 3 routing information.

Now, putting this two terms together we obtain Load balancing with CEF. Cisco IOS software basically supports two modes of load balancing: On per-destination or per-packet basis.
In per-destination mode all packets for a given destination are forwarded along the same path. Usage of only one path will lead to a unequal usage of the lines, as the packets to the same destination will use only one line, leaving the other unused. This can be a problem in a small environment where let’s say that a location is having 2 x E1 connection. In per-destination mode, all the packets to one destination will use one E1 connection, and the other line will remain unused. In addition to this, if this one destination is a server on which most on the site’s users will connect to, the per-destination mode will lead to an exhaust of the bandwidth available on one E1 line leaving the other one empty. On the other hand, on more developed location, where there are more destination available, the per-destination method will not have too much impact about the usage of the lines, as in this case the traffic will be split for the multiple destinations over multiple paths.

The per-packet mode guarantees equal load across all links because the forwarding process determines the outgoing interface for each packet by looking up the route table and picking the least used interface. If you read until now, than most probably you will say that this method is the best and then why the per-destination mode is the default one. Some issues with the per-packet mode would be that this method will almost always result in out-of-order packets, as advised by Ivan Pepelnjak, on his blog. What I can tell you is that this out-of-order packets has not a big impact on low-speed environment where TCP stack can deal with this problem. On the other hand on high-speed environment where video or voice traffic is expected, you can have big problems and it’s better to avoid the usage of per-packet load-balancing. Also, Cisco is advising that this ensures equal utilization of the links, but is a processor intensive task and impacts the overall forwarding performance. How much will impact on the CPU / resources usage? I cannot tell exactly because this depend on the task and traffic that the device has to handle.

OK, so if this per-packet mode is not so great, than why we should use it? The answer is that in some particular topologies or environments you cannot use other method of load-balancing and you are in desperate need of such mechanism. Below you will see an example of what kind of topology can force us to use per-packet load-balancing.

Before you configure load-balancing, you have to be sure that IP CEF is enabled on your router. In case that it is disabled, please enable it:

configure terminal
ip cef

If you want to fine tune the IP CEF load-balancing algorithm you can do this with the command:

configure terminal
ip cef load-balancing algorithm “name parameter”

where for “name” you have 3 choices:
original – Sets the load sharing algorithm to the original based on a source and destination hash.
tunnel – Sets the load sharing algorithm for use in tunnel environments or in environments where there are only a few IP source and destination address pairs.
universal – Sets the load sharing algorithm to the universal algorithm that uses a source and destination, and ID hash.
Skipping the tunnel option which you should use only if you are sure that you need it, the other choices would be universal and original. Original algorithm use IP addresses to generate the 4-bit hash. On the other hand universal algorithm add a router specific information to the hash leading to a more complex development of the hash value. Since universal option is the default one, exception the case where you know what you are doing, you should not change this value.

To enable IP CEF load-balancing on per-destination base, you don’t have to modify anything, as it’s enabled by default. For per-packet mode, you have to use the following commands:

configure terminal
interface FastEthernet x/y
ip load-balancing per-packet

One scenario where I had to use per-packet load-balancing is the one below.
Let’s assume that we have client (c-ubuntu-1, 10.10.20.100) which is sending traffic to one server (s-ubuntu-1,10.10.10.100). The routing protocol is already configured in such way that CE1 and CE2 routers are announcing, through OSPF, 2 default routes (equal cost) to SC device. The rest of the routing part is assure by BGP towards PE routers and another IGP protocols, but this has no importance for the topic discussed here.

SC1#sh ip route | i 110/1
O*E2 0.0.0.0/0 [110/1] via 172.29.190.237, 02:35:46, Vlan23
[110/1] via 172.29.190.229, 02:35:46, Vlan13

On the SC devices I used L3 interface vlan 23 to connect to CE2 and vlan 13 to connect to CE1. With the CEF enabled and per-destination load-balancing mechanism, only one  path (either vlan 23 or vlan 13) was used, leading to only one WAN serial connections to be used. Since this was client to server traffic, and quite a lot, from time to time one WAN connection was exhausted while the other one  remains unused. You can see this, in the excerpt below:

SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100    -> 10.10.10.100   : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100    -> 10.10.10.100   : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100    -> 10.10.10.100   : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100    -> 10.10.10.100   : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100    -> 10.10.10.100   : Vlan23 (next hop 172.29.190.237)

After I enabled the per-packet load-balancing, the situation has changed since both lines to CE routers where used, leading to a equal utilization of the WAN lines:

SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan13 (next hop 172.29.190.229)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan13 (next hop 172.29.190.229)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan13 (next hop 172.29.190.229)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan23 (next hop 172.29.190.237)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan13 (next hop 172.29.190.229)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan13 (next hop 172.29.190.229)
SC#sh ip cef exact-route 10.10.20.100 10.10.10.100
10.10.20.100 -> 10.10.10.100 : Vlan23 (next hop 172.29.190.237)

I had to choose this method to force the equal usage of the WAN links and I rely on TCP stack to solve the out-of-order packets as there is not so high traffic over this interfaces.