Cisco STP forward-time command trick

This is more a topic for exams, like Cisco CCIE, but also it can appear in real-world environments.

Consider the following simple topology:
Simple L2 topology

SW1 is the spanning-tree root bridge for all VLANs. Imagine that you have a request which ask you that when a port becomes active, no matter of VLAN, it should wait 10 seconds until it transition to forwarding state.

You look into configuration mode and spot the “spanning-tree .. forward-time..” command. OK, simple enough, you go there and type:

spanning-tree vlan 1-4094 forward-time 10

Task completed. This is what I also thought. Looks pretty straight forward, but it’s a mistake which I learned from and I hope you will read this before doing it.
Lucky for me it was just a test lab not the real CCIE lab, but if I don’t teach this kind of things or spend the entire day fine-tuning STP, I tend to forget.

Back to this example, let’s see on the SW2 what’s happening after I did configure the above command on SW1. By the way, you need to configure this command only on the STP root bridge because the downstream devices will inherit the values.

On the SW2 I shutdown the root interface to force the alternate interface (which is in blocking state) to transition to forward state and check the output of “debug spanning-tree events” :

00:55:15: STP: VLAN0001 new root port Fa0/2, cost 19
00:55:15: STP: VLAN0001 Fa0/2 -> listening
...
00:55:25: STP: VLAN0001 Fa0/2 -> learning
...
00:55:35: STP: VLAN0001 sent Topology Change Notice on Fa0/2
00:55:35: STP: VLAN0001 Fa0/2 -> forwarding

From :15 seconds when the port entered Listening state until :35 seconds when it went into Forwarding state there are 20 seconds. We were required to have 10 seconds not 20. That’s not good.

Going back and checking the parameters of a STP for a particular VLAN did not give me too much information to clarify the issue:

show spanning-tree vlan 1 | i Forward
Hello Time   3 sec  Max Age 10 sec  Forward Delay 10 sec

OK, after reading again carefully the documentation I came to the understanding that value added to this command is applied to each state towards the forwarding state.
The STP Port States are: Blocking, Listening, Learning, Forwarding and Disabled.

When I did shutdown the root port, the alternate port when immediately to Listening state then waited for 10 seconds until it transition to Learning state and again 10 Seconds to achieve Forwarding state.

The correct solution was obvious now to decrease the forward-time to 5 seconds.

spanning-tree vlan 1-4094 forward-time 5

Let’s check again:

01:11:55: STP: VLAN0001 new root port Fa0/2, cost 19
01:11:55: STP: VLAN0001 Fa0/2 -> listening
...
01:12:00: STP: VLAN0001 Fa0/2 -> learning
...
01:12:05: STP: VLAN0001 sent Topology Change Notice on Fa0/2
01:12:05: STP: VLAN0001 Fa0/2 -> forwarding

Now the total time is 10 second.

I hope this will help others to avoid my mistakes.


EtherChannel over 802.1q Tunneling

Consider the following topology:
EtherChannel over 802.1q Tunneling Topology

We have one Customer with two distributed locations (SW1, R1 and SW2, R2) connected over Provider backbone. What we want to create is something like this:

EtherChannel over Provider L2 cloud

If Provider support 802.1q and L2 tunneling we can achieve a nice Etherchannel between our 2 remote locations with direct CDP visibility. Also STP and VTP is supported, just like when these SW1 and SW2 switches are directly connected.

First, lets configure SW1 and SW2 Customer devices.

On the three interfaces connected to provider devices we want to configure LACP Etherchannel:

SW1 / SW2 Customer

interface FastEthernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active
!
interface FastEthernet0/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active
!
interface FastEthernet0/3
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active

Next we will configure the SW1 and SW2 ports connected to R1 and R2 devices:

SW1 / SW2 Customer

vtp mode transparent
vtp domain Customer
vlan 100
 name End2End
!
interface Fa0/10
switchport mode access
switchport access vlan 100

Of course another approach can be taken in terms of VTP, like having Server / Client configuration, but this was the simplest one to illustrate here.

Let’s add some IP addresses on the two routers R1 and R2:

R1 Customer

interface fa0/0
ip address 10.0.0.1 255.255.255.0

R2 Customer

interface fa0/0
ip address 10.0.0.2 255.255.255.0

Our job, as Customer, is done. What about the Provider configuration? Here is where “the magic” happens.

To provide our Customer with three end to end 802.1q tunnels, we need to create three VLANs, assign them to the interfaces pointing to Customer SW1 and SW2 and enable the 802.1q tunnels.

SW1 / SW2 Provider

vlan 10
vlan 20
vlan 30
!
interface FastEthernet0/1
 switchport access vlan 10
 switchport mode dot1q-tunnel
!
interface FastEthernet0/2
 switchport access vlan 20
 switchport mode dot1q-tunnel
!
interface FastEthernet0/3
 switchport access vlan 30
 switchport mode dot1q-tunnel

Of course SW1 and SW2 from Provider should have 802.1q trunk enable and allow the tranport of VLANs 10, 20 and 30:

SW1 / SW2 Provider

int fa0/4
switchport trunk mode dot1q
switchport mode trunk
switchport trunk allowed vlan 10,20,30

OK, we have the dot1q tunneling enabled now:

SW1 / SW2 Provider

show dot1q-tunnel

dot1q-tunnel mode LAN Port(s)
-----------------------------
Fa0/1
Fa0/2
Fa0/3

Still, the Customer wants Etherchannel functionality, CDP visibility and the ability to transport own VLAN information (remember we did configure Vlan 100 on the interface of SW1 / SW2 Customer pointing to R1 / R2). Let’s enable also these ones:

SW1 / SW2 Provider

interface FastEthernet0/1
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable
!
interface FastEthernet0/2
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable
!
interface FastEthernet0/3
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable

Perfect, now let’s do some “show” commands to see that everything is working.

SW1 / SW2 Customer

show etherchannel 1 summary | b Group
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/1(P)   Fa0/2(P)   Fa0/3(P)
show spanning-tree vlan 100

VLAN0100
  Spanning tree enabled protocol ieee
  Root ID    Priority    32868
             Address     0011.20ab.6180
             Cost        9
             Port        56 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)
             Address     0014.a86b.f600
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/10              Desg FWD 19        128.3    P2p
Po1                 Root FWD 9         128.56   P2p

OK, the Etherchannel is UP and the STP is showing correct values. Let’s see if we can do a simple “ping” from R1 to R2

R1#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/24 ms

The Customer is happy, but what about the Provider, what does it see on the L2 infrastructure?

SW1 / SW2 Provider

show spanning-tree vlan 100
Spanning tree instance(s) for vlan 100 does not exist.

So, the Provider has no idea about Vlan 100 used by the Customer. This is because STP BPDUs from SW1 / SW2 Customer are tunneled inside dot1q-tunnel and hidden by the metro tags 10, 20 and 30.

One note for real life example, the Provider needs to support at least MTU 1504 so that Customer does not deal with packet fragmentation.


MAC address tracking with NMS

Let’s assume that I have a port open in my Cisco switch for guest users and I would like to keep track of the new mac-addresses that are added or existing mac-addresses that are removed from that port.

I have a NMS (Network Management System), capable of receiving SNMP, traps that will send me an e-mail alert whenever one of the above mentioned situations happens.

For this test I did use the following IP addresses:
– Cisco switch L3 interface from where the SNMP traps will be send: Lo10, 1.1.1.1 /32
– Cisco switch L2 port where I want to track the mac-address change: Fa0/10
– NMS IP address: 172.31.82.254
– SNMP community string: Cisco

OK, this is a simple example, but it can be useful in some situation or for Cisco exam preparation.

First let’s bring the Lo10 interface UP and configure the IP address. Please keep in mind that without a L3 interface the SNMP configuration will raise an alarm and the system will not work.

interface Loopback10
 ip address 2.2.2.2 255.255.255.255

Next, I’ll configure the SNMP part:

SW(config)#snmp-server host 172.31.82.254 traps Cisco
SW(config)#snmp-server enable traps mac-notification

Following is the actual configuration for mac-address change notification:

SW(config-if)#snmp trap mac-notification change ?
  added    Enable Mac Address added notification for this port
  removed  Enable Mac Address removed notification for this port

The above lines show the possible options. You can use them both if you want to receive notification for adding and removing of the mac-address or any of these options separately. I did use both for this test.

Let’s verify if the configuration is applied correctly:

SW#show mac address-table notification change interface fa0/10
MAC Notification Feature is Disabled on the switch
Interface            MAC Added Trap MAC Removed Trap
---------            -------------- ----------------
FastEthernet0/10     Enabled        Enabled

I find this helpful if you have open ports which require special attention and for some reason you cannot use some technique like dot1x or lock them on static mac-address.


Cisco 2600: How many IPv4 prefixes can fit in BGP table

Today I wanted to try and see how many IPv4 prefixes can a Cisco 2600 accommodate in BGP table both global and in VRF table. I have lying around a Cisco 2621XM with 64MB of RAM, so I said to stress it a bit like in the old days when it was productive.

I did use the bgp_simple tool to test with the latest full Internet prefixes table (dated 2013.03.09). For those of you not familiar with bgp_simple, it’s a script capable of BGP peering and route injection. I would recommend to check the script webpage as it’s an awesome tool.

First, I did use the Global table to store all BGP prefixes. The router issued a %SYS-2-MALLOCFAIL error after receiving 44791 prefixes and few seconds later it dropped the BGP neighborship peering.

BB1#sh ip bgp sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.182.1   4 65000 1194845      95  1181910  360    0 00:10:57    44791

BB1#
*Feb 10 21:22:49.383: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x8006A11C, alignment 0 
Pool: Processor  Free: 64176  Cause: Not enough free memory 
Alternate Pool: None  Free: 0  Cause: No Alternate pool 
 -Process= "BGP Router", ipl= 0, pid= 32 -Traceback= 0x8119EA40 0x80057D14 0x80059F18 0x8005A75C 0x8006A120 0x8006AE9C 0x8192D52C 0x80DFDD34 0x80DFE708 0x80DFF60C 0x82BA5A60 0x80DFF6E4 0x80DD8694 0x802435AC 0x80246CB0
*Feb 10 21:23:00.395: %BGP-5-ADJCHANGE: neighbor 192.168.182.1 Down No memory
*Feb 10 21:23:00.515: %BGP-3-NOTIFICATION: sent to neighbor 192.168.182.1 3/1 (update malformed) 0 bytes 
BB1# FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0037 0200 0000 1C40 0101 0040 020E 0206 FDE8 9465 12EC 1D31 00D1 5821 4003 04C0 A8B6 0117 433F E0

For second test I did create VRF A and the BGP neighborship was established over an interface in that VRF, so all prefixes injected by bgp_simple arrived in the VRF table. This time it took only 33321 prefixes for the router to spit an error and drop the BGP connection.

BB1#sh ip bgp vpnv4 vrf A sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.182.1   4 65000  439682      31   323732  534    0 00:09:24    33321

*Feb 10 21:37:22.323: %SYS-2-MALLOCFAIL: Memory allocation of 2920 bytes failed from 0x8006A11C, alignment 0 
Pool: Processor  Free: 8584  Cause: Memory fragmentation 
Alternate Pool: None  Free: 0  Cause: No Alternate pool 
 -Process= "BGP Router", ipl= 0, pid= 32 -Traceback= 0x8119EA40 0x80057D14 0x80059F18 0x8005A75C 0x8006A120 0x8006AE9C 0x80DEB7B0 0x80DCC800 0x80E3DCC8 0x80E3B62C 0x80E2A244 0x80E2AE30 0x80DD0B14 0x80DD7F08 0x802435AC
*Feb 10 21:37:31.011: %BGP-5-ADJCHANGE: neighbor 192.168.182.1 vpn vrf A Down No memory
*Feb 10 21:37:31.251: %BGP-3-NOTIFICATION: sent to neighbor 192.168.182.1 3/1 (update malformed) 0 bytes 
BB1#sh ip bgp vpnv4 vrf A sum | b Nei FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0052 0200 0000 3740 0101 0040 020A 0204 FDE8 2236 1AAE 18B7 4003 04C0 A8B6 01C0 081C 18B7 03E9 18B7 A028 1AAE 3E80 1AAE 3F6A 1AAE 59D9 1AAE 89EA 2236 1AAE 1440 3B90

For those of you curious about why it took less prefixes to exhaust memory in VRF vs non-VRF, the reason is that in VRF you have more memory consumption per IPv4 prefix. Ivan Pepelnjak explained it very well in his blog post about running Internet in VRF.

I’ll check what other old Cisco routers I have around to see their BGP table limit.


Overlay MPLS over provider backbone

Imagine that you have two or more sites which you want to connect together using MPLS technology. You cannot afford dark fiber and your Service Provider cannot offer you L2 connections of any kind. The only thing your SP can offer is L3 transport. Still, you want to build your own MPLS environment and there is no way to convince your SP to enable CsC.

I will use the following topology to demonstrate one of the possibilities to build an overlay MPLS over SP backbone.

Overlay MPLS over SP

The provider has already his MPLS backbone configured (P1, PE1, PE2 and PE3) and it’s offering you, as customer, IP transport over it’s backbone. Usually, from customer perspective, you don’t get to see the SP backbone, but just for reference, it is using ISIS for IGP, MP-BGP and MPLS VPN to transport our prefixes.

From IP prefixes allocation I’m using “xy” in the third octet (x – lower router number, y – higher router number) and “z” in the last octet (router number) with a /24 mask.

We have three locations named CPE1, CPE2 and CPE3. Currently between my CPE and provider PE I have enabled BGP, but you can use any protocol (even static) if your SP is able to route your IP prefixes over its backbone. On each CPE device I have a Loopback interface and its IP address will be the only prefix you announce (through BGP in this demonstration) to SP.

Let’s establish the BGP connection from our CPE to SP PE. As I’m playing the role of customer here, only the CPE exhibits will be shown:

CPE5

interface Loopback0
 ip address 5.5.5.5 255.255.255.255
!
router bgp 65001
 bgp router-id 5.5.5.5
 bgp log-neighbor-changes
 neighbor 10.0.35.3 remote-as 65000
 neighbor 10.0.35.3 description R3PE3
 neighbor 10.0.35.3 timers 5 20
 !
 address-family ipv4
  neighbor 10.0.35.3 activate
  no auto-summary
  no synchronization
  network 5.5.5.5 mask 255.255.255.255
 exit-address-family

CPE6

interface Loopback0
 ip address 6.6.6.6 255.255.255.255
!
router bgp 65001
 bgp router-id 6.6.6.6
 bgp log-neighbor-changes
 neighbor 10.0.26.2 remote-as 65000
 neighbor 10.0.26.2 description R2PE2
 neighbor 10.0.26.2 timers 5 20
 !
 address-family ipv4
  neighbor 10.0.26.2 activate
  no auto-summary
  no synchronization
  network 6.6.6.6 mask 255.255.255.255
 exit-address-family

CPE7

interface Loopback0
 ip address 7.7.7.7 255.255.255.255
!
router bgp 65001
 bgp router-id 7.7.7.7
 bgp log-neighbor-changes
 neighbor 10.0.47.4 remote-as 65000
 neighbor 10.0.47.4 description R4PE4
 neighbor 10.0.47.4 timers 5 20
 !
 address-family ipv4
  neighbor 10.0.47.4 activate
  no auto-summary
  no synchronization
  network 7.7.7.7 mask 255.255.255.255
 exit-address-family

BGP neighborship suppose to be up now and on each CPE I should receive the Loopback prefixes of the other two CPE devices.

R5CPE5#sh ip bgp sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.35.3       4 65000     126     125        2    0    0 00:10:04        0
!
R6CPE6#sh ip bgp sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.26.2       4 65000     136     135        2    0    0 00:10:50        0
!
R7CPE7#sh ip bgp sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.47.4       4 65000     134     134        4    0    0 00:10:46        0

BGP neighborship is up alright, but where are my prefixes? State/PfxRcd is 0, when it should show 2.
I did that on purpose.

Notice that we are using the same AS number on all our sites. I think you already know that the rule in BGP is that if our own ASN is seen in the AS-Path of a particular IP prefix, BGP will not install that prefix in BGP table. This is fixable:
1. We ask our provider to have a little “as-override” command in its BGP configuration for our neighbor
2. We use different ASN on each site (assuming that we are using private ASN)
3. We configure “allowas-in” on BGP neighborship with SP

Basically you may use any of the three methods (or other if you can think of any other), but in my case I don’t want to ask the SP nor I want to change my ASN scheme. I’ll go with the third option and be careful not run into loop issues (consider this is an Enterprise environment I think it’s doable).

CPE5

router bgp 65001
  neighbor 10.0.35.3 allowas-in

CPE6

router bgp 65001
  neighbor 10.0.26.2 allowas-in

CPE7

router bgp 65001
  neighbor 10.0.47.4 allowas-in

Let’s check again and do some testing. I will use CPE5

R5CPE5#sh ip route bgp
     6.0.0.0/32 is subnetted, 1 subnets
B       6.6.6.6 [20/0] via 10.0.35.3, 00:04:02
     7.0.0.0/32 is subnetted, 1 subnets
B       7.7.7.7 [20/0] via 10.0.35.3, 00:04:02
!
R5CPE5#ping 6.6.6.6 source 5.5.5.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 5.5.5.5 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/44/64 ms

R5CPE5#ping 7.7.7.7 source 5.5.5.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 5.5.5.5 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/52/96 ms

Next part involves the creation of Tunnels interfaces, to have a full mesh connection between the three sites, enable of IGP / MPLS and creation of a second Loopback interface which we will use later for the iBGP configuration. I did chose IS-IS for IGP and LDP for MPLS. The new Loopback interface will be routed using the IS-IS protocol.

A note from my side. Since I had limited number of routers, my CPE devices will be kind of P / PE / CE router in my overlay MPLS demonstration.

CPE5

int Tun56
tunnel source lo0
tunnel destination 6.6.6.6
ip address 192.168.56.5 255.255.255.0
mpls ip
ip router isis
!
int Tun57
tunnel source lo0
tunnel destination 7.7.7.7
ip address 192.168.57.5 255.255.255.0
mpls ip
ip router isis
!
int Lo1
ip address 55.55.55.55 255.255.255.255
!
router isis
net 47.0005.0005.0005.0005.00
passive-interface lo1
is-type level-2-only

CPE6

int Tun56
tunnel source lo0
tunnel destination 5.5.5.5
ip address 192.168.56.6 255.255.255.0
mpls ip
ip router isis
!
int Tun67
tunnel source lo0
tunnel destination 7.7.7.7
ip address 192.168.67.6 255.255.255.0
mpls ip
ip router isis
!
int Lo1
ip address 66.66.66.66 255.255.255.255
!
router isis
net 47.0006.0006.0006.0006.00
passive-interface lo1
is-type level-2-only

CPE7

int Tun57
tunnel source lo0
tunnel destination 5.5.5.5
ip address 192.168.57.7 255.255.255.0
mpls ip
ip router isis
!
int Tun67
tunnel source lo0
tunnel destination 6.6.6.6
ip address 192.168.67.7 255.255.255.0
mpls ip
ip router isis
!
int Lo1
ip address 77.77.77.77 255.255.255.255
!
router isis
net 47.0007.0007.0007.0007.00
is-type level-2-only
passive-interface lo1	

I will use CPE5 for some show commands output and to check that everything is running fine:

R5CPE5#ping 192.168.56.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.56.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/48/52 ms

R5CPE5#ping 192.168.57.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.57.7, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/40/52 ms

!
R5CPE5#sh isis topology

IS-IS paths to level-2 routers
System Id            Metric     Next-Hop             Interface   SNPA
R5CPE5               --
R6CPE6               10         R6CPE6               Tu56        *Tunnel*       
R7CPE7               10         R7CPE7               Tu57        *Tunnel* 
!
R5CPE5#show mpls interfaces 
Interface              IP            Tunnel   Operational
Tunnel56               Yes (ldp)     No       Yes         
Tunnel57               Yes (ldp)     No       Yes 
!
R5CPE5#show mpls ldp neighbor 
    Peer LDP Ident: 6.6.6.6:0; Local LDP Ident 5.5.5.5:0
	TCP connection: 6.6.6.6.64820 - 5.5.5.5.646
	State: Oper; Msgs sent/rcvd: 16/16; Downstream
	Up time: 00:07:47
	LDP discovery sources:
	  Tunnel56, Src IP addr: 192.168.56.6
        Addresses bound to peer LDP Ident:
          10.0.26.6       6.6.6.6         192.168.56.6    192.168.67.6    
    Peer LDP Ident: 7.7.7.7:0; Local LDP Ident 5.5.5.5:0
	TCP connection: 7.7.7.7.11545 - 5.5.5.5.646
	State: Oper; Msgs sent/rcvd: 16/16; Downstream
	Up time: 00:07:26
	LDP discovery sources:
	  Tunnel57, Src IP addr: 192.168.57.7
        Addresses bound to peer LDP Ident:
          10.0.47.7       7.7.7.7         192.168.57.7    192.168.67.7  

I will create now two VRF instances as I want to separate Financial department traffic from the Technical one.

On all three CPE devices:

ip vrf FIN
rd 65001:1
route-target import 65001:1
route-target export 65001:1
!
ip vrf TEK
rd 65001:2
route-target import 65001:2
route-target export 65001:2

Configuration of iBGP depends on the previous Loopback1 interfaces, so be sure that you have that interfaces reachable through IS-IS. Because it’s an iBGP and I don’t want to type the same command over and over, I will use peer-groups.

CPE5

router bgp 65001
!
neighbor OM peer-group
neighbor OM remote-as 65001
neighbor OM timers 5 20
neighbor OM description Overlay-MPLS
neighbor OM update-source lo0
!
address-family vpnv4
neighbor 66.66.66.66 peer-group OM
neighbor 77.77.77.77 peer-group OM

CPE6

router bgp 65001
neighbor OM peer-group
neighbor OM remote-as 65001
neighbor OM timers 5 20
neighbor OM description Overlay-MPLS
neighbor OM update-source lo1
!
address-family vpnv4
neighbor 55.55.55.55 peer-group OM
neighbor 77.77.77.77 peer-group OM

CPE7

router bgp 65001
neighbor OM peer-group
neighbor OM remote-as 65001
neighbor OM timers 5 20
neighbor OM description Overlay-MPLS
neighbor OM update-source lo1
!
address-family vpnv4
neighbor 55.55.55.55 peer-group OM
neighbor 66.66.66.66 peer-group OM	

We should check that everything is up. I will use again CPE5:

R5CPE5#show ip bgp vpnv4 all sum | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
66.66.66.66     4 65001      46      46        1    0    0 00:03:30        0
77.77.77.77     4 65001      31      31        1    0    0 00:02:18        0

Finally we are getting somewhere. In real world you will have the CPE routers connected to downstream devices, using subinterfaces in particular VRF and so on. I’m short on devices, so I will use some additional Loopback interfaces and add them to VRF FIN and TEK for testing.

CPE5

int Lo51
ip vrf forwarding FIN
ip address 10.51.51.51 255.255.255.255
!
int Lo52
ip vrf forwarding TEK
ip address 10.52.52.52 255.255.255.255
!
router bgp 65001
!
address-family ipv4 vrf FIN
network 10.51.51.51 mask 255.255.255.255
!
address-family ipv4 vrf TEK
network 10.52.52.52 mask 255.255.255.255

CPE6

int Lo61
ip vrf forwarding FIN
ip address 10.61.61.61 255.255.255.255
!
int Lo62
ip vrf forwarding TEK
ip address 10.62.62.62 255.255.255.255
!
router bgp 65001
!
address-family ipv4 vrf FIN
network 10.61.61.61 mask 255.255.255.255
!
address-family ipv4 vrf TEK
network 10.62.62.62 mask 255.255.255.255

CPE7

int Lo71
ip vrf forwarding FIN
ip address 10.71.71.71 255.255.255.255
!
int Lo72
ip vrf forwarding TEK
ip address 10.72.72.72 255.255.255.255
!
router bgp 65001
!
address-family ipv4 vrf FIN
network 10.71.71.71 mask 255.255.255.255
!
address-family ipv4 vrf TEK
network 10.72.72.72 mask 255.255.255.255

To check if everything is working fine, I will use CPE5 for some tests:

R5CPE5#sh ip route vrf FIN | b Ga
Gateway of last resort is not set

     10.0.0.0/32 is subnetted, 3 subnets
B       10.61.61.61 [200/0] via 66.66.66.66, 00:04:42
C       10.51.51.51 is directly connected, Loopback51
B       10.71.71.71 [200/0] via 77.77.77.77, 00:02:46
!
R5CPE5#sh ip route vrf TEK | b Ga
Gateway of last resort is not set

     10.0.0.0/32 is subnetted, 3 subnets
B       10.62.62.62 [200/0] via 66.66.66.66, 00:04:59
C       10.52.52.52 is directly connected, Loopback52
B       10.72.72.72 [200/0] via 77.77.77.77, 00:03:04
!
R5CPE5#ping vrf FIN 10.71.71.71 source 10.51.51.51

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.71.71, timeout is 2 seconds:
Packet sent with a source address of 10.51.51.51 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/44/48 ms
!
R5CPE5#ping vrf TEK 10.62.62.62 source 10.52.52.52

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.62.62.62, timeout is 2 seconds:
Packet sent with a source address of 10.52.52.52 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/47/52 ms
!
R5CPE5#show mpls forwarding-table 
Local  Outgoing    Prefix            Bytes tag  Outgoing   Next Hop    
tag    tag or VC   or Tunnel Id      switched   interface              
16     Pop tag     192.168.67.0/24   0          Tu57       point2point  
       Pop tag     192.168.67.0/24   0          Tu56       point2point  
17     Pop tag     66.66.66.66/32    0          Tu56       point2point  
18     Pop tag     77.77.77.77/32    0          Tu57       point2point  
19     Aggregate   10.51.51.51/32[V] 1040                               
20     Aggregate   10.52.52.52/32[V] 520                                

You may wonder why somebody would put together such a complex configuration. There may be multiple reasons, beyond the scope of this example, but I would like to add MPLS TE, Encrypted Site-to-Site traffic with route manipulation, independent configuration from the SP, learning purposes and many more.

Can we encounter problems with this configuration? Well, yes.
If the provider has a very low MTU size support, you may get a lot of fragmentation. Also maintenance and operation of Tunnels may be tricky for a very large environment, but there are solutions to limit the number of tunnels. Still the benefits exit.

To enumerate one benefit from real world. Applying this configuration in Enterprise environment, you have the possibility to change your SP without too much of a dazzle as long as your new provider can transport the IP address of your primary Loopback interface. The rest, stays the same.

Please let me know if you have questions or if something in my explanation is wrong.