Cisco WLAP and WLC fail to create CAPWAP connection

Last days I’ve encounter an issue when some of the Wireless Lightweight Access Points (WLAP) just disappeared from the Wireless LAN Controller (WLC).

I saw before these kind of problems before and usually, whatever the reason, the WAP cannot discover the WLC. It was not the case now, everything seems to be in order both in IP connectivity and correct parameters to point the WAP to correct WLC.

Looking back now, the problem is generated by an obvious issue, but back then it took me a bit to troubleshoot the issue. I’ll share my findings so others can resolve it quickly in case they hit this problem.

The WLC logs didn’t point out an obvious reason. Maybe is due to log volume and the fact that this particular WLC had other WAP which were working fine. Just couple of them suddenly disappeared.
I went the other way, and start troubleshooting from the WAP. Once I got remote access to the WAP (yes, it had an IP address and was reachable) the logs showed something like this:

WAP#
*Oct 17 19:54:55.757: %DOT11-7-AUTH_FAILED: Station MAC_ADDRESS Authentication failed
*Oct 17 19:54:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246
*Oct 17 19:54:56.352: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP
*Oct 17 19:54:56.352: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246
*Oct 17 19:56:01.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246
*Oct 17 19:56:01.364: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP
*Oct 17 19:56:01.364: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246

Was obvious that something is wrong with the CAPWAP tunnel and seemed to be related to the DTLS since the certificate unknown alert was present.

I’ll spare you the research around to figure it out, but finally I came to find this Field Notice: FN – 63942 – Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration – Software Upgrade Recommended – Cisco which in turn pointed to this BUG CSCuq19142. The BUG says that a WAP will fail to join a WLC if the SSC (self signed certificate) or MIC (manufactured installed certificate) has an expired date.

Going back to WAP CLI to check the MIC (SSC is not the case), it seemed that the suggested command “show crypto pki certificates” was not available. At least it seemed…

You need to add another command “debug capwap console cli” before to issue the “show crypto pki certificates“:

WAP# debug capwap console cli
WAP# show crypto pki certificates
!! removed output!!
Certificate
  Status: Available
  Certificate Serial Number (hex): HEX_VALUE
  Certificate Usage: General Purpose
  Issuer:
    cn=Cisco Manufacturing CA
    o=Cisco Systems
  Subject:
    Name: AP_NAME
    [email protected]
    cn=AP_NAME
    o=Cisco Systems
    l=San Jose
    st=California
    c=US
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/cmca.crl
  Validity Date:
    start date: 07:21:37 UTC Oct 13 2012
    end   date: 07:31:37 UTC Oct 13 2022
  Associated Trustpoints: Cisco_IOS_MIC_cert
  Storage:
!! removed output !!

If you check the validity date, seems this AP had a 10 years anniversary, which is also the default expiration date for the MIC installed certificate.

Checking the Field Notice above, it recommends to upgrade the WLC OS, but a lot of OS versions are affected, so in the meantime I went with the suggest workaround:

WLC> config ap cert-expiry-ignore mic enable

The WLC will ignore the MIC with expired date and as result the WAP will immediately join the WLC.

I hope this basic explanation and the quick workaround will help somebody if they run into the same issue.

IPsec VPN Mikrotik to Cisco

Not long ago I wrote an article on how to configure an IPsec VPN using Mikrotik and Linux devices. For today, I will replace the Linux device with a Cisco. I did test the entire construct in GNS3 integrated with Mikrotik.

The topology looks like this:

IPsec VPN Mikrotik Cisco

The red line represent the IPsec VPN tunnel.
Please note the used IP addresses. In this way the below configuration will be easier to understand.

Mikrotik Configuration

1. Firewal rules

By default, the Mikrotik comes with the INPUT channel that drop the connection incoming on ether1-gateway (which is the WAN interface). You need to be sure that at least the IPsec packets are able to be accepted inbound on the WAN interface, so the below rules needs to be placed before the rule dropping packets (the Firewal rules are checked top-down)

On INPUT channel allow the following on the interface facing Internet
– Port 500/UDP
– Port 4500/UDP
– Proto 50
– Proto 51
It may be that you don’t need all these ports, but you can close them later. You can check logs if you want to troubleshoot.

On NAT channel, SRCNAT you need have the rule involving interesting traffic (local LAN subnets for example) before NAT masquerade.
You need to add a rule with ACCEPT source LOCAL_LAN (192.168.88.0/24 in this example) destination REMOTE_LAN (192.168.0.0/24 in this example).

On Console the configuration looks like this:

CLI

!
ip firewall filter add chain=input proto=ipsec-ah action=accept place-before=0
ip firewall filter add chain=input proto=ipsec-esp action=accept place-before=0
ip firewall filter add chain=input proto=udp port=500 action accept place-before=0
ip firewall filter add chain=input proto=udp port=4500 action accept place-before=0
!
ip firewall nat add chain=srcnat src-address=192.168.88.0/24 dst-address=192.168.0.0/24 action=accept place-before=0

2. The IPsec Proposal

GUI

IP > IPsec > Proposals

Name: MyProposal
Auth. Algorithm: sha1
Encr. Algorithm: aes-256 cbc
PFS Group: none

CLI

ip ipsec proposal add name=MyProposal auth-algorithms=sha1 enc-algorithms=aes-256-cbc pfs-group=none

3. The IPsec Policy

GUI

IP > IPsec > Policies

SRC ADDR: 192.168.88.0/24
DST ADDR: 192.168.0.0/24
Protocol: all
Action: Encrypt
Level: require
IPsec protocols: esp
Tunnel: check
SA SRC: 10.0.0.2
SA DST: 192.168.23.3
Proposal: MyProposal

CLI

ip ipsec policy add src-address=192.168.88.0/24 dst-address=192.168.0.0/24 protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=10.0.0.2 sa-dst-address=192.168.23.3 proposal=MyProposal

4. The IPsec Peer

GUI

IP > IPsec > Peers

Address: 192.168.23.3
Port: 500
Auth. Method: pre shared key
Passive: not checked
Secret: MYKEY
Policy Template Group: default
Exchange mode: main
Send Initial Contact: checked
NAT Traversal: checked
My ID: Auto - empty
Proposal Check: obey
Hash Algorithm: sha1
Encryptions Algorithm: aes-256
DH Group: modp1024
Generate policy: no

CLI

ip ipsec peer add address=192.168.23.3 port=500 auth-method=pre-shared-key secret=MY_KEY exchange-mode=main send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-256 dh-group=modp1024 generate-policy=no

Cisco configuration

1. Crypto ISAKMP Policy

crypto isakmp policy 1
encr aes 256
authentication pre-share
group 2

You can specify also the hash as sha1, but this is the default method on Cisco, so no extra line will appear.

2. Crypto ISAKMP neighbor

crypto isakmp key MYKEY address 10.0.0.2 no-xauth

3. Crypto IPsec transformation set

crypto ipsec transform-set MYTRANSFORMSET esp-aes 256 esp-sha-hmac
 mode tunnel

4. Crypto map

crypto map MYCRYPTOMAP 10 ipsec-isakmp
 description Mikrotik VPN
 set peer 10.0.0.2
 set transform-set MYTRANSFORMSET
 match address ACLTRAFF

5. Access-list for interesting traffic

ip access-list extended ACLTRAFF
 permit ip 192.168.0.0 0.0.0.255 192.168.88.0 0.0.0.255

6. Interface config

int fa1/0
 description Internet facing interface
 crypto map MYCRYPTOMAP

The settings (like encryption algorithm) can be tuned to fit your requirements.

If you have any questions or something is unclear please let me know in Comments.

GNS3 1.2.1 installation on Ubuntu 14.04

As mentioned in an earlier post GNS3 is moving ahead fast. Currently at version 1.2.1 the GNS3 is looking great. Compared with the version 1.0 Beta 1 which I had installed, the 1.2.1 is not only more stable, but it has the Menu more clean and compact. For example now there is only one Preferences menu where you can adjust all your settings.

During the installation of 1.0 Beta 1 I made some notes in Evernote and it prove to be very useful as the installation was pretty messy. With 1.2.1 I did the same thing, but the installation was very smooth. Still, I said that if I made those notes maybe I should share them for those interested in a quick installation. A more complete guide can be found on GNS3 Community.

1. Download GNS3 1.2.1

Head over to http://www.gns3.com/, create and account and download the bundle archive for Linux.

If you for some reason you don’t want to create an account, you may download each package individually from https://github.com/GNS3

The following lines will assume that you have the bundle archive.

2. Install Ubuntu 14.04 dependencies

$ sudo apt-get install libpcap-dev uuid-dev libelf-dev cmake
$ sudo apt-get install python3-setuptools python3-pyqt4 python3-ws4py python3-netifaces python3-zmq python3-tornado
$ sudo apt-get install unzip 

3. Unzip the bundle archive

$ unzip GNS3-1.2.1.source.zip

You should see 5 packages in GNS3-1.2.1 folder:
dynamips-0.2.14.zip
gns3-server-1.2.1.zip
gns3-gui-1.2.1.zip
iouyap-0.95.zip
vpcs-0.6.zip

4. Install Dynamips

$ unzip dynamips-0.2.14.zip
$ cd dynamips-0.2.14
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install

To check if the correct version is install:

$ dynamips | grep version

You should see in the output 0.2.14

5. Install GNS3 Server

$ unzip gns3-server-1.2.1.zip
$ cd gns3-server-1.2.1
$ sudo python3 setup.py install

To check if the GNS3 Server is installed correctly:

$ gns3server

If you see some output other than an error, than you’re fine.

6. Install GNS3 GUI

$ unzip gns3-gui-1.2.1.zip
$ cd gns3-gui-1.2.1
$ sudo python3 setup.py install

To test if the installation is working:

$ gns3

You should see a graphical interface of GNS3 launched.

At this moment you have a working GNS3 environment if you want only want to test Cisco hardware emulators. I strongly recommend to continue and install also the rest of the components. Who knows when you’ll need them

7. Install IOUyap (Optional, if you will use IOU images)

$ unzip iouyap-0.95.zip
$ cd iouyap-0.95.zip
$ make
$ sudo make install

To test the installation:

$ iouyap -h

If you encounter an error, please check the [Update 1] section at the bottom of this article.

8. Install VPCS (Optional, if you want to use VirtualPC)

$ unzip vpcs-0.6.zip
$ cd vpcs-0.6/src
$ ./mk.sh 64
$ cp vpcs /usr/bin/

For the third line, the 64 represent 64bit, as my Ubuntu 14.04 is build on 64bit.
The values can be:
– 32 or i386 for 32bit OS
– 64 or amd64 for 64bit OS

Please be sure to use the correct one for your OS.

To test the VPCS:

$ vpcs

You should see a Virtual PC being launched. Leave the console with letter q.

9. Install VirtualBox (Optional, if you want to launch VMs)

Download the correct version for your system from https://www.virtualbox.org/wiki/Linux_Downloads. The following lines will assume an Ubuntu 14.04 64bit OS.

$ apt-get install dkms
$ dpkg -i virtualbox-4.3_4.3.20-96996~Ubuntu~raring_amd64.deb

You can also use the instructions at https://www.virtualbox.org/wiki/Linux_Downloads and go for an APT installation.The choice is yours.

10. Install Qemu (Optional, if you want to use qemu images)

$ sudo apt-get install qemu

11. Install IOU (Optional, if you want to use IOU images)

I’m not a legal matter expert, and the usage of IOU is sort of grey area. Because of this, I’m not going to cover this chapter.

You’re ready to go. Start the GNS3 GUI:

$ gns3

Some things to check before going live:

  • check in the menu Edit > Preferences to set your desired Paths (in General sections) and to check the paths for the binaries (dynamips, vpcs, iou, virtualbox…)
  • add the IOS, virtualbox vm, iou images
  • in case of Cisco hardware emulators don’t forget to find the IdlePC value (when you add the IOS image or later with the start of your first router with a certain image) otherwise your CPUs will cry.

If something does not work as described or you need help please let me know in Comments.

[Update 1]

If you get the following error during installation of iouyap:

GNS3-1.2.2.source/iouyap-0.95 $ make
gcc -g -DDEBUG -Wall -c -o iouyap.o iouyap.c
iouyap.c:40:23: fatal error: iniparser.h: No such file or directory
#include
^
compilation terminated.
make: *** [iouyap.o] Error 1

Try to install the iniparser as follows:

sudo apt-get install flex bison

then

cd /tmp
curl -L https://github.com/ndevilla/iniparser/archive/master.tar.gz | tar -xz
cd iniparser*
make

and finally iouyap

cd /tmp
curl -L https://github.com/GNS3/iouyap/archive/master.tar.gz | tar -xz
cd iouyap*
bison -ydv netmap_parse.y
flex netmap_scan.l
gcc -Wall *.c -I /tmp/iniparser*/src -L /tmp/iniparser* -o iouyap -liniparser -lpthread
strip --strip-unneeded iouyap
sudo mv iouyap /usr/local/bin

Thanks to mweisel @ forum.gns3.net for this update!

Cisco BGP soft-reconfiguration and received-routes relation

A while ago I received the following question:

“Why I’m not seeing the prefixes received from the BGP peer when using the show ip bgp neighbors x.x.x.x received-routes while the soft-reconfiguration inbound is not enabled?”

I must admit that I had to stop and think for a second before giving my response.

Simple BGP

For the above diagram I have a simple BGP configuration:

router bgp 65301
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 neighbor 192.168.0.2 remote-as 65302
 neighbor 192.168.0.2 timers 5 15
 !
 address-family ipv4
  network 1.1.1.1 mask 255.255.255.255
  network 2.2.2.2 mask 255.255.255.255
  network 3.3.3.3 mask 255.255.255.255
  neighbor 192.168.0.2 activate
 exit-address-family
router bgp 65302
 bgp log-neighbor-changes
 neighbor 192.168.0.1 remote-as 65301
 neighbor 192.168.0.1 timers 5 15

Focus is on the secondary router. I’m trying to see what prefixes I receive from my BGP neighbor, so I rapidly hit the following command:

R2#sh ip bgp neighbors 192.168.0.1 received-routes
% Inbound soft reconfiguration not enabled on 192.168.0.1

OK, that’s not good. Am I missing the command? Let’s see:

R2#sh ip bgp neighbors 192.168.0.1 ?
  advertised-routes  Display the routes advertised to a BGP neighbor
  dampened-routes    Display the dampened routes received from neighbor (eBGP
                     peers only)
  flap-statistics    Display flap statistics of the routes learned from
                     neighbor (eBGP peers only)
  paths              Display AS paths learned from neighbor
  policy             Display neighbor polices per address-family
  received           Display information received from a BGP neighbor
  received-routes    Display the received routes from neighbor
  routes             Display routes learned from neighbor
  |                  Output modifiers

-> received-routes – Display the received routes from neighbor

OK, I know the soft-reconfiguration inbound is not enabled, but what has this to do with the fact that the command is not showing me what routes I receive from BGP neighbor?

Let’s recall what “soft-reconfiguration inbound” command actually does.

BGP soft-reconfiguration inbound
*Cisco BGP-4 Command and Configuration Handbook

According to the above explanation, if you have an inbound policy (like a route-map) applied to a BGP neighbor and you change that policy, you need to clear the BGP session before it take effect. This is the procedure without having “soft-reconfiguration inbound” configured. I remember it like this and most of the network engineers out there remember this behavior associated with the “soft-reconfiguration inbound” command.

Still, I just want to see what routes I received from BGP neighbor and according with “sh ip bgp neighbors 192.168.0.1 received-routes” description is the right command . I have no inbound policy on R2 for R1 BGP neighbor.

A less remembered fact about the “soft-reconfiguration inbound” command is that when added, the router begins to store updates from the specified neighbor. These updates are unmodified by any existing inbound policies so that the router can correctly apply the new policies when soft reconfiguration is triggered.

Where are these updates stored?

In the BGP Adj-RIB-in (Adjacent Routing Information Base, Incoming) table. So, what the “received-routes” command does actually is looking in the BGP Adj-RIB-in table for the received routes from the BGP neighbor. I consider that the “received-routes” command has an ambiguous explanation leading to confusion.

When “soft-reconfiguration inbound” is not present, the BGP router does not store anything in Adj-RIB-in. Rather it process the update and discard the Adj-RIB-in table, but not before adding the information in the Loc-RIB (Local Routing Information Base) table. Knowing these facts of course the BGP router returns an error when trying to check the received prefixes using the “received-routes” command.

To check actually what’s received from BGP peer and stored in the Loc-RIB (after being processed by inbound policies) use only the “routes” parameter in the command:

R2#sh ip bgp neighbors 192.168.0.1 routes | b Net
     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       192.168.0.1              0             0 65301 i
 *>  2.2.2.2/32       192.168.0.1              0             0 65301 i
 *>  3.3.3.3/32       192.168.0.1              0             0 65301 i

Total number of prefixes 3

-> routes – Display routes learned from neighbor

The output is what exists in the Loc-RIB table, after processed by the inbound policy.

Let me show you an example of the above explanation.

I’ll apply the “soft-reconfiguration inbound” first:

R2(config-router)#neighbor 192.168.0.1 soft-reconfiguration inbound

Now I’ll check again the received routes:

R2#sh ip bgp neighbors 192.168.0.1 received-routes  | b Net
     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       192.168.0.1              0             0 65301 i
 *>  2.2.2.2/32       192.168.0.1              0             0 65301 i
 *>  3.3.3.3/32       192.168.0.1              0             0 65301 i

Total number of prefixes 3

OK, so I have three prefixes, reflected in the Adj-RIB-in table.
Checking next the Loc-RIB tables (so the routes installed after being processed by inbound policies):

R2#sh ip bgp neighbors 192.168.0.1 routes | b Net
     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       192.168.0.1              0             0 65301 i
 *>  2.2.2.2/32       192.168.0.1              0             0 65301 i
 *>  3.3.3.3/32       192.168.0.1              0             0 65301 i

Total number of prefixes 3

The Adj-RIB-in and Loc-RIB tables are identical.

Now, I’ll apply an inbound policy that will filter the 3.3.3.3 prefix.

R2(config)#ip prefix-list LIST permit 3.3.3.3/32
R2(config)#route-map INBOUND deny 10
R2(config-route-map)#match ip address prefix-list LIST
R2(config-route-map)#route-map INBOUND permit 1000
R2(config-route-map)#router bgp 65302
R2(config-router)#neighbor 192.168.0.1 route-map INBOUND in

OK, we have the “soft-reconfiguration inbound” in place, so the inbound policies should be applied automatically denying the 3.3.3.3 prefix.

R2#sh ip bgp neighbors 192.168.0.1 received-routes | b Net
     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       192.168.0.1              0             0 65301 i
 *>  2.2.2.2/32       192.168.0.1              0             0 65301 i
 *   3.3.3.3/32       192.168.0.1              0             0 65301 i

The above output is what we receive from BGP peer. Notice that the 3.3.3.3 prefix is still there, in the Adj-RIB-in table, as the inbound policies are not applied yet. The only visible change is the missing > sign (best).

R2#sh ip bgp neighbors 192.168.0.1 routes | b Net
     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       192.168.0.1              0             0 65301 i
 *>  2.2.2.2/32       192.168.0.1              0             0 65301 i

Now we can see that the inbound policy is working fine as the 3.3.3.3 prefix is not installed in the Loc-RIB table. This is also the explanation why the > (best) sign was missing from the 3.3.3.3 prefix in Adj-RIB-in.

I hope you understood the logic behind the confusion which these commands “received-routes” and “routes” and their explanation in IOS is creates.

Please let me know in Comments if you have any questions.

EGP

Today I came across an old Cisco router with original IOS image. Big surprise (at least for me) when I did check what routing protocols are supported on this router:

EGP protocol

I was out of the game, or better not even yet had discover the networking games, when the EGP was still out there and available to be configured on the Cisco routers.

I hope to bring a smile on your face or some nostalgic memories when you’ll see this :)