Cisco WLAP and WLC fail to create CAPWAP connection

Last days I’ve encounter an issue when some of the Wireless Lightweight Access Points (WLAP) just disappeared from the Wireless LAN Controller (WLC).

I saw before these kind of problems before and usually, whatever the reason, the WAP cannot discover the WLC. It was not the case now, everything seems to be in order both in IP connectivity and correct parameters to point the WAP to correct WLC.

Looking back now, the problem is generated by an obvious issue, but back then it took me a bit to troubleshoot the issue. I’ll share my findings so others can resolve it quickly in case they hit this problem.

The WLC logs didn’t point out an obvious reason. Maybe is due to log volume and the fact that this particular WLC had other WAP which were working fine. Just couple of them suddenly disappeared.
I went the other way, and start troubleshooting from the WAP. Once I got remote access to the WAP (yes, it had an IP address and was reachable) the logs showed something like this:

WAP#
*Oct 17 19:54:55.757: %DOT11-7-AUTH_FAILED: Station MAC_ADDRESS Authentication failed
*Oct 17 19:54:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246
*Oct 17 19:54:56.352: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP
*Oct 17 19:54:56.352: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246
*Oct 17 19:56:01.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246
*Oct 17 19:56:01.364: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP
*Oct 17 19:56:01.364: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246

Was obvious that something is wrong with the CAPWAP tunnel and seemed to be related to the DTLS since the certificate unknown alert was present.

I’ll spare you the research around to figure it out, but finally I came to find this Field Notice: FN – 63942 – Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration – Software Upgrade Recommended – Cisco which in turn pointed to this BUG CSCuq19142. The BUG says that a WAP will fail to join a WLC if the SSC (self signed certificate) or MIC (manufactured installed certificate) has an expired date.

Going back to WAP CLI to check the MIC (SSC is not the case), it seemed that the suggested command “show crypto pki certificates” was not available. At least it seemed…

You need to add another command “debug capwap console cli” before to issue the “show crypto pki certificates“:

WAP# debug capwap console cli
WAP# show crypto pki certificates
!! removed output!!
Certificate
  Status: Available
  Certificate Serial Number (hex): HEX_VALUE
  Certificate Usage: General Purpose
  Issuer:
    cn=Cisco Manufacturing CA
    o=Cisco Systems
  Subject:
    Name: AP_NAME
    [email protected]
    cn=AP_NAME
    o=Cisco Systems
    l=San Jose
    st=California
    c=US
  CRL Distribution Points:
    http://www.cisco.com/security/pki/crl/cmca.crl
  Validity Date:
    start date: 07:21:37 UTC Oct 13 2012
    end   date: 07:31:37 UTC Oct 13 2022
  Associated Trustpoints: Cisco_IOS_MIC_cert
  Storage:
!! removed output !!

If you check the validity date, seems this AP had a 10 years anniversary, which is also the default expiration date for the MIC installed certificate.

Checking the Field Notice above, it recommends to upgrade the WLC OS, but a lot of OS versions are affected, so in the meantime I went with the suggest workaround:

WLC> config ap cert-expiry-ignore mic enable

The WLC will ignore the MIC with expired date and as result the WAP will immediately join the WLC.

I hope this basic explanation and the quick workaround will help somebody if they run into the same issue.

Docker image – Python for network engineers

Lately I’m looking more and more into Python, with respect to automation implementations useful for network engineers. In the learning process I’ve used different materials, like the excellent video trainings Python Programming for Network Engineers from David Bombal which are available free on Youtube.

This training in particular relies on a Ubuntu Docker image in order to support Python learning following interaction with Cisco devices in GNS3. Everything is great, just that the image doesn’t contain all necessary tools (like Paramiko, Netmiko, Ansible…). As you can guess, whenever you close / open the Project in GNS3, all the installed packages installed in the Ubuntu Docker image are gone.

Since we’re talking automation, I got bored to install the necessary tools everytime I wanted to start a new project or I had to close GNS3 for some reason. I’ve tried to find a Docker image that suits my needs, but I couldn’t (please point me to one if you know it).

So, I’ve build a Docker image, based on Ubuntu 16.04, which contains the necessary tools to start learning Python programming oriented for network engineers:

  • Openssl
  • Net-tools (ifconfig..)
  • IPutils (ping, arping, traceroute…)
  • IProute
  • IPerf
  • TCPDump
  • NMAP
  • Python 2
  • Python 3
  • Paramiko (python ssh support)
  • Netmiko (python ssh support)
  • Ansible (automation)
  • Pyntc
  • NAPALM

The above list can be extended, but I would like to keep it to the minimum necessary (I want to keep the image size at decent level).

If you’re interested, please find the image at: https://hub.docker.com/r/yotis/ubuntu1604-pfne/, or you can download it:

$ docker pull yotis/ubuntu1604-pfne

I’ve tested the image for couple of days and it works fine. However if something doesn’t work as expected, please let me know and I’ll try to fix it.

For those using GNS3 is possible to import the image above directly into GNS3 using the PFNE Appliance.

Cisco ASA packet capture showing bidirectional traffic flow

Recently I had to troubleshoot some communication issues via a Cisco ASA device and the packet capture on the IOS comes in handy for this task.

When you have a lot of traffic over ASA and you’re interested in a particular IP address, the basic packet capture lesson says that you should configure an access-list to limit the captured packets for the interesting traffic only.

Let’s assume that I have a particular interest for the traffic to and from the IP address 10.0.0.10.

I created a standard ACL to match only the traffic related to 10.0.0.10:

access-list TS standard permit host 10.0.0.10

Afterward I attached the created ACL to a packet capture on a particular interface (let’s call it “lan”).

capture TSHOOT access-list TS interface lan

You can find the above lines in almost any how-to regarding packet capture on Cisco ASA.

Checking the capture I noticed that traffic is unidirectional captured:

FW# show capture TSHOOT

4 packets captured

   1: 20:15:32.757010       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   2: 20:15:33.759283       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   3: 20:15:34.761374       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   4: 20:15:35.823748       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request

This is not enough to troubleshoot complex communication scenarios.

Ok, maybe the standard ACL is not enough, so I tried to use extended one where 10.0.0.10 is source on one line and destination on another:

access-list TS extended permit ip host 10.0.0.10 any
access-list TS extended permit ip any host 10.0.0.10

This should do it…just that it doesn’t.

capture TSHOOT access-list TS interface lan

ERROR: Capture doesn't support access-list  containing mixed policies

Hmm, maybe it does not work with two lines in the ACL. I removed one, same error.

I was looking around to find a way to do it, but I couldn’t. This is why I wrote this article. Maybe my googling skills are not so good, as I’m sure it has to be an example somewhere out there.

However, here how I did it.

I gave up using the ACL. No, I’m was not going to capture the entire traffic :) Instead, I used inline restrictions for the IP address that I’m interested in.

capture TSHOOT interface lan match ip host 10.0.0.10 any

The result looks good now:

FW# show capture TSHOOT

8 packets captured

   1: 20:19:07.222553       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   2: 20:19:07.223392       802.1Q vlan#10 P0 10.0.0.10 > 192.168.0.10: icmp: echo reply
   3: 20:19:08.229953       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   4: 20:19:08.230670       802.1Q vlan#10 P0 10.0.0.10 > 192.168.0.10: icmp: echo reply
   5: 20:19:09.229327       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   6: 20:19:09.230121       802.1Q vlan#10 P0 10.0.0.10 > 192.168.0.10: icmp: echo reply
   7: 20:19:10.252321       802.1Q vlan#10 P0 192.168.0.10 > 10.0.0.10: icmp: echo request
   8: 20:19:10.253130       802.1Q vlan#10 P0 10.0.0.10 > 192.168.0.10: icmp: echo reply

The packet capture shows now bidirectional traffic flow.

I hope you’ll find this useful during troubleshooting.

Mikrotik L2TP with IPsec for mobile clients

I got some questions about how to configure Mikrotik to act as L2TP Server with IPsec encryption for mobile clients. I know this is not exactly in the line of this blog oriented on enterprise networks, but it’s network technology in the end so I’ll try to cover it here.

Before we start, please make sure that your Mikrotik build-in firewall is configured in such way that it can accept packets on the WAN interface. You can check my article on IPsec VPN Mikrotik to Cisco for firewall configuration.

Another important part is that I’m using RouterOS v6.24 in the below scenario. In earlier versions some configurations are a bit different, but you’ll figure it out as I will explain where is really important.

1. Add a new IP Pool

It’s not mandatory if you already have a IP Pool, but I assume you don’t and we need to add one.

GUI

IP > Pool

Add a new pool

Name: L2TP-Pool
Adresses: 172.31.86.1-172.31.86.14
Next Pool: None

CLI

/ip pool add name="L2TP-Pool" ranges=172.31.86.1-172.31.86.14

L2TP Configuration

1. Configure L2TP Profile

Before adding a new L2TP Server, we need to add a new L2TP Profile. We can use also the default one, but I don’t like to mix things.

GUI

PPP > Profiles

Name: l2tp-profile
Local Address: L2TP-Pool
Remote Address: L2TP-Pool
DNS Server: 8.8.8.8
Change TCP MSS: yes
Use Encryption: required

The rest of values can be left on default value.

CLI

/ppp profile add name=l2tp-profile local-address=L2TP-Pool remote-address=L2TP-Pool use-encryption=required change-tcp-mss=yes dns-server=8.8.8.8

2. Add a L2TP-Server

GUI

PPP > Interface > L2TP Server

Enabled: Checked
Max MTU: 1460
Max MRU: 1460
Keepalive Timeout: 30
Default Profile: mschap2
Use IPsec: Checked
IPsec Secret: MYKEY

CLI

/interface l2tp-server server set authentication=mschap2 default-profile=l2tp-profile enabled=yes ipsec-secret=MYKEY max-mru=1460 max-mtu=1460 use-ipsec=yes

3. Add PPP Secrets

GUI

PPP > Secrets

Enabled: Checked
Name: MYUSER
Password: MYPASSWORD
Service: l2tp
Profile: l2tp-profile

Let the rest as default.

CLI

/ppp secret add name=MYUSER password=MYPASSWORD service=l2tp profile=l2tp-profile

IPsec Configuration

On IPsec configuration, you can use the default configuration (like Proposals) but I would suggest to let those as default and add your new ones. In case that you already have some IPsec configuration which is already working and using the default configuration we don’t want to mess with that.

1. IPsec Proposals

GUI

IPsec > Proposals

Enabled: Checked
Name: L2TP-Proposal
Auth. Algorithm: sha1
Encr. Algorithm: 3des, aes-256 cbc
PFS Group: none

CLI

/ip ipsec proposal add name=L2TP-Proposal auth-algorithms=sha1 enc-algorithms=3des,aes-256-cbc pfs-group=none

Something to mention here. In version previous than 6.xx, you can pick only one encryption algorithm, if I remember correctly. You cannot add multiple algorithms (like 3des and aes-256 above). If this is the case, be sure to stay with 3des. I know it offer less security, but for some reason I could not force Microsoft Windows to work on L2TP via aes-256.

2. IPsec Peers

GUI

IPsec > Peers

Enabled: Checked
Address: 0.0.0.0
Auth. Method: pre shared key
Secret: MYKEY
Policy Template Group: default
Exchange Mode: main l2tp
Send Initial Contact: Checked
NAT Traversal: Checked
My ID: auto
Proposal check: obey
Hash Algorithm: sha1
Encryption Algorithm: 3des, aes-256
DH Group: modp1024
Generate policy: port override

CLI

/ip ipsec peer add address=0.0.0.0/0 port=500 auth-method=pre-shared-key secret="MYKEY" generate-policy=port-override exchange-mode=main-l2tp
 send-initial-contact=yes nat-traversal=yes hash-algorithm=sha1 enc-algorithm=3des,aes-256 dh-group=modp1024

IMPORTANT

The value of the Secret field above, MUST be the same as in L2TP Configuration, Step 2.
Also, if your RouterOS support only one encryption algorithm, then pick 3des.

3. IPsec Policies

GUI

Enabled: Checked
Src. Address: ::/0
Dst. Address: ::/0
Protocol: 255(all)
Template: Checked
Group: default
Action: encrypt
Level: require
IPsec Protocols: esp
Tunnel: Not checked
SA Src. Address: 0.0.0.0
SA Dsr. Address: 0.0.0.0
Proposal: L2TP-Proposal

CLI

/ip ipsec policy add src-address=::/0 dst-address=::/0 protocol=all template=yes group=default action=encrypt level=require ipsec-protocols=esp tunnel=no sa-src-address=0.0.0.0 sa-dst-address=0.0.0.0 proposal=L2TP-Proposal

Below, I’ll add two examples how to configure the iPhone and Microsoft Windows to work with the above configuration.

iPhone

Go to Settings, VPN section and Add VPN Configuration…

It will look like this:

iPhone L2TP Configuration

The Server is the public IP address or FQDN of your Mikrotik. Account and Password are the one defined in L2TP Configuration Step 3. (MYUSER and MYPASSWORD in the example above). Secret , is the IPsec Secret Key defined in L2TP Configuration Step 2. and IPsec Configuration Step 2. (MYKEY in the example).

PC with Microsoft Windows

1. Add a new VPN connection

Add New VPN Connection

2. Pick the option Use my Internet connection

Chose VPN type

3. Add Mikrotik L2TP Server details

Add L2TP Mikrotik details

4. Add the user and password

Add L2TP user

Add this point Windows 7 force me to hit Connect. I will not work yet. Please follow the next steps.

You need to reach the Properties of your new VPN connection.

5. Configure the VPN Security settings.

Be sure to have the settings like in image below, to force encryption and use mschap2 protocol.

L2TP VPN Security

6. Set the IPsec Secret key

Hit the Advanced button and set the IPsec key

L2TP IPsec Key

Hit Connect and it will work. If you have questions please be sure to add them to Comments.

IPsec VPN Mikrotik to Cisco

Not long ago I wrote an article on how to configure an IPsec VPN using Mikrotik and Linux devices. For today, I will replace the Linux device with a Cisco. I did test the entire construct in GNS3 integrated with Mikrotik.

The topology looks like this:

IPsec VPN Mikrotik Cisco

The red line represent the IPsec VPN tunnel.
Please note the used IP addresses. In this way the below configuration will be easier to understand.

Mikrotik Configuration

1. Firewal rules

By default, the Mikrotik comes with the INPUT channel that drop the connection incoming on ether1-gateway (which is the WAN interface). You need to be sure that at least the IPsec packets are able to be accepted inbound on the WAN interface, so the below rules needs to be placed before the rule dropping packets (the Firewal rules are checked top-down)

On INPUT channel allow the following on the interface facing Internet
– Port 500/UDP
– Port 4500/UDP
– Proto 50
– Proto 51
It may be that you don’t need all these ports, but you can close them later. You can check logs if you want to troubleshoot.

On NAT channel, SRCNAT you need have the rule involving interesting traffic (local LAN subnets for example) before NAT masquerade.
You need to add a rule with ACCEPT source LOCAL_LAN (192.168.88.0/24 in this example) destination REMOTE_LAN (192.168.0.0/24 in this example).

On Console the configuration looks like this:

CLI

!
ip firewall filter add chain=input proto=ipsec-ah action=accept place-before=0
ip firewall filter add chain=input proto=ipsec-esp action=accept place-before=0
ip firewall filter add chain=input proto=udp port=500 action accept place-before=0
ip firewall filter add chain=input proto=udp port=4500 action accept place-before=0
!
ip firewall nat add chain=srcnat src-address=192.168.88.0/24 dst-address=192.168.0.0/24 action=accept place-before=0

2. The IPsec Proposal

GUI

IP > IPsec > Proposals

Name: MyProposal
Auth. Algorithm: sha1
Encr. Algorithm: aes-256 cbc
PFS Group: none

CLI

ip ipsec proposal add name=MyProposal auth-algorithms=sha1 enc-algorithms=aes-256-cbc pfs-group=none

3. The IPsec Policy

GUI

IP > IPsec > Policies

SRC ADDR: 192.168.88.0/24
DST ADDR: 192.168.0.0/24
Protocol: all
Action: Encrypt
Level: require
IPsec protocols: esp
Tunnel: check
SA SRC: 10.0.0.2
SA DST: 192.168.23.3
Proposal: MyProposal

CLI

ip ipsec policy add src-address=192.168.88.0/24 dst-address=192.168.0.0/24 protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=10.0.0.2 sa-dst-address=192.168.23.3 proposal=MyProposal

4. The IPsec Peer

GUI

IP > IPsec > Peers

Address: 192.168.23.3
Port: 500
Auth. Method: pre shared key
Passive: not checked
Secret: MYKEY
Policy Template Group: default
Exchange mode: main
Send Initial Contact: checked
NAT Traversal: checked
My ID: Auto - empty
Proposal Check: obey
Hash Algorithm: sha1
Encryptions Algorithm: aes-256
DH Group: modp1024
Generate policy: no

CLI

ip ipsec peer add address=192.168.23.3 port=500 auth-method=pre-shared-key secret=MY_KEY exchange-mode=main send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-256 dh-group=modp1024 generate-policy=no

Cisco configuration

1. Crypto ISAKMP Policy

crypto isakmp policy 1
encr aes 256
authentication pre-share
group 2

You can specify also the hash as sha1, but this is the default method on Cisco, so no extra line will appear.

2. Crypto ISAKMP neighbor

crypto isakmp key MYKEY address 10.0.0.2 no-xauth

3. Crypto IPsec transformation set

crypto ipsec transform-set MYTRANSFORMSET esp-aes 256 esp-sha-hmac
 mode tunnel

4. Crypto map

crypto map MYCRYPTOMAP 10 ipsec-isakmp
 description Mikrotik VPN
 set peer 10.0.0.2
 set transform-set MYTRANSFORMSET
 match address ACLTRAFF

5. Access-list for interesting traffic

ip access-list extended ACLTRAFF
 permit ip 192.168.0.0 0.0.0.255 192.168.88.0 0.0.0.255

6. Interface config

int fa1/0
 description Internet facing interface
 crypto map MYCRYPTOMAP

The settings (like encryption algorithm) can be tuned to fit your requirements.

If you have any questions or something is unclear please let me know in Comments.