Last days I’ve encounter an issue when some of the Wireless Lightweight Access Points (WLAP) just disappeared from the Wireless LAN Controller (WLC).
I saw before these kind of problems before and usually, whatever the reason, the WAP cannot discover the WLC. It was not the case now, everything seems to be in order both in IP connectivity and correct parameters to point the WAP to correct WLC.
Looking back now, the problem is generated by an obvious issue, but back then it took me a bit to troubleshoot the issue. I’ll share my findings so others can resolve it quickly in case they hit this problem.
The WLC logs didn’t point out an obvious reason. Maybe is due to log volume and the fact that this particular WLC had other WAP which were working fine. Just couple of them suddenly disappeared.
I went the other way, and start troubleshooting from the WAP. Once I got remote access to the WAP (yes, it had an IP address and was reachable) the logs showed something like this:
WAP# *Oct 17 19:54:55.757: %DOT11-7-AUTH_FAILED: Station MAC_ADDRESS Authentication failed *Oct 17 19:54:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246 *Oct 17 19:54:56.352: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP *Oct 17 19:54:56.352: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246 *Oct 17 19:56:01.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC_IP peer_port: 5246 *Oct 17 19:56:01.364: %DTLS-5-ALERT: Received FATAL : Certificate unknown alert from WLC_IP *Oct 17 19:56:01.364: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to WLC_IP:5246
Was obvious that something is wrong with the CAPWAP tunnel and seemed to be related to the DTLS since the certificate unknown alert was present.
I’ll spare you the research around to figure it out, but finally I came to find this Field Notice: FN – 63942 – Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration – Software Upgrade Recommended – Cisco which in turn pointed to this BUG CSCuq19142. The BUG says that a WAP will fail to join a WLC if the SSC (self signed certificate) or MIC (manufactured installed certificate) has an expired date.
Going back to WAP CLI to check the MIC (SSC is not the case), it seemed that the suggested command “show crypto pki certificates” was not available. At least it seemed…
You need to add another command “debug capwap console cli” before to issue the “show crypto pki certificates“:
WAP# debug capwap console cli WAP# show crypto pki certificates !! removed output!! Certificate Status: Available Certificate Serial Number (hex): HEX_VALUE Certificate Usage: General Purpose Issuer: cn=Cisco Manufacturing CA o=Cisco Systems Subject: Name: AP_NAME [email protected] cn=AP_NAME o=Cisco Systems l=San Jose st=California c=US CRL Distribution Points: http://www.cisco.com/security/pki/crl/cmca.crl Validity Date: start date: 07:21:37 UTC Oct 13 2012 end date: 07:31:37 UTC Oct 13 2022 Associated Trustpoints: Cisco_IOS_MIC_cert Storage: !! removed output !!
If you check the validity date, seems this AP had a 10 years anniversary, which is also the default expiration date for the MIC installed certificate.
Checking the Field Notice above, it recommends to upgrade the WLC OS, but a lot of OS versions are affected, so in the meantime I went with the suggest workaround:
WLC> config ap cert-expiry-ignore mic enable
The WLC will ignore the MIC with expired date and as result the WAP will immediately join the WLC.
I hope this basic explanation and the quick workaround will help somebody if they run into the same issue.