Troubleshooting with Wireshark [Riverbed lab kit]

A while ago I attended a Wireshark webinar from Riverbed in which they presented the tool, some beginner and intermediate users troubleshooting scenarios and some lab kit.

Now I got an e-mail that they made it available for download at http://www.riverbed.com/wireshark-virtual-tour

Part of this Lab Kit were available in the Virtual World Tour 2014 webinar on Troubleshooting with Wireshark, held by Laura Chappell and Gerald Combs.

Kit is a free collection of Wireshark training, trace files and tips and tricks for troubleshooting your network. This lab kit contains the following:

  • Nine Network Analysis Training videos
  • Troubleshooting Checklist
  • Sample Network Analysis Report
  • Practice Trace Files
  • Laura’s Wireshark Troubleshooting Profile
  • Chapter Excerpts from Laura Chappell’s new book “Troubleshooting with Wireshark Locate the Source of Performance Problems”


    • If you are interested in troubleshooting with Wireshark, I think this would be a nice place to start. The presentation page of Wireshark Virtual Tour looks a little bit like comics marketing style, but the materials are pretty good.

    Live digital threats

    I wanted to bring these free tools to my blog to keep track of them, as most probably I will forget about, and to share with anybody interested in gathering information in the security areas.

    First tool was revealed to my in tweet from Greg Ferro:

    https://twitter.com/etherealmind/status/392606499044098049

    As usually I trust his words, I said to give it a try. I can tell that this tool is very interesting. According to the explanation found on http://www.digitalattackmap.com/faq/ :

    The Digital Attack Map presents data gathered and published by Arbor Networks ATLAS® global threat intelligence system. ATLAS sources its data worldwide from 270+ ISP customers who have agreed to share anonymous network traffic and attack statistics. Data is updated hourly and can also be found in Arbor’s ATLAS Threat Portal.

    Below you have an embedded version of this map, but better access the Digital Attack Map website.

    Second tool caught my attention from the Digital Attack Map explanation and it’s called Arbor’s ATLAS Threat Portal.

    Atlat Threat Portal

    This tool is explained as:

    The ATLAS portal today is a public resource that delivers a sub-set of the intelligence derived from the ATLAS sensor network on host/port scanning activity, zero-day exploits and worm propagation, security events, vulnerability disclosures and dynamic botnet and phishing infrastructures.

    If the first tool is related to network attacks, the second one is covering more topics like botnets, phishing, scans and more.
    You can access Arbor’s ATLAS Threat Portal here.


    Cisco switches and smartport macros

    Smartport macros are not more than some templates you can define on Cisco switches that will apply the same configuration on multiple ports. It’s not a subject that needs too many discussions, but it can be useful for your Cisco certification preparation or real life Cisco switch administration.

    Configuration is very simple and it goes something like this:

    macro name ACCESS-PORT
switchport mode access
switchport access vlan 6
switchport voice vlan 7
spanning-tree portfast
spanning-tree bpdufilter enable
@

    After this you apply the macro to a port or a range of ports:

    interface range fa0/1 - 6
macro apply ACCESS-PORT

    That’s it :)

    A less known fact is that Cisco switches are having some predefined smartport macros, which can be really helpful. The smartport macros which you configure can be spotted with a simple “show running-config” command. This is not the case for the default smartport macros which cannot be seen in the running-config, so you may not be aware that they exist.

    The default smartport macros can be seen using the following commands:

    SW1#show parser macro brief
    default global   : cisco-global
    default interface: cisco-desktop
    default interface: cisco-phone
    default interface: cisco-switch
    default interface: cisco-router
    default interface: cisco-wireless

    This will show you only a summary of the default smartport macros. If you want to see what are they configure to do, check the following command:

    SW1# show parser macro
Total number of macros = 7
--------------------------------------------------------------
Macro name : cisco-global
Macro type : default global
# Enable dynamic port error recovery for link state failures.
errdisable recovery cause link-flap
errdisable recovery interval 60
 
# Config Cos to DSCP mappings
mls qos map cos-dscp 0 8 16 24 32 46 46 56
 
# Enable aggressive mode UDLD on all fiber uplinks
udld aggressive
 
# Enable Rapid PVST+ and Loopguard
spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree extend system-id
--------------------------------------------------------------
Macro name : cisco-desktop
Macro type : default interface
# macro keywords $access_vlan
# Basic interface - Enable data VLAN only
# Recommended value for access vlan should not be 1
switchport access vlan $access_vlan
switchport mode access
 
# Enable port security limiting port to a single
# MAC address -- that of desktop
switchport port-security
switchport port-security maximum 1
 
# Ensure port-security age is greater than one minute
# and use inactivity timer
switchport port-security violation restrict
switchport port-security aging time 2
switchport port-security aging type inactivity
 
# Configure port as an edge network port
spanning-tree portfast
spanning-tree bpduguard enable
--------------------------------------------------------------
Macro name : cisco-phone
Macro type : default interface
# Cisco IP phone + desktop template
 
# macro keywords $access_vlan $voice_vlan
 
 
# VoIP enabled interface - Enable data VLAN
# and voice VLAN
# Recommended value for access vlan should not be 1
switchport access vlan $access_vlan
switchport mode access
 
# Update the Voice VLAN value which should be
# different from data VLAN
# Recommended value for voice vlan should not be 1
switchport voice vlan $voice_vlan
 
# Enable port security limiting port to a 2 MAC
# addressess -- One for desktop on data vlan and
# one for phone on voice vlan
switchport port-security
switchport port-security maximum 2
 
# Ensure port-security age is greater than one minute
# and use inactivity timer
switchport port-security violation restrict
switchport port-security aging time 2
switchport port-security aging type inactivity
 
# Enable auto-qos to extend trust to attached Cisco phone
auto qos voip cisco-phone
 
# Configure port as an edge network port
spanning-tree portfast
spanning-tree bpduguard enable
--------------------------------------------------------------
Macro name : cisco-switch
Macro type : default interface
# macro keywords $native_vlan
# Access Uplink to Distribution
# Do not apply to EtherChannel/Port Group
switchport trunk encapsulation dot1q
 
# Define unique Native VLAN on trunk ports
# Recommended value for native vlan should not be 1
switchport trunk native vlan $native_vlan
 
# Update the allowed VLAN range such that it
# includes data, voice and native VLANs
switchport trunk allowed vlan ALL
 
# Hardcode trunk
switchport mode trunk
 
# Configure qos to trust this interface
auto qos voip trust
 
# 802.1w defines the link as pt-pt for rapid convergence
spanning-tree link-type point-to-point
--------------------------------------------------------------
Macro name : cisco-router
Macro type : default interface
# macro keywords $native_vlan
# Access Uplink to Distribution
switchport trunk encapsulation dot1q
 
# Define unique Native VLAN on trunk ports
# Recommended value for native vlan should not be 1
switchport trunk native vlan $native_vlan
 
# Update the allowed VLAN range such that it
# includes data, voice and native VLANs
switchport trunk allowed vlan ALL
 
# Hardcode trunk
switchport mode trunk
 
# Configure qos to trust this interface
auto qos voip trust
mls qos trust dscp
 
# Ensure fast access to the network when enabling the interface.
# Ensure that switch devices cannot become active on the interface.
spanning-tree portfast trunk
spanning-tree bpduguard enable
--------------------------------------------------------------
Macro name : cisco-wireless
Macro type : default interface
# macro keywords $native_vlan
# Access Uplink to Distribution
switchport trunk encapsulation dot1q
 
# Define unique Native VLAN on trunk ports
# Recommended native vlan should NOT be 1
switchport trunk native vlan $native_vlan
 
# Update the allowed VLAN range such that it
# includes data, voice and native VLANs
switchport trunk allowed vlan ALL
 
# Hardcode trunk and disable negotiation to speed up convergence
switchport mode trunk
switchport nonegotiate
 
# Configure qos to trust this interface
auto qos voip trust
mls qos trust cos
 
# Ensure that switch devices cannot become active on the interface.
spanning-tree bpduguard enable
--------------------------------------------------------------
Macro name : VLAN_146
Macro type : customizable
switchport mode access
switchport access vlan 146
spanning-tree bpdufilter enable
--------------------------------------------------------------

    To be honest I never used them like this, but they were a pretty good starting point to customize new smartport macros.

    If you are rather interested in the Cisco switch interface macro command, I did write a post on this topic some years ago and you can read it here.


    Cisco STP forward-time command trick

    This is more a topic for exams, like Cisco CCIE, but also it can appear in real-world environments.

    Consider the following simple topology:
    Simple L2 topology

    SW1 is the spanning-tree root bridge for all VLANs. Imagine that you have a request which ask you that when a port becomes active, no matter of VLAN, it should wait 10 seconds until it transition to forwarding state.

    You look into configuration mode and spot the “spanning-tree .. forward-time..” command. OK, simple enough, you go there and type:

    spanning-tree vlan 1-4094 forward-time 10

    Task completed. This is what I also thought. Looks pretty straight forward, but it’s a mistake which I learned from and I hope you will read this before doing it.
    Lucky for me it was just a test lab not the real CCIE lab, but if I don’t teach this kind of things or spend the entire day fine-tuning STP, I tend to forget.

    Back to this example, let’s see on the SW2 what’s happening after I did configure the above command on SW1. By the way, you need to configure this command only on the STP root bridge because the downstream devices will inherit the values.

    On the SW2 I shutdown the root interface to force the alternate interface (which is in blocking state) to transition to forward state and check the output of “debug spanning-tree events” :

    00:55:15: STP: VLAN0001 new root port Fa0/2, cost 19
00:55:15: STP: VLAN0001 Fa0/2 -> listening
...
00:55:25: STP: VLAN0001 Fa0/2 -> learning
...
00:55:35: STP: VLAN0001 sent Topology Change Notice on Fa0/2
00:55:35: STP: VLAN0001 Fa0/2 -> forwarding

    From :15 seconds when the port entered Listening state until :35 seconds when it went into Forwarding state there are 20 seconds. We were required to have 10 seconds not 20. That’s not good.

    Going back and checking the parameters of a STP for a particular VLAN did not give me too much information to clarify the issue:

    show spanning-tree vlan 1 | i Forward
Hello Time   3 sec  Max Age 10 sec  Forward Delay 10 sec

    OK, after reading again carefully the documentation I came to the understanding that value added to this command is applied to each state towards the forwarding state.
    The STP Port States are: Blocking, Listening, Learning, Forwarding and Disabled.

    When I did shutdown the root port, the alternate port when immediately to Listening state then waited for 10 seconds until it transition to Learning state and again 10 Seconds to achieve Forwarding state.

    The correct solution was obvious now to decrease the forward-time to 5 seconds.

    spanning-tree vlan 1-4094 forward-time 5

    Let’s check again:

    01:11:55: STP: VLAN0001 new root port Fa0/2, cost 19
01:11:55: STP: VLAN0001 Fa0/2 -> listening
...
01:12:00: STP: VLAN0001 Fa0/2 -> learning
...
01:12:05: STP: VLAN0001 sent Topology Change Notice on Fa0/2
01:12:05: STP: VLAN0001 Fa0/2 -> forwarding

    Now the total time is 10 second.

    I hope this will help others to avoid my mistakes.


    EtherChannel over 802.1q Tunneling

    Consider the following topology:
    EtherChannel over 802.1q Tunneling Topology

    We have one Customer with two distributed locations (SW1, R1 and SW2, R2) connected over Provider backbone. What we want to create is something like this:

    EtherChannel over Provider L2 cloud

    If Provider support 802.1q and L2 tunneling we can achieve a nice Etherchannel between our 2 remote locations with direct CDP visibility. Also STP and VTP is supported, just like when these SW1 and SW2 switches are directly connected.

    First, lets configure SW1 and SW2 Customer devices.

    On the three interfaces connected to provider devices we want to configure LACP Etherchannel:

    SW1 / SW2 Customer

    interface FastEthernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active
!
interface FastEthernet0/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active
!
interface FastEthernet0/3
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active

    Next we will configure the SW1 and SW2 ports connected to R1 and R2 devices:

    SW1 / SW2 Customer

    vtp mode transparent
vtp domain Customer
vlan 100
 name End2End
!
interface Fa0/10
switchport mode access
switchport access vlan 100

    Of course another approach can be taken in terms of VTP, like having Server / Client configuration, but this was the simplest one to illustrate here.

    Let’s add some IP addresses on the two routers R1 and R2:

    R1 Customer

    interface fa0/0
ip address 10.0.0.1 255.255.255.0

    R2 Customer

    interface fa0/0
ip address 10.0.0.2 255.255.255.0

    Our job, as Customer, is done. What about the Provider configuration? Here is where “the magic” happens.

    To provide our Customer with three end to end 802.1q tunnels, we need to create three VLANs, assign them to the interfaces pointing to Customer SW1 and SW2 and enable the 802.1q tunnels.

    SW1 / SW2 Provider

    vlan 10
vlan 20
vlan 30
!
interface FastEthernet0/1
 switchport access vlan 10
 switchport mode dot1q-tunnel
!
interface FastEthernet0/2
 switchport access vlan 20
 switchport mode dot1q-tunnel
!
interface FastEthernet0/3
 switchport access vlan 30
 switchport mode dot1q-tunnel

    Of course SW1 and SW2 from Provider should have 802.1q trunk enable and allow the tranport of VLANs 10, 20 and 30:

    SW1 / SW2 Provider

    int fa0/4
switchport trunk mode dot1q
switchport mode trunk
switchport trunk allowed vlan 10,20,30

    OK, we have the dot1q tunneling enabled now:

    SW1 / SW2 Provider

    show dot1q-tunnel
 
dot1q-tunnel mode LAN Port(s)
-----------------------------
Fa0/1
Fa0/2
Fa0/3

    Still, the Customer wants Etherchannel functionality, CDP visibility and the ability to transport own VLAN information (remember we did configure Vlan 100 on the interface of SW1 / SW2 Customer pointing to R1 / R2). Let’s enable also these ones:

    SW1 / SW2 Provider

    interface FastEthernet0/1
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable
!
interface FastEthernet0/2
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable
!
interface FastEthernet0/3
l2protocol-tunnel point-to-point lacp
l2protocol-tunnel cdp
l2protocol-tunnel stp
no cdp enable

    Perfect, now let’s do some “show” commands to see that everything is working.

    SW1 / SW2 Customer

    show etherchannel 1 summary | b Group
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/1(P)   Fa0/2(P)   Fa0/3(P)
    show spanning-tree vlan 100
 
VLAN0100
  Spanning tree enabled protocol ieee
  Root ID    Priority    32868
             Address     0011.20ab.6180
             Cost        9
             Port        56 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
 
  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)
             Address     0014.a86b.f600
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300
 
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/10              Desg FWD 19        128.3    P2p
Po1                 Root FWD 9         128.56   P2p

    OK, the Etherchannel is UP and the STP is showing correct values. Let’s see if we can do a simple “ping” from R1 to R2

    R1#ping 10.0.0.2
 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/24 ms

    The Customer is happy, but what about the Provider, what does it see on the L2 infrastructure?

    SW1 / SW2 Provider

    show spanning-tree vlan 100
Spanning tree instance(s) for vlan 100 does not exist.

    So, the Provider has no idea about Vlan 100 used by the Customer. This is because STP BPDUs from SW1 / SW2 Customer are tunneled inside dot1q-tunnel and hidden by the metro tags 10, 20 and 30.

    One note for real life example, the Provider needs to support at least MTU 1504 so that Customer does not deal with packet fragmentation.