Cisco | IPNET - Part 7

Cisco: Use TTCP to test together with TTCPW or JPerf

In one my previous posts, I shown you how to test a connection using a hidden Cisco IOS tool called TTCP. A few days ago I run into an issue. I had to test a TCP connection to a remote Cisco router, but I had not other router on which to initiate the TTCP connection. As explained in Testing TCP Connection post, to use TTCP you need 2 Cisco routers.

Now, I found 2 new ways to do the testing without the need of having 2 routers, but just one. Maybe you already know this methods, but for those who don’t please keep on reading.

First, there is a Windows tool called TTCPW (download here) (actually you can download also the code, and I think it’s possible to compile and run it under Linux as well). This TTCW tool have the same option like Cisco TTCP and can work together without any issue.
On Cisco router, issue the ttcp command, and keep the regular parameters (we are not interested for now in fine tuning). Below I set the Cisco router to be the receiver:

On PC side, you download TTCPW and use the same settings. Basically to transmit you only need to input ttcpw.exe -t or -r “ip.address” and that’s it Of course you can tune the settings to meet your needs. Just type ttcpw.exe to see all the settings.

The second tool that you can use with Cisco TTCP is IPerf (text mode) or JPerf (Java graphical mode). Just fill in the IP address and the port (5001 if default) and you’re ready to go:

Of course there are some limitations on JPerf to TTCP compared to JPerf to JPerf testing. One of then is that you cannot use parallel streams, if you want to stress the connection. To overcome this limitation, I do the following.
Open 2 or 3 connection to the Cisco router where TTCP will run. Start one each connection one TTCP daemon with different ports (e.g. assuming 3 connection than ports 5001, 5002 and 5003). Then on the client start 3 JPerf (Iperf) with the same IP address but different ports (you can take the one below). In this way you can stress the connection a little bit.

Cisco tips: Track down communication issues – Part 2

In the 1st Part of this series, I’ve described the most common steps that you should follow to troubleshoot a total lack of communication between a Layer 2 device (Cisco switch) and an end user connected device. As I promised here is the second part, in which I’ll try to show you what you can check when you have no problem with connection, but still you encounter a degradation in service. By this degraded service, I understand a scenario when you have packet loss for example, or intermitent connection which will affect communication and more than sure will make user users not very happy.

We will stick with the same scenario when a end user device is connected to a Cisco switch. Remember that until now, we just troubleshoot at the Layer 1 and Layer 2. Today we will stick in the same area, so nothing directly related to IP, routing protocols or complex networking environment.

Scenario 2: You have an end device connected to a switch and you have degraded communication

a) Check for errors on the interface:

In this example there is no errors, but if you find something there, you may want to keep an eye on this port. Try to issue the above command couple of times to see if the errors are increasing in real time, as this is the worst case possible and you should take action immediately. Error on the interface can be caused by faulty interface on the switch or on the other end, ethernet cable issue or wrong configuration

b) Check the interface queue and drop packets

Interface queue is very important and you should check it during your troubleshooting process. With the above command you can see how many packets are in the input / output queue, which is the transmit and receive rate and very important if you have packets dropped from input and output queue. Usually this happens when there is a lot of load on the switch and it cannot process as quick as it’s needed all the packets. This lead us to the next step.

c) Check the CPU load on the switch

The command output is longer but most interesting for this example are the first 2 rows which show load in 60 seconds and in 60 minutes. If you have there peaks up to 100, then it’s bad and the device is having some issues that need to be fixed.

d) Identify what process is keeping the CPU busy

Most of the time, this is easy to read and to see what process is taking all your CPU power. When you see there Fifo Error Detection with 100% than you have to think that maybe there is something wrong with the queue on one of the interfaces and try to find which one is having problem. This is not straighforward and you have to check a lot of things, but can be helpful. To be honest, I see a lot of engineers just reloading the device and then problem is solved (if it was due to a hardware issue and not a configuration mistake).

e) Check for memory issue on the switch

Again, if you run out of memory, bad things can happend to your device and as well to the communication with device connected to the switch. Reloading of the device solved about 90% of this kind of problems. I don’t recommend just unplug the power cable as soon as you see a memory problem. First have a look, maybe there is something you can fix without reloading the device.

f) Check for problems with storm-control implementation

In one of previous posts I have explained how you can use storm-control to limit the available bandwidth on a Cisco switch interface. In the example above I set this bandwidth to 1 % from the available one gigabit (I know is stupid, but imagine a typo mistake). Imagine what effect will this have on the traffic. Everything above 1 % is keeped in the queue until this is full and then silent discarded.

e) As a general rule, have a look into the logs (maybe this should be first step!)

If there are a lot of Spanning-tree reconfiguration, interface flapping or anything else that looks suspicious, be sure to check on this as you can find there the root cause for your problems.

Do you have any other tips in regard to this topic? Anything else you check and can be added here? Be sure to comment below and your suggestion will be taken into consideration.

Cisco Borderless Network – Phase 2

Everywhere where I turn my look in the last days I hear about the quick coming of Phase 2 of Cisco Borderless Network. If you are interested, you can register for the event on Cisco website .

I tried to search some documentation to understand what is Cisco Borderless Network and which is the big difference from Unified Communication , but all I could find is mostly marketing related documents which promise the next network miracle, everything interconnected, controlled and monitored from distance.

Then I turn myself to the Cisco blogging community to see what’s there, but also it seems that the things are not so clear there as well. Everybody know about the 5 phases of the new Cisco service:

Phase 1: Borderless Network Services – Delivering innovations IN and ON the network that optimize network availability, performance, and security.
Phase 2: Borderless User Services – Embedding key services spanning mobility, security, and application performance across all elements of the network.
Phase 3: Borderless Policy – Implementing a unified policy framework for managing security, identity, and access to the network and network resources.
Phase 4: Borderless Integration Framework – Bringing end-to-end network-to-endpoint intelligence through open, extensible interfaces into the network.
Phase 5: Borderless Experience – Converging services and systems to provide a superior customer and employee experience regardless of location, device, or application.

but I which also look more like a marketing ad, than network related topics. On phase 1, I was lucky enough and I could find some direction on technical blog of Ivan Pepelnjak, and in this phase it seems that everything was about the new ISR G2 release. Ivan have some objective observation about the Phase 1 of Cisco Borderless Network:

All the embedded “WAN” ports are Gigabit Ethernet uplinks. Good.
They claim up to 5-times higher performance than the previous routers. Average. The ISR series was launched in 2004 and Moore’s law predicts 5.8-times increase.
Lots of the old interface modules are supported. Amazing; I’m just hoping it doesn’t hurt the performance.
They’ve replaced the old half-hearted attempts to include an x86 generic application platform within a router with the Service Ready Engine (another great marketing invention … sounds so much better than a Linux blade) modules, having up to 4GB of RAM and 1TB of hard disk. I don’t want to know how the people who bought the old AXP platform feel reading these specs.

Read more about Ivan’s analysis on his blog.

Now going back to the Phase 2, Jim Duffy on NetworkWorld.com Cisco subnet is presuming that this will include :

innovations in Cisco’s switching portfolio and how they are relevant to the company’s business
service-enabling solutions for video, energy management and trusted access
a competitive switching offer for the price-sensitive market segment
enhanced support and services

but still there is enough fog around the subject. I think I will register for the Phase 2 European event, on March 18 maybe I will get some more clues about Cisco Borderless Network.

Skipping all this marketing and technical stuff, I’m wondering if the world is really prepared for this. And when I’m saying world I mean networks. I really don’t know what to say. I mean I know it’s cool and maybe positive from financial perspective to turn off light in some remote location, but how your network will support this. If you turn your look around you’ll see that network engineers have more stressing problems with day by day operation like VoIP services, slow data transfer between data centers and LAN security, just to name a couple of my problems.

Another problem is the word of the day: costs. How much it will cost to implement all of this and will companies be interested in implementing such a solution? I know that from marketing perspective everything looks great, because that’s their job, to make it look great, but network engineers might have a different opinion.

If you have some news or thoughts in regard to this topic, please use with trust the comment form.

Cisco tips: Track down communication issues – Part 1

You know how sometimes you plug in everything, configure device / ports and then the un-expected result is “not working”. Then you start to troubleshoot, which is a good point, but very important is where are you looking for the root cause of your issue.

A lot of people who have communication issue start by issuing a ping from one end to the other one. This is a good approach when somebody is reporting service issue (e.g. my webserver doesn’t work) as with the ping you can see immediately if is a communication issue or a server one. This scenario ussualy occurs when you already have a working environment and after a while somebody encounter a service issue.

Instead when you just deployed a new connection and you are having issue with it, ping is not the best approach. In the following article I will try to show you some good steps proven to be effective when you start troubleshooting. For today I will take as example a faulty communication between a Cisco switch and an end device (server or user device). This involes in the first step basic Layer 2 troubleshooting.

Scenario 1: You have an end device connected to a switch and you have no communication

a) Check the interface connection status:

OK – port is connected and protocol shows up status

Not OK – check the faulty port as it is in shutdown state

Not OK – port is not connected, protocol shows down status
You can check for cable error (damage, faulty plug, unplugged) or ask the owner of the remote device to check it.

If you found any error in the above step, try to fix them now. If the interface is connected, but still not working, follow on to the next steps.

b) Check speed and duplex settings
-if you have auto-negotiation here and it fails, you will end with an interface in down status
-again if you have static settings here, check to be the same on both sides.
-for more pro and cons regarding auto-negotiation vs static, please see Greg Ferro’s article

c) Check the interface switchport configuration:

There is no right and wrong configuration here, but I can point you to check the following:
– very important, check if the access VLAN is the right one
– if you have switchport auto-negotiation enabled, check to see that this is correctly achieved
– if you have switchport static configuration, check to have the correct settings for your needs
– if you need trunking (to an end device), check to allow the necessary VLANs on that trunk
– if you use private VLANs, pay attention to the configuration of primary and secondary vlan and right association

d) Check security on the Access port (BPDU guard, port-security, mac-address access-list…):

– port is in shutdown state due to Security Violation (1); The mac-address that you see there is the one coming to the port from the other end, but also you can see that there is a Configured Mac Address (1) on the port; Most probably the one configured on the port does not match the one from the network.

-if the result is like above one, you may want to check if that mac-address access-list allow communication from end device mac-address

– if your interface status is like this, you might have an issue with spanning-tree BPDUguard being enabled on the interface; I know I said that this is end device connected, but what if the user needed an extra port, and he connected there a switch? You always have to assume what’s the worst and check for possible issue.

f) Check the Spanning-tree protocol on the switch port

– your output might look different than the above one, but be sure to have there FWD (Forwarding) status if the port is connected to an end device.
– again it is very rare not to be in FWD status, but if the device has some strange bridging capabilities or user added another device in the middle, like a switch than you can identify a problem with Spanning-tree.

This are the basic stuff that I checked in regard to Layer 2 topology when I have no communication to the end host. In the next part, I will do a short presentation of the scenario when you have communication to end device, but the connection encounter traffic deprecation.

Check the 2nd Part of this series which deals with communication deprecation at Layer 2.

Do you have any other tips in regard to this topic? Anything else you check and can be added here? Be sure to comment below and your suggestion will be taken into consideration.

OSPF: Area range vs Summary address

It seems like an easy one! I mean what could be so hard about area range and summary address command? You will be surprised how many people tend to forget this things or to apply them when not necessary or even worst where is their place to be added. It’s basic knowledge, but sometimes exactly this basic knowledge give us headache.

In the following line, I will try to explain briefly the difference between Area range and Summary address command. First let’s have a look at the following topology:

We have mixed network domain with EIGRP and OSPF. From documentation we know that along other routers in OSPF domain we have 2 important routers:
ASBR (Autonomous System Boundary Router) – One device at the edge of OSPF domain, that receive routes from another non-OSPF domain (e.g. RIP, EIGRP…)
ABR (Area Border Router) – One device inside the OSPF domain that assure Inter-area communication (e.g. area 0 and area 1)

You will find situation when only one physical device will be ASBR and ABR router. The functionality and rules remain the same, just that instead of having 2 device you have only one that connect to another non-OSPF domain and in the same time to multiple OSPF areas.

Why we would use this 2 commands? Well one simple explanation is that we want to summarize advertised subnets to reduce the total number of routes present in the routing table. Less routes means less overhead and load for a router.

Let’s have a look to the routers presented in the above topology. The dynamic IP routing protocols (EIGRP and OSPF) are already configured and functional. For the IP network clouds, I have used Loopback interfaces.

R1:

On R1 the 2 IP subnets (192.168.1.0/24 and 192.168.2.0/24) are present in EIGRP routing protocol.

R2:

On R2, there is already a basic redistribution between OSPF and EIGRP. Also notice that the IP subnet which connect R2 and R3 is routed in OSPF area 0 (zero)

R3:

As well on R3, the IP subnet between R2 and R3 is present in OSPF area 0 (zero) and the other 2 subnets (10.10.1.0 /24 and 10.10.2.0 /24) are in OSPF area 1 (one). Now, if you remember from my older post, if you have a Loopback interface with a IP address (doesn’t matter what netmask) this will be always advertised in OSPF as /32. I did a little trick to be sure that they are still advertised in OSPF as presented under Loopack 1 (10.10.1.1 /24) and Loopback 2 (10.10.2.1 /24). You want to know how, check this post.

Until now, we saw how the routers are configured. Let’s have a look how the routing table appear now on this routers.
R1:

Among other routes, please notice that we receive the 2 networks from R3 each of them with a /24 prefix. Keep this in mind as it’s important for later.

R2:

On R2 we have both the EIGRP and OSPF routes, each with a /24 prefix. In some minutes we will change this. Let’s check the last router.

R3:

Here we have the 2 prefixes from EIGRP domain, again with /24.

After reviewing all the necessary information let’s apply the configuration. I will start with Area range command. For example in our case we want to advertise only one subnet from Area 1 to Area 0 in OSPF. We will combine the 2 x /24 subnets in one /22.
Why /22 and not /23? Because we have 10.10.1.0 /24 and 10.10.2.0 /24 and 10.10.1.0/23 will not be a valid prefix. Keep in mind that you have to stick to the subnetting rules. Indeed with this /22 we will “catch” also 10.10.0.0/24 and 10.10.3.0/24 in our range, but being in a test environment this is not a problem for now. In the real world you have to take care about discontinuous networks and to apply summarization only when it’s possible.

On our R3 router we should apply the following configuration:

R3
configure terminal
router ospf 1
area 1 range 10.10.0.0 255.255.252.0

If we check now R2, we should see:

Only one /22 subnet. IP subnet summarization is successfully taking place.

What about Summary address command? Remember that we redistribute from EIGRP into OSPF, so if you have a look about, right now in the OSPF cloud we have 2 /24 subnets imported from EIGRP domain. We want to summarize this 2 addresses in a /22 (the same reason like explained above). For this we will use the Summary address command. In short explanation, this command is only used on ASBR routers, when you want to summarize IP subnets imported from a non-OSPF domain.
On R2, we a apply the following configuration:

configure terminal
router ospf 1
summary-address 192.168.0.0 255.255.252.0

Let’s check what we receive on R1 and R3 after summarization:

We could see clear the advantage. Summary route 192.168.0.0/22 is present on R3 and again 10.10.0.0 /22 is on R1. This is the nice part and we achieved what we wanted.

Very important!
What about the blue line? That a big problem. Maybe your idea was that I did some mistake when using .1. and .2. in the third octet the subnet. It was more easier to use .0. and .1. and then I could summarize easy with /23. But I wanted to show you a hidden danger which lies beyond the summarization. Remember that I said I did just a basic redistribution between EIGRP and OSPF? Forget that! In test environment is OK, but in real ones not. Always try to reduce the amount of redistributed subnets between 2 domains to only the necessary one using route-maps or route tagging. This should be mandatory, when redistributed summarized networks which are discontinuous.

Back to the blue line. We redistribute from EIGRP to OSPF, where we did the summarization. Due to the fact that we used 192.168.1.0/24 and 192.168.2.0/24 we had to summarize to /22, which include also the non-present subnets 192.168.0.0/24 and 192.168.3/24. Then we redistribute from OSPF to EIGRP. See the loop? EIGRP-OSPF-EIGRP. Since EIGRP see there a /22 prefix which include the 2 additional /24 subnets 192.168.0.0 and 192.168.3.0 of which R1 has no clue about, the router install this route in it’s routing table, thinking that R2 is the gateway for the 2 prefixes above. It sound complicated but I tried to simplify the explanation as much as I could. If we were using route-maps or route tagging this situation could be avoided.