[Solution] Speed / Duplex auto-negotiation fails between Cisco and Tandberg

In the last weeks I was working closely with a Cisco Telepresence team to identify a issue regarding poor performance of the video systems. We did find pretty quickly the issue as being the failure of auto-negotiation of Speed and Duplex on the connection between Cisco switch port and Tandberg endpoint devices.

This was the easy part. We though it will be fixed in minutes, but after a few days we did recognized that there is something we do not understand. We did change the settings everywhere to have auto-negotiation on, but we still had problems. For example with Cisco and Tandberg ports set on auto-negotiation on both sides, I’ve seen the most uncommon results:
– Cisco 1000Mbps – Tandberg 1000Mbps = negotiation 1000Mbps / Full
– Cisco 1000Mbps – Tandberg 100Mbps = negotiation 1000Mbps / Full on Cisco + 100Mbps / Full on Tandberg
– Cisco 100Mbps – Tandberg 100Mbps = negotiation 100Mbps / Half on Cisco + 100Mbps / Full on Tandberg

These are just a few of the strange results that we got. Myself as part of the network team I turn my attention to search bugs in IOS, configuration issues, faulty hardware. The Telepresence team was doing their job to search on their systems. Nothing was working.

We turn our attention to TAC engineers. They did try to simulate in a lab environment our problems, but failed. Their system were not having this kind of issue. Internet, search engines and boards could not help as well. I was about to think that we are somewhere in the Bermuda triangle and we are the only one with this kind of problem.

Then the solution came from a Cisco engineer when we least expect it. I quote from his e-mail in which he gave us some suggestion to try:

Are you aware that Tandberg endpoints running newer versions of software 
need to be rebooted before changes to speed settings take effect? 
This can sometimes cause confusion.

We stopped for a second and ask “Did we reload any Tandberg device during troubleshooting sessions?” The answer was “No”. After reload all devices, one by one, everything was working expect a few devices.

We discovered that these Tandberg devices didn’t want to auto-negotiate because of lack of a Cat6 cabling. It seems that all 8 wires need to be there and connected. So, if you have a cable that is patched to transport data and telephony for example to spare some wires, then you may be in trouble.

Why did I add this thing here? For sure it will bring some ironic smile on some faces, but I like to learn from my mistakes or from not paying close attention to some small line in the documentation. OK, if I made you laugh it’s fine, but the reason of this article is different. When I did search Internet for possible solution, I could not find anywhere a line with “reload the damn Tandberg device after you modify Speed / Duplex” settings.


[Cisco Live] SDN controller interview

The original name of this video is “SDN controller DEMO”. I think the “demo” word there is a bit inappropriate used, as actually is more like a Cisco marketing video than demo. Don’t be so surprised, you know how Cisco promote their products.

Just my 2 cents about the SDN/OpenFlow trend that is coming up these days. I don’t mind innovation, I’m glad if and when I can get in contact with new technologies, but what disturb me is that in a lot of presentations that I’ve seen until now, SDN is presented like the magic wand that does everything with point and click.
No knowledge needed, no network understanding, no effort to see where or how the packets travel through network, you just have to point and click, slice the network however you want, plug the toy and you’re ready to go. If this is all true, then someone please explain to me with I’m busting my…head to learn and really understand what’s actually going on in the network. I really hope this kind of presentations implies that this is a new product that needs to be promoted and if they make it sound too complex, nobody will buy it.

Disclaimer: This video is not mine and I don’t claim any rights on it. My thanks go to Jimmy Ray Purse, TechWiseTV, Networking 101 Show, Cisco and last but not least to YouTube for hosting it and let us embed this video.

Juniper, first steps after power-on the device

As you know from my previous posts, I’m trying to find time to gain some Juniper knowledge. During this “quest” I will add here some basic things about how to start working with Juniper devices. For now I know only the basics of Juniper configuration, but I hope that soon you’ll find here some more challenging scenarios.

I have a basic topology that you’ll find below. The scenario is already prepare to have some tasks which suppose integration between Juniper and Cisco environment.

Let’s assume that I did power on the two boxes J1 and J2 and now I’m connected to J1 through a console cable. After the boot sequence I’m left with something like this:

Tue Jun 12 11:46:06 UTC 2012
 
Amnesiac (ttyd0)
 
login:

All platforms running the Junos OS have only the root user configured by default, without any password. Let’s introduce that username and see what’s happening:

login: root
Password:
 
--- JUNOS 9.4R2.9 built 2009-03-25 07:50:02 UTC
root@%

What I have now in front is actually the shell of the FreeBSD OS. JunOS is based on the FreeBSD OS. If you ever interacted with a Linux based system, then you can run here specific linux commands. For example:

root@% ls
.snap           boot            jail            modules         sbin
COPYRIGHT       config          kernel          opt             staging
altconfig       data            libexec         packages        tmp
altroot         dev             mfs             proc            usr
bin             etc             mnt             root            var
root@% 
root@% 
root@% 
root@% ps u
USER   PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root  1153  0.0  0.2  1492   936  v0  Is+  11:46AM   0:00.02 /usr/libexec/getty
root   941  0.0  0.4  2636  2176  d0- S    11:46AM   0:00.14 /usr/sbin/eventd -
root  1264  0.0  0.2  1676  1252  d0  Is   11:50AM   0:00.05 login [pam] (login
root  1265  0.0  0.5  3872  2744  d0  S    11:51AM   0:00.21 -csh (csh)
root  1289  0.0  0.2  1612   996  d0  R+   11:54AM   0:00.01 ps u

OK, you got my point. To get from the FreeBSD shell to JunOS CLI, you need to enter the following:

root@% cli
root>

What you see now is the Operational Mode. In this mode the user can run basic and troubleshooting commands (like traceroute, ping…). You can get a list of commands using the ? (question mark):

root> ?
Possible completions:
  clear                Clear information in the system
  configure            Manipulate software configuration information
  file                 Perform file operations
  help                 Provide help information
  monitor              Show real-time debugging information
  mtrace               Trace multicast path from source to receiver
  op                   Invoke an operation script
  ping                 Ping remote target
  quit                 Exit the management session
  request              Make system-level requests
  restart              Restart software process
  set                  Set CLI properties, date/time, craft interface message
  show                 Show system information
  ssh                  Start secure shell on another host
  start                Start shell
  telnet               Telnet to another host
  test                 Perform diagnostic debugging
  traceroute           Trace route to remote host

If you want to compare the Operational Mode is somehow like Privilege level 1 under Cisco CLI. Still I have the feeling that Operational Mode offer a wider area of commands and more powerful than Cisco CLI Privilege level 1. I may be mistaken.

All platforms running the Junos OS come with a factory-default configuration. All factory-default configurations
allow access using the root account without any password. Nevertheless to activate a configuration you have first to set the password root password.Factory-default configurations can vary from one platform family to another or even between the different models
within the same platform family.
My default configuration looks like:

root> show configuration 
version 9.4R2.9;
system {
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}

I this first post my target is to set the hostname of the Juniper devices. To accomplish this step, I need to go first into configuration mode:

root> configure 
Entering configuration mode
The configuration has been changed but not committed
 
[edit]
root#

and then set the hostname:

root# set system host-name J1 
 
[edit]

If I look at the system prompt, it still shows root#, so it doesn’t quite seems to work. This is because I have to commit to activate the configuration:

root# commit 
[edit]
  'system'
    Missing mandatory statement: 'root-authentication'
error: commit failed: (missing statements)
 
[edit]
root#

Well, this didn’t work as expected. The most important thing that I learned when I started with Juniper is that before I can activate any configuration (commit) I need to set the password for the root user:

root# set system root-authentication plain-text-password              
New password:
Retype new password:
 
[edit]
root#

Let me try to commit one more time, after setting the root password:

root# commit 
commit complete
 
[edit]
root@J1#

You can see that the prompt did change into root@HOSTNAME# (in my case this is root@J1#). If you look again to the system configuration. I will exist the Configuration Mode and have another look at my config file:

root@J1> show configuration 
## Last commit: 2012-06-12 12:38:22 UTC by root
version 9.4R2.9;
system {
    host-name J1;
    root-authentication {
        encrypted-password "$1$DKpYj/Nd$TVFTars5T2.oM3y5eyp520"; ## SECRET-DATA
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}
 
root@J1>

The host-name and root password appears now in the active configuration.

That’s it for today. Until next post, I will add the basic configuration for J2 and the Cisco, so I can go to basic interface configuration and connectivity check.


Selective BGP Dampening and parameters tuning

Some time ago, I wrote about BGP Dampening and how this feature can improve the stability of the network. A lot happened since then and during my experience with different service providers I have seen that BGP dampening can help in the same measure at it can harm your network. An endless discussion can be started on this topic, but this is not what I want to do here.

One thing that I did learn is that fine tuning of any feature can help a lot in some cases making the difference between stable network and a total disaster. In regard to BGP dampening, I have the following scenario. Imagine that you would like to use BGP dampening, but only for some networks, which are proven to be more stable than others. I will base my example on the following scenario:

Task

Router 1 in the above scenario has three networks that are advertised into BGP:
L0 – 1.1.1.0 /24
L1 – 11.11.11.0 /24
L2 – 111.111.111.0 /24
Imagine this are being remote networks and that 11.11.11.0 /24 is very unstable. To simulate an unstable network that triggers BGP, shut / no shut multiple times.

On Router 2, we want to use the BGP Dampening feature, but only for this network. We know already that we can do something like:

conf t
router bgp 200
bgp dampening

This will enable the BGP dampening feature. We can even fine tune some parameters like:

conf t
router bgp 200
bgp dampening 15 750 5000 30

In this way we increase the limit at which a route will be dampened (5000) and decrease the maximum dampening time to 30 minutes. Unfortunately this parameters are applied globally and all routes (stable and unstable) will play by this rules.

Solution

Going back to the idea of this post, use of selective BGP Dampening, we can configure Router 2 like this:

conf t
access-list 11 permit 11.11.11.0 0.0.0.255
!
route-map DAM permit 10
match ip address 11
set dampening 15 750 2000 60
!
route-map DAM deny 1000

What we just did is to match the unstable prefix in an ACL. Use the ACL in a route-map with permit policy. Mandatory set the dampening parameters. They can be the same as original values, but if you don’t set anything here, you will meet the following error when trying to apply the BGP dampening.

%BGP-3-BADROUTEMAP: Bad parameters in the route-map DAM applied for Dampening

At the end we have a deny policy in the same route-map to avoid matching any other prefixes. We can not apply it to BGP:

conf t
router bgp 200
bgp dampening route-map DAM

We want to check that BGP Dampening feature is activated:

R2#sh ip bgp dampening parameters
 dampening 15 750 2000 60 (route-map DAM 10)
  Half-life time      : 15 mins       Decay Time       : 2320 secs
  Max suppress penalty: 12000         Max suppress time: 60 mins
  Suppress penalty    :  2000         Reuse penalty    : 750

By the way, if you check the output immediately after applying the BGP dampening feature, you might see the following error:

% dampening reconfiguration in progress for IPv4 Unicast

Verification

Let us see if there are any flaps on going:

R2#sh ip bgp dampening flap-statistics 
 
R2#

Now we can shut / no shut L1 interface on R1 to create an instability of this network. After doing so couple of times we can see that the BGP dampening is active:

R2#sh ip bgp dampening flap-statistics | b Net
   Network          From            Flaps Duration Reuse    Path
 h 11.11.11.0/24    10.1.12.1       1     00:00:22          100

If we continue to play with shut / no shut, soon we will see that 11.11.11.0 /24 is marked as dampened:

R2#sh ip bgp dampening dampened-paths | b Net
   Network          From             Reuse    Path
*d 11.11.11.0/24    10.1.12.1        00:06:29 100 i

Now I want to prove that the same BGP dampening policies does NOT apply to other networks like 111.111.111.0 /24. I will try to play the same shut / no shut game with L2 on R1. After 5 minutes of this game I can see the following:

R2#sh ip bgp dampening dampened-paths | b Net
   Network          From             Reuse    Path
*d 11.11.11.0/24    10.1.12.1        00:02:09 100 i
 
R2#sh ip bgp dampening flap-statistics | b Net
   Network          From            Flaps Duration Reuse    Path
*d 11.11.11.0/24    10.1.12.1       3     00:07:51 00:01:49 100 
 
R2#sh ip bgp | b Net  
   Network          Next Hop            Metric LocPrf Weight Path
*> 2.2.2.0/24       0.0.0.0                  0         32768 i
*> 3.3.3.0/24       10.1.23.3                0             0 300 i
*d 11.11.11.0/24    10.1.12.1                0             0 100 i
*> 111.111.111.0/24 10.1.12.1                0             0 100 i

You can see that 111.111.111.0 /24 does not appear in any dampening report.

I tried this in multiple scenarios and every time I got the expected result. If you test this and get different results, please let me know in comments and we can discuss.


Fiber Optic basics

I think all network engineers touched, if not used / patched, as least one time a fiber optic patch cord. As a network engineer you might not necessarily need to understand HOW fiber optic is working. It’s there and it’s working. You just need to plug the patch cord and that that’s it.

Anyway, for interested network engineers (or geeks) like me, to understand how fiber optic works might be a fun way to spend 10 minutes of my life. Of course, to have in-deep knowledge of FO, at the level which allow you to design applications for this transport medium, you need to read a little bit more than the video above.

Disclaimer: This video is not mine and I don’t claim any rights on it. My thanks go to Jimmy Ray Purse, TechWiseTV, Networking 101 Show, Cisco and last but not least to YouTube for hosting it and let us embed this video.