ESXi VM – The CPU has been disabled by the guest operating system

For some weeks now, a couple of my virtual machines on ESXi would stop working out of nowhere. They were completely unresponsive (including via the ESXi VM Console). Nothing would help, except a shutdown / start of the VM. Just to find out later that, randomly, the VM would become unresponsive again.

The only human readable information about these failures was in the ESXi host Events and was saying something like this (among other things):

 The CPU has been disabled by the guest operating system

One other thing which I should mention is that all my VM encountering this issue where Linux based, mainly Ubuntu 20.04 as OS distribution.

Not much to work with, but I gave it a try and searching for the error did point me to this VMware KB: https://kb.vmware.com/s/article/2000542

The KB is clearly accurate, just that it didn’t help me at all to resolve my problem. The troubleshooting process explain in the KB lead me to a dead end.

Other web resources (for the above error) pointed to articles which explained a procedure for VMware Workstation / Player. Not my case, since I’m using ESXi.

More research done, which took a while – that’s why I’m writing this article, hopefully others with this problem will find it easier – pointed to a BUG. Seems this BUG is a particular case between my VM Linux kernel and the version of the ESXi I’m using currently.

I’ve arrived to this VMware KB https://kb.vmware.com/s/article/2151480 which was a game changer. In my case this KB was hard to find, because the title – Linux VM fails with the error “kernel BUG at drivers/net/vmxnet3/vmxnet3_drv.c:1413!” (2151480) – is completely different than the error I was seeing and which I used searching the web.

Skipping the long output at the beginning of the KB, I saw something interesting in lower part of the page:

This issue occurs due to a bug in VMXNET3 vNIC backend which is part of the vmkernel. This issue occurs if the following conditions are met:

    Linux VM is running kernel >= 4.8
    HW version of VM is >=13
    ESXi version is 6.5

All the above fits my scenario, VMXNET3 as vNIC, Kernel 5.4, VM HW version 13 and ESXi 6.5

Like in most of BUG cases, the obvious solution is upgrade. Same here:

This issue is resolved in VMware ESXi 6.5 U1

Just that I cannot upgrade now for various reasons.

So, I’ve decide to look into the workarounds.

Second workaround on the page seems to be more simple and I don’t even have to restart the VM:

ethtool -G ethX rx-mini 0

Of course replace the ethX with your interface name.

Worked like a charm without any visible side-effects.

The other workaround is also doable, but I didn’t want to modify the .vmx file

Power off the virtual machine
         
Edit the vmx file and add the below parameter:
vmxnet3.rev.30 = FALSE
         
Power on the virtual machine

Now I’m just curious if I would encounter the same issues using another vNIC adapter type, like E1000 or E1000E instead of VMXNET3. Maybe I’ll give it a try…

Ubuntu image for EVE-NG – Python for network engineers

Lately I’ve started working more and more with EVE-NG to test various network scenarios, automation and in general to try and learn something everyday.

If you’re familiar with EVE-NG, you know where to find various Linux images which you can download and install . Very helpful indeed, however all of them are coming without any pre-installed tools which I need for network oriented tests. I need Python, IPerf, Ansible, various Python libraries for network automation, etc.
Basically every time when I setup a new lab in EVE-NG, I need to make sure that the Linux image has a connection to Internet to download all these tools. Doable, but too much time consuming.

Lately EVE-NG has the Pro version, where you have Docker images which support some of the tools for a network engineer needs to test automation. If you already have EVE-NG Pro, then maybe this is a bit redundant. However if you’re still using the Community version, it may sounds interesting.

I’ve developed the Ubuntu (18.04) image using the same tools that you can find in my Docker image (Ubuntu 16:04 Pfne):
* If you’re not sure what I’m talking about, please read my previous post.

  • Openssl
  • Net-tools (ifconfig..)
  • IPutils (ping, arping, traceroute…)
  • IProute
  • IPerf
  • TCPDump
  • NMAP
  • Python 2
  • Python 3
  • Paramiko (python ssh support)
  • Netmiko (python ssh support)
  • Ansible (automation)
  • Pyntc
  • NAPALM

The image is hosted on my Firstdigest Project at Sourceforge.
If you are in a hurry, download directly using this link: Ubuntu 18.04 Pfne for EVE-NG.

For convenience here are the steps, but if you run into trouble be sure to check the EVE-NG Documentation.

  • Download the image
  • Using favorite SFTP Client (WinSCP, FileZilla) connect to your EVE-NG and upload the image to the location: /opt/unetlab/addons/qemu/
  • Connect via SSH to your EVE-NG machine and go to location:
cd /opt/unetlab/addons/qemu/
  • Unzip your uploaded image file.
tar xzvf linux-ubuntu-server-18.04-pfne.tar.gz
  • Remove the archived image file (be sure to have a copy somewhere to avoid you have to download it again)
rm -f linux-ubuntu-server-18.04-pfne.tar.gz
  • Fix permissions
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions

The image comes with the following predefined username and password (security was not the point here):

User: root
Password: root
User: pfne
Password: pfne

With this image you have everything ready for your tests. You want to test QoS? Just design a network and two (client / server pair) machine using this image and push some packets with IPerf. Or maybe you want to test some automation. Here you have it, just start playing with.

Btw, I assume you have the EVE-NG installed. If not and you’re into learning topics, I’ll advise you to install this great application. You can start with Community version which is free (and honestly has enough features for most of the self-teaching engineers out there) and if you feel like go with the Pro version.

Let me know if you find it useful. In case of problems, please comment and I’ll try to help in my spare time.

IPsec VPN Mikrotik to Linux

After writing the Mikrotik IPsec VPN article and I got some questions about how Mikrotik will work with a Linux device to build an IPsec VPN. I did notice that the questions were more oriented for a copy / paste solution, so I’ll provide one that it’s working. If you need more details about why the solution is like it this, please let me know.
Also don’t forget to customize the solution as you need.

I’ll start with the same topology like in the last post, just that the right side now it’s a Linux device.

Mikrotik-IPsec-VPN-Linux

Please consider the minimum ports needed to be open on your firewall from my earlier article. Just don’t forget to open these ports also on the Linux device.

First let’s configure the Mikrotik.

The IPsec Proposal

CLI

ip ipsec proposal add name=MyProposal auth-algorithms=sha1 enc-algorithms=aes-256-cbc pfs-group=none

GUI

IP > IPsec > Proposals

Name: MyProposal
Auth. Algorithm: sha1
Encr. Algorithm: aes-256 cbc
PFS Group: none

The IPsec Policy

CLI

ip ipsec policy add src-address=192.168.0.0/24 dst-address=172.30.0.0/24 protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=11.11.11.11 sa-dst-address=22.22.22.22 proposal=MyProposal

GUI

IP > IPsec > Policies

SRC ADDR: 192.168.0.0/24
DST ADDR: 172.30.0.0/24
Protocol: all
Action: Encrypt
Level: require
IPsec protocols: esp
Tunnel: check
SA SRC: 11.11.11.11
SA DST: 22.22.22.22
Proposal: MyProposal

The IPsec Peer

CLI

ip ipsec peer add address=22.22.22.22 port=500 auth-method=pre-shared-key secret=my_preshared_key exchange-mode=main send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=3des,aes-128 dh-group=modp1024 generate-policy=no

GUI

IP > IPsec > Peers

Address: 22.22.22.22
Port: 500
Auth. Method: pre shared key
Passive: not checked
Secret: my_preshared_key
Policy Template Group: default
Exchange mode: main
Send Initial Contact: checked
NAT Traversal: checked
My ID: Auto - empty
Proposal Check: obey
Hash Algorithm: sha1
Encryptions Algorithm: 3des aes-128
DH Group: modp1024
Generate policy: no

Now, the Linux part. I’m using Ubuntu, but I’m not going to advocate here for one flavour or another. So, just use any device with Linux or you try solutions such as Amazon AWS. Install Openswan (compile it from source or install via your Linux flavour package system).

The main file for Openswan is ipsec.conf. For me this file is in /etc, but I assume it can reside in another location.

For the above example, the ipsec.conf file looks like this:

version 2.0

# basic configuration
config setup
        nat_traversal=yes
        oe=off
        protostack=netkey
        force_keepalive=yes
        keep_alive=60
        #nhelpers=0

# Add connections here
conn mikrotik-to-linux
        authby=secret
        auto=start
        type=tunnel
        left=22.22.22.22
        leftid=22.22.22.22
        leftsourceip=172.30.0.1
        leftsubnet=172.30.0.0/24
        right=11.11.11.11
        rightsubnet=192.168.0.0/24
        rightid=11.11.11.11
        pfs=no
        forceencaps=yes
        ike=aes256-sha1;modp1024
        phase2=esp
        phase2alg=aes256-sha1

You need to associate the keyword “left” with “local” and “right” with “remote” and it will be easier to read the configuration above.

Also in the /etc location I have another file called ipsec.secrets which has the pre-shared secret key:

22.22.22.22 11.11.11.11 : PSK "my_preshared_key"

This is the minimal configuration that I need to apply to have the IPsec VPN up and running. I’m sure that it can be fine tuned to add more security or features, but that is not in scope of this post.

As always if you have problems please let me know in Comments.

How to emulate ASA in Ubuntu 9.10 and GNS3

Cisco ASA

Brainbump.net has an excellent and very complete how to emulate ASA using just the following components:

  • Ubuntu 9.10 – 32 bit Edition
  • GNS3 v0.7 RC1 tgz
  • Dynamips 0.2.8-RC2 binary for Linux x86 platforms
  • Qemu-0.11.0 tar.gz
  • Qemu-0.11.0 Patch
  • ASA Binary Version 8.0(2) – (asa802-k8.bin)

How-to is divided in 3 video tutorial parts for easy understanding and start with the most basic installation on GNS3 under Ubuntu 9.10 and continue with the actually configuration on the emulation.
If you are interested in security or you just want to test ASA and don’t have access to real hardware you definetely will want to try Brainbump.net tutorial.

READ THE FULL TUTORIAL on Brainbump.net


Cisco: Use TTCP to test together with TTCPW or JPerf

In one my previous posts, I shown you how to test a connection using a hidden Cisco IOS tool called TTCP.  A few days ago I run into an issue. I had to test a TCP connection to a remote Cisco router, but I had not other router on which to initiate the TTCP connection. As explained in Testing TCP Connection post, to use TTCP you need 2 Cisco routers.

Now, I found 2 new ways to do the testing without the need of having 2 routers, but just one. Maybe you already know this methods, but for those who don’t please keep on reading.

First, there is a Windows tool called TTCPW (download here) (actually you can download also the code, and I think it’s possible to compile and run it under Linux as well). This TTCW tool have the same option like Cisco TTCP and can work together without any issue.
On Cisco router, issue the ttcp command, and keep the regular parameters (we are not interested for now in fine tuning). Below I set the Cisco router to be the receiver:

Cisco TTCP
Cisco TTCP with default settings

On PC side, you download TTCPW and use the same settings. Basically to transmit you only need to input ttcpw.exe -t or -r “ip.address” and that’s it  Of course you can tune the settings to meet your needs. Just type ttcpw.exe to see all the settings.

TTCPW
TTCPW help

The second tool that you can use with Cisco TTCP is IPerf (text mode) or JPerf (Java graphical mode). Just fill in the IP address and the port (5001 if default) and you’re ready to go:

Jperf with TTCP
Jperf with TTCP

Of course there are some limitations on JPerf to TTCP compared to JPerf to JPerf testing. One of then is that you cannot use parallel streams, if you want to stress the connection. To overcome this limitation, I do the following.
Open 2 or 3 connection to the Cisco router where TTCP will run. Start one each connection one TTCP daemon with different ports (e.g. assuming 3 connection than ports 5001, 5002 and 5003). Then on the client start 3 JPerf (Iperf) with the same IP address but different ports (you can take the one below). In this way you can stress the connection a little bit.