VCSA, 503 Service Unavailable – possible fix

My ESXi hosting the VCSA crashed for whatever reason and after reboot the VCSA was displaying a “503 Service Unavailable” error.

What I was seeing actually was a blabbering long line:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fa69401a900] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

The ESXi hosting my VCSA is not the fastest in the world, so I’ve waited a while, but the error was still there. Searching the Interne returned a lot of possible root causes for this errors, ranging from simple to complex one (like duplicate database table entry where you have to manually touch the postgresql instance).

I didn’t want to jump directly into touching things like the database, so I started with something more simple.

Below is what worked for me, maybe you’ll find it useful and can try before going into advanced troubleshooting.

I’ve connected to the VCSA CLI using the root credentials.

[email protected]'s password:
Connected to service

* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"

Command>

Launched BASH by typing shell at the Command> prompt.

Now I have a Linux like CLI terminal.

Next step I’ve ran

service-control --status --all

which resulted in the following output:

root@vcsa [ ~ ]# service-control --status --all
Running:
 lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-eam vmware-rhttpproxy vmware-sca vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vsphere-client
Stopped:
 applmgmt pschealth vmcam vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-psc-client vmware-rbd-watchdog vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsphere-ui

I’m not a certified expert in VCSA, but this doesn’t look good. Too many stopped services.

So, I just give it a try to see if I can start them by running

service-control --start --all

The next output is a long one, but basically it will check what services are up and start the ones which are stopped

root@vcsa [ ~ ]# service-control --start --all
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2020-04-06T17:31:57.180Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
2020-04-06T17:31:57.185Z   Done running command
2020-04-06T17:31:57.188Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.188Z   Running command: ['/sbin/service', u'lwsmd', 'status']
2020-04-06T17:31:57.213Z   Done running command
Successfully started service lwsmd
2020-04-06T17:31:57.217Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
2020-04-06T17:31:57.589Z   Done running command
2020-04-06T17:31:57.593Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.593Z   Running command: ['/sbin/service', u'vmafdd', 'status']
2020-04-06T17:31:57.617Z   Done running command
Successfully started service vmafdd
2020-04-06T17:31:57.621Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
2020-04-06T17:31:57.627Z   Done running command
2020-04-06T17:31:57.630Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.630Z   Running command: ['/sbin/service', u'vmdird', 'status']
2020-04-06T17:31:57.654Z   Done running command
Successfully started service vmdird
2020-04-06T17:31:57.657Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
2020-04-06T17:31:57.663Z   Done running command
2020-04-06T17:31:57.667Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.667Z   Running command: ['/sbin/service', u'vmcad', 'status']
2020-04-06T17:31:57.690Z   Done running command
Successfully started service vmcad
2020-04-06T17:31:57.694Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
2020-04-06T17:31:57.700Z   Done running command
2020-04-06T17:31:57.703Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.703Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2020-04-06T17:31:57.727Z   Done running command
Successfully started service vmware-sts-idmd
2020-04-06T17:31:57.730Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
2020-04-06T17:31:57.736Z   Done running command
2020-04-06T17:31:57.739Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.740Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']
2020-04-06T17:31:57.763Z   Done running command
Successfully started service vmware-stsd
2020-04-06T17:31:57.767Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
2020-04-06T17:31:57.773Z   Done running command
2020-04-06T17:31:57.777Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.777Z   Running command: ['/sbin/service', u'vmdnsd', 'status']
2020-04-06T17:31:57.801Z   Done running command
Successfully started service vmdnsd
2020-04-06T17:31:57.805Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
2020-04-06T17:31:57.812Z   Done running command
2020-04-06T17:31:57.815Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2020-04-06T17:31:57.815Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2020-04-06T17:31:57.839Z   Done running command
2020-04-06T17:31:57.843Z   Running command: ['/usr/bin/systemctl', 'daemon-reload']
2020-04-06T17:31:57.927Z   Done running command
2020-04-06T17:31:57.927Z   Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-psc-client.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true']
2020-04-06T17:31:57.943Z   Done running command
Successfully started service vmware-psc-client
Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start statsmonitor services. Error: Operation timed out

The last line above is not too encouraging, “failed” keywords is not something to I wanted to see in the output. I was thinking my attempt didn’t work.

However checking the service status again, I’ve seen the following:

root@vcsa [ ~ ]# service-control --status --all
Running:
 applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha

This was for sure better than before.

I gave it a try by opening the https://vcsa.local.domain and there it was, the webpag working fine.

I’m not sure exactly why the restart of the VCSA resulted in some services not to start properly, but seems that a kick will do the job.