My ESXi hosting the VCSA crashed for whatever reason and after reboot the VCSA was displaying a “503 Service Unavailable” error.
What I was seeing actually was a blabbering long line:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fa69401a900] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
The ESXi hosting my VCSA is not the fastest in the world, so I’ve waited a while, but the error was still there. Searching the Interne returned a lot of possible root causes for this errors, ranging from simple to complex one (like duplicate database table entry where you have to manually touch the postgresql instance).
I didn’t want to jump directly into touching things like the database, so I started with something more simple.
Below is what worked for me, maybe you’ll find it useful and can try before going into advanced troubleshooting.
I’ve connected to the VCSA CLI using the root credentials.
[email protected]'s password: Connected to service * List APIs: "help api list" * List Plugins: "help pi list" * Launch BASH: "shell" Command>
Launched BASH by typing shell at the Command> prompt.
Now I have a Linux like CLI terminal.
Next step I’ve ran
service-control --status --all
which resulted in the following output:
root@vcsa [ ~ ]# service-control --status --all Running: lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-eam vmware-rhttpproxy vmware-sca vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vsphere-client Stopped: applmgmt pschealth vmcam vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-psc-client vmware-rbd-watchdog vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsphere-ui
I’m not a certified expert in VCSA, but this doesn’t look good. Too many stopped services.
So, I just give it a try to see if I can start them by running
service-control --start --all
The next output is a long one, but basically it will check what services are up and start the ones which are stopped
root@vcsa [ ~ ]# service-control --start --all Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True 2020-04-06T17:31:57.180Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd'] 2020-04-06T17:31:57.185Z Done running command 2020-04-06T17:31:57.188Z Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.188Z Running command: ['/sbin/service', u'lwsmd', 'status'] 2020-04-06T17:31:57.213Z Done running command Successfully started service lwsmd 2020-04-06T17:31:57.217Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd'] 2020-04-06T17:31:57.589Z Done running command 2020-04-06T17:31:57.593Z Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.593Z Running command: ['/sbin/service', u'vmafdd', 'status'] 2020-04-06T17:31:57.617Z Done running command Successfully started service vmafdd 2020-04-06T17:31:57.621Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird'] 2020-04-06T17:31:57.627Z Done running command 2020-04-06T17:31:57.630Z Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.630Z Running command: ['/sbin/service', u'vmdird', 'status'] 2020-04-06T17:31:57.654Z Done running command Successfully started service vmdird 2020-04-06T17:31:57.657Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad'] 2020-04-06T17:31:57.663Z Done running command 2020-04-06T17:31:57.667Z Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.667Z Running command: ['/sbin/service', u'vmcad', 'status'] 2020-04-06T17:31:57.690Z Done running command Successfully started service vmcad 2020-04-06T17:31:57.694Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd'] 2020-04-06T17:31:57.700Z Done running command 2020-04-06T17:31:57.703Z Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.703Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status'] 2020-04-06T17:31:57.727Z Done running command Successfully started service vmware-sts-idmd 2020-04-06T17:31:57.730Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd'] 2020-04-06T17:31:57.736Z Done running command 2020-04-06T17:31:57.739Z Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.740Z Running command: ['/sbin/service', u'vmware-stsd', 'status'] 2020-04-06T17:31:57.763Z Done running command Successfully started service vmware-stsd 2020-04-06T17:31:57.767Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd'] 2020-04-06T17:31:57.773Z Done running command 2020-04-06T17:31:57.777Z Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.777Z Running command: ['/sbin/service', u'vmdnsd', 'status'] 2020-04-06T17:31:57.801Z Done running command Successfully started service vmdnsd 2020-04-06T17:31:57.805Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client'] 2020-04-06T17:31:57.812Z Done running command 2020-04-06T17:31:57.815Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings. 2020-04-06T17:31:57.815Z Running command: ['/sbin/service', u'vmware-psc-client', 'status'] 2020-04-06T17:31:57.839Z Done running command 2020-04-06T17:31:57.843Z Running command: ['/usr/bin/systemctl', 'daemon-reload'] 2020-04-06T17:31:57.927Z Done running command 2020-04-06T17:31:57.927Z Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-psc-client.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true'] 2020-04-06T17:31:57.943Z Done running command Successfully started service vmware-psc-client Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start statsmonitor services. Error: Operation timed out
The last line above is not too encouraging, “failed” keywords is not something to I wanted to see in the output. I was thinking my attempt didn’t work.
However checking the service status again, I’ve seen the following:
root@vcsa [ ~ ]# service-control --status --all Running: applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui Stopped: vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-statsmonitor vmware-vcha
This was for sure better than before.
I gave it a try by opening the https://vcsa.local.domain and there it was, the webpag working fine.
I’m not sure exactly why the restart of the VCSA resulted in some services not to start properly, but seems that a kick will do the job.