So ESXi upgrades can be done in a variety of ways.
You could use the Update manager, use the ISO in IPMI – but then the most fun way (for learning) is how to do an upgrade using the command line – and having fun in the process!
Installs and Fun? I hear you say? Well, YES, if you can break the install midway for fun and pleasure 🙂 Well how is THAT possible? I see you wonder? And what is the POINT of it all? To which I say – What is the point of Life itself? 🙂
So here I am logged in to my test subject for the day – a healthy lab machine running, well, the below
vmware -vl
VMware ESXi 7.0.0 build-15843807
VMware ESXi 7.0 GANow, the point is, we are looking to do an upgrade/update over the wire! So to do that, the bare necessities are, an ESXi host and the hosts able to talk to the world outside!
So to start the upgrade, we need a couple of things – the exact build of ESXi we need and the name of the package at VMware’s depot
For this, I will be using “ESXi-7.0U1b-17168206-standard”
So here goes steps are below
- Start the upgrade
- Poke around in the guts of ESXi 7.0.1 and see what the upgrade looks like from ESXi’s perspective
- If possible, recreate a simple install error and find out what that looks like
This will help us understand the upgrade process when a real world upgrade goes belly up!
Part 1 – Starting the upgrade
So to start the upgrade fire up your trusty command line/ssh tool and log in to the ESXi host. You can use Putty or xTerm etc – I am just using the Terminal on my Mac
So once we log in, let’s check the system time and the current version the node is on
vmware -lv gives you the version and the date command shows the time
vmware -vl
VMware ESXi 7.0.0 build-15843807 <-------
VMware ESXi 7.0 GA
date
Sun Jan 31 14:33:38 UTC 2021 <--------Ensure the host has access to the internet
ping google.com
PING google.com (142.250.67.78): 56 data bytes
64 bytes from 142.250.67.78: icmp_seq=0 ttl=114 time=27.470 ms
64 bytes from 142.250.67.78: icmp_seq=1 ttl=114 time=27.467 ms
64 bytes from 142.250.67.78: icmp_seq=2 ttl=114 time=27.510 msNow that is done, check the free space on the ESXi localsystem to ensure you have free space to install
vdf -h
Tardisk Space Used
vmx.v00 116M 116M
vim.v00 139M 139M
tpm.v00 24K 22K
sb.v00 171M 171M
s.v00 68M 68M
i40en.v00 716K 713K
ixgben.v00 612K 608K
nmlx5cor.v00 1M 1M
nfs_vaai.v00 156K 155K
bnxtnet.v00 688K 685K
bnxtroce.v00 324K 323K
brcmfcoe.v00 2M 2M
brcmnvme.v00 124K 123K
elxiscsi.v00 548K 546K
elxnet.v00 636K 635K
i40iwn.v00 484K 480K
iavmd.v00 196K 195K
igbn.v00 320K 319K
iser.v00 260K 259K
lpfc.v00 2M 2M
lpnic.v00 636K 635K
lsi_msgp.v01 552K 549K
lsi_msgp.v02 512K 511K
mtip32xx.v00 256K 252K
ne1000.v00 636K 633K
nenic.v00 264K 261K
nfnic.v00 576K 573K
nhpsa.v00 612K 611K
nmlx4_co.v00 784K 781K
nmlx4_en.v00 732K 730K
nmlx4_rd.v00 340K 338K
nmlx5_rd.v00 292K 288K
ntg3.v00 116K 115K
nvme_pci.v00 120K 117K
nvmerdma.v00 168K 164K
nvmxnet3.v00 196K 193K
nvmxnet3.v01 172K 168K
pvscsi.v00 124K 121K
qcnic.v00 300K 297K
qedentv.v00 3M 3M
qedrntv.v00 2M 2M
qfle3.v00 2M 2M
qfle3f.v00 1M 1M
qfle3i.v00 368K 367K
qflge.v00 500K 498K
rste.v00 828K 825K
sfvmk.v00 648K 647K
smartpqi.v00 364K 362K
vmkata.v00 204K 202K
vmkfcoe.v00 1008K 1006K
vmkusb.v00 1M 1M
vmw_ahci.v00 236K 234K
crx.v00 12M 12M
elx_esx_.v00 2M 2M
btldr.v00 1M 1M
esx_dvfi.v00 488K 484K
esx_ui.v00 14M 14M
esxupdt.v00 1M 1M
tpmesxup.v00 12K 11K
weaselin.v00 2M 2M
loadesx.v00 56K 53K
lsuv2_hp.v00 72K 70K
lsuv2_in.v00 28K 26K
lsuv2_ls.v00 1M 1M
lsuv2_nv.v00 16K 13K
lsuv2_oe.v00 16K 13K
lsuv2_oe.v01 16K 13K
lsuv2_oe.v02 16K 13K
lsuv2_sm.v00 56K 54K
native_m.v00 2M 2M
qlnative.v00 2M 2M
vdfs.v00 12M 12M
vmware_e.v00 188K 187K
vsan.v00 46M 46M
vsanheal.v00 7M 7M
vsanmgmt.v00 21M 21M
xorg.v00 3M 3M
imgdb.tgz 1M 1M
state.tgz 68K 64K
loadesx-2316334 76K 73K
esxupdt-2316334 1M 1M
tpmesxup-2316334 12K 11K
weaselin-2316334 2M 2M
-----
Ramdisk Size Used Available Use% Mounted on
root 32M 12M 19M 38% --
etc 28M 560K 27M 1% --
opt 32M 0B 32M 0% --
var 48M 544K 47M 1% --
tmp 256M 6M 249M 2% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
crx 1024M 0B 1024M 0% --
configstore 32M 44K 31M 0% --
configstorebkp 32M 44K 31M 0% --
hostdstats 1053M 2M 1050M 0% --
esx-update-2316334 14M 1M 12M 11% --
stagebootbank 4094M 134M 3959M 3% --
vibdownload 126M 1M 124M 1% --
and
df -h
Filesystem Size Used Available Use% Mounted on
NFS 12.0T 7.9G 12.0T 0% /vmfs/volumes/vmContainer1
VMFS-5 19.8G 1.0G 18.7G 5% /vmfs/volumes/NTNX-local-ds-18SM6F100193-D
VFFS 31.5G 4.0G 27.5G 13% /vmfs/volumes/OSDATA-601589af-b059815f-b96c-ac1f6b626d76
vfat 4.0G 162.8M 3.8G 4% /vmfs/volumes/BOOTBANK1
vfat 4.0G 162.8M 3.8G 4% /vmfs/volumes/BOOTBANK2So start the install, the following commands can be used
Open the firewall and allow http access
esxcli network firewall ruleset set -e true -r httpClientStart the actual install – note \ denotes the command is one single line
esxcli software profile update -d \ https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw- \ depot-index.xml -p ESXi-7.0U1b-17168206-standardSo at this point, the upgrade has started. But the screen will be blank
Part 2 – Monitoring the upgrade
So ESXi upgrades using the above method has the following workflow
- File is downloaded – if the filename is correct in the VMware depot
- Downloaded files are extracted in the local partition of the ESXi host
- Files are then installed
- Logs are written in the background denoting the progress and/or failure if any
So to monitor progress, open a new ssh session, hop on to the host and scan the esxupdate logs
In my case, the following is seen:
Image profile modifications are done, files are copied, extracted and non needed vibs are removed. Then, files are stored locally for the actual upgrade
2021-01-31T14:23:08Z esxupdate: 2316334: root: INFO: Command = profile.update
2021-01-31T14:23:08Z esxupdate: 2316334: root: INFO: Options = {'level': None, 'dryrun': False, 'depot': ['https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml'], 'profile': 'ESXi-7.0U1b-17168206-standard', 'noliveinstall': False, 'viburl': None, 'addon': None, 'nosigcheck': False, 'force': False, 'nohwwarning': False, 'softwarespec': None, 'nomaintmode': False, 'nameid': None, 'oktoremove': False, 'pending': None, 'downgrade': None, 'proxy': None, 'updateonly': False, 'rebooting': False, 'baseimageversion': None}
2021-01-31T14:23:09Z esxupdate: 2316334: HostImage: INFO: Installers initiated are {'live': <vmware.esximage.Installer.LiveImageInstaller.LiveImageInstaller object at 0xf0671a1390>, 'locker': <vmware.esximage.Installer.LockerInstaller.LockerInstaller object at 0xf0672a4358>, 'boot': <vmware.esximage.Installer.BootBankInstaller.BootBankInstaller object at 0xf067293ef0>}
2021-01-31T14:23:09Z esxupdate: 2316334: DepotCollection: DEBUG: Downloading depot index.xml from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
2021-01-31T14:23:09Z esxupdate: 2316334: downloader: INFO: Opening https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml for download
2021-01-31T14:23:11Z esxupdate: 2316334: DepotCollection: DEBUG: Downloading vendor index.xml from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vmw-esx-index.xml
2021-01-31T14:23:11Z esxupdate: 2316334: downloader: INFO: Opening https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vmw-esx-index.xml for download
2021-01-31T14:23:12Z esxupdate: 2316334: DepotCollection: DEBUG: Downloading metadata.zip from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vmw-ESXi-5.0.0-metadata.zip
2021-01-31T14:23:12Z esxupdate: 2316334: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vmw-ESXi-5.0.0-metadata.zip to /tmp/tmpte68r34q
2021-01-31T14:23:15Z esxupdate: 2316334: Metadata.pyc: INFO: Reading metadata zip /tmp/tmpte68r34q
2021-01-31T14:23:15Z esxupdate: 2316334: Metadata.pyc: DEBUG: Processing file vendor-index.xml
2021-01-31T14:23:15Z esxupdate: 2316334: Metadata.pyc: DEBUG: Processing vib xml esx-base-2102922490.xml
2021-01-31T14:23:15Z esxupdate: 2316334: Metadata.pyc: DEBUG: Processing vib xml net-e1000e--1572903360.xml
2021-01-31T14:26:17Z esxupdate: 2316334: imageprofile: INFO: Adding VIB VMware_locker_tools-light_11.1.1.16303738-16850804 to ImageProfile ESXi-7.0U1b-17168206-standard
2021-01-31T14:26:17Z esxupdate: 2316334: imageprofile: DEBUG: VIBs {'VMware_locker_tools-light_11.0.5.15389592-15843807'} are replaced by VIB VMware_locker_tools-light_11.1.1.16
303738-16850804, removing them from ImageProfile ESXi-7.0U1b-17168206-standard
2021-01-31T14:26:17Z esxupdate: 2316334: imageprofile: DEBUG: VIB VMW_bootbank_sfvmk_2.0.0.1004-3vmw.700.1.0.15843807 is being removed from ImageProfile ESXi-7.0U1b-17168206-sta
ndard
2021-01-31T14:33:13Z esxupdate: 2316334: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsi-msgpt35/VMW_bootbank_lsi-msgp
t35_13.00.13.00-2vmw.701.0.0.16850804.vib to /tmp/vibdownload/VMW_bootbank_lsi-msgpt35_13.00.13.00-2vmw.701.0.0.16850804.vib
2021-01-31T14:33:13Z esxupdate: 2316334: downloader: INFO: Proxies configured: None
2021-01-31T14:33:15Z esxupdate: 2316334: esximage.Vib: DEBUG: Verifying VIB VMW_bootbank_lsi-msgpt35_13.00.13.00-2vmw.701.0.0.16850804 signature #1
2021-01-31T14:33:15Z esxupdate: 2316334: esximage.Vib: DEBUG: Total of 164451 bytes were written.
2021-01-31T14:33:15Z esxupdate: 2316334: HostImage: INFO: Attempting to download VIB loadesx
2021-01-31T14:33:15Z esxupdate: 2316334: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/loadesx/VMware_bootbank_loadesx_7
.0.1-0.15.17168206.vib to /tmp/vibdownload/VMware_bootbank_loadesx_7.0.1-0.15.17168206.vib
Part 3 – Analysing failures
This process would have continued until there was a network event or any other catastrophic event that would cause installs to fail. Here, I have just used dd to create a large file in the /scratch partition
dd if=/dev/zero of=large_file bs=4k \ iflag=fullblock,count_bytes count=40GSo immediately my scratch partition gets filled up and the install fails
2021-01-31T14:35:33Z esxupdate: 2316334: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/loadesx-2316334
2021-01-31T14:35:33Z esxupdate: 2316334: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/esxupdt-2316334
2021-01-31T14:35:33Z esxupdate: 2316334: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/tpmesxup-2316334
2021-01-31T14:35:33Z esxupdate: 2316334: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2316334
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: Traceback (most recent call last):
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-17168206/bora/build/esx/release/vmvisor/esxupdate/lib64/python3.5/site-packages/vmware/esxi
mage/Installer/BootBankInstaller.py", line 1175, in _SysStorageBootPartAction
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/systemStorag
e/esxboot.py", line 222, in installBootloader
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/systemStorag
e/esxboot.py", line 94, in installBiosBootloader
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/systemStorag
e/vfat.py", line 808, in mcopy
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/systemStorag
e/vfat.py", line 780, in _mtools
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 383,
in run
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676,
in __init__
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228,
in _execute_child
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR: OSError: [Errno 28] No space left on device <----------------
2021-01-31T14:35:33Z esxupdate: 2316334: root: ERROR:
This is what comes up in the console where the update command was running
esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0U1b-17168206-standard
[InstallationError]
Failed to update bootloader: [Errno 28] No space left on device
cause = [Errno 28] No space left on device
vibs = ['VMware_bootbank_esx-base_7.0.1-0.15.17168206']
Please refer to the log file for more details.Note the error actually says please refer to the log file 🙂 Which is kinda funny, as if a normal Joe or Jane knew where log files were and which ones to check, you wouldn’t be reading this far into this blog 🙂 🙂 🙂
Anyways, so now that we know what upgrades look like – (upgrades are simple pleasant things to work with actually!) here’s wishing everyone here learnt something!
Stay tuned for more!
