Home > Blogs > VMware vSphere Blog > Tag Archives: ESX

Tag Archives: ESX

Quickest Way to Patch an ESX/ESXi Using the Command-line

By William Lam, Sr. Technical Marketing Engineer

As you know, when it comes to automating patch management for your vSphere infrastructure, we highly recommend leveraging our vSphere Update Manager (VUM) which is part of the vSphere vCenter Suite to help simplify the update process. Though not all environments have the luxury of running vCenter Server to manage their ESX(i) hosts. An example of this could be 1-2 hosts running at a ROBO (remote office/branch office) site or single test/dev host in a home or office lab where VUM is not available.

However, it is still possible to patch/upgrade your ESX(i) host using the command-line without the need of VUM, but you will have to manually identify the patch dependencies and ensure host compliance.

Depending on the version of ESX or ESXi you are running, you may have several options that could include local and/or remote command-line utilities that are available in following four forms:

  • ESX Service Console
    • esxupdate – Local utility found on classic ESX hosts to manage/install patches
  • ESXi Shell
    • ESXCLI – Local utility found on ESXi 5.0 hosts that can be used manage/install patches
  • vCLI (Windows/Linux or use vMA)
    • vihostupdate35 – Remote utility to manage/install patches for ESXi 3.5
    • vihostupdate – Remote utility to manage/install patches for ESX(i) 4.0 & 4.1
    • ESXCLI – Remote utility to manage/install patches for ESXi 5.0 (patch capability introduced in vSphere 5 for ESXi 5.0 hosts only)
  • PowerCLI(Windows)
    • InstallVMHostPatch – Remote utility using PowerCLI to manage/install patches for ESX(i) 4.0 and 4.1

Note: If you are using vSphere Hypervisor (Free ESXi), you will not be able to leverage any of the the remote CLI’s but you can still use the local CLI.

Here is a table summarizing all available command-line options based on the version of ESX(i) you are running:

Hypervisor Version Local Command vCLI Remote Command PowerCLI Remote Command
ESX 3.5 esxupdate −−bundle=<zip> update N/A N/A
ESXi 3.5 N/A vihostupdate35
−−bundle=<zip> −−install
N/A
ESX 4.0 esxupdate −−bundle=<zip> update vihostupdate <<bundle=<zip> −−install Install-VMHostPatch
ESXi 4.0 N/A vihostupdate −−bundle=<zip> −−install Install-VMHostPatch
ESX 4.1 esxupdate −−bundle=<zip> update vihostupdate −−bundle=<zip> −−install Install-VMHostPatch
ESXi 4.1 N/A vihostupdate −−bundle=<zip> −−install Install-VMHostPatch
ESXi 5.0 esxcli software vib update −−depot=/vmfs/volumes/[datastore]/<zip> esxcli software vib update −−depot=/vmfs/volumes/[datastore]/<zip> Install-VMHostPatch
Or Get-ESXCLI with the local command referenced in this table.

Note: When you download patches from VMware, there is an associated VMware KB article and it provides a link to the patch management documentation. You should always refer to that for more details and information for different methods of applying a patch.

Here is an example of using esxupdate on a classic ESX host. The patch bundle needs to be uploaded to ESX host using scp or winSCP and then specifying the full path on the command-line:

$ esxupdate −−bundle=ESX400-200907001.zip update

Here is an example of using the remote vihostupdate utility for an ESXi host, you will need to specify the ESXi host using the −−server parameter and −−username/−−password for remote authenication. You may choose to leave off −−password and you will be prompted to enter your credentials. The patch bundle does not need to be uploaded to ESXi host, it can reside on the system that is running the vihostupdate command. During the execution, the patch bundle will automatically be transfered to the host:

$ vihostupdate −−server [ESXI-FQDN] −−username [USERNAME] −−bundle=ESXi410-201011001.zip −−install

Here is an example of using the local esxcli utility for an ESXi 5.0 host. The patch bundle needs to be uploaded to ESXi host using scp or winSCP and then specifying the full path on the command-line:

$ esxcli software vib update −−depot=/vmfs/volumes/datastore1/ESXi500-201112001.zip

Here is an example of using the remote esxcli utility for an ESXi 5 host, you will need to specify the ESXi host using the −−server parameter and −−username/−−password for remote authenication. You may choose to leave off −−password and you will be prompted to enter your credentials. The patch bundle needs to be uploaded to ESXi host using scp/winSCP or vCLI’s vifs utility and then specifying the full path on the command-line:

$ vifs −−server [ESXI-FQDN] −−username [USERNAME] -p ESXi500-201112001.zip “[datastore1] ESXi500-201112001.zip”
$ esxcli −−server [ESXI-FQDN] −−username [USERNAME] software vib update −−depot=/vmfs/volumes/datastore1/ESXi500-201112001.zip

Note: In ESXi 5, −−depot only supports local server path or remote URL. The latter is to help centralize the location of your patches and help reduce manual transfer. This is why you need to transfer the patch to host if you do not have a patch depot.

Here is an example of using Install-VMHostPatch utility for an ESXi host:

Get-VMHost ESXI-FQDN | Set-VMHost -State Maintenance
$DS = Get-VMHost ESXI-FQDN | Get-Datastore datastore1
Copy-DatastoreItem C:\tmp\ESXi500-201112001\ $DS.DatastoreBrowserPath -Recurse
Get-VMHost ESX-FQDN | Install-VMHostPatch -Hostpath “/vmfs/volumes/datastore1/ESXi500-201112001/metadata.zip”

Note: The Install-VMHostPatch cmdlet does have a -LocalPath parameter for you to specify a local path to the patch. For larger files it is recommended you use the Copy-Datastore cmdlet to upload the file to a datastore on the host and then the -HostPath parameter as can be seen in the example above.

As you can see over the releases, we have had several methods of patching a host using the command-line both locally/remotely and it may not always be intuitive. When we converged to only ESXi with the release of vSphere 5.0, you will see that patching from the command-line has also converged to a single command-line utility using ESXCLI and a common patch format called a VIB. ESXCLI was first introduced in vSphere 4.0 and it had some limited capabilities. With vSphere 5.0, it has been significantly enhanced and now supports patching as one of it’s many capabilities. The syntax and expected output is exactly the same if you execute ESXCLI locally or remotely on an ESXi host with the exception of the remote authentication that is required for a remote execution. This should provide for a better user experience and consistency going forward.

An alternative method to patching from the command-line if you do not have VUM is using VMware Go, which is an online service (SaaS) provided by VMware. VMware Go can help manage your ESXi host but it also provides a patching capability similar to that of VUM.

Get notification of new blog postings and more by following lamw on Twitter:  @lamw


PSODs and VMware HA

Despite our best efforts here at VMware, there are occasions where a PSOD may occur.  PSODs can occur for a wide range of reasons, including out of memory and hung CPU conditions.  In a HA protected environment, if a PSOD does occur you would expect that the VMs that were running on the host that encountered the PSOD would be failed over to another host.  There does exist a corner case where this may not be the case.  Let me explain why:
 
Before I gointo details here, let me first make absolutely clear a very important point.  What I am about to describe is a rare corner case.  Odds are you likely have not seen it, nor will you.  It is possible, however, and for those who have experienced it, I hope that this makes things a bit clearer.
 
ESX, as you know, has what we refer to as the ‘COS’, or Service Console.  Sometimes, the COS may become unresponsive.  This can be due to a variety of reasons, but commonly it is due to an issue withmemory.  For example, there may be a memory leak in a process, a third party application consuming a large amount of memory, or excessive numbers of processes running in the COS.
 
When the COS becomes unresponsive or throws an error, the VMkernel detects this and starts to generate a core dump.  This core dump is invaluable for troubleshooting and basically consists of an entire dump of memory as well as some additional overhead.  Of course, this is all compressed so as to save space.  During the time that the VMkernel is performing the dump, it tries to keep everything exactly like it was when the problem with the COS occurred until after the dump completes.  This includes the locks on any datastores used by VMs that were running.  Once this process is finished, the user will see a PSOD.
 
Normally, this core dump occurs very rapidly – less than a minute in most cases.  In rare cases though, it might take longer.  I’ve heard rumors that some people have seen these core dumps take as long as 20 minutes.  This is where the problem starts for people with VMware HA enabled.
 
In this scenario, only the COS has an issue.  VMkernel and everything else is completely functional.  This means that it can still respond to ICMP pings and the like.  Since the COS is not functional though, VMware HA will detect this as a failure of the system and try to restart the VMs that were hosted on another system in the cluster.   However, the‘failed’ system is not completely failed until it completes the core dump.   If VMware HA tries to start a VM on another host, it will fail as that VM is still locked, or in use by the failed system.  VMware HA tries multiple times to restart the VMs.  If for some reason though, the core dump takes an extreme amount of time to complete, VMware HA may time out and give up trying to restart the VMs.  The end result here is that once the failed system does PSOD, it appears that VMware HA failed, as the VMs are not restarted.
 
How can you prevent this from happening to you? 
 
One possible action is to not run anything within the COS.  By doing this, you will eliminate the possibility of an application or script that has not been thoroughly tested by VMware from causing issues within the COS.  
 
Another solution would be to disable the ability of vmkernel to perform a core dump.  This is not a very viable solution for many, as doing this eliminates critical information needed to perform a RCA.  Thus, you might not be able to get to the root cause of your problem.  I’d only recommend doing this in the rare case where you have a known issue with a server but are unable to fix it immediately. 
 
The simplest solution is to use ESXi.  As ESXi provides for a simpler more secure environment without a COS, this problem simply doesn’t exist. 
 
For more information, I would recommend looking at the following KB articles:

VMware ESX and ESXi 4.1 Comparison
Configuring an ESX host to capture a Service Console coredump
Understanding a "Lost Heartbeat" purple diagnostic screen
Understanding an "Oops" purple diagnostic screen