Home > Blogs > Support Insider > Tag Archives: ESXi

Tag Archives: ESXi

How to revert an ESXi host to a previous version using DCUI

This video demonstrates how to revert an ESXi host to a previous version using the Direct User Console Interface (DCUI).

VMware recommends that you back up the configuration data before proceeding with any changes.

Reverting an ESXi host is only available if the host was updated using one of the following methods:

  • VIB installation or removal
  • Profile installation or removal
  • ESXi host updated using VMware Update Manager
  • ESXi host updated from a ISO

Video:

A look at All Paths Down in vSphere

Karthick SivaramakrishnanToday we have a guest post from Karthick Sivaramakrishnan, who is a 3 year veteran at VMware. His primary field of expertise is vSphere Storage and Site Recovery Manager.

This blog post is centered around how ESXi handles unscheduled storage disconnects on vSphere 5.x and 6.x. An unscheduled storage disconnect means some issue in the vSphere environment has led to All-Paths-Down (APD) for a datastore.  An APD situation will be seen when ESXi host does not have any path to communicate with a lun on the storage array.

ESXi host can encounter an APD under several conditions. As a result, we may end up having VMs running on a given datastore go down, the host could get disconnected from vCenter, and in worst cases ESXi could become unresponsive.

From vSphere version 5.x and onwards, we are able to discern whether a disconnect is permanent or transient. Ideally a transient disconnect leads to All Paths Down state and ESXi expects the device to have a temporary disconnect. When we see permanent device loss or PDL the device is expected to have a non-recoverable issue like a hardware error or the lun is unmapped.

In the below example we see all iSCSI datastores are in inactive state.

Datastores

To determine what caused this issue we see ESXi logs, particularly vmkernel and vobd. This issue will be evident in the vmkernel logs.

vmkernel log

2017-01-10T13:04:26.803Z cpu1:32896)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6000eb31dffdc33a0000000000000028] has entered the All Paths Down state.

2017-01-10T13:04:26.818Z cpu0:32896)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6000eb31dffdc33a000000000000002a] has entered the All Paths Down state.

vobd log

2017-01-10T13:04:26.905Z: [scsiCorrelator] 475204262us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.6000eb31dffdc33a0000000000000028. Path vmhba33:C0:T1:L0 is down. Affected datastores: “Green”.

2017-01-10T13:04:26.905Z: [scsiCorrelator] 475204695us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.6000eb31dffdc33a000000000000002a. Path vmhba33:C0:T0:L0 is down. Affected datastores: “Grey”.

From these logs we understand that ESXi host has lost connectivity to the datastore. Any virtual machines using the affected datastore may become unresponsive. In this example while the datastores was mounted on ESXi, we lost the network uplink on the nic that was used for iSCSI connection. This was a transient issue and the datastore came up once the network uplink was restored.

In the below example we see Datastore Black is in inactive state.

Datastore view missing

If we look into the logs to determine whats going on we see these events.

Vmkernel.log

2017-01-09T12:42:09.365Z cpu0:32888)ScsiDevice: 6878: Device naa.6000eb31dffdc33a0000000000000063 APD Notify PERM LOSS; token num:1

2017-01-09T12:42:09.366Z cpu1:32916)StorageApdHandler: 1066: Freeing APD handle 0x430180b88880 [naa.6000eb31dffdc33a0000000000000063]

2017-01-09T12:49:01.260Z cpu1:32786)WARNING: NMP: nmp_PathDetermineFailure:2973: Cmd (0xc1) PDL error (0x5/0x25/0x0) – path vmhba33:C0:T3:L0 device naa.6000eb31dffdc33a0000000000000063 – triggering path evaluation

2017-01-09T12:49:01.260Z cpu1:32786)ScsiDeviceIO: 2651: Cmd(0x439d802ec580) 0xfe, CmdSN 0x4b7 from world 32776 to dev “naa.6000eb31dffdc33a0000000000000063” failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.

2017-01-09T12:49:01.300Z cpu0:40210)WARNING: NMP: vmk_NmpSatpIssueTUR:1043: Device naa.6000eb31dffdc33a0000000000000063 path vmhba33:C0:T3:L0 has been unmapped from the array

After some time passes you will see this message:

2017-01-09T13:13:11.942Z cpu0:32872)ScsiDevice: 1718: Permanently inaccessible device :naa.6000eb31dffdc33a0000000000000063 has no more open connections. It is now safe to unmount datastores (if any) and delete the device.

In this case the lun was unmapped from the array for this host and that is not a transient issue. Sens data 0x5 0x25 0x0 corresponds to “LOGICAL UNIT NOT SUPPORTED” which indicates the device is in Permanent Device Loss (PDL) state. Once ESXi knows the device is in PDL state it does not wait for the device to return back.

ESXi only checks ASC/ASCQ and if it happens to be 0x25/0x0 or  0x68/0x0, it marks device as PDL.

VMware KB 2004684 has in-depth information around APD and PDL situations. It also talks about planned and unplanned PDL. You can read it here: Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x (2004684)

Further on in the hostd logs you will see some additional events that will correlate to storage connection.  Look for the below event id’s.

Event ID : esx.problem.storage.connectivity.lost

datestores3

“esx.problem.storage.connectivity.lost” event indicates a loss in connectivity to the specified storage device.  Any virtual machines using the affected datastore may become unresponsive.

Event ID : esx.problem.scsi.device.state.permanentloss

datastores4

“esx.problem.scsi.device.state.permanentloss” event indicates a permanent device loss.

Top 20 ESXi articles for December 2016

Top 20Here is our Top 20 ESXi articles list for December 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Product offerings for vSphere 6.x
  2. Uploading diagnostic information for VMware using FTP
  3. How to purchase and file Pay Per Incident support for VMware products
  4. Downloading and licensing vSphere Hypervisor (ESXi 6.x)
  5. Uploading diagnostic information for VMware through the Secure FTP portal
  6. Downloading, licensing, and using VMware products
  7. Using the VMware Knowledge Base
  8. Support Contracts FAQs
  9. ESXi host fails with PSOD when using Intel Xeon Processor E5 v4, E7 v4, and D-1500 families
  10. Snapshot removal task stops at 99% in ESXi/ESX
  11. How to consolidate snapshots in vSphere 5.x/6.x
  12. Unable to power off the virtual machine in an ESXi host
  13. “esxcli software vib” commands to patch an ESXi 5.x/6.x host
  14. Creating a persistent scratch location for ESXi 4.x/5.x/6.x
  15. Determining Network/Storage firmware and driver version in ESXi 4.x and later
  16. Unable to delete the virtual machine snapshots
  17. Investigating virtual machine file locks on ESXi
  18. How to unmount a LUN or detach a datastore device from ESXi hosts
  19. ESXi 5.x with E1000e adapter fails with purple diagnostic screen
  20. Recreating a missing virtual machine disk descriptor file

Top 20 ESXi articles for November 2016

Top 20Here is our Top 20 ESXi articles list for November 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Using the VMware Knowledge Base
  2. How to purchase and file Pay Per Incident support for VMware products
  3. Downloading, licensing, and using VMware products
  4. Uploading diagnostic information for VMware using FTP
  5. ESXi host fails with PSOD when using Intel Xeon Processor E5 v4, E7 v4, and D-1500 families
  6. Correlating build numbers and versions of VMware products
  7. Build numbers and versions of VMware ESXi/ESX
  8. Collecting diagnostic information for VMware products
  9. Snapshot removal task stops at 99% in ESXi/ESX
  10. Collecting diagnostic information for ESX/ESXi hosts and vCenter Server using the vSphere Web Client
  11. “vm-support” command in ESX/ESXi to collect diagnostic information
  12. “esxcli software vib” commands to patch an ESXi 5.x/6.x host
  13. Recreating a missing virtual machine disk descriptor file
  14. Support Contracts FAQs
  15. ESXi 5.x with E1000e adapter fails with purple diagnostic screen
  16. Uploading diagnostic information for VMware through the Secure FTP portal
  17. Determining Network/Storage firmware and driver version in ESXi 4.x and later
  18. Creating a persistent scratch location for ESXi 4.x/5.x/6.x
  19. Licensing VMware vCenter Site Recovery Manager
  20. Investigating virtual machine file locks on ESXi

Top 20 ESXi articles for October 2016

Top 20Here is our Top 20 ESXi articles list for October 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Using the VMware Knowledge Base
  2. Uploading diagnostic information for VMware using FTP
  3. How to purchase and file Pay Per Incident support for VMware products
  4. Collecting diagnostic information for VMware products
  5. ESXi host fails with PSOD when using Intel Xeon Processor E5 v4, E7 v4, and D-1500 families
  6. Support Contracts FAQs
  7. Uploading diagnostic information for VMware through the Secure FTP portal
  8. Snapshot removal task stops at 99% in ESXi/ESX
  9. Correlating build numbers and versions of VMware products
  10. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x
  11. End of General Support for vSphere 5.0 and vSphere 5.1
  12. “vm-support” command in ESX/ESXi to collect diagnostic information
  13. Build numbers and versions of VMware ESXi/ESX
  14. Recreating a missing virtual machine disk descriptor file
  15. Investigating virtual machine file locks on ESXi
  16. “esxcli software vib” commands to patch an ESXi 5.x/6.x host
  17. Downloading, licensing, and using VMware products
  18. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x, and ESXi 6.x
  19. Creating a persistent scratch location for ESXi 4.x/5.x/6.0
  20. How to unmount a LUN or detach a datastore device from ESXi hosts

How to recreate a missing vmdk AKA virtual machine disk descriptor file

This video tells you how to recreate a missing virtual machine disk descriptor, also know as the vmdk file.

Note: VMware recommends to attempt to restore the missing descriptor file from backups if possible. If this is not possible, proceed with recreating the virtual machine disk descriptor file.

Watch the video to see the steps taken to create a virtual machine disk descriptor file:

  • Identify the size of the flat file in bytes.
  • Create a new blank virtual disk that is the same size as the original. This serves as a baseline example that is modified in later steps.
  • Rename the descriptor file (also referred to as a header file) of the newly-created disk to match the name of the original virtual disk.
  • Modify the contents of the renamed descriptor file to reference the flat file.
  • Remove the leftover temporary flat file of the newly-created disk, as it is not required.


This procedure will not work on virtual disks configured with a Para-virtualized SCSI controller in the virtual machine

Top 20 ESXi articles for September 2016

Top 20Here is our Top 20 ESXi articles list for September 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Recreating a missing virtual machine disk descriptor file
  2. Troubleshooting an ESXi/ESX host in non responding state
  3. vMotion of virtual machine fails on certain models of Intel Processor
  4. VMware End User License Agreements
  5. Licensing VMware vCenter Site Recovery Manager
  6. Unable to delete the virtual machine snapshots
  7. End of General Support for vSphere 5.0 and vSphere 5.1
  8. Restarting the Management agents in ESXi
  9. Connectivity to a VMFS5 datastore is lost when using VAAI ATS heartbeat
  10. Unmanaged workload is detected on datastore running SIOC
  11. Resolving the CID mismatch error: The parent virtual disk has been modified since the child was created
  12. “maximum consolidate retries was exceeded for scsix:x” error in ESXi
  13. How to register/add a VM to the Inventory in vCenter Server
  14. Downloading, licensing, and using VMware products
  15. Update sequence for vSphere 6.0 and its compatible VMware products
  16. ESXi 5.5 Update 3b and later hosts are not manageable after an upgrade
  17. Collecting diagnostic information for VMware products
  18. “LINT1 motherboard interrupt” error in an ESX/ESXi host
  19. Using the esxcli storage vmfs unmap command to reclaim VMFS deleted blocks on thin-provisioned LUNs
  20. Understanding lost access to volume messages in ESXi 5.5/6.x

Top 20 ESXi articles for August 2016

Top 20Here is our Top 20 ESXi articles list for August 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Collecting diagnostic information for VMware ESX/ESXi
  2. Snapshot removal task stops at 99% in ESXi/ESX
  3. Support Contracts FAQs
  4. Investigating virtual machine file locks on ESXi/ESX
  5. PSOD with Intel Xeon CPU E5-26xx v4 CPUs
  6. How to purchase and file Pay Per Incident support for VMware products
  7. Using the VMware Knowledge Base
  8. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x, and ESXi 6.x
  9. Uploading diagnostic information for VMware through the Secure FTP portal
  10. Restarting the Management agents in ESXi
  11. Correlating build numbers and versions of VMware products
  12. Licensing VMware vCenter Site Recovery Manager
  13. “esxcli software vib” commands to patch an ESXi 5.x/6.x host
  14. Product offerings for vSphere 5.x
  15. ESXi 5.x with E1000e adapter fails with purple diagnostic screen
  16. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x
  17. Recreating a missing virtual machine disk descriptor file
  18. How to unmount a LUN or detach a datastore device from ESXi hosts
  19. Collecting diagnostic information for VMware products
  20. Creating a persistent scratch location for ESXi 4.x/5.x/6.0

Top 20 ESXi articles for July 2016

Top 20Here is our Top 20 ESXi articles list for July 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Uploading diagnostic information for VMware through the Secure FTP portal
  2. Support Contracts FAQs
  3. Commands to monitor snapshot deletion in ESXi/ESX
  4. How to purchase and file Pay Per Incident support for VMware products
  5. Uploading diagnostic information for VMware using FTP
  6. Downloading, licensing, and using VMware products
  7. Licensing VMware vCenter Site Recovery Manager
  8. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x, and ESXi 6.x
  9. ESXi 5.x with E1000e adapter fails with purple diagnostic screen
  10. Recreating a missing virtual machine disk descriptor file
  11. Using the VMware Knowledge Base
  12. Product offerings for vSphere 5.x
  13. ESXi hosts are no longer manageable after an upgrade
  14. Installing patches on an ESXi 5.x/6.x host from the command line
  15. Enabling or disabling VAAI ATS heartbeat
  16. Restarting the Management agents in ESXi
  17. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x
  18. Consolidating snapshots in vSphere 5.x/6.0
  19. “maximum consolidate retries was exceeded for scsix:x” error in ESXi
  20. Build numbers and versions of VMware ESXi/ESX

Top 20 ESXi articles for June 2016

Top 20Here is our Top 20 ESXi articles list for June 2016. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. VMware ESXi 5.x host experiences a purple diagnostic screen mentioning E1000PollRxRing and E1000DevRx
  2. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x, ESXi 5.x, and ESXi 6.x
  3. Commands to monitor snapshot deletion in VMware ESXi/ESX
  4. ESXi 5.5 Update 3b and later hosts are no longer manageable after upgrade
  5. Restarting the Management agents on an ESXi or ESX host
  6. Recreating a missing virtual machine disk descriptor file
  7. Identifying and addressing Non-Maskable Interrupt events on an ESX/ESXi host
  8. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x
  9. Snapshot consolidation in VMware ESXi 5.5.x and ESXi 6.0.x fails with the error: maximum consolidate retries was exceeded for scsix:x
  10. Powering off a virtual machine on an ESXi host
  11. Correlating build numbers and versions of VMware products
  12. Updating an ESXi/ESX host using VMware vCenter Update Manager 4.x and 5.x
  13. Update sequence for vSphere 6.0 and its compatible VMware products
  14. ESXi 5.5 or 6.0 host disconnects from vCenter Server with the syslog.log error: Unable to allocate memory
  15. Enabling or disabling VAAI ATS heartbeat
  16. ESXi 6.0 hosts become unresponsive when joined to an Active Directory domain
  17. Enabling EVC on a cluster when vCenter Server is running in a virtual machine
  18. Unable to delete the virtual machine snapshot due to locked files
  19. Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions)
  20. Reverting to a previous version of ESXi