All Paths Down (APD) is an issue which has come up time and time again, and has impacted a number of our customers. Let's start with a brief description about what All Paths Down (APD) actually is & how it occurs, & what impact it has on the host. Then we'll get into how we have improved the behaviour in 5.0.
A brief overview of APD
APD is what occurs on an ESX host when a storage device is removed in an uncontrolled manner from the host (or the device fails), and the VMkernel core storage stack does not know how long the loss of device access will last. A typical way of getting into APD would be a Fiber Channel switch failure or (in the case of an iSCSI array) a network connectivity issue. Note that if careful consideration is given to the redundancy of your multi-pathing solution and our best practices for setting up switch and HBA redundancy is followed, it could help a lot in avoiding this situation.
The APD condition could be transient, since the device or switch might come back. Or it could be permanent in so far as the device might never come back. In the past, we kept the I/O queued indefinitely, and this resulted in I/Os to the device hanging (not acknowledged). This became particularly problematic when someone issued a rescan of the SAN from a host or cluster which was typically the first thing customer tried when they found a device was missing. The rescan operation caused hostd to block waiting for a response from the devices (which never comes). Because hostd is blocked waiting on these responses, it can't be used by other services, like the vpx agent (vpxa) which is responsible for communication between the host and vCenter. The end result is the host becoming disconnected from vCenter. And if the device is never coming back, well we're in a bit of a pickle! 🙁
It should also be noted that hostd could also grind to a halt even without a rescan of the SAN being initiated. The problem is that hostd has a limited number of worker threads. If enough of these threads get stuck waiting for I/O to a device that is not responding, hostd will eventually be unable to communicate to anything else, including healthy devices, because it doesn't have any free worker threads to do any work.
APD Handling before vSphere 5.0
To alleviate some of the issues arising from APD, a number of advanced settings were added which could be tweaked by the admin to mitigate this behaviour. Basically this involved not blocking hostd when it was scanning the SAN, even when it came across a non-responding device (i.e. a device in APD state). This setting was automatically added in ESX 4.1 Update 1 and ESX 4.0 Update 3. Previous versions required customers to manually add the setting. This is discussed in greater detail in KB 1016626.
APD Handling Enhancements in vSphere 5.0 – Introducing Permanent Device Loss (PDL)
In vSphere 5.0, we have made improvements to how we handle APD. First, what we've tried to do is differentiate between a device to which connectivity is permanently lost, and a device to which connectivity is transiently or temporarily lost. We now refer to a device which is never coming back as a Permanent Device Loss (PDL).
APD is now considered a transient condition, i.e. the device may come back at some time in the future.
PDL is considered a permanent condition where the device is never coming back.
As mentioned earlier, I/O to devices which were APD would be queued indefinitely. With PDL devices (those devices which are never coming back), we will now fail the I/Os to those devices immediately. This means that we will not end up in a situation where processes such as hostd get blocked waiting on I/O to these devices, which also means that we don't end up in the situation where the host disconnects from vCenter.
This begs the question – how do we differentiate between devices with are APD or PDL?
The answer is via SCSI sense codes. SCSI devices can indicate PDL state with a number of sense codes returned on a SCSI command. One such sense code is 5h / ASC=25h / ASCQ=0 (ILLEGAL REQUEST / LOGICAL UNIT NOT SUPPORTED). The sense code returned is a function of the device. The array is in the best position to determine if the requests are for a device that no longer exists, or for a device that just has an error/problem. In fact, in the case of APD, we do not get any sense code back from the device.
When the last path to the device in PDL returns the appropriate sense code, internally the VMkernel changes the device state to one which indicates a PDL. It is important to note that ALL paths must be in PDL for the DEVICE to become PDL. Once a device is in this state, commands issued to that device will return with VMK_PERM_DEV_LOSS. In other words, we now fail the I/Os immediately rather than have I/Os hanging. This will mean that with PDL, hostd should never become blocked, and no hosts should disconnect from vCenter. Hurrah!
None of these distinctions between APD and PDL are directly visible to the Virtual Machines that are issuing I/Os. SCSI commands from a VM to an APD/PDL device are simply not responded to by the VMkernel, and when they timeout, it elicits a retransmit attempt from the host. In other words, VMs retry their I/O indefinitely. That is why in most cases, if the device which was in APD does comes back, the VMs continue to run from where they left off – this is one of the really great features of virtualization in my opinion (and has saved many an admin who inadvertently disconnected the wrong cable or offlined the wrong device) 🙂
Best Practice to correctly remove a device & avoid APD/PDL
There has never been an intuitive or well defined way to remove a storage device from an ESX in the past. Now we have a controlled procedure to do this in 5.0.
5.0 introduces two new storage device management techniques. You now have the ability to mount/unmount a VMFS volume & attach/detach a device. Therefore, if you want to remove the device completely, the first step is to unmount the volume. In this example, I have a NetApp device on which a VMFS-5 filesystem has been created. First, get the NAA id of the underlying LUN as you will need this later. This can be found by clicking on the Properties of the datastore, & looking at the extent details:
Before proceeding, make sure that there are no running VMs on the datastore, that the datastore is not part of a datastore cluster (unused by Storage DRS), is not used by vSphere HA as a heartbeat datastore & does not have Storage I/O Control enabled.
With the NAA id noted, right click on the volume in the Configuration > Storage > Datastores view, and select Unmount:
When the unmount operation is selected, a number of checks are done on the volume to ensure that it is in a state that allows it to be unmounted, i.e. no running VMs, Storage I/O Control not enabled, not used as a Heartbeat Datastore, not managed by Storage DRS.
If the checks pass, click OK & then the volume is unmounted.
The CLI command to do an unmount is esxcli storage filesystem unmount if you prefer to do it from the ESXi shell. When that the volume is safely unmounted, the next step is to detach it from the host. This can also done either via the CLI or via the UI in the Configuration > Storage window, but the view must be changed to Devices rather than Datastores. Click on the Devices button, select the correct device using the NAA id noted previously, right click and select Detach:
The detach will check that the volume is indeed in a state that allows it to be detached:
The same task can be done via the ESXi shell. The command that I need to use to do a detach is esxcli storage core device set –state=off. Note how the Status changes from on to off in the following commands:
~ # esxcli storage core device list -d naa.60a98000572d54724a34642d71325763 naa.60a98000572d54724a34642d71325763 Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d54724a34642d71325763) Has Settable Display Name: true Size: 3145923 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.60a98000572d54724a34642d71325763 Vendor: NETAPP Model: LUN Revision: 7330 SCSI Level: 4 Is Pseudo: false Status: on Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Thin Provisioning Status: yes Attached Filters: VAAI Status: unknown Other UIDs: vml.020000000060a98000572d54724a34642d713257634c554e202020
~ # esxcli storage core device set –state=off -d naa.60a98000572d54724a34642d71325763
~ # esxcli storage core device list -d naa.60a98000572d54724a34642d71325763 naa.60a98000572d54724a34642d71325763 Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d54724a34642d71325763) Has Settable Display Name: true Size: 3145923 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: Vendor: NETAPP Model: LUN Revision: 7330 SCSI Level: 4 Is Pseudo: false Status: off Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Thin Provisioning Status: yes Attached Filters: VAAI Status: unknown Other UIDs: vml.020000000060a98000572d54724a34642d713257634c554e202020 ~ #
This device is now successfully detached from the host. It remains visible the UI at this point:
You can now do a rescan of the SAN and this will safely remove the device from the host.
APD may still occur
The PDL state is definitely a major step in handling APD conditions. The unmount/detach mechanism should also alleviate certain APD conditions. However there is still a chance that APD conditions can occur. For instance, if the LUN fails in a way which does not return the sense codes expected by PDL, then you could still experience APD in your environment. VMware is continuing to work on the APD behaviour to mitigate the impact it might have on your infrastructure.
Detecting PDL Caveat
There is one important caveat to our ability to detect PDL. Some iSCSI arrays map a Lun to Target as a one-to-one relationship, i.e. there is only ever a single LUN per Target.
In this case, the iSCSI arrays do not return the appropriate SCSI sense code, so we cannot detect PDL on these arrays types.
However most other storage arrays on our HCL should be able to provide SCSI sense code to enable the VMkernel detect PDL.
Follow these blogs and more VMware Storage information on Twitter: VMwareStorage
