vSphere 6.0 introduces a powerful new feature as part of vSphere HA called VM Component Protection (VMCP). VMCP protects virtual machines from storage related events, specifically Permanent Device Loss (PDL) and All Paths Down (APD) incidents.
Permanent Device Loss (PDL)
A PDL event occurs when the storage array issues a SCSI sense code indicating that the device is unavailable. A good example of this is a failed LUN, or an administrator inadvertently removing a WWN from the zone configuration. In the PDL state, the storage array can communicate with the vSphere host and will issue SCSI sense codes to indicate the status of the device. When a PDL state is detected, the host will stop sending I/O requests to the array as it considers the device permanently unavailable, so there is no reason to continuing issuing I/O to the device.
All Paths Down (APD)
If the vSphere host cannot access the storage device, and there is no PDL SCSI code returned from the storage array, then the device is considered to be in an APD state. This is different than a PDL because the host doesn’t have enough information to determine if the device loss is temporary or permanent. The device may return, or it may not. During an APD condition, the host continues to retry I/O commands to the storage device until the period known as the APD Timeout is reached. Once the APD Timeout is reached, the host begins to fast-fail any non-virtual machine I/O to the storage device. This is any I/O initiated by the host such as mounting NFS volumes, but not I/O generated within the virtual machines. The I/O generated within the virtual machine will be indefinitely retried. By default, the APD Timeout value is 140 seconds and can be changed per host using the Misc.APDTimeout advanced setting.
VM Component Protection (VMCP)
vSphere HA can now detect PDL and APD conditions and respond according to the behavior that you configure. The first step is to enable VMCP in your HA configuration. This settings simply informs the vSphere HA agent that you wish to protect your virtual machines from PDL and APD events. In the spirit of keeping things dead simple, it’s as easy as a clicking a checkbox. To see for yourself, head on over to the Feature Walkthrough site and see just how simple this really is.
Cluster Settings -> vSphere HA -> Host Hardware Monitoring – VM Component Protection -> Protect Against Storage Connectivity Loss.
The next step is configuring the way you want vSphere HA to respond to PDL and ADP events. Each type of event can be configured independently. These settings are found on the same window that VMCP is enabled by expanding the Failure conditions and VM response section.
Response for Datastore with Permanent Device Loss (PDL)
There are three actions that can be taken in response to a PDL event. These choices are pretty simple since a PDL event is black and white.
Disabled – No action will be taken to the affected VMs. Issue events – No action will be taken against the affected VMs, however the administrator will be notified when a PDL event has occurred. Power off and restart VMs – All affected VMs will be terminated on the host and vSphere HA will attempt to restart the VMs on hosts that still have connectivity to the storage device.
Response for Datastore with All Paths Down (APD)
There are few more options available for an APD response. This is because the device state is unknown and may only be temporarily unavailable.
Disabled – No Action will be taken to the affected VMs. Issue events – No action will be taken against the affected VMs, however the administrator will be notified when an APD event has occurred. Power off and restart VMs (conservative) – vSphere HA will not attempt to restart the affected VMs unless it has determined there is another host that can restart the VMs. The host experiencing the APD will communicate with the HA master to determine if there is sufficient capacity to power on the affected workloads. If the master determines there is sufficient capacity, the host experiencing the APD will terminate the VMs so they can be restarted on a healthy host. If the host experiencing the APD cannot communicate with the vSphere HA master, no action will be taken. Power off and restart VMs (aggressive) – vSphere HA will terminate the affected VMs even if it cannot determine that another host can restart the VMs. The host experiencing the APD will attempt communicate with the HA master to determine if there is sufficient capacity to power on the affected workloads. If the HA master is not reachable, it will be unknown if there is sufficient capacity available to restart the VMs. In this scenario, the host takes the risk and terminates the VMs so they can be restarted on the remaining healthy hosts. However, if there is not sufficient capacity available, vSphere HA may not be able to recover all of the affected VMs. This is common in a network partition scenario where a host cannot communicate with the HA master to get a definitive response to the likelihood of a successful recovery.
Delay for VM failover for APD
Once the APD Timeout has been reached (default: 140 seconds) VMCP will wait an additional period of time before taking action against the affected VMs. By default, the waiting period is 3 minutes. In other words, VMCP will wait 5m:20s before taking action against VMs. The sum of the APD Timeout and the Delay for VM Failover is also known as the VMCP Timeout.
Response for APD recovery after APD timeout
This setting will instruct vSphere HA to take a certain action if an APD event is cleared after the APD timeout was reached but before the Delay for VM failover has been reached.
Disabled – No action will be taken against the affected VMs. Reset VMs – The VMs will be reset on the same host. (Hard reset)
This option is available because some applications or guest operating systems may be in an unstable condition after losing connection with storage services for an extended period of time. This setting will instruct vSphere HA how to handle this situation.
VMCP Recovery Workflow
Figure 1. VMCP Recovery Workflow
VMCP Recovery Timeline
Figure 2. VMCP Recovery Timeline
T=0s: A storage failure is detected. VMCP will start the recovery workflow. T=0s: For a PDL event, the recovery process immediately starts. VMs will be restarted on healthy hosts in the cluster. T=0s: For an APD condition, the APD Timeout timer starts. T=140s: The host declares an APD Timeout and will begin to fast fail non-virtual machine I/O to the unresponsive storage device. By default, this is 140 seconds. T=320s: The VMCP Timeout. This is 3 minutes after the APD Timeout has been reached. vSphere HA will start the APD recovery response. T=140s-T=320s: The period after an APD Timeout, but before the VMCP Timeout. VMs may become unstable after losing access to storage for an extended period of time. If an APD is cleared in this time period, the option to reset the VMs is available.
Summary
VMCP is a long-awaited feature that provides protection against datastore accessibility failures that affect the virtual machines running on a host in a vSphere HA cluster. Hopefully this article will help explain the various configuration options available that are appropriate for your specific environment.
