HCI clusters use local storage devices

Hyperconverged Infrastructure (HCI) clusters powered by VMware vSAN consist of multiple hosts with local storage devices in each host contributing to a shared vSAN datastore. This model combines the cost savings of using local storage devices with the benefits of shared storage. However, when a vSAN host is offline due to planned or unplanned downtime, compute and shared datastore capacities are impacted until the host is back online. This is one of the reasons we recommend maintaining a minimum of 25-30% free space or “slack space” on the vSAN datastore. VMware Knowledge Base article 2108740 is an example of one of the many places this recommendation is made. Adequate free space provides capacity for vSAN to perform tasks such as VM snapshots, re-balance drive capacity utilization, and host maintenance mode.

Digging into this a bit more, it is important to understand how data is distributed across hosts in a vSAN datastore. A storage policy is applied to objects such as a VM’s virtual disks. vSAN uses the rules in the storage policy to determine how many copies of an object should be created. These rules also define how the copies are placed to avoid an object becoming inaccessible a due to drive or host failure. The image below shows a scenario where a virtual disk has a RAID-1 mirroring policy applied that enables the object to remain accessible in the event of a single failure (Failures to Tolerate = 1).

In the scenario above, there are two mirrored components that contain the VM’s data and a small witness component. A minimum of two of these components must be active for the virtual disk to be accessible. For example, if host .163 is offline, the two components on Host .164 and Host .161 are still active and the virtual disk object remains accessible. This is just one scenario. There are many combinations and levels of fault tolerance with vSAN architecture and storage policies.

A host in maintenance mode is similar to a host being offline

This blog article focuses on maintenance mode as some administrators do not realize that a host in maintenance mode is similar to a host being offline from a compute and storage capacity perspective. Placing a host into maintenance mode in an HCI cluster means that compute and storage capacity is reduced until the host exits maintenance mode.

Here is vSAN datastore capacity before putting a host into maintenance mode…

… and after the host is put into maintenance mode…

Note that the total vSAN datastore capacity has been reduced from 5.82TB to 4.37TB. That is a 25% reduction, which is what we would expect if one node in a 4-node cluster is offline.

The screen shot below shows the status of one of the components located on a host that is in maintenance mode.

The virtual disk object remains accessible as two of the three components are active. However, if Host .164 fails while Host .161 is in maintenance mode, that virtual disk would not be accessible until Host .164 comes back online.

Lesson learned and recommendations

The lesson to be learned here is that maintenance mode effectively removes a host’s compute and storage resources from an HCI cluster until the host exits maintenance mode. With that in mind, here are a few points and operational recommendations to consider when using maintenance mode:

• Maintain 25-30% free space or “slack space” for operations such as VM snapshots, component rebuilds, and maintenance mode.
• Use vSAN Health to verify all hosts and the vSAN cluster are healthy before putting a host into maintenance mode. If there are existing issues, resolve them before putting a host into maintenance mode.
• Avoid having more than one host (per cluster) in maintenance mode at any given time.
• Placing a vSAN host into maintenance mode reduces compute and vSAN datastore capacity until the host exits maintenance mode.
• The resilience to drive and host failure will likely be reduced for some objects unless the “Full data migration” maintenance mode option is selected. The default maintenance mode option is “Ensure accessibility.”
• Read this blog article that provides details on the behavior and impact of maintenance mode options: A Closer Look at vSAN Maintenance Mode
• Here is another recommended article: vSAN Maintenance Mode – RAID-1 and RAID-5 using “Ensure Accessibility”
• Use the “Full data migration” maintenance mode option if the host will be offline for more than 60 minutes. This option will help ensure redundant copies of data are migrated to other hosts to maintain resilience and compliance with storage policies. It is likely more time will be required for the host to enter maintenance mode using the “Full data migration” option. Why 60 minutes? 60 minutes is the default setting for the CLOM Repair Delay timer – see VMware Knowledge Base article 2075456 for details on that setting.

• If you want to mitigate this corner-case risk (a host failing while another is in maintenance mode), consider a storage policy with Failures to Tolerate (FTT) = 2. That policy might be assigned permanently to your most important VMs or assigned temporarily when performing maintenance on a cluster. In either case, make sure you understand the impact to capacity consumption. As an example, a 200GB virtual disk with a policy assigned where RAID-1 mirroring is used and FTT = 2 can consume up to 600GB of raw storage capacity in a standard vSAN cluster. You will also need a proper vSAN cluster design – basically, more hosts –  to support higher FTT rules. The vSAN Design and Sizing Guide is very helpful when making design decisions.

@jhuntervmware on Twitter