Which do you choose, safety or productivity? That’s a trick question; the answer is if either wins, both lose. The balance between safety and productivity is usually referenced in the context of workplace safety. The small gains made in production by shortcutting safety seem to be worthwhile until an accident occurs. Then the gains disappear or even become losses in many cases. This concept also applies to the balance between managing safety (background operations I/O) and productivity (VM I/O) in your storage cluster. If your background operations (i.e resync operations, rebalancing etc.) consume all your I/O then your application performance will suffer. Likely, if your application consumes all your I/O and your backend cannot safely maintain availability then your gains made by starving your backend become losses.
Awareness and control of primary and secondary I/O operations are one of the many benefits of having a distributed storage system like vSAN integrated directly into the hypervisor. The way vSAN handles this balance has evolved throughout the past few releases. In vSAN 5.5 and 6.0 limited resynchronization control could be implemented at the CLI. In earlier releases, this could be manually throttled. In vSAN 6.6 and ESXi 6.0 Patch 2, a static throttle for rebuilds was implemented. Manual throttling was certainly a powerful feature but left the decision of balancing safety and productivity to the user.
Let’s stick with the workplace safety analogy for a minute. The manufacturing industry is a great example of a place where safety measures are a critical part of the success of a business. However, in a recent survey from a manufacturing media resource Automation World, respondents admitted two reasons why safety guidelines are not always followed: they don’t want to disrupt production and the procedures are inconvenient. It would seem that the balance between safety and productivity could shift depending on the employee’s values. In vSAN 6.6.1 automated throttling of traffic was introduced. This introduced a general windowing capability based on virtual machine latency. With vSAN 6.7, Adaptive Resync takes advantage of VMware’s abilities to optimize and tune vSAN in a way that achieves an all new level of consistency of VM performance under a wide variety of conditions.
Adaptive Resync
Adaptive Resync found in vSAN 6.7 is a new mechanism to properly regulate the flow of resynchronization I/O versus VM I/O during periods of contention. It represents a new level of capabilities and control that allows vSAN to share storage bandwidth resources by I/O class.
In order to provide a mechanism to ensure the proper priority and fairness of I/Os under a variety of conditions, the solution needed to be able to achieve the following:
- Establish and identify different classes of I/O as they occur.
- Have a way to regulate bandwidth for classes of I/O at various layers in the storage stack.
- Have logic that can intelligently decipher conditions of I/O types, to dynamically place back pressure and regulate as needed.
vSAN 6.7 distinguishes four types of I/O, and has a pending queue for each I/O class, as shown here:
- VM I/O represents all read and write requests from the guest VM. This includes VM I/Os that are committed or fetched from multiple hosts as a part of the assigned storage policy.
- Resync I/O represents all I/O activities that occur as the result of resynchronization, rebuild, or repair operations.
- Metadata I/O represents all I/O related to the management of the objects that comprise a VM, such as witness traffic, as well as cluster monitoring and membership activities.
- Namespace I/O relates to all I/O related to the namespace object of the VM.
vSAN manages and regulates these various I/O types using the dispatch\fairness scheduler. So long as there is no contention, all I/O types are pulled from their respective queues. During periods of contention, the scheduler can determine from the bandwidth regulator the rate of ingress, and can continue to reduce the inflow of I/Os until the congestion signals stop growing. When I/Os exceed the advertised available bandwidth, the scheduler will ensure that resync I/Os are assigned no less than approximately 20% of the available bandwidth, leaving the remaining 80% for VM and other I/O classes. Again when there is no contention these limits are not applied.
Let’s look at an example. A VM is hard at work consuming approximately 75% of the total bandwidth available. Then a change is made to a VM Storage Policy that requires some background resync activity. This causes some contention. As a result, the vSAN dispatch\fairness scheduler ensures the resync activity consumes no more than 20% of the available I/O leaving the remaining 80% for the VM workload. Once the VM generates less I/O removing the contention, Adaptive Resync allows the resync activity to consume the remaining bandwidth.
- Period 1: This represents the VM I/O activity being able to freely use all available bandwidth while no resync activity is occurring.
- Period 2: This represents resync activity beginning to occur while VM I/O activity is still busy. As the aggregate bandwidth of both I/O types near 100%, VM I/Os are slowed slightly in order to grant 20% of the bandwidth available for resync activity.
- Period 3: This represents VM I/O and resync I/O activity that activity that is still very busy, exceeding the advertised bandwidth, and both maintain their designated allocation.
- Period 4: This represents VM I/O activity where VMs are generating less I/O activity. Adaptive Resync allows resync activity to consume the remaining available bandwidth.
In the illustration below a host failure was initiated to demonstrate vSAN’s ability to manage availability. vSAN determines it needs to rebuild components for the affected VMs to maintain the number of Failures to Tolerate dictated by the VM Storage Policy. As a result, resync IOPS increase. Later, VM IO activity increases causing contention. Adaptive resync balances the resync and VM actvity until there is no longer contention.
- Period 1: This represents the resync I/O activity being able to freely use all available bandwidth while no VM activity is occurring.
- Period 2: This represents VM activity beginning to occur while resync I/O activity is still busy. As the aggregate bandwidth of both I/O types near 100%, resync I/Os are slowed slightly in order to grant 80% of the bandwidth available for VM activity.
- Period 3: This represents VM I/O and resync I/O activity that activity that is still very busy, exceeding the advertised bandwidth, and both maintain their designated allocation.
- Period 4: This represents VM I/O activity where VMs are generating less I/O activity. Adaptive Resync allows resync activity to consume the remaining available bandwidth and finish rebuilding the absent components.
In summary
Adaptive Resync in vSAN 6.7 introduces the mechanics necessary to implement a fully intelligent, adaptable flow control mechanism for managing resync I/O and VM I/O. The functional behavior of Adaptive Resync could be summarized as:
- When no resync activity is occurring, VM I/Os can consume up to 100% of the available bandwidth.
- When resync and VM I/O activity is occurring, and the aggregate bandwidth of the I/O classes is below the advertised available bandwidth, neither I/O class will be throttled.
- When resync and VM I/O activity is occurring, and the aggregate bandwidth of the I/O classes exceeds the advertised bandwidth, resync I/Os are assigned no less than approximately 20% of the bandwidth, allocating approximately 80% for VM I/Os.
The idea of “production vs. safety” must give way to the vision of “safe production.” The Adaptive Resync capabilities of vSAN 6.7 maximize the ability for resynchronization traffic to use as much bandwidth as possible while not imparting performance implications on the respective VMs powered by vSAN. For detailed information on vSAN Adaptive Resync and all other vSAN related information be sure to go to storagehub.vmware.com.
@vPedroArrow
