While VMware has been announcing a myriad of new capabilities that are a part of the Express Storage Architecture in vSAN 8, several enhancements make vSAN 8 enticing for existing vSAN clusters using the Original Storage Architecture, or OSA.
Perhaps the most significant of these enhancements for the OSA in vSAN 8 is the increase in the maximum capacity of caching/buffering devices that vSAN can recognize – increased from 600GB to 1.6TB per disk group. Let’s learn how this can benefit customers making the transition to vSAN 8 using the OSA.
Write Buffer Limits in Previous Versions of vSAN (OSA)
The vSAN OSA provides performance and capacity through a two-tier architecture, where a caching/buffering tier exists for increased levels of performance to augment the storage devices in the capacity tier. As described in the post Write Buffer Sizing in vSAN when Using the Very Latest Hardware the vSAN OSA uses disk groups to achieve this result. While a disk group could have many capacity devices it was limited to one caching/buffering device per disk group, and vSAN would recognize a buffer capacity of no greater than 600GB per disk group.
For customers, this meant that sometimes their buffering devices were not being used to their fullest extent because market conditions prevented them from purchasing anything smaller than a 1.2TB to 1.6TB buffer device. As capacity devices increased in density or workloads became more intense, this put additional burdens on the cluster because the buffer devices remained at a fixed limit of 600GB. Customers could reduce the effects of the 600GB limit by adding more disk groups, but this was not always possible.
For environments powering demanding workloads with large working sets, or large capacity tiers using slower, value-based flash devices, the smaller buffers would tend to result in increased latency, and compromises in performance consistency. Smaller buffers are more likely to destage frequently overwritten, or read data (aka “hot data”), placing burdens on both the buffering tier and capacity devices.
A Larger Logical Write Buffer Limit for vSAN 8 OSA
The OSA in vSAN 8 increases the logical address space of a write buffer device to 1.6TB. For those customers using these larger storage devices as a write buffer, this nearly triples the effective buffer capacity, all through a software upgrade.
What does a larger write buffer mean for you and your workloads? Simply put, better performance! How? A vSAN cluster will likely see improved performance in the following ways:
- Improve performance of writes. A larger buffer capacity allows more data that can be written at a burst level of performance, maintaining a lower level of latency, as opposed to the slower, steady-state rate of the capacity tier.
- Improves performance consistency. Larger buffers allow more hot data to reside in this faster tier. A larger working set of data living in a fast tier means a larger percentage of hot data is accessible at a fast rate. This is known as temporal locality, and is a common factor in system optimizations.
- Improve performance of reads. With vSAN All-flash configurations, while the buffering tier is designed to ingest writes, all read requests will check the buffer first before fetching the data from the capacity tier.
- Reduces I/O demand on the capacity tier. When more hot data can reside in a buffer tier, less demand is placed on the capacity tier, reducing I/O amplification. Frequent overwrites can remain in the buffer and many read requests can be serviced by the buffer – both of which reduce the demand on the capacity tier devices.
- Reduced reboot times. This improvement comes through a feature introduced in vSAN 7 U1 that allowed in-memory metadata tables to be saved to persistent media upon a host restart – speeding up the restart process. A larger buffer allows for more metadata mapping to be saved, and thus a more efficient host restart process.
- Improved Resynchronization performance. A larger buffer can absorb more resynchronization activities on a host, reducing the time to completion.
Figure 1. How a larger buffer device can hold more hot data for more workloads, or higher-demanding workloads.
Recommendation: When in doubt, use an NVMe-based storage device of 1.6TB as the write buffer. This is a simple way to maximize the performance benefit of a large write buffer while simplifying your design process.
Taking Advantage of this Improvement
This improvement only applies to vSAN clusters running an all-flash configuration, using the OSA. Hybrid vSAN clusters using the OSA are still limited to a 600GB write buffer. Customers who wish to take advantage of this enhancement must meet certain requirements, which include, but may not be limited to:
- vCenter Server must be running version 8, and vSAN hosts must be running vSAN 8, using the OSA.
- Enable the feature using an advanced setting per knowledge base article: KB 89485
- vSAN cluster must be using the latest On-Disk Format version in vSAN 8.
- Existing disk groups must be recreated after the advanced setting is enabled on the vSAN hosts.
Note that enabling a larger write buffer will consume more memory on the host: Up to 5GB more per disk group per host.
This write buffer enhancement only applies to the OSA in all-flash configurations. The ESA in vSAN 8 does not use the construct of disk groups, or the use of dedicated caching devices. For more information on the vSAN Express Storage Architecture, see An Introduction to the vSAN Express Storage Architecture.
Summary
vSAN 8, using the Original Storage Architecture, nearly triples the logical limit of the buffer capacity to a maximum of 1.6TB, allowing users to take advantage of this higher-density, high-performance devices to deliver improved levels of performance consistency to their clusters running the vSAN OSA. More capacity at the buffer tier means that workloads can run at the higher performing rate for longer periods, and it can reduce the strain on the value-based capacity tier devices, as a larger amount of working set data can reside in the buffer.
