Designing vSAN Disk groups – All Flash Cache Ratio Update

For More information on this topic, see the vSAN Design Guide found at core.vmware.com

vSAN cache recommendations for vSAN 5.5 were 10% of used capacity. With the addition of all-flash in 6.0 it stayed at 10% and Cormac here explains the math behind this. Now if you look closely you’ll notice a few issues as vSAN and flash drives have progressed.

  1. The 10% ratio guidelines was based on a number of assumptions (over-write ratio’s, write endurance being your only bottleneck and not latency consistency, or throughput capabilities).
  2. A 600GB write buffer partition was discussed as a bottleneck in sizing cache and that a drive couldn’t use more than this for endurance/performance.
  3. It assumes that the capacity tier will scale cache overwrites and endurance usage with its capacity growth

The reality is that:

Workload read write ratio and randomness 70% or higher random read workloads can make do with much less cache (800GB for larger hosts, 200GB for small ones). They provide more time for de-stage to catch up. Higher ratio’s of writes can do with slightly less cache but putting 5TB of Cache in a host is unnecessary for performance or endurance reasons. The size of the drive does contribute to performance results.

Drives above 600GB are supported and work just fine.  The write buffer partition size only matters in the context of burst non-restaged writes. Outside of niche streaming write workloads, and deploying several TB’s for use cases outside of these workloads is “excessive”. The higher endurance drives still have a lot of value in reducing write latency, but there are other limits that come into play. Drives are virtualized internally and Duncan explains this well here.

Capacity Tier size does not matter for sizing the write cache. Capacity tier devices are becoming more resilient. With existing endurance optimizations, customer sampling, and testing it has been identified that the capacity of these devices is not relevant in sizing going forward.

New Cache guidance will factor in target performance, and workload IO mix. The following chart will explain this. Note that this chart assumes 2 disk groups and TOTAL cache for both devices combined. More Disk groups can add more performance.  


Screen Shot 2017-01-18 at 12.10.48 PM

Normal mixed workloads (70/30 Read/Write; Random) – can make do with much less cache (800GB total for highest performance, 200GB for lower performance demands). They provide more time for destaging to catch up with de-stage and garbage collection. In this example, a AF-8 could use 2 x 400GB Write buffer drives, while a smaller AF-6 would need only 2 x 200GB devices.

Mid-range workloads (Greater than 30% write but still random) – Can make do with less cache. 1.2TB for the most performance, and 400GB for lower demands. A slightly larger write buffer helps the SSD keep up with sustained de-stage speeds.  An AF-8 would use a 2 x 600GB Write buffer drives, and a smaller single disk AF-6 would need 2 x 400GB Write buffer devices.

High write throughput workloads (100% sequential writes) – Leverage the most cache. 1.6TB for pushing the most performance, and 600GB for lesser demands. A Larger write buffer helps the SSD keep up with sustained endurance load. An AF-8 would use 2 x 800GB devices on the highest end.

Note these numbers come from performing testing existing vSAN but there was some headroom for future features, functionality and changes built into these recommendations.


I’m an existing vSAN All Flash Customer on 6.2/6.1/6.0.  Am I impacted?  Do I need to change my cache sizing

No, we will support your current sizing of 1:10 of usable capacity

Not an issue since the new cache size requirements are less than or equal to the current requirements

The new guidelines for AF-4, AF-6 and AF-8 match the current guidelines

I’ve deployed or am going to deploy a Hybrid vSAN cluster.  Am I impacted? 

No, the revised sizing guidelines are applicable only to All Flash

Hybrid will continue to use 1:10 caching to usable capacity ratio

Complications associated with dedup/compression/EC sizing does not exist in hybrid

The “write” load issue (associated with All-Flash) does not exist with hybrid as All-Flash drives much higher I/O to caching tier

Will the TCO & Sizing Tool change with the new guidelines?

Yes, we are working on updating the TCO & Sizing tool with the new guidelines based on the premise of max write throughput for All-Flash.

I have a cache drive of size > 1.4TB.  Can I use it for the caching tier in vSAN All-Flash?

Yes, it is fully supported although it may not be economical from a cost perspective.

VSAN cycles through All cells in a cache SSD so any excess capacity (greater than recommended size) is used for endurance and helps in extending the life of the drive.

Does the size of the capacity tier matter when sizing cache for All-Flash vSAN?

No. It does not.  The new cache sizing is based on testing of sustained writes per day at fixed ratio’s that VSAN and has nothing to do with the usable capacity on the capacity tier.

Do deduplication or compression ratios impact sizing of the cache tier?

No. It does not.

What about the 600GB cache limitation for All-Flash?  Does that apply here?

No. The 600GB is an ”internal limit” of how vSAN fills up the write cache buffer.

This should not be confused with the cache sizes recommended in the revised guidelines chart.

Will Ready Node configurations change with new guidelines?

No, the current ready node configurations for standard workloads don’t change as the cache requirements are the same as the new guidelines.  For sequential or write-intensive workloads, use the new formula to size cache.


10 comments have been added so far

  1. Hi John,

    I may have missed something but I did not see the new formula for the sizing calculation. Can you share?

    Thank you,

  2. Hi John, do you know if VSAN Ready Nodes configurations are going to include this new considerations in the short term?

  3. Does this still hold true for vSAN 6.6? Are there any updated design guidelines? Link, ” Designing vSAN Disk groups – All Flash Cache Ratio Update – VMware News and Updates” is password protected, can you share it with me on vikratnr@vmware.com.

Leave a Reply

Your email address will not be published.