Why vSAN ESA Only Supports NVMe Solid State Storage Devices

Why can’t spinning disks be used with vSAN ESA? The recent hardware shortages with servers and storage have encouraged customers to look at every conceivable way to lower costs, and this question occasionally comes up in that effort. Some customers contend that their application demands simply aren’t very high, so it seems entirely plausible that spinning disks could be good enough in vSAN ESA. To be clear, spinning disks will not work with, and are not supported in vSAN ESA. This post aims to help you understand why.

The idea of using spinning disks comes from distant memories of their use in enterprise storage systems. These systems, such as vSAN OSA hybrid configurations, and hybrid storage arrays would typically use higher performance NAND flash to buffer incoming writes in order to be written to the spinning disk. It was a novel approach that reflected the hardware of the time, and software that was designed for that type of architecture. While it had its limits, it seemed like a reasonable approach at that time, so why isn’t it a good approach now?

The Problem with Legacy Media, Bus Protocols, and Architectures

Since the early 2000s, spinning media used either SATA or SAS interfaces. SATA used a serial ATA command set to pass storage commands to a device, while SAS used SCSI. But both were designed assuming the characteristics and limitations of these devices, such as the magnetic platters, the actuator assembly, and the logic board.

While SATA was generally intended for consumer applications, and SAS was found more often in the enterprise, the storage industry was in a race to the bottom in attempts to win over market share. New storage offerings in the early 2010s put pressure on storage vendors to drive down costs of their storage solutions for rapidly growing virtualized environments.

But there were unavoidable limits that were difficult to ignore.

Physics

Spinning disk always has been, and will always be bound by physical limits of a platter spinning at thousands of revolutions per second and the magnetic heads that were responsible for reading and writing the data. Whether they were 7,200 RPM SATA drives, or SAS drives that spun at 10,000 or 15,000 RPMs, rotational latencies prevented any substantial improvement in performance. Storage solutions would stripe data across multiple disks and employ other tricks to work around the limits of spinning media, but it was a lot of effort for only incremental levels of improvement.

Protocols

At the heart of any storage device is an interface specification paired with its storage protocol, sometimes referred to as a bus protocol. But SATA and SAS have inherent problems. Since these protocols were intended for rotational media, they had command sets that were complex and consumed an unnecessary amount of CPU cycles to process a read or write command.

SATA is a half-duplex interface, meaning that communication can only occur in one direction at a time. It had a terribly low threshold of pending I/O commands that it could retain: 1 queue of a maximum of 32 commands, known as a “queue depth” of 32. SAS is full-duplex, and had an incrementally larger queue depth of 256, but suffered from the same fundamental issues as SATA devices. These limits became more of an issue as spinning disks increased in capacity – which were prone to higher demands from an environment. But the real problems started to appear when SAS and SATA were used with NAND flash, since many of the physical limits of spinning disk no longer existed. The inferior characteristics of SAS and SATA were hampering the native abilities of NAND flash.

Architecture

SAS and SATA devices attach to a physical host bus adapter (HBA). Sometimes referred to as a storage controller, this HBA would pass I/O commands and payload through this controller to the respective storage device. In a shared storage environment, this external controller may be responsible for several storage devices — funneling all I/O activity through a single device. Not only was it a common bottleneck for already slow storage devices, but it often masked other issues.

This bottleneck became especially painful when solid-state drives (NAND flash SSDs) entered the scene. The functional performance of NAND flash was often bound by the bus protocol, the HBA, or a combination of both. This left some people underwhelmed with their initial experience of NAND flash. These inherent limitations are one of the reasons why vSAN ESA does not support SAS or SATA-based flash devices.

Enter NVMe Paired with Modern NAND Flash

NVMe was the interface specification built specifically for solid state storage that completely changed the capabilities of storage devices. By the early 2010s, it was apparent that the days of spinning disks were numbered, but also that there had been enough practical experience with SATA and SAS on flash that clearly pointed to the interface as the next biggest bottleneck.

The NVMe specification aimed to resolve the challenges commonly associated with spinning disk and flash devices using SAS or SATA interfaces.

• NVMe devices plug directly into a PCIe Bus. This allows for massive amounts of storage payload to be transferred. A PCIe Gen 5 device using 4 lanes can transmit nearly 16GB/s throughput.
• Each NVMe device has its own controller built onto the device. NVMe does not use a shared HBA or controller. This eliminates the funneling/bottleneck effect of a shared controller.
• Multiple queues using large queue depths. Instead of just having a single queue, the controllers built into each NVMe device can have up to 64,000 queues, each with a queue depth of up to 64,000 commands.
• A simplified command set is built for flash. This lean command set eliminates much of the technical debt of SAS and SATA command sets, which reduces resource consumption.
• NVMe assumes the use of solid state storage. NVMe is most commonly used with NAND flash but was also found with other technologies like 3D XPoint. Most importantly, it helps eliminate the physical limits of spinning disk.

The differences noted above allow NVMe to deliver extraordinary performance with tremendous efficiency. NVMe only takes about one-third the CPU overhead per I/O when compared to SATA, and about one-third to one-half the CPU overhead per I/O when compared to SAS. And since it also eliminates the funneling effect with traditional SATA and SAS devices, it can provide tremendous parallelism that not only delivers high levels of performance, but can ensure that performance remains consistent at nearly all levels of utilization.

Note: vSAN ESA only supports NVMe devices plugged directly into the PCIe bus. vSAN ESA does not support the use of Tri-Mode controllers.

Performance and Data Availability Comparison

Given the limits of the media and the bus protocols, when we compare NVMe to spinning disk, rotational media can be up to 10,000 times slower on random throughput, over 10,000 times slower on random IOPS, and up to 2,000x slower as measured by latency. It is these kinds of performance differences paired with the efficiencies that make storage devices using NVMe the only choice for vSAN ESA.

And remember that performance is not just about ensuring your workloads operate at the levels you expect, but it is also about data availability. vSAN distributes data across hosts to make it highly available. Its self-healing capabilities rely on the timely ability to evacuate data or rebuild data on other hosts in the event of a planned or unplanned event. Having just 1/2000th to 1/10000th the storage device performance to perform these operations would be like cutting your network performance by the same proportion. The impact on data availability would be substantial, and even a simple evacuation of a host could take days instead of minutes.

Cost Comparison

The reasons why some speculate about the use of spinning disks in vSAN ESA is their cost and availability. But therein lies the truth worth paying attention to the most. Spinning disks remain relatively cheap and readily available because they are so inferior that they cannot be used in meaningful ways for production workloads in the data center. In other words, nobody wants them, and the market prices reflect that.

Spinning disks paired with legacy interfaces were so deficient that the vSAN Original Storage Architecture (OSA) could not support advanced space efficiency features like erasure coding, deduplication and compression. This is important to remember because it changes the financial equation when comparing the cost per TB of physical space of a spinning disk versus flash.

For a very simplistic cost comparison of a cost per GB or TB, let’s suppose the street price of NVMe storage is at an exorbitant amount of 8x the street price for spinning disk. Let’s see how this compares.

• Spinning disk: For every unit of data stored using a spinning disk, it would take 3x the amount of capacity for resilience. This is because it must use simple data mirroring to provide resilience as opposed to space efficient erasure coding. It would also not be able to use in-line data compression, or post processed global deduplication.
• NVMe storage: For every unit of data stored using NVMe based flash, it takes just 1.5x the amount of capacity for resilience. Now let’s assume a very modest data reduction ratio of 4:1 after data compression and deduplication, and that unit of data stored resiliently is just 0.375 the size of its native size.

	Street price per unit of capacity	Unit of data stored	Unit of storage consumed after resilience	Unit of storage consumed after data reduction	Effective cost per unit of data stored resiliently
Spinning disk	$1	1	3	3	$3
NVMe storage	$8	1	1.5	0.375 (4:1)	$3

While the example above ends up being about the same price per unit of storage capacity purchased, the performance of NVMe storage can be thousands of times faster and more consistent than spinning disk. Not only does this provide the performance you’d expect for your guest workloads, it can help vSAN ensure the data remains resilient during planned and unplanned events.

This is most certainly an over simplistic theoretical comparison. Anything built using spinning disks would need to have flash devices purchased for the sole purpose of caching, and there would be other costs to these configurations, including additional storage controllers. One would be so physically constrained by these old devices that they may require a larger number of them which reduces physical space and power efficiency.

Even if it were possible, would the extreme step of investing in antiquated hardware with dramatically inferior capabilities feel like a good long-term decision when NAND shortages are alleviated? That type of decision would certainly have repercussions at a later time.

Reducing Storage Costs the Right Way

vSAN ESA was designed specifically for use with NVMe storage devices. While it does not support physical media like spinning disk which used legacy bus protocols like SATA and SAS, using NVMe storage devices allows the architecture to give you the ability to store more data with less physical hardware.

• No external storage controllers to create an I/O funneling effect, and reducing costs
• All devices used for vSAN ESA contribute to capacity (no dedicated caching devices)
• Optimized data compression
• Supreme global deduplication
• Optimal Erasure Coding
• Fewer hosts, devices, networking and power requirements to achieve performance objectives

And remember, vSAN ESA supports lower cost Read-Intensive NVMe devices, and high density, low cost QLC storage devices for some use cases.

But wait, you may ask yourself, why not just use spinning media with vSAN OSA? While support of vSAN OSA on all-flash configurations continues, the deprecation of hybrid clusters in vSAN OSA was announced in the VCF 9.0 release notes.

Summary

While the intentions are most certainly good, spinning disks for a modern enterprise storage system are the wrong way to solve the problem of hardware scarcity and its associated cost. They are simply unable to provide the capabilities necessary to support a modern storage system running your workloads.

@vmpete