In the past series of posts, we’ve seen how performance of vSAN comes from not only the physical bandwidth capabilities of the network used to connect vSAN hosts, but also by its design. When using vSAN ESA, higher bandwidth networks paired with efficient network designs can help workloads fully exploit today’s server hardware. In the spirit of providing the best network possible for your vSAN environment, you may wonder if there is anything else that can be done with the network to improve the performance of vSAN. This post will discuss vSAN over RDMA, and whether it is right for you and your environment.
The concepts described here build off the information found in the posts: “vSAN Networking – Network Topologies,” “vSAN Networking – Network Oversubscription.” “vSAN Networking – Optimal Placement of Hosts in Racks,” “vSAN Networking – Teaming for Performance” and “vSAN Networking – Teaming for Redundancy.”
vSAN over RDMA Overview
vSAN uses IP-based Ethernet networks for communicating between hosts. Ethernet frames (Layer 2) are the logical substrate that deliver TCP-based communication to and from the hosts with their respective payload. vSAN data payload sits inside of these packets in the same manner as other data payload types. For many years, TCP over Ethernet has provided an extraordinarily reliable, and durable method of network communication for a wide variety of traffic types. Its durability is unprecedented, as it can withstand extremely poor designs and conditions, yet still communicate successfully.
But that extraordinary flexibility and durability has tradeoffs. The additional layers of logic used to acknowledge packets, retry dropped packets and account for all types of poor connectivity conditions does introduce resource overhead and variability in packet delivery when compared to a lossless, deterministic protocol like Fibre Channel. While this can reduce throughput and increase latency, it is generally most perceptible in poorly designed environments, and typically insignificant in an environment designed correctly.
To help offset some of the characteristics of TCP-based networking over Ethernet, one can use vSAN over RDMA over Converged Ethernet (specifically, RoCE v2). This is a technology that still uses Ethernet, but strips out some of the unnecessary complexities of TCP, offloads communication activities from the CPU to the hardware, and provides direct memory access for processes. The simpler stack frees up CPU cycles for guest workloads, and lowers latency across the wire. For vSAN, not only does it improve top-line performance, but it improves the consistency of that performance.
Figure 1. Using vSAN with RDMA over Converged Ethernet (RoCE v2) in vSAN.
RDMA can be enabled on a vSAN cluster by enabling the capability within the cluster configuration in the vSphere Client UI. This assumes all of the prerequisite steps have been taken to ensure your host NICs and switches are prepared for using RDMA. See your NIC and switch vendor’s documentation on the necessary steps to enable RDMA across these devices.
If there is any single issue with the RDMA configuration, such as a host in a cluster that is no longer able to communicate via RDMA, the entire cluster will automatically fail back to TCP over Ethernet.
Recommendation. Only consider RDMA if you are using vSAN ESA. While support of vSAN over RDMA has existed since vSAN 7 U2, it is a technology that will provide the most benefit to the high-performance capabilities of ESA, found in vSAN 8 and newer.
As noted in “Designing a vSAN Network,” the use of vSAN over RDMA does introduce additional requirements, limitations and considerations. This includes but is not limited to:
- vSAN ReadyNodes must use NICs certified for use with RDMA.
- Switches compatible with, and configured for RDMA, including settings such as data center bridging (DCB) and priority flow control (PFC).
- Cluster size of no greater than 32 hosts.
- Must use one of the following teaming policies: “Route based on originating virtual port,” or “Route based on source MAC hash.” The use of LACP or IP Hash is not supported with RDMA.
- Preferable to use dedicated NIC ports for RDMA instead of mixing TCP and RDMA on the same uplink.
- Not compatible with 2-Node clusters, stretched clusters, vSAN datastore sharing, and vSAN storage clusters.
- In VCF 5.2, vSAN over RDMA is not supported. It is not integrated into the SDDC Manager workflows, and does not provide any way to configure RDMA for vSAN powered clusters. Any subsequent attempts at configuring RDMA with a vSAN cluster using vCenter Server is not supported in VCF 5.2.
For more guidance on installing RDMA for use with vSAN, see KB 382163: Configuring RDMA for vSAN.
Performance Improvements that Come from vSAN over RDMA
Assuming the comparison of two clusters using the same hardware, vSAN over RDMA can deliver better performance than vSAN using TCP over Ethernet. In the post by Intel: “Make the Move to 100GbE with RDMA on VMware vSAN with 4th Gen Intel Xeon Scalable Processors” they found impressive improvement depending on the conditions.
Recommendation: Use RDTBench as a way to test RDMA and TCP connections between hosts. It can also serve as a great way to validate a working configuration prior to deploying a high-performance cluster into production.
Is Fibre Channel Really a Gold Standard?
There is a reason Fibre Channel has typically been held in high regard by storage administrators. The Fibre Channel protocol is purpose-built to do one thing very well: Transport storage traffic. It has a thinner stack built specifically for the task of providing a consistent, low latency transport for storage. Deterministic networking using Fibre Channel works more like a single unit, where every component is predetermined and works in unison.
But Fibre Channel and other protocols that assume lossless networks also have their own cost that shouldn’t go unnoticed. Due to its significant expense, any Fibre Channel related purchases tend to dominate the budget and take away from other networking investments. This creates a pervasive sunk cost that can unintentionally influence future decisions. Fibre Channel fabrics lack the flexibility provided by Ethernet networks to extend across a variety of network topologies.
While Fibre Channel was intended to run over a lossless physical transport, unintended consequences can occur when there are disruptions in a fabric. Forward Error Correction (FEC) was added to the 32GFC specification to combat small transient disruptions, but any fabric that grows in scale can grow in complexity, making lossless transports more challenging.
Fibre Channel’s advantage is not about absolute speed, but rather the predictably that it can deliver data from point to point. You can see in the comparison below that even when accounting for a roughly 10% overhead when transmitting vSAN traffic using TCP over Ethernet, that commodity Ethernet standards can easily meet or exceed the throughput capabilities of Fibre Channel.
Figure 2. Comparing Fibre Channel speeds to vSAN ESA using TCP over Ethernet.
Note that references such as “32GFC” and 25GbE Ethernet represent branded names, and not the precise effective throughput across the wire. Each standard has an overclocked baud rate to account for the respective overheads. With Ethernet, the effective throughput will depend on the type of traffic transported. 40GbE is not shown, as it has been largely obsolete since 2017.
Meanwhile, Ethernet has shown a resurgence, spurred on by AI infrastructures that need high levels of performance without the fragility of a traditional lossless network. Ethernet was designed to accommodate the practical realities of the data center, where changes in environmental conditions and hardware failures are inevitable.
The commodity pricing of 100GbE, and the availability of 400GbE hardware (with 800GbE around the corner) make Ethernet extremely attractive. Even traditional storage vendors have recently stated that more and more customers heavily invested in Fibre Channel are now looking at Ethernet as their next storage fabric. The announcement of Broadcom’s Tomahawk 6 chip that delivers 102.4 Terabits/sec within a single chip is a good indicator that the future of high-performance is with Ethernet. With vSAN ESA, most of the perceived performance costs of TCP over Ethernet can be offset by a good design that does not consist of any network oversubscription, and native high-bandwidth capabilities of the hardware. This is evident in the post: “vSAN ESA Beats Performance of Top Storage Array for Large Financial Firm” where vSAN ESA, using TCP over Ethernet easily outperformed a top storage array that used Fibre Channel.
Is TCP over Ethernet Good Enough?
For those who have a good network design using higher bandwidth networking and no network oversubscription, vSAN using TCP over Ethernet will be good enough for most environments, and is the best starting point for new vSAN clusters. This recommendation aligns well for customers running vSAN in a VCF 5.2 environment, given its current lack of support for vSAN over RDMA. vSAN over RDMA may drive higher levels of performance, but the requirements and limitations of RDMA may not make it the best fit for your environment.
You can, however, make vSAN’s storage fabric provide performance and consistency more like a deterministic Fibre Channel fabric. This would include:
- Good network design. A good design of an Ethernet network will deliver much higher throughput and lower latency. A proper non-blocking spine-leaf topology that delivers line-rate bandwidth from host to host, without any oversubscription will reduce dropped packets and other delays that hinder performance. Proper placement of vSAN hosts that comprise a cluster will also improve networking efficiency and performance.
- Higher bandwidth. Retire those old switches! They have no place in your data center anymore. The use of higher bandwidth switches and NICs ensure that your workloads can push read and write commands and data payload freely without contention. The key for consistent delivery in an Ethernet network is to prevent Ethernet or TCP from running into situations where it must retry the delivery of frames or packets due to insufficient or unreliable resources.
- NIC and switch tuning. NICs and switches often have default configurations that are not necessarily optimized for performance. This might be an option if you want to explore improving performance without enabling RDMA, and after you have addressed the first two recommendations. The Performance Best Practices for VMware vSphere 8.0 U1 paper shows some examples of optional tuning.
Further information on vSAN networking may be found with the vSAN Network Design Guide. For VCF environments, see “Network Design for vSAN for VMware Cloud Foundation.”
Summary
As the performance capabilities of vSAN become more powerful, it becomes that much more reliant on the network to deliver high levels of performance, consistently. Even though vSAN over RDMA can deliver better performance when there is contention in the protocol stack, or the CPU, a high-performing vSAN cluster can certainly be built using traditional TCP over Ethernet.
