25Gb Networking vs 100Gb Networking with VMware vSAN

Over the past four years, the VMware vSAN Express Storage Architecture (ESA) has demonstrated extraordinary performance capabilities for our customers. But, as noted in a series of blog posts, a distributed storage system like vSAN relies not just on the hardware of the hosts, but the network fabric that connects them. The post: “What to Look for in Network Switches for VMware vSAN” details many of the attributes of network switches that matter to vSAN. But how much of an impact can some of these characteristics have on a modern vSAN cluster?

One customer recently stress tested their environment to evaluate how different network switches impacted performance, and were kind enough to share those results with us. A large percentage of environments running vSAN ESA use 25GbE Top of Rack (ToR) switches throughout their environment with great success, as 25GbE switches offer a nice blend of high performance at a good value. This customer wanted to see how 100GbE switches with larger port buffers compared their 25GbE switches.

The result? For this customer’s environment a 100GbE switch with ultra-deep port buffers (8GB) yielded 23x better performance than a 25GbE switch with smaller port buffers (4GB) when performing synthetic testing. This is an astonishing difference. Let’s look at the testing and the results more closely to understand why this difference was so significant, and how it can help you maximize vSAN’s potential in your own environment.

Motives and Test Environment

The customer’s inspiration for comparing network switchgear with vSAN came when performing synthetic tests on a newly deployed vSAN ESA cluster. Their 25Gb network switches were demonstrating behavior that indicated they were unable to cope with vSAN’s ability to push extraordinary amounts of I/O through the links connected to hosts, especially high throughput test cases.

The objective for this customer was clear. They wanted to stress test the cluster and its network to determine not only what was theoretically possible for this specific cluster, but how much the network switchgear had played a role in the result. They were not trying to emulate VM I/O characteristics or load levels from production environments.

Test Environment

The environment and approach to testing consisted of the following:

• vSAN version: vSAN 8 U3 (ESA).
• Cluster host count: 4 hosts.
• Cluster data services: vSAN Data-at-Rest Encryption enabled.
• Storage devices in hosts: Qty 5, 7.68TB NVMe storage devices.
• Host NICs: 100Gb, auto-sensing to 25Gb or 100Gb based on link speed.
• Storage policy: RAID-5 erasure coding.
• Synthetic I/O generator: HCI Bench running multiple tests, where each test used different I/O sizes ranging from 4K to 256K. All tests used a 50/50 read/write ratio and ran for a duration of 30 minutes.

To produce a proper comparison, the synthetic tests ran on the cluster connected to a 25Gb switch with 4GB port buffers, and repeated the tests on a 100Gb switch with 8GB port buffers. The test method, the host hardware, and the software configuration all remained the same during the comparison.

Test Results

The observations below highlight the relative gain when running a 100Gb switch with 8GB port buffers versus a 25Gb switch with 4GB port buffers using the customer’s specific hardware and configuration. It should not suggest you will see identical results in your own environment. The results below reflect the testing performed using larger I/O sizes, since that is typically the best way to determine the limitations of a switch.

Figure. Test results comparing switches with different bandwidth and port buffer sizes.

• IOPS: 23x improvement.
• Throughput: 23x improvement.
• Latency: 96% reduction.
• Network and buffer congestion: Pause frames went from significant when using the lower performing switch to zero when using the higher performing switch.

Why is there such a significant (23x) performance difference between the 25Gb switch and the 100Gb switch? While the 100Gb switch provides about 4x the theoretical bandwidth across a single link compared to the 25Gb switch, the larger port buffers allow microbursts of incoming traffic to be absorbed in the buffer without inducing pause frames, which is an indicator of congestion. As noted in “What to Look for in Network Switches for VMware vSAN” congestion control mechanisms of a network can drastically restrict practical performance from the theoretical maximum link speeds. This is precisely what was occurring with the 25Gb switch with smaller port buffers initially used with the cluster.

Another noteworthy observation when performing these tests is that when using a 100Gb switch with ultra-deep buffers, the link utilization never exceeded 85%. This is a good indication that at no time was the network the bottleneck for these specific hosts, and offered sufficient headroom to ensure stable and consistent performance.

Recommendation: Oversize your network with higher throughput, ultradeep buffer switches so that you rely on the sophisticated and efficient schedulers found in vSphere and vSAN to share resources, instead of the primitive congestion control mechanisms found with network communication.

But not all switches are created equally. Some switches target different use cases, and may not have the port buffer sizes or processing power appropriate for high throughput conditions. Sometimes it can be difficult to compare specifications across switch manufacturers even though a specification like port buffer depth may be identical. This is because different architectures are used to achieve an end result. For more information, see the post: “What to Look for in Network Switches for VMware vSAN.”

What Faster Network Switching Can Mean to You

Does this mean this environment will see a 23x improvement in VM performance for real workloads? Likely not, as these synthetic tests were intended to only serve as a stress test, and do not reflect the characteristics of real workloads. You may have vSAN ESA clusters using 25Gb switches that deliver very good performance and easily satisfy the demand of the workloads on the cluster. However, the practical benefits that higher performance network switches can deliver may be better than you realize.

• Faster Repair times in the event of a host outage. During planned or unplanned host maintenance, vSAN can resynchronize data much faster, and allow it to regain the prescribed level of resilience.
• Better performance for guest VMs. Guest VM performance will likely improve, especially under bursts of concurrent activity across several VMs, as well as high throughput conditions, such as large sequential I/O activity.
• Improved performance consistency for guest VMs. An often overlooked aspect of performance monitoring, this faster networking will reduce the deviation in performance when workloads are at their peak. The network will be contending less often, which will improve the consistency of the cluster.

While the performance benefits are obvious, the ability for a vSAN cluster to self-heal more quickly shouldn’t be overlooked. It is an easy way to make vSAN even more robust for planned and unplanned maintenance events.

100Gb networking can be more expensive than its 25Gb counterpart, but there are subtleties that bring them much closer in price than you think. With 100Gb networking, the effective cost is reduced due to a smaller hardware footprint.

• Fewer links. Hosts with 100Gb NICs often need fewer uplinks to serve the needs of the host. Fewer NICs also means a reduced number of ports used with the ToR switches.
• Potentially fewer hosts. Higher performance capabilities per host may also mean fewer hosts. This will drastically reduce your overall hardware footprint, power and cooling, software licensing, and associated network costs.

In other words, less hardware equals a simpler configuration and more savings for you.

Recommendation: If you are currently using 25Gb network switches, evaluate the possibility of specifying 100Gb NICs for your hosts in your next server hardware refresh. Those NICs will automatically sense the speed of the switches, and will set you up for immediate performance improvements when the environment is upgraded to faster switches.

This post is a part of a multi-part series on network topics related to vSAN, which can be found in the links below:

• vSAN Networking – Network Topologies
• vSAN Networking – Network Oversubscription
• vSAN Networking – Optimal Placement of Hosts in Racks
• vSAN Networking – Teaming for Performance
• vSAN Networking – Teaming for Redundancy
• vSAN Networking – Is RDMA Right for You?
• What to Look for in Network Switches for VMware vSAN
• 25Gb Networking vs 100Gb Networking with VMware vSAN

Summary

vSAN supports a variety of network speeds to provide maximum flexibility. While 25Gb networking with vSAN can be a great option for many environments, a customer’s test results show how 100Gb networking with ultra-deep port buffers can unlock the full potential of vSAN in a resource intensive environment.

– @vmpete