Accelerated Data Plane Performance Using Enhanced Data Path in NUMA Architecture

Authors: Jambi Ganbar (Sr. Technical Solutions Manager, NFV), Jubin Thomas (NSX Information Experience team)

Overview

Some workloads demand accelerated and predictable networking performance. Our Network Functions Virtualization (NFV) customers and some of our financial, media, and high-performance computing (HPC) customers deploy these workloads. These workloads process a lot of network traffic. Network traffic in the virtual domain relies heavily on CPU cycles and the number of CPU cores available on the host. These CPU resources are used by the workload to perform its task and by the hypervisor layer to deliver network traffic to and from the application.

In this blog, we discuss the configuration required to achieve accelerated data plane performance in modern multiple NUMA architecture hosts. This blog accompanies a new white paper we just released on the subject. The white paper can be found here.

With the introduction of NSX-T version 2.2, we added a new mode of operations in the NSX-controlled virtual distributed switch. We refer to this switch as N-VDS. This new mode is called Enhanced Data Path and is often indicated as N-VDS (E). N-VDS (E) is one of the core building blocks in achieving accelerated data plane performance.

When you configure your VM and the N-VDS (E) in the way we describe below, you achieve vertical NUMA alignment. This term refers to ensuring that your memory and CPU resources in the physical domain are aligned to the resources in the virtual domain. This alignment helps to provide accelerated networking performance by maintaining NUMA boundaries.

To achieve accelerated data plane performance, configure the following components in N-VDS (E):

First, assign logical cores to N-VDS (E).
Second, ensure that VMs associated to this N-VDS (E) are configured with Latency Sensitivity set to High. These VMs also need to fulfill their prerequisites in order for the Latency Sensitivity to take effect. N-VDS (E) will try to align VMs to logical cores on the same NUMA node.
Third, select Load Balanced Source teaming policy, which is NUMA aware, to align the VM to physical NIC on the same NUMA node.

To understand the mechanics behind the process, continue reading.

NOTE: You can leverage N-VDS (E) capabilities in NSX-T running on vSphere 6.7 hosts or later versions. You also require specific N-VDS Enhanced Data Path NIC drivers.

Know your Host

To verify the NUMA configuration of the host, such as the total number of sockets, logical cores available, and other details run the following command: # exscli ens memory get
N-VDS in Enhanced Data Path mode requires a specific poll mode driver. To identify NICs that support Enhanced Data Path, see the VMware Compatibility Guide. The list of supported drivers will grow over time.
Today’s servers are architected with multiple NUMA nodes. It is important to plan the number of logical cores to be configured on these NUMA nodes. The number of logical cores is important when configuring NSX-T Transport Node in Enhanced Data Path mode as they are assigned during this phase. You can distribute the total number of logical cores and NUMA nodes among different N-VDS switches. A host can have more than one N-VDS switch.
Install drivers to enable NIC cards to run Enhanced Data Path mode. You can download and install the drivers from My VMware page. Important to note that while loading the Enhanced Data Path drivers, the existing drivers need to be unloaded first.
Use a VMXNET3 virtual network adapter for your VMs vNICs.

Configure logical cores on N-VDS switches

When you configure N-VDS in the Enhanced Data Path mode, you can control the number of logical cores that can be used by the N-VDS (E) on a specific NUMA node. The number of logical cores per NUMA node should be based on the traffic type and the NIC throughput capacity.

This provides for clear resource separation between the infrastructure and the VNF.

Ensure vCPUs of a VMs remain NUMA local

To avoid having the vCPUs of your data plane VM be assigned from a different NUMA node, configure the VMs with Latency Sensitivity = High.

N-VDS (E) only associates low-latency VMs with specific CPU cores on the same NUMA node. These cores provide designated compute resources, which enable faster packet processing.

NOTE: Even if you set latency sensitivity to high, it is not a satisfactory condition. You can come across a scenario where the host is completely loaded so you do not have enough free cores in a single NUMA node to power on the VM. In this case, the local scheduler will straddle the VM vCPUs across both NUMA modes. In this case, the VM works under suboptimal conditions.

Align VMs to Physical NICs on the same NUMA node

To align VMs to physical NICs on the same NUMA node, do the following:

Select the load balanced source teaming policy in the uplink profile that is applied to the transport node. This teaming policy makes the best effort to align VM and physical NIC on the same NUMA node.

This alignment means that the physical NICs and logical cores are aligned on the same NUMA node. Now, the VM, logical cores and physical NICs are in complete vertical NUMA alignment. NUMA

Once you configure the above components, logical cores, VM resources, and physical NICs are aligned on the same NUMA node. It is important to vertically align logical cores, VM resources, and physical NICs because only then can you avoid crossing the QPI bus which is a high-latency operation. costly operation. If anyone of the components are not vertically aligned, then the network performance suffers.

In the next section, we explore how traffic flows over NUMA aware Enhanced Data Path.

Traffic flow over NUMA aware Enhanced Data Path

The following procedure is a step-by-step flow of packets sent from VM 2 on ESXi 1 host to VM 3 on ESXi 2 host.

Assuming that the VM 2 and VM 3 are set to High Latency Sensitivity, the following steps describes the traffic path for the packet from VM 2 to VM 3.

With VM 2 set to High Latency Sensitivity, let’s assume that the Enhanced Data Path load balancer assigns logical core 2 to VM 2.
When VM 2 sends a packet to VM 3, it first goes to the virtual NIC on VM 2. From there, it is then sent to the hypervisor of ESXi 1 host.
Logical core 2 receives packet from virtual NIC of VM 2.
The Load Balanced Source teaming policy which is NUMA aware, associates a physical NIC to the VM on the same NUMA node.
In this case, the teaming policy associates VM2 to LAG A. And logical core 2 delivers packet to LAG A.
One of the NICs in LAG A delivers the packet to NIC 4.
NIC 4 then delivers the packet to the virtual NIC of VM 3.
On NUMA node 3, the logical core 3 continuously polls NIC 4 for any packets that need to be delivered to VM 3.
When it receives the packet, logical core 3 delivers the packets to the virtual NIC of VM 3.

Traffic flow when NUMA awareness is misplaced

NUMA awareness is not counted for by the teaming policy if:

– A VM has affinity with more than one NUMA node

– A LAG profile has NIC cards associated with multiple NUMA nodes

Let’s see how traffic flows in such conditions.

Let’s consider traffic flow from VM 2 to VM 4:

VM 2 has NUMA affinity with NUMA 1 and NUMA 2.
When VM 2 sends packets to reach VM4, the teaming policy does not know which NUMA affinity to respect. So, it can arbitrarily select NIC 1 or LAG A as the physical NIC to transfer the packet.
Depending upon the NIC availability, packets from VM 2 are switched to VM 4.

Let us consider traffic flow from VM 1 to VM 3:

VM1 has NUMA affinity with only NUMA 1.
Since VM1 is set for High Latency Sensitivity, the Enhanced Data Path load balancer associated VM 1 to logical core 1.
The Load Balanced Source teaming policy makes the best effort to bind VM 1 to NIC 1 and packets are sent from NIC 1 to the destination VM 4.

Monitor Performance

To monitor network performance, use the esxtop utility. See, Using the esxtop utility.

Summary

The N-VDS switch in Enhanced Data Path mode can be configured to align VMs to logical cores and the Load Balanced Source teaming policy in NSX-T 2.3 aligns VMs to NICs to improve the network performance.