SAP HANA on vSphere 8 with 4th Gen Intel® Xeon® Scalable processors (Sapphire Rapids) gained support last year. Initially, deploying SAP HANA as ‘half-socket’ sized VMs wasn’t supported by SAP, limiting deployment to a maximum of 2 SAP HANA production VMs on a 2-socket Sapphire Rapids server.
Despite the excellent performance of vSphere 8 virtualized SAP HANA VMs on Sapphire Rapids CPUs, our testing revealed higher deviations when co-deploying SAP VMs on the same CPU (half-socket VMs). To address this, we utilized Intel sub-NUMA clustering (SNC), a feature available since Intel Skylake CPUs. After successful re-runs of SAP-defined deviation tests—measuring deviations significantly below 5% compared to a specified baseline—with two SAP HANA VMs sharing a CPU socket on an SNC-2 enabled Sapphire Rapids CPU-based vSphere 8 ESXi hosts, it’s now supported to deploy two SAP HANA production VMs per socket on a 2-socket Sapphire Rapids SNC-2 enabled vSphere 8 ESXi host system.
The adoption of SNC-2 in SAP HANA accounts for the increased density of processors, memory controllers, processor interconnects, and supporting infrastructure within a single chip as CPU size decreases. While increasing the number of CPU cores enhances performance, it also increases data transfer times within the CPU chip, affecting memory-sensitive applications like SAP HANA and SNC helps to mitigate these impacts. Please refer to following Intel web page for details on SNC and refer SAP note 3372365 for the SAP SNC-2 support details.
Utilizing SNC-2 with SAP HANA requires no special considerations from the SAP HANA side. SAP HANA automatically detects up to 4 NUMA Nodes on a 2-socket, SNC-2 enabled server and optimizes accordingly for the underlying architecture.
The test summary depicted below illustrates increased fluctuations when running a baseline SAP HANA VM with SAP HANA noise VMs. Regardless of full- or half-socket configurations and SNC-2 utilization, SAP HANA perceives one NUMA node in this setup.
The figures below illustrate supported deployment options for virtual SAP HANA with SNC-2. Options 1 to 4 are supported by both SAP and VMware, while options 5 and 6 are not supported due to suboptimal NUMA node placement of the SAP HANA VMs.
Note: Full-Socket SAP HANA VM support remains available and unchanged with the Sapphire Rapids platform. The sole restriction pertains to half-socket SAP HANA VMs, which necessitate SNC-2 activation. Additionally, SAP confines SNC-2 support to 2-socket SPR hosts exclusively.
Legend:
| Option 1: 2 VMs per NUMA node, 4 VMs in total per ESXi host
|
Option 1 shows 4 SAP HANA VMs running on an SNC-2 enabled 2-socket SPR ESXi host.
Each VM operates on a sub-NUMA node that is fully isolated, providing exclusive access to all available CPU resources. VM size is based on the SAP HANA reference configuration for 2-socket Sapphire Rapids systems with 60 core CPUs with 2 TB per CPU socket:
|
| Option 2: 1 VM per NUMA node, 2 VMs in total per ESXi host
|
Option 2 shows 2 SAP HANA VMs running on an SNC-2 enabled host.
SAP supports expanding an SAP HANA VM across two sub-NUMA nodes to utilize a full CPU socket. This is analogous to when a single CPU socket VM spans two CPU sockets on an older CPU generation. SAP HANA is NUMA-aware and optimizes memory access based on memory latencies. VM size is based on the SAP HANA reference configuration for 2-socket Sapphire Rapids systems with 60 core CPUs with 2 TB per CPU socket:
|
| Option 3: 1 VM across two NUMA nodes / physical CPU sockets
|
Option 3 shows a 4-SNC 2-node wide single SAP HANA VM running on an SNC-2 enabled host.
On an SNC-2 enabled 2-socket SAP HANA host, SAP supports spanning an SAP HANA VM across 4-SNC 2 nodes. In this configuration, SAP HANA detects a ‘4 NUMA node’ server and attempts to optimize memory latency based on NUMA locality. VM size is based on the SAP HANA reference configuration for 2-socket Sapphire Rapids systems with 60 core CPUs with 2 TB per CPU socket:
|
| Option 4: 1 VM across two NUMA nodes / physical CPU sockets
|
Option 4 shows a supported configuration with 3 SAP HANA VMs, two single SNC-2 wide VMs and one VM spanning two SNC nodes.
VM size is based on the SAP HANA reference configuration for 2-socket Sapphire Rapids systems with 60 core CPUs with 2 TB per CPU socket:
|
| Option 5: No support for NUMA node / sockets crossing SNC-2 VMs
|
Not supported for SAP HANA VM deployments.
|
| Option 6: No support for NUMA node / sockets crossing SNC-2 VMs
|
Not supported for SAP HANA VM deployments.
|
Guidelines for deploying a SAP HANA VM on an SNC-2 enabled SPR ESXi host:
- SNC requires that the ESXi host memory be symmetrically populated.
- Enable SNC-2 and hyperthreading in the BIOS of the ESXi host.
- Use SNC-2 for SAP HANA VMs exclusively on 2-socket SPR hosts or later.
- Size the SAP HANA VM according to available logical threads and memory per CPU socket.
- Apply the ‘sched.nodeX.affinity=”Y”‘ VMX advanced parameter to all SNC-2 leveraging VMs to prevent unintended NUMA node migrations.
- Utilize SNC-2 primarily as a consolidation platform for half-socket VMs or SAP HANA VMs requiring low memory latency due to lower memory bandwidth associated with an SNC-NUMA node.
- SAP HANA VMs within a 2-socket SNC-2 host can get extended to up to 4-sub-numa nodes (refer to the previous figure for details).
- Address SNC-2 related performance issues by offline migrating SAP HANA VMs from SNC-2 enabled hosts to non-SNC enabled hosts and adjusting the VM configurations accordingly to the the non-SNC configuration.
- vMotion of SNC-2 leveraging VMs is only supported between SNC-2 enabled hosts. Avoid migrating SNC-2 configured VMs to non-SNC-2 hosts to prevent performance issues. Migration in the reverse direction is also unsupported for the same reason.
- Use vMotion host rules to prevent migration of SNC-2 VMs to non-SNC-2 ESXi hosts.
For more information, deployment, sizing and operation examples refer to the new SAP HANA on VMware vSphere best practices guide (2024 edition).
Summary
The performance boost observed in virtualized systems running on 2-socket Sapphire Rapids with vSphere 8, compared to older CPU generations, is remarkable. Half-Socket support on 2-socket Sapphire Rapids ESXi SNC-2 hosts now significantly enhances SAP HANA VM density and Total Cost of Ownership (TCO) per ESXi host. This enhancement facilitates the deployment of SAP HANA VMs, ranging from smaller half-socket configurations starting at 128 GB up to 1.5 TB (with TDI sizing, otherwise only 1 TB). Furthermore, the scalability from a single sub-NUMA node SAP HANA VM (half-socket) to a four sub-NUMA node VM, supporting up to 240 vCPUs and 6 TB of RAM (TDI sizing), accommodates a wide spectrum of SAP HANA system sizes and deployments.
For larger SAP HANA deployments, 4-socket Sapphire Rapids systems are now also supported by SAP with vSphere 8. Refer to the provided blog for further details.
