Leaving CPU Hot Add at its default setting of “off” is one of the performance best practices that we have for large VMs. From the Performance Best Practices Guide for vSphere 6.7 U2:
CPU Hot Add is a feature that allows the addition of vCPUs to a running virtual machine. Turning on this feature, however, turns off vNUMA for that virtual machine, resulting in the guest OS seeing a single vNUMA node. Without vNUMA support, the guest OS has no knowledge of the CPU and memory virtual topology of the ESXi host. This in turn could result in the guest OS making sub-optimal scheduling decisions, leading to reduced performance for applications running in large virtual machines. For this reason, enable CPU Hot Add only if you expect to use it. Alternatively, plan to power down the virtual machine before adding vCPUs, or configure the virtual machine with the maximum number of vCPUs that might be needed by the workload. If choosing the latter option, note that unused vCPUs incur a small amount of unnecessary overhead. Unused vCPUs could also cause the guest OS to make poor scheduling decisions within the virtual machine, again with the potential for reduced performance. For additional information see VMware KB article 2040375.
If you turn on CPU Hot Add, virtual NUMA is turned off. This means that the VM is not aware of which of its vCPUs are on the same NUMA node and might increase remote memory access. This removes the ability for the guest OS and applications to optimize based on NUMA and results in a possible reduction in performance.
Virtual NUMA (vNUMA) exposes NUMA topology to the guest operating system, allowing NUMA-aware guest operating systems and applications to make the most efficient use of the underlying hardware’s NUMA architecture. (For more information about NUMA, see page 27 in the Performance Best Practices Guide for vSphere 6.7 U2.)
To get an idea of what the performance impact can be by enabling CPU Hot Add, a simple test was run in our lab environment. This test found performance with the default setting of CPU Hot Add turned off performed from 2% to 8% better than when CPU Hot Add was turned on.
Test Details
VMware ESXi 6.7 Update 2 was installed on a four-socket Intel Xeon E7-4890 v2 (Ivy Bridge) server with 15 cores per socket and 1 TB of RAM. A virtual machine was created with 28 vCPUs and 512 GB which spanned span half of the host’s sockets. The VM was installed with Windows Server 2019 and SQL Server 2019. The VM had 4 paravirtual SCSI (PVSCSI) adapters with two virtual disks for data and two virtual disks for logs. All virtual disks were hosted on a flash-based Fibre Channel array.
DVD Store 3 (DS3)—an open source, transactional database workload that simulates an online store—was the test workload. The DS3 test database created was approximately 300 GB in size on disk. A separate VM with 20 vCPUs and 32 GB of RAM was used to run the DS3 driver program that simulates all the online store users’ activities and creates load on the SQL Server system.
One set of tests was run with CPU Hot Add turned off, which is the default setting, and one set of tests were run with CPU Hot Add turned on. This is a simple check box in the VM settings as shown below.
Quick note about Hot Plug vs Hot Add terminology: As seen in the screen shot above, this feature is sometimes referred to CPU Hot Plug and other times as CPU Hot Add. Both terms refer to the same thing and can be considered interchangeable. I personally like CPU Hot Add, so that will be used throughout this post.
When run with the default setting of CPU Hot Add turned off, the 28-vCPU VM has two NUMA nodes, as seen in the SQL Server Management Studio when looking at the Processor section of settings for the SQL Server database.
But when CPU Hot Add is turned on, it is seen as a single NUMA node as shown in the same SQL Server Management Studio screen.
In each set of test runs, an increasing amount of load was run against the database. This was accomplished by increasing the number of threads in the driver program until the maximum amount of throughput measured in terms of orders per minute (OPM) was achieved. Each test was run 3 times, and the throughput was averaged.
The default setting of CPU Hot Add turned off outperformed CPU Hot Add turned on by 2 to 8 percent across the load levels tested with this test workload.
This shows that turning on CPU Hot Add, which causes virtual NUMA to turn off, leads to a performance decrease. The performance impact will vary based on many factors, but it is expected that, generally, with VMs that span more than one NUMA node, performance will be better with the default setting of CPU Hot Add turned off.