Leaving CPU Hot Add at its default setting of disabled is one of the performance best practices that we have for large VMs. From the Performance Best Practices Guide for vSphere 6.7 U2:

CPU Hot Add is a feature that allows the addition of vCPUs to a running virtual machine. Enabling this feature, however, disables vNUMA for that virtual machine, resulting in the guest OS seeing a single vNUMA node. Without vNUMA support, the guest OS has no knowledge of the CPU and memory virtual topology of the ESXi host. This in turn could result in the guest OS making sub-optimal scheduling decisions, leading to reduced performance for applications running in large virtual machines. For this reason, enable CPU Hot Add only if you expect to use it. Alternatively, plan to power down the virtual machine before adding vCPUs, or configure the virtual machine with the maximum number of vCPUs that might be needed by the workload. If choosing the latter option, note that unused vCPUs incur a small amount of unnecessary overhead. Unused vCPUs could also cause the guest OS to make poor scheduling decisions within the virtual machine, again with the potential for reduced performance. For additional information see VMware KB article 2040375.

The reason for this is that if you enable CPU Hot Add, virtual NUMA is disabled. This means that the VM is not aware of which of its vCPUs are on the same NUMA node and might increase remote memory access. This removes the ability for the guest OS and applications to optimize based on NUMA and results in a possible reduction in performance.

Virtual NUMA (vNUMA) exposes NUMA topology to the guest operating system, allowing NUMA-aware guest operating systems and applications to make the most efficient use of the underlying hardware’s NUMA architecture. (For more information about NUMA, see page 27 in the Performance Best Practices Guide for vSphere 6.7 U2.)

To get an idea of what the performance impact can be by enabling CPU Hot Add, a simple test was run in our lab environment. This test found performance with the default setting of CPU Hot Add disabled performed from 2% to 8% better than when CPU Hot Add was enabled.

Test Details

VMware ESXi 6.7 Update 2 was installed on a four-socket Intel Xeon E7-4890 v2 (Ivy Bridge) server with 15 cores per socket and 1 TB of RAM.  A virtual machine was created with 28 vCPUs and 512 GB which spanned span half of the host’s sockets.  The VM was installed with Windows Server 2019 and SQL Server 2019.  The VM had 4 paravirtual SCSI (PVSCSI) adapters with two virtual disks for data and two virtual disks for logs.  All virtual disks were hosted on a flash-based Fibre Channel array.

DVD Store 3 (DS3)—which is an open source, transactional database workload that simulates an online store—was the test workload. The DS3 test database created was approximately 300 GB in size on disk.  A separate VM with 20 vCPUs and 32 GB of RAM was used to run the DS3 driver program that simulates all the online store users’ activities and creates load on the SQL Server system.

One set of tests were run with CPU Hot Add disabled, which is the default setting, and one set of tests were run with CPU Hot Add enabled.  This is a simple check box in the VM settings as shown below.

To disable or enable CPU Hot Add, go to Virtual Hardware > CPU > CPU Hot Plug, and then check or uncheck Enable CPU Hot Add.

Quick note about Hot Plug vs Hot Add terminology:  As seen in the screen shot above, this feature is sometimes referred to CPU Hot Plug and other times as CPU Hot Add. Both terms refer to the same thing and can be considered interchangeable. I personally like CPU Hot Add, so that will be used throughout this post.

When run with the default setting of CPU Hot Add disabled, the 28-vCPU VM has two NUMA nodes as seen in the SQL Server Management Studio when looking at the Processors section of settings for the SQL Server database.

In SQL Server Management Studio, if you have CPU Hot Add disabled, you’ll see NUMA nodes under Processor.

But when CPU Hot Add is enabled, it is seen as a single NUMA node as shown in the same SQL Server Management Studio screen.

With CPU Hot Add enabled, you’ll see only 1 NUMA node in SQL Server Management Studio.

In each set of test runs, an increasing amount of load was run against the database. This was accomplished by increasing the number of threads in the driver program until the maximum amount of throughput measured in terms of orders per minute (OPM) was achieved. Each test was run 3 times, and the throughput was averaged.

The default setting of CPU Hot Add disabled outperformed CPU Hot Add enabled by 2 to 8 percent across the load levels tested with this test workload.

When CPU Hot Add is disabled, performance increases 2-8%.

This shows that the enabling of CPU Hot Add, which causes the disabling of virtual NUMA, leads to a performance decrease. The performance impact will vary based on many factors, but it is expected that, generally, with VMs that span more than one NUMA node, performance will be better with the default setting of CPU Hot Add disabled.