When CPU Hot Add is disabled, performance increases 2-8%.

CPU Hot Add Performance in vSphere 6.7

Leaving CPU Hot Add at its default setting of “off” is one of the performance best practices that we have for large VMs. From the Performance Best Practices Guide for vSphere 6.7 U2:

CPU Hot Add is a feature that allows the addition of vCPUs to a running virtual machine. Turning on this feature, however, turns off vNUMA for that virtual machine, resulting in the guest OS seeing a single vNUMA node. Without vNUMA support, the guest OS has no knowledge of the CPU and memory virtual topology of the ESXi host. This in turn could result in the guest OS making sub-optimal scheduling decisions, leading to reduced performance for applications running in large virtual machines. For this reason, enable CPU Hot Add only if you expect to use it. Alternatively, plan to power down the virtual machine before adding vCPUs, or configure the virtual machine with the maximum number of vCPUs that might be needed by the workload. If choosing the latter option, note that unused vCPUs incur a small amount of unnecessary overhead. Unused vCPUs could also cause the guest OS to make poor scheduling decisions within the virtual machine, again with the potential for reduced performance. For additional information see VMware KB article 2040375.

If you turn on CPU Hot Add, virtual NUMA is turned off. This means that the VM is not aware of which of its vCPUs are on the same NUMA node and might increase remote memory access. This removes the ability for the guest OS and applications to optimize based on NUMA and results in a possible reduction in performance.

Virtual NUMA (vNUMA) exposes NUMA topology to the guest operating system, allowing NUMA-aware guest operating systems and applications to make the most efficient use of the underlying hardware’s NUMA architecture. (For more information about NUMA, see page 27 in the Performance Best Practices Guide for vSphere 6.7 U2.)

To get an idea of what the performance impact can be by enabling CPU Hot Add, a simple test was run in our lab environment. This test found performance with the default setting of CPU Hot Add turned off performed from 2% to 8% better than when CPU Hot Add was turned on.

Test Details

VMware ESXi 6.7 Update 2 was installed on a four-socket Intel Xeon E7-4890 v2 (Ivy Bridge) server with 15 cores per socket and 1 TB of RAM.  A virtual machine was created with 28 vCPUs and 512 GB which spanned span half of the host’s sockets. The VM was installed with Windows Server 2019 and SQL Server 2019.  The VM had 4 paravirtual SCSI (PVSCSI) adapters with two virtual disks for data and two virtual disks for logs.  All virtual disks were hosted on a flash-based Fibre Channel array.

DVD Store 3 (DS3)—an open source, transactional database workload that simulates an online store—was the test workload. The DS3 test database created was approximately 300 GB in size on disk.  A separate VM with 20 vCPUs and 32 GB of RAM was used to run the DS3 driver program that simulates all the online store users’ activities and creates load on the SQL Server system.

One set of tests was run with CPU Hot Add turned off, which is the default setting, and one set of tests were run with CPU Hot Add turned on.  This is a simple check box in the VM settings as shown below.

You can disable or enable CPU Hot Add by going into Virtual Hardware and open CPU, then look for CPU Hot Plug. Enable CPU Hot Add is next to it.
To turn off or turn on CPU Hot Add, go to Virtual Hardware > CPU > CPU Hot Plug, and then check or uncheck Enable CPU Hot Add.

Quick note about Hot Plug vs Hot Add terminology:  As seen in the screen shot above, this feature is sometimes referred to CPU Hot Plug and other times as CPU Hot Add. Both terms refer to the same thing and can be considered interchangeable. I personally like CPU Hot Add, so that will be used throughout this post.

When run with the default setting of CPU Hot Add turned off, the 28-vCPU VM has two NUMA nodes, as seen in the SQL Server Management Studio when looking at the Processor section of settings for the SQL Server database.

When run with the default setting of CPU Hot Add disabled, the 28-vCPU VM has two NUMA nodes as seen in the SQL Server Management Studio when looking at the Processors section of settings for the SQL Server database.
In SQL Server Management Studio, if you have CPU Hot Add turned off, you’ll see NUMA nodes under Processor.

But when CPU Hot Add is turned on, it is seen as a single NUMA node as shown in the same SQL Server Management Studio screen.

With CPU Hot Add enabled, you'll see only 1 NUMA node in SQL Server Management Studio.
With CPU Hot Add turned on, you’ll see only 1 NUMA node in SQL Server Management Studio.

In each set of test runs, an increasing amount of load was run against the database. This was accomplished by increasing the number of threads in the driver program until the maximum amount of throughput measured in terms of orders per minute (OPM) was achieved. Each test was run 3 times, and the throughput was averaged.

The default setting of CPU Hot Add turned off outperformed CPU Hot Add turned on by 2 to 8 percent across the load levels tested with this test workload.

When CPU Hot Add is disabled, performance increases 2 to 8 percent.
When CPU Hot Add is turned off, performance increases 2-8%.

This shows that turning on CPU Hot Add, which causes virtual NUMA to turn off, leads to a performance decrease. The performance impact will vary based on many factors, but it is expected that, generally, with VMs that span more than one NUMA node, performance will be better with the default setting of CPU Hot Add turned off.


6 comments have been added so far

    1. A large VM in this context is one that is spanning multiple physical NUMA nodes. A VM that is small enough to fit on a single NUMA node would not be considered large in this context. (Sorry I didn’t respond earlier – I must have missed the notification)


  1. Thanks Todd. I had the same question of how large VM you consider as large. Another question Can we turn off CPU hot add for all the VMs irrespective of large or small VMs so that we don’t need to put additional controls in infrastructure automation?

    1. Hot-add CPU is turned off by default – so it is probably already off on all of your VMs, unless selected to turn it on. We have seen customers that decide to enable it thinking that it would be great to be able to hot-cpus at some point in the future, without knowing about the potential performance cost. Here we are trying to inform about this feature in more detail, even though it is something that probably most VMs are not using.


  2. hi Todd,

    Thank you for the great article. May I ask you if it is possible to hot remove cpu on a VM, if yes, where in the vm settings we see that option. ( my sincere apologies for asking this question out of topic of this blog)


Leave a Reply

Your email address will not be published. Required fields are marked *