The number of concurrent operations—tasks that run at the same time—for VMware vCenter Server depends on a variety of limits that apply to vCenter globally, per datacenter, per host, and even per NIC. If you look in the configuration maximums (https://configmax.vmware.com/) or docs.vmware.com for vCenter concurrent operation limits, you’ll see a bunch of numbers—these are important to know, and we include relevant parts of that information here. But there are some special considerations you need to know about to fully understand these limits.
This video describes the limits and text follows.
Global limits for concurrent operations
Global limits apply to an entire vCenter server:
- vCenter can execute approximately 640 concurrent operations before incoming requests are queued.
- vCenter can support up to 2,000 concurrent sessions (authenticated logins via UI or API, also including remote consoles) before it rejects them.
Host limits for concurrent operations
There are also per-host limits. Let’s look at each operation’s capacity in terms of generic units. Each operation takes up a certain number of units on the host (or datastore, or NIC).
- Each ESXi 6.x or 7.x host has a total of 16 units available for concurrent operations at any given time.
- An operation consumes a portion of these units both on the source and destination host.
- Different operations consume different numbers of units:
- A Storage vMotion operation costs 8 units per host. If you are changing the datastore for a VM and not the host, then the host can be involved in 2 at a time, each of which costs 8 units, for a total of 16 units.
- A linked clone operation costs 1 unit, but you must have a pre-existing snapshot. If a snapshot doesn’t exist, a snapshot is created first. Creating this snapshot may slow down the initial batch of linked clones. Cloning a powered-on VM also requires taking a snapshot first. It still costs 2 units, but vCenter needs to take a snapshot first.
- A clone, relocate, or vMotion operation costs 2 units each on a given host. That is a cost of 2 units to the source and 2 units to the destination. For example, if you clone a VM from Host A to Host A (that is, just make a duplicate on the same host), this uses 4 units on that host. But if you clone a VM from Host A to Host B, then that is only 2 units on each host. More explanation on this follows.
Concrete examples
As we mention above, cloning VMs, relocating VMs, and migrating VMs with vMotion each take a different number of resources depending on the circumstances. Cloning a VM on a single host costs 4 units and cloning a VM from one host to another host costs 2 units per host. Let’s show some examples (also demonstrated in the video above) to clarify these points.
In our first example, let’s take Host A with a VM that we want to clone. We have another host (Host B) that we can clone it to, but let’s first look at what happens when we clone the VM to another VM on the same host, Host A.
⇑ Figure 1: Each host has 16 units for vCenter operations.
Starting out, with no clones, we have 16 potential units (shown as free slots in Figure 1) that we can use for cloning operations (or vMotion, or relocate). But if we decide to clone a new VM on Host A itself, then we consume two slots for Host A as the source and then 2 slots for Host A as the destination. We thus consume 4 total slots. Since a host has 16 units available, if we wanted to clone a number of VMs from Host A to Host A, then we can have only 4 cloning operations happening at the same time (Figure 2).
⇑ Figure 2: Cloning a VM to the same host results in 4 units used, so 4 cloning operations can happen at the same time.
However, if we clone a VM from Host A to Host B, then we can have 8 clones happening at the same time (Figure 3). Each clone takes up 2 slots on Host A and 2 slots on Host B, leaving 14 slots free on each host for more provisioning operations.
⇑ Figure 3: If you clone one VM from Host A to Host B, you take up 2 units per host.
So, if you want the best concurrency with cloning, you should spread out templates or VMs across several source hosts and then clone them to a different set of destination hosts.
Datastore limits for concurrent operations
- A datastore, by default, has a capacity of 128 units.
- A vMotion operation costs 1 unit, so you can do 128 at a time on a given datastore.
- A Storage vMotion operation costs 16 units, so you can do 8 at a time on a given datastore. Just a reminder: If you use vSAN for a cluster and have only 1 datastore, then your concurrency will be impacted by this limit.
NIC limits for concurrent operations
- A 1Gb NIC has a capacity of 4 units, so you can do 4 vMotion operations at a time from a given 1Gb NIC.
- A 10Gb NIC has a capacity of 8 units, so you can do 8 vMotion operations at a time from a given 10Gb NIC.
- A 25Gb NIC has as capacity of 8 units, so you can do 8 vMotion operations at a time from a given 25Gb NIC.
NOTE: See KB 2108824 for a note on how many vmknics to set up to get full line throughput. We also recommend dedicating one NIC (or port group) to use for vMotion, and another NIC (or port group) to use for vSphere provisioning. For information about NICs and port groups, see the vSphere Networking guide.
Conclusion
When you consider vCenter limits on simultaneous tasks, there are some things to consider:
- vCenter limits
- Host limits
- Datastore limits
- NIC capabilities and vCenter limits on those capabilities
If you run many tasks at the same time in vCenter (or have a script that does this), and you see significant queuing in vCenter, you should keep in mind, “What operations am I doing and where are they being limited?”