The number of concurrent operations—tasks that run at the same time—for VMware vCenter Server depends on a variety of limits that apply to vCenter globally, per datacenter, per host, and even per NIC. If you look in the configuration maximums (https://configmax.vmware.com/) or docs.vmware.com for vCenter concurrent operation limits, you’ll see a bunch of numbers—these are important to know, and we include relevant parts of that information here. But there are some special considerations you need to know about to fully understand these limits.
This video describes the limits and text follows.
Global limits for concurrent operations
Global limits apply to an entire vCenter server:
- vCenter can execute approximately 640 concurrent operations before incoming requests are queued.
- vCenter can support up to 2,000 concurrent sessions (authenticated logins via UI or API, also including remote consoles) before it rejects them.
Host limits for concurrent operations
There are also per-host limits. Let’s look at each operation’s capacity in terms of generic units. Each operation takes up a certain number of units on the host (or datastore, or NIC).
- Each ESXi 6.x or 7.x host has a total of 16 units available for concurrent operations at any given time.
- An operation consumes a portion of these units both on the source and destination host.
- Different operations consume different numbers of units:
- A Storage vMotion operation costs 8 units per host. If you are changing the datastore for a VM and not the host, then the host can be involved in 2 at a time, each of which costs 8 units, for a total of 16 units.
- A linked clone operation costs 1 unit, but you must have a pre-existing snapshot. If a snapshot doesn’t exist, a snapshot is created first. Creating this snapshot may slow down the initial batch of linked clones. Cloning a powered-on VM also requires taking a snapshot first. It still costs 2 units, but vCenter needs to take a snapshot first.
- A clone, relocate, or vMotion operation costs 2 units each on a given host. That is a cost of 2 units to the source and 2 units to the destination. For example, if you clone a VM from Host A to Host A (that is, just make a duplicate on the same host), this uses 4 units on that host. But if you clone a VM from Host A to Host B, then that is only 2 units on each host. More explanation on this follows.
Concrete examples
As we mention above, cloning VMs, relocating VMs, and migrating VMs with vMotion each take a different number of resources depending on the circumstances. Cloning a VM on a single host costs 4 units and cloning a VM from one host to another host costs 2 units per host. Let’s show some examples (also demonstrated in the video above) to clarify these points.
In our first example, let’s take Host A with a VM that we want to clone. We have another host (Host B) that we can clone it to, but let’s first look at what happens when we clone the VM to another VM on the same host, Host A.
⇑ Figure 1: Each host has 16 units for vCenter operations.
Starting out, with no clones, we have 16 potential units (shown as free slots in Figure 1) that we can use for cloning operations (or vMotion, or relocate). But if we decide to clone a new VM on Host A itself, then we consume two slots for Host A as the source and then 2 slots for Host A as the destination. We thus consume 4 total slots. Since a host has 16 units available, if we wanted to clone a number of VMs from Host A to Host A, then we can have only 4 cloning operations happening at the same time (Figure 2).
⇑ Figure 2: Cloning a VM to the same host results in 4 units used, so 4 cloning operations can happen at the same time.
However, if we clone a VM from Host A to Host B, then we can have 8 clones happening at the same time (Figure 3). Each clone takes up 2 slots on Host A and 2 slots on Host B, leaving 14 slots free on each host for more provisioning operations.
⇑ Figure 3: If you clone one VM from Host A to Host B, you take up 2 units per host.
So, if you want the best concurrency with cloning, you should spread out templates or VMs across several source hosts and then clone them to a different set of destination hosts.
Datastore limits for concurrent operations
- A datastore, by default, has a capacity of 128 units.
- A vMotion operation costs 1 unit, so you can do 128 at a time on a given datastore.
- A Storage vMotion operation costs 16 units, so you can do 8 at a time on a given datastore. Just a reminder: If you use vSAN for a cluster and have only 1 datastore, then your concurrency will be impacted by this limit.
NIC limits for concurrent operations
- A 1Gb NIC has a capacity of 4 units, so you can do 4 vMotion operations at a time from a given 1Gb NIC.
- A 10Gb NIC has a capacity of 8 units, so you can do 8 vMotion operations at a time from a given 10Gb NIC.
- A 25Gb NIC has as capacity of 8 units, so you can do 8 vMotion operations at a time from a given 25Gb NIC.
NOTE: See KB 2108824 for a note on how many vmknics to set up to get full line throughput. We also recommend dedicating one NIC (or port group) to use for vMotion, and another NIC (or port group) to use for vSphere provisioning. For information about NICs and port groups, see the vSphere Networking guide.
Conclusion
When you consider vCenter limits on simultaneous tasks, there are some things to consider:
- vCenter limits
- Host limits
- Datastore limits
- NIC capabilities and vCenter limits on those capabilities
If you run many tasks at the same time in vCenter (or have a script that does this), and you see significant queuing in vCenter, you should keep in mind, “What operations am I doing and where are they being limited?”
Great info!
So if I were to configure an ESXi host with 2 pnics with a vmotion vmk tied to each pnic, my total vmotion NIC units would be 16 but I’d still be limited to 8 concurrent vmotions because a vmotion has a cost of 2 at the host level and the host units remain at 16?
Correct. The rationale is that we want to make sure a host isn’t so busy doing management operations that the performance of its VMs suffers.
Using more pNICs will typically help vMotion performance, especially helpful in situations where the underlying network configuration is not optimal. For instance, to achieve a full line rate on a 40Gb/s vMotion network, we recommend configuring three vMotion vmknics on the 40Gb/s NIC. The use of multiple vMotion vmknics allows vMotion to create multiple vMotion worker threads, thus spreading the CPU load across multiple cores rather than saturating a single core and bottlenecking network bandwidth. Hence, “2 * 20GbE NICs” is recommended than using “1 * 40GbE NIC” if you are only using a single vmknic per pNIC.
Great post!
How does that work with relocation task? (changing datastore and host at the same time)
If you mean relocating a powered-on VM, that is the same as what I’m calling ‘Storage VMotion.’
As you said we have limitation on network and storage imagine our host has 1G network now want to vmotion from Host1 to Host2
1 – regardless network we can do max 16/2 = 8 concurrent vMotion but here also network is using so when while using 1G max concurrent vMotion is 4 because network has just 4 units is that correct ?
2 When we are doing vMotion between 2 host all of them are using such as host , network , datastore now how it calculate ? because that is migrate vm between 2 hosts and also migrate data between two datastores (shared storage) ?
Are you doing a VMotion of a powered-on VM across hosts _and_ datastores? If so, the Storage VMotion limits apply.
In the first part (Host limits for concurrent operations) you said :
A Storage vMotion operation costs 8 units per host . Is that your means we can do just one svmotion because it is using all 16 units
A vMotion operation costs 2 units each on a given host
in the second part (Datastore limits for concurrent operations) you said :
A Storage vMotion operation costs 16 units
A vMotion operation costs 1 unit
I am really confused why there are these differences?
Sorry, let me clarify:
1. Storage VMotion = changing both. host and datastore. This costs 8 units PER HOST. So you can do no more than 2 per host at a time.
2. Storage VMotion costs 16 units PER DATASTORE. In other words, you can do no more than 8 at a time on a given datastore. So if you have a vSanDatastore, it can have a total of 8 storage VMotions at once. So if your vSAN cluster has a total of 8 hosts, and you have 4 hosts each doing 2 storage VMotions to 4 other hosts, you have hit the a) per-host limit (2 storage VMotions per host) and b) the per-datastore limit of 8.
Hope this helps.
I am so sorry . I think I have some problem because some parts are confusing me. I understood follows . Are there correct ?
In all of follow scenarios we are using 10G network.
1 – A vm migrate from HostA to HostB (Just vm migrate without storage) it will consume 2 units from Host so maximum concurrent vmotion are 8 in this mode.
2- Just do svmotion from sharedstorage1 to sharedstorage2 (just storage migrate without move vm) it will consume 8 unit from Host so maximum concurrent svmotion is 2 in this mode.
3- A vm migrate from HostA to HostB also vm’s data will migrate from sharedstorage1 to sharedstorage2 . according to above descriptions it had to use 10 units (2units for migrate vm and 8 units for migrate vm’s data) but your are saying totally in this mode Host will consume 8 units . which one is correct ?
and also confuse about follow :
In the first part above “Host limits for concurrent operations” you said :
4- A Storage vMotion operation costs 8 units per host (I understood just do svmotion without change vm’s host) so maximum concurrent svmotion is 2 is that correct
5- A clone, relocate, or vMotion operation costs 2 units per host ( I understood just migrate vm from hostA to HostB ) it will consume 2 units is that correct ?
But
in the second part “Datastore limits for concurrent operations” you said :
6- A vMotion operation costs 1 unit (Cannot understand this can you give an example please? what is difference with above vMotion that you said it use 2 units)
7- A Storage vMotion operation costs 16 units (Cannot understand this can you give an example please? what is difference with above svMotion that you say it will consume 8 units)
Would you please help me to clarify ?
According to document from Host perspective
“A Storage vMotion operation costs 8 units per host. If you are changing the datastore for a VM and not the host”
So if vmA and vmB and vmC resided on the HostA and Datastore1(sharedstorage) now I can just do 2 concurrent Storage vMotion from Datastore1(sharedstorage) to Datastore2(sharedstorage)
and third VM will be wait an put in the queue.
1 – Is that correct ? If your answer is “Yes” I did it in my lab but could do svmotion simultaneously for 3vms that resided on HostA and none of them did not put in queue . Please see follow pic :
https://pasteboard.co/K9gh0so.jpg
If you are moving the datastore and NOT the host, and if the VM is powered-off, then this is considered a ‘relocate’ operation, and you can do up to 8 of those on a host (it costs 2 units per host). This is NOT Storage vMotion. Storage vMotion is moving both the host and the datastore when the VM is powered-on. I’m sorry for the confusion.
I am so sorry . I think I have some problem because some parts are confusing me. I understood follows . Are there correct ?
In all of follow scenarios we are using 10G network.
1 – A vm migrate from HostA to HostB (Just vm migrate without storage) it will consume 2 units from Host so maximum concurrent vmotion are 8 in this mode.
2- Just do svmotion from sharedstorage1 to sharedstorage2 (just storage migrate without move vm) it will consume 8 unit from Host so maximum concurrent svmotion is 2 in this mode.
3- A vm migrate from HostA to HostB also vm’s data will migrate from sharedstorage1 to sharedstorage2 . according to above descriptions it had to use 10 units (2units for migrate vm and 8 units for migrate vm’s data) but your are saying totally in this mode Host will consume 8 units . which one is correct ?
and also confuse about follow :
In the first part above “Host limits for concurrent operations” you said :
4- A Storage vMotion operation costs 8 units per host (I understood just do svmotion without change vm’s host) so maximum concurrent svmotion is 2 is that correct
5- A clone, relocate, or vMotion operation costs 2 units per host ( I understood just migrate vm from hostA to HostB ) it will consume 2 units is that correct ?
But
in the second part “Datastore limits for concurrent operations” you said :
6- A vMotion operation costs 1 unit (Cannot understand this can you give an example please? what is difference with above vMotion)
7- A Storage vMotion operation costs 16 units (Cannot understand this can you give an example please? what is difference with above vMotion)
Hi Babak.
Good questions. Let me see if I can answer them.
1. Correct: HostA to HostB is 2 units from host so max of 8 concurrent VMotions.
2. Correct. Changing datastores without changing the host ocnsumes 8 units per host, so max concurrency is 2 per host (since a host has 16 slots).
3. Storage VMotion from HostA/DS1 to HostB/DS2 has 2 limits to consider: the host cost when changing both host and datastore is 8 units, and the datastore cost is 16. These units are counted separately, so HostA is charged 8 units and DS1 is charged 16 units. HostB is charged 8 units and DS2 is charged 16 units.
4. Correct: maximum concurrent VMotions where you change both host and datastore is 2 per host (since the host cost is 8).
5. Correct. Clone, relocate, or VMotion costs 2 units per host.
6. VMotion costs 1 unit per datastore. Here is an example. If I had, say, a 64 Host cluster with a single datastore, and if I did 23VMotions each from Host0 to Host 63, Host1 to Host62, Host2 to Host61, etc. then no host would be saturated (each one is involved in only 3 VMotions, but I would be doing a total of 192 VMotions using that datastore. Since the datastore limit is 128, vCenter will throttle VMotions so I’m doing only 128 at once.
7. When I said Storage VMotion, I meant a VMotion that changes both the host and the datastore.
Hi!
thank you for the informative blog post. It is very useful in the design of my environment.
I only have one additional question that I could not answer with it.
What about the unit consumption of datastores in cold clone operations? Are these the same as storage vMotion? Or is there no unit limit for clone operations?
thanks in advance!
Max
A cold clone does not have a unit consumption on a datastore. It only. has a unit consumption on a host (both source and destination).
You can also check https://www.vmware.com/techpapers/2021/cloning-vSphere7-perf.html
Hi,
Is there any limit of simultaneous cross vcenter storage vmotion migrations? This is other than host, datastores and vmotion NIC.
Thanks in advance
Bharat
Please, check https://blogs.vmware.com/vsphere/2019/09/how-to-tune-vmotion-for-lower-migration-times.html and https://blogs.vmware.com/performance/2021/02/vcenter-limits-for-concurrent-vmotion.html
Sorry I missed this post. Let me look this up and get back to you when I can.
There is, to my knowledge, no such limit beyond the host/datastore/vmotion NIC limits.
Hi,
Interesting info !!
Two questions about it:
1.- Is there VMware documentation about this?
2.- How many credits costs a snapshot operation for a datastore where virtual disks of VM are located?
Thanks
Sorry, one more question:
Is there any limit at vSphere cluster level?
Thanks
Sorry, one more question: is there any limit at vSphere cluster (not host or datastore) level?
Thanks.
Hi Ravi, has this issue been addressed in newer versions of vsphere such as 8.x
We have vsphere 7.03 and Horizon 8 and ISCSI storage
When we adjust the Horizon provisioning threads above 8, the Virtual Center tasks start backing up.
Provisioning of VMs can exceed 60 minutes at time, while the provisioning tasks are running other tasks are blocked
We have a support ticket open but not getting to the bottom of the task/process backlog issues we are seeing
Any pointers are appreciated greatly
Thanks