Endpoint Limits for Concurrent REST Requests from vCenter APIs

Customers often ask us about the concurrency limits in vCenter APIs for operations like provisioning, tagging, or content-library changes. In this article, we answer this question by describing the flow of API commands through the vCenter Server appliance (VCSA, or simply vCenter), highlighting appropriate limits as commands traverse the VCSA stack.

There are two ways to make API calls that manipulate vCenter resources: using a REST client (aka vAPI) or a SOAP client. Each has different limits enforced at the various layers of the stack.

• The REST client leverages the vSphere Automation APIs. Some examples in this blog post include REST Client Initiates a VM Clone Request and REST Client List All Tags.
• The SOAP client calls are typically invoked using PowerCLI, Perl, Java, or Python scripts. These scripts can manipulate only provisioning, inventory, and host management. We include an example of a SOAP client action below; refer to SOAP Client Initiates a VM Clone Request.

This blog post is primarily concerned with concurrent vAPI/REST client requests. SOAP requests are limited to 2,000 concurrent sessions or 640 concurrent requests. For a complete treatment of the SOAP limits, see vSphere 7.0 U1 Tagging Performance Best Practices.

Table 1 below shows all the limits that affect REST client requests/user sessions and connections that can run concurrently. Figure 1 shows the API data flow and where in the flow the limits apply.

Table 1. This table describes the endpoint limits for concurrent REST requests from vCenter automation APIs. Refer to figure 1 to find the corresponding area the limit applies to.

Figure 1 below shows the flow of a REST client request through vCenter to the reverse HTTP proxy (Envoy) shown in the orange box, to the vAPI endpoint in the green box, and ending at the vAPI service providers in the blue boxes at the bottom of the figure.

Figure 1. Flow of a REST client request through vCenter 

The flow proceeds as follows:

• The request from the external clients first arrives at vCenter’s reverse proxy–known as Envoy–which acts as the gatekeeper for all incoming requests. There is a limit of 2048 concurrent HTTP or HTTPS requests at Envoy (#1 and #2 in figure 1). Note: The Envoy service brokers both internal services as well as those external to vCenter. Internal service communication occurs over HTTP, and so there is a limit of 2048 concurrent connections or sessions at Envoy in this case.

• Envoy determines whether the request must be sent to the vAPI endpoint (if it’s from a REST client) or directly to the API service providers (if the request is from a SOAP client). For REST requests, which are routed to the vAPI endpoint:
  • The maximum number of connections is 550 (#3 in figure 1).
  • There is a rate limiter of 1500 requests per second to the vAPI endpoint by the Jetty webserver (#4 in figure 1).
  • Envoy then sends the REST client requests onto the vAPI endpoint, which is an API aggregator for a number of infrastructure services provided by different vCenter components (API service providers, vpxd, the content library, or tagging services, as described below). It also includes Jetty as the webserver. Envoy sends SOAP client requests directly to the API service providers or vpxd. SOAP clients cannot manipulate the content library or tagging services.
• API service providers (general) (aka vAPI service providers) – These are vCenter services that provide the actual infrastructure operations required by the clients. Examples include appliance management, token service, vpxd_svcs, compute policy, trust management, certificate management, and the vmware_vmon service, which manages the lifecycle of Platform Services Controller and vCenter.
  • vpxd – This is the vCenter service responsible for provisioning, inventory, and host management.
  • Content library – This stores VM templates, vApp templates, and other types of files. It can share the stored files across multiple vCenter instances in the same or different locations.
  • Tag – A tag lets you attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects. A tag is a label that you can apply to objects in the vSphere inventory.

There is a limit of 1000 sessions that the vAPI endpoint can create (#5 in figure 1). There is also a queue that limits the number of simultaneous sessions, which allows approximately 160 sessions in vCenter 7.x (#7 in figure 1).

API Data Flow

In this section, we address the flow of vCenter APIs in more detail. A general flow is first to verify that the requestor is a valid user (authentication), then to determine that this user has access to requested objects (for example, does a user have permissions to clone a particular VM), and finally to carry out the request. The following sections show some examples of popular vCenter operations.

REST Client Initiates a VM Clone Request

The API data flow is shown in figure 2 below.

Figure 2. Flow of a REST client (vAPI) initiating a VM clone request; the numbers correspond to the flow description below

The flow occurs as follows:

1. Envoy receives the REST request. The VM clone request can be issued as either blocking or non-blocking.
   • Blocking requests are those which wait for the long-running operation to complete, and then return the result to the calling function. An example of a blocking request for VM Clone is https://{api_host}/api/vcenter/vm?action=clone

• • Non-blocking requests are those that do not wait for the long-running operation to complete. Once the request is issued to vCenter, it returns a Task ID. This Task ID is then used to track the status of the request. An example of a non-blocking request for VM Clone is https://{api_host}/api/vcenter/vm?action=clone&vmw-task=true

For a blocking vAPI VM Clone request, there is a limit of 10 concurrent blocking tasks, which are covered in Maximum blockingTasks.

2. For REST-based requests, Envoy redirects the request to the vAPI endpoint where it checks for authentication.
3. The vAPI endpoint then moves the request to the appropriate service provider (in this case, vpxd), which checks for authorization, then performs the VM clone request.
4. The response is propagated back to the calling module. For a non-blocking VM clone request, a Task ID is sent back, which you can use to poll the status of that task.

SOAP Client Initiates a VM Clone Request

The flow is shown in figure 3 below.

Figure 3. SOAP client flow for initiating a VM clone request; the numbers correspond to the flow description below

The flow occurs as follows: 

1. Envoy receives the non-blocking VM clone request.
2. Envoy redirects the request to the appropriate service provider (vpxd) directly, which checks for authentication and authorization and then performs the necessary task. 
3. Because it is a non-blocking VM clone request, the response (Task ID) is sent back to the calling module, Envoy. You can then use this Task ID to poll for the status of the VM Clone task.

REST Client List All Tags

The flow is shown in figure 4 below.

Figure 4. A diagram of the flow for when a REST client initiates a “list all tags” request; the numbers correspond to the flow description below

The flow occurs as follows:

1. Envoy receives the request to list all the tags (https://{api_host}/api/cis/tagging/tag).
2. Envoy redirects the request to the vAPI endpoint, where it checks for authentication.
3. The vAPI endpoint forwards the request to the appropriate service provider, which checks for authorization and then lists all the tags.
4. The response is then propagated back to the calling module.

Further Description of vAPI Endpoint Limits

Now that we have established how a vAPI request flows through vCenter, let’s look at some sample data flows to understand further why the limits of the vAPI endpoint matter and how it impacts the client workloads.

maxSessions

A session is equivalent to a user login; that is, a new session is created every time the user logs in.

The maxSessions parameter controls the total number of sessions supported by the vAPI endpoint, which is 1000 by default. Upon exceeding this limit, additional logins or attempts to create new sessions will be rejected.

• Say you have 10 users (user1 to user10), who successfully establish 100 sessions with a REST client each on vCenter, which leads to 1000 user sessions being created at the vAPI endpoint.
• Once the total sessions hit the count of 1000, no further sessions will be allowed at the vAPI endpoint.
• When user11 attempts to create a new session (the 1001st session), it leads to a 503 Service Unavailable error.
• Existing sessions will function as usual.

Figure 5. The limit of maxSessions for the vAPI endpoint is 1000

maxSessionsPerUser

The maxSessionsPerUser parameter controls the total number of sessions per user supported by the vAPI endpoint, which is 550 by default. Upon exceeding this limit, additional logins or attempts to create new sessions for that particular user will be rejected.

• Say user1 has 550 sessions (REST) successfully established on vCenter.
• If the total sessions of user1 hit the 550 sessions mark, then no further sessions can be created for this user at the vAPI endpoint.
• Now, when user1 attempts to create a new session (551^st session for user1), it leads to a 503 Service Unavailable error.
• Already existing sessions of user1 will function as usual.

Figure 6. The maximum concurrent sessions per user at the vAPI endpoint is 550; the 551^st session errors out with 503 Service Unavailable

Maximum blockingTasks

For this example, let’s use the VM Clone operation as the target of a blocking call, which has a limit of 10 requests.

Let’s say user1 attempts to run 11 blocking versions of the VM Clone operations (REST) simultaneously. (Or, it could also be multiple users attempting to run more than 10 blocking calls in parallel.)

vCenter starts rejecting the VM Clone requests due to the concurrent blocking requests and returns a 503 Service Unavailable whenever the concurrent blocking tasks request limit exceeds 10. 

Figure 7. The maximum number of concurrent blocking tasks at the vAPI endpoint is 10; any further requests error out with 503 Service Unavailable

maxHTTPS Connections at Envoy

Most HTTPS connections refer to the TCP connection being issued by the client. As previously mentioned, a session is equivalent to a login. Users can create multiple sessions over a single TCP connection.

Envoy maintains separation in limits for HTTP and HTTPS requests, each being limited to a maximum of 2048 connections. Most HTTP requests are internal; that is, communication between providers and components of vCenter—for example, a vCenter connection with tagging service or storage profile services.

Envoy has a maximum HTTPS connection limit of 2048, and the maximum number of connections from Envoy to the vAPI endpoint is limited to 550 by default.

Here we look at a specific scenario (shown in figure 8), where the maximum HTTPS connections at Envoy (2048) are exhausted even before the maximum vAPI endpoint connections limit (550) is reached.

• First, we established over 1750 SOAP connections to vCenter. 
• Once these connections were established and their respective operations were in progress, we then attempted 550 REST connections to vCenter. 
• Since Envoy already had over 1750 HTTPS connections, there wasn’t enough room to accommodate all 550 connections to the vAPI endpoint. This hit the limit and resulted in a 503 Service Unavailable error.

Figure 8. The maximum number of HTTPS connections at Envoy is 2048

maxConnections to the vAPI endpoint

For incoming REST requests at Envoy, Envoy creates connections to the vAPI endpoint to serve these requests. The maximum number of these connections, from Envoy to the vAPI endpoint, is limited to 550. 

Figure 9. The maximum number of connections from Envoy to the vAPI endpoint is 550

Conclusion

In this blog, we discussed the concurrency limits for vCenter APIs. We showed there are limits at multiple layers of the stack, depending on the endpoint (vCenter, tagging, content library, etc.), and these limits are a combination of session limits, per-user concurrent calls, and overall limits to the number of requests per second. Limits in the lower levels of the stack affect the limits at the top layers of the stack, so you should know which layers you are working with and what the limits in each layer are.

References

• REST (vAPI) automation APIs
  https://developer.vmware.com/apis/vsphere-automation/latest/
• SOAP APIs
  https://developer.vmware.com/apis/968/vsphere
• SDK programming guide
  https://developer.vmware.com/docs/6611/vmware-vsphere-web-services-sdk-programming-guide 

Acknowledgments

The author thanks Venu Uppalapati, Ravi Soundararajan, and Julie Brodeur for their reviews and edits to this blog post.