Many customers are using the vRealize Operations Manager REST API for a variety of use cases. The API offers the ability to interact with vR Ops programmatically to enable automation use cases, data mining for overall enterprise planning and injecting metrics and properties into monitored resources for expanded reporting and alerting. In addition to native HTTP REST calls, customers can also leverage the PowerCLI module for vR Ops beginning with version 6.3R1 of PowerCLI.
One frequent question from customers is around API performance. Obviously, as you operationalize extensibility with vR Ops using the API this is a valid question. The good news is that our Quality Engineering team regularly tests new builds of vR Ops for API functionality and performance. In this blog post, I will cover the scale and recent results of that testing as well as best practices for using the API to ensure performance. I will also cover how you can monitor API performance in your own environment.
Performance Tests of the public REST APIs of vRealize Operations
I reached out to Ruzan Hovhannisyan, Member of Technical Staff on VMware’s vR Ops ESO team. Ruzan provided details on how they validate the API performance and shared some recent results. One of the test environments is a single large vRealize Operations node monitoring 5,000 objects and 3 million metrics. This instance was deployed in July, 2014 and is updated with new builds weekly. There are a number of API tests run against this environment, but allow me to focus on just a few to provide some idea of the responsiveness we see for a configuration of this size.
- Getting stats of 3mln metrics for 1 day in average takes 201254ms (3.3 minutes)
- Getting stats of 3mln metrics from the beginning of deployment till today in average takes 453856 ms (7.6 minutes)
- Getting stats of 3mln metrics for 5 minutes (the call was triggered every 5 minutes 1 day continuously) in average takes 1sec.
To be honest, these tests were run by Ruzan at my request to validate what a customer SHOULD experience. Honestly, this is the most intensive REST request I can imagine for the API and one of the more common use cases. Customers like the data that vR Ops provides but want to integrate with existing systems or load into data mining for resource planning.
But, the bottom line is that it is reasonable to expect good performance from API calls, even for retrieval of historical data (in this case over 2 years!). Even grabbing fresh metrics for all 5,000 resources every five minutes wasn’t a problem for vR Ops REST API.
Best Practices for Best Performance
Given that the REST API is proven performant, how can customers make sure they are getting the very best possible performance from their instance’s APIs? For this, I enlisted the help of Nandu Mallapragada, Staff Engineer on VMware’s R&D team. Nandu suggested that customers make sure to leverage the token-based authentication method in their code or scripts. The use of basic authentication for successive REST calls can add significant overall delays because basic authentication does a full lookup of the credentials each time it is called. Token-based, on the other hand, provides a one-time token with a 30 minute expiration that allows successive calls to avoid these delays.
Another frequent mistake, is making more calls than needed. Using your code to retrieve a set of data from vR Ops and then parsing that response is much more efficient that making successive calls to retrieve individual data points. For example, if you would like to have the set of CPU related metrics from all virtual machine resources, use the POST /api/resources/stats/query with a request payload. Below is a screenshot from Postman where I am making such a request (the double curly braces indicate a variable in Postman).
As a result, I get a nice JSON response that is easily navigated by my code.
This is going to be much more efficient than making successive REST calls for each VM or metric.
How to Measure the REST API Performance in Your Environment
Ruzan pointed out that the test results are readily available in the vRealize Operations user interface (and for that matter, via the REST API). These API calls are part of the vRealize Operations self-monitoring metric set. Simply navigate to Environment > vRealize Operations Clusters and from there select a node to expose the vRealize Operations Manager Suite API resource.
This resource contains a metric group, API Calls, that you can use to monitor performance of specific API endpoints. As shown below, I have charted the API call POST api/resources/{id}/stats/ average response time in milliseconds.
Using this, you can create a dashboard to check on the performance of your API calls from third-party systems.
