Auto Deploy performance boost with reverse proxy caches

The latest release of VMware vSphere includes beneficial improvements to Auto Deploy, including a new graphical user interface, a new deployment workflow, and various manageability and operational enhancements. One of the operational advancements is a dramatically simplified caching capability.

There are several reasons why you might consider adding reverse proxy caching to your Auto Deploy infrastructure. For starters, this design will reduce the load on the vCenter Server Appliance and Auto Deploy service, freeing up resources for other processes. Secondly, the boot time of individual stateless VMware ESXi hosts is modestly improved – saving about 30 seconds in a typical setup, possibly more in a heavily-loaded environment. And finally, you can potentially boot far more stateless hosts concurrently without overwhelming the VCSA.

When changing critical infrastructure components, resiliency would naturally be top of mind. I’m glad to report that the new reverse proxy design does not create a single point of failure, since you can deploy multiple proxy servers that are tried in a round-robin sequence with no load balancers. Furthermore, if all proxies happen to become unavailable, the stateless clients fail gracefully back to the default behavior of accessing the Auto Deploy server directly. This is a welcome improvement over previous releases! Just keep in mind that the caches are only for performance optimization and not for redundancy of the overall stateless infrastructure – the Auto Deploy server is still in charge and must be online for successful host boot operations.

Instant Reverse Proxy Container

If you like the sound of these benefits, then it’s easy enough to test this design by quickly deploying a Docker container configured for the task. Create one or two Linux VMs running Docker (I’m using PhotonOS in my lab) and fire up the Nginx container that I published on Hub:

In the above example, the proxy will listen on port 5100 and fetch any requested ESXi image files from your existing Auto Deploy server located at 10.197.34.22. Run this container on each VM that will act as a proxy, and make note of their IP addresses for the next part.

Connectivity Test

Before you configure Auto Deploy to use these new caches, it’s a good idea to verify connectivity. One way to do this is to watch the Nginx log file while manually requesting a file from the cache.

To watch the Nginx log, get the id of the container and use the docker logs –f  command:

Then, request the Auto Deploy tramp file from another system, like so:

Confirm that the proxy responds immediately with the above output. If it does not, go back and double-check addresses, ports, and other potential connectivity problems.  Also observe the log file that is being tailed for a corresponding hit.

Activate the Proxies

Now that you have one or more reverse proxies up and running, it’s time to configure Auto Deploy to start using them. This is done by using PowerCLI, through a new cmdlet introduced in the 6.5 release. Run the following command for each proxy you need to register, adjusting the address accordingly:

Check the configuration by running Get-ProxyServer,  and if necessary, remove a proxy from rotation with the Remove-ProxyServer  cmdlet.

At this point, any stateless hosts that boot will use the cache. You can verify the configuration by accessing the Auto Deploy diagnostics web interface:

Click on any listed host, then on the diagnostics page that will appear, click Get iPXE Configuration. Check the resulting config for the multiple-uris directive and lines beginning with uris -a that point to your proxy caches, like so:

Action!

Boot or reboot stateless hosts, and they will access the proxy caches. You can monitor requests coming to the Auto Deploy server and to the caches to verify the changes have taken effect. Note that the first time a host boots, the proxy will need to fetch all the files from Auto Deploy to cache them. After that, everything but a small set of non-cacheable files will be served from the caches.

The caches are easy to monitor through the docker logs command, as described above. It’s’ also pretty simple to watch key activity on the Auto Deploy (VCSA) system. Try the following command with and without the caches enabled if you want to get a feel for the boot time reduction in your environment:

From Concept to Production

The example above is a proof of concept, intended to help you understand how to configure and monitor the effects of reverse proxies. For most production datacenters, it would be wise to create a proxy server that is equipped with SSL certificates so that the traffic between hosts and the proxies can be encrypted. The Nginx SSL configuration is straightforward, but beyond the scope of this article. You can also read how I created the container if you want to use that as a reference.

Summary

The new reverse proxy cache feature in Auto Deploy 6.5 is very easy to set up and will boost performance without introducing additional failure points to your vSphere infrastructure. Docker containers running Nginx offer a simple way to demonstrate the concept in your environment.