Posted by Kyle Gleed, Sr. Technical Marketing Architect, 6 June 2012
One thing that I really like about ESXi is that it's small. The full ESXi 5.0 image is just under 150MB, if you include VMware tools it's around 300MB. Having such a small image not only helps keep things very secure, but it also makes it very easy to deploy. So easy in fact that it's becoming increasingly popular to find customers deploying ESXi "on the fly" when the host is booted, a practice commonly referred to as "stateless" ESXi. There are different ways to implement stateless ESXi, but typically it involves PXE booting the hosts, installing the ESXi image over the network, and using scripts, host profiles, or a combination of the two to apply the host configuration. Stateless ESXi has several key benefits:
No requirement for a dedicated boot device. ESXi is booted over the network directly into the host's memory.
Eliminate configuration drift. Because the host is booted from a centralized base image and the configuration re-applied each time the host reboots, each reboot it's like doing a fresh install.
Centralized image and configuration management. You only need to maintain one central image and one central set of configuration files/scripts for all your hosts.
In vSphere 5.0 VMware formally introduced Auto Deploy as its official solution for running stateless ESXi. Auto deploy leverages gPXE to network boot ESXi hosts where a rules engine running on an Auto Deploy server uses attributes learned from the booting host to determine what software should be installed, what configuration should be applied and where in vCenter the host should be connected.
Whether you are using Auto Deploy or have implemented a custom stateless ESXi solution one thing you really need to be careful about is making configuration changes and applying software updates to your stateless hosts.
Configuration Changes
Each time you reboot a stateless host it boots from a centralized base image and the host's configuration is re-applied. This means each time you reboot your hosts it is like doing a fresh install in that the host will always boot to a consistent image and configuration. While this is great when it comes to eliminating configuration drift, it can present challenges when you do need to make changes, and you need those changes need to persist across reboots. Consider an example where you want to change the number of ports on a standard vSwitch. Changing the number of vSwitch ports requires rebooting the host, but in the case of stateless ESXi when you reboot the change will be lost as the host will boot from the centralized image and get reconfigured back to it's original state. As such, when running stateless ESXi for host changes to persist across reboots, such as changing the number of vSwitch ports, you need to first update the source scripts (or host profile) and then reboot the host. So remember, when working with stateless ESXi hosts you should avoid making host level changes directly on the host. Instead changes should always be made to the source scripts/host profile that are used to configure the host. Failure to follow this rule will result in changes being lost the next time you reboot.
Patching
Another area to consider is patching. Using traditional tools, like vSphere Update Manger (VUM), don't work well in a stateless environment because, just like with configuration changes, any patches or updates applied to a stateless host will be lost the next time you reboot. Here again, when running stateless ESXi in order for software changes to persists across reboots you need to first update the source image and then reboot your hosts.
While at first glance it might appear that running stateless makes applying configuration changes and software updates a bit more complicated, in truth it actually makes things quite easy. Consider a large cluster with 12 hosts, rather than having to apply the same configuration change 12 times, or install the same patch to 12 separate hosts, you can simply make the configuration change once, or apply the software patch once (to the source image) and then reboot the hosts. This also helps to avoid common mistakes like inadvertently forgetting to make a change or apply a patch to one host in the cluster. Because you've updated the source configuration the changes will always persist even when you reboot. In addition, by leveraging vMotion together with DRS to migrate VMs off the hosts prior to rebooting it's very easy to apply configuration changes and roll out software updates with no VM downtime. Another nice benefit of stateless ESXi is that in the rare event a change or software update introduces a problem that requires undoing, rolling back is as simple as removing the update from the source and rebooting your hosts again.
Stateless ESXi provides many benefits and is quickly becoming the boot method of choice for many. However, don't forget when moving to a stateless architecture that you should avoid making changes to individual hosts. Instead you will need to get into the habit of always updating the source from which your hosts will boot and get their configuration. After all, centralized image and configuration management is a key benefit of running stateless ESXi.
