Posted on 16 Aug 2012 by Kyle Gleed, Sr. Technical Architect, VMware
When using Auto Deploy to provision ESXi hosts there is no requirement for a local boot disk or any kind of local storage. When you boot a "diskless" host using Auto Deploy the host’s "scratch" location will default to /tmp/scratch (i.e. ramdisk), which is not recommended. This basically means your log files are running in memory and will be lost each time you reboot. The same thing happens when you boot your hosts using USB/SD – do to write sensitivity with USB/SD ESXi will not use them for scratch, even if there is plenty of available space, and when there is no other local storage scratch will go to /tmp/scratch.
How can you tell where your hosts scratch directory is located? Log on to the ESXi shell and run an "ls -al" from the root directory. Look for the "/scratch" directory, if it's linked to /tmp/scratch it's in memory. If it's linked to a VMFS volume then it's on a persistent storage device.
Normally, this isn't a big deal because if you find that your scratch is in memory it’s easy to change the location to a persistent datastore (http://kb.vmware.com/kb/1033696). However, while this KB works when booting off of USB/SD it doesn’t work when using Auto deploy.
One of the requirements for setting the scratch location is that each host needs to have it’s own scratch directory, which means each host requires a different setting for the ScratchConfig.ConfiguredScratchLocation. As Auto Deploy uses host profiles to configure the host, this means you would need to set this as part of the host’s answer file, but I just discovered that you can't set a unique ScratchConfig.ConfiguredScratchLocation using a host profile. This means if you are using Auto Deploy with diskless hosts your scratch will always be in /tmp/scratch, and there’s no way to change it.
So what should you do if you find yourself in this boat? I’ve found two possible workarounds:
Add a local disk to your Auto Deploy hosts and format it as VMFS. The disk will need to be at least 4GB in size. When the host boots it will see the local VMFS and will put scratch on it. This is done automatically, there's nothing you need to configure. Of course this mitigates some of the benefits of going stateless as it requires allocating local storage for each hosts.
A better workaround, and what I recommend, is to setup a remote syslog server. This way you don’t have to hassle with allocating local storage and you get all the benefits of remote logging. The downside I guess would be the need to configure a remote syslog host. Keep in mind that VMware provides a free syslog collector with the vCenter install media that can be used. Here’s some more information on how to configure remote logging with ESXi: http://kb.vmware.com/kb/2003322.
Note, when using a remote a syslog server be aware that if the logging stops there's nothing to warn you of this by default. Here's a post from William Lam (@lamw) at virtuallyGhetto that shows how to setup notification using a vCenter alarm. Good stuff…
