At VMworld I was surprised at the number of questions I was asked related to booting ESXi off of USB/SD. Unfortunately, the vSphere docs largely assume you will use a local disk and don't really cover USB/SD, as such it’s not always easy to find answers to common questions. Below are some answers to several of the more popular USB/SD related questions.
1. What size USB/SD should I use?
The minimum disk size required to install ESXi is 1GB. When booting USB/SD there is little benefit to using a larger device because any space beyond the 1st GB will go unused. When choosing a USB/SD boot device, it’s not the size of the device that is important but the reliability of the device. Be sure to use good quality USB/SD devices.
2. How long will a USB/SD device last?
Unlike a local disk or SAN LUN, USB/SD devices are sensitive to excessive amounts of I/O as they tend to wear over time. This naturally raises a concern about the life span of the USB/SD device. When booting from USB/SD keep in mind that once ESXi is loaded, it runs from memory and there is very little ongoing I/O to the boot device. The only reoccurring I/O is when the host configuration is routinely saved to the USB/SD device, which by default is done once every 10 minutes. Based on how often you reboot the host and install patches it is expected that a good quality USB should last for several years.
3. Where does the scratch partition get placed when booting from USB?
Because USB/SD devices are sensitive to high amounts of I/O the installer will not place the scratch partition on a USB/SD device. Instead, the installer first scans for a local 4GB vfat partition, if it doesn’t find one it will then scan for a local VMFS volume on which to create a scratch directory. If no local vfat partition or VMFS volume is found, as a last resort the installer will put the scratch partition in “/tmp/scratch” (i.e. put scratch on the local ramdisk). If this happens it’s a good idea to manually reconfigure the scratch partition after the install.
4. When manually configuring the scratch partition where should I put it?
You can place the scratch partition on any available datastore (NFS, FC, iSCSI, local). Use the vSphere client to configure the ScratchConfig.ConfiguredScratchLocation to a valid VMFS directory (requires a reboot).
5. What are there performance considerations needed when choosing a location for the scratch partition?
Performance of the scratch partition is not critical. Scratch is used for log files, core dumps, support log bundles, and as a temporary working space. It’s not imperative that the scratch partition be on a high performing device. The concern is to have scratch on a reliable, persistent location.
6. What is the impact of running with a scratch partition vs. a scratch volume, vs.leaving scratch on ramdisk?
Operationally, there is no impact from running scratch on a partition vs. as part of a VMFS volume, or on ramdisk. The important issue is to avoid having scratch in memory as this can complicate troubleshooting because the log files will be lost each time the host reboots.
7. How do I know if my scratch partition is running in memory vs. on a persistent datastore?
You can check the location of the scratch partition using the vSphere client by selecting the host and choosing Configuration -> Advanced Settings -> ScratchConfig (see screen shot above).
Another easy way to check the location of the scratch partition is to logon to the ESXi shell and list the /scratch directory. The scratch directory will be linked to the assigned disk partition or VMFS volume. If /scratch is linked to "/tmp/scratch", your scratch partition is in memory and should be changed.
8. If I setup the ESXi Syslog Collector and Dump Collector can I leave the scratch partition in memory?
If you setup the ESXi syslog and dump collectors (new in vSphere 5.0) and are relying on remote logging for your logs and dump files, you may consider leaving the scratch partition in memory (/tmp/scratch). However, do keep in mind that if you ever run into a situation where the /tmp directory fills up you could run into problems. Also, be sure to factor in how a network issue could impact your ability to access log files and troubleshoot issues.
9. What are the limitations when running embedded ESXi vs. installed ESXi?
There are no limitations. The only difference is with embedded ESXi the server vendor has pre-installed ESXi on a USB/SD device for you. The thing to keep in mind is that vendors who provide embedded ESXi will often include additional software packages with their embedded ESXi image, this can be an issue when doing upgrades as you will either need to get an updated image from the vendor (which could introduce a delay in getting access to patches and updates) or you may need to use the ESXi Image Builder CLI to manually add the extra packages to the ESXi image you download from VMware.
10. Can I perform a scripted install to a USB/SD device?
Yes, the ability to perform a scripted install to USB/SD was added in ESXi 5.0.
