Home > Blogs > VMware vSphere Blog


Booting ESXi off USB/SD

At VMworld I was surprised at the number of questions I was asked related to booting ESXi off of USB/SD.  Unfortunately, the vSphere docs largely assume you will use a local disk and don't really cover USB/SD, as such it’s not always easy to find answers to common questions.  Below are some answers to several of the more popular USB/SD related questions.

 

1.  What size USB/SD should I use?

The minimum disk size required to install ESXi is 1GB. When booting USB/SD there is little benefit to using a larger device because any space beyond the 1st GB will go unused.  When choosing a USB/SD boot device, it’s not the size of the device that is important but the reliability of the device.  Be sure to use good quality USB/SD devices.

 

2.  How long will a USB/SD device last?

Unlike a local disk or SAN LUN, USB/SD devices are sensitive to excessive amounts of I/O as they tend to wear over time.  This naturally raises a concern about the life span of the USB/SD device.  When booting from USB/SD keep in mind that once ESXi is loaded, it runs from memory and there is very little ongoing I/O to the boot device. The only reoccurring I/O is when the host configuration is routinely saved to the USB/SD device, which by default is done once every 10 minutes.  Based on how often you reboot  the host and install patches it is expected that a good quality USB should last for several years.

 

3.  Where does the scratch partition get placed when booting from USB?

Because USB/SD devices are sensitive to high amounts of I/O the installer will not place the scratch partition on a USB/SD device.  Instead, the installer first scans for a local 4GB vfat partition, if it doesn’t find one it will then scan for a local VMFS volume on which to create a scratch directory.  If no local vfat partition or VMFS volume is found, as a last resort the installer will put the scratch partition in “/tmp/scratch” (i.e. put scratch on the local ramdisk).  If this happens it’s a good idea to manually reconfigure the scratch partition after the install.

 

4.  When manually configuring the scratch partition where should I put it?

You can place the scratch partition on any available datastore (NFS, FC, iSCSI, local).  Use the vSphere client to configure the  ScratchConfig.ConfiguredScratchLocation to a valid VMFS directory (requires a reboot).

  A1

5.  What are there performance considerations needed when choosing a location for the scratch partition?

Performance of the scratch partition is not critical.  Scratch is used for log files, core dumps, support log bundles, and as a temporary working space.  It’s not imperative that the scratch partition be on a high performing device.  The concern is to have scratch on a reliable, persistent location.

 

6.  What is the impact of running with a scratch partition vs. a scratch volume, vs. leaving scratch on ramdisk?

Operationally, there is no impact from running scratch on a partition vs. as part of a VMFS volume, or on ramdisk.  The important issue is to avoid having scratch in memory as this can complicate troubleshooting because the log files will be lost each time the host reboots.

 

7.  How do I know if my scratch partition is running in memory vs. on a persistent datastore?

You can check the location of the scratch partition using the vSphere client by selecting the host and choosing Configuration -> Advanced Settings -> ScratchConfig (see screen shot above).

Another easy way to check the location of the scratch partition is to logon to the ESXi shell and list the /scratch directory.  The scratch directory will be linked to the assigned disk partition or VMFS volume.  If /scratch is linked to "/tmp/scratch", your scratch partition is in memory and should be changed.

A2

8. If I setup the ESXi Syslog Collector and Dump Collector can I leave the scratch partition in memory?  

If you setup the ESXi syslog and dump collectors (new in vSphere 5.0) and are relying on remote logging for your logs and dump files, you may consider leaving the scratch partition in memory (/tmp/scratch).  However, do keep in mind that if you ever run into a situation where the /tmp directory fills up you could run into problems.  Also, be sure to factor in how a network issue could impact your ability to access log files and troubleshoot issues.

 

9.  What are the limitations when running embedded ESXi vs. installed ESXi?  

There are no limitations.  The only difference is with embedded ESXi the server vendor has pre-installed ESXi on a USB/SD device for you.  The thing to keep in mind is that vendors who provide embedded ESXi will often include additional software packages with their embedded ESXi image, this can be an issue when doing upgrades as you will either need to get an updated image from the vendor (which could introduce a delay in getting access to patches and updates) or you may need to use the ESXi Image Builder CLI to manually add the extra packages to the ESXi image you download from VMware.

 

10.  Can I perform a scripted install to a USB/SD device?

Yes, the ability to perform a scripted install to USB/SD was added in ESXi 5.0.

12 thoughts on “Booting ESXi off USB/SD

  1. Robert QC

    I try to use a 128GB USB stick as to have a ESXi to go. It shutdown by itself on the boot process. Is there a MAXIMUM USB size to be able to run a ESXi 5.1 ?

    Reply
  2. Leo

    I am booting ESXi 5.1.0 from Verbatim USB stick and on every reboot the configuration is lost. I am not sure why it cannot preserve the configuration on the usb device. It also is stuck for 7 minutes during the boot on Running usbarbitrator start. Maybe this is causing it to loose the configuration. I am booting ESXi on Dell XPS 15 laptop. Any idea how to fix this?
    Thanks. Leo

    Reply
    1. vegara

      I had a similar problem to yours running the hypervisor on a shuttle pc thought a USB stick (long delay with the usbarbitrator + no configuration saved with reboots). The whole problem was solved when i switched the usb stick from a usb 3.0 port to a usb 2.0 port (i remember that i could not get the instalation done using the usb 3.0 port as well). Since i also have a Dell XPS 15 laptop, i suggest that you use the USB stick with the eSata/USB 2.0 port and not the usb 3.0 (blue ones). btw: why are you running a hypervisor on a laptop?

      Reply
  3. charlie hay

    i utilized a 16G, 2G worked fine. when i changed to 16G, formatting broke. so it appeared that installer wanted datastore somewhere other than usb. i inserted a hd into server. reran installer, pointed datastore to hd, works now. let me know if you have any clues ot how large of usb key is permitted.

    Reply
  4. Pingback: vSphere 4.1 to 5.1 lessons learned | Tech Talk

  5. Gaurav

    Hey thanks Kyle….But I have a Question If i have installed esxi 5.0 on 1 GB USB and I want to upgrade Is it possible

    Reply
  6. Pingback: نصب Esx روی USB

  7. Ashraf

    Hi,

    I’ve a question and wondering if any one can help me with my issue. I built a ESXi 4 some times back and used to boot from USB drive. Somehow i lost USB drive and unable to boot the Server. Can any one pleases guide me on booting the Server?

    Thank you,

    Reply
    1. Jason Wohl

      Not responsible for any data loss, but I believe that if you were to use the same installer disk or use an installer disk of the same version and just install it onto another usb stick then you will be able to boot the server and keep all your datastores. Make sure you don’t reformat anything during install and you should be fine

      Reply
  8. Pingback: TinkerTry IT @ home | Rufus takes under a minute to create a bootable USB flash drive for ESXi installation

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>