posted

2 Comments

In a previous blog post, vSphere 7 – ESXi System Storage Changes, we discussed the changes to the ESXi system storage layout. How the various partitions are consolidated into fewer, larger partitions that are expandable as well. A lot of inquiries came in for more information on what happens when upgrading to vSphere 7. Let’s take a closer look at what happens to the ESXi system storage when upgrading to vSphere 7.

Storage Requirements Upgrades

The boot media requirements differ between a new vSphere 7 install, and an upgrade to vSphere 7. As mentioned in the first blog post, there’s a requirement for boot media to run a 4 GB storage device at minimum, when upgrading to vSphere 7.  Even though 4 GB boot media devices are supported, let me emphasize that it is good practice to adhere to the boot media requirements for a fresh vSphere 7 installation (8 GB for USB or SD devices, 32 GB for other boot devices). 32 GB or higher boot devices are recommended, check out this KB article for more information.

All the scenarios in this diagram are supported when upgrading to vSphere 7. Again, the recommended boot device would be a high endurance disk or flash device.

Partition Layout

To quickly recap what’s in the previous blog post, let’s look at how the partition layout changed between vSphere 6.x and vSphere 7. The small & large core-dump, locker, and scratch disk are consolidated into the new ESX-OSData partition.

Whether you freshly install or upgrade to vSphere 7, the partition layout as shown in the diagram above is applied. This partitioning reflects what happens in the vSphere upgrade process when the ESXi system storage media is HDD or SSD. The (system storage related) upgrade steps are:

  1. Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
  2. Cleanup all system partitions, non-datastore partitions are not destroyed.
  3. If the upgrade media does not have an existing VMFS partition, the upgrade process creates a new GPT partition lay-out.
  4. Create partitions (book-banks and ESX-OSData)
  5. Restore the contents from RAM to the appropriate partitions.

Upgrade Scenarios

But what happens from a ESXi system storage perspective if you have ESXi installed on a USB or SD device together with a HDD/SSD or you when you have a USB-only system?

Upgrade Scenario : USB with HDD

When the storage media is a USB  or SD card, and the scratch partition is on HDD or SDD storage media, the upgrade process is as follows:

  1. Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
  2. Cleanup all system partitions, non-datastore partitions are not destroyed.
  3. If the upgrade media does not have a VMFS partition, create a GPT partition layout.
  4. Create partitions (book-banks and ESX-OSData)
    • The dedicated scratch partition is converted to the ESX-OSData partition
  5. Restore the contents from RAM to the appropriate partitions.

In this scenario, the scratch partition on the hard drive is converted to ESX-OSDATA. Its size is limited to 4 GB because of VFAT restrictions. This size might be too small for customers, who have large memory systems and require a large core dump file. In this case, customers can take the following actions:

  • Create a core dump file on a datastore. To create a core dump file on a datastore, see the KB article 2077516.
  • Assign scratch to a directory in the datastore. To assign the scratch location to a directory in a datastore, see KB article 1033696.

Upgrade Scenario : USB-only

Having a USB or SD-only device setup in your ESXi host, you can still upgrade to vSphere 7 if the storage device is at least 4 GB. Although a higher endurance and capacity device is strongly recommended. See this KB article for more insights storage endurance requirements. To support the 4GB minimum when upgrading to vSphere 7, there’s a couple of things happening with the storage partition layout.

Note: using vSAN in a cluster with ESXi hosts that have more than 512GB of memory require larger than 4 GB boot devices (when upgrading to vSphere 7) because of a larger core dump partition.

In the scenario of using a 4 GB boot device and no local disk is found, ESXi is running in a so-called ‘degraded mode‘. This means ESXi is not running in an optimal state, with some functionalities disabled. Also, running in a degraded state could mean ESXi loses its state on a power cycle. Solving this requires adding a local disk or flash device and run the instructions found in KB article 77009.

This diagram shows the typical setup when upgrading using USB boot media. This scenario is also applicable for a setup where the scratch location points to a datastore. For security reasons, ESX-OSData cannot exist in those locations. The upgrade steps using USB or SD media are:

  1. Request remediation of unavailable scratch volume:
    • The user is prompted to create a compatible scratch volume or add a spare disk. The upgrade process terminates if the user chooses to remediate.
  2. If remediation is ignored, then a fallback mode will be used:
    • ESX-OSData is created on USB.
      • USB flash MUST be >= 4GB otherwise upgrade will terminate because VMFS requires at least 1.3GB. This space is necessary for pre-allocation of the core file, vmtools and vSAN traces.
    • RAM-disk is used for frequently written data.
    • Subsystems that require persistence storage of data, implement an equivalent backup.sh capability to allow buffered saving of the data from RAM-disk to ESX-OSData
    • This is a highly degraded mode of operation with boot messages displaying so. The user must accept that there is potential for data to be lost because of the use of a RAM-disk which may be storing data that ESXi considers to be persistent across reboots.

The backup.sh script is run at regular intervals to save the system state and sticky bit files to the boot-bank.

First Boot Tasks

After the ESXi upgrade, or a new installation, the first boot tasks of the ESXi host are executed. These include:

  • Scan for unpartitioned disks and partition them as datastores.
  • Create symbolic links to file systems, ensuring that the necessary namespaces for subsystem are available.
  • Initialize subsystems to reference the correct storage location. For example, logs and core dump locations.

To Conclude

Thanks goes out to our engineering team for providing the information. The goal of this blog post is for you to have a better understanding of what is happening with the ESXi system storage when upgrading to vSphere 7. A key takeaway here is to make sure you meet the storage requirements when installing or upgrading to vSphere 7, and to use a recommended boot device.