Upgrading 2-node vSAN Clusters from 6.7U3 to 7U1 with a Shared Witness Host

Shared Witness Host for 2-node vSAN Clusters

vSAN 7 Update 1 supports a shared witness host for 2-node vSAN clusters. The Shared Witness for 2-Node vSAN Deployments blog article discusses the topic in more detail. In this article, we will take a look at upgrading a 2-node deployment from vSAN 6.7U3 to vSAN 7U1 including witness host consolidation. A vSAN shared witness host for 2-node clusters reduces compute and storage requirements for multiple 2-node vSAN clusters. This lowers cost and complexity–especially in cases where you have tens or hundreds of 2-node deployments.

Shared Witness Host for 2-node vSAN Clusters

vSAN currently supports up to 64 2-node configurations per shared witness. As with many supported limits, just because you can doesn’t mean you should. I recommend starting with a consolidation ratio of 10-20 2-node clusters per shared witness host. Take some time to confirm everything is running well before performing further consolidation. Also consider higher consolidation ratios might call for a larger shared witness host appliance. The good news is it is easy to add, change, and remove vSAN witness hosts since they are virtual appliances.

Verify Your Upgrade Path and Read the Release Notes

Below we see a couple of 2-node vSAN clusters running in datacenter-01. Each cluster has a witness host. The vCenter Server version is 6.7U3. The ESXi version is 6.7U3.

2-node vSAN clusters with witness hosts

It is important to verify the vCenter Server and ESXi upgrade paths are supported. VMware Knowledge Base (KB) article vSphere Back-in-Time Release Upgrade Restriction (67077)  provides guidance on supported upgrade paths.

In a few cases there is no upgrade path from 6.7U3 to 7.0U1. This sometimes happens if an update for a previous version comes out after the latest version. For example, vCenter Server 6.7U3l (build 17138064 on Nov. 19, 2020) was released after vCenter Server 7.0U1a (build 17004997 on Oct. 22 2020). In this case, you would need to wait until an update is released for vCenter Server 7.0U1 that supports an upgrade from vCenter Server 6.7U3l.

Always read the release notes on the product version you are upgrading to before performing an upgrade. Also, see KB articles vSphere 7 Upgrade Best Practices (78205) and Update sequence for vSphere 7.0 and its compatible VMware products (78221).

Perform the vCenter Server Upgrade

The vCenter server upgrade documentation is here. Ensure DNS and NTP are configured properly throughout your environment. IMPORTANT: Verify health checks for vSphere and vSAN are all green. Fix any issues before proceeding.

vSAN Skyline Health

I am not going to cover the vCenter Server upgrade in detail since this procedure is already covered in the documentation. I did an upgrade to the most recent release of vCenter Server to get the latest features, fixes, and enhancements.

Upgrading with vSphere Lifecycle Manager in vSphere 7

After upgrading vCenter Server, I am now running version 7.0U1a and I have two 2-node vSAN clusters running ESXi 6.7U3. Lifecycle management in vSphere 7 is much better with the introduction of vSphere Lifecycle Manager (vLCM). Similar to the vCenter Server upgrade, I will not get into the details of how to configure and run vLCM as that is covered in the vSphere documentation. For this article, I created an Upgrade Baseline in vLCM using the iso for ESXi 7.0U1. I assigned the new baseline to the clusters I plan to upgrade. After attaching the baseline and checking for compliance, I see that the cluster is, of course, not in compliance with the ESXi 7.0U1 baseline.

vLCM cluster non-compliant

Keep in mind the vSAN witness host virtual appliance is not part of the cluster. A witness host must be updated separately. You will also need to turn off vSphere HA prior to remediating a cluster. vSphere HA admission control commonly prevents the hosts in a 2-node cluster from entering maintenance mode, which means the remediation will fail. I have also seen a few cases where you have to put each host in maintenance mode and perform the host upgrade–one at a time, of course. This is discussed in the vSphere 6.7 documentation. Be sure to re-enable vSphere HA when the upgrades are complete.

After the two hosts in the cluster are upgraded, you might notice vSAN Skyline Health recommending on-disk format upgrades. You might also see the recommendation to upgrade the witness host software (ESXi) version. The link to upgrade the on-disk format is grayed out as we need to upgrade the witness host first.

vSAN Skyline Health disk format upgrade

I place the witness host in maintenance mode, attached the 7.0U1 baseline, and performed the upgrade. After the upgrade of the witness host from 6.7 to 7.0, you will notice a warning about the embedded OEM license that comes with a vSAN witness host.

vSAN witness host license assignment warning

That is because the embedded 6.7 license key is not compatible with version 7. However, the witness host is still functioning properly and the 2-node cluster is protected by the witness functionality. Now is a good time to deploy a new 7.0U1 witness host that can be shared across multiple 2-node clusters. The new appliance will contain an embedded OEM license that is compatible with ESXi 7.

If you try to change the witness host to a host running version 7.0U1 before upgrading the on-disk format, you will likely see the error below. You must upgrade the on-disk format of the cluster before switching to the new 7.0U1 witness host.

vSAN disk format converter legacy objects

Finish the Upgrade and Consolidate Witness Hosts

The on-disk format upgrade can be completed when the two hosts and the witness host are running the same version. After the on-disk format upgrade is complete, you can then change the witness host to a newly deployed 7.0U1 witness host. The new witness host can be assigned to multiple 2-node clusters. Once the witness host has been changed from the old one to the new one, you can power off the old witness host and delete it to reclaim compute and storage capacity.

Last, but not least, verify the vSphere and vSAN health checks are all ok. Remedy any issues that you discover to help ensure stability and performance across all of your vSAN clusters. As always, open a support request (SR) with VMware Global Support Services (GSS) if you run into issues that aren’t easy to resolve.

For larger-scale operations and upgrades, changing the vSAN witness host can be automated using PowerCLI. See PowerCLI vSAN Witness Replacement to get started.