Upgrading to VMware Virtual SAN 6.0

Virtual SAN 6.0 introduced new changes to the structural components of its architecture. One of those changes is a new on-disk format which delivers better performance and capability enhancements. One of those new capabilities allows vSphere Admins to perform in-place rolling upgrades from Virtual SAN 5.5 to Virtual SAN 6.0 without introducing any application downtime.

Upgrading an existing Virtual SAN 5.5 cluster to Virtual SAN 6.0 is performed in multiple phases and it requires the re-formating of the of all of the magnetic disks that are being used in a Virtual SAN cluster. The upgrade is defined as a one-time procedure that is performed from RVC command line utility with a single command.

Upgrade Phase I: vSphere Infrastructure Upgrade

This phase of the upgrade is all components are upgraded to the vSphere 6.0 release. All vCenter Servers and ESXi hosts and all infrastructure related components need to be upgraded to version their respective and corresponding 6.0 software release. Any of the vSphere supported procedures for the individual components is supported.

• Upgrade vCenter Server 5.5 to 6.0 first (Windows or Linux based)
• Upgrade ESXi hosts from 5.5 to 6.0 (Interactive, Update Manager, Re-install, Scripted Updates, etc)
• Use Maintenance Mode (Ensure accessibility – recommended for reduced times, Full data migration – not recommended unless necessary

Upgrade Phase II: Virtual SAN 6.0 Disk Format Conversion (DFC)

This phase is where the previous on-disk format (VMFS-L) is replaced on all of the magnetic disk devices with the new on-disk format (VSAN FS). The disk format conversion procedures will reformat the disk groups and upgrade all of the objects to the new version 2. Virtual SAN 6.0 provides supports for both the previous on-disk format of Virtual SAN 5.5 (VMFS-L) as well as its new native on-disk format (VSAN FS).

While both on-disk formats are supported, it is highly recommended to upgrade the Virtual SAN cluster to the new on-disk format in order to take advantage of the performance and new available features. The disk format conversion is performed sequentially performed in a Virtual SAN cluster where the upgrade takes place one disk group per host at a time. The workflow illustrated below is repeated for all disk groups on each host before the process moves onto another host that is a member of the cluster.

Before initiating the disk format conversion procedure (VSAN upgrade) ensure that all of the vSphere and Virtual SAN 5.5 cluster pre-requisites are satisfied and the cluster is prepared for the disk format conversion.

• vCenter Server 5.5 has been upgraded to vCenter 6.0
• All the ESXi hosts that are members of a Virtual SAN 5.5 cluster have been successfully upgraded to ESXi 6.0.
• Cluster add disk to storage claim rule is set to manual

Upgrade Procedure

The disk format conversion is manually initiated by a vSphere admin from the RVC console. Login to the vCenter Server and start RVC as illustrated in the image below.

As a precautionary step before initiating the disk format conversion from RVC, check the status of the cluster components listed below and make sure that everything checks out ok.

• Cluster Health – make sure that all host in the cluster have the right and matching product, VSAN cluster enabled status, Cluster info, Storage Info, Disk Mappings, and Network Information. From RVC use the vsan.cluster_info command.

• Cluster Disk Status – make sure that the disks of all of the hosts in the cluster are all healthy and currently formatted with on-disk format v1 (VMFS-L). From RVC use the vsan.disks_stats command.

• Inaccessible Objects, and Out of Sync Virtual Machines Status – make sure that all objects and virtual machines in the cluster are accessible and there are no virtual machines vmx files out of sync. From RVC use the vsan.check_state command.

In the event an error is reported by any of the check points identified above, correct them before proceeding with the on-disk format conversion. Failing to do so will halt the disk format conversion.

If everything checks out in good health initiate the upgrade procedure by using the RVC command vsan.v2_ondisk_upgrade. By default, the upgrade operation moves all components from the disk groups before formatting. This operation requires spare capacity as well as additional hosts in order to maintain virtual machines availability compliance (FTT=1) during the upgrade.

In the event there isn’t enough available capacity or enough hosts in the cluster to maintain virtual machines availability compliance (4 or more hosts) use the  allow-reduced-redundancy option. This option does not evacuate all the data on the disk groups. It only evacuates enough data to ensure the objects are still available. The use of the allow-reduced-redundancy option is not recommended unless is absolutely necessary.

For example in a three host cluster with single disk group per host. The allow-reduce-redundany options exposes the risks of a potential outage as virtual machines will be unprotected for the duration of the upgrade. The DEFAULT MODE is the RECOMMENDED option to upgrade.

The are a series of pre-checks and validation operations that take place before the disk format conversion is initiated. The actions listed below are performed by the vsan.v2_ondisk_upgrade command as part of the upgrade procedure:

• All hosts in the cluster are connected to vCenter Server
• All hosts have been upgraded to ESXi 6.0
• All hosts are part of the same Virtual SAN cluster
• All objects are accessible in the cluster
• There are no network partition in the cluster
• Identify unhealthy magnetic disks in the cluster
• No mismatch between Virtual SAN cluster and vCenter Server
• No host with auto-claim storage
• On-disk format version

Its important to note that once all of the pre-checks are performed the Virtual SAN Cluster Monitoring, Membership, and Directory Service (CMMDS) will not allow any ESXi 5.5.x hosts join the cluster.