Now that we have covered the considerations to backup each component in your vSphere Environment, it’s time to think about the rollback or restore process. In the previous post we discussed how to backup each individual component, in this post we will continue the conversation and discuss considerations to restore or rollback your vSphere Environment components.
Platform Services Controller / vCenter Server
When things don’t go right with an update, upgrade or configuration change of your Platform Services Controller or vCenter Server you need to understand the proper steps it takes to rollback. In these scenarios sometimes there may have been a failure in the process, or after a successful upgrade was completed, an incompatibility was found which would result in a rollback. A reason we see the need to rollback a vCenter Server is when a supporting solution such as a backup vendor or monitoring vendor is not compatible. In this case you will need to revert to the previous version.
Scenario 1: Embedded vCenter Server Appliance (No Enhanced Linked Mode)
When you need to rollback a Standalone vCenter Server with an Embedded Platform Services due to a failed upgrade or migration—shutdown the new vCenter Server Appliance, power on the original and rejoin to the domain if it previously was.
In the event the rollback occurs due to a patch or configuration change, the rollback process will include restoring from the file-based or image-based backup taken.
Scenario 2: Embedded vCenter Server Appliance in Enhanced Linked Mode
In case you happen to have multiple Embedded vCenter Servers in Enhanced Linked Mode, the rollback and recovery of this is a bit different than Scenario 1.
As the Embedded vCenter Server contains PSC related data we cannot just revert a snapshot of a vCenter Server running in linked mode. There are pre-built checks in the vCenter Server that will look for inconsistencies and if any major issues occur the PSC service could go into read-only mode.
In order to properly restore an Embedded vCenter Server that is in Enhanced Linked Mode we must perform a restore that was taken with an Image or File-Based Backup. The File-Based Restore wizard contains logic to detect if the node was in linked mode and instead of restoring the stale data, it will perform a reconciliation and synchronize the PSC data from another vCenter Server in the SSO domain instead of restoring it. If you restore an Embedded vCenter Server from an image-based backup you must log into the VAMI and manually run reconciliation.
Scenario 3: External Platform Services Controller
As we discussed in Scenario 2 if a vCenter Server had PSC data we did not want to restore old data. The same applies to a standalone External PSC, if we need to rollback a single PSC we will shut down the node and unregister it from the SSO domain. We will then redeploy the PSC using the same FQDN and IP and allow it to synchronize the data. The only time a PSC should be restored from backup is if all PSC’s within an SSO domain are unavailable.
Scenario 4: External vCenter Server
Last but not least as we wrap up we will talk about restoring an External vCenter Server. When you need to rollback an External vCenter Server due to a failed upgrade or migration—shutdown the new vCenter Server Appliance, power on the original and rejoin to the domain if it previously was. In the event you need to roll back from a patch or configuration change an External vCenter Server contains no PSC data, it is fine to perform a restore from an image based or file-based backup.
ESXI Server
Certain scenarios may require you to rollback an ESXi Server. Whether there was an issue with an update, upgrade or software installation we will cover two scenarios to rollback an ESXi Server.
Scenario 1: Reverting using the Built-in Recovery
The most common method used to rollback an ESXi Host is the built-in recovery. Using the Direct Console User Interface (DCUI) you have the ability to revert to a previous installation using the altbootbank. This is the simpler of the two scenarios, but a previous version is not always available. In the event a previous version is unavailable you will need to proceed with restoring using a configuration backup.
Scenario 2: Reverting using Configuration Backup
The next method to recover an ESXi Host is to restore a previous configuration. If an altbootbank is not available to use the built-in recovery you must first install ESXi Server from media. Using the configuration backup, you can perform a restore to bring the host back to its configured state. In some environments host profiles are not used, which is why having a configuration backup can make a restore simpler.
VMware Tools / VM Compatibility
Downgrading of VMware Tools is not supported. In the event of a need to revert to a previous version of VMware Tools, you must first uninstall the newer version of VMware Tools and then install the older version of VMware Tools.
Unlike VMware Tools there is no process to perform an in-place downgrade of VM Compatibility / Virtual Hardware. To downgrade your virtual machine hardware version you have 3 options:
Revert to a previous backup or snapshot of the Virtual Machine prior to upgrading virtual hardware.
Create a new virtual machine mirroring the original specifications using a previous hardware version and attach the existing disk from the virtual machine.
Use VMware Converter to perform a VM to VM conversion choosing a previous hardware version for the destination.
Storage (VMFS/VSAN)
When performing a rollback of a VMFS-6 volume it follows the same steps as an upgrade to VMFS-6. When Migrating VMFS-5 to VMFS-6 there was no in-place upgrade. The same applies to a downgrade. In order to downgrade a VMFS-6 volume to VMFS-5 you must first create a new VMFS-5 Volume and then perform a storage migration of Virtual Machines.
Generally, it is not recommended to downgrade a vSAN Disk Format. However, in some scenarios you may wish to format a vSAN Disk Group with a legacy format version. This process includes the ability to evacuate a disk group and choose a legacy disk version by modifying an advanced setting and then recreating a disk group. This is only applicable to vSAN 6.x versions.
Network (VDS)
Last but not least let’s review the scenario of having to rollback a previous vSphere Distributed Switch version. If you have ever performed an upgrade you may notice on the screen, that a distributed switch cannot be downgraded, however there is a process in order to perform a restore which will allow us to go back to a previous distributed switch version.
In order to “restore” a distributed switch configuration you must first create a Distributed Switch at the correct target version and choose the option to import configuration preserving the original distributed switch and port group identifiers. If you would attempt to restore a distributed switch configuration on top of the existing, it would leave the current distributed switch version intact.
Although we don’t like to anticipate any issues in our vSphere Environment, it is important to not only take regular backups of our environment, but to also practice restoring outside of any lifecycle activities. When thinking about backing up and restoring vSphere Components it is important to consider the different parts of your environment and the impact restoring that component will have in your environment.
If you have any questions, comments or feedback, feel free to leave them below or reach out on twitter at @davidstamen.
