Upgrading VMware Cloud Foundation™ is a common activity among VMware Technical Account Management customers.
As a Technical Account Manager (TAM), I have been recently asked by one of my customers to help them with their VMware Cloud Foundation Upgrade Plan. As we were hitting these vulnerabilities, VMSA-2022-0030 and VMS-2022-033, the question asked was do we need to upgrade the full stack or not?
Before answering that question and providing the upgrade process, there are a few clarifications which need to be addressed.
- Bill of Materials
- VMware Cloud Foundation™ SDDC Manager™ Patching Process
- Async Patch Tool
- Preparing for Recovery for Applied Patches
Clarifications
Bill of Materials (BOM):
VMware Cloud Foundation includes a prescriptive bill of materials (BOM) which conform to VMware Validated Design and are fully tested for compatibility.
VMware Cloud Foundation SDDC Manager automates the life cycle management of all components deployed through VMware Cloud Foundation.
Each release of VMware Cloud Foundation includes an updated BOM. VMware Cloud Foundation SDDC Manager is responsible for updating software components to match the updated BOM. It is important that all software updates are performed using VMware Cloud Foundation SDDC Manager to ensure full compatibility and consistency with inventory information.
VMware Cloud Foundation SDDC Manager Patching Process
The first thing that happens is that update bundles are released by VMware. These bundles are released on a regular basis to the VMware depot for download. These are all tested and validated by VMware to work together as a single integrated platform.
Once available, VMware Cloud Foundation SDDC Manager provides notifications when a new update is available for download applicable to the environment that it is operating in.
A pre-check can then be run prior to starting an update, which ensures the components being upgraded are healthy and optimal. The pre-check will notify the user of any potential issues before starting the upgrade.
Using this information, updates can be scheduled to run immediately, or they can be scheduled on a specific date and time to run within the chosen maintenance windows.
Repeat until all patches are completed and new bundles are released. VMware Cloud Foundation Lifecycle Management provides a lot of flexibility and granularity in terms of how to patch the full software stack within the VMware Cloud Foundation platform.
Async Patch Tool:
The Async Patch Tool enables patches for individual components via VMware Cloud Foundation SDDC Manager and makes recovering from patching easy.
It will support async patching for VCF 4.2.1 and later.
Async Patch Tool only enables the Patch and doesn’t perform an actual patching. Once the patch for a component has been enabled, the upgrade (patching) would still be done in a traditional way using the VMware Cloud Foundation SDDC Manager UI (LCM).
For example, you could use the Async Patch Tool to get a VMware vCenter Server® patch that addresses a critical security issue as described in a VMware Security Advisory (VMSA).
You use the Async Patch Tool to download the patch and upload it to the internal LCM repository on the VMware Cloud Foundation SDDC Manager appliance. Then you use the VMware Cloud Foundation SDDC Manager UI to apply the patch.
The Async Patch Tool has two modes, online and offline, meaning if you don’t have internet connectivity at your site for whatever reason you can still use this tool to patch like the LCM offline bundle utility.
Patches can then be applied via VMware Cloud Foundation SDDC Manager’s LCM automation and you do not need to apply the patch directly via a product interface.
Therefore, patched components will have different versions than those listed in the Bill of Materials (BOM).
Preparing for Recovery for Applied Patches: To ensure that forward upgrades can proceed successfully after a patch has been applied, the Async Patch Tool automates version aliasing and inventory updates, ensuring correct configuration. Run the Enable Upgrade workflow using the tool and specify the target VMware Cloud Foundation version for the forward upgrade path you wish to enable.
As VMware Cloud Foundation releases are blocked once SDDC has deviated from the VMware Cloud Foundation release BOM, upgrade enabler does the version configuration changes on the VMware Cloud Foundation LCM service so that the next applicable release becomes available for upgrade. This step is performed after you have applied all the enabled patches.
For more info check VCF Async Patch Tool
By that we solved the below challenges:
- Keeping up with the high velocity of critical security patches and internal policies for rapid rollout
- Long qualification cycles that make it impractical to simply skip upgrade and consume security patches bundled with new releases.
Full Stack Upgrade or Patching – Which is the Right Approach?
To decide if a full stack upgrade is required or if only patching the individual component, we will refer to this Knowledge Base article.
In this scenario, Current Versions are:
| Software Component | Version | Build Number |
| VMware Cloud Foundation SDDC Manager | 4.4.1.1 | 19766960 |
| VMware vCenter® Server Appliance™ | 7.0 Update 3g | 19480866 |
| VMware ESXi™ | 7.0 Update 3g | 19482537 |
| VMware NSX-T™ Data Center | 3.1.3.7.4 | 19762317 |
| VMware vRealize Suite Lifecycle Manager | 8.6.2 PSPAK 3 | 19447709 |
Our Target Version:
VMSA-2022-0030 details vulnerabilities in vCenter Server 6.7 and 7.0 and VMware ESXi 6.7 and 7.0. These vulnerabilities are remediated in the releases vCenter Server 7.0 Update 3i and ESXi 7.0 Update 3i.
VMSA-2022-0033 details vulnerabilities in VMware ESXi 7.0. These are remediated in VMware ESXi 7.0 Update 3i.
For example, if VMware Cloud Foundation is on Version 4.4.1 as in this scenario, looking at the KB article and then at VMware Cloud Foundation Version 4.4.1, we can see that the target vCenter Patch and ESXI Patch apply to VMware Cloud Foundation 4.4.x as below.
In this scenario, the answer is that we don’t need to upgrade the full stack.
How to Download Async Patch Tool
Download the Async Patch Tool to a computer that has access to the VMware Cloud Foundation SDDC Manager appliance.
- Log in to VMware Customer Connect™ and browse to the Download VMware Cloud Foundation page.
- In the Select Version field, select your current version of VMware Cloud Foundation.
- Click Drivers and Tools.
- Expand VMware Cloud Foundation Tools and click Go To Downloads in the Async Patch Tool row.
- Click Download Now.
rm -r /home/vcf/asyncPatchTool
rm -r <Output directory>
Async Patch Online Mode:
- Create the asyncPatchTool directory: mkdir /home/vcf/asyncPatchTool
- Copy the Async Patch Tool file (vcf-async-patch-tool-<version>.tar.gz) that you downloaded to the /home/vcf/asyncPatchTool directory.
- Navigate to /home/vcf/asyncPatchTool and extract the contents of vcf-async-patchtool-<version>.tar.gz
Once you have untarred the tar file, you should see three folders: bin, conf, and lib. - Set the permissions for the asyncPatchTool directory as below:
- Navigate to /home/vcf/asyncPatchTool/bin.
- Run the following command: ./vcf-async-patch-tool –listAsyncPatch –du customer_connect_email
(Replace customer_connect_email with your VMware Customer Connect email address)
• Enable an async patch by running the following command:
./vcf-async-patch-tool -e –patch product:version –du customer_connect_email –sddcSSOUser SSOuser –sddcSSHUser vcf –it ONLINE
• Enter Y to confirm that you are running the latest version of the Async Patch Tool.
Read the information and enter Y to acknowledge the pre-requisites.
When it reaches “Current upload status” stage you will start to see tasks appear in VMware Cloud Foundation SDDC Manager.
After the async patch is successfully applied, use the Async Patch Tool to deactivate the patch.
After you apply an async patch, and one or more of the VMware Cloud Foundation component versions deviates from the BOM, you must use the Async Patch Tool to enable an upgrade to a later version of VMware Cloud Foundation.
You cannot enable async patches or enable VMware Cloud Foundation upgrades if your VMware Cloud Foundation instance already has any async patches enabled.
• Run the below command and replace SSOuser with the SSO user account, for example, [email protected].
./vcf-async-patch-tool –disableAllPatches –sddcSSOUser SSOuser –sddcSSHUser vcf
Please confirm: is AP Tool running the latest version(Y/N)?: Y
CEIP is disabled on your SDDC environment, do you want to enable CEIP only for Async Patch Tool (Y/N)? N
Then enter the required passwords.
Async Patch Offline Mode:
If your VMware Cloud Foundation SDDC Manager appliance does not have a connection to the internet, you can run the Async Patch Tool from a computer that does. Download an async patch, copy the patch and the Async Patch Tool to the VMware Cloud Foundation SDDC Manager appliance, and enable the patch. You can then use the VMware Cloud Foundation SDDC Manager UI to apply the patch to all workload domains.
Check steps needed to apply an Async Patch to VMware Cloud Foundation in Offline Mode.
This VMware Cloud Foundation demo covers the process step by step.
The next section explains that if you are going to do a full stack upgrade, you need to apply an async patch for the VMware vCenter Appliance and hosts.
For Full Stack Upgrade:
The management domain in your environment must be upgraded before you upgrade VI workload domains. In order to upgrade to VMware Cloud Foundation 4.5, all VI workload domains in your environment must be at VMware Cloud Foundation 4.2.1 or higher. If your environment is at a version lower than 4.2.1, you must upgrade the workload domains to 4.2.1 and then upgrade to
4.5.
Upgrade Component Order:
- VMware Cloud Foundation SDDC Manager is the first component that must be updated.
- The management domain components are usually second to be updated.
- Finally, workload domains are typically the last to be upgraded.
- The order may change based on the product requirements. VMware Cloud Foundation SDDC Manager always follows a validated path that is supported by VMware Engineering teams for the release.
- vRealize Components are updated optionally through VMware Aria Suite Lifecycle™ (formally VMware vRealize Suite Lifecycle Manager) starting from VMware Cloud Foundation 4.4
We will list first the current versions.
Current Versions
| Software Component | Version | Build Number |
| VMware Cloud Foundation SDDC Manager | 4.4.1.1 | 19766960 |
| VMware vCenter Server Appliance | 7.0 Update 3g | 19480866 |
| VMware ESXi | 7.0 Update 3g | 19482537 |
| VMware NSX-T Data Center | 3.1.3.7.4 | 19762317 |
| VMware vRealize Suite Lifecycle Manager | 8.6.2 PSPAK 3 | 19447709 |
Target Versions after Full Stack Upgrade
| Software Component | Version | Build Number |
| VMware Cloud Foundation SDDC Manager | 4.5 | 20612863 |
| VMware vCenter Server Appliance | 7.0 Update 3h | 20395099 |
| VMware ESXi | 7.0 Update 3h | 20328353 |
| VMware NSX-T Data Center | 3.2.1.2 | 20541212 |
| VMware vRealize Suite Lifecycle Manager | 8.8.2 | 20080494 |
Versions after applying Async Patch to the vCenter and Host
| Software Component | Version | Build Number |
| VMware Cloud Foundation SDDC Manager | 4.5 | 20612863 |
| VMware vCenter Server Appliance | 7.0 Update 3i | 20845200 |
| VMware ESXi | 7.0 Update 3i | 20842708 |
| VMware NSX-T Data Center | 3.2.1.2 | 20541212 |
| VMware vRealize Suite Lifecycle Manager | 8.8.2 | 20080494 |
Before you can upgrade VMware Cloud Foundation, you must download the upgrade bundles for each VMware Cloud Foundation component that requires an upgrade.
Upgrade Bundle:
Update bundles are used during an upgrade. An update bundle contains everything needed to update the appropriate VMware Cloud Foundation software components in the management domain or in a VI workload domain. An upgrade bundle is applied to the management domain first before it can be applied to workload domains.
There are also two ways to get patches Online Lifecycle Management and Offline Lifecycle Management. Let’s explore each.
Online Lifecycle Management:
Step 1: Authenticate to the VMware depot.
Select Bundle Management then click on the My VMware Account link.
Step 2: Download applicable bundles.
Step 3: Install the bundle, or schedule for install.
For more info check Online Bundle Download
Offline Lifecycle Management:
Used when VMware Cloud Foundation SDDC Manager does not have access to the VMware depot. Manually download update bundles and transfer them to VMware Cloud Foundation SDDC Manager Command line tool, shipped with VMware Cloud Foundation SDDC Manager (/home/vmware/vcf/lcm/lcm-tools)
Step 1: Generate marker file on VMware Cloud Foundation SDDC Manager
Step 2: Copy tool and marker file to computer with internet access
Step 3: Execute tool to download applicable update bundles
Step 4: Transfer bundle(s) to VMware Cloud Foundation SDDC Manager repository
Step 5: Install the bundle, or schedule for install
For more info check Offline Bundle Download for VMware Cloud Foundation
PRECHECK
Before we start, we need to pre-check to validate each component in the SDDC making sure that any known issues are addressed prior to seeing a failure during upgrade.
• Navigate to the Updates/Patches tab of the management domain and click PRECHECK.
VMware Cloud Foundation SDDC Manager Upgrade
- VMware Cloud Foundation SDDC Manager is in control of the update process and versions.
- This occurs in two phases:
- VMware Cloud Foundation SDDC Manager Upgrade Phase – This is where all Services on the VMware Cloud Foundation Appliance are updated. The console is unavailable, and an upgrade Splash Screen is shown.
- Configuration Drift Upgrade Phase – This is where any last-minute configuration changes or bug fixes are applied to the VMware Cloud Foundation SDDC Manager configuration. Note that there can be more than one configuration drift update, although this is not frequent. The console is not locked in this phase; however, once the update has started, workflows cannot be run until it is completed.
Before we start, we need to pre-check to validate each component in the SDDC, making sure that any known issues are addressed prior to seeing a failure during upgrade.
- Navigate to the Updates/Patches tab of the management domain and click PRECHECK.
- Then go to Available Updates section, which displays the offline bundle that you uploaded to VMware Cloud Foundation SDDC Manager before starting the upgrade.
- Click UPDATE NOW. The first available update would always be the VMware Cloud Foundation SDDC Manager update. The update can be either scheduled or can be initiated immediately.
- Click VIEW UPDATE ACTIVITY to view the detailed tasks then click Finish.
Check out this demo which covers upgrading the VMware Cloud Foundation SDDC Manager.
VMware NSX-T Data Center Upgrade
NSX-T Data Center Manager update becomes available after the VMware Cloud Foundation SDDC Manager is successfully updated. Follow the same VMware Cloud Foundation SDDC Manager update process to complete the NSX-T Data Center Manager upgrade.
- There are four stages to an NSX-T Data Center upgrade that are performed by VMware Cloud Foundation SDDC Manager.
- NSX Upgrade coordinator upgrade and preparation
- NSX Edge Cluster Upgrade(s)
- Host Cluster Upgrade
- NSX Management Cluster
Check steps needed for Upgrade NSX-T Data Center
VMware vCenter Server Upgrade
vCenter Server update becomes available after the NSX-T Data Center Manager is successfully updated.
- Navigate to the Updates/Patches tab of the domain you are upgrading.
- Run the upgrade precheck.
- In the Available Updates click Update Now
VMware ESXi Upgrade
ESXi host upgrades are the second thing that takes the longest time aside from NSX-T Data Center because a host evacuation and maintenance mode cycle is required.
By default, the upgrade process upgrades the ESXi hosts in all clusters in a workload domain in parallel. If you have multiple clusters in the management domain or in a VI workload domain, you can select the clusters to upgrade. You can also choose to upgrade the clusters in parallel or sequentially.
Check this link for more information on upgrading VMware ESXI.
You can also view this VCF LCM Upgrade ESXi v1demo which covers the process.
VMware Aria Suite Lifecycle Upgrade
Once VMware Cloud Foundation 4.4 is deployed, VMware Aria Suite Lifecycle component patching and upgrades are lifecycle managed through vRSLCM and no longer appear in VMware Cloud Foundation SDDC Manager as part of an updated VMware Cloud Foundation bundle.
This allows VMware Cloud Foundation 4.4 environments to run any “compatible” updated VMware Aria product.
This allows VMware Cloud Foundation 4.4 environments the flexibility to be upgraded to the most current versions of VMware Aria without having to wait for an updated VMware Cloud Foundation version (bill of materials) to be released.
vRSLCM is still initially deployed into a VMware Cloud Foundation using VMware Cloud Foundation SDDC Manager automated workflows and is VMware Cloud Foundation aware. This process has also not changed with VMware Cloud Foundation 4.4.
vRSLCM needs to be running VRSLCM 8.6.2 to support vRSLCM Flexible Upgrades which allow newer “compatible” versions of VMware Aria Suite Lifecycle products to be installed moving forward.
vRLCM Upgrade Process:
– First, we need to know if we need to download vRSLCM PSPAK before upgrading VRLCM to 8.8.2 or not, by checking this Knowledge Base article.
| 4.4.0.0 4.4.1.0 | 8.6.2 | 4 | Allows upgrade to 8.8.2 or 8.10 for VCF 4.5 | 8.6.x |
As you can see, in order to upgrade to 8.8.2, we need to have PSPACK 4
Note: To check which PSPACK you have: Go to vRealize Life cycle Manager UI, navigate to Settings > Product Support Pack > Check the build
It was showing Build Number: 19447709 which is PSPACK 3
In that case, follow this link to upgrade PSPACK to 4.
– Second, we need to upgrade VMware Aria Suite Lifecycle to 8.8.2
Upgrade Methods:
We have 3 methods as seen in the below picture.
Online:
- Go to settings > My VMware > Add My VMware Account > then authenticate to the VMware depot
- Then go to Settings > System upgrade > Online > Download applicable bundle
Offline using CD ROM:
- Click this link to download the iso file from the VMware Customer Connect
- Use winscp and move the file to one of the datastore seen by the host holding vRLCM VM
- Attach the iso to the virtual Machine
- Go to VMware Aria Lifecycle Suite user interface > Settings > System Upgrade >CDROM
(Note: you need to disable FIPS Mode) - Press “Check for Upgrades” > Upgrade
Once the desired version of vRSLCM is running, vRSLCM does a check of all installed vRealize components and provides the operator a selection of compatible upgradable options which have been validated for the VMware Cloud Foundation version being used.
The VMware Cloud Foundation Operator is then able to download and install the latest VMware Aria builds within the VMware Cloud Foundation environment. They can choose to upgrade individual VMware Aria components without having to upgrade all of the components in the VMware Aria Suite™.
For example: VMware Aria Operations™ can be upgraded without having to upgrade VMware Aria Automation™ or VMware Aria Operations™ for Logs.
Before upgrading any VMware Aria component, you should check the Interoperability guide.
