Overview
The VCF Async Patch tool can be used to patch individual BOM products on vSAN Ready Nodes and VxRail environments. The BOM products that can be individually async patched include:
VCF on vSAN Ready Nodes: VC, NSX, ESXi
VCF on VxRail: VC, NSX, ESXi/VxRail composite bundle
The Async Patch Tool is supported with VMware Cloud Foundation 4.2.1 and later
Future VCF upgrade version: VCF releases that are patched with async patches of BOM products can be upgraded to future VCF releases as stated in https://kb.vmware.com/s/article/88287
Operating System: Supported with Linux (includes Cygwin support) and Windows (includes WSL support) environment
Flowchart
Commands
Remove Older version of the tool and configure TCP keepalives
How do I remove older version of the tool?
user: vcf
Remove older version of tool
$ rm -r /home/vcf/asyncPatchTool (default directory)Additional options $ rm -r <outputdirectory> |
How do I configure keepalives?
User: vcf
Configure Keepalives
Modify properties to below$ vim ~/.ssh/config TCPKeepAlive yes ServerAliveInterval 30 |
Download Async Patch Tool
Online – can download bundles to SDDC manager to connect to depot.vmware.com.
How do I download async patch tool?
user: vcf
Download Tool
| 1. Log into VMware customer connect and select your current version of VCF 2. Click “Drivers & Tools” 3. Expand VMware Cloud Foundation Tools and click Go To Downloads in the Async Patch Tool row and download the tool Refer commands below for 4. Extract the tool to SDDC Manager (online) or DMZ machine (offline) which has connectivity to depot.vmware.com 5. Ensure tool has right permissions |
How do I install the tool on the SDDC Manager?
User: vcf
$ mkdir /home/vcf/asyncPatchTool$ cp vcf-async-patch-tool-<version>.tar.gz /home/vcf/asyncPatchTool(Copy the Async Patch Tool file (vcf-async-patch-tool-<version>.tar.gz) that you downloaded in step 1 to the /home/vcf/asyncPatchTool directory) $ tar -xvf vcf-async-patch-tool-<version>.tar.gz(Navigate to /home/vcf/asyncPatchTool and extract the contents of vcf-async-patch-tool-<version>.tar.gz) Set the permissions for the asyncPatchTool directory $ cd /home/vcf/$ chmod -R 755 asyncPatchTool$ chown -R vcf:vcf asyncPatchTool |
Demo
List Patches
How do I list patches available for async patching in the Async Patch Tool?
user: vcf
List Patches
$ cd /home/vcf/asyncPatchTool/bin$ ./vcf-async-patch-tool --listAsyncPatch --depotUser ${DEPOT_USER}--depotUser ${DEPOT_USER} VMware Customer Connect email address--sku ${SKU_TYPE} Supported values VCF, VCF_ON_VXRAIL--${PRODUCT_TYPE} Supported values NSX_T_MANAGER,VCENTER,ESX_HOST--outputDirectory ${OUTPUT_DIRECTORY} Specify a location for the download; default /root/apToolBundles--proxyServer, --ps Connect to the internet through a proxy server; --proxyServer FQDN:portPost Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password |
What does it output?
The tool will list a table of async patches and their details to the console in human-readable format:
List Option
Demo
Download Async Patch / Enable an Async Patch / Upload Patch to the SDDC manager
How do I enable async patch on my environment?
The Async Patch Tool downloads the patch and uploads it to the internal LCM repository on the SDDC Manager appliance
user: vcf
$ cd /home/vcf/asyncPatchTool/binVSRN $ ./vcf-async-patch-tool -e --patch ${PRODUCT_TYPE}:${PRODUCT_VERSION} --du ${DEPOT_USER} --sddcSSOUser ${SSOuser} --sddcSSHUser ${SDDC_SSH_USER} --it ${INSTANCE_TYPE}VxRail $ ./vcf-async-patch-tool -e --patch ${PRODUCT_TYPE}:${PRODUCT_VERSION} --du ${DEPOT_USER} --pdu ${PARTNER_DEPOT_USER} --sddcSSOUser ${SSOuser} --sddcSSHUser vcf --it ${INSTANCE_TYPE}--${PRODUCT_TYPE}:${PRODUCT_VERSION} Product and Version of the parch retrieved from the "list patch" command. If the product type is VX_MANAGER, enter your Dell EMC Depot user name and password. (VxRail only)--depotUser ${DEPOT_USER} VMware Customer Connect email address--sddcSSOUser ${SSOuser} SSO user account, for example, [email protected]--sddcSSHUser ${SDDC_SSH_USER} SDDC SSH account, for example vcf--pdu ${PARTNER_DEPOT_USER} Dell EMC Depot email address. (VxRail only)--proxyServer, --ps Connect to the internet through a proxy server; --proxyServer FQDN:port--it ${INSTANCE_TYPE} ONLINE/OFFLINEPost Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y to acknowledge prerequisites- Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter the password for the super user (vcf) account – Enter the password for the root user account – Enter the password for the SSO user account- Enter your VMware Customer Connect (Depot) password – Enter Dell EMC Depot user name and password if the product type is VX_MANAGER |
Demo
Log in to the SDDC Manager UI and apply the async patch to all workload domains
The patches that were enabled show up in the SDDC Manager. This should be run as a regular upgrade from the SDDC Manager.
Disable all Patches
How do I disable the patches?
After the patches have been applied from the SDDC manager, they need to be disabled using the AP tool
user: vcf
$ cd /home/vcf/asyncPatchTool/bin$ ./vcf-async-patch-tool --disableAllPatches --sddcSSOUser ${SDDC_SSO_USER} --sddcSSHUser ${SDDC_SSH_USER}--sddcSSOUser ${SDDC_SSO_USER} SSO user account, for example, [email protected]--sddcSSHUser ${SDDC_SSH_USER} SDDC SSH account, for example vcf Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter the password for the super user (vcf) account – Enter the password for the root user account – Enter the password for the SSO user account |
Demo
Enable Future Upgrade/Download Future Bundles/Upload future bundles to SDDC Manager (Upgrade from VCF 4.x to 4.y)
How do I enable future upgrade?
user: vcf
$ cd /home/vcf/asyncPatchTool/binVSRN $ ./vcf-async-patch-tool --enableVCFUpgrade ${TARGET_VCF_VERSION} --du ${DEPOT_USER} --sddcSSOUser ${SDDC_SSO_USER} --sddcSSHUser ${SDDC_SSH_USER} --it ${INSTANCE_TYPE} VxRail $ ./vcf-async-patch-tool --enableVCFUpgrade ${TARGET_VCF_VERSION} --du ${DEPOT_USER} --pdu ${PARTNER_DEPOT_USER} --sddcSSOUser ${SDDC_SSO_USER} --sddcSSHUser ${SDDC_SSH_USER} --it ${INSTANCE_TYPE} --${TARGET_VCF_VERSION} Target version of VMware Cloud Foundation--depotUser ${DEPOT_USER} VMware Customer Connect email address--sddcSSOUser ${SDDC_SSO_USER} SSO user account, for example, [email protected]--sddcSSHUser ${SDDC_SSH_USER} SDDC SSH account, for example vcf --pdu ${PARTNER_DEPOT_USER} Dell EMC Depot email address. (VxRail only)--proxyServer, --ps Connect to the internet through a proxy server; --proxyServer FQDN:port--${INSTANCE_TYPE} ONLINE/OFFLINE--outputDirectory ${OUTPUT_DIRECTORY} Location of transferred artefacts from DM-Z machine to SDDC-M in case for offline customers. This is optional for online SDDC-M environments. Default output path: /root/apToolBundles Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Read the information and enter Y to acknowledge the pre-requisitesEnter the password for the super user (vcf) account – Enter the password for the root user account – Enter the password for the SSO user account – Enter your VMware Customer Connect (Depot) password – If the product type is VX_MANAGER, enter your Dell EMC Depot user name and password (VxRail only) – Enter Y or N to choose whether or not to download vRealize bundles The Async Patch Tool determines which bundles are required, downloads the bundles, and uploads them to the internal LCM repository on the SDDC Manager appliance |
Demo
Log in to the SDDC Manager UI and Upgrade to a future VCF version
Standalone commands
Enable CEIP
Enables or disables telemetry collection for data relevant to Async Patch Tool operations. This is a one-time operation that will configure the tool for all future operations
$ ./bin/vcf-async-patch-tool –ceip “<Boolean>”
Help
Lists the different types of options supported by the tool
${AP_TOOL_DIR}/bin/vcf-async-patch-tool -h
Inventory Sync
This operation updates the VCF inventory of NSX-T, ESXi and VC with the accurate information of the versions run by the actual products, keeping the record that the VCF instance is up-to-date. This option should be exercised by the customers when the customers have done any out of band upgrades.
${AP_TOOL_DIR}/bin/vcf-async-patch-tool –performInventorySync –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser ${SDDC_SSH_USER}
Demo
Enable async patch
${AP_TOOL_DIR}/bin/vcf-async-patch- tool –enableAsyncPatch –patch
${PRODUCT_TYPE}:${PRODUCT_VERSION} –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser
${SDDC_SSH_USER} –instanceType ${INSTANCE_TYPE} [ONLINE/OFFLINE]
Additional options
–depotUser ${DEPOT_USER} : Required to be specified for online SDDC-M environments
–pdu ${PARTNER_DEPOT_USER} : Required to be specified for online Vxrail SDDC-M environments
–outputDirectory ${OUTPUT_DIRECTORY} : Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional for online SDDC-M environments. Default output path: /root/apToolBundles
Post-check
This option can be used to verify if the patch enablement has been completed successfully or failed. It internally will ensure that all the requested patches have been uploaded to LCM and are showing as available for upgrade.
${AP_TOOL_DIR}/bin/vcf-async-patch-tool –enableAsyncPatch –postcheck –patch
${PRODUCT_TYPE}:${PRODUCT_VERSION} –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser
${SDDC_SSH_USER} –instanceType ${INSTANCE_TYPE} [ONLINE/OFFLINE]
Additional options
–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments
–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments
–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments
Pre-check
Validate system environment is able to perform enable patch.
${AP_TOOL_DIR}/bin/vcf-async-patch-tool –enableAsyncPatch –precheck –patch
${PRODUCT_TYPE}:${PRODUCT_VERSION} –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser
${SDDC_SSH_USER} –instanceType ${INSTANCE_TYPE} [ONLINE/OFFLINE]
Additional options
–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments
–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments
–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments
–productType, –ptype <String>: Product type, ESX_HOST,NSX_T_MANAGER, VCENTER listAsyncPatch in order to filter the list by product type.
–proxyServer, –ps <String>: Used when internet connectivity is only available through a proxy server. Provide proxy server addressand port in ‘<FQDN:port>’ format.
To Note:
–depotPassword <String> MyVMware login password. Should be specified in quotes if any special characters are included
–depotUser, –du <String> MyVMware login user name. Should be specified in quotes if any special characters are included
Troubleshooting
Log Location
Log for the Async patch tool is async_patch_tool.log. Tail -f to see log details. The tool prints the INFO or above level logs to the console. The tool prints the current location of the log file when the process is running, copies over the log files to /var/log/vmware/vcf/lcm/tools/asyncpatchtool directory once the tool finishes the execution to allow SoS collection
Disabling All Patches Ends Unexpectedly with Failure Waiting for LCM Service to come up
The script used to clean up bundles in the disable patch workflow intermittently gets stuck and exits out.
In this scenario, there is a chance that LCM was never restarted if the script exited unexpectedly.
If this occurs, ensure the LCM service is up and running correctly and retry AP Tool operation
Enable Future Upgrade on VxRail fails with Exception
partnerBundleMetadata.json file does not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/localsoftwareCompatibilitySets.json file does not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/local |
Make sure the partnerBundleMetadata.json and softwareCompatibilitySets.json are correctly placed in /nfs/vmware/vcf/nfs-mount/bundle/depot/local
Make sure a permission of 755 on the above location , for the vcf_lcm user
Invalid Permissions Issue
If the output directory was copied over to the sddc VM without setting proper ownership/permissions, the tool will fail when uploading bundles with error similar to:
2022-04-27 14:12:12.147 [ERROR] Unexpected error occurred uploading bundle {"status":500,"code":"Internal Server Error","message":"INSUFFICIENT_BUNDLE_DELETE_PERMISSIONS; /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests/bundle-47505.manifest file can not be deleted due to insufficient permissions. vcf_lcm user must have read and write access to /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests directory or upload bundle files from any directory where vcf_lcm user have read and write access."} |
Unwanted bundles are enabled on environment and cleanup has to be performed
If the patches enabled using AP tool are required to be cleaned up, please login your SDDC VM as root user and run disable all patches command:
Error Message
vcf@sddc-manager [ ~ ]# {asyncPatchTool}/bin/vcf-async-patch-tool --disableAllPatches --ssou {ssoUsername} |
Account locked issue
The tool uses root credentials for performing the operations such as config property update, etc as required for the operations.
If there are multiple attempts with either blank or invalid password, the user account is locked on SDDC VM. Follow the steps below to reset the number of failed logins by the root user.
Reset failed root login attempts
1. Login as root into the vCenter shell.2. Execute - pam_tally2 --user=root --reset |
Invalid Permissions Issue
To fix the error, ensure that the output directory has proper vcf:vcf 755 permissions:
vcf@sddc-manager [ ~ ]# chmod -R 755 {apToolBundlesDir}vcf@sddc-manager [ ~ ]# chown -R vcf:vcf {apToolBundlesDir} |
Links
Documentation: https://docs.vmware.com/en/VMware-Cloud-Foundation/services/ap-tool/GUID-49818DF1-94EA-4C85-8CB6-6EFFCE5F8060.html
Patch Support : https://kb.vmware.com/s/article/88287
