The VCF Async Patch tool can be used to patch individual BOM products on vSAN Ready Nodes and VxRail environments. The BOM products that can be individually async patched include:
VCF on vSAN Ready Nodes: VC, NSX, ESXi
VCF on VxRail: VC, NSX, ESXi/VxRail composite bundle
The Async Patch Tool is supported with VMware Cloud Foundation 4.2.1 and later
Future VCF upgrade version: VCF releases that are patched with async patches of BOM products can be upgraded to future VCF releases as stated in https://kb.vmware.com/s/article/88287
Operating System: Supported with Linux (includes Cygwin support) and Windows (includes WSL support) environment
Flowchart
Commands
Remove Older version of the tool and configure TCP keepalives
Modify properties to below: $ vim ~/.ssh/config TCPKeepAlive yes ServerAliveInterval 30
Download Async Patch Tool
Offline – Needs a DMZ machine which can connect to depot.vmware.com. Use linux machine
How do I download async patch tool
1. Log into VMware customer connect and select your current version of VCF 2. Click “Drivers & Tools” 3. Expand VMware Cloud Foundation Tools and click Go To Downloads in the Async Patch Tool row and download the tool
Refer commands below for: 4. Extract the tool to SDDC Manager (online) or DMZ machine (offline) which has connectivity to depot.vmware.com 5. Ensure tool has right permissions
$ cp vcf-async-patch-tool-<version>.tar.gz ${APTool_Install_Directory}/asyncPatchTool (Copy the Async Patch Tool file (vcf-async-patch-tool-<version>.tar.gz) that you downloaded in step 1 to the ${APTool_Install_Directory}/asyncPatchTool directory)
$ tar -xvf vcf-async-patch-tool-<version>.tar.gz (Navigate to ${APTool_Install_Directory}/asyncPatchTool and extract the contents of vcf-async-patch-tool-<version>.tar.gz)
Set the permissions for the asyncPatchTool directory $ cd ${User home folder} $ chmod -R 755asyncPatchTool
Demo
List Patches – Offline mode
How do I list patches available for async patching in the Async Patch Tool?
user: <DMZ user>
List Patches
$ cd /{APTool_Install_Directory}/asyncPatchTool/bin $ ./vcf-async-patch-tool --listAsyncPatch --depotUser ${DEPOT_USER}
Additional options – Examples --sku Supported values VCF, VCF_ON_VXRAIL --productType Supported values NSX_T_MANAGER,VCENTER,ESX_HOST --outputDirectory Specify a location forthe download; default/root/apToolBundles --proxyServer, --ps Connect to the internet through a proxy server; specify the FQDN and port of the proxy server. For example, --proxyServer FQDN:port
Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password
What does it output?
The tool will list a table of async patches and their details to the console in human-readable format:
List Option
Demo
Copy Async Patch Tool to SDDC Manager- Offline Mode
user: vcf (SSH to SDDC manager FQDN)
Copy the entire contents of the Async Patch Tool directory from the computer with internet access to the /home/vcf/asyncPatchTool directory on the SDDC Manager appliance
--product:version Product and version of the parch retrieved from the "list patch"command customer_connect_email VMware Customer Connect email address --sku_type VCF or VCF_ON_VXRAIL --pdu dell_emc_depot_email Dell EMC Depot email address (VxRail only) --outputDirectory Specify a location forthe download; default${USER_HOME_DIR}/apToolBundles --proxyServer, --ps Connect to the internet through a proxy server; --proxyServer FQDN:port
Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Too – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password – Enter Dell EMC Depot user name and password if the product type is VX_MANAGERThe Async Patch Tool downloads the patch and required artifacts (for example, the LCM manifest)
Demo
Copy Patch to SDDC manager – Offline mode
user: vcf
Copy the entire output directory (specified in above download command, for example: apToolBundles) to the SDDC Manager appliance. You can select any location that has enough free space available, for example, /nfs/vmware/vcf/nfs-mount/.)
--product:version Product and version of the parch retrieved from the "list patch"command --SSO user SSO user account, forexample, [email protected] --outputDirectory Specify a location forthe download; default${USER_HOME_DIR}/apToolBundles. Recommend specifying it to /nfs/vmware/vcf/nfs-mount/apToolBundles
Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Read the information and enter Y to acknowledge the pre-requisites – Enter the password for the super user (vcf) account – Enter the password for the root user account – Enter the password for the SSO user account – The Async Patch Tool uploads the patch to the internal LCM repository on the SDDC Manager appliance
Demo
Log in to the SDDC Manager UI and apply the async patch to all workload domains.
The patches that were enabled show up in the SDDC Manager. This should be run as a regular upgrade from the SDDC Manager.
Disable all Patches – Offline
user: vcf
SSH in to the SDDC Manager appliance using the vcf user account Navigate to /home/vcf/asyncPatchTool/bin
Run the following command: ./vcf-async-patch-tool --disableAllPatches --sddcSSOUser SSOuser --sddcSSHUser vcf --SSO user SSO user account, forexample, [email protected]
Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter the password for the super user (vcf) accountEnter the password for the root user accountEnter the password for the SSO user account
Demo
Download Future Bundles for Enable Upgrade – DMZ Offline Mode
user: DMZ user
Download Future Upgrade Bundles
$ cd /{APTool_Install_Directory}/asyncPatchTool/bin
--product:version Product and version of the parch retrieved from the "list patch"command customer_connect_email VMware Customer Connect email address target_VCF_version VCF version that the customer wants to upgrade to current_VCF_version Current VCF version of customer SDDC --sku_type VCF or VCF_ON_VXRAIL --pdu dell_emc_depot_email Dell EMC Depot email address (VxRail only) --outputDirectory Specify a location forthe download; default${USER_HOME_DIR}/apToolBundles --proxyServer, --ps Connect to the internet through a proxy server; --proxyServer FQDN:port
Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Too – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password – Enter Dell EMC Depot user name and password if the product type is VX_MANAGERThe Async Patch Tool downloads the patch and required artefacts (for example, the LCM manifest)
Demo
Standalone commands
Help
Lists the different types of options supported by the tool
${AP_TOOL_DIR}/bin/vcf-async-patch-tool -h
Inventory Sync
This operation updates the VCF inventory of NSX-T, ESXi and VC with the accurate information of the versions run by the actual products, keeping the record that the VCF instance is up-to-date. This option should be exercised by the customers when the customers have done any out of band upgrades
This option can be used to verify if the patch enablement has been completed successfully or failed. It internally will ensure that all the requested patches have been uploaded to LCM and are showing as available for upgrade.
–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments
–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments
–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments
Pre-check
Validate system environment is able to perform enable patch.
–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments
–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments
–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments
–productType, –ptype <String>: Product type, ESX_HOST,NSX_T_MANAGER, VCENTER listAsyncPatch in order to filter the list by product type.
–proxyServer, –ps <String>: Used when internet connectivity is only available through a proxy server. Provide proxy server addressand port in ‘<FQDN:port>’ format.
To Note:
–depotPassword <String> MyVMware login password. Should be specified in quotes if any special characters are included
–depotUser, –du <String> MyVMware login user name. Should be specified in quotes if any special characters are included
The download operation of enable patch downloads additional SDDC Hot Patch bundles. These bundles may be required to patch your SDDC to successfully apply the async patch on your environment.
Troubleshooting
Log Location
Log for the Async patch tool is async_patch_tool.log. Tail -f to see log details. The tool prints the INFO or above level logs to the console. The tool prints the current location of the log file when the process is running, copies over the log files to /var/log/vmware/vcf/lcm/tools/asyncpatchtool directory once the tool finishes the execution to allow SoS collection
Disabling All Patches Ends Unexpectedly with Failure Waiting for LCM Service to come up
The script used to clean up bundles in the disable patch workflow intermittently gets stuck and exits out.
In this scenario, there is a chance that LCM was never restarted if the script exited unexpectedly.
If this occurs, ensure the LCM service is up and running correctly and retry AP Tool operation
Enable Future Upgrade on VxRail fails with Exception
partnerBundleMetadata.json filedoes not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/local softwareCompatibilitySets.json filedoes not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/local
Make sure the partnerBundleMetadata.json and softwareCompatibilitySets.json are correctly placed in /nfs/vmware/vcf/nfs-mount/bundle/depot/local
Make sure a permission of 755 on the above location , for the vcf_lcm user
Invalid Permissions Issue
If the output directory was copied over to the sddc VM without setting proper ownership/permissions, the tool will fail when uploading bundles with error similar to:
2022-04-2714:12:12.147[ERROR] Unexpected error occurred uploading bundle {"status":500,"code":"Internal Server Error","message":"INSUFFICIENT_BUNDLE_DELETE_PERMISSIONS; /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests/bundle-47505.manifest file can not be deleted due to insufficient permissions. vcf_lcm user must have read and write access to /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests directory or upload bundle files from any directory where vcf_lcm user have read and write access."}
Unwanted bundles are enabled on environment and cleanup has to be performed
If the patches enabled using AP tool are required to be cleaned up, please login your SDDC VM as root user and run disable all patches command: Error Message
The tool uses root credentials for performing the operations such as config property update, etc as required for the operations.
If there are multiple attempts with either blank or invalid password, the user account is locked on SDDC VM. Follow the steps below to reset the number of failed logins by the root user.
Reset failed root login attempts
1. Login as root into the vCenter shell. 2. Execute - pam_tally2 --user=root --reset
Invalid Permissions Issue
To fix the error, ensure that the output directory has proper vcf:vcf 755 permissions:
