Overview

The VCF Async Patch tool can be used to patch individual BOM products on vSAN Ready Nodes and VxRail environments. The BOM products that can be individually async patched include:

VCF on vSAN Ready Nodes: VC, NSX, ESXi

VCF on VxRail: VC, NSX, ESXi/VxRail composite bundle

The Async Patch Tool is supported with VMware Cloud Foundation 4.2.1 and later

Future VCF upgrade version: VCF releases that are patched with async patches of BOM products can be upgraded to future VCF releases as stated in https://kb.vmware.com/s/article/88287

Operating System: Supported with Linux (includes Cygwin support) and Windows (includes WSL support) environment 

Flowchart

Commands

Remove Older version of the tool and configure TCP keepalives

How do I remove older version of the tool?

user: vcf

Remove older version of tool

$ rm -r /home/vcf/asyncPatchTool (default directory) Additional options $ rm -r <outputdirectory>

How do I configure keepalives?

User: vcf

Configure Keepalives

Modify properties to below: $ vim ~/.ssh/config        TCPKeepAlive yes            ServerAliveInterval 30

Download Async Patch Tool 

Offline – Needs a DMZ machine which can connect to depot.vmware.com. Use linux machine

How do I download async patch tool

1. Log into VMware customer connect and select your current version of VCF 2. Click “Drivers & Tools” 3. Expand VMware Cloud Foundation Tools and click Go To Downloads in the Async Patch Tool row and download the tool Refer commands below for: 4. Extract the tool to SDDC Manager (online) or DMZ machine (offline) which has connectivity to depot.vmware.com 5. Ensure tool has right permissions

How do I install the tool on the DMZ server?

OFFLINE – Install tool on DMZ machine

User: <DMZ user>

Extract Patch

$ mkdir ${APTool_Install_Directory}/asyncPatchTool $ cp vcf-async-patch-tool-<version>.tar.gz ${APTool_Install_Directory}/asyncPatchTool (Copy the Async Patch Tool file (vcf-async-patch-tool-<version>.tar.gz) that you downloaded in step 1 to the ${APTool_Install_Directory}/asyncPatchTool directory) $ tar -xvf vcf-async-patch-tool-<version>.tar.gz (Navigate to ${APTool_Install_Directory}/asyncPatchTool and extract the contents of vcf-async-patch-tool-<version>.tar.gz) Set the permissions for the asyncPatchTool directory $ cd ${User home folder} $ chmod -R 755 asyncPatchTool

Demo

List Patches – Offline mode 

How do I list patches available for async patching in the Async Patch Tool?

user: <DMZ user>

List Patches

$ cd /{APTool_Install_Directory}/asyncPatchTool/bin $ ./vcf-async-patch-tool --listAsyncPatch --depotUser ${DEPOT_USER}                       Additional options – Examples --sku                  Supported values VCF, VCF_ON_VXRAIL --productType          Supported values NSX_T_MANAGER,VCENTER,ESX_HOST --outputDirectory      Specify a location for the download; default /root/apToolBundles --proxyServer, --ps    Connect to the internet through a proxy server; specify the FQDN and port of the proxy server. For example, --proxyServer FQDN:port Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password

What does it output?

The tool will list a table of async patches and their details to the console in human-readable format:

List Option

Demo

Copy Async Patch Tool to SDDC Manager- Offline Mode

user: vcf (SSH to SDDC manager FQDN)

Copy the entire contents of the Async Patch Tool directory from the computer with internet access to the /home/vcf/asyncPatchTool directory on the SDDC Manager appliance scp -r ${DMZ_AP_DOWNLOADED_DIR}/asyncPatchTool vcf@${SDDC_IP/SDDC_FQDN}:/home/vcf Set the permissions for the asyncPatchTool directory  cd /home/vcf/  chmod -R 755 asyncPatchTool  chown -R vcf:vcf asyncPatchTool

Demo

Download async BOM patch – DMZ Offline mode

user: DMZ user

Download Patch

$ cd /{APTool_Install_Directory}/asyncPatchTool/bin VSRN $ ./vcf-async-patch-tool -d --patch product:version --du customer_connect_email --sku sku_type --outputDirectory  VxRail $ ./vcf-async-patch-tool -d --patch product:version --du customer_connect_email --sku sku_type --pdu dell_emc_depot_email --outputDirectory --product:version           Product and version of the parch retrieved from the "list patch" command customer_connect_email     VMware Customer Connect email address --sku_type                   VCF or VCF_ON_VXRAIL --pdu dell_emc_depot_email   Dell EMC Depot email address (VxRail only) --outputDirectory            Specify a location for the download; default ${USER_HOME_DIR}/apToolBundles --proxyServer, --ps          Connect to the internet through a proxy server; --proxyServer FQDN:port Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Too – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password – Enter Dell EMC Depot user name and password if the product type is VX_MANAGERThe Async Patch Tool downloads the patch and required artifacts (for example, the LCM manifest)

Demo 

Copy Patch to SDDC manager – Offline mode

user: vcf

Copy the entire output directory (specified in above download command, for example: apToolBundles) to the SDDC Manager appliance. You can select any location that has enough free space available, for example, /nfs/vmware/vcf/nfs-mount/.) scp -r ${USER_HOME_DIR}/apToolBundles vcf@${SDDC_IP/SDDC_FQDN}:/nfs/vmware/vcf/nfs-mount Set permissions chmod -R 755 /nfs/vmware/vcf/nfs-mount/apToolBundles SSH in to the SDDC Manager appliance using the vcf user account Navigate to /nfs/vmware/vcf/nfs-mount  cd  /nfs/vmware/vcf/nfs-mount  (If you copied the output directory to a different location, navigate to that directory instead) Run the following commands:  chmod -R 755 apToolBundles  chown -R vcf:vcf apToolBundles

Demo

Enable Patch – Offline mode

user: vcf

$ cd /home/vcf/asyncPatchTool/bin VSRN $ ./vcf-async-patch-tool -e --patch product:version --sddcSSOUser SSOuser --sddcSSHUser vcf --outputDirectory bundleDirectory --it OFFLINE VxRail $ ./vcf-async-patch-tool -e --patch product:version --sddcSSOUser SSOuser --sddcSSHUser vcf --outputDirectory bundleDirectory --it OFFLINE --product:version           Product and version of the parch retrieved from the "list patch" command --SSO user                 SSO user account, for example, [email protected] --outputDirectory          Specify a location for the download; default ${USER_HOME_DIR}/apToolBundles. Recommend specifying it to /nfs/vmware/vcf/nfs-mount/apToolBundles Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Read the information and enter Y to acknowledge the pre-requisites – Enter the password for the super user (vcf) account – Enter the password for the root user account – Enter the password for the SSO user account – The Async Patch Tool uploads the patch to the internal LCM repository on the SDDC Manager appliance

Demo

Log in to the SDDC Manager UI and apply the async patch to all workload domains.

The patches that were enabled show up in the SDDC Manager. This should be run as a regular upgrade from the SDDC Manager. 

Disable all Patches – Offline

user: vcf

SSH in to the SDDC Manager appliance using the vcf user account Navigate to /home/vcf/asyncPatchTool/bin Run the following command:  ./vcf-async-patch-tool --disableAllPatches --sddcSSOUser SSOuser --sddcSSHUser vcf   --SSO user             SSO user account, for example, [email protected] Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Tool – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter the password for the super user (vcf) accountEnter the password for the root user accountEnter the password for the SSO user account

Demo

Download Future Bundles for Enable Upgrade – DMZ Offline Mode

user: DMZ user

Download Future Upgrade Bundles

$ cd /{APTool_Install_Directory}/asyncPatchTool/bin VSRN $ ./vcf-async-patch-tool -d --targetVcfVersion target_VCF_version --sourceVcfVersion current_VCF_version --du customer_connect_email --sku sku_type --outputDirectory VxRail $ ./vcf-async-patch-tool -d --targetVcfVersion target_VCF_version --sourceVcfVersion current_VCF_version --du customer_connect_email --sku sku_type --pdu dell_emc_depot_email --outputDirectory --product:version            Product and version of the parch retrieved from the "list patch" command customer_connect_email       VMware Customer Connect email address target_VCF_version           VCF version that the customer wants to upgrade to current_VCF_version          Current VCF version of customer SDDC --sku_type                   VCF or VCF_ON_VXRAIL --pdu dell_emc_depot_email   Dell EMC Depot email address (VxRail only) --outputDirectory            Specify a location for the download; default ${USER_HOME_DIR}/apToolBundles --proxyServer, --ps          Connect to the internet through a proxy server; --proxyServer FQDN:port Post Input – Enter Y to confirm that you are running the latest version of the Async Patch Too – Enter Y or N to choose whether or not to participate in the VMware Customer Experience Improvement Program (CEIP) – Enter your VMware Customer Connect (Depot) password – Enter Dell EMC Depot user name and password if the product type is VX_MANAGERThe Async Patch Tool downloads the patch and required artefacts (for example, the LCM manifest)

Demo

Standalone commands

Help

Lists the different types of options supported by the tool

${AP_TOOL_DIR}/bin/vcf-async-patch-tool -h

Inventory Sync

This operation updates the VCF inventory of NSX-T, ESXi and VC with the accurate information of the versions run by the actual products, keeping the record that the VCF instance is up-to-date. This option should be exercised by the customers when the customers have done any out of band upgrades

${AP_TOOL_DIR}/bin/vcf-async-patch-tool –performInventorySync –sddcSSOUser

${SDDC_SSO_USER} –sddcSSHUser ${SDDC_SSH_USER}

Demo

Post-check

This option can be used to verify if the patch enablement has been completed successfully or failed. It internally will ensure that all the requested patches have been uploaded to LCM and are showing as available for upgrade.

${AP_TOOL_DIR}/bin/vcf-async-patch-tool –enableAsyncPatch –postcheck –patch

${PRODUCT_TYPE}:${PRODUCT_VERSION} –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser

${SDDC_SSH_USER} –instanceType ${INSTANCE_TYPE} [ONLINE/OFFLINE]

Additional options

–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments

–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments

–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments

Pre-check

Validate system environment is able to perform enable patch.  

${AP_TOOL_DIR}/bin/vcf-async-patch-tool –enableAsyncPatch –precheck –patch

${PRODUCT_TYPE}:${PRODUCT_VERSION} –sddcSSOUser ${SDDC_SSO_USER} –sddcSSHUser

${SDDC_SSH_USER} –instanceType ${INSTANCE_TYPE} [ONLINE/OFFLINE]

Additional options                    

–depotUser ${DEPOT_USER}: Required to be specified for online SDDC-M environments     

–pdu ${PARTNER_DEPOT_USER}: Required to be specified for for online Vxrail SDDC-M environments    

–outputDirectory ${OUTPUT_DIRECTORY}: Required to be specified for for offline SDDC-M environments. This should be the location of transferred artefacts from DM-Z machine to SDDC-M. This is optional arg for online SDDC-M environments

–productType, –ptype <String>: Product type, ESX_HOST,NSX_T_MANAGER, VCENTER listAsyncPatch in order to filter the list by product type.   

–proxyServer, –ps <String>: Used when internet connectivity is only available through a proxy server. Provide proxy server addressand port in ‘<FQDN:port>’ format.    

To Note:

–depotPassword <String>             MyVMware login password. Should be specified in quotes if any special characters are included

–depotUser, –du <String>           MyVMware login user name. Should be  specified in quotes if any special  characters are included       

The download operation of enable patch downloads additional SDDC Hot Patch bundles. These bundles may be required to patch your SDDC to successfully apply the async patch on your environment.  

Troubleshooting

Log Location

Log for the Async patch tool is async_patch_tool.log. Tail -f to see log details. The tool prints the INFO or above level logs to the console. The tool prints the current location of the log file when the process is running, copies over the log files to /var/log/vmware/vcf/lcm/tools/asyncpatchtool directory once the tool finishes the execution to allow SoS collection

Disabling All Patches Ends Unexpectedly with Failure Waiting for LCM Service to come up

The script used to clean up bundles in the disable patch workflow intermittently gets stuck and exits out.

In this scenario, there is a chance that LCM was never restarted if the script exited unexpectedly. 

If this occurs, ensure the LCM service is up and running correctly and retry AP Tool operation

Enable Future Upgrade on VxRail fails with Exception

partnerBundleMetadata.json file does not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/local softwareCompatibilitySets.json file does not exist at location /nfs/vmware/vcf/nfs-mount/bundle/depot/local

Make sure the partnerBundleMetadata.json and softwareCompatibilitySets.json are correctly placed in /nfs/vmware/vcf/nfs-mount/bundle/depot/local 

Make sure a permission of 755 on the above location , for the vcf_lcm user

Invalid Permissions Issue

If the output directory was copied over to the sddc VM without setting proper ownership/permissions, the tool will fail when uploading bundles with error similar to:

2022-04-27 14:12:12.147 [ERROR] Unexpected error occurred uploading bundle {"status":500,"code":"Internal Server Error","message":"INSUFFICIENT_BUNDLE_DELETE_PERMISSIONS; /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests/bundle-47505.manifest file can not be deleted due to insufficient permissions. vcf_lcm user must have read and write access to /nfs/vmware/vcf/nfs-mount/apToolBundles/manifests directory or upload bundle files from any directory where vcf_lcm user have read and write access."}

Unwanted bundles are enabled on environment and cleanup has to be performed

If the patches enabled using AP tool are required to be cleaned up, please login your SDDC VM as root user and run disable all patches command:
Error Message

vcf@sddc-manager [ ~ ]# {asyncPatchTool}/bin/vcf-async-patch-tool --disableAllPatches --ssou {ssoUsername}

Account locked issue

The tool uses root credentials for performing the operations such as config property update, etc as required for the operations.

If there are multiple attempts with either blank or invalid password, the user account is locked on SDDC VM. Follow the steps below to reset the number of failed logins by the root user.

Reset failed root login attempts

1. Login as root into the vCenter shell. 2. Execute - pam_tally2 --user=root --reset

Invalid Permissions Issue

To fix the error, ensure that the output directory has proper vcf:vcf 755 permissions:

vcf@sddc-manager [ ~ ]# chmod -R 755 {apToolBundlesDir} vcf@sddc-manager [ ~ ]# chown -R vcf:vcf {apToolBundlesDir}

Links

Documentation: https://docs.vmware.com/en/VMware-Cloud-Foundation/services/ap-tool/GUID-49818DF1-94EA-4C85-8CB6-6EFFCE5F8060.html

Patch Support : https://kb.vmware.com/s/article/88287