Home > Blogs > VMware Support Insider > Author Archives: Rick Blythe
Rick Blythe

About Rick Blythe

Social Media Program Manager for VMware, Rick Blythe manages the Twitter handle @vmwarekb and curates the Support Insider Blog.

Popular tweets

Here’s an interesting top 20 list. In the last 30 days, these were the tweets form our @vmwarekb account that got the most sharing from our customers. Perhaps you missed something on our list you’d be interested in.

Oh, and if you don’t follow us on Twitter, why not?

How to restart the Management agents on a VMware vSphere ESXi or ESX host
Poor network performance when using VMXNET3 adapter for routing in a Linux guest operating system (2077393)
Configuring the ESXi host with Active Directory authentication
Configuring the ESXi host with Active Directory authentication (2075361)
ESXi host initiates ARP Broadcast storm to NFS server (2080034)
Veeam virtual machine backups fail with the error: The host is not licensed for this feature (2080352)
Downgrading device drivers in VMware ESXi 5.x (2079279)
ESXi 5.5 … purple … screen error:#PF Exception 14 in world 33426: vmkeventd IP 0x418002c71507 addr 0×0 (2061842)
Storage Controllers previously supported for VSAN that are no longer supported (2081431)
VMware ESXi 5.1, Patch Release ESXi510-201406001 (2077640)
VM loses network connectivity during migration … in vCNS 5.1.4 and 5.5.2 and NSX for vSphere 6.0.4 (2080479)
Copying a file is slow on HTTP connections in vCenter Server (2081624)
VMware ESXi host in the vSphere Distributed Switch (vDS) are out of sync (2081052)
Preparing Windows 2008 R2 SP1 Server as a desktop to be deployed by Horizon DaaS (Desktone) (2080765)
Adding an Integrated Active Directory (IWA) Identity Source without the vSphere Web Client for vCenter SSO (2063424)
Upgrade paths and product compatibility for PowerCLI versions that feature OpenSSL security fixes (2082132)
Upgrading VMware vCenter Server 5.5 to a 5.5.x version using Simple Install fails (2074676)
VMware ESXi 5.1, Patch ESXi-5.1.0-20140604001-standard (2077642)
Dell EqualLogic Multipathing Extension Module (MEM) in View environments storage performance degradation (2078451)
Booting the ESXi host fails at Initializing scheduler (2077712)

VMware Support Options

Ever wondered what all of your options are when it came to technical support from VMware?

Listen to David Hulbert as he provides a 3 minute overview of VMware Technical Support, describing all of the levels of support options available.

 

Using vSphere ESXi Image Builder to create an installable ISO that is not vulnerable to Heartbleed

Here is a follow-up post from Andrew Lytle, member of the VMware Mission Critical Support Team. Andrew is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

VMware recently released updates to all products affected by the vulnerability dubbed “Heartbleed” (CVE-2014-0160): http://www.vmware.com/security/advisories/VMSA-2014-0004.html

As per KB article: Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160 (2076665), the delivery method for this code change in the VMware ESXi product is through an updated ESXi vSphere Installation Bundle (VIB). VIBs are the building blocks of an ESXi image. A VIB is akin to a tarball or ZIP archive in that it’s a collection of files packaged into a single archive.

Typically a new ESXi ISO file will be made available only during major revisions of the product (Update 1, Update 2, etc). If you need an ESXi 5.5 ISO which is already protected from Heartbleed, you can make your own ISO easily using vSphere PowerCLI.

The PowerCLI ImageBuilder cmdlets are designed to make custom ESXi ISOs which have asynchronous driver releases pre-installed, but it can also be used in a situation like this to make an ISO which lines up with a Patch Release instead of a full ESXi Update Release.

In this post we will cover both the ESXi 5.5 GA branch, as well as the ESXi 5.5 Update 1 branch. Choose the set of steps which will provide the ISO branch you need for your environment.

Creating an ISO based on ESXi 5.5 GA (Pre-Update 1)

These steps are for downloading the requirements for creating an ISO which is based on the ESXi 5.5 “GA” release, which was originally released 2013-09-22.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Install PowerCLI through the Windows MSI package, and copy the zip files to a handy location. For the purposes of this example, I will copy these files to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404020.zip
    1-1

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has Version of 5.5.0-0.15.1746974 and the Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base
    1-2

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140401020s-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-heartbleed.iso
    1-3

Creating an ISO based on ESXi 5.5 Update 1

These steps are for creating an ISO which is based on the ESXi 5.5 “Update 1” release, which was originally released 2014-03-11.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Copy the zip files to a handy location. For the purposes of this example, I will copy it to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404001.zip
    2-1

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has the Version of 5.5.0-1.16.1746018 and Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base
    2-2

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140404001-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-update1-heartbleed.iso
    2-3

Installing the ESXi ISO

The ISO file which was created in this steps can be used in exactly the same manner as the normal VMware ESXi 5.5 ISO. It can be mounted in a remote management console, or burned to a CD/DVD for installation.

Top 20 Articles for April 2014

Here is our Top 20 KB list for April 2014. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. Response to OpenSSL security issue CVE-2014-0160/CVE-2014-0346 a.k.a: “Heartbleed” (2076225)
  2. VMware ESXi 5.x host experiences a purple diagnostic screen mentioning E1000PollRxRing and E1000DevRx (2059053)
  3. Installing Windows in a virtual machine using VMware Fusion Easy Install (1011677)
  4. Installing async drivers on VMware ESXi 5.0, 5.1, and 5.5 (2005205)
  5. Re-pointing and re-registering VMware vCenter Server 5.1 / 5.5 and components (2033620)
  6. Resolving OpenSSL Heartbleed for VMware vCenter Server 5.5 (2076692)
  7. Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160 (2076665)
  8. Purging old data from the database used by VMware vCenter Server 4.x and 5.x (1025914)
  9. Troubleshooting Fusion virtual machine performance issues (1015676)
  10. Investigating virtual machine file locks on ESXi/ESX (10051)
  11. Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x hosts (2004605)
  12. Uninstalling and manually installing VMware Tools in VMware Fusion (1014522)
  13. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and ESXi 5.x (1027206)
  14. Resetting the VMware vCenter Server 5.x Inventory Service database (2042200)
  15. Installing VMware Tools in a Fusion virtual machine running Windows (1003417)
  16. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x (2004684)
  17. Manually deleting linked clones or stale virtual desktop entries from the View Composer database in VMware View Manager and Horizon View (2015112)
  18. Upgrading to vCenter Server 5.5 best practices (2053132)
  19. Installing or upgrading to ESXi 5.5 best practices (2052329)
  20. Installing vCenter Server 5.5 best practices (2052334)

Patching ESXi 5.5 for Heartbleed without installing Update 1

On April 19th, VMware released a series of patches for ESX 5.5 and ESX 5.5 Update 1 to re-mediate the CVE dubbed “Heartbleed” (CVE-2014-0076 and CVE-2014-0160).

VMware also recently announced that there was an issue in the newest version of ESXi 5.5 (Update 1 and later), which can cause difficulties communicating with NFS storage. This NFS issue is still being investigated, and customers are encouraged to subscribe to KB article: Intermittent NFS APDs on ESXi 5.5 U1 (2076392) for updates.

Due to the confluence of these two unrelated issues, you might find yourself trying to patch ESXi to protect yourself from the Heartbleed vulnerability, while at the same time trying to avoid installing ESXi 5.5 Update 1.

Here is the information from the VMware Knowledge Base on the topic:

2

The note at the bottom is the key. Stated simply, if you are…

  • Using NFS storage
  • Concerned about patching to Update 1 due to change control
  • Not already running ESXi 5.5 Update 1 (build-1623387)

… then you should patch your install for the Heartbleed issue and at the same time stay at ESX 5.5 by applying Patch Release ESXi550-201404020, and not ESXi550-201404001.

An Explanation of Patch Release Codes

To better understand the Patching process in a VMware environment, it is valuable to understand the codes which are used in VMware Patch Releases.

When VMware releases a patch, or a series of patches, they are bundled together in what is known as a Patch Release. A Patch Release will have a coded name which is formed using the following structure. I have added braces to demonstrate the different sections better in each example.

[PRODUCT]-[YEAR][MONTH][THREE DIGIT RELEASE NUMBER]

For example, the patch release for ESXi 5.5 that was released in January 2013 would be coded like this (without the explanatory braces):

[ESXi550]-[2013]-[01][001]

As part of a Patch Release, there will be at least one Patch. Each Patch is given a Patch (or Bulletin) ID. Patch IDs are similarly structured to Patch Release codes, but also have a two letter suffix. For Security Bulletins, the prefix will be SG. For Bug Fix Bulletins, the prefix will be BG.

For example, the two Patch IDs which were released to patch Heartbleed are:

[ESXi550]-[2014][04][401]-[SG]
[ESXi550]-[2014][04][420]-[SG]

Note that the only difference in the Patch IDs here is in the three digit release number (401 vs 420).

Patching with VMware Update Manager

There are a number of methods for patching ESXi hosts, and the most commonly used is VMware Update Manager (VUM). VUM will present a pair of Dynamic Baselines which will be automatically updated when patches are available. The danger in this case is that VUM may show you both the Pre-Update 1 patch, as well as the Post-Update 1 patch. If you are not careful as to which patches you apply, you might accidentally end up patching your host to Post-Update 1.

Here are the patches which were released on April 19th, as seen in VUM. The Update 1 patch is highlighted in red, while the Pre-Update 1 patch is marked in green.

1

Note: VMware also released two other ESXi 5.5 patches on April 19th, as part of Patch Release but these are not related to the Heartbleed vulnerability in any fashion. (ESXi550-201404402-BG, and ESXi550-201404403-BG).

Creating a Fixed Baseline

Patching a host using ESXi550-201404420-SG (Pre-Update 1), while avoiding ESXi550-201404401-SG (Post-Update 1) requires the use of a Fixed Baseline in Update Manager.

  1. Start in the Update Manager Admin view.
  2. Select the Baselines and Groups tab.
  3. Click Create… in the Baselines column.
    3
  4. Give the new Baseline a descriptive Name (and optionally a Description).
    4
  5. Click Next.
  6. For Baseline type, select Fixed.
    5
  7. Use the Search feature to find the only Patch we want to apply. You will need to select the Patch ID option from the dropdown menu to ensure the search scans for the appropriate column.
    6
  8. Enter the Patch ID into the search field: ESXi550-201404420-SG and click Enter to search.
  9. Select the Patch which shows up in the filtered list, and click the Down Arrow to move it into the selected Baselines.
    7
  10. Click Next and confirm that the Patch ESXi550-201404420-SG is the only one selected.
    8
  11. Click Finish.

The Baseline is now created and available for use.

Remediating a Host using the Fixed Baseline

Once the Fixed Baseline has been created, we can use it to Scan and Remediate an ESXi host.

  1. Select the host you wish to patch, and place it into Maintenance Mode.
  2. Click the Update Manager tab.
  3. Make sure that there are no Dynamic Baselines attached to the host you wish to patch. Detach any baselines which are currently attached:
    Critical Host Patches (Predefined)
    Non-Critical Host Patches (Predefined)
    Any other Custom Baselines which you have created
  4. Click the Attach link.
    9
  5. Select the newly created Baseline and click Attach.
    10
  6. Click the Scan link and make sure Patches and Extensions is selected. Click Scan again.
    11
  7. When you are ready to patch the host, select Remediate.
  8. Complete the Remediation wizard.

Once the host is patched, it will reboot automatically.

Patching an ESXi host manually via the command line

Another option to patch an ESXi host is to use the esxcli command line tool. The patch files required are the same. For more information on how to proceed with this route, refer to the vSphere 5.5 Documentation under the heading Update a Host with Individual VIBs.

References

Author: Andrew Lytle
As a member of the VMware Mission Critical Support Team, Andrew Lytle is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

ALERT: vCAC 6.0.x tenants become inaccessible and identity stores disappear

VMware Support AlertVMware has become aware of an issue that occurs after 90 days after deployment of a template in vCloud Automation Center (vCAC) 6.0.x, tenants become inaccessible and identity stores disappear due to expiration of the tenant admin password. For more information, see the article below.

For further information and updates, please refer to KB article: vCloud Automation Center 6.0.x tenants become inaccessible and identity stores disappear (2075011) in your problem description.

Note: Any updates to this issue will be reflected in the aforementioned KB article. To be alerted when this article is updated, click Subscribe to Document in the Actions box on the KB article page.

Top 20 Articles for March 2014

Here is our Top 20 KB list for March 2014. This list is ranked by the number of times a VMware Support Request was resolved by following the steps in a published Knowledge Base article.

  1. VMware ESXi 5.x host experiences a purple diagnostic screen mentioning E1000PollRxRing and E1000DevRx
  2. Downloading and Installing VMware Fusion
  3. Installing async drivers on VMware ESXi 5.0, 5.1, and 5.5
  4. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and ESXi 5.x
  5. Installing Windows in a virtual machine using VMware Fusion Easy Install
  6. Troubleshooting Fusion virtual machine performance issues
  7. Upgrading to vCenter Server 5.5 best practices
  8. Investigating virtual machine file locks on ESXi/ESX
  9. Re-pointing and re-registering VMware vCenter Server 5.1 / 5.5 and components
  10. Uninstalling and manually installing VMware Tools in VMware Fusion
  11. Installing vCenter Server 5.5 best practices
  12. Purging old data from the database used by VMware vCenter Server 4.x and 5.x
  13. Troubleshooting Fusion virtual machine startup issues
  14. Installing or upgrading to ESXi 5.5 best practices
  15. Creating a persistent scratch location for ESXi 4.x and 5.x
  16. Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x hosts
  17. Broadcom 5719/5720 NICs using tg3 driver become unresponsive and stop traffic in vSphere
  18. Update sequence for vSphere 5.5 and its compatible VMware products
  19. Installing VMware Tools in a Fusion virtual machine running Windows
  20. Methods of upgrading to vCenter Server 5.5

ALERT: Response to Heartbleed OpenSSL security issue

heartbleedThis week, a new vulnerability was discovered affecting SSL, a protocol most of the Internet uses to encrypt and secure communications. The VMware Security Engineering, Communications, and Response group (vSECR) is investigating the OpenSSL issue dubbed “Heartbleed”. For information on which VMware products may be affected and resolution/remediation steps, refer to the two KB articles at the bottom of this post.

For the curious, we would like to quickly explain why this particular vulnerability could be a risk across the Internet. The bug — dubbed “Heartbleed” — allows anybody to read the memory on a system that is supposed to be protected by SSL.

An anonymous attacker could potentially steal any information from an SSL-secured communication when the issue is not addressed. Best practices dictate that websites and web service providers should always use SSL-encrypted communication when dealing with sensitive information like usernames, passwords, and bank info. Heartbleed could breach that information to anybody who knows how to extract it without leaving a trace.

Scheduled Maintenance April 11

VMware will be performing a system upgrade to several VMware Support Web applications on Friday, April 11th, 2014 from 7:00PM to 7:45PM Pacific Time. If you need to file a high severity support request while the upgrade is in progress, please call VMware Technical Support for assistance. Because of this maintenance window, you may experience longer than normal wait times on the phone. We encourage you to submit your lower severity support issues via the online case logging option once the website becomes available again.

While this upgrade is in progress, you will be unable to:

  • Access or manage your VMware account
  • Submit support requests online
  • Download, purchase or register VMware products
  • Manage VMware product licenses
  • Download, purchase or register VMware products Manage VMware product licenses
  • Access VMware Communities

Please note this maintenance window does not affect the VMware Knowledgebase or the various Product Support Centers where you may find articles and notes that will help you resolve issues you may encounter.

We appreciate your patience during this maintenance period. These system upgrades are part of our commitment to continued service improvements and will help VMware better serve your needs.

Top 20 vSphere 5.5 Support Topics

Here’s our Top 20 vSphere 5.5 Knowledgebase articles for vSphere 5.5 and VMware Hypervisor 5.5

These KBs address the bulk of calls into our call centers for these products. See anything familiar in this list?

  1. VMware ESXi 5.x host experiences a purple diagnostic screen mentioning E1000PollRxRing and E1000DevRx
  2. Installing async drivers on ESXi 5.0, 5.1, and 5.5
  3. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and ESXi 5.x
  4. Collecting diagnostic information for VMware ESX/ESXi using the vSphere Client
  5. Re-pointing and re-registering VMware vCenter Server 5.1 / 5.5 and components
  6. Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x hosts
  7. Upgrading to vCenter Server 5.5 best practices
  8. Installing or upgrading to ESXi 5.5 best practices
  9. Investigating virtual machine file locks on ESXi/ESX
  10. Creating a persistent scratch location for ESXi 4.x and 5.x
  11. Reducing the size of the vCenter Server database when the rollup scripts take a long time to run
  12. Broadcom 5719/5720 NICs using tg3 driver become unresponsive and stop traffic in vSphere
  13. Methods for upgrading to ESXi 5.5
  14. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x
  15. Installing vCenter Server 5.5 best practices
  16. Restarting the Management agents on an ESXi or ESX host
  17. Powering off a virtual machine on an ESXi host
  18. Migrating the vCenter Server database from SQL Express to full SQL Server
  19. Resetting the VMware vCenter Server 5.x Inventory Service database
  20. Methods of upgrading to vCenter Server 5.5