Home > Blogs > Support Insider > Category Archives: From the Trenches

Category Archives: From the Trenches

Issues creating Desktops in VDI, and what to do about it

Connection Server rebootWe want to highlight some mitigation techniques and a handy KB article today for those of you managing Horizon View desktops. We’re talking about those occasions when no desktop can be created or recomposed in your vdi environment and no tasks submitted from Connection Brokers are acted upon by Virtual Center server.

Our Engineering team has been hard at work fixing many of the underlying causes of this happening and the latest release of View have all but eliminated these issues. However, if these issues show up in latest View releases, then we ask everyone to follow the specific steps in this KB: Pool settings are not saved, new pools cannot be created, and vCenter Server tasks are not processed in a Horizon View environment (2082413)

This KB contains several main steps, the first one of which is collecting the bundled logs from all connection brokers in the vdi environment and recording the time this issue was first observed. Steps 2 to 6 are basic steps that can potentially help address the issue but if issues persist, then step 7 requests opening a support case and submitting the collected bundle logs in step 1 alongside the recorded time when the issue was first observed. You might also include any other useful information, such as whether any recent changes were made to the environment.

When opening your support case, please note this KB article number in the description the case. That helps us get right on point ASAP.

Step 8 is what should address this issue without any connection broker reboot as it causes the stoppage of all View services in all View connection brokers and then restarting them.

If step 8 does not resolve your issue, then the last step (9) involves reboot of all connection serves and this has always addressed the issue in our experience.

Fresh vSphere 6 KB articles!

vSphere 6.0 has been out now for a few weeks and you early adopters have been busy kicking the tires. We’ve heard some very encouraging things about this release ie: the web client improvements. It’s always interesting and top of mind for us to see what issues emerge in everyone’s environments and we monitor support requests coming into support as well as social media to see what customers run into.

Here’s an fresh list of Knowledgebase articles we’ve created to address some of these inquiries. Familiarize yourself with the list and of course share with your colleagues using the buttons on this page.

Database compatibility issues during upgrade

Deprecated VMFS volume errors

Backup failures/CBT mem heap issues

Replace certificates for vSphere 6.0

Decommissioning a vCenter Server or Platform Services Controller

When Linked Clones Go Stale

One of the biggest call drivers within our VMware View support centers revolves around linked clone pools. Some of your users may be calling you to report that their desktop is not available. You begin to check your vCenter and View Administrator portal and discover some of the following symptoms:

  • You cannot provision or recompose a linked clone desktop pool
  • You see the error:
    Desktop Composer Fault: Virtual Machine with Input Specification already exists
  • Provisioning a linked clone desktop pool fails with the error:
    Virtual machine with Input Specification already exists
  • The Connection Server shows that linked clone virtual machines are stuck in a deleting state
  • You cannot delete a pool from the View Administrator page
  • You are unable to delete linked clone virtual machines
  • When viewing a pools Inventory tab, the status of one or more virtual machines may be shown as missing

There are a number of reasons this might happen, and KB: 2015112 Manually deleting linked clones or stale virtual desktop entries from the View Composer database in VMware View Manager and Horizon View covers resolving this topic comprehensively, but let’s discuss a bit of the background around these issues.

When a linked clone pool is created or modified, several backend databases are updated with configuration data. First there is the SQL database supporting vCenter Server, next there is the View Composer database, and thirdly the ADAM database. Let’s also throw in Active Directory for good measure. With all of these pieces each playing a vital role in the environment, it becomes apparent that should things go wrong, there may be an inconsistency created between these databases. These inconsistencies can present themselves with the above symptoms.

Recently a new Fling was created to address these inconsistencies. If you’re not acquainted with Flings, they’re tools our engineers build to help you explore and manipulate your systems. However, it’s important to remember they come with a disclaimer:

“I have read and agree to the Technical Preview Agreement. I also understand that Flings are experimental and should not be run on production systems.”

If you’re just in your lab environment though, they are an excellent way to learn and understand the workings of your systems at a deeper level. Here is the Fling: ViewDbChk. For production systems we recommend following the tried and true procedures documented in KB 2015112. The KB includes embedded videos to help walk you through the steps.

Using vSphere ESXi Image Builder to create an installable ISO that is not vulnerable to Heartbleed

Here is a follow-up post from Andrew Lytle, member of the VMware Mission Critical Support Team. Andrew is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

VMware recently released updates to all products affected by the vulnerability dubbed “Heartbleed” (CVE-2014-0160): http://www.vmware.com/security/advisories/VMSA-2014-0004.html

As per KB article: Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160 (2076665), the delivery method for this code change in the VMware ESXi product is through an updated ESXi vSphere Installation Bundle (VIB). VIBs are the building blocks of an ESXi image. A VIB is akin to a tarball or ZIP archive in that it’s a collection of files packaged into a single archive.

Typically a new ESXi ISO file will be made available only during major revisions of the product (Update 1, Update 2, etc). If you need an ESXi 5.5 ISO which is already protected from Heartbleed, you can make your own ISO easily using vSphere PowerCLI.

The PowerCLI ImageBuilder cmdlets are designed to make custom ESXi ISOs which have asynchronous driver releases pre-installed, but it can also be used in a situation like this to make an ISO which lines up with a Patch Release instead of a full ESXi Update Release.

In this post we will cover both the ESXi 5.5 GA branch, as well as the ESXi 5.5 Update 1 branch. Choose the set of steps which will provide the ISO branch you need for your environment.

Creating an ISO based on ESXi 5.5 GA (Pre-Update 1)

These steps are for downloading the requirements for creating an ISO which is based on the ESXi 5.5 “GA” release, which was originally released 2013-09-22.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Install PowerCLI through the Windows MSI package, and copy the zip files to a handy location. For the purposes of this example, I will copy these files to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404020.zip

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has Version of 5.5.0-0.15.1746974 and the Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140401020s-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-heartbleed.iso

Creating an ISO based on ESXi 5.5 Update 1

These steps are for creating an ISO which is based on the ESXi 5.5 “Update 1” release, which was originally released 2014-03-11.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Copy the zip files to a handy location. For the purposes of this example, I will copy it to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404001.zip

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has the Version of 5.5.0-1.16.1746018 and Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140404001-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-update1-heartbleed.iso

Installing the ESXi ISO

The ISO file which was created in this steps can be used in exactly the same manner as the normal VMware ESXi 5.5 ISO. It can be mounted in a remote management console, or burned to a CD/DVD for installation.

Patching ESXi 5.5 for Heartbleed without installing Update 1

On April 19th, VMware released a series of patches for ESX 5.5 and ESX 5.5 Update 1 to re-mediate the CVE dubbed “Heartbleed” (CVE-2014-0076 and CVE-2014-0160).

VMware also recently announced that there was an issue in the newest version of ESXi 5.5 (Update 1 and later), which can cause difficulties communicating with NFS storage. This NFS issue is still being investigated, and customers are encouraged to subscribe to KB article: Intermittent NFS APDs on ESXi 5.5 U1 (2076392) for updates.

Due to the confluence of these two unrelated issues, you might find yourself trying to patch ESXi to protect yourself from the Heartbleed vulnerability, while at the same time trying to avoid installing ESXi 5.5 Update 1.

Here is the information from the VMware Knowledge Base on the topic:


The note at the bottom is the key. Stated simply, if you are…

  • Using NFS storage
  • Concerned about patching to Update 1 due to change control
  • Not already running ESXi 5.5 Update 1 (build-1623387)

… then you should patch your install for the Heartbleed issue and at the same time stay at ESX 5.5 by applying Patch Release ESXi550-201404020, and not ESXi550-201404001.

An Explanation of Patch Release Codes

To better understand the Patching process in a VMware environment, it is valuable to understand the codes which are used in VMware Patch Releases.

When VMware releases a patch, or a series of patches, they are bundled together in what is known as a Patch Release. A Patch Release will have a coded name which is formed using the following structure. I have added braces to demonstrate the different sections better in each example.


For example, the patch release for ESXi 5.5 that was released in January 2013 would be coded like this (without the explanatory braces):


As part of a Patch Release, there will be at least one Patch. Each Patch is given a Patch (or Bulletin) ID. Patch IDs are similarly structured to Patch Release codes, but also have a two letter suffix. For Security Bulletins, the prefix will be SG. For Bug Fix Bulletins, the prefix will be BG.

For example, the two Patch IDs which were released to patch Heartbleed are:


Note that the only difference in the Patch IDs here is in the three digit release number (401 vs 420).

Patching with VMware Update Manager

There are a number of methods for patching ESXi hosts, and the most commonly used is VMware Update Manager (VUM). VUM will present a pair of Dynamic Baselines which will be automatically updated when patches are available. The danger in this case is that VUM may show you both the Pre-Update 1 patch, as well as the Post-Update 1 patch. If you are not careful as to which patches you apply, you might accidentally end up patching your host to Post-Update 1.

Here are the patches which were released on April 19th, as seen in VUM. The Update 1 patch is highlighted in red, while the Pre-Update 1 patch is marked in green.


Note: VMware also released two other ESXi 5.5 patches on April 19th, as part of Patch Release but these are not related to the Heartbleed vulnerability in any fashion. (ESXi550-201404402-BG, and ESXi550-201404403-BG).

Creating a Fixed Baseline

Patching a host using ESXi550-201404420-SG (Pre-Update 1), while avoiding ESXi550-201404401-SG (Post-Update 1) requires the use of a Fixed Baseline in Update Manager.

  1. Start in the Update Manager Admin view.
  2. Select the Baselines and Groups tab.
  3. Click Create… in the Baselines column.
  4. Give the new Baseline a descriptive Name (and optionally a Description).
  5. Click Next.
  6. For Baseline type, select Fixed.
  7. Use the Search feature to find the only Patch we want to apply. You will need to select the Patch ID option from the dropdown menu to ensure the search scans for the appropriate column.
  8. Enter the Patch ID into the search field: ESXi550-201404420-SG and click Enter to search.
  9. Select the Patch which shows up in the filtered list, and click the Down Arrow to move it into the selected Baselines.
  10. Click Next and confirm that the Patch ESXi550-201404420-SG is the only one selected.
  11. Click Finish.

The Baseline is now created and available for use.

Remediating a Host using the Fixed Baseline

Once the Fixed Baseline has been created, we can use it to Scan and Remediate an ESXi host.

  1. Select the host you wish to patch, and place it into Maintenance Mode.
  2. Click the Update Manager tab.
  3. Make sure that there are no Dynamic Baselines attached to the host you wish to patch. Detach any baselines which are currently attached:
    Critical Host Patches (Predefined)
    Non-Critical Host Patches (Predefined)
    Any other Custom Baselines which you have created
  4. Click the Attach link.
  5. Select the newly created Baseline and click Attach.
  6. Click the Scan link and make sure Patches and Extensions is selected. Click Scan again.
  7. When you are ready to patch the host, select Remediate.
  8. Complete the Remediation wizard.

Once the host is patched, it will reboot automatically.

Patching an ESXi host manually via the command line

Another option to patch an ESXi host is to use the esxcli command line tool. The patch files required are the same. For more information on how to proceed with this route, refer to the vSphere 5.5 Documentation under the heading Update a Host with Individual VIBs.


Author: Andrew Lytle
As a member of the VMware Mission Critical Support Team, Andrew Lytle is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

New book: Getting Started with VMware Fusion

One of our own, Michael Roy has just published his first book: Getting Started with VMware Fusion, written to help readers to get started running Windows on their Mac the right way.

Michael talks about how to import your physical PC into the virtual world, and provides practical examples of how to keep your new Virtual Machine secure, backed up, and running smoothly.

Going a bit deeper, he teaches you about snapshots explaining their great uses, and also using Linked Clones in VMware Fusion Professional.

Michael Roy started at VMware working on VMware Fusion version 2 in 2009, where he co-led a world-class global support team, giving customers the help they needed to get the most out of VMware Fusion. He currently specializes in Technical Marketing for Hybrid Cloud Services.

iSCSI Storage and vMotion VLAN Best Practices

We got a question this morning on twitter from a customer asking for our best practices for setting up iSCSI storage and vMotion traffic on a VLAN.

The question caused a bit of a discussion here amongst our Tech Support staff and the answer it seems is too long to fit into a Tweet! Instead, here’s what you need to know if you are working on the best design for your VLANS.

iSCSI and vMotion on the same pipe (VLAN) is a big no-no unless you are using multiple teamed 1GbE uplinks or 10GbE uplinks with NIOC to avoid the two stomping on one another.

While vMotion traffic can be turned off/on/reconfigured on the fly, iSCSI traffic does not  handle any changes to the underlying network (though great improvements have been made 5.1/5.5) on the fly. You will need to take a maintenance window to reconfigure how you want your VLANs to function – especially for the iSCSI network – and then (more than likely) perform a rolling reboot of all hosts. If iSCSI traffic is already VLAN’d off, you should just leave the iSCSI traffic where it is as to avoid taking down the whole environment and just move the vMotion network to a separate VLAN.

That said, here is our most recent iSCSI Best Practice Setup guide from Cormac Hogan. Also see: vMotion Best Practice Setup guide.

Here are the pertinent pages in our documentation on the subject:

pubs.vmware.com…rking-guide.pdf – Page 187


pubs.vmware.com…orage-guide.pdf – Page 75

Installing async drivers in ESXi 5.x

One thing that catches a few customers up is the process of installing async drivers in their ESXi host … We have a KB article on the topic here, but there is more than one method to choose from and preparation steps involved. Since these steps might seem a little tricky, we decided a quick, live video explaining the topic might help many of you.

We called upon Kiwi Ssennyonjo to walk us through the salient points.

Again, the full KB article can be found here: Installing async drivers on ESXi 5.0/5.1/5.5 (2005205)

Using VisualEsxtop to troubleshoot performance issues in vSphere

What is VisualEsxtop?

VisualEsxtop is a new performance monitoring tool that was recently posted on the VMware Labs Flings project. On Flings, apps and tools built by VMware engineers for fun are available for download. The intent is to make VMware Administrators’ lives easier in their daily work.

Note:  VisualEsxtop is not an official VMware tool. For support and feedback please contact VMware Labs : http://labs.vmware.com/contact-us

VisualEsxtop is an enhanced version of resxtop and esxtop. VisualEsxtop can connect to VMware vCenter Server or ESX hosts, and display ESX server stats with a better user interface and more advanced features.

How to install VisualEsxtop ?

  1. Download visualEsxtop.zip from http://labs.vmware.com/flings/visualesxtop
  2. Unzip visualEsxtop.zip to folder
  3. Make sure Java 1.6 is in the PATH.
    1. On windows, to verify if Java is in the path,
      Click on Start > run > TYPE cmd and press ENTER > TYPE java and press Enter.
      If java is not in path, you will notice error like this –
    2. If JDK 1.6 or later is already installed on your machine but not in the path, here is how you add it (Instructions for Windows 7, other versions might be slightly different)
      -Go to Control Panel > System and Security > System >
      -Click Advanced system settings
      -Click on Environmental Variables
      -Click on New
      -Under Edit User Variable type the following and click OK
      Variable name: path
      Variable value: The path to the JDK 1.6 binary folder (C:\Program Files (x86)\Java\jdk1.6.0_14\bin\  for example)
    3. Then, open cmd again (Start > Run> Type cmd) and type “java”. This should successfully return usage options for java command.

What can you do with VisualEsxtop?

  • Real-time Performance monitoring of individual ESX(i) hosts or vCenter Server. The default interval (5 seconds) is modifiable. Type Ctrl+N and change to the new value
  • Multiple sessions to different hosts or same host at the same time. This comes in very handy when you are comparing stats between hosts or between multiple views/fields.
  • Flexible counter selection and filtering. This is in my opinion the best feature of this tool. You can filter results to get specific outputs. The examples in the next section will show you how to.
  • Save data to a batch file. You can now pick and choose relevant tabs and fields and also chose intervals, number of snapshots for the output. Type Ctrl+S to get the save option
  • Load batch output and replay them. Type Ctrl+B to load a saved csv file.
  • Line chart for selected performance counters
  • Embedded tooltip for counter description
  • Color coding for important counters

Running the tool:

  1. Run visualEsxtop.sh (Linux) or visualEsxtop.bat (Windows) from the extracted files.  (Note: Type export JAVA_OPTS=-Xmx2048m  if loading large amounts of data)
  2. On the VMWARE VisualEsxtop window, select File > Connect to Live server.
  3. Choose the IP address of the host or vCenter and the credentials to connect.

Examples of using the tool:

Example 1:

The example below is listing only the devices that have DAVG value of above 20ms. The filter used is DAVG/cmd under Disk World tab.  Typically we do not want DAVG (device driver level  latency) to be over 20ms for lengthy period. Note that by using the filter, it is very easy to list only the devices that currently have high latency values.

Example 2:

The example below is listing the vmkernels and virtual machines that are currently running on vmnic0. The filter used is TEAM-PNIC  under Network tab. You can also sort by %DRPTX , %DRPRX to filter for any devices reporting packet drops. Note that the vmnic number will only show up if the uplink ports are not in an etherchannel binding.

Tips on working with Charts:

  • To build a new chart, under Chart tab click twice on Object Types to start to select fields
  • To add a field to the chart, expand the related object and click twice on the field
  • To remove a field from the current chart, click twice on the field from the bottom left window pane
  • The chart allows you to add any fields from any objects at the same time. You have to be the judge regarding what fields are relevant. For example, listing DAVG and Physical CPU Core Util% in the same chart may not provide much value.

Chart view (tab) screenshot:

Where do I get tips on Troubleshooting performance on ESXi ?

Details about different fields in esxtop tool can be found in this communities blog post: Interpreting esxtop Statistics.

There are many great articles and tips on performance troubleshooting in the VMware Knowledgebase. Here are a couple that I recommend to give you a start –

Announcing: Support Insider Live

Announcing Support Insider LiveWe are delighted to announce a new video initiative from the Knowledge Management team at VMware that will bring you tips straight from the mouths of our front-line Technical Support Engineers.

Support Insider Live videos are short, to the point nuggets of wisdom from those who work on customer issues daily. Ranging from basic to advanced, each video will address one idea, probably answering a question you’ve asked yourself.

Videos are hosted on our VMware KBTV channel on YouTube in a new playlist called Support Insider Live. Each video will be blogged about here if you wish to consume them that way. Our first video answers the question, “What is a PSOD?”

We’d love to hear your feedback! Have any requests?