Home > Blogs > VMware Support Insider > Category Archives: From the Trenches

Category Archives: From the Trenches

Using vSphere ESXi Image Builder to create an installable ISO that is not vulnerable to Heartbleed

Here is a follow-up post from Andrew Lytle, member of the VMware Mission Critical Support Team. Andrew is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

VMware recently released updates to all products affected by the vulnerability dubbed “Heartbleed” (CVE-2014-0160): http://www.vmware.com/security/advisories/VMSA-2014-0004.html

As per KB article: Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160 (2076665), the delivery method for this code change in the VMware ESXi product is through an updated ESXi vSphere Installation Bundle (VIB). VIBs are the building blocks of an ESXi image. A VIB is akin to a tarball or ZIP archive in that it’s a collection of files packaged into a single archive.

Typically a new ESXi ISO file will be made available only during major revisions of the product (Update 1, Update 2, etc). If you need an ESXi 5.5 ISO which is already protected from Heartbleed, you can make your own ISO easily using vSphere PowerCLI.

The PowerCLI ImageBuilder cmdlets are designed to make custom ESXi ISOs which have asynchronous driver releases pre-installed, but it can also be used in a situation like this to make an ISO which lines up with a Patch Release instead of a full ESXi Update Release.

In this post we will cover both the ESXi 5.5 GA branch, as well as the ESXi 5.5 Update 1 branch. Choose the set of steps which will provide the ISO branch you need for your environment.

Creating an ISO based on ESXi 5.5 GA (Pre-Update 1)

These steps are for downloading the requirements for creating an ISO which is based on the ESXi 5.5 “GA” release, which was originally released 2013-09-22.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Install PowerCLI through the Windows MSI package, and copy the zip files to a handy location. For the purposes of this example, I will copy these files to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404020.zip
    1-1

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has Version of 5.5.0-0.15.1746974 and the Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base
    1-2

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140401020s-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-heartbleed.iso
    1-3

Creating an ISO based on ESXi 5.5 Update 1

These steps are for creating an ISO which is based on the ESXi 5.5 “Update 1” release, which was originally released 2014-03-11.

Step 1: Download the Required Files

When creating a custom ESXi image through Image Builder, we need to start by downloading the required files:

Copy the zip files to a handy location. For the purposes of this example, I will copy it to C:\Patches\

Step 2: Import the Software Depot

  • Add-EsxSoftwareDepot C:\Patches\ESXi550-201404001.zip
    2-1

Step 3: Confirm the patched version (optional)

If you wish to confirm the esx-base VIB (which includes the Heartbleed vulnerability code change) is added correctly, you can confirm the VIB has the Version of 5.5.0-1.16.1746018 and Creation Date of 4/15/2014.

  • Get-EsxSoftwarePackages –Name esx-base
    2-2

Step 4: Export the Image Profile to an ISO

  • Export-EsxImageProfile –ImageProfile ESXi-5.5.0-20140404001-standard –ExportToISO –FilePath C:\Patches\ESXi5.5-update1-heartbleed.iso
    2-3

Installing the ESXi ISO

The ISO file which was created in this steps can be used in exactly the same manner as the normal VMware ESXi 5.5 ISO. It can be mounted in a remote management console, or burned to a CD/DVD for installation.

Patching ESXi 5.5 for Heartbleed without installing Update 1

On April 19th, VMware released a series of patches for ESX 5.5 and ESX 5.5 Update 1 to re-mediate the CVE dubbed “Heartbleed” (CVE-2014-0076 and CVE-2014-0160).

VMware also recently announced that there was an issue in the newest version of ESXi 5.5 (Update 1 and later), which can cause difficulties communicating with NFS storage. This NFS issue is still being investigated, and customers are encouraged to subscribe to KB article: Intermittent NFS APDs on ESXi 5.5 U1 (2076392) for updates.

Due to the confluence of these two unrelated issues, you might find yourself trying to patch ESXi to protect yourself from the Heartbleed vulnerability, while at the same time trying to avoid installing ESXi 5.5 Update 1.

Here is the information from the VMware Knowledge Base on the topic:

2

The note at the bottom is the key. Stated simply, if you are…

  • Using NFS storage
  • Concerned about patching to Update 1 due to change control
  • Not already running ESXi 5.5 Update 1 (build-1623387)

… then you should patch your install for the Heartbleed issue and at the same time stay at ESX 5.5 by applying Patch Release ESXi550-201404020, and not ESXi550-201404001.

An Explanation of Patch Release Codes

To better understand the Patching process in a VMware environment, it is valuable to understand the codes which are used in VMware Patch Releases.

When VMware releases a patch, or a series of patches, they are bundled together in what is known as a Patch Release. A Patch Release will have a coded name which is formed using the following structure. I have added braces to demonstrate the different sections better in each example.

[PRODUCT]-[YEAR][MONTH][THREE DIGIT RELEASE NUMBER]

For example, the patch release for ESXi 5.5 that was released in January 2013 would be coded like this (without the explanatory braces):

[ESXi550]-[2013]-[01][001]

As part of a Patch Release, there will be at least one Patch. Each Patch is given a Patch (or Bulletin) ID. Patch IDs are similarly structured to Patch Release codes, but also have a two letter suffix. For Security Bulletins, the prefix will be SG. For Bug Fix Bulletins, the prefix will be BG.

For example, the two Patch IDs which were released to patch Heartbleed are:

[ESXi550]-[2014][04][401]-[SG]
[ESXi550]-[2014][04][420]-[SG]

Note that the only difference in the Patch IDs here is in the three digit release number (401 vs 420).

Patching with VMware Update Manager

There are a number of methods for patching ESXi hosts, and the most commonly used is VMware Update Manager (VUM). VUM will present a pair of Dynamic Baselines which will be automatically updated when patches are available. The danger in this case is that VUM may show you both the Pre-Update 1 patch, as well as the Post-Update 1 patch. If you are not careful as to which patches you apply, you might accidentally end up patching your host to Post-Update 1.

Here are the patches which were released on April 19th, as seen in VUM. The Update 1 patch is highlighted in red, while the Pre-Update 1 patch is marked in green.

1

Note: VMware also released two other ESXi 5.5 patches on April 19th, as part of Patch Release but these are not related to the Heartbleed vulnerability in any fashion. (ESXi550-201404402-BG, and ESXi550-201404403-BG).

Creating a Fixed Baseline

Patching a host using ESXi550-201404420-SG (Pre-Update 1), while avoiding ESXi550-201404401-SG (Post-Update 1) requires the use of a Fixed Baseline in Update Manager.

  1. Start in the Update Manager Admin view.
  2. Select the Baselines and Groups tab.
  3. Click Create… in the Baselines column.
    3
  4. Give the new Baseline a descriptive Name (and optionally a Description).
    4
  5. Click Next.
  6. For Baseline type, select Fixed.
    5
  7. Use the Search feature to find the only Patch we want to apply. You will need to select the Patch ID option from the dropdown menu to ensure the search scans for the appropriate column.
    6
  8. Enter the Patch ID into the search field: ESXi550-201404420-SG and click Enter to search.
  9. Select the Patch which shows up in the filtered list, and click the Down Arrow to move it into the selected Baselines.
    7
  10. Click Next and confirm that the Patch ESXi550-201404420-SG is the only one selected.
    8
  11. Click Finish.

The Baseline is now created and available for use.

Remediating a Host using the Fixed Baseline

Once the Fixed Baseline has been created, we can use it to Scan and Remediate an ESXi host.

  1. Select the host you wish to patch, and place it into Maintenance Mode.
  2. Click the Update Manager tab.
  3. Make sure that there are no Dynamic Baselines attached to the host you wish to patch. Detach any baselines which are currently attached:
    Critical Host Patches (Predefined)
    Non-Critical Host Patches (Predefined)
    Any other Custom Baselines which you have created
  4. Click the Attach link.
    9
  5. Select the newly created Baseline and click Attach.
    10
  6. Click the Scan link and make sure Patches and Extensions is selected. Click Scan again.
    11
  7. When you are ready to patch the host, select Remediate.
  8. Complete the Remediation wizard.

Once the host is patched, it will reboot automatically.

Patching an ESXi host manually via the command line

Another option to patch an ESXi host is to use the esxcli command line tool. The patch files required are the same. For more information on how to proceed with this route, refer to the vSphere 5.5 Documentation under the heading Update a Host with Individual VIBs.

References

Author: Andrew Lytle
As a member of the VMware Mission Critical Support Team, Andrew Lytle is a Senior Support Engineer who is specializes in vCenter and ESXi related support.

New book: Getting Started with VMware Fusion

One of our own, Michael Roy has just published his first book: Getting Started with VMware Fusion, written to help readers to get started running Windows on their Mac the right way.

Michael talks about how to import your physical PC into the virtual world, and provides practical examples of how to keep your new Virtual Machine secure, backed up, and running smoothly.

Going a bit deeper, he teaches you about snapshots explaining their great uses, and also using Linked Clones in VMware Fusion Professional.

Michael Roy started at VMware working on VMware Fusion version 2 in 2009, where he co-led a world-class global support team, giving customers the help they needed to get the most out of VMware Fusion. He currently specializes in Technical Marketing for Hybrid Cloud Services.

iSCSI Storage and vMotion VLAN Best Practices

We got a question this morning on twitter from a customer asking for our best practices for setting up iSCSI storage and vMotion traffic on a VLAN.

The question caused a bit of a discussion here amongst our Tech Support staff and the answer it seems is too long to fit into a Tweet! Instead, here’s what you need to know if you are working on the best design for your VLANS.

iSCSI and vMotion on the same pipe (VLAN) is a big no-no unless you are using multiple teamed 1GbE uplinks or 10GbE uplinks with NIOC to avoid the two stomping on one another.

While vMotion traffic can be turned off/on/reconfigured on the fly, iSCSI traffic does not  handle any changes to the underlying network (though great improvements have been made 5.1/5.5) on the fly. You will need to take a maintenance window to reconfigure how you want your VLANs to function – especially for the iSCSI network – and then (more than likely) perform a rolling reboot of all hosts. If iSCSI traffic is already VLAN’d off, you should just leave the iSCSI traffic where it is as to avoid taking down the whole environment and just move the vMotion network to a separate VLAN.

That said, here is our most recent iSCSI Best Practice Setup guide from Cormac Hogan. Also see: vMotion Best Practice Setup guide.

Here are the pertinent pages in our documentation on the subject:

pubs.vmware.com…rking-guide.pdf – Page 187

and

pubs.vmware.com…orage-guide.pdf – Page 75

Installing async drivers in ESXi 5.x

One thing that catches a few customers up is the process of installing async drivers in their ESXi host … We have a KB article on the topic here, but there is more than one method to choose from and preparation steps involved. Since these steps might seem a little tricky, we decided a quick, live video explaining the topic might help many of you.

We called upon Kiwi Ssennyonjo to walk us through the salient points.

Again, the full KB article can be found here: Installing async drivers on ESXi 5.0/5.1/5.5 (2005205)

Using VisualEsxtop to troubleshoot performance issues in vSphere

What is VisualEsxtop?

VisualEsxtop is a new performance monitoring tool that was recently posted on the VMware Labs Flings project. On Flings, apps and tools built by VMware engineers for fun are available for download. The intent is to make VMware Administrators’ lives easier in their daily work.

Note:  VisualEsxtop is not an official VMware tool. For support and feedback please contact VMware Labs : http://labs.vmware.com/contact-us

VisualEsxtop is an enhanced version of resxtop and esxtop. VisualEsxtop can connect to VMware vCenter Server or ESX hosts, and display ESX server stats with a better user interface and more advanced features.

How to install VisualEsxtop ?

  1. Download visualEsxtop.zip from http://labs.vmware.com/flings/visualesxtop
  2. Unzip visualEsxtop.zip to folder
  3. Make sure Java 1.6 is in the PATH.
    1. On windows, to verify if Java is in the path,
      Click on Start > run > TYPE cmd and press ENTER > TYPE java and press Enter.
      If java is not in path, you will notice error like this -
    2. If JDK 1.6 or later is already installed on your machine but not in the path, here is how you add it (Instructions for Windows 7, other versions might be slightly different)
      -Go to Control Panel > System and Security > System >
      -Click Advanced system settings
      -Click on Environmental Variables
      -Click on New
      -Under Edit User Variable type the following and click OK
      Variable name: path
      Variable value: The path to the JDK 1.6 binary folder (C:\Program Files (x86)\Java\jdk1.6.0_14\bin\  for example)
    3. Then, open cmd again (Start > Run> Type cmd) and type “java”. This should successfully return usage options for java command.

What can you do with VisualEsxtop?

  • Real-time Performance monitoring of individual ESX(i) hosts or vCenter Server. The default interval (5 seconds) is modifiable. Type Ctrl+N and change to the new value
  • Multiple sessions to different hosts or same host at the same time. This comes in very handy when you are comparing stats between hosts or between multiple views/fields.
  • Flexible counter selection and filtering. This is in my opinion the best feature of this tool. You can filter results to get specific outputs. The examples in the next section will show you how to.
  • Save data to a batch file. You can now pick and choose relevant tabs and fields and also chose intervals, number of snapshots for the output. Type Ctrl+S to get the save option
  • Load batch output and replay them. Type Ctrl+B to load a saved csv file.
  • Line chart for selected performance counters
  • Embedded tooltip for counter description
  • Color coding for important counters

Running the tool:

  1. Run visualEsxtop.sh (Linux) or visualEsxtop.bat (Windows) from the extracted files.  (Note: Type export JAVA_OPTS=-Xmx2048m  if loading large amounts of data)
  2. On the VMWARE VisualEsxtop window, select File > Connect to Live server.
  3. Choose the IP address of the host or vCenter and the credentials to connect.

Examples of using the tool:

Example 1:

The example below is listing only the devices that have DAVG value of above 20ms. The filter used is DAVG/cmd under Disk World tab.  Typically we do not want DAVG (device driver level  latency) to be over 20ms for lengthy period. Note that by using the filter, it is very easy to list only the devices that currently have high latency values.

Example 2:

The example below is listing the vmkernels and virtual machines that are currently running on vmnic0. The filter used is TEAM-PNIC  under Network tab. You can also sort by %DRPTX , %DRPRX to filter for any devices reporting packet drops. Note that the vmnic number will only show up if the uplink ports are not in an etherchannel binding.

Tips on working with Charts:

  • To build a new chart, under Chart tab click twice on Object Types to start to select fields
  • To add a field to the chart, expand the related object and click twice on the field
  • To remove a field from the current chart, click twice on the field from the bottom left window pane
  • The chart allows you to add any fields from any objects at the same time. You have to be the judge regarding what fields are relevant. For example, listing DAVG and Physical CPU Core Util% in the same chart may not provide much value.

Chart view (tab) screenshot:

Where do I get tips on Troubleshooting performance on ESXi ?

Details about different fields in esxtop tool can be found in this communities blog post: Interpreting esxtop Statistics.

There are many great articles and tips on performance troubleshooting in the VMware Knowledgebase. Here are a couple that I recommend to give you a start –

Announcing: Support Insider Live

Announcing Support Insider LiveWe are delighted to announce a new video initiative from the Knowledge Management team at VMware that will bring you tips straight from the mouths of our front-line Technical Support Engineers.

Support Insider Live videos are short, to the point nuggets of wisdom from those who work on customer issues daily. Ranging from basic to advanced, each video will address one idea, probably answering a question you’ve asked yourself.

Videos are hosted on our VMware KBTV channel on YouTube in a new playlist called Support Insider Live. Each video will be blogged about here if you wish to consume them that way. Our first video answers the question, “What is a PSOD?”

We’d love to hear your feedback! Have any requests?

Troubleshooting when virtual machine operations are greyed out

Hello, I am a Technical Support Engineer at VMware. I would like to talk about an issue I worked on recently with a customer that may come up for other users. The Support Request (SR) came to me with this description: “The Virtual Machine options are not working”. I had to ask myself, “What do you mean by ‘not working’”? I quickly called the customer and we got a remote session going so that I could see for myself what the problem was.

The problem was with only one Virtual Machine in that vCenter instance. I found that the virtual machine options were disabled (greyed-out) when I right-clicked the problematic virtual machine.

Note: Click an image to see the full-sized version.

The customer at this point mentioned that he had tried to take a backup using third-party software. He saw this error message in the “Recent Tasks” pane:

Error: Another task in progress

My initial hunch was that there was a job running for the virtual machine in the background, but I needed to verify it wasn’t a permissions problem.

Permissions are not set correctly

First I checked the permissions given to that virtual machine. The customer had a very small environment, so I did not spend too much time checking the complete permissions given for the entire vSphere environment. Instead, I went through the following steps:

Note: If your environment is large and you have multiple vSphere administrators with different permissions, permissions on a particular virtual machine machine might be incorrect/missing. In this case, the vSphere root administrator needs to ensure that there are sufficient permissions given for you to administer the virtual machine, at all levels.

  1. Check the Permissions tab for the virtual machine, to make sure that your name is listed there with the proper permission assigned. If your name is not there, but an AD/Local group is listed, then make sure that your name is added to the AD/Local group.Here, the user Testuser has the Virtual Machine Administrator role:
  2. If the permissions are defined at the host, cluster, Datacenter, or vCenter level:
      1. Apply the required permissions to the user/group.
      2. Make sure that the permission is propagated to all the objects.
      3. Make sure that your name is listed in the local/AD group.Here, the group Virtual Machine Administrator has the Virtual Machine Administrator role:

  3. Roles can also be assigned in the folder level. Go to the VMs and Templates view and make sure the folder where the Virtual Machine located has all required privileges assigned.In the below example, VM1,VM2 and VM3 are located in the folder Test. The Virtual Machine Administrator group is assigned with “Virtual Machine Administrator” role.
    In the examples above, the role “Virtual Machine Administrator” is not a default role, it is created with privileges to Administrate Virtual Machines. To know what privileges are required, I always refer to the Required Privileges for Common Tasks section in: Virtual Machine Administration Guide.

With that set of checks, we can now go on to the next step, and this comes back to my hunch because of the error message which I noticed in the “Recent Task”.

Tasks Running in the Background

We should always be aware that certain jobs for any given Virtual Machine may be running in the background which will not allow any changes/operations in the Virtual Machine. These background jobs cannot be seen in the “Task and Event” tab in the vCenter server or in the “Recent Tasks”.

The method we look for these background jobs is by logging into the host via SSH where the problem Virtual Machine is running. Here are the steps to find any running jobs which cannot be seen in the vSphere client.

1. Log into the ESX/ESXi host using a SSH client.
2. Run the following command to list all the Virtual Machines registered in the host. We are running this specifically to find the vmid for the problem Virtual Machine.

vim-cmd vmsvc/getallvms

The output appears as:

3. Check the tasks running on the Virtual Machine by running the command:

vim-cmd vmsvc/get.tasklist vmid

In our example, we run the command against the Virtual Machine VM3. It indicates there are 2 jobs/tasks running on that Virtual Machine.

The output shows as below if there is no jobs/tasks running.

4. Now you you have a choice to either wait for the jobs to complete, or restart the Management agent to terminate this job.

In this particular support case I was working we found a snapshot job that was triggered by the backup agent running in the Virtual Machine. I stopped the job by restarting the management agent on that host. For this customer, this solved the problem, but there is one more reason this might happen. I had a word with one my colleagues, and he pointed out that you could also encounter this if you have an invalid entry in your VMX file. Let’s go one step further and show you how to check this.

Invalid entries in VMX file

This type of issue might happen if a vmx file has invalid parameters or blank lines in it. You can resolve this issue by manually removing the invalid arguments or deleting the blank lines.

Caution: If incorrectly done, your virtual machine may fail to start or operate incorrectly. Always take a backup of your .vmx file before modifying it.

  1. Open the .vmx file using any text editor.
  2. Search for any blank lines and delete them.
    Note
    : To delete a single line using vi editor, press d twice.
  3. Compare the .vmx file with a working virtual machine .vmx file and see if there are any invalid arguments.
  4. To apply the changes, reload the .vmx file by running the command:
vim-cmd vmsvc/reload vmid

Note: The Default location of the vmx file is:

Vmfs/volumes/name_of_the_datastore/vm_name/vm_name.vmx

Summary

Symptom:
Virtual machine operations are grayed out

Possible Causes:

  • Permissions are not set correctly
  • Tasks Running in the Background
  • vmx file is corrupted

For further information on the steps used in this article, refer to: Troubleshooting when virtual machine options are grayed out in vSphere Client (2048748)

Hope this blog post helps you out a little bit. Have a great day!

Attempting to Sysprep a Virtual Machine with IE10 fails

We just received word from one of our support engineers working on the front-lines that some customers are reporting problems wherein Sysprep fails on Virtual Machines that have IE10 on them.

The issue is not a VMware bug, it’s an Registry location problem, but customers might still reach for the phone to call us first. We thought we’d try and get the word out today to try and save you some steps (and time).

The problem manifests itself when you attempt to Sysprep a Virtual Machine that has IE10 on it.  In the Sysprep setupact.log log file (located at C:\Windows\System32\Sysprep\Panther), you see:

Error[0x0f0085]SYSPRP LaunchDll:Could not load DLL C:\Windows\SysWOW64\iesysprep.dll[gle=0x000000c1]

The registry entry for the location of certain files is incorrect. For further details and resolution, we refer you to KB article: Sysprep fails on Virtual Machine installed with IE10 (2051620).

VMware vFabric Postgres Cheat-sheet

Here’s a cool pdf download for you. It’s a cube note (or cheat sheet) that you can use when troubleshooting Postgres issues with VMware vFabric, straight from our awesome team of storage support engineers.

VMware vFabric Postgres Chart
VMware vFabric Postgres Chart