VMware ESXi Chronicles

Kyle Gleed, Sr. Technical Marketing Manager, VMware

For folks who are still running the classic ESX hypervisor there is a new fling available to help smooth your migration to ESXi.  The ESX System Analyzer scans existing ESX hosts looking for issues that could affect your transition to ESXi.  It produces a nice report that you can use to assess your readiness and identify any potential risks that you should be aware of before moving to ESXi.  Check it out, I think you'll like it. 

Kyle Gleed, Sr. Technical Marketing Manager, VMware

Duncan Epping recently posted some great info on creating custom VIB files (How to create your own .vib files, Some more nuggets about handling vib files).  With custom VIBs making their way into the community this got me thinking that a quick refresher on VIB security would be helpful.  A while back I posted a VIB overview blog in which I discussed how signature files are used to help not only identify if a VIB is officially supported, but also to protect the against any malicious tampering of the VIBs contents.  While custom VIBs definitely have their place (see KB 2007381), do exercise caution when adding them to your ESXi image profiles.  Here's a quick recap of the section on VIB security:

The signature fileis an electronic signature used to verify the level of trust associated with the VIB.  The acceptance level not only helps protect the integrity of the VIB, but it also identifies who created the VIB and the amount of testing and verification that has been done. There are four acceptance levels:

• VMwareCertified:  VIBs created and tested by VMware.  VMware Certified VIBs undergo thorough testing by VMware.

• VMwareAccepted:  VIBs created by a VMware partners that are approved by VMware.  VMware relies on partners to perform the testing, but VMware verifies the results.

• PartnerSupported:  VIBs created and tested by a trusted VMware partner.  The partner performs all testing.  VMware does not verify the results.

• CommunitySupported:  VIBs created by individuals or partners outside of the VMware partner program.  These VIBs do not undergo any VMware or trusted partner testing and are not supported by VMware or its partners. 

All VMware and partner supported VIBs must be signed by a VMware trusted authority, this helps ensure the security of the VIB by preventing any unauthorized tampering of its contents.   Community supported VIBs do not need to be signed, but they are still required to have an empty signature file.  Be careful when using CommunitySupported VIBs as their contents are not tested, monitored or controlled. 

Coinciding with the VIB acceptance levels, ESXi Image Profiles also have an acceptance level.  When the image is created it is assigned one of the four acceptance levels.  Any VIBs added to the image must be at the same acceptance level or higher.  This helps ensure that non-supported VIBs don’t get mixed in with supported VIBs when creating and maintaining ESXi images.

Kyle Gleed, Sr. Technical Marketing Manager, VMware

I often get asked for advice on upgrading VMware Tools and Virtual Hardware following an ESXi host upgrade.  While the actual task of upgrading a VM is straight forward, the challenge comes because upgrading the tools/virtual hardware requires rebooting the VM, and with many VMs hosting production workloads there’s a lot of sensitivity around VM downtime.  

In the past VMware has recommended upgrading the VM tools/virtual hardware anytime you upgrade your ESXi host.  This meant following an ESXi host upgrade with a lot of VM upgrades and reboots.  However, starting with vSphere 5.0 this is no longer the case.  In 5.0 VMware has extended the tools/virtual hardware support matrix to allow VMs with older versions of tools/virtual hardware to run on newer ESXi host.  This allows customers to evaluate the added features and capabilities provided with the newer tools/virtual hardware versions and make an informed decision on whether or not to upgrade their VMs, and to avoid unnecessary upgrades.

The table below shows the virtual hardware support matrix for vSphere 5.0.  As you can see vSphere 5.0 supports VMs running virtual hardware versions 4, 7, and 8.  Additional information about virtual hardware support in 5.0 and how to upgrade a VMs virtual hardware is available in the vSphere Virtual Machine Administration Guide (page 85).

In addition, the VMware Tools support matrix for each ESXi version is available from the VMware Product Interoperability Matrix.  Note that vSphere 5.0 supports VMs runnig VMware Tools version 4.0 and above.  In addition, VMware Tools 5.0 is also supported on older ESX/ESXi hosts (4.0 and above).

Kyle Gleed, Sr. Technical Marketing Manager, VMware

Heads up for folks running ESXi 4.0 update 2.  You will need to apply ESXi 4.0 Update 3 prior to upgrading vCenter to 5.0.  For more info check out KB article 2007269.

Kyle Gleed, Sr. Technical Marketing Manager, VMware

I was recently looking for a quick way to verify that all my VMs have been updated with the latest versions of tools and virtual hardware.  I was thinking I would need to use a PowerCLI or vCLI script to do this, but was pleasantly surprised when I discovered that I can easily get this info from the vSphere client.

Simply select your data center or cluster and go to the "Virtual Machines" tab.  Here you can customize the columns that are displayed by right clicking on any of the column headings.  From the list of available options select "VM Version" and VMware Tools Version Status". 

Once the new columns were added I was able to drag and drop the column headings to change the display order, making it possible to see the VM name, virtual hardware version, and VMware Tools version in one view without having to use the horizontal scroll bar.  

End result is just want I needed, a list of my VMs along with the virtual hardware and VMware Tools versions - and no scripting was required!

Kyle Gleed, Sr. Technical Marketing Manager, VMware
 
I often get asked if it's better to upgrade an ESX/ESXi 4.x host or just do a fresh install of ESXi 5.0.  Before I answer this question lets review the differences between an upgraded host and a freshly installed host.

1.  Boot disk partition format:  When upgrading, the boot disk retains the older MBR (Master Boot Record) partition format where on a new install the boot disk is formatted as GPT (GUID Partition Table).  The key here being that the new GPT format enables you to use LUNs larger than 2TB and up to 64TB.  The larger LUN size support is typically not an issue for ESXi boot disks.

2.  VMFS volume version:  When you upgrade, the existing boot disk VMFS-3 volume is preserved.  On a fresh ESXi 5.0 install a new VMFS-5 volume is created.   Keep in mind that you can always upgrade the VMFS-3 volume to VMFS-5 after the host upgrade.  To understand the implications of upgrading a VMFS-3 volume to VMFS-5 check out this blog. 

The above two issues apply to both ESX and ESXi hosts.  ESX hosts have one additional difference to consider:

 
3.  Location of the ESXi scratch partition:    When you do a fresh ESXi install a 4GB VFAT partition is created for the scratch partition (assuming you are installing to a local or SAN disk).  However, when you upgrade, instead of a dedicated disk partition a scratch directory is created on the VMFS datastore.  While the location of /scratch is different, there is no operational significance to using a VMFS directory compared to a disk partition.  Also, keep in mind that you can always change the location of /scratch at any time.
 
With these considerations in mind, I don't see any disadvantages to upgrading a host compared to doing a fresh install.  What's more, I think upgrading has advantages over doing a fresh install: (1) the upgrade preserves the host configuration eliminating the need to manually reconfigure each host, and (2) the data on the VMFS datastore is preserved eliminating the need to manually migrate data off the boot disk or having to rely on backup and restore tools.  This not only helps to speed up the upgrade process, but also help reduce the risks of losing data or running into configuration errors while reconfiguring the hosts. 

As such, I personally tend to recommend doing in-place upgrades, especially if you have a lot of ESX/ESXi hosts to upgrade.   However, if you only have a small number of hosts and are comfortable doing the manual reconfiguration, doing a fresh install works as well.  It's largely an issue of personal preference and which approach is convenient for you.
 
For more information on upgrading to ESXi 5.0 check out the vSphere Upgrade Best Practices white paper.

There are so many new features and capabilities in vSphere 5.0 that it's becoming increasingly difficult to keep up with everything.  When it comes to getting a quick introduction on a new feature I've found a great resource is the VMware TechPub's Channel on YouTube.   The channel contains several short videos (~3 minutes) that provide a high level overview of various features and capabilities.  If you have a question about a new feature and what it does, this is a great place to start.

http://www.youtube.com/user/VMwareTechPubs


Here's an example of just some of the videos available today (and there are more coming):

ESX/ESXi Convergence

Using Image Builder CLI

Auto Deploy Architecture

Using Host Profiles

Troubleshooting Smart Cards in VMware View

vSphere Network I/O Control

Kyle Gleed, Sr. Technical Marketing Manager, VMware

(Note original post was updated on 28 Oct 2011 to better clarify ESX/ESXi 3.5 support in a mixed cluster.)

Running different versions of ESX/ESXi in an vCenter 5.0 HA/DRS cluster is supported.  Frank Denneman recently posted a good blog on this.  You may ask why would anyone want to run a mixed cluster?  Usually, this is done to facilitate rolling upgrades.  If you have a large 32-node cluster it's not practical to upgrade all 32 hosts at once, so instead you can leverage the mixed cluster support to upgrade two or three hosts at a time and "roll" the upgrade through the cluster until all 32 hosts are upgraded.

While mixed clusters are supported there are some things to watch for, specifically the VMware tools and virtual hardware versions of your VMs.  The table below provides a summary of the VMware tools and virtual hardware versions that are supported on both vSphere 4.x and 5.0.

   
From the table we can see that:

1.  VMs with virtual hardware and VMware tools version 3 are not supported on ESXi 5.0 hosts.

2.  VMs with virtual hardware version 8 are not supported on pre-5.0 hosts.

3.  VMFS-5 is not supported on pre-5.0 hosts.

With these limitations in mind, my recommendations for running mixed clusters are as follows:

1.  Verify VM virtual hardware and VMware tools versions before mixing 3.5 and 5.0 hosts in the same cluster.  VMs with VMware tools version 3 and virtual hardware version 3 are not supported on ESXi 5.0.  To avoid potential pitfalls, be sure to upgrade VM hardware versions to version 4 and VMware tools version to 3.5 before mixing  3.5 and 5.0 hosts in the same cluster.

2.  Do not upgrade virtual hardware versions while running in a mixed mode.  Once you upgrade a VMs virtual hardware version to 8, it can no longer run on a pre-5.0 ESX/ESXi hosts.  In addition, there is no option to undo the upgrade or revert back to an earlier virtual hardware version.  As such, while running a mixed cluster you should avoid upgrading the virtual hardware version of your VMs to version 8 until after all hosts have been upgraded to ESXi 5.0.

3.  Do not upgrade VMFS-3 volumes to VMFS-5 while running in mixed mode.  Wait until after all the hosts in the cluster are running ESXi 5.0 to upgrade VMFS volumes.  Upgrading to VMFS-5 will prevent any pre-5.0 hosts from accessing the filesystem.  Also, note that the upgrade to VMFS-5 is permanent, there is no way to revert an upgraded VMFS volume back to VMFS-3.

4.  Do upgrade VMware Tools to the latest version.  Unlike the virtual hardware version, the newer VMware tools 5.0 version is fully supported on older ESX/ESXi 4.x hosts.   As there are many improvements included with the latest version of VMWare tools it's always a good idea to upgrade as soon as possible.  Note however, VMware Tools 4.0 is also fully supported on ESXi 5.0 so it's not required to upgrade the VMware tools right away.  If you have 3.5 hosts in your cluster you should wait until all hosts are running at ESX/ESXi 4.x or higher before upgrading VMware tools to version 5.0.

Conclusion

So in conclusion, running mixed ESX/ESXi versions in an HA/DRS cluster is supported but be careful not to mix older VMs running virtual hardware 3 or VMware tools 3 in the same cluster with ESXi 5.0 hosts.  It is okay (if not recommended) to upgrade VMware Tools  while running in a mixed mode as long as all the hosts are running ESX/ESXi 4.x or higher, but avoid upgrading virtual hardware and VMFS volumes until after all hosts are running ESXi 5.0.

When using Auto Deploy you have two options for managing the IP addresses of your ESXi hosts:  (1) use static reservations in DHCP or (2) use an answer file.  I'll go over each of these options.
 
1.  I like to use static DHCP reservations as this eliminates the extra step of having to pre-populate an answer file for each host.  With a static IP reservations the DHCP server assigns an IP address based on the host's MAC address, this ensures that each time the host boots the DHCP server will always assign the same IP address.  Because the host always gets the same IP address from the DHCP server there is no need to reconfigure the hosts' network using an answer file when the host profile is applied.

Configuring static IP reservations in DHCP is very easy.  Simply provide the MAC address of the primary NIC being used for the management network and enter the IP address you want to be assigned.  The steps to configure static IP reservations on a Windows DHCP server can be found in the vSphere Installation and Setup Guide.

2.  Another approach for configuring IP addresses for your Auto Deployed ESXi hosts is to create an answer file.  Answer files are a new feature in host profiles introduced in vSphere 5.0.  Where host profiles are used to store configuration parameters that are common to many hosts, answer files store info that is unique for each host, such as IP addresses. 

The downside to using answer files is that you must first do an initial deployment of each host in order to manually populate each host's answer file.  An example of how to do this is available in the vSphere Installation and Setup Guide.  However, once the answer file has been created it will be used to automatically reconfigure the host's network during all future reboots without additional user intervention.

Note that it is possible to preconfigure the answer files using scripts.  This can eliminate the need to manually pre-configure each hosts but it does require some scripting skills.  Refer to this blog for more information on how to accomplish this.
 
I tend to prefer using static IP reservations in DHCP, but either approach will work.   I know many customers who prefer using answer files as it allows them to maintain their ESXi host IP addresses on their own without having to involve the DHCP administrator each time they deploy new hosts or want to make changes.