(Note original post was updated on 28 Oct 2011 to better clarify ESX/ESXi 3.5 support in a mixed cluster.)
Running different versions of ESX/ESXi in an vCenter 5.0 HA/DRS cluster is supported. Frank Denneman recently posted a good blog on this. You may ask why would anyone want to run a mixed cluster? Usually, this is done to facilitate rolling upgrades. If you have a large 32-node cluster it's not practical to upgrade all 32 hosts at once, so instead you can leverage the mixed cluster support to upgrade two or three hosts at a time and "roll" the upgrade through the cluster until all 32 hosts are upgraded.
While mixed clusters are supported there are some things to watch for, specifically the VMware tools and virtual hardware versions of your VMs. The table below provides a summary of the VMware tools and virtual hardware versions that are supported on both vSphere 4.x and 5.0.
From the table we can see that:
1. VMs with virtual hardware and VMware tools version 3 are not supported on ESXi 5.0 hosts.
2. VMs with virtual hardware version 8 are not supported on pre-5.0 hosts.
3. VMFS-5 is not supported on pre-5.0 hosts.
With these limitations in mind, my recommendations for running mixed clusters are as follows:
1. Verify VM virtual hardware and VMware tools versions before mixing 3.5 and 5.0 hosts in the same cluster. VMs with VMware tools version 3 and virtual hardware version 3 are not supported on ESXi 5.0. To avoid potential pitfalls, be sure to upgrade VM hardware versions to version 4 and VMware tools version to 3.5 before mixing 3.5 and 5.0 hosts in the same cluster.
2. Do not upgrade virtual hardware versions while running in a mixed mode. Once you upgrade a VMs virtual hardware version to 8, it can no longer run on a pre-5.0 ESX/ESXi hosts. In addition, there is no option to undo the upgrade or revert back to an earlier virtual hardware version. As such, while running a mixed cluster you should avoid upgrading the virtual hardware version of your VMs to version 8 until after all hosts have been upgraded to ESXi 5.0.
3. Do not upgrade VMFS-3 volumes to VMFS-5 while running in mixed mode. Wait until after all the hosts in the cluster are running ESXi 5.0 to upgrade VMFS volumes. Upgrading to VMFS-5 will prevent any pre-5.0 hosts from accessing the filesystem. Also, note that the upgrade to VMFS-5 is permanent, there is no way to revert an upgraded VMFS volume back to VMFS-3.
4. Do upgrade VMware Tools to the latest version. Unlike the virtual hardware version, the newer VMware tools 5.0 version is fully supported on older ESX/ESXi 4.x hosts. As there are many improvements included with the latest version of VMWare tools it's always a good idea to upgrade as soon as possible. Note however, VMware Tools 4.0 is also fully supported on ESXi 5.0 so it's not required to upgrade the VMware tools right away. If you have 3.5 hosts in your cluster you should wait until all hosts are running at ESX/ESXi 4.x or higher before upgrading VMware tools to version 5.0.
Conclusion
So in conclusion, running mixed ESX/ESXi versions in an HA/DRS cluster is supported but be careful not to mix older VMs running virtual hardware 3 or VMware tools 3 in the same cluster with ESXi 5.0 hosts. It is okay (if not recommended) to upgrade VMware Tools while running in a mixed mode as long as all the hosts are running ESX/ESXi 4.x or higher, but avoid upgrading virtual hardware and VMFS volumes until after all hosts are running ESXi 5.0.
