VI Team Blog

The following “top tips” highlight some issues that can arise in a VI deployment. They cover things which are sometimes hard to diagnose, or which might result in a problem weeks or months after some seemingly innocuous action. It is meant to shed some insight on “latent” issues, that is, those which don’t result in immediate warnings or errors when the root cause event occurs. These have been collected from customer experience gathered over time by the VI team, and will be posted in two parts.  We welcome your comments on these and any other “gotchas” that you might have encountered.

1. Make sure DNS is fully configured. This includes ensuring proper, consistent configuration for all of the following: short name, fully-qualified name, forward lookup, and reverse lookup. Otherwise, you'll see ESX hosts intermittently disconnect from VirtualCenter, and HA might not work properly.

2. Don’t use Virtual SMP for applications which don’t need it. Most applications are single-threaded and therefore cannot benefit from more than one virtual CPU. Assigning just a single CPU to VMs maximizes the physical CPU utilization of ALL of your cores, and avoids underutilized cores. If your applications were converted from running on 2 physical servers, don’t assume they need to – they might have been running on the smallest practical server configuration available. Start with a single VCPU, and then monitor the performance to see whether increase the number of virtual CPUs actually makes a difference.

3. Make sure you monitor the "% ready" metric. There's one new, key metric in managing virtualization environments that is doesn't exist in physical environments: ready time. Ready time measures, for a given VM, the amount of time that a VM is ready to run on the physical CPU but processor cycles are unavailable.  In a properly loaded system, ready time should remain near zero, although percentages less than five present no significant problem.  As ready time climbs to double digit percentages, the applications are lacking a significant portion of the CPU cycles they are requesting.  This usually happens as a result of an overly aggressive consolidation, and can be solved in various ways (reducing the number of VM's running, reducing the use of virtual SMP, adding memory or other resources in case swapping is occurring, etc.). For more information see this performance study: Ready Time Observations.
   

4. Watch your snapshot space growth.  Because snapshots live on your disk and grow over time, you want to be careful that you have enough spare capacity on your disk. Every snapshot consists of a “REDO” file; for the most recent snapshot, all new disk writes associated with the VM are recorded to this file. A REDO file has the potential in the extreme to grow to be the size of the original disk, and the REDO file of every snapshot that you maintain continues to occupy disk space. You want to make sure that you have enough "headroom" on your datastore to handle such growth over time.  Operations that might dramatically increase the size of your snapshots include the following: an OS service pack update, application reinstall, or a disk defrag inside the VM.

5. Make sure the SQL Server Agent is up and running on the VirtualCenter DB. VirtualCenter depends on Microsoft SQL Server Agent to perform stats rollups. However, VirtualCenter does not have the ability to ensure this service is running on the DB server. If the user has it disabled, or the service is shut down at some point, the VI Client will not show expected stats (weekly, monthly…).  In addition, since daily data is not rolled up, it accumulates in the database, thus degrading performance and consuming more and more space.

6. Team your management NICs if using VMware High Availability (VMware HA). This will help you avoid false alarms (i.e. false VMware HA failovers of VM's) in situations when you temporarily lose connectivity between your ESX hosts (e.g. when there's a momentary network outage, or even during a network switch maintenance operation).

Part 2 coming next week. [Update: see Top Tips for Deploying VI, Part 2]

--The VI Team

Which storage protocol to choose?

The most common storage related questions we are being asked today are:

• What is the best choice for running VI3 on shared storage?
• Should we use Fibre Channel (FC), iSCSI or NFS?

The answer to these questions will depend on a number of variables and as such the same answer will not be the same for each environment. VMware currently supports deployment of VI3 on all three of those storage protocol choices, as well as on local ESX server storage, and is focused on enabling customers to be successful at leveraging the benefits each of those choices available for the virtualization environment. Although differences exist in which VMware features and functions are available on them, the current approach is to remove as many of those differences as possible so that customers can have more choices available to them.

There are many industry perceptions that exist when comparing FC to Ethernet based storage (iSCSI and NFS) and these generally apply to both the physical and virtual deployments. Even though virtualization does not resolve the differences which exist between these protocols, VMware is focused on providing as level playing field as possible when it comes to using the different storage protocols.

What is supported?

Today some VMware features, functions and products are available on FC but are not an option when using NFS or iSCSI.

Summary of current feature support across protocols:

Type	Boot VM	Boot ESX Server	VMotion HA/DRS	RDM	MSCS Cluster	VCB	SRM	SVM	Lab Mgr	Stage Mgr	Life Cycle Mgr
Fibre Channel	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
iSCSI	Yes	Yes*	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
NAS	Yes	No	Yes	No	No	Yes	No	Not Yet	Yes	Yes	Yes
Local Storage	Yes	Yes	No	Yes	No	No	No	Not Yet	Yes	Yes	Yes

*  Boot from hardware iSCSI only (not supported from software iSCSI)

VMware is working to removing as many of those differences as possible so that the selection criterion for choosing a storage protocol is based on the differences that apply to both the physical and virtual world instead of what VMware features and products are available on one protocol but not another.

VMware will also address these differences in an order determined by the proportion of our customer deployments on each storage protocols. We have a great deal of work to do to complete this vision. But that is were we are headed.

Are there performance differences?

Performance is one of many considerations when choosing a storage protocol for the virtualization environment. However performance in a VI3 deployment is a multi­dimensional measurement that varies based on many factors. The number of ESX serves in the VMware cluster, number of VMs sharing a common pool of storage, block size, random vs. sequential ratio and read vs. write ratio. Today most benchmarks and comparisons tend to measure only one dimension and do not represent a typical multi ESX server infrastructure with multiple VMs running on them. The performance difference found in the physical world when comparing protocols is consistent with what is found in the world of virtualization. A VMware paper which provides some more details on storage protocol performance comparison is posted on our website. This paper shows that all network storage connection options available to ESX Server are all capable of reaching a level of performance limited only by the media and storage devices. When compared in terms of CPU costs, Fibre Channel and hardware iSCSI are the most CPU efficient because they offload the processing to the HBA, but in cases in which CPU consumption is not a concern, software iSCSI and NFS can also be part of a high performance solution.

Conclusions

One needs to consider not which protocol is the best choice for deployment of virtualization, but instead which virtualization solution provides the best support for multiple protocols? VMware not only supports the use of many storage protocol choices, but also provides a means to move VMs from a datastore on one protocol to a datastore on a different protocol while the VM remains up and running. Storage VMotion provides the means by which if one selects one protocol for a VI3 deployment, we enable switching to another protocol with much less disruption than any other virtualization solution offering available today.

While Storage VMotion is fully supported with FC today, we are headed in the direction of having it supported across all VI3 storage connectivity options (FC, iSCSI, NAS and local storage) so customers can seamlessly move from one type of storage to another.

So to close out on the question of which protocol is the best choice, the answer is all of them that VMware VI3 supports. And if you change your mind, or the choice first made was not the right one, VMware is providing the ability for one to move virtual machines to another storage protocol without huge hassle. In short, Vmware is all about being storage protocol agnostic.

If there is something you think I need to cover, let me know, please email me at: pmanning@vmware.com