Another guest post today from Simon Todd, who is a Tech Support Engineer in our Cork, Ireland office who specializes in Storage issues.

Anyone using snapshots has probably run into this issue once, twice or many times:

When trying to perform an online consolidation there is not enough space on the Datastore to perform the consolidation.

This is mainly due to the way snapshots are consolidated. To illustrate this, let’s look at an example where we have a virtual machine residing on a 600Gb VMFS Datastore that has a 500Gb base disk, and that virtual machine has 8 levels of snapshots of the following sizes:

As you can see we are consuming an additional 64Gb on top of the original base disk leaving us at risk to run out of space on the Datastore. The logical thing to do is consolidate these snapshots. So what happens if we consolidate these?  Well, prior to vSphere 4.0 Update 2 the snapshot consolidate process would take the latest snapshot, merge that into the previous snapshot, then merge that combined snapshot and merge it into the previous one, and so on and so forth until complete.

If each of the snapshots contains unique changes that do not exists in previous snapshots then the following diagram would illustrate the worst case scenario for Datastore usage:

As you can see there’s a potential for the snapshot consolidate process to consume 153Gb more than your Datastore allows. This process has been the same all the way back from when the snapshot process was first introduced.

vSphere 4.0 Update 2 changes

As of vSphere 4.0 Update 2 the snapshot consolidation operation has changed, instead of rolling the snapshots into each other, the snapshot process now takes the oldest snapshot and consolidates that into the base disk, removes the snapshot and then processes the next oldest snapshot and merges that into the base disk, so after all the snapshots are consolidated into the base disk they are removed. This not only saves on required disk space, but it also performs better:

Today we have a guest post from Ganesh Shetty, who is a Tech Support Engineer in our Bangalore office who specializes in Licensing issues. He’s currently focused on writing Knowledgebase articles. Today, he brings us some good advice advice in minimizing licensing headaches.

If you previously used Virtual Infrastructure 3, you may remember using license files to license ESX hosts. Although vSphere 4 does not use license files, this does not apply if you are managing ESX 3.x hosts using virtual center 4.x, as most of you may know. We still have a lot of customers using a mixed environment, comprising of ESX 3.x, ESX 4.x hosts managed by Virtual Center 4.x. This introduces some challenges in licensing the Environment.

There are several errors associated with an improperly licensed ESX 3.x host managed by a Virtual Center 4.x. In this post, I will be discussing a few steps to keep in mind when licensing ESX 3.x hosts being managed by a Virtual Center 4.x and I’ll also give you a few tips to fix errors related to this.

License files in place

The first and foremost step is to ensure that your license files are in place. Quite often the source of a lot of licensing issues is a corrupt license file. When in doubt about a license file, it is always better to delete and regenerate that license file. You can refer the knowledge base article Creating a Single Host and Centralized License file on the License Portal (1005132) for a detailed explanation on how to generate server based license files.

If regenerating the license file is not an option, you can check if the license file is valid or not by pasting its content in the server based license checker . Open the URL: http://www.vmware.com/checklicense/ and paste the contents of your license file, in the empty box. Click on validate.

This will tell you if the license file is valid or not. You need not be including a virtual center 2.x license file since you are using virtual center 4.x, so you may get an error message ‘No VC Management License found’ when validating the license file. This error can be ignored. The screenshot below indicates a valid instance of an Enterprise license file for 2 processors.

Notes:

• If you generate a VI3 Enterprise license, you will not see the option ‘Enterprise’ in the server based license checker. Instead you see 2 Standard + 2 vMotion +2 DRS + 2 HA + 2 VC Agent + 2 VCB licenses.
• The license checker can only be used to validate ‘Server based’ license files and not ‘Host based’ license files.

It is always recommended to not modify the contents of a license file. Manual consolidation of many license files into one file is possible, however, if done incorrectly, can corrupt the license file. As seen in the screenshots below, this license file is corrupt and will certainly give you an error when licensing the host. In such a scenario you must regenerate the license file.

You can also refer the knowledge base article Troubleshooting an error when uploading a license file (1005440) for additional causes of an invalid license file.

The license file is coded in UNIX and contains a number of stanzas which starts with ‘INCREMENT’. The header which follows the word ‘INCREMENT’ indicates the license type (e.g. Foundation, Standard, Enterprise etc.) and the quantity. It is helpful to understand these headers and what they denote. This will come in handy when performing a ‘Status Enquiry’ in the license server. I’ll show you how it is done later in this post.

The KB article Understanding VI3.5 Licensing: Server- and Host-Based Licensing Models (1003295) provides a detailed explanation on what these headers denote in a license file. I have listed a few important ones to keep in mind

· INCREMENT VC_ESXHOST VMWARELM 2005.05 permanent 2 :

2 processors of Virtual Center Agent

· INCREMENT VC_VMOTION VMWARELM 2005.05 permanent 2 :

2 processors of vMotion

· INCREMENT VC_DAS VMWARELM 2005.05 permanent 2 :

2 processors of High Availability

· INCREMENT VC_DRS VMWARELM 2005.05 permanent 2 :

2 processors of DRS

· INCREMENT PROD_ESX_FULL VMWARELM 2005.05 permanent 2

2 processors of Standard

· INCREMENT ESX_FULL_BACKUP VMWARELM 2005.05 permanent 2 :

2 processors of VCB

· INCREMENT PROD_ESX_STARTER VMWARELM 2005.05 permanent 2 :

2 processors of Foundation

After you have ensured the license file is valid, save the license file with a .lic extension.

Installing the License Server

If you have already installed the license server and need to upload the license file, simply follow steps 1 to 9 in the KB article Installing Licenses for ESX Server 3.x (1001383)

After you have generated (and saved a copy) of your license file, ensure you have downloaded the license server. The standalone version of the License server is available for download in the VI3 download section. When installing the license server for the first time, you are asked to browse to the source of the license file. Point to the location of the license file and complete the installation.

There have been reported issues of the standalone version of the license server not installing in Windows Server 2008. If you run into this, follow the KB article Unable to install the license server on Windows Server 2008 (1018118) to obtain a workaround.

After uploading the license file to the license server, it is recommended you restart the license server by clicking on Stop Server, Start Server and Reread License File, in the order shown here:

There are some occasions where you get the error message ‘cannot stop the server’ after clicking on Stop Server and subsequently the error message ‘Cannot ReRead license file’ when you click on ReRead License File. In such situations, reinstallation of the license server will fix the issue.

vCenter Server Settings

After the license file has bee
n reread successfully, Login to the virtual center using vSphere Client. Click on Home, vCenter Server Settings and ensure that you have entered the IP address or the name of the machine where the license server is installed. It is also necessary that you check the option Reconfigure ESX 3 hosts using license servers to use this server.

Note: Please ensure that the Virtual Center 4.x is licensed with a valid license key.

To license the ESX 3.x host follow steps 14 and 15 in the KB article Installing Licenses for ESX Server 3.x (1001383) . Finally, ensure that the ESX host is pointing to the correct IP address/name.

If you still get an error not enough licenses, check if the licenses available is greater than or equal to the processor socket count of your ESX 3.x host. This can be done by clicking on Configuration, Processors (Under Hardware) and look at the count next to Processor Sockets

In odd instances I have noticed that there will be sufficient licenses but the ESX 3.x host has not picked up the licenses from the license server. In such scenarios you can check if the issued license files have been assigned or not. This can be done by clicking on Server Status on the license server and then clicking on Perform Status Enquiry

As previously mentioned above, headers in the license file denote the license type and also its quantity. In the above screenshot, it indicates that the licenses and its various features have been issued but not assigned to the ESX 3.x host. This is a good indicator as to why you get an error ‘Not Enough Licenses’ despite having sufficient licenses. Sometimes, only one feature may show up as ‘not assigned’ while the other features may have successfully been assigned to the ESX host. For example VC_ESXHOST (VC Agent) may show up as ‘not assigned’. This means that the ESX host is not able to pick the VC Agent licenses and hence will not allow you to add the ESX host to the Virtual Center (since VC Agent licenses are required to add an ESX host to the Virtual Center) by throwing up an error ‘Not Enough Licenses’.

To fix this, restart the license server from Services.msc. The licenses will start reassigning themselves from scratch and whatever feature was not assigned will be successfully assigned by the license server.

Many KB articles have been referenced here, an they’re all well worth checking out when you have licensing problems.

Today we have a guest post from Duncan Armstrong, who is a Tech Support Engineer in our Canadian office who specializes in Fault and Storage issues. He’s currently focused on writing Knowledgebase articles as we want to capture as much real-world troubleshooting know-how as possible and share it with everyone. Today, he wants to give us some advice.

Since joining VMware Technical Support several years ago, I have frequently found myself pondering the decision making processes behind some of the various storage configurations that have been employed in VMware Infrastructure/vSphere environments. One particular configuration came up more often than I liked, and while unfortunate, it was hardly surprising to see these deployments/configurations result in unplanned outages or even impact recovery efforts. The version of ESX didn’t matter, nor did the type of storage used (NAS/SAN, iSCSI or FC). This issue did not discriminate any particular technology or branding. It seemed to punish those who preferred to organize their datastores in such a fashion.

I’ll get to the point: Dedicated Guest Operating System datastores are not a good idea, no matter how organized they may appear to be. The same sentiment can sometimes be shared with dedicated Data Disk datastores.

What? Why are these used?

I have the impression these “split” configurations—dedicated Guest OS datastores and Data Disk datastores—were created more for virtual machine file organization, than for performance or even space optimization. Customers with these setups do know very well where their guests and configuration files are (Guest OS Datastore), and where all their data drives are (their remaining Data Datastores). However deployments or consultations that push for this kind of setup are really quite wrong or misinformed.

There are a several ways these configurations can result in problems.

Event and time-based load spikes

After an outage or problem, the upswing of bringing your guests back online is going to take a longer amount of time; all of the reads and writes required for the mass startups are isolated to the few Guest OS Datastores that reside on your storage. Expect load to be maximized until all the guests have completed their startup sequences.

Once these operating systems are up, your guest system drive loads will start to drop, and likely begin to rise from the production data disks. The latter is really quite fine, but the former may translate into extended startup times following an outage.

System disks aside, having too many production-load guest Data disks on a single datastore would likely result in similar performance problems during production hours and especially backup schedules. Or you may see a consistent performance decline as a result of overloading a given disk set throughout your uptime.

I also see some customers dedicate an entire datastore for single virtual machine disks, configuring for a one Datastore to one Guest Data Disk ratio setup. This is also suboptimal and wastes space; delta files are not stored on these locations. The space left on each Data Disk Datastore is never utilized. That may not necessarily be a problem if you planned for it, but you’re also unnecessarily consuming more LUN ID numbers (if you are using VMFS or SAN storage). ESX supports up to 256; check VMware documentation, or specifically the Maximums and Limitations document for more information: Configuration Maximums for VMware vSphere 4.1.

Note: Thin-provisioned disks in vSphere are an exception; they start small and grow over time.

Over-commitment of ESX Server memory results in the utilization of existing Virtual Machine Swap (.vswp) files, which by default reside in the same directory as your virtual machines’ configuration files/directories. Having all your virtual machines across multiple hosts require heavy read/write activity on a common LUN won’t likely be a good experience.

Unexpected space consumption

Like the last point above, you may or may not already expect Virtual Machine Swap (.vswp) files to be created, consuming additional Guest OS Datastore space after virtual machine power-on (1:1 space required to virtual machine memory allocation. An 8GB RAM virtual machine uses an 8GB .vswp file unless if you set up memory reservations). Powering on 20+ Guest OSes with 4GB of memory each on the same datastore results in 80GB of virtual machine swap files being created.

The real killer in these setups is when snapshotting a virtual machine. Each virtual machine disk’s delta file (redo log) is created in the same directory as the virtual machine’s configuration file/directory, much like the .vswp file, above. A titular "OS Datastore" in these storage configurations, by role, effectively contains all of the delta files at once. This quickly becomes a problem, where such a datastore is usually fairly small to begin with (to save room on the SAN/NAS for "Data," of course).

With the actual data virtual disks sitting quite large, and commonly showing demanding performance and space consumption characteristics (the latter is due to increased additional delta/redo logging), they often cause the complete exhaustion of storage space on the "Guest OS Datastore," where the deltas are created and grow, to accommodate for disk changes. Of particular note are delta disks for databases or other "volatile" virtual machines where a lot of delta in a short time frame is going to be created. Datastore space exhaustion results in an outage for every virtual machine that continues to require additional space on the affected datastore. They cannot resume until there is additional space.

Space exhaustion while on snapshots is a common call-generator when this setup is used. Addressing these issues is somewhat trivial for VMware, but is largely demanding on expertise, familiarity with how snapshots function, and really, just thinking on your feet.

Conclusion

Please avoid this storage configuration. If you already have one like this, think about how you can start spreading out your operating system disks, even just a bit more. Mixing up the OS and Data disks isn’t inherently a Bad Thing, generally speaking, but there are always exceptions to the norm out there. So understand what you’re storing, their usage characteristics, and spread out your loads (even across a given day).

Further, by having some room to spare on each of your datastores (nothing is too tight and nothing is over-utilized now, right?), you have some breathing room later if, for any reason, you run into problems. Something I like to be able to anticipate is a mass exodus of all virtual machines from a datastore to my remaining ones. Do I have capacity for this? Or how about: Can I clone the largest virtual machine I have and put it on one or more destination datastores, in the event of an emergency?

I hope that helps a bit, or at least gives you some ideas. Or even more ideally, comfort you in the decisions you have already made to avoid this kind of setup. Don’t get caught setting something like this up. Yes, it will work fine if you avoid using features like snapshots (and accept the performance quirks/behaviors), but it could be done better, and with less effort.

This nice nugget landed in my email basket today – Eric Sloof, vExpert, Certified VMware Trainer, and evangelist extraordinaire, recently presented at the Dutch VMUG event his talk: VMware vSphere Advanced Troubleshooting.

Eric always attracts a crowd and this time was no different. Approximately 600 attendees wanted to hear how to best troubleshoot vSphere issues in production environments. Eric’s knowledge goes quite deep as you’ll soon find out by downloading his slide deck for the presentation. If performance problems plague your datacenter, Eric covers the various metrics you need starting with esxtop and then diving deeper from there. You’ll also gain a grip on things like memory limits and reservation, storage bottlenecks and finding the reason for dropped network packets. Fifty gorgeous slides of awesomeness!

If you missed his presentation, and you understand Dutch, you can watch it here:

Read: VMware vSphere Advanced Troubleshooting by Eric Sloof