Home > Blogs > VMware vSphere Blog > Monthly Archives: March 2011

Monthly Archives: March 2011

The missing link for scripted installs, adding your ESXi host to vCenter

It was bound to happen at some point and this morning William Lam published a script which enables you to add a host to your vCenter server during the scripted install. Now I have seen many cool scripts from William passing by over the last year or two but I feel that although this was probably not the most difficult one to write is is a brilliant piece of work. I tried this myself 18 months ago during a project and got stuck and decided the time needed did not weigh up against the costs associated. Thanks William for getting the job done.

Here's an exerpt from William's post, head over to his blog for the script!

How to automatically add ESX(i) host to vCenter in Kickstart

While recently updating my Automating Active Directory Domain Join in ESX(i) Kickstart article, it reminded me an old blog post by Justin Guidroz who initially identified a way to add an ESXi host to vCenter using python and the vSphere MOB. The approach was very neat but was not 100% automated as it required some user interaction with the vSphere MOB to identify certain API properties before one could potentially script it within a kickstart installation.

I decided to revisit this problem as it was something I had investigated awhile back. There are numerous ways on getting something like this to work in your environment, but it all boils down to your constraints, naming convention and provisioning process. If you have a well defined environment and utilizing a good naming structure and can easily identify which vCenter a given ESX(i) host should be managed from, then this can easily be integrated into your existing kickstart with minor tweaks. This script was tested on vCenter 4.1 Update 1 and ESXi 4.1 and 4.1 Update1.

Under the Covers with Storage vMotion

Hi,

I recently posted on how how vMotion works and figured it would be good to follow-up with a similar blog covering Storage vMotion (svMotion). 

Many people think svMotion is new, but the ability to migrate a running VMs disk files to a new datastore (DS) was first introduced in VI 3.0 as an upgrade tool to help with VMFS upgrades.  In VI 3.5 it was officially given the name Storage vMotion, but only had CLI support. GUI support was finally added in 4.0 and with 4.1 there were several performance improvements. 

Under the covers svMotion is a 6-step process.  Once a VM has been selected to have its disk files moved to a new DS using svMotion the following will take place:

1.    The VM home directory (config, log, swap, snapshots) is copied to the destination DS

2.    A “shadow” VM gets started on the destination DS using the copied files.  The “shadow” VM idles waiting for the copying of the VM disk file(s) to complete

3.    An initial copy of the VMs disk file(s) is made to the target DS, during the copy changes made to the source are tracked (change block tracking)

4.    svMotion iteratively repeats this process of copying the changed blocks from the source DS to the destination DS

5.    When the amount of outstanding changed blocks is small enough, svMotion invokes a Fast Suspend and Resume (FSR) of the VM (similar to vMotion) to transfer the running VM over to the idling shadow VM.  Like regular vMotion, this transfer will normally happen so quickly that it will be completely transparent to the VM.

6.    After the FSR completes the old home directory and VM disk files are deleted from the source DS.

With that let’s consider a few common questions:

Q.  Do I need to backup my VM prior to using svMotion?

A.  No, svMotion is as safe and reliable as it's vMotion counterpart.  Where vMotion transfers memory, svMotion transfer storage.  They employ the same FSR logic to transfer control of the running VM. 

Q:  Do we check for adequate free disk space on the destination DS before beginning Storage vMotion?

A:  Yes, checks are made before we initiate the svMotion to ensure that adequate disk space is available on the destination DS.  If there is not enough space, the svMotion request fails with no impact to the running VM. 

Q:  What happens to the VM if you run out of storage on the destination DS?

A:  If an out-of-space condition is hit svMotion will clean up the destination DS and the VM will continue to run on the source. 

Q:  What happens if the iterative coping of the changed blocks fails (i.e the VM is very write intensive)?

A:  If the VM is generating disk I/O faster than svMotion can copy, the svMotion will eventually fail leaving the VM running on the source.

Q:  Does all the svMotion traffic get copied over the vMotion network?

A.  No, a DataMover (DM) module built into the VMkernel, which is optimized for transferring large disk files, is used to copy the disk data. The swap files are also copied using the DM.

Q:  What is the threshold when the number of outstanding changed blocks is small enough to proceed with stunning the VM and switching to the destination DS?

A:  It depends on the disk copy throughput. svMotion continuously monitors the copy throughput during each copy iteration, when it detects that the time needed to copy the outstanding dirty blocks is less than the target downtime (default 5 secs), it proceeds with the switchover.  Note that the 5 seconds is only the time to copy the remaining disk blocks, it does not include the FSR time.

Regards,

-Kyle

Technology and Company behavior during a crisis – a great example!

Hello all,

This blog you are reading now is generally supposed to direct you to other VMware marketing materials.  As you can tell by what I do in this blog I provide answers to questions that customers ask me.  Sometimes I share things I think you should know about our products.  And today, there is something really different. 

I have experience in difficult times and places and as such a blog that I read yesterday really resonated with me.  When I help customers with protecting their business assets and technology with VMware's products, I try to remind them that people are their strength and that needs to be part of their plans.  The article below talks about how important technology was for communication, and how well a company responded for its employees.  Both are good lessons and I think also interesting reading.

Check out this blog for the story – a very nice story in a hard time – http://kevinrose.com/blogg/2011/3/14/apples-role-in-japan-during-the-tohoku-earthquake.html .

Michael

Need your help … What would you like to see …

Hello all,

We are currently just starting our work on the marketing 'collateral' we need for our upcoming releases.  The products I work with include VMware Data Recovery (vDR) and Site Recovery Manager (SRM).  So that means I need to produce a technical What's new deck for our partners and internal technical sales people for each of those two products.  That is pretty easy.  Where it gets interesting, is I also need to do an evaluators guide for each of them.  An evaluators guide is suppose to be a simple and easy introduction to the key features of the product it is about. 

You may want to check out the current ones – both from before my time here in Technical Marketing – vDR and SRM.

Why I am writing here today, is simple.  We have been told by outside customers who use these guides that they need to be better.  Some of the comments that are clearly great ideas include checklists so that you do not need to read as much, and can find what you need easier.  So I will include checklists in my guides. 

But I am hopeful that someone might read this blog and be willing to share what they think might bake our evaluator guides more useful.  I would really appreciate any comments that can help me do a better job. 

These two guides I need to write are my first.  This is your chance to get guides that you really need!

Thank you,

Michael

Ops changes part 7 – Upgrading Firmware

Upgrading firmware on any platform has always been a cumbersome task. When we asked a select group of customers what they expected to be most difficult when migrating to ESXi some answered Hardware Firmware Upgrades. The main reason for this was the fact that agents (some unsupported) were installed in the Service Console and they were used to upgrade the firmware. With ESXi that approach will no longer work due to the absence of a Service Console. Firmware however will still need to upgraded periodically.

When doing a little survey on twitter about how people did their firmware upgrades three methods stood out:

  1. Hardware vendor bootable CD-ROM/DVD
  2. Hardware vendor vCenter Plugin or management application
  3. PXE Boot of small linux distro / vendor's upgrade CD/DVD

The first option, bootable CD-ROM/DVD, is the most cumbersome solution but also fairly simple to use. Every vendor has a free ISOs available for download that contain all the latest and greatest firmware versions. These are usually categorized per server model and can be used to boot a host off. This is however a manual task and is most likely something one would use in SMB type of environments going up to 10 / 15 hosts.

Some hardware vendors also provide management plugin's like for instance Dell. Mike Laverick wrote an extensive article on this topic and it shows that there is a lot of value to be found in these. In the case of Dell the plugin provides more than just firmware upgrades and it is definitely worth investigating. Note that not only Dell offers a plugin for vCenter but many vendors do of which some have the capability of upgrading firmware and others don't but my guess is most will be releasing a version that will include this at some point:

Some vendors also have a centralized management application to manage their hardware end-to-end which also includes the capabilities to upgrade firmware. I personally find these solutions the most elegant as they offer a single pane of glass like for your hardware and it enables you to manage several layers. Cisco with its UCS platform was one of the first to deliver on this and of course there are the likes of IBM who have Systems Director which enables you to manage multiple platforms.

The last option that I wanted to discuss was PXE Booted Linux distro's. I have seen a couple of people mentioning this on twitter. Basically what they do is they use a tiny linux distribution (JeOS) and PXE boot their server with that distribution. This distribution contains the latest versions of the firmware or calls back to a central repository and automatically updates the server. Especially in very large environments a solution like this can be used as a uniform mechanism to deploy new versions of firmware across multiple hardware vendors as HP would not allow you to upgrade Dell servers with their tools. By developing your own tool this is of course possible.

As shown there are many ways to deal with firmware upgrades for your environment. It is difficult to come with a single approach that will work for all environments as the size of the environment and the type of hardware used will dictate which options are available. Based on the type of hardware used, the availability of plugins or a centralized solution and business requirements a decision should be made. We do advise to manage the firmware level consistently and follow the hardware vendor's recommendations to avoid running into any interdependency issues. We also recommend when acquiring new hardware to look into the level of integration and the mechanisms that can be leveraged around managing your hardware as especially in converged shared platforms availability and manageability is key to the success of your IT department.

vSphere – Now There’s an App for That!

Today VMware launched a new vSphere Client for the Apple iPad.  Its available now for download at the Apple App Store. We’re excited about expanding vSphere management to the iPad, enabling users to view key performance metrics and perform essential tasks in a simplified interface on the go.  No more running to the nearest laptop, firing it up, logging into VPN, drilling down into the vSphere Client every time you need to check on your virtual machines.  Just launch the iPad app wherever you have connectivity and a few taps of the finger later, you’re done!

With our initial release of the vSphere Client for iPad, you can:

  • Search for vSphere hosts and virtual machines in your vSphere environment 

Screenshot 01

  • Monitor the performance of vSphere hosts and virtual machines

  Screenshot 03

 

  • Manage virtual machines with the ability to start, stop and suspend
  • View and restore your virtual machines’ snapshots

  Screenshot 02

 

  • Reboot vSphere hosts or put them into maintenance mode
  • Diagnose vSphere hosts and virtual machines using built-in tools


Screenshot 04

The goal of the iPad client is not to replace or duplicate all the features of the existing vSphere Client, but rather to enable you to do the most essential tasks and see the most relevant performance data through a clean, consumable – portable – interface.  That being said, we’ll be adding more features in future releases.

Check out the videos below to see how to setup, configure and use the iPad client.

 

The vSphere Client for the Apple iPad is only available through the Apple App Store and requires the free vCMA appliance. To learn more visit the VMware community site.

After you have had a chance to try it, let us know what you think.

vCenter HA Survey

Customer feedback is critical for us at VMware to understand our customer's needs.  It helps to shape the direction of the products and features as well as help us identify any pain points.  To this end, we have a new survey that we invite you to take concerning the availability of vCenter.  I'd invite you to take the short survey and let us know your thoughts!

You can access the survey here.

Ops changes part 6 – Quick troubleshooting tips

When ever an issue arises on an ESX host people often rush to the Service Console and start typing various commands to figure out what is wrong. Of course some of the commands are not available with ESXi and you might not even have access to the ESXi console. Many will resort to the vCLI/vMA or even PowerCLI and that works perfectly fine. Especially the vCLI/vMA is geared towards those who have experience with ESX command-line troubleshooting. You will have all "esxcfg-*" commands to your disposal and of course resxtop; which will cover 95% of those cases where commandline details are required.

I want to stress that ESXi was not built for console access, although we do provide access to the console and it works fine. The idea around ESXi is to have a lean hypervisor which is managed from the outside versus the inside and VMware has provided multiple tools to do so. The first and foremost being of course vCenter Server or the vSphere Client. Many problems can be solved simply by using the vSphere Client connected to a host directly or through vCenter Server itself. The first KB article in the list below is a good example of how vCenter can be used to troubleshoot an inaccessible virtual machine. Something that people tend to forget is that the vSphere Client can also be used to read log files, there is no need to open up a console session for that shown below and explained in the second article in the list:

  1. Open a browser and enter the URL http://<vCenter hostname>, where <vCenter hostname> is the IP or fully qualified domain name for the vCenter Server.
  2. Provide administrative credentials when prompted.
  3. Click the Browse datastores in the vCenter inventory link.
  4. Navigate the webpages until you reach the appropriate datacenter, datastore, and folder as noted in step 1.
  5. Click the link to the appropriate log file, and open it with your preferred editor.

On the topic of log files, for those who never worked with ESXi the location is slightly different than you are used to:

  • The VMkernel, vmkwarning, and hostd logs are located at /var/log/messages
  • The Host Management service (hostd = Host daemon) log is located at /var/log/vmware/hostd.log\
  • The vCenter Agent log is located at /var/log/vmware/vpx/vpxa.log
  • The System boot log is located at /var/log/sysboot.log
  • The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_<hostname>-xxx.log

Note that the /var/log/messages is a combination of all logs out there except for the HA log. You will need to monitor open that up seperately when troubleshooting HA related issues. Also be noted that the HA logfiles aren't part of the Syslog mechanism either unfortunately. Knowing the logfiles and the type of info you can get from it is key when troubleshooting. I encourage everyone to get familiar with it when you have the time to do so, as under pressure you don't want to find yourself fiddling around in the wrong location or logfile when you have 4 managers and your director watching over your shoulder if you have fixed it already or not.

After you have dived into the log files make sure you check the Knowledge Base. Our Knowledge Base has an excellent set of articles which can be used to troubleshoot very specific issues or at least lead you into the right direction. I have listed some of the most common issues and used KB's including a link to the article below for your convenience:

  1. Restart the management agents on an ESXi host (1003490)
  2. Determining why a single virtual machine is inaccessible (1018834).
  3. Determining why a virtual machine was powered off or restarted (1019064).
  4. Determining why multiple virtual machines are inaccessible (1019000).
  5. Troubleshooting virtual machine network connection issues (1003893).
  6. Interpreting virtual machine monitor and executable failures (1019471).
  7. Determining why a virtual machine does not respond to user interaction at the console (1017926).
  8. Using Tech Support Mode in ESXi 4.1 (1017910)
  9. Determining why a VMware ESXi host is inaccessible (1019082)
  10. Determining why a VMware ESXi host was powered off or restarted (1019238).
  11. Determining why a VMware ESXi host does not respond to user interaction (1017135).
  12. Enabling serial-line logging for an ESXi host (1003900).
  13. Using performance collection tools to gather data for fault analysis (1006797).
  14. Using hardware NMI facilities to troubleshoot unresponsive hosts (1014767)
  15. Interpreting a VMware ESX host purple diagnostic screen (1004250).
  16. Troubleshooting VMware High Availability (HA) (1001596).

In some cases however it might be required or desirable to log in to Technical Support Mode (yes this is fully supported) and work directly from the ESXi shell. The ESXi shell as many of you know also contains all esxcfg-* commands, the invaluable esxcli command and of course some shell commands that are required for troubleshooting. Some  of those commands are obvious, others are less obvious. I have listed several below to make things easier.

The one many complained about in the past but actually is available is vmkping. Vmkping can be used to do basic network troubleshooting, but also for instance to validate if jumbo frames can be used by simply adding the size of the packet:

vmkping -s 9000 <ipaddress>

One that many bumped into in the ESXi 4.0 time frame was the lack of a mount command. This mount command was actually available, but as part of busybox:

/usr/bin/busybox mount

In 4.1 though the "mount" command has been linked to busybox itself enabling you to just use "mount. The same applies to for instance fdisk. Fdisk will enable you to validate the partition setup. It has helped me many times in the past to validate that partitions were still marked as "VMFS" when someone accidentally presented VMFS volumes to Windows machines which immediately resignatured the disks. Again under 4.0 fdisk is not available as a binary but is available through "busybox", and in 4.1 is available as a link. (Most of these links are located in /usr/sbin)

/usr/bin/busybox fdisk -l

Another thing that I have done in the past regularly when I needed to evacuate a host is place the host in maintenance mode. With ESXi you can do this as follows:

vim-cmd hostsvc/maintenance_mode_enter

And of course you can also exit maintenance mode:

vim-cmd hostsvc/maintenance_mode_exit

What about listing all VMs and stopping a specific one?

vim-cmd vmsvc/getallvms

vim-cmd vmsvc/poweroff <vm id>

These are just examples to show the power of vim-cmd. Many try to avoid using it, but really it is not overly complex and it gets the job done fairly simple. It can be difficult sometimes to figure out the syntax but than again if you can't find figure it someone else probably has, google it.

Something that I was asked about this week which can also come in handy when troubleshooting memory issues is the following command which will give you the memory utilization of the hypervisor components:

vdf -ph

These commands are just a couple of examples of what is possible within the ESXi shell, although we do generally recommend to avoid logging in to the ESXi shell (via remote or local tech support mode) and prefer to use the alternatives we offer it will work fine. In general troubleshooting hasn't changed much due to the full support of the "Technical Support Mode" feature, the remote command line utilities (vCLI or the vMA) and of course vCenter or the vSphere Client.

Ops changes part 5 – Scratch partition

When implementing ESXi the first noticeable difference in the architecture is the partition layout.With ESX classic you always had the ability to customize your partition layout and many used the obvious "Linux" partition scheme with a /tmp, /opt, /var and /home. This however as changed with ESXi and there isn't actually much left to discuss… well except for the Scratch Partition that is.

The question of course is what is it used for and when is it created and can I modify it?

The Scratch Partition is used for a couple of things:

  1. Store vm-support output (if scratch partition is unavailable the output will be stored in memory)
  2. During first boot the partition will be configured through syslog for retaining logiles
  3. Userworld swapfile (when enabled)

In the past, as some might remember, you needed to create this partition yourself. As of vSphere 4 Update 1 this is done automatically for you when you install ESXi Installable and the installer detects a disk a partition will be created. You might wonder why the change in behavior, well the reason for it being of course to have an area where detailed troubleshooting information can be stored. I guess our documentation explains it fairly well:

The scratch partition is not required. It is used to store vm-support output, which you need when you create a support bundle. If the scratch partition is not present, vm-support output is stored in a ramdisk. This might be problematic in low-memory situations, but is not critical. For ESXi Installable, the partition is created during installation and is thus selected. VMware recommends that you leave it unchanged.

As stated earlier, not only vm-support output but also logfiles and userworld swap (if enabed). All of this means that although not specifically required as mentioned in our documentation, from an operational perspective there is very much a requirement to have a scratch partition in place. So if in any case a scratch partition is not automatically created, for instance when using USB/SD as boot device, make sure you define your scratch partition, either on local or on remote storage. Of course vSphere offers you multiple ways of doing this:

Tech Support Mode / Scripted Install:

vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/storage1/<unique-name>

vCLI:

vicfg-advcfg.pl –server <hostname> –username root -s /vmfs/volumes/storage1/<unique-name> ScratchConfig.ConfiguredScratchLocation

PowerCLI:

Set-VMHostAdvancedConfiguration -VMHost (Get-VMHost <fqdn-hostname>) -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/storage1/<unique-name>"

Thanks to Alan Renouf for providing the line of PowerCLI code. By the way, Alan also provided a line of code to list all the host and the current Scratch Location, might come in handy some day:

New-VIProperty -ObjectType VMHost -Name ConfiguredScratchLocation ` -Value {($Args[0] | Get-VMHostAdvancedConfiguration ScratchConfig.ConfiguredScratchLocation).Values } ` -Force

Get-VMHost | Select Name, ConfiguredScratchLocation

vCenter:

  • ESXi host
  • Configuration
  • Software
  • Advanced Settings
  • ScratchConfig
  • ScratchConfig.ConfiguredScratchLocation

Please note that the location needs to be unique for every ESXi host and again, if it has been set by the installer I recommend to leave it to the default settings. Also don't forget to reboot your host after making changes to the Scratch Partition setting.

 

New Book: Professional vSphere 4: Implementation and Management

Cover

I'm always on the lookout for new technical books with a VMware focus.  It looks like Cody Bunch and Patrick Ancillotti are working on a new one called "Professional vSphere 4: Implementation and Management".  It is expected to be released later this year (May 18th acording to Amazon.com).  From the standard description:

"VMware vSphere, is VMware's first cloud operating system, able to manage large pools of virtualized computing infrastructure, including software and hardware. Building on the power of VMware? Infrastructure, VMware vSphere dramatically reduces capital and operating costs, and increases control over IT infrastructures while preserving the flexibility to choose any OS, application and hardware. This book provides vSphere administrators with essential guidance in implementing and managing VMware's vSphere product suite. This guidance has been battle tested in virtual environments big and small. It also provides actionable advice for the day to day operational maintenance and common pitfalls to avoid for first time vSphere administrators. Best of all, the book includes an introductory version of Train Signal's vSphere training DVD that provides additional hours of vSphere training."