Home > Blogs > VMware Support Insider > Monthly Archives: July 2010

Monthly Archives: July 2010

Scheduled Maintenance July 31

VMware will be performing a system upgrade to several VMware Web applications on July 31, 2010. Maintenance will begin at 12:01 PM Pacific Time and end July 31, 2010 at approximately 03:00 PM Pacific Time.

While this upgrade is in progress, you will be unable to:

  • Access or manage your VMware account
  • Submit support requests online 
  • Download, purchase or register VMware products Manage VMware product licenses.
  • Access to VMware Communities

If you need to file a support request while the upgrade is in progress, call VMware Technical Support for assistance.

We appreciate your patience during this maintenance period. These system upgrades are part of our commitment to continued service improvements and will help VMware better serve your needs.

VMware View Support with vCenter/vSphere 4.1

With the recent release of VMware vSphere 4.1 and VMware vCenter 4.1 we wanted to clarify support for these platforms with the current shipping versions of VMware View.

As hardware vendors continue to improve processor hardware, VMware is taking steps to improve the performance and utilization of this hardware. As such, we have moved to a 64-bit architecture for vSphere 4.1. View 4.5 will be the first version of View to support this new architecture, so customers should plan their upgrade paths accordingly.

Please note that View Composer is the only component of View which will not run on vSphere’s 64-bit architecture. Customers who are not using Composer should be fine to upgrade. Support for vSphere 4.1 and vCenter 4.1 are planned with upcoming version of VMware View including View 4.5.

Additional information about View compatibility with vSphere 4.1 and vCenter 4.1 can be found in the Knowledge Base Article: vSphere support for View Manager 1011292

Customers can also refer to the vSphere Compatibility Matrix: http://www.vmware.com/resources/compatibility/docs/vSphere_Comp_Matrix.pdf

View compatibility information can also be found in the View Manager Administrative Guide: http://www.vmware.com/support/pubs/view_pubs.html

VMworld is approaching quickly

If you’ve been procrastinating in getting yourself registered for VMworld this summer in San Francisco here is your friendly reminder. This year’s VMworld 2010 is being held in San Francisco August 30 through September 2. At the conference you can participate in hundreds of Breakout Sessions and cutting-edge Hands-on-Labs that don’t require pre-registration.

The ever popular Genius Bar returns this year where you can sit and have a face to face chat with various experts. From our own team, Rick Blythe and Kevin Mitts will be demonstrating our new Social Support initiatives including KBTV, the Knowledge Base, and more. Register now.

Register Now

How to upgrade to VMware vCenter Server 4.1 using the Data Migration tool

Our first video covering vSphere 4.1 is now live. The video compliments KB article 1022137 vSphere 4.1 upgrade pre-installation requirements and considerations. and describes the process for using our Data Migration tool to upgrade VMware vCenter Server 4.1

VMware vCenter 4.1 is part of the VMware vSphere 4.1 product suite and the Data Migration tool allows you to migrate your vCenter Server 4.0 configuration, since VMware has now entirely gone to a 64-bit platform. 64-bit brings significant performance benefits, but it also introduces some challenges that need to be considered before upgrading.

Sit back, grab that cup of java and take in a little KBTV.

How to upgrade to VMware vCenter Server 4.1 using the Data Migration tool

 

The Data Migration tool is provided with your vCenter Server 4.1 installation media.

Questions, Questions, Questions

Mike Bean

Hello Virtual World!

I’m writing this week’s column a little early, so my fan (Hi Mom!) will have something to read next week, as I am taking a short vacation. It’s a personal tradition of mine that I never work my birthday, haven’t for many years. (I’m stubborn like that!) So the ESX systems of the world will have to keep running without my meddling for a time!

It’s one of the peculiar ironies of my life that I have a tendency to learn things too late to really make use of them. One could very easily argue that I don’t have a monopoly on that. It’s a natural by-product of aging and wisdom, and happens to everyone. More specifically, in my last job I ran a SAN reliability lab. It was my job to run 30 some odd servers engaged in more or less random read/write operations to approx 2400 hard drives. When they broke, I was expected to call the attention of the local engineering staff to the problem. Over the course of this job, I learned something. To all my friends who are engineers, I apologize, but I’m about to characterize your entire profession under a stereotype. Engineers don’t like, “it’s broke.”

Engineers like data, they like hard facts. They like information so reliable you can practically scratch a window with it. They like recorded, quantifiable statistics they can browse, copy, tweak, manipulate, reproduce and experiment with. More often than not, they like this kind of information, because it’s easy to fix, and even if they can’t fix it, it gives them something to build on. The really insidious part of all of this is the attitude is infectious. After I started this job at VMware, I realized I’d caught the virus.

It’s super common that we’ll get requests virtually every day from folks who’ve observed some sort of anomalous behavior from their servers which they must account for, and they expect our help in doing that. We do our best to provide that help when it’s asked for, and we frequently ask for diagnostic data/vm-support bundles from the affected host. In fact, I would go so far as to suggest that it’s not a bad idea to automatically include these logs with EVERY support request you send to VMware. (VMware KB 1008524) However, there seems to be a lot of common misconceptions about what it is we can actually learn from those logs, here are a few of them. (Listed in no particular order)

#1 My VM’s are behaving strangely! Check the logs please!

This is a common question, and it’s worth exploring, but it’s important to try to not put all your eggs in one basket, because ESX does not do a lot of logging at the operating system level! By and large, we concentrate on setting up the sandbox, and what the kids do within it, is their business. That’s not to say we can’t learn things about events on a VM by examining its corresponding vmware.log, but it’s important to have realistic expectations. If you find yourself in the position of having to call VMware with this question, be prepared for us to ask you some questions right back.

How precisely is it behaving strangely? How long has it been behaving strangely? What’s changed? (There is ALWAYS SOMETHING, even if we don’t yet know what). What are the characteristics of the VM? What operating system does it use? How often does it show this kind of behavior? How many VM’s are showing this behavior? (What do they have in common?)

(Wow, if any of the engineering staff from my previous job are reading this column, they’re probably laughing themselves silly right now.)

#2 There are timestamp gaps in my logs! ESX must have crashed!

This is another common one. It’s true that ESX is a prodigious logger, and under normal operating circumstances, there will never be a time when it doesn’t have something to say, but if there are gaps, you can’t necessarily assume the host was down. All that a gap proves is that the system was in a state that ESX couldn’t write to its own system files. There are multiple ways that might happen; hardware failure is the most common. I spoke with someone just this morning with substantial gaps in his logs; he’d discovered bad dimms on the host.

Gaps in the system log imply that something’s wrong with the hardware, but they don’t always PROVE it. The point is, when ESX CRASHES, you’ll know it. That’s how it was designed. You’ll get one of our famous “purple screens” (VMware KB 1006802 or KB 1009525 = examples). A purple screen is ESX’s way of saying that it’s run out of options. There are no remaining courses of action that don’t pose substantial risk of data corruption, and for obvious reasons, ESX doesn’t “roll” like that. When the system hangs, it’s reasonable to conclude that ESX can’t write to its system files. If it can’t write to system files, then it’s reasonable to assume we can’t write a crash dump either. So if you find yourself in this state, BEFORE you reboot the host (which we do understand you must do), ask yourself some fairly critical questions. After all, if you don’t ask them, we’ll probably ask you instead.

What’s the date, time, and fully qualified server name? Are there any scheduled jobs running in the environment? (such as backup jobs or batch jobs? )

  • Is the service console pingable?
  • Is VMkernel pingable?
  • Are the virtual machines pingable?
  • Can I connect to the host in a VSphere client with Virtual Center?
  • Can I connect to the host in a VSphere client without Virtual Center?
  • Can I connect to the physical console either directly, or via ILO?

If ANY of these things are true, don’t reboot right away, give us a call, we may be able to help figure out a way out!

#3 My ESXi host broke, Check the logs!

We (VMware’s technical support engineers) almost always flinch when we get these requests. I covered this in a previous blog, but it’s worth reiterating.

ESXi lives in RAMdisk!

We can certainly understand a need for urgency, and if you need to recover your systems, you may have to reboot, but you also need to understand, because ESXi does live in RAMdisk its VERY COMMON that the logs will contain NOTHING prior to the reboot. Please understand, be aware of the risk, and don’t be surprised if your technician tells you there’s nothing to retrieve. ESXi HAD to give up something to get its footprint size. Our engineers aren’t magicians; they worked hard to cram as many functions as they did into such a small memory footprint. The bottom line is some hard choices had to be made, and some of the logging and diagnostic capability didn’t make the cut. That’s why it’s that much more important to use the capabilities ESXi does have, namely, syslog and the vMA (vSphere management assistant). Trust me, if you use ESXi, put them both in your tool belt. It’ll only take ONE outage incident for an external syslog server to pay for itself. It might not happen today, or tomorrow, but someday you’ll be glad you had them.

That’s all for today, barring acts of god, the economy, VMware’s management and Prince (who apparently believes the internet is “over” http://tinyurl.com/232jyv2) I’ll see you all in a few weeks.

Live well!

entia non sunt multiplicanda praeter necessitatem

Another post from guest blogger Mike Bean. A little Latin, a little logic, and the lowly service console.

Hello virtual world!

At the risk of singing an old refrain, I was unavoidably detained from my self-imposed one blog a week schedule. Due to circumstances beyond anyone’s control, my caseload has increased by an order of magnitude. Between you and me, I think my manager’s just trying to make sure I don’t have time to write any more blogs! Jokes aside, that’s really just the nature of the business we’re in, tech support is the information technology version of firefighting. Some days you’re playing cards in the station, next thing you know you’re hip deep in multiple five-alarms all over the county.

Can we speak candidly? Make no mistake about it, tech support kinda sucks. It’s very high stress, very high turnover rate, and generally fairly unappreciated. My intent here is not to solicit sympathy, but rather to paint an accurate picture. So why does anyone take the job?

Personally I tend to think it does have some advantages. It keeps the mind nimble. I can say with complete sincerity, that VERY RARELY do I leave the apartment with any real expectation of what the day has in store for me. More than once a customer has asked me, “So, have you ever seen anything like this?” Usually this is right about the time I’m frowning at my monitor with cartoon question marks spawning from my head. You really never know how the day will end. True story, one minute, I was fairly proud of myself. I’d managed to make a fairly tricky diagnosis concerning a half dozen some odd VM’s shutting down the night before. The next minute, I was getting REALLY BASIC elementary level instruction in unix/linux compression from our escalation engineers! (ah, humility, my old friend, it’s like you never left me!)

Despite all this seemingly random chaos, every now and then, patterns tend to emerge. This week’s blog, is in fact, brought to you by last week’s pattern, which, as it turns out, was (drum roll)

BROKEN SERVICE CONSOLES

Ladies and gentlemen, I’m going to ask as nicely as I know how, PLEASE! Tread lightly on your service consoles! Obviously, I can’t know what ESX’s developers had in mind. Regardless, I’ll gladly make the argument that the service console is NOT INTENDED for day to day use. As I’ve alluded to in a previous column, they are a maintenance hatch. It’s unrealistic to expect our customers not to use them, but, please, have the clear understanding that there-is-no nested maintenance hatch, within the maintenance hatch! I met one gentleman this week whose security department had pushed out untested authentication files to his ESX service consoles, effectively breaking the root user! (I’m still a little foggy on how we fixed that one, but I think it involved several sacred rites and sacrifices to the Aztec sun gods!)

The fact is, everyday at VMware, people care A LOT about stability, and they work hard to ensure that, BUT our QA department can’t test every possible package combination for compatibility and long term stability. It’s not realistically possible. So consider please, a possible rule of thumb. If you’re thinking about adding packages to your service console that weren’t there natively, please automatically assume that you’re adding a package VMware has NOT TESTED. There may be cases where that’s not true, but I think you’ll find it’s far more accurate as a rule then an exception.

I can recall a marathon session with a customer whose hosts weren’t making it more then 3 or 4 hours without crashing. Through old fashioned, traditional logic & process of elimination we narrowed it down a third party package on his consoles. When I followed up on the case several days later, he explained the software had been pushing out some sort of faulty active directory policy that was finally isolated as the source of the crashes!

entia non sunt multiplicanda praeter necessitatem

Occam In case my readers don’t speak Latin, I won’t make you guess what that means, if you recognize it, more power to you! It’s commonly known as Occam’s razor. "Entities must not be multiplied beyond necessity", or the more common interpretation = “the simplest explanation is usually the correct one.” Occam has all sorts of applications. It’s a 14th century logic principle, that’s considered a scientific guideline by some, a mathematics principle by others, and a philosophy rule by crazy tech support engineers, (like myself). My co-workers can testify to this, I actually wrote it on the whiteboard in my cubicle, as a reminder to myself.

By and large, we’ll leave philosophy to the philosophers, but I would suggest to you now that Occam is an EXCELLENT principle to follow when it comes to modifications of the service console. If, by any circumstance, the service console is completely unrecoverable, there is NO option but reinstallation. I ALWAYS encourage caution. The less you modify your console the better, and if you MUST modify it, FOLLOW Occam and do so only with the greatest of care. The simpler you keep your configuration, the more reliable the host will be, and the better the chances it’ll be there when you need it!

Live well!

Useful vSphere 4.1 Knowledgebase Articles

We’ve noticed a few blogs (such as this one) advertising lists of our recent Knowledgebase articles pertaining to the release of vSphere 4.1

We thought a comprehensive list would be great idea.  Here is every KB article posted for the vSphere 4.1 release.

http://kb.vmware.com/kb/1023163 Installing a VMware Service Manager license
http://kb.vmware.com/kb/1017910 Using Tech Support Mode in ESXi 4.1
http://kb.vmware.com/kb/1017928 Downloading the license file fails
http://kb.vmware.com/kb/1018178 ESX virtual machine monitor encounters a critical error and shuts down
http://kb.vmware.com/kb/1018180 Cannot power on a virtual machine and you receive an Admission Check Failed error
http://kb.vmware.com/kb/1018181 Attempting to remove a host from the inventory when it is connected to vDS generates an error
http://kb.vmware.com/kb/1018183 vSphere 4 license expires and you cannot register your product
http://kb.vmware.com/kb/1019144 vCenter Server 4.1 fails to install or upgrade with the error: This installation package is not supported by this processor type
http://kb.vmware.com/kb/1019738 Upgrading to Site Recovery Manager 4.0 Update 1 fails with the error: can’t open perl script "e:\program": no such file or directory
http://kb.vmware.com/kb/1019890 Virtual machines fail to start with a timeout error when performing a test failover
http://kb.vmware.com/kb/1020121 Powering on virtual machines fails with the error: MAX VCPUs limit is reached
http://kb.vmware.com/kb/1021491 Network configuration of a recovered virtual machine during Site Recovery Manager failover fails with the log error: Network Device Not Found
http://kb.vmware.com/kb/1021635 Migrating to the vCenter Server 4.1 database
http://kb.vmware.com/kb/1021695 Update Manager 4.1 patch repository features
http://kb.vmware.com/kb/1021769 Configuring  IPv6 with ESX and ESXi 4.1
http://kb.vmware.com/kb/1021779 Working with firewall rules in ESX 4.x
http://kb.vmware.com/kb/1021801 Location of ESXi log files
http://kb.vmware.com/kb/1021806 Location of log files for VMware products
http://kb.vmware.com/kb/1021827 A recovery or test recovery plan fails with virtual machines that have a pre-boot message
http://kb.vmware.com/kb/1021829 Deleted networks continue to appears in the Site Recovery Manager Configure Test Network dialog
http://kb.vmware.com/kb/1021919 Site Recovery Manager shadow virtual machines on the recovery site lose their vDS settings after a test failover
http://kb.vmware.com/kb/1021953 I/O Statistics in vSphere 4.1
http://kb.vmware.com/kb/1021970 Overview of Active Directory integration in ESX 4.1 and ESXi 4.1
http://kb.vmware.com/kb/1021976 vStorage APIs for Array Integration FAQ
http://kb.vmware.com/kb/1022030 ALUA parameters in the output of ESX/ESXi 4.1 commands
http://kb.vmware.com/kb/1022077 Support for the scsi_device_reprobe() API in VMKLinux in ESX 4.0 and 4.1
http://kb.vmware.com/kb/1022101 Installing ESX 4.1 and vCenter Server 4.1 best practices
http://kb.vmware.com/kb/1022104 Upgrading to ESX 4.1 and vCenter Server 4.1 best practices
http://kb.vmware.com/kb/1022137 vSphere 4.1 upgrade pre-installation requirements and considerations
http://kb.vmware.com/kb/1022140 Upgrading ESX 4.0 to ESX 4.1
http://kb.vmware.com/kb/1022178 Solution Licensing
http://kb.vmware.com/kb/1022256 vCenter Server 4.1 network port requirements
http://kb.vmware.com/kb/1022263 Deploying ESXi 4.1 using the Scripted Install feature
http://kb.vmware.com/kb/1022289 Changing the number of virtual CPUs per virtual socket in ESX/ESXi 4.1
http://kb.vmware.com/kb/1022290 USB support for ESX/ESXi 4.1
http://kb.vmware.com/kb/1022303 Avocent ACS6000 Virtual Serial Port Concentrator
http://kb.vmware.com/kb/1022308 Troubleshooting ESXi 4.1 Scripted Install errors
http://kb.vmware.com/kb/1022536 Using vShieldZones 1.0 with ESX 4.1
http://kb.vmware.com/kb/1022585 Network I/O Resource Management in vSphere 4.1 with vDS
http://kb.vmware.com/kb/1022587 Excluding a physical NIC from network resource management
http://kb.vmware.com/kb/1022590 Load Based Teaming in vSphere 4.1
http://kb.vmware.com/kb/1022842 Changes to DRS in vSphere 4.1
http://kb.vmware.com/kb/1022843 Changes to VMware High Availability in vSphere 4.1
http://kb.vmware.com/kb/1022844 Changes to Fault Tolerance in vSphere 4.1
http://kb.vmware.com/kb/1022851 Changes to vMotion in vSphere 4.1
http://kb.vmware.com/kb/1022867 Performing local administration of vCenter Server Heartbeart from an account that did not perform the installation
http://kb.vmware.com/kb/1022869 Joining a vCenter Server Heartbeat instance to a Linked Mode Group and Isolating a vCenter Server Instance from a Linked Mode Group when protected by vCenter Server Heartbeat
http://kb.vmware.com/kb/1022874 Downgrading vCenter Server Heartbeat to a previous version
http://kb.vmware.com/kb/1022877 Uninstalling VMware vCenter Server Heartbeat
http://kb.vmware.com/kb/1023026 Creating a static route for the VMware Channel
http://kb.vmware.com/kb/1023118 Changes to VMware Support Options in vSphere 4.1
http://kb.vmware.com/kb/1024066 TCP Offload features on all NICs must be disabled before insta
lling vCenter Server Heartbeat
http://kb.vmware.com/kb/1021920 Site Recovery Manager registering virtual machine task fails during failover with the error: Creating VM timeout
http://kb.vmware.com/kb/1022091 Troubleshooting Storage I/O Control
http://kb.vmware.com/kb/1023990 VMware ESX and ESXi 4.1 Comparison

VMware ALERT: VMware View Composer 2.0.x is not supported in a vSphere vCenter Server 4.1

There is an issue that prevents View Composer from working with vSphere 4.1 that customers need to be aware of. View Composer will not run on a 64-bit OS, and vSphere 4.1 only supports 64-bit OS’s. The release notes don't mention this and refers users to the VMware View Support page for compatibility information.

The Knowledgebase Team has prepared KB article: vSphere support for View Manager (1011292), and an alert has been placed on the View Support page to alert customers of this issue.

This Knowledge Base article will be updated if new information becomes available. If you have been affected by this, please read the KB.

We apologize for any inconvenience this may have caused you. If you know how to spread the word to your friends and colleagues, please do so.

EMC Ionix Customers Transitioning to VMware Support July 1st

In case you were unaware, EMC Ionix customers became fully supported by VMware on July 1st.  Our Customer Service, Licensing and Technical Support teams are trained and ready to help from our Colorado, California, Australia, UK, and Bangalore locations.

We’d like to draw your attention to our GSS Acquisition Support page to help with this transition. This page is a great place to start familiarizing yourselves with key information and self-help resources, before exploring further on www.vmware.com/support

New VMware vCenter Lab Manager Video Tutorial Series!

Today we have a post about VMware vCenter Lab Manager from Graham Daly, who is a tech support engineer in our Cork, Ireland office, currently focusing on content creation.

VMware vCenter Lab Manager is an application that provides a rapid provisioning portal and image library management system to automate the setup and teardown of multi-machine software configurations. Lab Manager leverages VMware vSphere and VMware vCenter to provide virtual infrastructure resources to multiple teams, projects, and geographies from a central location.

Lab Manager is an application that operates at the top end of a 3-tier system. Lab Manager sits on top of the vCenter Server and the ESX infrastructure or in other words, VMware vSphere. This means that before you actually start installing or configuring Lab Manager, you will need to have your vSphere environment configured.

KB article (1020915) is going to act as a central location or a “one-stop-shop” for tutorial style videos which will discuss and demonstrate the various different topics/aspects of the Lab Manager product.

As new videos become available, they will be added to the article.

The first three videos in the series are:

The next videos in the series will be:

  • Managing Users and Groups within vCenter Lab Manager
  • Networking within vCenter Lab Manager

You won’t want to miss these videos, so now is a good time to subscribe to this blog.