Home > Blogs > VMware Support Insider > Category Archives: Management

Category Archives: Management

Troubleshooting when virtual machine operations are greyed out

Hello, I am a Technical Support Engineer at VMware. I would like to talk about an issue I worked on recently with a customer that may come up for other users. The Support Request (SR) came to me with this description: “The Virtual Machine options are not working”. I had to ask myself, “What do you mean by ‘not working’”? I quickly called the customer and we got a remote session going so that I could see for myself what the problem was.

The problem was with only one Virtual Machine in that vCenter instance. I found that the virtual machine options were disabled (greyed-out) when I right-clicked the problematic virtual machine.

Note: Click an image to see the full-sized version.

The customer at this point mentioned that he had tried to take a backup using third-party software. He saw this error message in the “Recent Tasks” pane:

Error: Another task in progress

My initial hunch was that there was a job running for the virtual machine in the background, but I needed to verify it wasn’t a permissions problem.

Permissions are not set correctly

First I checked the permissions given to that virtual machine. The customer had a very small environment, so I did not spend too much time checking the complete permissions given for the entire vSphere environment. Instead, I went through the following steps:

Note: If your environment is large and you have multiple vSphere administrators with different permissions, permissions on a particular virtual machine machine might be incorrect/missing. In this case, the vSphere root administrator needs to ensure that there are sufficient permissions given for you to administer the virtual machine, at all levels.

  1. Check the Permissions tab for the virtual machine, to make sure that your name is listed there with the proper permission assigned. If your name is not there, but an AD/Local group is listed, then make sure that your name is added to the AD/Local group.Here, the user Testuser has the Virtual Machine Administrator role:
  2. If the permissions are defined at the host, cluster, Datacenter, or vCenter level:
      1. Apply the required permissions to the user/group.
      2. Make sure that the permission is propagated to all the objects.
      3. Make sure that your name is listed in the local/AD group.Here, the group Virtual Machine Administrator has the Virtual Machine Administrator role:

  3. Roles can also be assigned in the folder level. Go to the VMs and Templates view and make sure the folder where the Virtual Machine located has all required privileges assigned.In the below example, VM1,VM2 and VM3 are located in the folder Test. The Virtual Machine Administrator group is assigned with “Virtual Machine Administrator” role.
    In the examples above, the role “Virtual Machine Administrator” is not a default role, it is created with privileges to Administrate Virtual Machines. To know what privileges are required, I always refer to the Required Privileges for Common Tasks section in: Virtual Machine Administration Guide.

With that set of checks, we can now go on to the next step, and this comes back to my hunch because of the error message which I noticed in the “Recent Task”.

Tasks Running in the Background

We should always be aware that certain jobs for any given Virtual Machine may be running in the background which will not allow any changes/operations in the Virtual Machine. These background jobs cannot be seen in the “Task and Event” tab in the vCenter server or in the “Recent Tasks”.

The method we look for these background jobs is by logging into the host via SSH where the problem Virtual Machine is running. Here are the steps to find any running jobs which cannot be seen in the vSphere client.

1. Log into the ESX/ESXi host using a SSH client.
2. Run the following command to list all the Virtual Machines registered in the host. We are running this specifically to find the vmid for the problem Virtual Machine.

vim-cmd vmsvc/getallvms

The output appears as:

3. Check the tasks running on the Virtual Machine by running the command:

vim-cmd vmsvc/get.tasklist vmid

In our example, we run the command against the Virtual Machine VM3. It indicates there are 2 jobs/tasks running on that Virtual Machine.

The output shows as below if there is no jobs/tasks running.

4. Now you you have a choice to either wait for the jobs to complete, or restart the Management agent to terminate this job.

In this particular support case I was working we found a snapshot job that was triggered by the backup agent running in the Virtual Machine. I stopped the job by restarting the management agent on that host. For this customer, this solved the problem, but there is one more reason this might happen. I had a word with one my colleagues, and he pointed out that you could also encounter this if you have an invalid entry in your VMX file. Let’s go one step further and show you how to check this.

Invalid entries in VMX file

This type of issue might happen if a vmx file has invalid parameters or blank lines in it. You can resolve this issue by manually removing the invalid arguments or deleting the blank lines.

Caution: If incorrectly done, your virtual machine may fail to start or operate incorrectly. Always take a backup of your .vmx file before modifying it.

  1. Open the .vmx file using any text editor.
  2. Search for any blank lines and delete them.
    Note
    : To delete a single line using vi editor, press d twice.
  3. Compare the .vmx file with a working virtual machine .vmx file and see if there are any invalid arguments.
  4. To apply the changes, reload the .vmx file by running the command:
vim-cmd vmsvc/reload vmid

Note: The Default location of the vmx file is:

Vmfs/volumes/name_of_the_datastore/vm_name/vm_name.vmx

Summary

Symptom:
Virtual machine operations are grayed out

Possible Causes:

  • Permissions are not set correctly
  • Tasks Running in the Background
  • vmx file is corrupted

For further information on the steps used in this article, refer to: Troubleshooting when virtual machine options are grayed out in vSphere Client (2048748)

Hope this blog post helps you out a little bit. Have a great day!

New Network port diagram for vSphere 5.x

Over the past few weeks we have been working on constructing a brand new network diagram, depicting ports in use for vSphere 5.x

These diagrams have been very popular in the past and we hope you like this one too! We created Knowledgebase article: Network port diagram for vSphere 5.x (2054806) as a container for the pdf diagram. The pdf also lists all of the ports used in tabular format.

If you’d like to see more of these, tell us in the comments section below!

Network port diagram for vSphere 5

Alternate download location.

Note: This information provided is on a best effort basis. VMware will endeavor to update the diagram as new releases come out.

Physical or Appliance – Upgrading to vCenter Server 5.1

The other day we received this question from a customer via Twitter:

@VMwareCares planning to upgrade to 5.1 from 5.0 vcenter. What’s recommended physical or appliance? Ups and downs side of each?

We thought a few more of you might have the same questions so we decided we would take the opportunity to explain the differences between vCenter server and vCenter appliance and under what situation which one should be opted for, over the other.

The vCenter Server Appliance (vCSA) is a preconfigured Linux-based virtual machine optimized for running vCenter Server and associated services. Versions 5.0.1 and 5.1 of the vCSA uses PostgreSQL for the embedded database instead of IBM DB2, which was used in vCenter Server Appliance 5.0 The vCSA embedded postSQL DB supports 5 hosts / 50 virtual machines, with an Oracle DB the vCSA can support 1000 hosts and 10,000 vms. If you configure your vCSA to use an external instance of Single Sign On (SSO), the external SSO instance must be hosted on another vCenter Server Appliance; it cannot be hosted on a Windows machine.

vCenter Server can be installed on a windows Guest OS and can be connected to Oracle or Microsoft SQL. SSO can be installed on the same Guest OS or can be on a different machine. It should be noted that patching of of the vCenter Appliance is not supported.

Below is a table listing more of the differences between the products.

Features vCenter Server vCenter Server Appliance
Guest OS Any Supported Guest OS Preconfigured Linux-based virtual machine (64-bit SUSE Linux Enterprise Server 11)
Database Supported Versions SQL Server and Oracle. PostgreSQL (built-in ) can have 5 hosts and 50 Virtual Machines.Supported External Oracle database.
System Requirement 2 vCPU and 4 GB RAM 2 vCPU and 4 GB RAM
Platform Physical or virtual machine Virtual Appliance
Installation Using binary provided in .zip or .ISO Deploying OVF
Update Manager Can be installed on same vCenter Server or on separate Guest OS. Separate install
Single Sign On (SSO) Can be installed on same vCenter Server or separate Guest OS. Pre-installed.
Networking IPv6 and IPv4 Support IPv4 Support
Linked Mode Supported Not Supported
SRM (Site Recovery Manager) Compatible with SRM Compatible with SRM
vSphere Web Client Can be installed on same vCenter server or separate machine. Pre-Installed.
Syslog Server Can be installed on vCenter Server or separate server and configured using plug-in. Pre-installed and does not have plug-in.
ESXi Dump collector Can be installed on vCenter Server or on a separate Guest OS. Pre-installed and does not have plug-in.
Multi-site SSO Supported Not Supported. Basic SSO only.
VSA (vSphere Storage Appliance Supported Not Supported
VMware View Supported Not Supported

SSL Certificate Automation Tool version 1.0.1

Last month we announced a new SSL Certificate Automation tool to help everyone with the implementation of custom certificates. Yesterday, we released the second version of it (version 1.0.1). This is a minor update which aims to simplify the replacement of certificates further by adding Certificate Signing Request (CSR) functionality to the tool. This functionality allows a user to quickly generate certificate requests (and consequently the private keys) for submission to the Certificate Authority.  The CSR functionality was the largest portion of manual steps, and as a result the update reduces the number of steps by over 15.

In addition, there are several minor bug fixes which were fixed which impacted tool functionality.

For further details and to download the latest version of the SSL tool see: Deploying and Using the SSL Certificate Automation Tool (2041600)

We hope these additions provide useful for everyone!

The Inside Scoop: Maintenance tips for your vSphere Database

Today we have the third edition of our blog series The Inside Scoop. In this installment we will look at vSphere Databases and more specifically some helpful tips for maintaining them.

In order to obtain some real world perspective, we met up with some of our Technical Support Engineers at our support center in Cork, Ireland and mainly asked them two questions:

  1. What are the most common issues they deal with concerning vSphere Databases?
  2. What advice do they have for ensuring that a vSphere Database is maintained?

Here is what they had to say….

Common Issues

The two most common issues that come into our Technical Support teams are:

  1. Database Corruption
  2. Database Performance

These are really the two biggest issues that customers encounter with their SQL databases in their vSphere environments.

Many a database administrator has nightmares about database corruption and when an incident comes along quite often many hours are spent by the DBA trying to rescue the situation. Sadly, database corruption is something that just happens; nobody plans to have it.

If you are or were a system administrator or a database administrator at some point during your career, chances are that there was probably a time when you learned the hard way about not having a recent database backup.

However it is not all doom and gloom when it comes to database corruption incidents. The impact and headaches of such a corruption incident can be minimized and reduced by simply applying and enforcing a policy of regular database backups. Taking regular database backups will not fix the corrupted database but at least your road to recovery will be a much better and less painful one.

Along with database corruption the other big generator for support requests is that of database performance. A database is like the heart of the environment and just like a heart, if it is in a bad or a poorly maintained condition then it is going to experience performance issues.

The vSphere database is what manages and runs the jobs and processes that take place within the environment in any given moment. The speed at which the vSphere environment can run effectively and efficiently is quite often determined by the health of the database. If your database is unhealthy, then chances are you will notice performance impacts within your environment.

What symptoms should I look out for?

Symptoms of database corruption would include the vCenter Server failing to start or crashing on particular tasks.

Symptoms for database performance related issues can be more varied, however some common ones include:

  • The vCenter Server taking a long time to start up
  • Tasks taking a long time to complete or are timing out

Some Helpful Database Maintenance Tips

When it comes to database corruption scenarios the best thing that you really can have is a recent backup. This will save a lot of time and heartache when it comes to restoring your environment and the more recent the backup the better as it will minimize the loss of data.

In regards to database performance issues, prevention really is the best cure and so here are some steps and measures which will help to reduce or prevent your environment from encountering poor database performance:

  1. Monitor scheduled database jobs to ensure they are running correctly – For more information, refer to KB article: Checking the status of vCenter Server performance rollup jobs (2012226)
  2. Collect Stats
  3. Rebuild Indexes – For more information, refer to KB article: Rebuilding indexes to improve the performance of SQL Server and Oracle vCenter Server databases (2009918)
  4. Delete old data – For more information, refer to KB article: Reducing the size of the vCenter Server database when the rollup scripts take a long time to run (1007453)
  5. Monitor Database Growth – For more information, refer to KB article:
    Determining where growth is occurring in the vCenter Server database (1028356)

A pdf document on vCenter Server Database Best Practices is available: VMware vCenter Server 5.1 Database Performance Improvements and Best Practices for Large-Scale Environments

Introducing … RSS updates from the VMware Compatibility Guide / HCL

In the past, administrators wishing to determine whether their specific hosts, guest operating systems, storage arrays, or other hardware were on the official VMware Compatibility Guide / HCL had to continually return to our page to see if VMware had added support or added any updates.

The new and improved VMware Compatibility Guide makes it easier to get updates about specific hardware by providing RSS feed subscriptions. To subscribe to updates about an item:

  1. Search for the item you’re interested in.
  2. From the search results, click the model name.
  3. In the Model Detail seciton of the page, at the bottom-right corner, click the rss feed link.

You’ll be prompted for a way to subscribe to the feed, based on your browser and configuration. Updates will be sent via RSS when the page for this model is updated.

For more information about the Compatibility Guide / HCL, see our KB article Using the VMware HCL and Product Interoperability Matrixes (2006028).

Introducing the vCenter Certificate Automation Tool 1.0

Fresh out of development today VMware has a new tool to help everyone with the implementation of custom certificates. The vCenter Certificate Automation Tool 1.0, will help customers update certificates needed for running vCenter Server and supporting components. This is primarily of interest to customers who use custom certificates either generated internally from Corporate CAs, or from public CA’s like VeriSign.

To add a little background information various components within vSphere and the vCenter platform use certificates for identifying themselves as well as for secure communication with external software entities (browsers, API clients).  These can broadly be classified into the following categories:

  1. Secure token Service Certificate – Certificate used by vCenter Single Sign On (SSO) for encryption tokens
  2. Solution User Certificates – Certificates used by each solution to identify themselves as users to SSO
  3. SSL Certificates  – certificates needed for SSL communication for the UI and API layer
  4. Host Certificates – These certificates are deployed in each ESXi host and used for secure vCenter to ESXi communication.

Note: The new certificate tool automates the updating of certificates in the management layer only (a, b, c above). This tool does NOT handle replacement of certificates in ESXi hosts.

The vCenter Certificate Automation Tool aims to automate the process of uploading certificates and restarting the following components within the vCenter Platform:

  1. vCenter Server
  2. vCenter Single Sign On
  3. vCenter Inventory Service
  4. vSphere Web Client
  5. vCenter Log Browser
  6. VMware Update Manager (VUM)
  7. vCenter Orchestrator (VCO)

For more information on how to download, install, and use the tool, refer to KB article: Deploying and Using the SSL Certificate Automation Tool (2041600).

Trending Issues in ESXi and vCenter Server

As we spoke about in this announcement, here are our currently trending KB articles related to ESXi and vCenter Server. These are the issues the majority of users are encountering. See anything familiar?

The Inside Scoop: Troubleshooting tips for common issues with vCenter Server Single Sign-On (SSO)

Good morning folks,

Today we have the second edition of our new blog series The Inside Scoop. In this installment we will be focusing on the topic: Troubleshooting tips for common issues with vCenter Server Single Sign-On (SSO).

With the release of vSphere 5.1 the virtualization world was introduced to vCenter Server Single Sign-On. Along with the incredible new features and abilities that Single Sign-On offered, there was also some confusion around the installation and configuration of SSO amongst our user communities.

We met up with some of our front line Technical Support Engineers at our support center in Cork, Ireland and quizzed their brains for a little while in regards to Single Sign-On, and in order to get an insight into the common configuration issues that they see on a weekly basis, and also get some of their recommendations on how to troubleshoot common issues.

The  primary area of expertise of these Engineers is our vSphere product family, more specifically, our vCenter Server product. They spend a large portion of their time helping customers troubleshoot the issues that they are encountering in their environments. Well, here is what they had to say…

Check your Identity Sources

One of the most frequent questions we receive from new users of Single Sign-On is:

Why am I unable to see the expected identity sources in my identity sources list?

The answer is actually quite a straight forward one. Identity sources will only be automatically discovered during the SSO installation if the user running the installation was a Domain Administrator. If a Local User ran the installation, then automatic discovery would not occur.

Another issue which pops up from time to time is when identity sources have been manually added by a user but the Domain Alias was never specified.

Ensure that you specify the Domain Alias when manually adding an identity source as without it the “Use Windows Session Credentials” option will not work. Also, it is worth noting that you cannot modify the Domain Alias after an identity source has been created. You will need to delete the existing identity source and re-create in order to add the Domain Alias.

Some users have also reported issues where they are experiencing longer than expected wait times upon logging into the vCenter Server system. In most cases, this can be easily resolved by adding the identity source to the list of Default Domains and moving it to the top of the list. This will speed up the authentication times as the first in the list will be the identity source that is queried first upon login. Good tip!

Note: Be sure to click the Save icon in order to save the changes to the Default Domains.

Don’t forget the Database Scripts!

When installing vCenter Single Sign-On you are presented with two options concerning which type of database you want to use for the Single Sign-On installation. These options are:

  • Install a local Microsoft SQL Server 2008 R2 Express instance
  • Use an existing supported database

Quite often we see Single Sign-On installations where this step is overlooked. If you choose to use the an existing support database option, you must run two SQL scripts prior to being able to install Single Sign-On. The install wizard will prompt you to perform this action. The two scripts are located on the vCenter Server 5.1 Installation Media in the \Single Sign On\DBScripts\SSOServer\schema\<DB>\ directory.

The two scripts are:

  • rsaIMSLite<DBName>SetupTablespaces.sql
  • rsaIMSLite<DBName>SetupUsers.sql

Clicking “Next” and progressing to the next section in the installation wizard without first having run the above mentioned database scripts, would result in an error message such as the one shown below because the installer will not be able to communicate with the database due to the fact that the database is not even there!

The log file quoted in the above error dialog box will contain an error message similar to the following:

ERROR  1217[main] – com.vmware.vim.installer.core.logging.CoreLoggerImpl.error(?:?) – Failed to established connection :com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user ‘RSA_DBA’

In order to ensure that your vSphere 5.1 environment gets off to the best start possible, whether performing an fresh install or upgrading from an existing vSphere 5.0.x version, it would be best to familiarize yourself with the various best practices and methods of installation and upgrading. The following VMware Knowledge Base articles would be a great place to start:

Avoid unnecessary changes

Whenever we receive support requests from our user community complaining of symptoms such as “vCenter Server failed to start after restarting the Single Sign-On server”, one of the first areas we consider looking at is whether or not something has changed on the system where the Single Sign-On server is installed. If the system where the Single Sign-On server is installed is changed, this will cause the SSO server to fail to start and in turn will cause the vCenter Server from starting.

What kind of changes are we talking about?

For example:

  • When updates are applied to the operating system
  • The machine name changes
  • The machine is added or removed from an Active Directory domain
  • If you clone or change the parameters of a virtual machine where SSO is installed (such as the amount of RAM, the number of CPUs, the MAC address)

When it comes to the machine where the Single Sign-On installation is situated, it would be best advised to avoid making system changes on that machine whether it be a physical or virtual machine.

For additional information relating to this issue, check out VMware Knowledge Base article: After making a change or restarting Single Sign On server system, vCenter Server 5.1.x fails to start (2036170).

Single Sign-On Passwords

Sometimes we receive support requests from users looking for assistance relating to failed login attempts. The users report that they are no longer able to login with their Single Sign-On administrator credentials into their vCenter Server systems using the vSphere Web Client. Users who report this problem usually also see the following error message.

User account is locked. Please contact your administrator

Being unable to login using your SSO administrator credentials can occur for several reasons but we find that most common causes are either:

  • Too many failed login attempts
  • The password for the admin@system-domain user has been changed
  • The Single Sign-On passwords have expired

For security purposes, the administrator account is automatically locked out if there are too many failed login attempts. By default, vCenter SSO allows for three failed login attempts before an account is locked out. These policies can be modified and the instructions for doing so are found in VMware Knowledge Base article Configuring and troubleshooting vCenter Single Sign On password and lockout policies for accounts (2033823).

At the time of installation the password for the admin@system-domain user is defined. This also defines the password for the “Master” password. It is worth noting that changing the password for the user admin@system-domain does not change the Master password. The Master password continues to remain the same as the original password that was defined at installation time and for this reason it is recommended that you always keep a note of this password as you may need it in an emergency. If your SSO administrator password needs to be unlocked or reset, see VMware Knowledge Base article Unlocking and resetting the vCenter Single Sign On (SSO) administrator password (2034608) for details.

Single Sign On account passwords expire after 365 days, including the password for admin@system-domain user. To reset SSO user account passwords, refer to VMware Knowledge Base article Resetting an expired password in VMware Single Sign On (SSO) (2035864).

Watch out for Non-ASCII characters

In the initial releases of vCenter Server 5.1, Single Sign-On installations may have failed due to SSO administrator passwords containing non-ASCII characters.  This issue has since been resolved with the release of VMware vCenter Server 5.1.0a, but if you are running on an older release and are seeing errors such as “Error 29133 Administrator login error” or “Error 29158 Node package creation error“, then we would recommend that you update your system to either of the more recent 5.1.0a or 5.1.0b releases as soon as you can. Additional information concerning this issue is available in KB article: vSphere 5.1 Single Sign On (SSO) installation fails with error: Error 29133. Administrator login error (2035820).

Well, that concludes our Inside Scoop edition for this week. We hope this will be of some help to you in keeping your vCenter Server Single Sign-On environments running as smoothly as possible. Be sure to check back with us again in the coming weeks for the next edition of The Inside Scoop!

Mismatched VMware build numbers

We recently heard from a customer:

“I went ahead and installed vCenter 5.1.0b (all components were upgraded, SSO, Inventory Service, vCenter and Web Client). When I launched vCenter and check its build number it doesn’t show the same build number that is on the ISO file name.  The ISO file name had a build number of 947939 but vCenter shows Build 947673.  Is that right? If so, why the mismatch?”

This is due to our package build methodology.

All builds generated in our internal build server come with a unique build number. For instance, VPX is a key product build generated through our internal build server, where all Virtual Center related changes are included. Then, VIMISO gets generated, but it does so by packing multiple components like VPX,SSO,NGC,VIC, VCO and others so VIMISO comes with it’s own unique number that applies to all of the included components.

When we connect NGC/VIC to Virtual Center Server, its displays the VPX Build number whereas the file name shows the VIMISO build number.

Now that we know why this occurs, to look at the numbers you really should be concerned with, refer to KB article: Correlating vCenter Server and ESXi/ESX host build numbers to update levels (1014508).