Home > Blogs > Support Insider > Category Archives: Inside Scoop

Category Archives: Inside Scoop

Important KB updates for current NSX for vSphere users -May 2016 Edition

NSXOur NSX support team would like all of our customers to know about important KB updates for current NSX for vSphere issues. Here’s what’s new and trending-

Please take note of key updates to the following important End of General Support and End of Availability events:

New and important issues:

NSX for Multi-Hypervisor:

New master playbook KBs:

How to track the top field issues:

 

The Inside Scoop: Maintenance tips for your vSphere Database

Today we have the third edition of our blog series The Inside Scoop. In this installment we will look at vSphere Databases and more specifically some helpful tips for maintaining them.

In order to obtain some real world perspective, we met up with some of our Technical Support Engineers at our support center in Cork, Ireland and mainly asked them two questions:

  1. What are the most common issues they deal with concerning vSphere Databases?
  2. What advice do they have for ensuring that a vSphere Database is maintained?

Here is what they had to say….

Common Issues

The two most common issues that come into our Technical Support teams are:

  1. Database Corruption
  2. Database Performance

These are really the two biggest issues that customers encounter with their SQL databases in their vSphere environments.

Many a database administrator has nightmares about database corruption and when an incident comes along quite often many hours are spent by the DBA trying to rescue the situation. Sadly, database corruption is something that just happens; nobody plans to have it.

If you are or were a system administrator or a database administrator at some point during your career, chances are that there was probably a time when you learned the hard way about not having a recent database backup.

However it is not all doom and gloom when it comes to database corruption incidents. The impact and headaches of such a corruption incident can be minimized and reduced by simply applying and enforcing a policy of regular database backups. Taking regular database backups will not fix the corrupted database but at least your road to recovery will be a much better and less painful one.

Along with database corruption the other big generator for support requests is that of database performance. A database is like the heart of the environment and just like a heart, if it is in a bad or a poorly maintained condition then it is going to experience performance issues.

The vSphere database is what manages and runs the jobs and processes that take place within the environment in any given moment. The speed at which the vSphere environment can run effectively and efficiently is quite often determined by the health of the database. If your database is unhealthy, then chances are you will notice performance impacts within your environment.

What symptoms should I look out for?

Symptoms of database corruption would include the vCenter Server failing to start or crashing on particular tasks.

Symptoms for database performance related issues can be more varied, however some common ones include:

  • The vCenter Server taking a long time to start up
  • Tasks taking a long time to complete or are timing out

Some Helpful Database Maintenance Tips

When it comes to database corruption scenarios the best thing that you really can have is a recent backup. This will save a lot of time and heartache when it comes to restoring your environment and the more recent the backup the better as it will minimize the loss of data.

In regards to database performance issues, prevention really is the best cure and so here are some steps and measures which will help to reduce or prevent your environment from encountering poor database performance:

  1. Monitor scheduled database jobs to ensure they are running correctly – For more information, refer to KB article: Checking the status of vCenter Server performance rollup jobs (2012226)
  2. Collect Stats
  3. Rebuild Indexes – For more information, refer to KB article: Rebuilding indexes to improve the performance of SQL Server and Oracle vCenter Server databases (2009918)
  4. Delete old data – For more information, refer to KB article: Reducing the size of the vCenter Server database when the rollup scripts take a long time to run (1007453)
  5. Monitor Database Growth – For more information, refer to KB article:
    Determining where growth is occurring in the vCenter Server database (1028356)

A pdf document on vCenter Server Database Best Practices is available: VMware vCenter Server 5.1 Database Performance Improvements and Best Practices for Large-Scale Environments

The Inside Scoop: Troubleshooting tips for common issues with vCenter Server Single Sign-On (SSO)

Good morning folks,

Today we have the second edition of our new blog series The Inside Scoop. In this installment we will be focusing on the topic: Troubleshooting tips for common issues with vCenter Server Single Sign-On (SSO).

With the release of vSphere 5.1 the virtualization world was introduced to vCenter Server Single Sign-On. Along with the incredible new features and abilities that Single Sign-On offered, there was also some confusion around the installation and configuration of SSO amongst our user communities.

We met up with some of our front line Technical Support Engineers at our support center in Cork, Ireland and quizzed their brains for a little while in regards to Single Sign-On, and in order to get an insight into the common configuration issues that they see on a weekly basis, and also get some of their recommendations on how to troubleshoot common issues.

The  primary area of expertise of these Engineers is our vSphere product family, more specifically, our vCenter Server product. They spend a large portion of their time helping customers troubleshoot the issues that they are encountering in their environments. Well, here is what they had to say…

Check your Identity Sources

One of the most frequent questions we receive from new users of Single Sign-On is:

Why am I unable to see the expected identity sources in my identity sources list?

The answer is actually quite a straight forward one. Identity sources will only be automatically discovered during the SSO installation if the user running the installation was a Domain Administrator. If a Local User ran the installation, then automatic discovery would not occur.

Another issue which pops up from time to time is when identity sources have been manually added by a user but the Domain Alias was never specified.

Ensure that you specify the Domain Alias when manually adding an identity source as without it the “Use Windows Session Credentials” option will not work. Also, it is worth noting that you cannot modify the Domain Alias after an identity source has been created. You will need to delete the existing identity source and re-create in order to add the Domain Alias.

Some users have also reported issues where they are experiencing longer than expected wait times upon logging into the vCenter Server system. In most cases, this can be easily resolved by adding the identity source to the list of Default Domains and moving it to the top of the list. This will speed up the authentication times as the first in the list will be the identity source that is queried first upon login. Good tip!

Note: Be sure to click the Save icon in order to save the changes to the Default Domains.

Don’t forget the Database Scripts!

When installing vCenter Single Sign-On you are presented with two options concerning which type of database you want to use for the Single Sign-On installation. These options are:

  • Install a local Microsoft SQL Server 2008 R2 Express instance
  • Use an existing supported database

Quite often we see Single Sign-On installations where this step is overlooked. If you choose to use the an existing support database option, you must run two SQL scripts prior to being able to install Single Sign-On. The install wizard will prompt you to perform this action. The two scripts are located on the vCenter Server 5.1 Installation Media in the \Single Sign On\DBScripts\SSOServer\schema\<DB>\ directory.

The two scripts are:

  • rsaIMSLite<DBName>SetupTablespaces.sql
  • rsaIMSLite<DBName>SetupUsers.sql

Clicking “Next” and progressing to the next section in the installation wizard without first having run the above mentioned database scripts, would result in an error message such as the one shown below because the installer will not be able to communicate with the database due to the fact that the database is not even there!

The log file quoted in the above error dialog box will contain an error message similar to the following:

ERROR  1217[main] – com.vmware.vim.installer.core.logging.CoreLoggerImpl.error(?:?) – Failed to established connection :com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user ‘RSA_DBA’

In order to ensure that your vSphere 5.1 environment gets off to the best start possible, whether performing an fresh install or upgrading from an existing vSphere 5.0.x version, it would be best to familiarize yourself with the various best practices and methods of installation and upgrading. The following VMware Knowledge Base articles would be a great place to start:

Avoid unnecessary changes

Whenever we receive support requests from our user community complaining of symptoms such as “vCenter Server failed to start after restarting the Single Sign-On server”, one of the first areas we consider looking at is whether or not something has changed on the system where the Single Sign-On server is installed. If the system where the Single Sign-On server is installed is changed, this will cause the SSO server to fail to start and in turn will cause the vCenter Server from starting.

What kind of changes are we talking about?

For example:

  • When updates are applied to the operating system
  • The machine name changes
  • The machine is added or removed from an Active Directory domain
  • If you clone or change the parameters of a virtual machine where SSO is installed (such as the amount of RAM, the number of CPUs, the MAC address)

When it comes to the machine where the Single Sign-On installation is situated, it would be best advised to avoid making system changes on that machine whether it be a physical or virtual machine.

For additional information relating to this issue, check out VMware Knowledge Base article: After making a change or restarting Single Sign On server system, vCenter Server 5.1.x fails to start (2036170).

Single Sign-On Passwords

Sometimes we receive support requests from users looking for assistance relating to failed login attempts. The users report that they are no longer able to login with their Single Sign-On administrator credentials into their vCenter Server systems using the vSphere Web Client. Users who report this problem usually also see the following error message.

User account is locked. Please contact your administrator

Being unable to login using your SSO administrator credentials can occur for several reasons but we find that most common causes are either:

  • Too many failed login attempts
  • The password for the admin@system-domain user has been changed
  • The Single Sign-On passwords have expired

For security purposes, the administrator account is automatically locked out if there are too many failed login attempts. By default, vCenter SSO allows for three failed login attempts before an account is locked out. These policies can be modified and the instructions for doing so are found in VMware Knowledge Base article Configuring and troubleshooting vCenter Single Sign On password and lockout policies for accounts (2033823).

At the time of installation the password for the admin@system-domain user is defined. This also defines the password for the “Master” password. It is worth noting that changing the password for the user admin@system-domain does not change the Master password. The Master password continues to remain the same as the original password that was defined at installation time and for this reason it is recommended that you always keep a note of this password as you may need it in an emergency. If your SSO administrator password needs to be unlocked or reset, see VMware Knowledge Base article Unlocking and resetting the vCenter Single Sign On (SSO) administrator password (2034608) for details.

Single Sign On account passwords expire after 365 days, including the password for admin@system-domain user. To reset SSO user account passwords, refer to VMware Knowledge Base article Resetting an expired password in VMware Single Sign On (SSO) (2035864).

Watch out for Non-ASCII characters

In the initial releases of vCenter Server 5.1, Single Sign-On installations may have failed due to SSO administrator passwords containing non-ASCII characters.  This issue has since been resolved with the release of VMware vCenter Server 5.1.0a, but if you are running on an older release and are seeing errors such as “Error 29133 Administrator login error” or “Error 29158 Node package creation error“, then we would recommend that you update your system to either of the more recent 5.1.0a or 5.1.0b releases as soon as you can. Additional information concerning this issue is available in KB article: vSphere 5.1 Single Sign On (SSO) installation fails with error: Error 29133. Administrator login error (2035820).

Well, that concludes our Inside Scoop edition for this week. We hope this will be of some help to you in keeping your vCenter Server Single Sign-On environments running as smoothly as possible. Be sure to check back with us again in the coming weeks for the next edition of The Inside Scoop!

The Inside Scoop: Common configuration pitfalls and issues our vCenter Server users are encountering

Good morning folks.

Fresh off the press, here is the first edition of our new blog series The Inside Scoop. See the introductory post for this series for some additional background information.

This first edition focuses on the topic Common configuration pitfalls and issues our vCenter Server users are encountering.

In a somewhat of a journalistic fashion, we met up with some of our front line Technical Support Engineers at our Support Center in Cork, Ireland and sat down with them so that we could pick their brains for a while. These Support Engineer’s primary area of expertise is for our vSphere product family, more specifically, our vCenter Server product. They spend a large portion of their time helping customers troubleshoot the issues that they are encountering in their environments.

When we met up with these very helpful individuals, we had one question on our minds: what are the common vCenter Server configuration issues that are seen most often coming into the Technical Support teams.

Well, enough talking from us. Here are the most common configuration pitfalls and issues our vCenter Server users are encountering according to our Technical Support Engineers.

vCenter Server Statistics levels set too high

Often times we receive Support Requests from users complaining of poor or slow performance in their vCenter Server environments, specifically after the vCenter Server Statistics levels have been increased. Other symptoms can include:

  • The VMware VirtualCenter Server service takes longer to start or may even fail to start
  • The vCenter Server database increases significantly in size
  • The database transaction log increases significantly in size
  • The performance data rollup jobs and stored procedures take a long time to run and may fail
  • Database jobs may fail
  • Performance graphs display gaps

The default statistic level is set to “1” which is the lowest setting. Sometimes this is increased to higher levels within an environment for various reasons, more than likely troubleshooting, but is forgotten, and not decreased after the fact. This can lead to a higher load being placed on the vCenter Server and the amount of data which the vCenter Server rollup jobs need to process will increase. As a result, vCenter Server might experience performance effects such as slowness, database job delays, failures, and even service interruptions.

Realistically, the Statistics level should only be increased for troubleshooting purposes and only for a short period of time, no longer than 24 hours, and should be reduced again as soon as possible in order to avoid these kinds of performance issues.

Your best bet would be to always ensure that the Statistics levels remain at the default level and only increase when absolutely necessary. For additional information see the following VMware Knowledge Base articles:

 vCenter Server performance data (Tasks & Events) growing too large

Another common issue that is seen through various incoming Support Requests is when  vCenter Server experiences performance issues due to the Tasks and Events data  growing too large, consuming disk space and ultimately leading to vCenter Server failures.

Some of the symptoms which users report are as follows:

  • vCenter Server service starts but eventually crashes
  • A slow down in performance is noticed regarding vCenter Server
  • Database jobs are failing
  • The database transaction logs are working correctly
  • Viewing the data Disk usage by tables report shows the VPX_EVENT and VPX_EVENT_ARG tables utilizing the maximum space
  • You may notice excessive growth of the VPX_EVENT table, and this error message in the Microsoft SQL Event log:

Could not allocate space for object ‘dbo.VPX_EVENT’.’VPXI_EVENT_USERNAME’ in database ‘VCDB’ because the ‘PRIMARY’ filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup

vCenter Server stores performance data in the vCenter Server database. Over time, data collection results in growth of the database files. This issue is actually pretty easy and straight forward to resolve and can be prevented from reoccurring again.

vCenter Server has a Database Retention Policy setting that allows you to specify when vCenter Server tasks and events should be deleted. VMware Knowledge base article Purging old data from the database used by vCenter Server (1025914) discusses this topic and provides instructions concerning the Database Retention Policy. There is also an embedded KBTV video in that KB article which demonstrates purging old data from the vCenter Server database.

vCenter Server SQL database transaction logs growing to excessive sizes

Thirdly and lastly for today is the issue of the Database transaction logs filling up. This is another very common issue reported by our vCenter Server users. Users often report symptoms such as:

  • The transaction log of the vCenter Server database grows to an excessive size
  • The vCenter Server service fails to start
  • The Windows Event logs state – Faulty Application : vpxd.exe
  • The vCenter Server vpx.log may show an error similar to to the following when connected to a SQL database

ODBC error: (42000) – [Microsoft][ODBC SQL Server Driver][SQL Server]The log file for database ‘<database>’ is full. Back up the transaction log for the database to free up some log space

Many users opt to use Microsoft’s SQL database for their vCenter Server installation. SQL Server offers administrators a choice of recovery models, which is the primary factor that determines transaction log disk space requirements. The default recovery model on the full version of SQL is Full Recovery. This recovery model has the potential to consume all available disk space if appropriate database maintenance is not performed. It is best practice to schedule regular backups of the database and transaction log to avoid unnecessary growth.

More often than not, the vCenter Server will crash when all of the disk space is being consumed by the transaction logs. Also, if the database does not have enough space to write to the transaction log, the vCenter Server service will fail to start. VMware Knowledge Base article Troubleshooting transaction logs on a Microsoft SQL database server (1003980) is the first port of call if you are encountering such symptoms. This article clearly describes what is happening and what you can do to resolve this issue.

Our Technical Support Engineers will often recommend using the Simple Recovery model unless your business requirement dictates otherwise. There are some limitations to using the Simple Recovery model and these are described within the Additional Information section of the above mentioned Knowledge Base article, so be sure to read up on them and familiarize yourself first before committing to a particular recovery model. Other Knowledge Base articles which may be of interest here include:

That concludes our Inside Scoop edition for this week and we hope this will be of some help and assistance to you in regards to keeping your vCenter Server environments running as smoothly as possible. Be sure to check back with us again in the coming weeks for the next edition of The Inside Scoop which will focus on Top tips for troubleshooting SSO related issues.

Introducing the Inside Scoop!

Normally when you see posts from me on this blog I am sharing our latest KBTV video production. Well, I will still be doing that, so no change on that front, however, I will be starting something new next week.

Next week will see the birth of a new series of posts entitled The Inside Scoop. In this series I will be acting as somewhat of a “journalist” wherein I will be meeting with several of our front-line Technical Support staff and interviewing them. The purpose is to to bring you some great insights and topical based information concerning our products straight from the very people who help many of you on a weekly basis through our Technical Support community.

Who best to offer you advise and tell you the DOs and DONTs of our products than the people who spend their entire day researching, investigating, analyzing and fixing the technical issues with our products. Oh, and our guests will surely point out some of our best Knowledgebase content too.

Over the coming weeks expect to see topics such as:

  • Top tips for troubleshooting SSO related issues
  • Must-have resources for troubleshooting vSphere related issues
  • Database maintenance tips
  • Common configuration mistakes our Desktop product users are making
  • Troubleshooting failed ThinApps
  • And much more

Our first post next week will feature Common configuration mistakes our vCenter Server users are making.

So stay tuned and keep an eye out for these blog posts.