Home > Blogs > Virtualize Business Critical Applications > Monthly Archives: March 2011

Monthly Archives: March 2011

Citrix XenApp on VMware Best Practices

Desktop application delivery and management can be tedious and time-consuming. Many organizations have chosen to leverage application virtualization and take a software as a service (SaaS) approach to desktop applications. By deploying software such as Citrix XenApp IT organizations are able to centralize the management, administration and delivery of popular Windows-based applications. This model of centralized administration and delivery of services is not new to VMware customers who have for years used virtualization technology to consolidate server workloads.

This guide provides information about deploying Citrix XenApp in a virtualized environment powered by VMware vSphere™. Key considerations are reviewed for architecture, performance, high availability, and design and sizing of virtualized workloads, many of which are based on current customer deployments of XenApp application servers on VMware. This guide is intended to help IT Administrators and Architects successfully deliver virtualized applications using Citrix XenApp on VMware vSphere.

The following topics are covered in detail:

  • VMware ESX host best practices for Citrix XenApp
  • Virtual hardware, guest operating system, and XenApp best practices
  • Monitoring performance
  • vSphere enhancements for deployment and operations

Download the complete best practices guide at the link below:

www.vmware.com/files/pdf/solutions/vmware-citrix-xenapp-best-practices-EN.pdf

-alex

Monitor SQL Server Performance on VMware

Virtualization adds new software layers and new types of interactions between SQL Server and the hardware components. While the general methodology for monitoring and troubleshooting SQL Server performance does not change, VMware provides additional tools for monitoring and troubleshooting at the physical host level. This is the first of a series of Monitoring and Troubleshooting SQL Server performance on VMware blogs. In this blog, I'd like to discuss how to effectively monitor SQL Server performance on VMware.

SQL Server and Windows provide a number of tools for monitoring performance. Those are still the primary tools to use for identifying problem areas, and resource bottlenecks with SQL Server running on VMware. The methodologies stay the same as performance monitoring SQL Server in a physical environment.

  • Performance Monitor: monitors overall resource usages. SQL Server specific counters: \SQLServer:*\*
  • SQL Server Profiler: monitors performance at T-SQL statement level. Common events of interests are: SQL:BatchStarting/SQL:BatchCompleted, RPC:Starting/RPC:Completed, Errors and Warnings (All), P:StmtStarting/Completed, SQL:StmtStarting/Completed
  • DMVs: monitors the internal health of SQL Server, sys.dm_*

For more information on using these tools for monitoring and troubleshooting SQL Server, see Troubleshooting Performance Problem in SQL Server 2008.

VMware provides two main tools for observing and collecting performance data at the host-level: the vSphere Client and the esxtop/resxtop utilities.

vSphere Client:

  • GUI interface, primary tool for observing performance and configuration data for one or more ESX/ESXi hosts
  • Does not require high levels of privilege to access the data

esxtop/resxtop

  • Command line utility that runs in interactive, batch, or replay mode
  • Gives access to detailed performance data of a single ESX/ESXi host
  • Provides fast access to a large number of performance metrics
  • Requires root-level access

     

Figure 1. Performance Chart Views with vSphere Client

Figure 2. resxtop CPU Metrics

 

Among the performance data exposed by host-level monitoring tools, the following is a list of key metrics that can help you quickly isolate issues to a specific resource area such as CPU, memory, storage, or network. See VMware Communities: Interpreting esxtop Statistics and vCenter Performance Counters for a full list of counters.

Note** The measurement units reported in esxtop/resxtop and vSphere Client may be different. See the above VMware community links for details.

Resource

Metric (esxtop/resxtop)

Metric (vSphere Client)

Host/Virtual Machine

Description

CPU

%USED

Used

Both

CPU used over the collection interval (%)

%RDY

Ready

Virtual Machine

CPU time spent in ready state

%SYS

System

Both

Percentage of time spent in the ESX Server VMKernel

Memory

Swapin, Swapout

Swapinrate, Swapoutrate

Both

Memory ESX host swaps in/out from/to disk (per virtual machine, or cumulative over host)

MCTLSZ (MB)

vmmemctl

Both

Amount of memory reclaimed from resource pool by way of ballooning

Disk

READs/s, WRITEs/s

NumberRead, NumberWrite

Both

Reads and Writes issued in the collection interval

DAVG/cmd

deviceLatency

Both

Average latency (ms) of the device (LUN)

KAVG/cmd

KernelLatency

Both

Average latency (ms) in the VMkernel, also known as "queuing time"

GAVG/cmd

TotalLatency

Both

Average latency (ms) in the guest. GAVG = DAVG + KAVG

Network

MbRX/s, MbTX/s

Received, Transimitted

Both

Amount of data transmitted per second

PKTRX/s, PKTTX/s

PacketsRx, PacketsTx

Both

Packets transmitted per second

%DRPRX, %DRPTX

DroppedRx, DroppedTx

Both

Dropped packets per second

Of the CPU counters, the total used time indicates system load. Ready time indicates overloaded CPU resources. A significant swap rate in the memory counters is a clear indication of a shortage of memory, and high device latencies in the storage section point to an overloaded or misconfigured array. Network traffic is not frequently the cause of most database performance problems, but it is an area to monitor to make sure the network bandwidth is there.

When SQL Server is running in a virtual environment, any time-based measurements reported with SQL Server monitoring tools or Windows Perfmon may be inaccurate if the host machine resources are over-committed. The accuracy of the in-guest tools would depend on the guest OS and kernel version being used, and the total load of the VMware ESX host. The vSphere tools provide more accurate reporting on those measurements. See vSphere Performance Troubleshooting for more information. For effective monitoring and troubleshooting of SQL Server performance problem on VMware use the SQL Server tools and Perfmon for monitoring SQL Server resource usage, server state, and internal health. Focus on identifying performance bottlenecks instead of time-base measurements and then further correlate performance data collected from the host using vSphere tools. By focusing on key performance metrics you can quickly isolate issues to a particular resource area.

For example, for identifying a CPU bottleneck, instead of looking at the % Processor Time Perfmon counter, you can focus on the Process Queue Length counter. If there are a high number of processor queues and SQL Server Wait Statistics is showing a high number of waiters for worker, it's obvious a processor bottleneck exists. You can then use vSphere Client to verify:

  1. If the number of virtual processors allocated to the SQL Server virtual machine is sufficient.
  2. If the Shares, reservation, and limits for the virtual machine CPU Resource Allocation are configure properly.
  3. If there is a host overload with sustained high %used, and high %rdy.

Similarly, you can effectively identify a disk bottleneck if the Disk Queue Length is high and a high number of SQL Server users are waiting on PAGEIOLATCH_EX, PAGEIOLATCH_SH. After a resource bottleneck is identified on the SQL Server virtual machine you can correlate statistics collected from the vSphere Client and esxtop/resxtop to identify any host level configuration issue that's affecting the SQL Server virtual machine performance.

Thanks for reading. On my next blog, I will discuss identifying and solving some common SQL Server on VMware performance problems. Stay tuned…

 

-Wanda

Wanda He, Technical Solutions Architect

 

Enterprise Java Applications on vSphere Best Practices

Many of our customers have run critical enterprise Java applications on vSphere successfully.  The best practices guide is often referred to by our customers when embarking on running Java applications on vSphere.

December last year version 2 of the best practices paper was released here: http://www.vmware.com/resources/techresources/1087,

The paper covers quite few best practices and definitely recommended for anyone doing deployments of Java applications on vSphere, however it is worthwhile to note that the top 3 most sought after best practices from this paper are as follows and based on customer interaction:

  • BP4 – VM Memory Sizing
  • BP5 – Setting Memory Reservation
  • BP6 – Using memory large pages

BP4 – VM memory Sizing: Whether you are using Windows or Linux as your guest OS, refer to the technical specification of the various vendors for memory requirements. It is common to see the guest OS allocated about 1GB in addition to the JVM memory size. However, each installation may have additional processes running on it, for example monitoring agents, and you need to accommodate their memory requirements as well.   Figure 2 shows the various segments of JVM and VM memory, and the formula summarizes VM Memory as:

VM Memory (needed) = guest OS memory + JVM Memory,

whereJVM Memory = JVM Max Heap (-Xmx value) + Perm Gen (-XX:MaxPermSize) + NumberOfConcurrentThreads * (-Xss)

The -Xmx value is the value that you found during load testing for your application on physical servers. This value does not need to change when moving to a virtualized environment. Load testing your application when deployed on vSphere will help confirm the best –Xmx value.  It is recommended that you do not overcommit memory because the JVM memory is an active space where objects are constantly being created and garbage collected. Such an active memory space requires its memory to be available all the time. If you overcommit memory ballooning or swapping may occur and impede performance.  ESX host employs two distinct techniques for dynamically expanding or contracting the amount of memory allocated to virtual machines. The first method is known as memory balloon driver (vmmemctl). This is loaded from the VMware Tools package into the guest operating system running in a virtual machine. The second method involves paging from a virtual machine to a server swap file, without any involvement by the guest operating system.

 

JavaHeapSegments 

BP5- Setting Memory Reservation: JVMs running on VMs have an active heap space requirement that must always be present in physical memory. Use the VMware vSphere Client to set the reservation equal to the needed VM memory.

Reservation Memory = VM Memory = guest OS Memory + JVM Memory

You may set this reservation to the active memory being used by the VM for a more efficient use of the amount of memory available. Or, a simpler approach is to set the reservation equal to the total configured memory of the VM.

 

BP6- Use memory large Pages:  Large memory pages help performance by optimizing the use of the Translation Look-aside Buffer (TLB), where virtual to physical address translations are performed. Use large memory pages as supported by your JVM and your guest operating system. The operating system and the JVM must be informed that you want to use large memory pages, as is the case when using large pages in physical systems.

  • Set the -XX:+UseLargePagesat the JVM level for Sun HotSpot.
  • On the IBM JVM it is -Xlp, and JRockit -XXlargePages.
  • You also need to enable this at the guest OS level. For information, see Large Page Performance: ESX Server 3.5 and ESX Server 3i v3.5.

High Availability for Exchange 2010 without DAG

With the release of Exchange 2010 the native clustering feature (Database Availability Group) is a significant improvement over what was available with all previous versions. Be that as it may, there will still be customers that simply don't want to cluster. Typically, not clustering the application would mean no high availability. VMware changed that with the introduction of VMware HA. By simply enabling this feature (which is literally a check box) all virtual machines, regardless of operating system or application, would be provided protection from an ESX host failure. To take it a step further you could enable VM Monitoring to protect against guest OS failures by monitoring VMware Tools running within the virtual machine. Agnostic protection of the guest OS and application is great, but the question that was consistently asked by our customers was "…what if my application fails?" vSphere 4.1 helps answer that.

With vSphere 4.1, we introduced an application programming interface (API) to provide third-party vendors the ability to integrate with VMware HA. The capability to allow application monitoring agents to interact with VMware HA is enabled per vSphere cluster with additional configuration options available per virtual machine. When enabled, this feature allows application monitoring agents to send application heartbeats to VMware HA. In the event of an application-level failure the application monitoring agent can take action to either bring the application back online, or stop the application heartbeat, causing VMware HA to initiate a restart of the virtual machine. Prior to vSphere 4.1, only VMware tools heartbeats could trigger VMware HA restarts.

To take advantage of the new API Symantec has released ApplicationHA. Using ApplicationHA Exchange administrators can provide application monitoring and availability without having to deploy an Exchange 2010 DAG.

ApplicationHA consists of three components:

  • ApplicationHA Console provides the interface between the Guest Component and vCenter Server. This component will be installed on a dedicated server (virtual or physical).
  • Guest Component is installed in the virtual machine where the application to be protected is running.
  • vSphere Client plug-in enables administrators to view the status of a monitored application and make basic configuration changes.

The Symantec Application HA agent for Exchange 2010 can monitor databases and bring them online and offline. In addition to protecting databases, Exchange 2010 services are also monitored. When an agent detects that a database or service has failed that component will be restarted. If the restart fails multiple times (configurable by the administrator) the agent reports the failure to VMware HA. VMware HA can then restart the virtual machine. The following Exchange 2010 services are monitored by the ApplicationHA agent:

  • Microsoft Exchange AD Topology Service
  • Microsoft Exchange Replication Service
  • Microsoft Exchange System Attendant
  • Microsoft Exchange Information Store
  • Microsoft Exchange Mail Submission

After the installation of the Guest Component in the Exchange 2010 mailbox VM the vSphere client can be used to configure application monitoring. The ApplicationHA Configuration Wizard allows the administrator to select the installed application to protect (one application per VM), and for Exchange 2010, databases to monitor.

With the application configured for monitoring administrators can use the vSphere client to display the status of the application, start and stop the application, re-configure monitoring, and enter maintenance mode.

.

The Settings link on the ApplicationHA tab allows the administrator to set additional configuration options.

  • App.RestartAttempts – the number of times that ApplicationHA must try to restart an application before declaring it faulted.
  • App.ShutdownGraceTime – maximum time in seconds that ApplicationHA will wait for an application to gracefully shutdown before declaring it faulted.
  • App.StartStopTimeout – maximum time in seconds that ApplicationHA will wait for an application to start or stop, after which an informational message is displayed stating that the operation is in progress in the background.

Deploying clustering technologies such as Microsoft Clustering Services within virtual machines is a supported configuration. Many customers have chosen not to take this approach for their critical applications due to limitations around using vSphere features such as VMware vMotion, DRS and in some cases HA. Using vSphere 4.1 and Symantec ApplicationHA can provide the needed application awareness in VMware HA while continuing to provide vSphere features in a supported fashion.

For more information about Symantec ApplicationHA including additional application support see the links below.

http://www.symantec.com/business/application-ha

http://sfdoccentral.symantec.com/appha/5.1/windows/pdf/appha_exch_agent.pdf

http://www.symantec.com/content/en/us/enterprise/white_papers/b-virtualizing_business_critical_apps_WP.en-us.pdf

-alex

Alex Fontana, Technical Solutions Architect