Home > Blogs > Virtualize Business Critical Applications > Monthly Archives: December 2011

Monthly Archives: December 2011

Protecting Virtual SAP Landscapes with VMware vShield App – Part 2 of 2

Example Firewall Configuration of an SAP vApp with vShield App

In the previous article, Part 1, we described the use cases for vShield App in a virtual  SAP environment. Here we demonstrate a simple vShield App  firewall configuration that quickly isolates a virtualized SAP landscape defined as a vApp in vCenter. It assumes the vShield Manager and App Appliances and vShield plug-in have been installed. The vShield App plug-in for vCenter allows you to create firewall rules via the vSphere client .

The following diagram shows the SAP vApp (called “SAP_MSSQL”) which consists of the following two virtual machines: SQL Server database and SAP Central Instance; application server instance. The only access into the SAP landscape will be via the SAP presentation client (SAP GUI) which in this example will use port 3200 (for background on SAP TCP/IP ports go to http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/4e515a43-0e01-0010-2da1-9bcc452c280b) . In this simple scenario we are not isolating the application from the database server.


This vApp is configured in the vCenter client and appears as follows in the vCenter “Hosts and Clusters”  inventory view .



The next three diagrams show vCenter screen captures that describe the three firewall configuration steps for this example:

1) Block all access into the landscape or vApp

2) Define the port for SAP GUI access

3) Allow SAP GUI access






The following whitepaper is available for further reading, “VMware vShield™ App: Protecting Virtual SAP Deployments” at http://www.vmware.com/files/pdf/techpaper/VMware-vShield-App-Protecting-Virtual-SAP-Deployments.pdf .

Sizing Virtual Machines for JVM workloads – Part 3

In today’s blog we would like to announce the that a new book, Enterprise Java Applications Architecture on VMware, has been released and is available from https://www.createspace.com/3632131.


Book Abstract

This book is the culmination of seven years of experience in running Java on VMware vSphere, both at VMware and at many VMware customer sites. In fact, many VMware customers run business critical enterprise Java applications on VMware vSphere and have achieved better TCO and SLAs. This book covers high level architecture and implementation details such as design and sizing, high-availability designs, automation of deployments, best practices, tuning, and troubleshooting techniques for enterprise Java applications on VMware.

Table of Contents

Chapter 1: Introduction
Chapter 2: Why Virtualize Enterprise Java Applications?
Chapter 3: Enterprise Java Applications on VMware
Chapter 4: Design and Sizing of Enterprise Java on VMware
Chapter 5: High-availability Designs of Enterprise Java
Chapter 6: Enterprise Java on VMware Best Practices
Chapter 7: UNIX-to-Linux Migration Considerations
Chapter 8: Run Effectively in Production
Chapter 9: Performance Study
Chapter 10: Application Modernization and vFabric
Chapter 11: Troubleshooting Primer
Chapter 12: FAQ – Enterprise Java Applications on vSphere

Target Audience

This book is targeted at IT professionals who are in search of implementation guidelines for running enterprise Java applications on VMware vSphere in production and QA/Test environments. The first three chapters provide CIOs, VPs, directors, and enterprise architects with key high-level business propositions for virtualizing enterprise Java applications. The remaining chapters are for developers and administrators who are interested in implementation details.

About the Author

Emad Benjamin has worked in the IT industry for the past twenty years. He graduated with a Bachelor of Electrical Engineering from the University of Wollongong. Earlier in his career, he was a C++ software engineer, then in 1997, he switched to programming with Java and has been focusing on Java ever since. He has extensive software development experience with companies such as, Cisco, Oracle, Citadel, BHP Steel, and others. For the past seven years, his main focus has been Java on VMware vSphere. Emad is currently at VMware focusing on all aspects of Java virtualization. He has shared his experience and Java virtualization best practices at many conferences and workshops around the world. You can connect with Emad on LinkedIn.




Sizing Virtual Machines for JVM Workloads – Part 2

Looking at a Sizing Example

Figure 4, shows the most commonly encountered JVM and virtual machine size. This may be a fairly busy JVM with 100 to 250 concurrent threads (actual thread count varies as it depends on the nature of the workload), 4GB of heap, approximately 4.5GB for the JVM process, and 0.5GB for the guest OS. This results in a total recommended memory reservation for the virtual machine of 5GB with 2 vCPU and 1 JVM process.

Figure 4. Most Commonly Encountered Configuration


Figure 5 takes a closer look at the sizing example within the Java process, the memory layout, and for various sizes..

Figure 5. 5GB RAM Virtual Machine with One JVM Process and Two CPUs


The general sizing equation can be summarized as follows:


Let’s assume that, through load testing, a JVM max heap (-Xmx) of 4096m has been determined as necessary. Proceed to size as follows:

  • Set -Xmx4096m and set –Xms4096m.
  • Set –XX:MaxPermSize=256m. This value is a common number and depends on the memory footprint of the class-level information within your Java application code base..
  • The other segment of NumberOfConcurrentThreads*(-Xss) depends mostly on the NumberOfConcurrentThreads the JVM will process, and the –Xss value you have chosen. A common range of –Xss is 128k-192k. If, for example, NumberOfConcurrentThreads is 100, then 100*192k => 19200k (assuming you set –Xss to 192k).

Note: The stack -Xss is application dependent, i.e. if the stack is not sized correctly you will get a StackOverflow. The default value is sometimes quite large, but you can size it down to help save on memory consumption.

  • Assume the OS has a requirement of about 500m to run as per the OS spec.
  • Total JVM memory (java process memory) = 4096m (-Xmx) + 256m (–XX:MaxPermSize) + 100*192k (NumberOfConcurrentThreads*-Xss) + “other mem”.
  • Therefore, JVM memory is approximately 4096m+256m+19.2m + “other mem” = 4371m + “other mem”.
  • Typically “other mem” is not significant. However, it can be quite large if the application uses lots of NIO buffers and socket buffers. Otherwise, assuming about 5 percent of the total JVM process memory (i.e., 4 to 5 percent of 4371=> assume 217m), should be enough, although proper load testing should be used to verify.
  • This implies that JVM process memory is 4371m+217m=4588m
  • To determine the virtual machine memory, assume you are using Linux with only this single Java process and no other significant process running. The total configured memory for the virtual machine translates to: VM memory = 4588m + 500m = 5088m.
  • Finally, you should set the virtual machine memory as the memory reservation. You can choose to set the memory reservation as 5088m. However, over time you should monitor the active memory used by the virtual machine that houses this JVM process and adjust the memory reservation to that active memory value, which could be less than 5088m.

Stay tuned. In Part 3 we will conclude with an overview of the recently released book, Enterprise Java Applications Architecture on VMware.





Protecting Virtual SAP Landscapes with VMware vShield App – Part 1 of 2

VMware vShield App is a hypervisor-level firewall that can be used in virtual SAP deployments to provide network isolation and facilitate quick firewall testing of SAP landscapes. SAP landscape consists of multiple SAP products (ERP, CRM, portal etc) with separate production and non-production environments.  The non-production systems can be further broken into categories such as development, sandbox, user acceptance testing, training, etc. Each landscape can consist of multiple virtual machines running different SAP databases and multiple application servers.  vShield allows these landscapes to be segregated into separate trust zones based on customer requirements.  The following diagram depicts what the security zone architecture could look like in a non production landscape example.


You can define security rules based on containers that are VMware vSphere® objects.  One such object is a vApp.  A vApp is a resource container for multiple virtual machines that work together as part of a multitier application. In the figure above, each zone can be configured as a vApp comprising all the associated virtual machines for a particular SAP landscape.  For example, a rule that denies any traffic from inside a vApp to a specific destination outside that vApp applies to all the virtual machines in that vApp.  Isolating landscapes addresses use cases like: end users with access to training systems can be protected from accessing the development and QA systems; new developers and project consultants who are coming up to speed in sandbox systems can be isolated from core development environments.

SAP administrators can benefit from vShield App in the following ways:

  • Organizing SAP landscapes into vApps and setting policies against these vApps allows administrators to focus on the landscape architecture and remove themselves from details such as IP addresses.
  • For SAP, an IP address change does not typically involve any change to the application configuration files—the same can be said of the firewall rules. Once firewall policies have been set against the vApp, creation of a new database and/or application server virtual machine inside of the vApp requires no extra firewall configuration. This is useful as SAP environments can be very dynamic, for example, creation of sandbox and training systems for new project personnel and end users.
  • Database server virtual machines can be isolated from application server virtual machines. While SAP developers mostly work within the SAP development environment, they may need access to the guest OS of the application server to work with interface files (to manage data that is transferred between separate SAP systems or between SAP and non-SAP apps). So developers may be granted access to the application server guest OS but firewall rules can block access to the guest OS of database server virtual machines which is typically restricted to the database and SAP Basis administrator.
  • vShield firewall policies apply to all ESXi hosts in the cluster so they are still enforced when a virtual machine is live migrated between hosts or a to a new host in the cluster (assuming that all hosts are protected by running the vShield App appliance).

In the second part of this blog series  we will demonstrate how a vApp consisting of a 3-tier SAP landscape can be protected using vShield App.


Sizing Virtual Machines for JVM Workloads

This week we will publish three blogs on the topic of sizing virtual machines for Java workloads. In the bogs we will discuss various sizing considerations, best practices, sizing limits, and the most common configuration used by our customers.

A sizing exercise in a virtualized environment for Java workloads is similar to that in physical environment. The only difference is that virtualized environment provides more flexibility, such as the ability to easily change the compute resource configuration. For more detailed information, we encourage you to review the Enterprise Java Applications on VMware – Best Practices Guide  (http://www.vmware.com/resources/techresources/1087)

Sizing Virtual Machines for JVM workloads – Part 1

Before delving into various sizing considerations we’ll provide some background information about the practical sizing limits of JVMs.

Background: JVM Practical Sizing Limits

Figure 1 illustrates the theoretical and practical sizing limits of Java workloads. These are critical limits that you need to be aware of when sizing JVM workloads.

Figure 1. Theoretical and Practical Sizing Limits of JVMs



The JVM theoretical limit is 16 exabytes, but there is no practical system that can provide this amount of memory, so we present this as the first theoretical limit.

  • The second limit is the amount of memory a guest OS can support. In most practical cases, this is several terabytes and depends on the operating system used.
  • The third limit is the ESXi 5 1TB RAM per virtual machine limit, which is ample for any workload that we have encountered.
  • The fourth limit (really, the first practical limit) is the amount of RAM that is cost-effective on typical ESXi hosts.
  • The fifth limit is the total amount of RAM across the server, and how this is divided into the number of NUMA nodes, where each processer socket will have one NUMA node worth of NUMA-local memory. The NUMA-local memory can be calculated as the total amount of RAM within the server divided by the number of processor sockets. We know that for optimal performance you should always size a virtual machine within the NUMA node memory boundaries. ESXi has many NUMA optimizations that come into play, but even so, it is best to stay NUMA local.

For example, if the ESX host has 256GB of RAM across two processor sockets, it has 2 NUMA nodes with 128GB (256GB/2) of RAM across each NUMA node. This implies that when you are sizing a virtual machine, it should not exceed the 128GB limit in order for it to be NUMA local.

The limits outlined above can help drive your design and sizing decision as to how practical and feasible it is to size large JVMs. However, there are other considerations that come with sizing very large JVMs such as GC tuning complexity and the knowledge needed to maintain large JVMs. In fact, most commonly sized JVMs within the VMware customer base are around 4GB of RAM for a typical enterprise Web application. On the other hand, larger JVMs exist, and we have customers that run large scale monitoring systems and large distributed data platforms on JVMs ranging from 4GB to 128GB.

With large JVMs comes the need to better understand GC tuning. VMware has helped many customers with their GC tuning activities, even though GC tuning on physical is no different than on virtual. The reason is that VMware has uniquely integrated vFabric Java and vSphere expertise into one spectrum, which has helped our customers optimally run many Java workloads on vSphere. When faced with the question of whether to vertically scale the size of the JVM and virtual machine, first consider a horizontal scale out approach. VMware has consistently found that our customers get better scalability with the horizontal scale out approach.

Furthermore, when sizing, it is helpful to categorize the size of the JVMs and virtual machines based on the Java workload types, as shown in Figure 2. 

Figure 2. Common JVM Sizes and Workload Categories


We usually find that the reason customers vertically scale a JVM is due to perceived simplicity of deployment and leaving existing JVM processes intact. Be aware of workload-related choices.

  • For example, a customer initially deploys one JVM process and as demand increases for more applications to be deployed, instead of horizontally scaling out by creating a second JVM and virtual machine, a vertical scale up approach is taken. As a consequence, the existing JVM is forced to vertically scale and carry many different types of workloads with varied requirements.
  • Keep in mind that some workloads, such as, a job scheduler, have a need for high throughput, while a public facing Web application demands fast response time. Stacking these types of applications on top of each other within one JVM complicates the GC cycle tuning opportunity. When tuning GC for higher throughput it is usually at the cost of decreased response time, and vice-versa.
  • You can achieve both higher throughput and better response time with GC tuning, but it unnecessarily extends the GC tuning activity. When faced with this deployment choice it is always best to split out different types of Java workloads into their own JVMs. One approach is to run the job scheduler type of workload in its own JVM and virtual machine, and the Web-based Java application on its own JVM and virtual machine.

In Figure 3, JVM-1 is deployed on a virtual machine that has mixed application workload types, which complicates GC tuning and scalability when attempting to scale up this application mix in JVM-2. A better approach is to split the Web application into JVM-3 and the job scheduler application into JVM-4 (that is, scaled out horizontally with the flexibility to vertically scale if needed). If you compare the vertical scalability of JVM-3 and JVM-4 versus vertical scalability of JVM-2 you will find JVM-3 and JVM-4 always scale better and are easier to tune. 

Figure 3. Splitting Workload Types to Improve Scalability


In Part 2 we will look at an actual sizing example with some practical numbers that you can directly apply. 




Oracle Databases on VMware vSphere – Part 3

In Part 2, we discussed on the deployment of Oracle 11gr2 RAC databases on VMware vSphere. In Part 3, let us discuss on how to live migrate (VMware vMotion) Oracle 11gr2 RAC node virtual machine during user workload.

Oracle 11gr2 RAC Node VM vMotion

Oracle DBAs who have virtualized Oracle 11gr2 RAC on VMware vSphere using VMFS can vMotion RAC Node VM from one physical server to another when the physical server requires maintenance or firmware upgrade (In the physical RAC the entire node will be down during the maintenance window). See below the 4 node Oracle 11gr2 RAC Node VM vMotion configuration and testing steps.

ESX Host


# of vCPUs

Memory (GB)

Oracle SGA (GB)

ESX1 – 12 Core, 192GB RAM





ESX2 – 12 Core, 192GB RAM





ESX3 – 12 Core, 192GB RAM





ESX4 – 12 Core, 192GB RAM





All four nodes are running with their respective ESX hosts. The SwingBench large scale order entry benchmark was run against all four nodes with 400 users logged in. CPU utilization on each node ~ 40-45% , ESX2 must be taken down for hardware maintenance (such as a firmware upgrade).

  • Migrate the Oracle RAC node VMORARAC2 from ESX2 ( to ESX4 ( so that ESX2 can be taken down for a firmware upgrade.

  • After the Hardware maintenance is completed on ESX2, move VMORARAC2 from ESX4 ( back to ESX2 (

The following are the observations during the vMotion testing:

  • CPU usage on all four ESX hosts was approximately 40-41%.
  • After migration of VMORARAC2 (RAC node 2) from ESX2 to ESX4, ESX4 had a CPU usage of 80-81%. CPU of this host was not saturated which shows that it was able to handle the loads of both RAC nodes.
  • Transactions per minute (TPM) are similar before and after vMotion migration.
  • Oracle database logs and cluster logs did not have errors during vMotion migration.

The following vCenter chart shows the CPU utilization of both the ESX2 ( and ESX4 ( host during vMotion test. CPU utilization for ESX4 is approximately 80-81%, which is the sum of the two Oracle RAC nodes (VMORARAC2 – approximately 40% and VMORARAC4 – approximately 41%).

Check out the Oracle 11gr2 RAC vMotion Demo by clicking below.

This concludes our three blogs for Oracle Databases on VMware. We welcome your comments and thanks for your time.

Oracle Databases on VMware vSphere – Part 2

In Part 1, we discussed on the best practices for deploying Oracle databases on VMware. In Part 2, let us discuss on how to deploy Oracle 11gr2 RAC on VMware vSphere using VMFS.

Oracle 11gr2 RAC Deployment on VMware vSphere using VMFS

Oracle DBAs planning to virtualize Oracle RAC on the VMware platform should work with a VMware and storage administrator to successfully deploy Oracle 11gr2 RAC on VMware vSphere using VMFS.

Most of the aspects of the virtualized installation are the same as with a physical RAC installation – See below deployment process diagram where the green colored steps represents VMware specific tasks.

Once the virtual machines are created and configured as shown in the below Oracle RAC Node 1 VM Properties (Clone from the first VM for multiple nodes), install Oracle 11gr2 Grid Infrastructure and RAC on the RAC VMs as you install on physical servers.

For more details on deployment guidelines for Oracle 11gr2 RAC on VMware vSphere click on the image below

Recommended Reading:

  • EMC IT's Virtual Oracle Deployment Framework – http://goo.gl/9mbLX EMC IT is one the largest Oracle deployment which is running on VMware vSphere
  • NetApp's Oracle Database 11g Release 2 Performance Using Data ONTAP 8.1 Operating in Cluster-Mode (4- Node Oracle 11gr2 RAC on vSphere compared with Bare Metal) – http://goo.gl/1AlSR
  • Read the blog post of Todd Muirhead, Oracle RAC Performance on vSphere 4.1: http://goo.gl/5tRNa
  • Read the blog post of Bob Goldsand –

In Part 3 we will look into how to live migrate (VMware vMotion) Oracle 11gr RAC Database VM during user workload.

Oracle Databases on VMware vSphere

This week we will publish three blogs on the topic of Oracle Databases on VMware. In this series, we will discuss on the following:

Part 1: Best practices for deploying Oracle Databases on vSphere,

Part 2: Oracle 11gr2 RAC deployment on VMware vSphere using VMFS and

Part 3: Oracle 11gr2 RAC Node VM vMotion – not possible in physical environment!

Oracle Databases on VMware Best Practices – Part 1

The successful deployment of Oracle databases on VMware vSphere is NOT significantly different from deploying Oracle databases on physical servers. Oracle DBAs who designs, implements, tests, operates, and maintains databases for an organization can fully leverage their current skill set while also delivering the benefits associated with virtualization. The following diagram depicts the general tasks for DBAs.

Proper configuration is critical for the optimal performance and operation of Oracle Databases. Below are some of the most common recommended best practices when deploying Oracle databases on VMware vSphere.

  • Create a computing environment optimized for vSphere.
  • Create golden images of optimized operating systems using vSphere cloning technologies.
  • Upgrade to vSphere ESX 4 or ESXi 5 – Follow vSphere Upgrade Best Practices
  • Allow vSphere to choose the best virtual machine monitor based on the CPU and guest operating system combination.
  • Use as few virtual CPUs (vCPUs) as possible until you know the workload.
  • Enable hyper-threading for Intel Core i7 processors.
  • For IP-based storage iSCSI and NFS, enable jumbo frames.
  • Create dedicated datastores to service database workloads.
  • Use vSphere VMFS for Oracle database deployments and make sure VMFS is properly aligned.
  • Use Oracle Automatic Storage Management (ASM).
  • Use your storage vendor's best practices documentation when laying out the Oracle database.
  • Avoid silos when designing the storage architecture.
  • Use paravirtualized SCSI adapters for Oracle datafiles with demanding workloads.
  • Use the VMXNET Family of paravirtualized network adapters.
  • Separate infrastructure traffic from VM traffic for security and isolation.
  • Use NIC teaming for availability and load balancing.
  • Take advantage of Net I/O Control to converge network and storage traffic onto 10GE.
  • Use VMware vCenter and/or the esxtop/resxtop utility for performance monitoring in the virtual environment.

In order to deploy Oracle Databases on VMware and meet performance goals, DBA's need to follow the Oracle Databases on VMware Best Practices Guide – This guide provides the recommended best practices for vSphere, Storage, Network and Performance monitoring, access the guide by clicking on the image below.

Also check out Michael Webster's blog post on Deploying Enterprise Oracle Databases on vSphere : http://goo.gl/wUafD

In Part 2 we will look into how to deploy Oracle 11gr2 RAC Databases on VMware vSphere using VMFS.