Home > Blogs > Virtualize Business Critical Applications
Featured post

Just Published – Virtualizing Active Directory Domain Services On VMware vSphere®

Announcing the latest addition to our series of prescriptive guidance for virtualizing Business Critical Applications on the VMware vSphere platform.

Microsoft Windows Active Directory Domain Services (AD DS) is one of the most pervasive Directory Services platforms in the market today. Because of the importance of AD DS to the operation and availability of other critical services, applications and processes, the stability and availability of AD DS itself is usually very important to most organizations.

Although the “Virtualization First” concept is becoming a widely-accepted operational practice in the enterprise, many IT shops are still reluctant to completely virtualize Domain Controllers. The most conservative organizations have an absolute aversion to domain controller virtualization while the more conservative organizations choose to virtualize a portion of the AD DS environment and retain a portion on physical hardware. Empirical data indicate that the cause of this opposition to domain controller virtualization is a combination of historical artifacts, misinformation, lack of experience in virtualization, or fear of the unknown.

The new Guide – Virtualizing Active Directory Domain Services On VMware vSphere® – is intended to help the reader overcome this inhibition by clearly addressing both the legacy artifacts and the new advancements in Microsoft Windows that help ameliorate the past deficiencies and make AD DS more virtualization-friendly – and safer.

With the release of Windows Server 2012, new features alleviate many of the legitimate concerns that administrators have about virtualizing AD DS. These new features, the latest versions of VMware® vSphere®, and recommended practices help achieve 100 percent virtualization of AD DS.

The Guide includes a number of best practice guidelines for administrators to help them optimally and safely deploy AD DS on VMware vSphere.

The recommendations in this guide are not specific to a particular set of hardware or to the size and scope of a specific AD DS implementation. The examples and considerations in this document provide guidance, but do not represent strict design requirements.

Similar to other “Best Practices” releases from VMware, this Guide is intended to serve as your companion and primary reference guidepost if you have any responsibility planning, designing, implementing and operating a virtualized Active Directory Domain Services instance in a VMware vSphere infrastructure.

This guide assumes a basic knowledge and understanding of vSphere and AD DS.
Architectural staff can use this document to understand the design considerations for deploying a virtualized AD DS environment on vSphere.
Engineers and administrators can use this document as a catalog of technical capabilities.
Management staff and process owners can use this document to help model business processes that take advantage of the savings and operational efficiencies achieved with virtualization.

You can download Virtualizing Active Directory Domain Services On VMware vSphere® here.

Featured post

Which vSphere Operation Impacts Windows VM-Generation ID?

In Windows Server 2012 VM-Generation ID Support in vSphere, we introduced you to VMware’s support for the new Microsoft’s Windows VM-Generation ID features, discussing how they help address some of the challenges facing Active Directory administrators looking to virtualize domain controllers.

One of the common requests from customers in response to the referenced article is a list of events and conditions under which an administrator can expect the VM-Generation ID of a virtual machine to change in a VMware vSphere infrastructure. The table below presents this list. This table will be included in an upcoming Active Directory on VMware vSphere Best Practices Guide.

Scenario VM-Generation ID Change
VMware vSphere vMotion®/VMware vSphere Storage vMotion No
Virtual machine pause/resume No
Virtual machine reboot No
HA restart No
FT failover No
vSphere host reboot No
Import virtual machine Yes
Cold clone Yes
Hot clone
Hot cloning of virtual domain controllers is not supported by either Microsoft or VMware. Do not attempt hot cloning under any circumstances.
New virtual machine from VMware Virtual Disk Development Kit (VMDK) copy Yes
Cold snapshot revert (while powered off or while running and not taking a memory snapshot) Yes
Hot snapshot revert (while powered on with a memory snapshot) Yes
Restore from virtual machine level backup Yes
Virtual machine replication (using both host-based and array-level replication) Yes

If you have a specific operation or task that is not included in the table above, please be sure to ask in the comments section.

Thank you.

The Case for SAP Central Services and VMware Fault Tolerance

What’s the CPU Utilization Of Standalone SAP Central Services in a Virtual Machine?

Since VMware came out with VMware Fault Tolerance (FT) we have considered the deployment option of installing SAP Central Services in a 1 x vCPU virtual machine protected by VMware FT. FT creates a live shadow instance of a virtual machine that is always up-to-date with the primary virtual machine. In the event of a hardware outage, VMware FT automatically triggers failover—ensuring zero downtime and preventing data loss. Central Services is a single-point-of-failure in the SAP architecture that manages transaction locking and messaging across the SAP system and failure of this service results in downtime for the whole system. Hence Central Services is a strong candidate for FT but FT currently only supports 1 x vCPU (vSphere 5.x), so some guidance is required on how many users we can support in this configuration. VMware has given technical previews of multi-vCPU virtual machines protected by FT at VMworld 2013/2014, but now, better late than never, here are the results of a lab test demonstrating the performance of standalone Central Services in a 1 x vCPU virtual machine.

We configured the following setup.


We have two virtual machines: a SAP dialog instance (processes OLTP requests)  and  database instance in one; and  Central Services in the second. The objective was to focus on performance of the standalone Central Services virtual machine that was servicing an ABAP based workload. For the ABAP stack Central Services is officially referred to as ABAP SAP Central Services (ASCS).  The goal of the test was to scale up to 1000 OLTP users running transactions in the SAP Sales and Distribution module.  The following metrics were measured (via vCenter): the maximum CPU usage of the ASCS virtual machine during the workload run; the maximum network usage of the ASCS virtual machine during the workload run. The following graph shows the results.


Summarizing the results:

  • At 1000 users the maximum CPU utilization of the ASCS virtual machine was 29%. This is a comfortable utilization to run in production. The number of SAP lock requests (referred to by SAP as enqueues and dequeues) was over 10000 per minute during one point in the run (eyeballed via SAP transaction SM12).
  • We can see a fairly linear relationship between CPU usage and the number of users.
  • Lock requests generate network traffic between the SAP dialog instance and the ASCS service – we see that the network usage is fairly linear as users are scaled up. Note: there are no lock requests between the SAP database and Central Services.

It is recommended to validate SAP workloads in a pre-production performance test before go-live. Customer tests/workloads will show different results to those shown above as the frequency of SAP lock requests depends on the transaction think times and customer specific business processes (the transaction think times used in this test were < 20 seconds).  Customer environments should also consider batch jobs running at the same time as OLTP activity that by design are generating a lot of locks e.g. mass update of business documents – this would create additional load on Central Services.

OLTP performance on Virtualized SQL 2014 with All Flash Arrays

TPC-C Benchmark is an on-line transaction processing (OLTP). (TPCC Main site) TPC-C uses a mix of five concurrent transactions of different types and complexity. The database is comprised of nine types of tables with a wide range of record and population sizes. TPC-C is measured as transactions per minute (TPM).

The goal of this exercise was to see if 1 million TPM can be achieved on virtualized SQL 2014 backed by an all Flash storage array for a TPC-C like test.  The TPC-C testing would be compared between two VM sizes (Within NUMA & Exceeding NUMA boundaries)

Continue reading

Effect of VAAI on cloning with All Flash Arrays:

Cloning virtual machines is an area where VAAI can provide many advantages. Flash storage arrays provide excellent IO performance. We wanted to see what difference VAAI makes in virtual machine cloning operations for “All Flash Arrays”.

The following components were used for testing VAAI performance on an all Flash storage array:

  1. Dell R910 server with 40 cores and 256 GB RAM
  2. Pure FA-400 Flash Array with two shelves that included 44 238 GB Flash drives and 8.2 TB usable capacity.
  3. Centos Linux Virtual Machine with 4 vCPU, 8 GB RAM,  16 GB OS/Boot Disk & 500 GB Data Disk all on the Pure Storage Array
  4. SW ISCSI on dedicated 10GBPS ports.

Test Virtual Machine:

The virtual machine used for testing was a generic Centos Linux based system with a second virtual data disk with 500GB Capacity.  To make the cloning process be truly exercised, we want this data disk to be filled with random data. Making the data random ensures that the data being copied is not repetitive in any way and is not easily compressed or de-duplicated.

 Preparing the Data Disk:

The following command was used to create a large 460 GB file with random data with “dd” command on Linux.

dd if=/dev/urandom of=/thinprov/500gb_file bs=1M count=4600000

The disk space used in the data disk is shown below and it contains only the random data file generated with dd command.

root@linux01 thinprov]# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/mapper/VolGroup00-LogVol00 10220744   2710700   6982480  28% /

/dev/sda1               101086     20195     75672  22% /boot

tmpfs                         4087224         0   4087224   0% /dev/shm

/dev/sdb1            516054864 469853428  19987376  96% /thinprov

 Tuning for VAAI and best performance:

VAAI can be enabled or disabled using the following settings: (1 enables, 0 Disables)

esxcli system settings advanced set –int-value 1 –o /DataMover/HardwareAcceleratedMove

esxcli system settings advanced set –int-value 1 -o /DataMover/HardwareAcceleratedInit

esxcli system settings advanced set –int-value 1 -o /VMFS3/HardwareAcceleratedLocking

esxcli system settings advanced set –int-value 1 -o /VMFS3/EnableBlockDelete

Adjust Maximum HW Transfer size for better copy performance:

esxcli system settings advanced set –int-value 16384 –option /DataMover/ 

For larger I/O sizes its found in experiments that settings IOPS to 1 have a positive effect on latency

esxcli storage nmp psp roundrobin deviceconfig set –d <device> -I 1 -t iops

On ESXi 5.5, DSNRO can be set on a per LUN basis!

esxcli storage core device set -d <device> -O 256

Set Disk SchedQuantum to maximum (64)

esxcli system settings advanced set –int-value 64 –o /Disk/SchedQuantum

Phase 1: Cloning with VAAI disabled:

For the first phase of the study VAAI was turned off and the settings validated. The cloning process was initiated for the Linux virtual machine and some of the key metrics were observed and captured at the storage array and in vCenter performance charts.

The cloning process was carefully monitored and the time for the cloning operation was observed to be 63 minutes.



The time in the chart between 2:06 and 3:09 PM represents the cloning operation shown as the blue area. There is a spike in latency (>2ms), IOPS (5000) and Bandwidth utilization around 420 MBPS during this cloning operation.


Phase 2: Cloning with VAAI Enabled:

For the second phase of the study VAAI was turned on and the settings validated. The cloning process was initiated for the Linux virtual machine and some of the key metrics were observed and captured at the storage array and in vCenter performance charts.

The cloning process was carefully monitored and the time for the cloning operation was observed to be 19 minutes.



The time in the chart between 3:54 and 4:13 PM represents the cloning operation shown as the blue area. There is a minimal spike in latency (0.5ms), IOPS (3000) and Bandwidth utilization around 10 MBPS during this cloning operation.


The performance chart for network usage does not correlate with the 10 MBPS average utilization during the cloning operation. The network utilization at the vSphere host level during the operation shows no increase in network utilization as was seen with the Non VAAI operation. This clearly shows that all the network activity occurs within the storage array with no impact the vSphere host.

Effect of VAAI on the cloning operation:

The observations highlight the huge impact that VAAI has on a large copy operation represented by a VM clone. A clone of a VM with 500 GB of random data benefits significantly through the use the use VAAI compliant storage as shown in the following table.



Arrays that are VAAI capable such as the Pure Storage array used in this study dramatically improves write intensive operations such as cloning by reducing time of impact, latency, IOPS and bandwidth consumed. This study shows that even all flash arrays that have fast disks with huge IOPS can significantly benefit from VAAI for cloning



Critical Factors to consider when virtualizing Business Critical Applications: (Part 2 of 2)


Availability of the applications is one of the important requirements for business critical applications. The vSphere platform provides capabilities to protect against hardware and software failures  to meet the stringent SLA requirements of business critical applications.

Virtualized applications can avail of features such as vSphere HA protects from HW failures by restarting virtual machines on surviving hosts if a physical host were to fail. vSphere FT  (Fault Tolerance) protects critical servers such as load balancers and other central servers with a small footprint (1vCPU) with zero downtime in the event of HW failures.

vSphere App HA for Application level protection for supported applications. Third party solutions that leverage vSphere Application Awareness API such as Symantec Application HA , NEVERFAIL, etc. layer on top of VMware HA and provide monitoring and availability  for most of the common business critical applications.



Traditional availability clusters like MSCS and Linux based clusters can be layered on top of vSphere HA, but quite often create restrictions relating to vMotion and other functionality in VMware environments. Currently the only use case that these solutions provide not provided by the Application HA solutions available on VMware is the ability to have reduced downtime during patch related activity requiring system reboot. If there is allowance for downtime for patch maintenance using traditional clusters can be avoided.


The ability to proactively manage business critical applications is a very important requirement. Virtualized business critical applications can leverage the vCenter Operations (VCOPS) suites of solutions to effectively manage their environment. VCOPS provides tactical (Right Now) and Strategic (Future Focused) capabilities to monitor performance and manage the capacity of the environment proactively.

VCOPS provides tactical monitoring and strategic planning


vCOPS1VCOPS also provides ability to perform root cause analysis on problems in the environment in a time efficient manner to reduce impact on business critical applications.

VCOPS smart alerts



vCOPS2Changes made to the environment can be correlated to the health, while also helping to diagnose the impact of the change on an application or environment.

Correlate events with changes made to environment



All major x86 applications are supported on VMware and actively being deployed by our customer.



Most application owners and DBAs virtualize Oracle or SAP for cloning. Creating application instances and refresh of data across environments such as production, development, test and QA is very time consuming and resource intensive.

With vSphere, any application can be templatized and easily cloned to be rolled out into these environments with minimal effort.  This agility provides application owners improved time to market of their solutions and increased productivity.

Easy cloning between production and other environments


AgilityDisaster Recovery:

One of the critical requirements for business critical applications is the need for disaster recovery with minimum possible downtime (Recovery Time Objective) and minimum loss of data (Recovery Point Objective).

Traditional disaster recovery for business critical applications with physical servers has many limitations such as the need for identical hardware in recovery site and downtime for production during testing. In a virtual environment, every virtual machine is encapsulated in a series of files that can be easily moved and instantiated in the recovery locations.

VMware Site Recovery Manager (SRM) provides a strong workflow engine that provides automation through centralized recovery plans, automated failover and failback and planned migration capabilities. SRM provides the ability to perform non-disruptive testing and leverages vSphere and Array based replication to replicate the data to the disaster recovery site.

Majority of the customers virtualizing business critical applications leverage SRM for their Disaster Recovery needs. SRM drastically reduces the RPO and RTO and helps them meet stringent business availability requirements.

SRM provides automated disaster recovery capabilities



The critical concerns of business critical application owners are adequately addressed by VMware’s suite of products and solutions. There is no reason not to virtualize your business critical applications. The trends shown below clearly attest to these capabilities.

Strong growth in Adoption of BCA virtualization


BCA_trendsPart 1 of series can be found at “Critical Factors to consider when virtualizing Business Critical Applications: (Part 1 of 2)


Impact of database licensing on Cluster Design for Production SAP virtualization

Type of Database licensing for SAP:

The type of licensing impacts the cluster design for SAP virtualization. SAP is supported on most common database platforms such as SQL, Oracle, DB2 and SYBASE. When customers procure SAP, they can choose to buy the database licensing through SAP or purchase it directly from the database vendor. This decision impacts the cluster design for virtualized SAP environments.

Let us look at these two scenarios and their impact on the design.

Scenario 1: Database License procured from the DB vendor for SAP:

Database vendors have differing but usually very restrictive policies regarding virtual machines running databases. The cost of licensing databases in the extreme case could force a customer to license for the entire cluster, even though  the database could be using only a small subset of the resources. Due to the uncertainty and the risk involved with DB licensing in this situation, it might be prudent to separate the entire database workload into its own cluster. By separating the entire database workload, the physical hardware used for databases can be isolated and licensed fully. Since only database workloads exist in this cluster one can achieve consolidation and efficiency for databases. The main disadvantage is the added overhead of having a separate cluster for databases. Since SAP landscapes have many modules with each module having its own individual database, creating a separate DB cluster with a good number of  hosts is  worthwhile and justified.

Dedicated Database Cluster for SAP

Dedicated Database Cluster for SAP








When there are no restrictions with licensing,  the typical cluster design methodology in vSphere environments espouses having an N+2 cluster.  An N+2 cluster would provide headroom for doing maintenance (One host at a time) and high availability (One host failure). These additional hosts can be costly for the database cluster due to the need to license all hosts.  In this situation the applications run in their own cluster, which typically is N+2.

SAP Applications in their own cluster

Dedicated APP Cluster for SAP










Most database vendors allow for a non licensed DB host, if the only purpose of these hosts is to function as a standby in the event of a failure. There are many conditions such as the number of actual days the standby takes over per year and other requirements that need to be met. vSphere clusters have a setting called dedicated failover host, which can be leveraged in database clusters to match the requirements of standby hosts.  One can potentially meet these conditions for standby node, by running all workloads in normal circumstances on licensed nodes with the dedicated failover node minimally being used only during actual failure or maintenance.

Scenario 2:  Database licensed through SAP along with SAP software:

When databases are licensed through SAP, there is no impact of database placement on licensing. This is akin to “named user” based licensing. There is a lot more flexibility to locate the database servers anywhere in the environment. Customers typically collocate the database servers along with the application servers for proximity and ease of management. The commonly used N+2 clusters can be leveraged in this scenario to allow for HA capacity even during maintenance.  All nodes can be fully utilized to run workloads.

SAP Applications and Databases in the same cluster











Cluster design in SAP environments can be impacted by the type of database licensing. Creating a dedicated database cluster for certain situation, can help meet many of the stringent licensing requirements, while still providing for consolidation and optimized utilization.



Critical Factors to consider when virtualizing Business Critical Applications: (Part 1 of 2)

Over the past few years, there has been significant acceleration in adoption of the VMware platform for virtualization of business critical applications. When vSphere 5 was introduced with its initial support for up to 32 vCPU many of the vertical scalability concerns that existed earlier were addressed. This has been increased to 64 processors with the later vSphere 5.x releases ensuring that more than 99% of all workloads will fit vertically.

Having personally worked in IT infrastructure for more than 20 years with a strong focus on implementing and managing business critical applications, I see a general reluctance from application owners to virtualize business critical applications. When virtualizing business applications there are many critical factors one should consider.  I seek to address the typical concerns of application owners about Virtualization with this multipart series on Virtualizing BCA.

CIOs and IT operations want to virtualize more because of the following advantages of virtualization:

  1. Infrastructure efficiency
  2. Simpler management
  3. Built-in availability
  4. Greater agility
  5. Simplified DR

But Application owners are usually reluctant because:

  1. Will my Application perform well? (Performance)
  2. Will it Scale? (Scalability)
  3. Can I meet my application SLAs? (Availability )
  4. Can I manage it effectively? (Manageability)
  5. Will my ISV support me? (Supportability)
  6. What’s in it for me? Will my application run better? (Agility & Time to Market)


Virtual performance is about 5-6% of physical for most business critical application workloads. The benefits of virtualization and productivity improvements overshadow the small overhead it introduces.

SAP Performance within 6% of Native:

SAP Performance on vSphere vs Physical


BCA1Exchange Server Performance:

Exchange Virtualization study discussed in Running Microsoft Apps on FlexPod for VMware  shows only a 5% performance difference between virtual and physical.

Exchange performance on vSphere



Database Performance:

SQL and Oracle performance close to Native as seen in the test results below.

SQL Throughput on vSphere vs Native

Oracle RAC performance vs Native on vSphere


It is proven that the performance of the common business critical applications are very close to their physical counterparts and performance should never be a concern for virtualizing them.  During virtualization there is usually a hardware refresh to the latest and greatest hardware with superior performance to existing systems. The small overhead of virtualization is easily offset while moving to this newer hardware as part of the virtualization migration.


With the vSphere 5.x platform, almost all workloads are amenable to virtualization.  VMware capabilities, such as vNUMA allows for NUMA aware operating systems and applications to run optimally even in virtual environments.

Scalability of vSphere


BCA5Workloads can scale up dynamically as demand increases with hot add capabilities for CPU, memory and storage resources as and when needed.  One can scale up or scale out based on the application requirements.

“Scale up” versus “Scale out”


These hot add capabilities are available only in virtualized environments and help right size environments and grow them dynamically, when needed without user downtime and loss of productivity. The following graphic shows the effect of increasing CPU on an Oracle DB server by 2 vCPU demonstrating the dynamic scale up capabilities of the vSphere platform.

Oracle Hot Add CPU Illustration



Part 2 of series can be found at  ”Critical Factors to consider when virtualizing Business Critical Applications: (Part 2 of 2)



SAP on VMware Sizing & Design Example

Recently in partner workshops I have come across some interesting discussions about the impact of hyper-threading and NUMA in sizing business critical applications on VMware. So here is an SAP example based on SAP’s sizing metric “SAPS” (a hardware-independent unit of measurement that equates to SAP OLTP throughput of Sales and Distribution users).  The examples here refer to vSphere scheduling concepts in this useful whitepaper The CPU Scheduler in VMware vSphere 5.1 .

SAP sizing requires the SAPS rating of the hardware which for estimation purposes can be obtained from certified SAP benchmarks published at http://www.sap.com/solutions/benchmark/sd2tier.epx . Let’s use certification 2011027  and assume that we plan to deploy on similar hardware as used in this benchmark. This is a virtual benchmark on vSphere 5 with the following result: 25120 SAPS (at ~100% CPU) for 24 vCPUs running on a server with 2 processors, 6 cores per processor and 24 logical CPUs as hyper-threading was enabled. This is a NUMA system where each processor is referred to as a NUMA node.  (Note cert 2011027 is an older benchmark, the SAPS values for vSphere on newer servers with faster processors would be different/higher, hence work with the server vendors to utilize the most recent and accurate SAPS ratings).

In this example I will design for application server virtual machines which as they scale out horizontally gives us the flexibility of choosing the number of vCPUs per virtual machine.  Now do we go with # of vCPUs = # of cores or # of vCPUs = number of logical CPUs? Let’s show an example for both. I will consider the following:

  • SAP sizing is typically conducted at 60-70% CPU and normal practice is to scale down the benchmark SAPS results, I will not bother with this and go with the 25120 SAPS at 100% CPU.
  • Size within the NUMA boundaries. In this two processor NUMA system example, there are two NUMA nodes each with one processor and memory. The access to memory in the same node is local; the access to the other node is remote. The remote access takes more cycles because it involves a multi-hop operation so keeping the memory access local improves performance.
  • For a 6 core NUMA node the virtual machine vCPU size should be a multiple divisor (or multiple) of 6 giving us 1, 2, 3 or 6 way VMs (see this VMware blog).
  • I assume workloads in all the virtual machines peak at the same time.

Let’s first show a design with # of vCPUs = # of cores i.e. no vCPU over-commit.

Example 1: # of vCPUs = # of cores, 2-way and 6-way app servers


With all the virtual machines under load simultaneously, the ESXi scheduler by default, with no specific tuning will: allocate a home NUMA node for the memory of each virtual machine; schedule vCPUs of each virtual machine on its home node thus maintaining local memory access; schedules each vCPU on a dedicated core to allow exclusive access to core resources. (Note that in physical environments such NUMA optimizations would require OS commands to localize the processing e.g. Linux command “numactl”) However the above configuration does not give us 25120 SAPS as not all the logical CPUs are being utilized as was the case in the benchmark. The hyper-threading performance boost for an SAP OLTP workload is about 24% (based on tests by VMware performance engineering – see this blog) so for # of vCPUs = # of cores we should theoretically drive about 25120/1.24 = 20258 SAPS. Also we can estimate about 20258/12 = 1688 SAPS per vCPU so the 2-way virtual machine is rated at 1688 x 2 = 3376 SAPS and the 6-way is 1688 x 6 = 10128 SAPS (@100% CPU in this example).  Are we “wasting” SAPS by not utilizing all the logical CPUs?  Technically yes but for practical purposes not a major issue because:

  • We have some CPU headroom which can be claimed back later after go-live when the virtual machines can be rebalanced based on the actual workload.  At this point vCPU over-commit may be possible as virtual machine workloads may not peak at the same time.
  • Hyper-threading benefit is dependent on the specific workload and while the 24% hyper-threading boost is based on an OLTP workload profile, the actual workload may be less impacted by hyper-threading for example:
    • CPU intensive online reporting
    • CPU intensive custom programs
    • CPU intensive batch jobs
    • SAP has created another metric in their sizing methodology referred to as SCU – Single Computing Unit of performance.  SAP has categorized different workloads/modules based on their ability to take advantage of hyper-threading. So some workloads may experience a hyper-threading benefit lower than 24%.

Now what if we need to drive the maximum possible SAPS from a single server – this is when we would need to configure # of vCPUs = # of logical CPUs. The following configuration can achieve the maximum possible performance.

Example 2: # of vCPUs = # of logical CPUs



In the above design the virtual machine level parameter “numa.vcpu.preferHT” needs to be set to true to override default ESXi scheduling behavior. Default behavior is where ESXi schedules the virtual machine across NUMA nodes when the number of vCPUs for a single virtual machine is greater than the number of cores in the NUMA node.  This results in vCPUs of a virtual machine being scheduled on a remote node relative to its memory location. This is avoided in the above example and performance is maximized because:  ESXi schedules all vCPUs of each virtual machine on the same NUMA node that contains the memory of the virtual machine thus avoiding the penalty of any remote memory access; all logical CPUs are being used thus leveraging the hyper-threading benefit (note vCPUs are sharing core resources so the SAPS per vCPU in this case is 25120/24 = 1047 at 100% CPU). This configuration is commonly used in the following situations: running a benchmark to achieve as much performance as possible (as was done for the app server virtual machines in the 3-tier vSphere SAP benchmark certification 2011044); conducting physical versus virtual performance comparisons. For practical purposes designing for # of vCPUs = # of logical CPUs may not be so critical. If we were to design for a 12-way app server (example 2 above), and actual workload was less than planned with lower CPU utilization, we would have plenty of vCPUs without the added gain from hyper-threading.  There are no hard rules so if desired, during the sizing phase, you can start with # vCPUs = # of cores or number of vCPUs = # of threads based on which approach you think best fits your needs.

Summarizing I have shown two sizing examples for SAP application server virtual machines in a hyper-threaded environment. In both cases sizing virtual machines within NUMA nodes helps with performance.  The SAPS values shown here are based on a specific older benchmark certification and would be different for modern day servers and more recent benchmarks.

Finally a thank-you to Todd Muirhead (VMware performance engineering), for his reviews and inputs.

Setting Multi Writer Flag for Oracle RAC on vSphere without any downtime

Oracle RAC Cluster using ASM for storage would require the shared disks to be accessed by all nodes of the RAC cluster.

The multi-writer option in vSphere allows the VMFS-backed disks to be shared by multiple VM’s simultaneously. By default, the simultaneous multi-writer “protection” is enabled for all .vmdk files ie all VM’s have exclusive access to their .vmdk files. So in order for all of the Oracle RAC VM’s to access the shared vmdk’s , the multi-writer protection needs to be disabled.

KB Article 1034165 provides more details on how to set the multi-writer option manually to allow VM’s to share vmdk’s (link below).

The above method would require the VM’s to be powered off before the multi-writer option can be set in the .vmx configuration file. This means that the Oracle RAC instance would have to be shutdown and the VM completely powered off before the option can be set leading to an Instance outage.

Oracle DBA’s would rather a method where they could add shared disks on the fly to a running Oracle RAC cluster without any downtime.

A colleague and I working for VMware PSO some time back (now I have moved under the Global Center Of Excellence group ) wrote a POWERCli script called the “addSharedDisk” which adds a new shared disk to the existing Oracle RAC nodes along with setting the multi-writer flag setting , setting the disks as independent-persistent and eager-zero thick formatting.

This script can be used to add shared disk on the fly while the Oracle RAC Clusters is up and running thus avoiding any outage.

- This is an generic script provided and can be customized as per your requirement
- This script is provided for educational purposes only , please use at your own risk
- No Support will be provided for this script
- All values provided in this script is for example only, you would need to provide real values

This script accepts at the command prompt
- The number of shared disks to be created( maximum 55)
- The number of nodes that you need this shared disk be added to

-the script places all shared VMDK files in the same datastore as the boot disk VMDK files. This can be customized to place shared vmdk’s on different datastores for IO load balancing
-the script can be further customized to place vmdk’s on different SCSI controllers.
-the script has the logic to check the network requirements of the Oracle RAC interconnect adapters(assumption of the script is 2 Private Adapters are deployed per VM) . (can be removed or commented out if not required)

The script can be found in the link below [https://communities.vmware.com/docs/DOC-24873]