Ten Reasons Why Oracle Databases Run Best on VMware

We’re really excited about the buzz around Oracle in virtualized environments. One of the best kept secrets is just how well Oracle performs on VMware ESX. This didn’t happen by accident – there are a number of features and performance optimizations in the VMware ESX server architecture, specifically for databases.

In this blog, I’ll walk through the top ten most important features for getting the best database performance. Here are a few of the performance highlights:

  • Near Native Performance: Oracle databases run at performance similar to that of a physical system
  • Extreme Database I/O Scalability: VMware ESX Server’s thin hypervisor layer can drive over 63,000 database I/Os per second (fifty times the requirement of a typical database)
  • Multi-core Scaling: Scale up using SMP virtual machines and multiple database instances
  • Large Memory : Scalable memory – 64GB per database, 256GB per host

We’ve continued to invest a great deal of work towards optimizing Oracle performance on VMware, because it’s already one of the most commonly virtualized applications. The imminent ESX 3.5 release is our best database platform to date, with several new advanced optimizations.

In this blog article we’d like to explain the unique and demanding nature of database applications such as Oracle produces and show the performance capabilities of ESX Server on this type of workload.

The Nature of Databases

Databases have some unique properties, such as a-large memory footprint. At the outset this can make them slightly more complex to virtualize well. However this has proven to be an opportunity, since we can optimize specifically for these defining properties.

  • Large Memory: Databases use large amounts of memory to cache their storage. A large cache is one of the most important performance criteria for databases, since it can often reduce physical I/O by 10-100 fold.
  • High Performance Block I/O: Databases read and write their data in fixed, block sized chunks. The I/Os are typically small, and operate at a very high rate on a small number of files or devices.
  • Throughput Oriented: Databases often have a large number of concurrent users, giving them natural parallelism and makes them ideally suited to take advantage of systems with multiple logical or physical processors.

Understanding and Quantifying Virtual Performance

The performance of a virtualized system should first be quantified in terms of latency and throughput, and then in terms of how efficiently resources are being used. For example, if a physical system is delivering 10000 transactions per minute at 500ms latency per transaction, then a virtualized system that is performing at 100% of native should provide the same level of throughput with acceptable latency characteristics. Secondary should be a metric of resource usage, which is a measure of how many additional physical resources were used to achieve the same level of performance. It’s sometimes overly easy to focus primarily on the CPU resource, when in reality memory and I/O are much more expensive resources to provision. This is becoming especially important going forward, as multicore CPUs continue to lower the cost per processor core, while memory cost remains at a premium.

More important for Oracle is the ability to scale up by taking advantage of multi-core CPUs, large memories, and the I/O throughput through the hypervisor to support the large number of disk spindles in the backend storage arrays.

Database Performance Myths

There are a few common myths about virtualizing databases:

  • Databases have a high overhead when virtualized: Virtualized Databases can perform at or near the speed of physical systems, in terms of latency and throughput. The virtualization overhead for typical real-world databases is minimal – for VMware ESX Server, we measured CPU overhead to be less than 10%.
  • Databases have too much I/O to be virtualized: Databases typically have a large number of small random  I/Os, and it is in theory possible to hit a scaling ceiling in the hypervisor layer. VMware ESX’s thin hypervisor layer can drive over 63,000 database I/Os per second, which is equivalent to more than 600 disk spindles of I/O  throughput. This is sufficient I/O scaling for even the largest databases on x86 systems.
  • Virtualization should only be used for smaller, non-critical applications: The ESX hypervisor is very robust: many customers are seeing over two years of uptime from ESX based systems. In addition, the ESX hypervisor remains stable, even if resources are overcomitted.

There isn’t one quick hit to make databases work well for a wide range of real-world applications – good performance is something that is earned from the long term discipline of focusing the lessons learned from many customer-oriented real-world database workloads, and applying those lessons across the architecture of the hypervisor.

Let’s take a quick walk through the specific features that you should look for in the hypervisor for good database performance.

1: High Performance I/O in VMware ESX

Throughput and latency of the I/O system are critical to performance of online transaction processing systems. Since transaction database systems operate on small data items at random places in the dataset, it’s important that we measure random I/O throughput (measured in I/O operations per second), rather than bandwidth (MB/s).


Figure.1 VMware ESX I/O Driver Model

Since the hypervisor logically resides between the database in the guest virtual machine and the backend storage, it is critical that the hypervisor’s I/O facilities scale up without any performance ceilings, and don’t add any appreciable latency. The I/O subsystem in VMware ESX shown in Figure.1 uses a direct driver model, so that there is minimal latency added by the virtualization stack. This is possible because I/O requests can be handled  in-line by the same processor as the requesting virtual machine (other architectures add substantial latency and CPU overhead when I/O is proxied via a heavy-weight domain-0 or parent-partition).


Figure.2 – Random I/O Throughput

(Average 4-CPU DB vs. VMware ESX 3.5

Oracle databases typically issue many small 4Kbyte or 8Kbyte sized I/Os in a random access pattern. For these I/O’s, a single typical disk can deliver somewhere in the order of 100-200 I/Os per second, depending on the rotational speed of the disk, though in practice, it’s best not to push the drives beyond 100 IOPS each. The throughput of the VMware ESX 3.5 hypervisor has been increased significantly, and shows that more than 60,000 I/Os per second can be sustained – the throughput of over 600 disks.

Results from an VMware study of its customers, showed that across 15,000 Oracle servers the average number of I/Os per second for a loaded 4 processor system is 1280, which is approximately the throughput of 15 disks. Since some workloads are more demanding than others, and some are bursty in nature, it’s important to have substantially more headroom. The throughput capability of the ESX’s I/O subsystem is sufficient for more than even the most demanding database.

2: Scale Up using Virtual SMP

VMware ESX can take advantage of systems with multiple physical
processors in two ways, by scaling out through multiple virtual
machines, and by scaling up each virtual machine to use more than one
physical processor. VMware ESX provides a Virtual SMP capability,
allowing up to four processors in each guest virtual machine, and up to
64 processors in the physical system.


Figure.3 – Virtual SMP

Since database workloads typically have a large number of concurrent
users, they are explicitly parallel and can easily process more than
one task at a time.

Oracle is able to take advantage of VMware’s Virtual SMP, so
performance can be scaled beyond a single processor for each virtual
machine. To demonstrate this, we ran several benchmarks with Oracle
database 10g Release 2, using the popular SwingBench on-line transaction processing
workload. Figure.5 shows the throughput of Oracle with an
increasing number of processors in a single virtual machine. The
benchmark measures transaction throughput, and shows 94% scaling as
additional processors are added. Incidentally, this is exactly the same as
the scaling we see on native, which is likely due to hardware and
database scaling artifacts.


Figure.5 – Scaling of  VMware Virtual SMP with a single instance of Oracle 10g R2

One of the key requirements for consolidation is good scalability with a large number of database instances. To show this, we ran multiple SMP instances of Oracle 10G on VMware ESX Server 3.5. Figure.3 shows the scaling of the VMware ESX platform when running
the open source DVDstore database benchmark. The benchmark is run using client-server mode, so that we can focus on the database tier . In this study, we scaled the benchmark using one through seven dual processor SMP Linux virtual machines, each with its own database instance. We’ll be posting further details of this benchmark on Vroom soon.

DVDstore Benchmark Scaling



Figure.4 – Scaling of multiple database instances on VMware ESX on a 16-core Sun x4600 M2






3: Scale up with Large Memory

Oracle databases love memory. The primary use is for caching pre-compiled SQL queries and caching blocks from disk in memory.

Database designers go to significant effort to avoid doing disk I/O when possible. This is because the latency of a disk I/O is substantially higher than the time a transaction will spend on the CPU. For example, a disk I/O takes on the order of 10 milliseconds, while the typical transaction takes just a few milliseconds of CPU time. If an I/O to disk can be cached in memory, it then could be serviced in a fraction of a millisecond. In addition, since disks are expensive, the cost of the storage systems for databases is often more affected by the I/Os per second that it can deliver, than by pure storage space. Lowering overall disk throughput can mean significantly lowering the cost of the system.

Larger memory sizes help Oracle by caching more disk blocks in memory. Consider this simple example: if a database system is using memory to cache it’s disk I/O, and is yielding a 90% cache hit rate, this means that one in every ten accesses causes a physical I/O. For 10,000 accesses a second, we would see 1,000 I/O’s per second. If we increase memory to improve the cache hit rate to 99%, then we reduce the I/Os to one in one-hundred, reducing the physical I/O by 10x to only 100 I/Os per second.

Often, over 80% of the memory used by the guest operating system is used by the Oracle disk block cache. A general rule of thumb is that the database cache be sized at 5-10% of the database size, and that doubling memory improves throughput by about 20%. This is obviously very workload dependent, but you can see that larger memory sizes help improve resource efficiency in other areas of the system, and that generally, more is better. For these reasons, large memory is very important for databases. VMware ESX 3.0 allowed 16GB of RAM per guest, and 3.5 increases the capability to 64GB per guest.

Due to the inherent gains in processor utilization through consolidation of workloads, we can squeeze more workloads onto a single system. This means that the average memory requirement per physical processor is on average twice that of a traditional unvirtualized system. To accommodate these growing requirements, we’ve pushed the memory scalability curve considerably in ESX 3.5, and now supports up to 256 Gigabytes of RAM on the new high-end systems from Sun, IBM and Unisys.

4: Large Pages in ESX 3.5 Hypervisor

Oracle databases have used large pages in the CPU’s MMU to optimize memory performance for some time. This facility is used with the operating system’s large page feature, typically for the large shared memory segments that hold the database’s disk block cache. Large pages are supported on Linux, Windows and Solaris guests. Oracle typically yields a 5-20% performance advantage with large pages, depending on the type of processor and the size of the configured memory.

Other x86 hypervisors don’t provide virtual large page capability, so this optimization is lost when the database is virtualized. The ESX 3.5 hypervisor provides advanced large-page support which allows the database to properly exploit the CPU’s large page capability.

5: ESX Optimization for NUMA systems

Many of the interesting new hardware systems today are implemented using non-uniform memory architectures. This means that not all memory is of uniform speed – accessing memory that is closer topologically to the processor is faster than memory that is further away.

To ensure optimal performance, the VMware ESX hypervisor allocates memory for the guest operating systems from physical memory near the CPU on which the guest resides on.

6: High-performance Paravirtualized Networking

Paravirtualization is a term used to describe when the guest operating system has some knowledge of the hypervisor, and can leverage this knowledge to optimize it’s execution in concert with the hypervisor. The VMware ESX hypervisor uses paravirtualized networking drivers in the guest operating system to provide high performance networking. These drivers are installed automatically through the VMware tools package at the time the guest is first powered on. Unlike CPU paravirtualization, paravirtualized drivers do not require any changes to the guest operating system – they are simply installed as transparent new drivers.


Figure.6 – Multi-NIC Scalability

VMware ESX can drive gigabit Ethernet at line rate, as demonstrated in the paper Networking Performance in Multiple Virtual Machines. The networking performance of ESX 3.5 has further improved by incorporating new stateless offload features, such as large-segment-offload (LSO)  and jumbo frames —  and now achieves near line-rate (9.9gbits/s) on 10Gbit Ethernet.

The performance of networking can be increased beyond that of one NIC by scaling across multiple NICs. Figure.6 shows the scaling of gigabit Ethernet performance as multiple NICs are added.

7. Use VMware ESX’s Page-Sharing to use less memory

The ESX hypervisor can safely share physical memory that has the same contents through a facility known as transparent page sharing. Through page sharing, the hypervisor arranges for a single physical page of memory to back multiple pages in the guest, so that just one copy of the data need reside in memory. Using this technique, the total amount of memory consumed is less than the sum of the parts. The hypervisor ensures full security isolation — if one guest modifies a page, then it get its own private copy.

This facility can be used effectively to save memory with Oracle in several ways. When there are multiple instances of a database running, the page sharing facility will automatically share the code portions of the operating system and the Oracle instance. This often results in saving in the order of a few hundred megabytes of memory per virtual machine.

When multiple databases are sharing similar data – for example, a shared reference table or multiple copies of the database for development purposes, ESX can automatically detect the duplicate disk blocks in the Oracle disk block cache, and arrange to share those. Thus, the database cache memory image can be transparently shared across database instances, and across virtual machines. This can result in a further saving in the order of tens of megabytes (at least the system tables will be the same), through several gigabytes, depending on the amount of common disk data between the instances.

As an additional benefit, some memory can be shared within each instance. This is often for zero pages.

8: Paravirtualized CPU


Figure.7 – Virtualization Techniques

There are various techniques used to virtualize the x86 instruction set, including binary translation, paravirtualization and hardware assist. Binary translation has long been used by the VMware hypervisor to provide near native performance for virtualization for many workloads. CPU Paravirtualization or hardware assist are two approaches that can be used to provide small optimizations for workloads with many system calls as well as providing certain memory optimizations. No single approach is best for all workloads, and in VMware ESX, different approaches are used for different workloads. Ole Agesen and Keith Adams help explain the different technologies in their paper about the performance of virtualization.


VMware ESX can optionally use paravirtualization for some guest operating system types. In a recent study, the performance of Oracle 10g R2 using the Swingbench online transaction processing workload on a paravirtualized Linux guest shows a moderate gain of 10% when using paravirtualized CPU interfaces.

9: The Best Oracle-Windows Performance

Since all of the key CPU, memory and I/O virtualization capabilities are in a portable layer of the hypervisor, the performance of Oracle on Windows guests is equivalent to that of Linux guests. Oracle on Windows is able to take advantage of large-pages, SMP, and I/O scalability as well as our high performance paravirtualized networking drivers.

For Linux, Solaris and Unix administrators, this means that you have the freedom to choose the OS which has the best tools to facilitate your deployment. For Windows administrators, it means that you can confidently run your Oracle databases on Windows, with the same levels of performance and scalability.

10: Universal 32-bit and 64-bit Guest Support

To take advantage of more than 3.5Gbytes of memory in a guest, databases need to be configured as 64-bit applications, and use a 64-bit capable operating system. VMware ESX allows mixed 32-bit and 64-bit guests concurrently, thu simplifying the deployment of 64-bit guests when needed.


To all the database administrators out there, watch for more VROOM posts about VMware performance with Oracle, the new  Oracle portal, which will contain plenty of good resources for virtualization of Oracle. Also, there is a new Oracle Discussion Forum – feel free to discuss Oracle performance over at the forum too. Virtual database performance has never been so good!


44 comments have been added so far

  1. Hello. I am new to VMware and the new ESX 3.x server. I want to clarify an issue with VMnix and VMkernel. According to literature, VMnix is based on a customised kernel of Redhat 7.2. This version came out nearly 5 years back and is based on the old Linux scheduler. Pre 2.6 the Linux kernel suffered from scalability at heavy loads mainly due to the scheduler. Ingo Molnar’s O(1) scheduler changed that and gave Linux tremendous scalability and performance under very heavy, enterprise class loads.
    My question is: what kernel version is the new ESX 3.x server based on?
    Asaf Maruf

  2. ESX is not based on the Linux Kernel at all. All the Linux kernel does is bootstrap the proprietary VMWARE ESX kernel called “VMKERNEL”, this is where all the scalability and stability is. The Linux kernel is almost not important, and in ESX 3i it doesn’t exist at all.

  3. thank you for the interesting document. Three small questions:
    – how supported is it? (by VMware, by Oracle)
    – are there many sites in production with Oracle/ESX?
    – is RAC working well (inside a physical host, across physical hosts)?
    thank you in advance,

  4. Your literature points to the service console of ESX Version 2.x. That console has been updated to RHEL3 in ESX/VI Version 3.x. Note the distinction between vmkernel and linux service console.The console is completely hidden in ESX 3i.

  5. Didn’t Oracle just announce that they will only support their products running on their new “Oracle VM” hypervisor?

  6. Hello everyone.
    I would like to know if someone has experience on SQL 2005 running over EXS 3.x server, and what about it´s performance
    Thanks in advance

  7. I think there is a small issue in your blog post. Quoting you:
    “VMware ESX’s thin hypervisor layer can drive over 63,000 database I/Os per second, which is equivalent to more than 600 disk spindles of I/O throughput.”.
    That is 105 IOps / disc. If you mean notebook hds, you are possibly right. But speaking of contemporary (server-) harddiscs, you are absolutely wrong.

  8. Are there tools available to help one test the performance of applications in a VMware enviroment prior to moving them into production?

  9. Richard,
    I have run both Swingbench and Benchmark Factory against an almost identical setup of a standalone and virtualized server. I am unfortunately not seeing similar performance numbers-the standalone box is outperforming its virtual counterpart by margins as high as 5x.
    Using the latest 3.5 host – any config setting I may be missing that would account for the difference?
    Much thanks in advance!

  10. This looks nice, but oracle’s recent move into the VM world has resulted in a rather nasty support issue:
    According to their FAQ, they explicitly state that none of their products are supported in any VM environment other than their own.
    They dont want you using ESX, which rocks. They want you to pay more for Oracle VM which is brand new and has stability and scalability issues.
    Hopefully folks will beat them back into submission to support ESX.

  11. Oracle is not certified for use with VMware! Of course it works fine, but it is not certified.
    Oracle VM is a FREE virtualization alternative, which is Xen based, and as far as Oracle Products is concerned is FULLY CERTIFIED.
    Thus, going with the FREE and CERTIFIED option when it comes to oracle installations, ORACLE VM is the obvious choice.
    Other than that VMware is a fine product. Just that ESX compared to Oracle VM is expensive and in several cases limited.

  12. This part has me perplexed:
    “the performance of Oracle on Windows guests is equivalent to that of Linux guests”
    Does this mean Oracle on a Linux guest has now been reduced to Windows performance? We are a large Oracle shop and our in-house testing with large installations (like a single RAC cluster driving 40 DB instances and several TBs of DASD) has shown Microsoft performance to lag behind Linux performance – sometimes quite dramatically. We are looking for a virtualization solution for some non-RAC standalone implementations, but VMWare hasn’t shown the performance we want and we’re leery of the stability of Xen at the moment. OracleVM is much cheaper than VMWare and is fully supported by Oracle so we may take a look at that, but we’d definitely need some assurances from Oracle on stability before we went live with anything.

  13. WFL3
    Oracle VM is built from XEN. Oracle does not have near term plans to run RAC on OVM. VMmotion capabilities are not projected near term either.
    I am investigating comparisons of IBM/VM/Oracle virtualization. As I get more info, I’ll try to come back and post.

  14. Anyone done a comparison for KVM/VMware/Oracle VM for a mixed environment?
    Looking for some guidelines to virtualize a Linux/Ux/Windows environment running Oracle Business Suites and 10g

  15. Wayne
    I look forward to your posts, Thanks.
    We are just now starting to give OracleVM a test drive, initial impressions look good. The price (FREE) is definitely right and if you’re an Oracle customer talk to your salesman about 24/7 support – it’s practically free as well.
    We’re also looking at zLinux on IBM big iron – I’m not as optimistic on this front after some initial rounds of benchmarking. Cost/performance ratios look dismal IMO so far.

  16. What are your thoughts on deploying this solution to small foot print servers like blades. HS21 or something like a DL380. Is there some recomendation on the size of the servers?

  17. I’m look at ESX to run a whole application: Oracle 10g on 64-bit Linux, Oracle OBIEE Web on 32-bit Win and 2 X OBIEE Analytics on 32-bit Win. So these could all run on the same server? When they talk to each other over the “network” would the traffic actually leave the server? Assume I have a single Fast Ethernet NIC.

  18. Working for government department where each and every word is confidential. Knowing that your shredded documents cannot be reassembled and that your confidential information is safe is worth every cent of the cost of a paper shredder

  19. The saleman told me from whom I bought the shredder. All office shredders roll on casters for convenient sharing among offices. Every shredder model has a 10-year warranty on cutting heads and can take staples and paper clips which saves office stationary too.

  20. Living in Florida there was an urgent need of shredder in the office so I called up the saleman who told me there was No sales tax on purchases delivered out of California! 10 year warranty on cutting heads & 1 year warranty on mechanical parts (parts only)!

  21. According to the investigations Andersen partner David Duncan allegedly headed an effort to destroy documents related to Enron after learning the Securities and Exchange Commission had requested financial records from the company.

  22. In a civil case, a judge can allow the jury to question a document-destroying party’s intentions. For example, judges in certain cases will tell jurors they should assume missing documents are harmful simply because they were destroyed–even if they never see the contents.

  23. The other big problem beyond support is that Oracle does NOT recognize the vCPU guest limit as a hard partition for licensing purposes so you would have to be licensed per seat, for all cores in the box, or have an enterprise agreement to be in compliance.

  24. aehhhhm sorry? Are you really believing what you’re writing there?
    1.) Oracle Databases are not supported on VMWare
    2.) You can run Oracle on VMWare BUT: you have to licences all sockets from the base system (not the VM)
    So if you have a VM with 2 virtual processor on a host with 8 CPUs you have to license 8 CPUs!!!
    3.) VMWare boosts the I/O over the memory. That is nice for performance, but what happens by a power failure ==> data loss

  25. Hi,
    I am stuck in installation process. During installation of clusterware on both node. It gave me error to not identified remote node. Even cluster verify is success.

  26. The numbers in this article are now quite out of date. But show even with ESX 3.5 the hypervisor could perform quite well. With the latest vSphere 5.1 release 64 vCPU’s and 1TB of RAM per VM are supported in addition to over 1million IOPS per VM (<2ms latency). The hypervisor is not your bottleneck. If there is an overhead it means you just need to add additional resources to the VM to compensate. Ensure your physical infrastructure meets your requirements and you'll get great performance plus all the other availability and quality of service capabilities that VMware vSphere offers.

  27. This information is now 14 years old and yet is still referenced on the latest vmware.com ‘Oracle Solutions with VMware’ page. Please update this page with more up-to-date performance reports or remove the link from the VMware page.

Leave a Reply

Your email address will not be published. Required fields are marked *