vSphere Oracle PMem

Oracle workloads using Intel Optane DC PMM in Memory Mode on VMware vSphere platform – An Investigation

 

 

 

Techopedia – “While scaling out involves adding more discrete units to a system in order to add capacity, scaling up involves building existing units by integrating resources into them”

Business-critical production Oracle workload are very resource intensive and large memory sizes are characteristics of production databases to ensure that the working set of the database fits into DRAM for fast access to data.

Let us look at the different options we have currently with the demand for memory increase for Oracle workloads –

  • Increased Cost with buying more DRAM ($$$) – cost to adding additional DRAM to beef up the system for large memory allocations
  • Memory Latency with sizing workloads across multiple NUMA nodes to take advantage of remote memory to fulfill memory needs – With NUMA servers, one has an option of accessing memory from remote NUMA nodes across QPI/UPI interconnects but …. there is memory latency that one would encounter with remote memory accesses.

Ok, let’s look at our choices – Increased cost ($$$) OR Memory Latency for critical workloads ?

 

 

 

 

Enter VMware vSphere platform with Intel DC Optane PMM in Memory Mode.

VMware vSphere usage of Intel Optane PMem in Memory Mode can offer increased memory capacity and TCO improvements for large memory bound workloads without having to buy expensive DRAM’s or size workloads across NUMA modes for added memory needs.

This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase.

 

 

Intel Optane DC Persistent Memory – Memory Mode

 

Intel Optane DC Persistent Memory has two modes – App Direct and Memory Mode.

When running in this mode, the Intel Optane DC Persistent Memory is presented to applications and operating systems as if it were ordinary volatile memory. But, under the hood, the higher performance DRAM is used as a cache for the persistent memory.

Memory Mode makes adoption of DCPMM technology easier because applications do not need to be modified. The affordable capacity frequently makes it more cost-efficient than adding additional DRAM. However, as with any cache architecture, there can be variability in performance depending on workload, ratio of DRAM to persistent memory, and dataset characteristic.

 

 

 

More information on Intel Optane DC Persistent Memory in Memory Mode can be found at vSphere Support for Intel’s Optane Persistent Memory (PMEM) (67645) and VMware vSphere Performance with Intel Optane Persistent Memory in Memory Mode – Balanced Profile.

 

 

 

Test Use case

 

This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase

This test case focused on comparing results between running Oracle workloads on 2 VM setup as shown below

  • VMware vSphere with Intel Optane PMEM in Memory Mode
  • VMware vSphere with workload spanning NUMA nodes (non-Memory mode)

Given the un-availability of a server in my lab with 768GB DRAM – we decided to compare the above 2 cases with the premise that performance of most VMs within a NUMA node will be better than performance across NUMA node given the memory latency across NUMA nodes.

So

  • if the performance of the workload in Memory mode is at par / better than the workload across NUMA nodes a.k.a Wide VM configuration AND
  • if performance of the workload across NUMA nodes a.k.a Wide VM configuration is acceptable , given the fact the need for additional memory triggered the need to add additional memory across NUMA nodes
  • then performance of the workload in Memory mode should be acceptable, advantages being no need to buy expensive DRAM

 

 

 

 

 

Test Bed

 

Details of the VMware ESXi servers in DRAM and Memory mode are shown as below.

  • DRAM Mode – ‘sc2esx64.vslab.local’ , VMware ESXi, 7.0.3, 19482537 with 4 sockets, 24 cores per socket, 1.5 TB DRAM and 3TB Persistent Memory
  • Memory Mode – ‘sc2esx66.vslab.local’, VMware ESXi, 7.0.3, 19482537 with 4 sockets, 24 cores per socket, 3TB Memory (1.5 TB DRAM is now a cache for the 3TB PMEM memory which is volatile memory in the memory mode)

 

 

 

 

2 identical VM’s were created, one running in DRAM mode and one in Memory mode.

  • DRAM Mode – VM ‘Oracle21C-OL8-DRAM’
  • Memory Mode – VM ‘Oracle21C-OL8-MemoryMode’

 

Both VM’s had 24 vCPUs, vRAM 740GB each with VM storage on All-Flash array.

The only difference between the 2 VM’s was

  • VM ‘Oracle21C-OL8-DRAM’ was a NUMA aware VM at the ESXI, OS and Oracle level
  • VM ‘Oracle21C-OL8-MemoryMode’ was a non-NUMA VM

 

 

 

 

The vmdk’s for the VM ‘Oracle21C-OL8-Customer’ are shown as below –

  • Hard Disk 1 (SCSI 0:0) – 80G for OS (/)
  • Hard Disk 2 (SCSI 0:1) – 80G for Oracle Grid Infrastructure and RDBMS binaries
  • Hard Disk 3 (SCSI 1:0) – 100G for Grid ASM (GRID_DG ASM Disk Group)
  • Hard Disk 4 (SCSI 1:1) – 200G for Oracle Database (DATA_DG ASM Diskgroup)
  • Hard Disk 5 (SCSI 2:0) – 1TB SLOB Tablespace (SLOB_DG ASM Diskgroup)
  • Hard Disk 6 (SCSI 3:0) – 100G for Redo logs (REDO_DG ASM Diskgroup)
  • Hard Disk 7 (SCSI 1:2) – 512G for FRA (FRA_DG ASM Diskgroup)

 

 

Details of the vmdk’s are shown as below

 

 

 

 

ESXI & VM NUMA configurations for VM’s in DRAM and Memory mode are shown as below

 

 

 

 

OS NUMA Layout for VM’s in DRAM and Memory mode is shown as below

 

 

 

A single instance database ‘ORA21C’ with multi-tenant option was provisioned with Oracle Grid Infrastructure (ASM) and Database version 21.5 (Database Release Update 21.5.0.0.220118) on O/S OEL 8.5 UEK.

Oracle ASM was the storage platform with Oracle ASMLIB for device persistence.

Oracle SGA & PGA set to 640GB and 20G respectively. All Oracle on VMware best practices were followed.

 

The ASM diskgroups are shown as below.

grid@oracle21c-ol8:+ASM:/home/grid> asmcmd lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  1048576    204799    51449                0           51449              0             N  DATA_DG/

MOUNTED  EXTERN  N         512             512   4096  4194304    524284   524192                0          524192              0             N  FRA_DG/

MOUNTED  EXTERN  N         512             512   4096  4194304    102396   102292                0          102292              0             N  GRID_DG/

MOUNTED  EXTERN  N         512             512   4096  1048576    102399    77471                0           77471              0             N  REDO_DG/

MOUNTED  EXTERN  N         512             512   4096  4194304   1048572    24472                0           24472              0             N  SLOB_DG/

grid@oracle21c-ol8:+ASM:/home/grid>

 

 

 

 

Test Case

 

SLOB 2.5.4.0 Oracle I/O workload generation tool kit was chosen as the load generator for this exercise.

We ran a Read-only test with the following SLOB parameters as shown below –

UPDATE_PCT=0
RUN_TIME=300
SCALE=600G
WORK_UNIT=32
REDO_STRESS=LITE

 

We kept the redo stress LITE as the intent of this exercise is to test DRAM v/s Memory Mode performance of Oracle workloads.

 

Remember, any performance data is a result of the combination of

  • hardware configuration
  • software configuration
  • test methodology & test tool
  • workload profile used in the testing

So

  • the performance metrics / improvements I got with my workload in my lab is in no way representative of any real production workload, which means the performance improvements for real world workloads will be better
  • this was by no means a benchmarking exercise.

 

 

 

 

Test Results

 

We ran the SLOB load generator with Oracle SGA in DRAM across 2 NUMA nodes v/s Oracle SGA in Memory Mode and compare the 2 run database metrics.

From an overall DB perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode

  • Logical reads (blocks) have increased from 10,176,122.9/sec to 10,363,868.4/sec
  • Executes (SQL) have increased from 296,379.6/sec to 304,222.0/sec

 

 

 

 

 

From a Database OS perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in DRAM in Memory Mode

  • Both VM’s have similar OS metrics from a Load Average and CPU Utilization perspective

 

 

 

 

 

From an OS CPU Utilization perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode

  • Both VM’s have similar OS metrics from a CPU Utilization perspective

 

 

 

 

 

Summary

 

In summary, we can see, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode

  • From an overall DB perspective
    • Logical reads (blocks) have increased
    • Executes (SQL) have increased

 

  • From a Database OS perspective
    • Both VM’s have similar OS metrics from a Load Average and CPU Utilization perspective

 

  • From an OS CPU Utilization perspective
    • Both VM’s have similar OS metrics from a CPU Utilization perspective

So performance of the above workload in Memory mode is acceptable compared to the performance of the workload across NUMA nodes a.k.a Wide VM configuration with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase.

 

 

Conclusion

  • This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase
  • This blog contains results that I got in my lab running a load generator SLOB against my workload, which will be way different than any real-world customer workload, your mileage may vary
  • Remember, any performance data is a result of the combination of hardware configuration, software configuration, test methodology & test tool and workload profile used in the testing.
  • Also, this was by no means a benchmarking exercise.

 

 

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

 

 

 

Comments

Leave a Reply