Techopedia – “While scaling out involves adding more discrete units to a system in order to add capacity, scaling up involves building existing units by integrating resources into them”
Business-critical production Oracle workload are very resource intensive and large memory sizes are characteristics of production databases to ensure that the working set of the database fits into DRAM for fast access to data.
Let us look at the different options we have currently with the demand for memory increase for Oracle workloads –
- Increased Cost with buying more DRAM ($$$) – cost to adding additional DRAM to beef up the system for large memory allocations
- Memory Latency with sizing workloads across multiple NUMA nodes to take advantage of remote memory to fulfill memory needs – With NUMA servers, one has an option of accessing memory from remote NUMA nodes across QPI/UPI interconnects but …. there is memory latency that one would encounter with remote memory accesses.
Ok, let’s look at our choices – Increased cost ($$$) OR Memory Latency for critical workloads ?
Enter VMware vSphere platform with Intel DC Optane PMM in Memory Mode.
VMware vSphere usage of Intel Optane PMem in Memory Mode can offer increased memory capacity and TCO improvements for large memory bound workloads without having to buy expensive DRAM’s or size workloads across NUMA modes for added memory needs.
This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase.
Intel Optane DC Persistent Memory – Memory Mode
Intel Optane DC Persistent Memory has two modes – App Direct and Memory Mode.
When running in this mode, the Intel Optane DC Persistent Memory is presented to applications and operating systems as if it were ordinary volatile memory. But, under the hood, the higher performance DRAM is used as a cache for the persistent memory.
Memory Mode makes adoption of DCPMM technology easier because applications do not need to be modified. The affordable capacity frequently makes it more cost-efficient than adding additional DRAM. However, as with any cache architecture, there can be variability in performance depending on workload, ratio of DRAM to persistent memory, and dataset characteristic.
More information on Intel Optane DC Persistent Memory in Memory Mode can be found at vSphere Support for Intel’s Optane Persistent Memory (PMEM) (67645) and VMware vSphere Performance with Intel Optane Persistent Memory in Memory Mode – Balanced Profile.
Test Use case
This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase
This test case focused on comparing results between running Oracle workloads on 2 VM setup as shown below
- VMware vSphere with Intel Optane PMEM in Memory Mode
- VMware vSphere with workload spanning NUMA nodes (non-Memory mode)
Given the un-availability of a server in my lab with 768GB DRAM – we decided to compare the above 2 cases with the premise that performance of most VMs within a NUMA node will be better than performance across NUMA node given the memory latency across NUMA nodes.
So
- if the performance of the workload in Memory mode is at par / better than the workload across NUMA nodes a.k.a Wide VM configuration AND
- if performance of the workload across NUMA nodes a.k.a Wide VM configuration is acceptable , given the fact the need for additional memory triggered the need to add additional memory across NUMA nodes
- then performance of the workload in Memory mode should be acceptable, advantages being no need to buy expensive DRAM
Test Bed
Details of the VMware ESXi servers in DRAM and Memory mode are shown as below.
- DRAM Mode – ‘sc2esx64.vslab.local’ , VMware ESXi, 7.0.3, 19482537 with 4 sockets, 24 cores per socket, 1.5 TB DRAM and 3TB Persistent Memory
- Memory Mode – ‘sc2esx66.vslab.local’, VMware ESXi, 7.0.3, 19482537 with 4 sockets, 24 cores per socket, 3TB Memory (1.5 TB DRAM is now a cache for the 3TB PMEM memory which is volatile memory in the memory mode)
2 identical VM’s were created, one running in DRAM mode and one in Memory mode.
- DRAM Mode – VM ‘Oracle21C-OL8-DRAM’
- Memory Mode – VM ‘Oracle21C-OL8-MemoryMode’
Both VM’s had 24 vCPUs, vRAM 740GB each with VM storage on All-Flash array.
The only difference between the 2 VM’s was
- VM ‘Oracle21C-OL8-DRAM’ was a NUMA aware VM at the ESXI, OS and Oracle level
- VM ‘Oracle21C-OL8-MemoryMode’ was a non-NUMA VM
The vmdk’s for the VM ‘Oracle21C-OL8-Customer’ are shown as below –
- Hard Disk 1 (SCSI 0:0) – 80G for OS (/)
- Hard Disk 2 (SCSI 0:1) – 80G for Oracle Grid Infrastructure and RDBMS binaries
- Hard Disk 3 (SCSI 1:0) – 100G for Grid ASM (GRID_DG ASM Disk Group)
- Hard Disk 4 (SCSI 1:1) – 200G for Oracle Database (DATA_DG ASM Diskgroup)
- Hard Disk 5 (SCSI 2:0) – 1TB SLOB Tablespace (SLOB_DG ASM Diskgroup)
- Hard Disk 6 (SCSI 3:0) – 100G for Redo logs (REDO_DG ASM Diskgroup)
- Hard Disk 7 (SCSI 1:2) – 512G for FRA (FRA_DG ASM Diskgroup)
Details of the vmdk’s are shown as below
ESXI & VM NUMA configurations for VM’s in DRAM and Memory mode are shown as below
OS NUMA Layout for VM’s in DRAM and Memory mode is shown as below
A single instance database ‘ORA21C’ with multi-tenant option was provisioned with Oracle Grid Infrastructure (ASM) and Database version 21.5 (Database Release Update 21.5.0.0.220118) on O/S OEL 8.5 UEK.
Oracle ASM was the storage platform with Oracle ASMLIB for device persistence.
Oracle SGA & PGA set to 640GB and 20G respectively. All Oracle on VMware best practices were followed.
The ASM diskgroups are shown as below.
grid@oracle21c-ol8:+ASM:/home/grid> asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 1048576 204799 51449 0 51449 0 N DATA_DG/
MOUNTED EXTERN N 512 512 4096 4194304 524284 524192 0 524192 0 N FRA_DG/
MOUNTED EXTERN N 512 512 4096 4194304 102396 102292 0 102292 0 N GRID_DG/
MOUNTED EXTERN N 512 512 4096 1048576 102399 77471 0 77471 0 N REDO_DG/
MOUNTED EXTERN N 512 512 4096 4194304 1048572 24472 0 24472 0 N SLOB_DG/
grid@oracle21c-ol8:+ASM:/home/grid>
Test Case
SLOB 2.5.4.0 Oracle I/O workload generation tool kit was chosen as the load generator for this exercise.
We ran a Read-only test with the following SLOB parameters as shown below –
UPDATE_PCT=0
RUN_TIME=300
SCALE=600G
WORK_UNIT=32
REDO_STRESS=LITE
We kept the redo stress LITE as the intent of this exercise is to test DRAM v/s Memory Mode performance of Oracle workloads.
Remember, any performance data is a result of the combination of
- hardware configuration
- software configuration
- test methodology & test tool
- workload profile used in the testing
So
- the performance metrics / improvements I got with my workload in my lab is in no way representative of any real production workload, which means the performance improvements for real world workloads will be better
- this was by no means a benchmarking exercise.
Test Results
We ran the SLOB load generator with Oracle SGA in DRAM across 2 NUMA nodes v/s Oracle SGA in Memory Mode and compare the 2 run database metrics.
From an overall DB perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode
- Logical reads (blocks) have increased from 10,176,122.9/sec to 10,363,868.4/sec
- Executes (SQL) have increased from 296,379.6/sec to 304,222.0/sec
From a Database OS perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in DRAM in Memory Mode
- Both VM’s have similar OS metrics from a Load Average and CPU Utilization perspective
From an OS CPU Utilization perspective, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode
- Both VM’s have similar OS metrics from a CPU Utilization perspective
Summary
In summary, we can see, running Oracle SGA in DRAM across 2 NUMA nodes versus Oracle SGA in Memory Mode
- From an overall DB perspective
- Logical reads (blocks) have increased
- Executes (SQL) have increased
- From a Database OS perspective
- Both VM’s have similar OS metrics from a Load Average and CPU Utilization perspective
- From an OS CPU Utilization perspective
- Both VM’s have similar OS metrics from a CPU Utilization perspective
So performance of the above workload in Memory mode is acceptable compared to the performance of the workload across NUMA nodes a.k.a Wide VM configuration with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase.
Conclusion
- This blog demonstrates how one can achieve similar performance running Oracle workloads on VMware vSphere with Intel Optane PMEM in Memory Mode as compared to running the same workloads on VMware platform by spreading memory across NUMA nodes (or adding expensive DRAM – this blog focused on comparing Memory Mode v/s NUMA access) to satisfy the memory demand – with the advantage of not needing to buy expensive DRAM or going across NUMA nodes for added memory increase
- This blog contains results that I got in my lab running a load generator SLOB against my workload, which will be way different than any real-world customer workload, your mileage may vary
- Remember, any performance data is a result of the combination of hardware configuration, software configuration, test methodology & test tool and workload profile used in the testing.
- Also, this was by no means a benchmarking exercise.
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html
