Wikipedia – Nicknamed the “Scarlet Speedster”, all incarnations of the Flash possess “super speed“, which includes the ability to run, move, and think extremely fast, use superhuman reflexes, and seemingly violate certain laws of physics.
This blog demonstrates how performance of Oracle Smart Flash Cache can be accelerated using Persistent Memory (PMEM) technology by using Intel Optane DC PMM in App Direct mode backed Oracle 21c Persistent Memory Filestore on VMware vSphere platform.
This blog is the 2nd blog in this series of accelerating Oracle workloads using Intel Optane DC PMM in App Direct mode backed Oracle 21c Persistent Memory Filestore on VMware vSphere platform.
The 1st blog of the series was Accelerate Oracle Redo performance using Intel Optane DC PMM backed Oracle 21c Persistent Memory Filestore on VMware vSphere
Yes, your Oracle workload is now truly a “Speed King”.
VMware vSphere Persistent Memory
VMs can be configured to use PMem on a standalone host, or in a cluster. In ESXi you can create VMs that are configured with PMem, and applications inside these VMs can take advantage of this increased speed. Once a VM is initially powered on, PMem is reserved for it regardless of whether it is powered on or off. This PMem stays reserved until the VM is migrated or removed.
Persistent memory can be consumed by virtual machines in two different modes. Legacy guest OSes can still take advantage of virtual persistent memory disk feature.
- Virtual Persistent Memory (vPMem) – Using vPMem, the memory is exposed to a guest OS as a virtual NVDIMM. This enables the guest OS to use PMem in byte addressable random mode.
- Virtual Persistent Memory Disk (vPMemDisk) – Using vPMemDisk, the memory can be accessed by the guest OS as a virtual SCSI device, but the virtual disk is stored in a PMem datastore.
More information on VMware vSphere Persistent Memory can be found here.
Intel Optane DC Persistent Memory
Intel® Optane DC Persistent Memory (DCPMM) memory is an emerging technology where non-volatile media is placed onto a Dual In-Line Memory Module (DIMM) and installed on the memory bus, traditionally used only for volatile memory. Intel’s DCPMM is the implementation of 3D Xpoint which coexists with DRAM on the memory bus. It can be used both as DRAM alternative but can also be used as a high performance and low latency storage device.
More information on Intel Optane DC Persistent Memory can be found here.
Oracle Smart Flash Cache
Oracle Database 11g Release 2 introduced a new database feature: Database Smart Flash Cache This feature allows customers to increase the effective size of the Oracle database buffer cache without adding more main memory to the system. For transaction-based workloads, Oracle database blocks are normally loaded into a dedicated shared memory area in main memory called the System Global Area (SGA).
Database Smart Flash Cache allows the database buffer cache to be expanded beyond the SGA in main memory to a second level cache on flash memory. Online Transaction Processing (OLTP) environments benefit from this technology. Benefits can include both increased transaction throughput and improved application response times
It is also supported in Real Application Cluster (RAC) environments. Database Smart Flash Cache can be applied to individual RAC nodes as required
More information on Oracle Smart Flash Cache can be found here
Oracle Persistent Memory Filestore
The Persistent Memory Database feature enables you to place database files in non-volatile memory. This feature supports a single-instance Oracle Database instance on PMEM Filestore.
PMEM Filestore is a pointer-switching PMEM file system that supports atomic updates of Oracle Database data blocks. PMEM Filestore implements the Filesystem in Userspace (FUSE) protocol, enabling Oracle DBAs to perform normal file-level maintenance. FUSE allows non-privileged (non-root) users, such as the Oracle Database software owner, to create and manage filesystems as well as the directories and files contained within them
Some of the benefits of using Persistent Memory Database:
- The PMEM Filestore provides atomic writes to full Oracle database blocks. This eliminates the need for media recovery due to partially written blocks after a power outage.
- Persistent Memory Database performs I/O to PMEM storage via memory copy. This is much faster than performing I/O via traditional operating system calls.
- Database queries save the traditional read from storage and memory copy into buffer cache because the Oracle Database server accesses data directly from persistent memory.
More information on Oracle 21c Persistent Memory Filestore can be found here.
Test Use case
This blog demonstrates how performance of Oracle Smart Flash Cache can be accelerated using Persistent Memory (PMEM) technology by using Intel Optane DC PMM in App Direct mode backed Oracle 21c Persistent Memory Filestore on VMware vSphere platform
Test Bed
Details of the VMware ESXi server is shown as below. An ESXi server Version 7.0.3, 18644231 server was deployed with 4 sockets, 24 cores per socket, 1.5 TB DRAM and 3TB Persistent Memory.
VM ‘Oracle21C-OL8-Customer’ was created created with 24 vCPU’s and 256GB memory with storage on All-Flash array with Oracle SGA & PGA set to 128G and 20G respectively. Init.ora ‘DB_FLASH_CACHE_SIZE’ parameter was set to 384G.
A single instance database ‘ORA21C’ with multi-tenant option was provisioned with Oracle Grid Infrastructure (ASM) and Database version 21.5 (Database Release Update 21.5.0.0.220118) on O/S OEL 8.5 UEK.
Oracle ASM was the storage platform with Oracle ASMLIB for device persistence.
The vmdk’s for the VM ‘Oracle21C-OL8-Customer’ are shown as below –
- Hard Disk 1 (SCSI 0:0) – 80G for OS (/)
- Hard Disk 2 (SCSI 0:1) – 80G for Oracle Grid Infrastructure and RDBMS binaries
- Hard Disk 3 (SCSI 1:0) – 100G for Grid ASM (GRID_DG ASM Disk Group)
- Hard Disk 4 (SCSI 1:1) – 200G for Oracle Database (DATA_DG ASM Diskgroup)
- Hard Disk 5 (SCSI 2:0) – 1TB SLOB Tablespace (SLOB_DG ASM Diskgroup)
- Hard Disk 6 (SCSI 3:0) – 100G for Redo logs (REDO_DG ASM Diskgroup)
- Hard Disk 7 (SCSI 1:2) – 512G for Smart Flash Cache (FLASH_DG ASM Diskgroup)
Details of the vmdk’s are shown as below
The ASM diskgroups are shown as below.
[root@oracle21c-ol8 ~]# oracleasm scandisks ; oracleasm listdisks
Reloading disk partitions: done
Cleaning any stale ASM disks…
Scanning system for ASM disks…
DATA_DISK01
FLASH_DISK01
GRID_DISK01
REDO_DISK01
SLOB_DISK01
[root@oracle21c-ol8 ~]#
In addition, 1 NVDIMM of 512G capacity is added to the VM, for use as PMEM in fdax mode in the GOS, as shown below.
ndctl command listing shows the PMEM device in a fsdax mode
[root@oracle21c-ol8 ~]# ndctl list
[
{
“dev”:”namespace0.0″,
“mode”:”fsdax”,
“map”:”dev”,
“size”:541163782144,
“uuid”:”5eca33a2-2990-4936-90f2-9f4adb15b190″,
“sector_size”:512,
“align”:2097152,
“blockdev”:”pmem0″
}
]
[root@oracle21c-ol8 ~]#
A partition is created on the PMEM device (pmem0) as shown below for creating a EXT4 dax mounted filesystem
[root@oracle21c-ol8 ~]# fdisk -lu /dev/pmem0
Disk /dev/pmem0: 504 GiB, 541163782144 bytes, 1056960512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9dad81a0
Device Boot Start End Sectors Size Id Type
/dev/pmem0p1 4096 1056960511 1056956416 504G 83 Linux
[root@oracle21c-ol8 ~]#
[root@oracle21c-ol8 ~]# mount | grep -i pmem0
/dev/pmem0p1 on /oracle_flash_dax type ext4 (rw,noatime,dax)
[root@oracle21c-ol8 ~]#
For Oracle Persistent Memory Filestore, the pre-requisite is to create a mount point for the file store. The final subdirectory name must be the same as the PMEM filestore name.
In this example , we created a directory ‘oraflash’ within a directory ‘oracle_flash’ and both owned by the Oracle user.
root@oracle21c-ol8 /]# ls -l
drwxrwxrwx 3 oracle oinstall 22 Feb 26 19:20 oracle_flash
oracle@oracle21c-ol8:ORA21C:/home/oracle> ls -l /oracle_flash
drwxrwxrwx 2 oracle oinstall 6 Feb 26 19:20 oraflash
More details on Oracle Persistent Memory Filestore can be found here.
Test Case
SLOB 2.5.4.0 Oracle I/O workload generation tool kit was chosen as the load generator for this exercise with following SLOB parameters set as below:
UPDATE_PCT=0
RUN_TIME=300
SCALE=200G
WORK_UNIT=32
REDO_STRESS=LITE
We kept the redo stress LITE as the intent of this exercise is to test Smart Flash cache read performance and study the performance improvement using PMEM technology
Test Results
Remember, any performance data is a result of the combination of
- hardware configuration
- software configuration
- test methodology & test tool
- workload profile used in the testing
So, the performance metrics / improvements I got with my workload in my lab is in no way representative of any real production workload , which means the performance improvements for real world workloads will be better. Also this was by no means a benchmarking exercise.
We ran the SLOB load generator with Smart Flash Cache on All-Flash Array v/s Smart Flash Cache on Oracle Persistent Memory Filestore and compare the 2 run database metrics.
We can see, by using Smart Flash Cache on Oracle PMEM Filestore v/s Smart Flash Cache on AFA
- Physical read (blocks) have increased from 145,632.3 per second (Smart Flash Cache on AFA) to 316,665.5 per second (Smart Flash Cache on Oracle PMEM Filestore)
- Read IO Requests have increased from 145,632.3 per second (Smart Flash Cache on AFA) to 316,665.5 per second (Smart Flash Cache on Oracle PMEM Filestore)
- Executes(SQL) have increased from 65,574.4 per second (Smart Flash Cache on AFA) to 142,534.5 per second (Smart Flash Cache on Oracle PMEM Filestore)
From a wait event perspective
- ‘db flash cache single block physical read’ reduced from 56.74 % (Smart Flash Cache on AFA) to 2.94 % (Smart Flash Cache on Oracle PMEM Filestore)
- Average wait time reduction from 2.19ms (Smart Flash Cache on AFA) to 45.40us (Smart Flash Cache on Oracle PMEM Filestore)
- ‘db flash cache multiblock physical read’ reduced from 31.69 % (Smart Flash Cache on AFA) to 6.66 % (Smart Flash Cache on Oracle PMEM Filestore)
- Average wait time reduction from 1.22ms (Smart Flash Cache on AFA) to 220.36us (Smart Flash Cache on Oracle PMEM Filestore)
Same information focusing on ‘db flash cache single block/multiblock physical read’ wait event reduction.
From a Smart Flash Cache performance perspective
- ‘physical read flash cache hits’ increased from 46,256,248 (Smart Flash Cache on AFA) to 100,868,011 (Smart Flash Cache on Oracle PMEM Filestore)
From an overall physical read perspective
- ‘physical read IO requests’ increased from 46,256,908 (Smart Flash Cache on AFA) to 100,868,416 (Smart Flash Cache on Oracle PMEM Filestore)
- ‘physical read requests optimized’ increased from 46,256,275 (Smart Flash Cache on AFA) to 100,868,024 (Smart Flash Cache on Oracle PMEM Filestore)
- ‘physical read total IO requests’ increased from 46,258,299 (Smart Flash Cache on AFA) to 100,869,753 (Smart Flash Cache on Oracle PMEM Filestore)
- ‘physical read total bytes’ increased from 378,958,690,304 (Smart Flash Cache on AFA) to 826,335,252,480 (Smart Flash Cache on Oracle PMEM Filestore)
- ‘physical read total bytes optimized’ increased from 378,931,404,800 (Smart Flash Cache on AFA) to 826,310,852,608 (Smart Flash Cache on Oracle PMEM Filestore)
- ‘physical reads cache’ increased from 46,256,897 (Smart Flash Cache on AFA) to 100,868,403 (Smart Flash Cache on Oracle PMEM Filestore)
From an IOStat by Function perspective:
- ‘Buffer Cache Reads’ increased from 43,065,274 (Smart Flash Cache on AFA) to 92,487,422 (Smart Flash Cache on Oracle PMEM Filestore)
From an OS perspective:
- overall database ‘%IO wait time’ has reduced from 38.3% (Smart Flash Cache on AFA) to 0% (Smart Flash Cache on Oracle PMEM Filestore)
From an OS CPU Utilization perspective:
- ‘overall %sys and %iowait utilization’ is drastically reducedfrom test run on Smart Flash Cache on AFA to test run on Smart Flash Cache on Oracle PMEM Filestore
- %iowait is reduced from 40% average (Smart Flash Cache on AFA) to 0% (Smart Flash Cache on Oracle PMEM Filestore)
From an OS IO request queue depth perspective:
- ‘blocked’ (Number of tasks currently blocked, waiting for I/O to complete) is drastically reduced from average 90% (Smart Flash Cache on AFA) to 0% (Smart Flash Cache on Oracle PMEM Filestore
Summary
In summary, we can see
- From an overall DB IOPS and SQL perspective
- Physical read (blocks) have increased from
- Read IO Requests have increased
- Executes (SQL) have increased
- From a wait event perspective
- ‘db flash cache single block physical read’ reduced
- ‘db flash cache multiblock physical read’ reduced
- From a Smart Flash Cache performance perspective
- ‘physical read flash cache hits’ increased
- From an overall physical read perspective
- ‘physical read IO requests’ increased
- ‘physical read requests optimized’ increased
- ‘physical read total IO requests’ increased
- ‘physical read total bytes’ increased
- ‘physical read total bytes optimized’ increased
- ‘physical reads cache’ increased
- From an IOStat by Function perspective:
- ‘Buffer Cache Reads’ increased
- From an OS perspective:
- overall database ‘%IO wait time’ has reduced
- From an OS CPU Utilization perspective:
- ‘overall %sys and %iowait utilization’ is drastically reduced
- From an OS IO request queue depth perspective
- ‘blocked’ (Number of tasks currently blocked, waiting for I/O to complete) is drastically reduced
Conclusion
- This blog is meant to raise awareness of how performance of Oracle Smart Flash Cache can be accelerated using Persistent Memory technology by using Intel Optane DC PMM in App Direct mode backed Oracle 21c Persistent Memory Filestore on VMware vSphere platform
- This blog contains results that I got in my lab running a load generator SLOB against my workload, which will be way different than any real-world customer workload, your mileage may vary
- Remember, any performance data is a result of the combination of hardware configuration, software configuration, test methodology & test tool and workload profile used in the testing.
- Also, this was by no means a benchmarking exercise.
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html
