I need more and more and …….!!! – The ever-growing demand of Business-Critical workloads especially Oracle workloads where CPU, Memory & IO intensive operations drives critical business transactions to achieve business SLA’ and targets in a timely fashion.
Enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s which can be easily achieved on vSAN 8 vSAN Express Storage Architecture (ESA).
Key points to take away from this blog
This blog is an exercise to showcase the performance advantages of using vSAN 8 vSAN Express Storage Architecture (ESA) for business-critical production Oracle workloads.
This blog also endeavors to compare a synthetic workload generator SLOB against vSAN OSA and vSAN ESA to get an idea of the performance improvement.
Disclaimer – This blog is not meant to be a performance benchmarking-oriented blog in any way. The test results published below are NOT official VMware Performance Engineering test results and in no way reflect real world workload metrics.
Remember, any performance data is a result of the combination of hardware configuration, software configuration, test methodology, test tool, and workload profile used in the testing below – so the performance improvement I got with the synthetic workload generator in my lab is in NO way representative of any real production Customer workload , which actually means the performance improvements for real production Customer workloads will be way much better.
vSAN Express Storage Architecture (ESA)
The vSAN Express Storage Architecture (ESA) is an optional, alternative architecture in vSAN that is designed to process and store data with all new levels of efficiency, scalability, and performance. This optional architecture is optimized to exploit the full capabilities of the very latest in hardware.
It was introduced in vSAN 8, and enhanced in vSAN 8 U1 and U2, it can be selected at the time of creating a cluster. The ESA in vSAN is an alternative to the Original Storage Architecture (OSA) found in all previous editions of vSAN, as well as an optional architecture in the very latest version.
vSAN ESA will unlock the capabilities of modern hardware by adding optimization for high-performance, NVMe-based TLC flash devices with vSAN, building off vSAN’s Original Storage Architecture(vSAN OSA). vSAN was initially designed to deliver highly-performant storage with SATA/SAS devices, the most common storage media at the time.
vSAN 8 will give our customers the freedom of choice to decide which of the two existing architectures (vSAN OSA or vSAN ESA) to leverage to best suit their needs.
More information on vSAN Express Storage Architecture (ESA) can be found at the
- Announcing vSAN 8
- vSAN Frequently Asked Questions (FAQ)
- vSAN Express Storage Architecture (ESA)
- An Introduction to the vSAN Express Storage Architecture
vSAN Express Storage Architecture (ESA) – New high-performance RAID – 5/6 erasure coding
Customers can expect RAID-5/6 to perform equal to RAID-1 with vSAN ESA. The new architecture will deliver optimal levels of resilience that are also space efficient while offering the highest levels of performance. All of this can be accomplished using RAID-6 with the assistance of the new log-structured filesystem and object format.
More information on vSAN ESA New high-performance RAID – 5/6 erasure coding can be found at in Announcing vSAN 8.
Test Bed
The Test bed is an 8 Node vSAN 8 Express Storage Architecture (ESA) Cluster with the setup as shown below –
- vCenter version was 8.0.2 build 22385739
- 8 Lenovo ThinkAgile VX7531 Node servers, 2 sockets, 28 cores per socket, Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz, 1TB RAM [ 2 physical NUMA nodes, each NUMA node with 28 cores and 512G ]
- VMware ESXi, 8.0.2, 22380479 with vSAN ESA All NVMe Storage
- Oracle 21.13 with Grid Infrastructure, ASM Storage and Linux udev (8k database block size)
- OEL UEK 8.9
Details of vSAN Express Storage Architecture (ESA) cluster ‘env175’ are shown as below. The ESA Cluster has 8 Lenovo ThinkAgile VX7531 Node servers as shown below.
Each Lenovo ThinkAgile VX7531 server has 2 sockets, 28 cores per socket, Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz, 1TB RAM as shown below.
Each Lenovo ThinkAgile VX7531 server has 6 internal NVMe drives, each server contributing 6 Internal NVMe drives to the vSAN ESA datastore capacity as shown below.
Lenovo ThinkAgile VX7531 server ‘env175-node1.pse.lab’ HBA & Internal NVMe drive details are as shown below.
Lenovo ThinkAgile VX7531 server ‘env175-node1.pse.lab’ networking details are shown as below. Every node in this cluster has 2 x 25Gb NIC (Management , vMotion, VM Network etc.) and 1x100Gb NIC for vSAN Traffic.
Lenovo ThinkAgile VX7531 server ‘env175-node1.pse.lab’ details are shown as below.
vSAN Express Storage Architecture (ESA) cluster Storage policies are shown as below.
Baseline Oracle Policy ‘Oracle ESA – FTT0 – NoRAID´ was created to JUST get baseline metrics for comparison purposes, besides that, no true production workload is ever setup with FTT=0 (Failures to Tolerate) and No RAID (no protection).
- Oracle ESA – FTT0 – NoRAID
- Oracle ESA – FTT1 – R5
- Oracle ESA – FTT2 – R6
Oracle VM ‘Oracle21C-OL8-DB_ESA’ details are shown as below.
The VM has 28 vCPU’s, 256 GB RAM, the single instance database ‘ORA21C’ was created with created with multi-tenant option & provisioned with Oracle Grid Infrastructure (ASM) and Database version 21.13 on O/S OEL UEK 8.9.
Oracle ASM was the storage platform with Linux udev for device persistence. Oracle SGA & PGA set to 64G and 10G respectively.
The database was deliberately set to NOARCHIVELOG mode as the intention was to drive maximum amount of IO without stressing ARCHIVELOG.
All Oracle on VMware platform best practices were followed.
Oracle VM ‘Oracle21C-OL8-DB_ESA’ has 4 vNVMe Controllers for added performance.
58 are assigned to the Oracle VM ‘Oracle21C-OL8-DB_ESA’ which are shown as below –
- NVME 0:0 – 80G for OS (/)
- NVME 0:1 – 80G for Oracle Grid Infrastructure & RDBMS binaries
- NVME 0:2 – 100G for GRID_DG
- NVME 0:3 – 200G for DATA_DG
- NVME 0:4 – 200G for DATA_DG
- NVME 0:5 – NVME 0:12 – 8 vmdk’s, each 25G – REDO_DG
- NVME 1:0 – 1:14 – 15 vmdk’s, each 50GB – SLOB_DG
- NVME 2:0 – 2:14 – 15 vmdk’s, each 50GB – SLOB_DG
- NVME 3:0 – 3:14 – 15 vmdk’s, each 50GB – SLOB_DG
Oracle VM ‘Oracle21C-OL8-Customer’ vmdk’s storage policy details are as below
- Test 1 – Test run was with ALL vmdk’s with Storage Policy ‘Oracle ESA – FTT0 – NoRAID’
- Test 2 – Test run was with ALL vmdk’s with Storage Policy ‘Oracle ESA – FTT1 – R5’
- Test 3 – Test run was with ALL vmdk’s with Storage Policy ‘Oracle ESA – FTT2 – R6’
Oracle ASM Disk Group details are as below:
grid@oracle21c-ol8-nvme:+ASM:/home/grid> asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 409600 117916 0 117916 0 N DATA_DG/
MOUNTED EXTERN N 512 512 4096 4194304 102400 102296 0 102296 0 N GRID_DG/
MOUNTED EXTERN N 512 512 4096 1048576 204800 140614 0 140614 0 N REDO_DG/
MOUNTED EXTERN N 512 512 4096 1048576 2304000 124998 0 124998 0 N SLOB_DG/
grid@oracle21c-ol8-nvme:+ASM:/home/grid>
ASM Diskgroup ‘SLOB_DG’ has 45 vmdk’s , each vmdk 50GB size with a total size of 2250GB (AU=1MB).
Test Use case
This blog is an exercise to showcase the performance advantages of using vSAN 8 vSAN Express Storage Architecture (ESA) for business-critical production Oracle workloads.
SLOB workload generator was run against a 2 TB SLOB tablespace ( 85 datafiles , each 25GB size , total size 2125G ) with 3 different storage policies.
- Oracle ESA – FTT0 – NoRAID
- Oracle ESA – FTT1 – R5
- Oracle ESA – FTT2 – R6
SLOB 2.5.4.0 was chosen as the load generator for this exercise with following SLOB parameters set as below:
- UPDATE_PCT=30
- RUN_TIME=300
- SCALE=25G
- WORK_UNIT=1024
Multiple SLOB schemas ( 80 schemas , 25G each , total size 2000GB ) were used to simulate a multi-schema workload model and the number of threads per schema was set to 20.
Work Unit size was set to 1024 to drive the maximum amount of IO without stressing REDO to study the performance metrics differences between the 3 different storage policies on vSAN ESA.
We ran multiple SLOB runs for the above use case and compared the Oracle metrics as shown in the table below.
Baseline Oracle Policy ‘Oracle ESA – FTT0 – NoRAID´ was created to JUST get baseline metrics for comparison purposes, besides that, no true production workload is ever setup with FTT=0 (Failures to Tolerate) and No RAID (no protection).
We can see that vSAN Express Storage Architecture (ESA) New high-performance RAID 5/6 erasure coding database metrics is comparable to the baseline.
As mentioned earlier, Customers can expect RAID-5/6 to perform equal to RAID-1 with vSAN ESA. The new architecture will deliver optimal levels of resilience that are also space efficient while offering the highest levels of performance. All of this can be accomplished using RAID-6 with the assistance of the new log-structured filesystem and object format.
Diving deeper into core database load profiles, we see similar database metrics for Executes (SQL), Transactions, Read/Write IOPS and Read/Write MB/secs for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs
- Executes (SQL) / second & Transactions / second are comparable across all 3 runs
- Read IO requests / second & Write IO requests / second are comparable across all 3 runs
- Read IO (MB) / second and Write IO (MB) / second are comparable across all 3 runs
Diving deeper into core database wait events, we see similar database wait events with similar wait times for ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
Looking at GOS statistics, we can see comparable performance metrics for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
In conclusion, we were able to get comparable performance metrics from both overall database and GOS perspective for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
OSA v/s ESA – Performance Improvements – based on my lab testing
As indicated earlier, this blog is not meant to be a performance benchmarking-oriented blog in any way.
The Final test was to understand the performance improvements offered by vSAN ESA over vSAN OSA using the the same SLOB workload.
The same vSAN ESA setup was re-purposed for vSAN OSA by tearing down vSAN ESA , using the same servers and the same set of internal devices in those servers and vSAN OSA was setup.
Details of the vSAN OSA are shown as below.
SLOB workload generator was run against a 2 TB SLOB tablespace ( 85 datafiles , each 25GB size , total size 2125G ) with 3 different storage policies.
- Oracle ESA – FTT0 – NoRAID
- Oracle ESA – FTT1 – R5
- Oracle ESA – FTT2 – R6
SLOB 2.5.4.0 was chosen as the load generator for this exercise with following SLOB parameters set as below:
- UPDATE_PCT=30
- RUN_TIME=300
- SCALE=25G
- WORK_UNIT=1024
Baseline Oracle Policy ‘Oracle ESA – FTT0 – NoRAID´ was created to JUST get baseline metrics for comparison purposes, besides that, no true production workload is ever setup with FTT=0 (Failures to Tolerate) and No RAID (no protection).
We compared the above metrics we got for vSAN ESA with the metrics for vSAN OSA as shown below.
We can see that vSAN Express Storage Architecture (ESA) New high-performance RAID 5/6 erasure coding database metrics has significant performance benefits compared to the vSAN OSA database metrics.
As mentioned earlier, Customers can expect RAID-5/6 to perform equal or even better to RAID-1 with vSAN ESA. The new architecture will deliver optimal levels of resilience that are also space efficient while offering the highest levels of performance. All of this can be accomplished using RAID-6 with the assistance of the new log-structured filesystem and object format.
In conclusion, we were able to get significant performance metrics from an overall database perspective for vSAN ESA database runs as compared to vSAN OSA.
Summary
This blog is an exercise to showcase the performance advantages of using vSAN 8 vSAN Express Storage Architecture (ESA) for business-critical production Oracle workloads.
This blog also endeavors to compare a synthetic workload generator SLOB against vSAN OSA and vSAN ESA to get an idea of the performance improvement.
Disclaimer – This blog is not meant to be a performance benchmarking-oriented blog in any way. The test results published below are NOT official VMware Performance Engineering test results and in no way reflect real world workload metrics.
Remember, any performance data is a result of the combination of hardware configuration, software configuration, test methodology, test tool, and workload profile used in the testing below – so the performance improvement I got with the synthetic workload generator in my lab is in NO way representative of any real production Customer workload , which actually means the performance improvements for real production Customer workloads will be way much better.
Baseline Oracle Policy ‘Oracle ESA – FTT0 – NoRAID´ was created to JUST get baseline metrics for comparison purposes, besides that, no true production workload is ever setup with FTT=0 (Failures to Tolerate) and No RAID (no protection.
We can see that vSAN Express Storage Architecture (ESA) New high-performance RAID 5/6 erasure coding database metrics is comparable to the baseline.
As mentioned earlier, Customers can expect RAID-5/6 to perform equal to RAID-1 with vSAN ESA. The new architecture will deliver optimal levels of resilience that are also space efficient while offering the highest levels of performance. All of this can be accomplished using RAID-6 with the assistance of the new log-structured filesystem and object format.
Diving deeper into core database load profiles, we see similar database metrics for Executes (SQL), Transactions, Read/Write IOPS and Read/Write MB/secs for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs
- Executes (SQL) / second & Transactions / second are comparable across all 3 runs
- Read IO requests / second & Write IO requests / second are comparable across all 3 runs
- Read IO (MB) / second and Write IO (MB) / second are comparable across all 3 runs
Diving deeper into core database wait events, we see similar database wait events with similar wait times for ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
Looking at GOS statistics, we can see comparable performance metrics for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
In conclusion, we were able to get comparable performance metrics from both overall database and GOS perspective for both ‘Oracle ESA – FTT1 – R5’ and ‘Oracle ESA – FTT2 – R6’ storage policy runs.
Re-running the same SLOB workload against a vSAN OSA setup, comprising of the same set of servers and same internal drives, we can see that vSAN Express Storage Architecture (ESA) New high-performance RAID 5/6 erasure coding database metrics has significant performance benefits compared to the vSAN OSA database metrics.
In conclusion, we were able to get significant performance metrics from an overall database perspective for vSAN ESA database runs as compared to vSAN OSA.
Acknowledgements
This blog was authored by Sudhir Balasubramanian, Senior Staff Solution Architect & Global Oracle Lead – VMware.
Conclusion
The ever-growing demand of Business-Critical workloads especially Oracle workloads where CPU, Memory & IO intensive operations drives critical business transactions to achieve business SLA’ and targets in a timely fashion.
Enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s which can be easily achieved on vSAN 8 vSAN Express Storage Architecture (ESA).
All Oracle on VMware vSphere collaterals can be found in the url below.
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html
