Business Critical Oracle Workloads have stringent IO requirements and enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s, all the way from on-premises to VMware Hybrid Clouds
With increasing need for data to be “fast and furious”, customers are turning to NVMe transfer protocol for accessing data quickly from flash memory storage devices such as solid-state drives (SSDs).
NVMe, or nonvolatile memory express, is a storage access protocol and transport protocol for flash and next-generation solid-state drives (SSDs) that delivers high throughput and fast response times for a variety of traditional and modern workloads. While all-NVMe server-based storage has been available for years, the use of non-NVMe protocols for storage networking can add significant latency.
To address this performance bottleneck, NVMe over TCP or NVMe/TCP, extends NVMe’s performance and latency benefits across TCP based network fabric.
We’ve heard from our customers that they would like to run NVMe/TCP because:
- TCP/IP is ubiquitous
- Well understood – TCP is probably the most common transport
- High performance – TCP delivers excellent performance scalability
- Well suited for large scale deployments and longer distances
- Actively developed – maintenance and enhancements are developed by major players
- Inherently supports in-transit encryption
More information on NVMe/TCP Support with VMware vSphere 7 Update 3 can be found here.
More information on VMware NVMe Concepts can be found here.
Key points to take away from this blog
This blog is not meant to be a performance benchmarking-oriented blog.
This blog is meant to raise the awareness of the simplicity and seamless use of NVMe/TCP storage backed datastores provided by Lightbits Storage to provide vmdk based storage for Oracle workloads.
Oracle workloads can take advantage of NVMe/TCP storage using VM vNVME controller backed vmdk’s on those NVMe/TCP storage enhancing performance with high throughput, low latency, fast response times and reduced CPU utilization.
From an Oracle workload or pretty much any Application on VMware platform, from a functional perspective, a vmdk carved form a NVMe/TCP backed Datastore is just a regular vmdk which then gets attached to the VM and gets consumed the same way as any regular vmdk on a VMFS /vVOL / NFS / vSAN datastore.
NVMe/TCP vSphere Support Extends to Lightbits Labs Storage with Intel
As a VMware Technology Alliance Partner and early design partner, Intel and Lightbits collaborated extensively with VMware on development and testing of the new vSphere NVMe/TCP feature.
The Lightbits software is fully optimized to run on Intel’s platform breadth of high-performance technologies for the data center such as Intel Xeon Scalable processors and Intel’s Ethernet 800 series Network Adapters (Optionally, Lightbits can also take advantage of Intel’s Optane Persistent Memory). All these technologies are now certified by VMware’s Day 0 software-defined storage solution for VMware vSphere 7 Update 3 release
Lightbits storage solution, running on Intel technologies, combined with vSphere with inbox support for NVMe/TCP delivers an end-to-end NVMe solution. Organizations with private clouds and hybrid clouds, as well as cloud service providers can now realize the performance, scalability, and cost-efficiency benefits of a combined solution from VMware, Lightbits, and Intel.
More information on NVMe/TCP with VMware vSphere & Lightbits Storage can be found at High Performance Disaggregated Storage for VMware
Lightbits Cloud Data Platform
Lightbits unique disaggregated software-defined architecture combines the advantages of an NVMe/TCP data fabric with Intelligent Flash Management and enterprise data services in an easy to consume solution on commodity hardware and standard networking
Lightbits software is a software-defined block storage solution offering hyperscale efficiency, performance and flexibility. It delivers composable, high performance, scale-out and redundant NVMe over TCP (NVMe/TCP) storage that performs like local flash.
Lightbits software for VMware provides a seamless integration via the vCenter plugin allowing applications and workloads to consume high-performance feature-rich NVMe storage for virtual machines and manage the LightOS cluster.
More Information on Lightbits software can be found here.
Read how to “Harness The Power Of NVMe/TCP To Get Highly Available Storage For Virtualized Applications” here.
Oracle Storage using vmdk’s on Lightbits NVMe/TCP storage
Oracle workloads on VMware platform can take advantage of various storage options available which includes running Oracle workloads on VMware datastores backed by Lightbits NVMe/TCP storage.
More information on storage options available for Oracle workloads on VMware platform can be found here.
From an Oracle workload perspective, once a vmdk is created on a Lightbits NVMe/TCP storage backed VMware Datastore and added to a Linux VM, when we login to the guest operating system, we have a choice of either using the newly created Linux device as
- File system
- Oracle Automatic Store Management (ASM) disk.
The rest of steps in partitioning the disks, creating ASM devices on them, adding them to ASM disk groups and creating Single Instance / RAC database is the same as in any physical architecture.
Test Use case
This test bed use case showcases the validation of Oracle Single Instance workloads for this setup below
- VMware ESXi, 8.0.0 Build 20513097
- Lightbits software
- Oracle 21.5 with Grid Infrastructure, ASM Storage and ASMLIB on Oracle Enterprise Linux (OEL) 8.7
Test Bed
The Test bed is a 3 Node vSphere Cluster with 3 ESXi servers, all version ESXi, 8.0.0 Build 20513097.
The server names are ‘sc2esx67.vslab. local’, ‘sc2esx68.vslab. local’ and ‘sc2esx69.vslab. local’.
The 3 ESXi servers are Intel NF5280M6 Ice Lake Servers, each server has 2 sockets, 32 cores per socket, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, with 256GB RAM.
The vCenter version was 8.0.0 build 20519528.
The 3 ESXi client servers are connected to the Lightbits servers via a 100Gbps Mellanox switch as shown below
Lightbits software for VMware provides a seamless integration via the vCenter® plugin allowing applications and workloads to consume high-performance feature-rich NVMe storage for virtual machines and manage the LightOS cluster.
Port binding for NVMe over TCP involves creating a virtual switch and connecting the physical network adapter and the VMkernel adapter to the virtual switch. Through this connection, the TCP adapter becomes bound to the VMkernel adapter.
The details of the VMkernel adapter and physical network adapter for server ‘sc2esx67’ are shown as below. The other 2 servers are configured in the same way.
More information on the port binding can be found here
The next step is to enable NVMe over TCP software adapters by installing an adapter that supports NVMe over TCP adapter.
The next step is to enable NVMe over TCP software adapters by installing an adapter that supports NVMe over TCP adapter. Details about server ‘sc2esx67’ as shown below. The other 2 servers are configured in the same way.
More information on how to enable NVMe over TCP software adapters can be found here.
Click on the Lightbits plugin to see the Lightbits Cluster setup as shown below.
The Cluster has the above ESXi servers as the connected ESXi ‘Client’ hosts.
The NVMe/TCP storage and datastore details are shown as below
Server ‘sc2esx67’ NVMe/TCP storage adapter details are shown as below. Servers ‘sc2esx68’ and ‘sc2esx69’ are set up the same.
Oracle VM ‘Oracle21C-OL8-Customer-NVMeTCP’ details are shown as below.
The VM has 16 vCPU’s, 100GB RAM, the single instance database ‘ORA21C’ was created with created with multi-tenant option & provisioned with Oracle Grid Infrastructure (ASM) and Database version 21.5 on O/S OEL 8.7 UEK.
Oracle ASM was the storage platform with Oracle ASMLIB for device persistence. Oracle SGA & PGA set to 80G and 10G, respectively.
All Oracle on VMware platform best practices were followed.
The vmdk’s for the Oracle VM ‘Oracle21C-OL8-Customer-NVMeTCP’ are shown as below –
- Hard Disk 1 (SCSI 0:0) – 80G for OS (/)
- Hard Disk 2 (SCSI 0:1) – 80G for Oracle Grid Infrastructure and RDBMS binaries
- Hard Disk 3 (SCSI 1:0) – 100G for ASM Disk Group GRID_DG
- Hard Disk 4 (SCSI 1:1) – 200G for ASM Disk Group DATA_DG
- Hard Disk 5 (NVME 0:0) – 1 TB for ASM Disk Group SLOB_DG
- Hard Disk 6 (SCSI 3:0) – 100G for ASM Disk Group REDO_DG
Details of Hard Disk 5 vmdk on NVMe/TCP Lightbits NVMe/TCP storage is shown as below.
From an Oracle workload or pretty much any Application on VMware platform, from a functional perspective, a vmdk carved form a NVMe/TCP backed Datastore is just a regular vmdk which then gets attached to the VM and gets consumed the same way as any regular vmdk on a VMFS /vVOL / NFS / vSAN datastore.
Oracle ASM disk details are as below:
[root@oracle21c-ol8 ~]# oracleasm listdisks
DATA_DISK01
GRID_DISK01
NVME_TCP_DISK01
REDO_DISK01
[root@oracle21c-ol8 ~]#
Oracle ASM disk group details are as below:
The OS details are shown as below
Summary
This blog is not meant to be a performance benchmarking-oriented blog.
This blog is meant to raise the awareness of the simplicity and seamless use of NVMe/TCP storage backed datastores provided by Lightbits Storage to provide vmdk based storage for Oracle workloads.
Oracle workloads can take advantage of NVMe/TCP storage using VM vNVME controller backed vmdk’s on those NVMe/TCP storage enhancing performance with high throughput, low latency, fast response times and reduced CPU utilization.
From an Oracle workload or pretty much any Application on VMware platform, from a functional perspective, a vmdk carved form a NVMe/TCP backed Datastore is just a regular vmdk which then gets attached to the VM and gets consumed the same way as any regular vmdk on a VMFS /vVOL / NFS / vSAN datastore.
This test bed use case showcases the validation of Oracle Single Instance workloads for this setup below
- VMware ESXi, 8.0.0 Build 20513097
- Lightbits software
- Oracle 21.5 with Grid Infrastructure, ASM Storage and ASMLIB on Oracle Enterprise Linux (OEL) 8.7
Acknowledgements
This blog was authored by Sudhir Balasubramanian, Senior Staff Solution Architect & Global Oracle Lead – VMware.
Many thanks to the following for proving their invaluable support in this effort
- Sagy Volkov & Gagan Gill, Lightbits
Conclusion
Business Critical Oracle Workloads have stringent IO requirements and enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s, all the way from on-premises to VMware Hybrid Clouds
With increasing need for data to be “fast and furious”, customers are turning to NVMe transfer protocol for accessing data quickly from flash memory storage devices such as solid-state drives (SSDs).
NVMe, or nonvolatile memory express, is a storage access protocol and transport protocol for flash and next-generation solid-state drives (SSDs) that delivers high throughput and fast response times for a variety of traditional and modern workloads. While all-NVMe server-based storage has been available for years, the use of non-NVMe protocols for storage networking can add significant latency.
To address this performance bottleneck, NVMe over Fabrics, or NVMe-oF, extends NVMe’s performance and latency benefits across network fabrics, including Ethernet.
We’ve heard from our customers that they would like to run NVMe/TCP because:
- TCP/IP is ubiquitous
- Well understood – TCP is probably the most common transport
- High performance – TCP delivers excellent performance scalability
- Well suited for large scale deployments and longer distances
- Actively developed – maintenance and enhancements are developed by major players
- Inherently supports in-transit encryption
All Oracle on VMware vSphere collaterals can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html
