Oracle vSAN

Oracle RAC on vSAN 6.7 P01 – No more Eager Zero Thick requirement for shared vmdk’s

Oracle RAC on vSAN 6.7 P01 – No more Eager Zero Thick requirement for shared vmdk’s

 

 

 

 

Read my lips – “No More Eager Zero Thick (EZT) requirement for Oracle RAC multi-writer disks starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)”

 

This blog addresses an important update for running Oracle RAC on VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001).

This applies to other clustered applications as well running on vSAN which uses multi-writer disks for clustering purposes e.g Oracle RAC, Redhat Clustering , Veritas Infoscale etc

 

 

Oracle RAC on VMware platform – Concepts

 

By default, in a VM, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So, in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled. The use case here is Oracle RAC.

 

 

 

Oracle RAC on VMware vSAN prior vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)

 

KB 2121181 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s on VMware vSAN .

2 requirements for sharing disks for an Oracle RAC cluster on VMware vSAN prior vSAN 6.7 P01

  • the shared disk/s have to be Eager Zero Thick provisioned
  • the shared disk/s need to have the multi-writer attribute set

In vSAN , the Eager Zero thick attribute can be controlled by a rule in the VM Storage Policy called ‘Object Space Reservation’ which need to be set to 100% which pre-allocates all object’s components on disk.

More details can be found in KB 2121181.

 

 

 

 

VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)  – release date DEC 5, 2019

 

VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001) release date DEC 5, 2019 , addresses a key limitation when provisioning shared disks with multi-writer for Oracle RAC cluster.

The current limitation is Oracle RAC shared vmdk’s need to be Eager Zero Thick for the multi-writer attribute to be enabled for the clustering software to work. This patch resolves this issue and now one can use thin provisioned vmdk’s with multi-writer attribute for Oracle RAC cluster starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001).

More details can be found here. The text relevant to Oracle RAC cluster is shown as below.

 

PR 2407141: You cannot provision virtual disks to be shared in multi-writer mode as eager zeroed thick-provisioned by using the vSphere Client

A shared virtual disk on a vSAN datastore for use in multi-writer mode, such as for Oracle RAC, must be eager zeroed thick-provisioned. However, the vSphere Client does not allow you to provision the virtual disk as eager zeroed thick-provisioned.  This issue is resolved in this release. You can share any type of virtual disks on the vSAN datastore in multi-writer mode.

 

Now there is only 1 requirement for sharing disks for an Oracle RAC cluster on VMware vSAN vSAN 6.7 P01

  • the shared disk/s need to have the multi-writer attribute set

 

What this means to customers is , with VMware vSAN 6.7 P01

  • Customers NO longer have to provision storage up-front when provisioning space for an Oracle RAC shared storage on VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)
  • RAC shared vmdk’s can be thin provisioned to start with and can slowly consume space as and when needed.

 

The Build numbers and versions of VMware vSAN can be found at KB 2150753

 

 

 

Key points to take away from this blog

 

  • Prior VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001),  Oracle RAC on vSAN requires
    • shared VMDKs to be Eager Zero Thick provisioned (OSR=100)
    • shared VMDKs to have the multi-writer attribute enabled
  • Starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001), Oracle RAC on vSAN does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
    • shared VMDKs can be thin provisioned (OSR=0)
    • shared VMDKs to have the multi-writer attribute enabled
  • For Existing Oracle RAC deployments, with VMware vSAN 6.7 P01 patch
    • Storage Policy change of a shared vmdk from OSR=100 to OSR=0 (and vice-versa if needed) can be performed ONLINE for existing RAC clusters without any RAC downtime

 

This blog focuses on validating the various test cases which was run on

  • 4 node VMware vSAN Clusters 6.7 build-14766710
  • 2 Node 18c Oracle RAC cluster on Oracle Linux Server release 7.6
  • Database Shared storage on Oracle ASM using Oracle ASMLIB

 

Details of the testing can be found below.

 

 

 

 

Setup – Oracle RAC on VMware vSAN 6.7 P01

 

The VMware vSAN ESXi 6.7 Patch Release ESXi670-201912001 setup is shown as below

  • 4 node VMware vSAN Clusters 6.7
  • Each node has 28 CPU’s, 384 GB memory
  • Each node has 2 vSAN disk groups
  • Each disk group has 1x750GB for cache and 1×1.75 TB for capacity, all drives NVMe
  • 2 Node 18c Oracle RAC cluster on Oracle Linux Server release 7.6
  • Database Shared storage on Oracle ASM using Oracle ASMLIB
  • Current shared vmdk Storage Policy – ‘Oracle RAC vSAN Storage Policy’
  • Create a new VM Storage Policy – ‘Oracle RAC vSAN – OSR 0’

 

The storage policies are shown as below. 2 vSAN Storage policies were created – “Oracle RAC vSAN Storage Policy” and “Oracle RAC vSAN – OSR 0”.

  • “Oracle RAC vSAN Storage Policy” was the existing Storage policy for Oracle RAC shared vmdk’s
  • “Oracle RAC vSAN – OSR 0” was the new Storage policy for Oracle RAC shared vmdk’s

 

  • Oracle RAC vSAN Storage Policy
    • Failures to tolerate – 1 failure – RAID-1 (Mirroring)
    • Object space reservation – Thick provisioning

 

 

 

 

  • Oracle RAC vSAN – OSR 0
    • Failures to tolerate – 1 failure – RAID-1 (Mirroring)
    • Object space reservation – Thin provisioning

 

 

 

A clone was created of an existing 2 Node 18c Oracle RAC cluster with cloned RAC VM’s ‘clone_pvrdmarac1’ and ‘clone_pvrdmarac2’ for this test case. Operating system Oracle Linux Server release 7.6, was chosen for the test cases.

 

This RAC cluster had shared storage on Oracle ASM using Oracle ASMLIB. The shared disks for the RAC shared vmdk’s were provisioned with OSR=100 as per KB 2121181.

 

In this RAC Cluster

  • each RAC VM has 8 vCPU’s, 96GB RAM
  • Both running 18c instances with Grid Infrastructure and RDBMS binaries with multi-tenancy option (1 PDB)
  • 1 ASM disk group DATA_DG with 1 ASM Disk of 2TB, EZT with MWF
  • ASM disk group was initially created with storage policy ‘Oracle RAC vSAN Storage Policy’

 

 

 

 

 

Details of RAC VM ‘clone_pvrdmarac1’  shared vmdk with multi-writer attribute and Storage Policy is shown as below. Observe the shared vmdk at SCSI 2:0 of size 2TB.

 

 

 

 

Details of RAC VM ‘clone_pvrdmarac2’  shared vmdk with multi-writer attribute and Storage Policy is shown as below. Observe ‘clone_pvrdmarac1’ shared vmdk at SCSI 2:0 of size 2TB.

 

 

 

 

Looking at the actual space consumed for RAC VM ‘clone_pvrdmarac1’, the shared vmdk shows up as 4TB , reason the VM storage policy had

  • OSR=100 , so Eager Zero Thick provisioned
  • Failures to tolerate (FTT) set to ‘1 failure – RAID-1 (Mirroring)’ , so 2 copies

 

 

All Cluster Services & Resource are up.

 

[root@prdmarac1 ~]# /u01/app/18.0.0/grid/bin/crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE prdmarac1 STABLE
ONLINE ONLINE prdmarac2 STABLE
ora.DATA_DG.dg
ONLINE ONLINE prdmarac1 STABLE
ONLINE OFFLINE prdmarac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE prdmarac1 STABLE
ONLINE ONLINE prdmarac2 STABLE
ora.chad
ONLINE OFFLINE prdmarac1 STABLE
ONLINE OFFLINE prdmarac2 STABLE
ora.net1.network
ONLINE ONLINE prdmarac1 STABLE
ONLINE ONLINE prdmarac2 STABLE
ora.ons
ONLINE ONLINE prdmarac1 STABLE
ONLINE ONLINE prdmarac2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prdmarac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE prdmarac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE prdmarac1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE prdmarac1 169.254.5.190 192.16
8.140.161,STABLE
ora.asm
1 ONLINE ONLINE prdmarac1 Started,STABLE
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
3 ONLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE prdmarac1 STABLE
ora.mgmtdb
1 ONLINE OFFLINE STABLE
ora.prdmarac.db
1 ONLINE ONLINE prdmarac2 Open,HOME=/u01/app/o
racle/product/18.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE prdmarac1 Open,HOME=/u01/app/o
racle/product/18.0.0
/dbhome_1,STABLE
ora.prdmarac1.vip
1 ONLINE ONLINE prdmarac1 STABLE
ora.prdmarac2.vip
1 ONLINE ONLINE prdmarac2 STABLE
ora.qosmserver
1 ONLINE ONLINE prdmarac1 STABLE
ora.scan1.vip
1 ONLINE ONLINE prdmarac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE prdmarac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE prdmarac1 STABLE
——————————————————————————–
[root@prdmarac1 ~]#

 

 

 

Test Suite Setup

 

4 test cases were performed against the RAC cluster. The test cases are

  • Using existing ASM Disk Group (DATA_DG)
    • Change Storage Policy of the shared vmdk’s OSR=100 to OSR=0 online without any RAC downtime
    • Change Storage Policy of the shared vmdk’s OSR=0 to OSR=100 online without any RAC downtime
    • Repeat the above 2 tests again just for consistency sake and check DB and OS error logs

 

SLOB Test Suite 2.4.2 was used with run time of 15 minutes with 128 threads

  • UPDATE_PCT=30
  • SCALE=10G
  • THINK_TM_FREQUENCY=0
  • RUN_TIME=900

 

All test cases were performed and the results were captured . Only Test Case 1 is published as part of this blog. The other test cases had the same exact outcome and hence is not included in this blog.

 

 

 

 

Test Case 1: Online change of Storage Policy for shared vmdk with no RAC downtime

 

 

As part of this test case, while the Oracle RAC was running, the Storage Policy of the shared vmdk was changed ONLINE without any downtime from ‘Oracle RAC vSAN Storage Policy’ to ‘Oracle RAC vSAN – OSR 0’.

 

1) Start SLOB load generator for 15 minutes against the RAC database with 128 threads against the SCAN to load balance between the 2 RAC instances

 

2) Edit settings for RAC VM ‘clone_pvrdmarac1’.

 

 

 

 

 

3) Change storage policy from ‘Oracle RAC vSAN Storage Policy’ to ‘Oracle RAC vSAN – OSR 0’ ie from OSR=100 to OSR=0

 

 

 

 

 

4) Edit settings for RAC VM ‘clone_pvrdmarac2’ and change shared vmdk Storage Policy from ‘Oracle RAC vSAN Storage Policy’ to ‘Oracle RAC vSAN – OSR 0’ ie from OSR=100 to OSR=0

 

 

 

 

 

5) Check both RAC VM’s Storage Policy Compliance

 

 

 

 

6) In the Datastore view,  if we check the RAC VM ‘clone_pvrdmarac1’ shared vmdk size, the Allocated Size is 2 TB but actual size on vSAN datastore is < 2TB.  This indicates that the shared vmdk is indeed thin-provisioned , not thick provisioned.

 

7) The O/S still reports the ASM disks with original size i.e 2 TB

 

[root@prdmarac1 ~]# fdisk -lu /dev/sda

Disk /dev/sda: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xcafbc3d9

Device Boot Start End Blocks Id System
/dev/sda1 2048 4294967294 2147482623+ 83 Linux
[root@prdmarac1 ~]#

[root@prdmarac2 ~]# fdisk -lu /dev/sda

Disk /dev/sda: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xcafbc3d9

Device Boot Start End Blocks Id System
/dev/sda1 2048 4294967294 2147482623+ 83 Linux
[root@prdmarac2 ~]#

 

 

 

8) No errors were observed in the database Alert log files and OS /var/log/messages file for both RAC instances

 

 

 

Summary of the test matrix along with the outcome for the various test cases are as below –

 

 

 

 

 

Summary

 

  • Prior VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001), Oracle RAC on vSAN requires the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
  • Starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001), Oracle RAC on vSAN does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled
    • shared VMDKs can be thin provisioned (OSR=0) for multi-writer mode to be enabled
  • For existing Oracle RAC deployments
    • Storage Policy change of a shared vmdk from OSR=100 to OSR=0 (and vice-versa if needed) can be performed ONLINE without any Oracle RAC downtime
  • All above test cases were performed and the results were captured . Only Test Case 1 is published as part of this blog. The other test cases had the same exact outcome and hence is not included in this blog.

 

All Oracle on vSphere white papers including Oracle on VMware vSphere / VMware vSAN / VMware Cloud on AWS , Best practices, Deployment guides, Workload characterization guide can be found in the url below

 

Oracle on VMware Collateral – One Stop Shop

http://www.vmw.re/oracle-one-stop-shop

 

 

Sudhir Balasubramanian

Sudhir Balasubramanian is a Senior Staff Solution Architect and Global Oracle Field Practice Lead specializing in Oracle on VMware Hybrid Multi-Clouds. In addition , Sudhir also focuses on educating VMware…

Related Articles