Much has been discussed and documented about Storage vMotion of Oracle RAC Clusters on VMware SDDC which can be found in the numerous blog posts available at Oracle on VMware Collateral – One Stop Shop.
Development, QA , Pre-Production RAC clusters form an important part of any RAC deployment environment and while they may not be as large as the production RAC clusters, they are deemed equally critical for the software life cycle process . Fortunately they are not constrained by the 100% uptime business SLA’s which means admins can choose to go another route of storage vMotioning Oracle RAC with minimal downtime in a very short time.
This blog focuses on storage vMotion of a 2 Node non-production Oracle RAC Cluster , Same vSphere Cluster, different Storage Array , with minimal downtime.
Let’s us first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion.
Oracle layout on vSphere – Single Instance and RAC
There are 3 main Oracle database deployment models, all of which are supported and can be deployed on the VMware SDDC stack.
- Oracle Single Instance
- Oracle Real Application Cluster
- Oracle RAC one node (uses the same concept of sharing vmdk/s using multi-writer flag)
The primary difference between Oracle Single instance and Oracle RAC vmdk/s setup is the vmdk sharing mode.
There are 3 values of disk sharing mode
- Unspecified
- No sharing
- Multi-writer – used for Clustering
Both Single Instance VM and RAC VM/s will have 2 genres of vmdk/s:
- non-database vmdk/s
- Operating System vmdk which houses the root file system (/)
- Oracle binaries vmdk which houses the Oracle binaries (Grid and RDBMS binaries) (/u01)
- database vmdk/s
- non-shared vmdk/s in case of Single Instance
- shared vmdk/s in case of RAC Cluster
- non-shared vmdk/s in case of Single Instance
VMware Storage vMotion
With VMware Storage vMotion technology, we can migrate VM’s to relocate the configuration file of a virtual machine and virtual disks, while the virtual machine is powered on, from one datastore to another.
The vSphere documentation details Storage vMotion Requirements and Limitations
Caveats with Multi-writer disks and Storage vMotion
The multi-writer capability for sharing vmdk/s between VM’s play an important role in Oracle RAC deployments on VMware SDDC.
VMFS is a clustered file system that disables (by default) multiple VM’s from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file..
The multi-writer option allows VMFS-backed disks to be shared by multiple VM’s. This option can be used to disable the protection for certain cluster-aware applications e.g Oracle Clusterware , where the applications ensure that writes originating from two or more different VM’s does not cause data loss and ensures data consistency and concurrency.
Among the unsupported actions for shared vmdk/s using multi-writer capability includes
- Storage vMotion
- Snapshots
- Changed Block Tracking (CBT)
Attempting a storage vMotion of an RAC VM will result in an error:
The supported and unsupported actions / features using multi-writer attribute can be found in the KB 1034165.
More information is available in the blog post Around the “Storage World” in no time – Storage vMotion and Oracle Workloads .
Storage vMotion of an Oracle RAC Cluster without ANY downtime
The blog Storage vMotion for Production Oracle RAC Workloads with NO downtime – Same vSphere Cluster , Different Storage Array details how we can Storage vMotion for Production Oracle RAC Workloads with NO downtime – Same vSphere Cluster , Different Storage Array.
The takeaways of this solution are
- Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk and will NOT migrate the Operating System or Oracle binaries vmdk.
- VMware storage vMotion will migrates ALL vmdk/s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but can ONLY be used for non-shared vmdk/s [KB 1034165]
- We can deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
Storage vMotion of an Oracle RAC Cluster with minimal downtime
For Development, QA , Pre-Production RAC clusters which are not constrained by the 100% uptime business SLA’s, admins can choose to go another route of storage vMotioning Oracle RAC with minimal downtime.
The steps to migrate a 2-node Oracle RAC with VM’s “VM1″ & VM2” hosting the 2 RAC instances respectively, with shared VMDKs with multi-writer attribute using vSphere Storage vMotion are outlined as follows:
- Shut down RAC cluster services on both RAC VMs (VM1 and VM2)
- Power off both RAC VMs (VM1 and VM2)
Make a copy the .vmx file for all RAC VM’s – append a subscript ‘ORIG’ at the end of all VM vmx files e.g cp VM2.vmx VM2.vmx.ORIG
Run the command on any RAC VM and save the output to a file , for e.g – cat VM1.vmx | grep -I multi-writer > multi-writer.out. The contents of the ‘multi-writer.out’ file will have entries containing the multi-writer setting e.g scsiX:Y.sharing = “multi-writer”
Save both files .vmx.ORIG and multi-writer.out to a backup location
- VM2 – Using Web Client , remove, but do not delete the disks , the shared VMDKs with multi-writer attribute only. leave the non-shared vmdk’s alone
- VM2 – Migrate VM2 with non-shared VMDKs to a different vSphere Cluster / different Datastore via vSphere Storage vMotion. On successful completion of Storage vMotion, VM2 is now on a different vSphere Cluster / different Datastore
- VM1 – Using Web Client, Remove the multi-writer attribute from VM1 shared EZT VMDKs – do not delete any shared vmdk’s
- VM1 – Migrate VM1 with both non-shared disks and original shared disks to same vSphere Cluster / Datastore as VM2 via vSphere Storage vMotion. On successful completion of Storage vMotion, VM1 is now on same vSphere Cluster / Datastore as VM2
- Observation – EZT-shared VMDKs will be converted to LZT when you migrate to a vSAN datastore using default Storage vMotion options. Beginning with VMware vSAN 6.7 Patch P01 (ESXi 6.7 Patch Release ESXi670-201912001), Oracle RAC on vSAN does NOT require shared VMDKs to be EZT (OSR=100) for multi-writer mode to be enabled
- VM1 – Using Web Client, Add multi-writer attribute to VM1 shared VMDKs. Power on VM1 and start cluster services to check the cluster consistency. After successful test of cluster consistency, power off VM1.
- VM2 -Using Web Client, Add the VM1 shared VMDKs with multi-writer attribute to VM2 at the same SCSI position as VM1.
- Power on VM1 and VM2.
- Start RAC Cluster services on VM1 and VM2.
This concept can be extended to an n-node Oracle RAC cluster in which n is greater or equal to two.
In vSphere 6.0, there is now a new sharing attribute on the Virtual Disk backing property which accepts one of two values: sharingMultiWriter or sharingNone for specifying the Multiwriter flag.
More can be read here.
Migration steps
We had a fully functional 2 Node Oracle RAC 12.2.0.1.0 Cluster running on OEL 7.4 with vmdk/s on NFS VAAI compliant datastore on VMware vSphere 6.5 . We would like to migrate this Oracle RAC cluster to a different storage to run some tests against the new storage.
The RAC VM’s are ‘pvrdmarac1’ and ‘pvrdmarac2’.
1. First , backup both RAC VM’s ‘pvrdmarac1’ and ‘pvrdmarac2’ .vmx files to a backup location.
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] cp pvrdmarac1.vmx pvrdmarac1.vmx.ORIG
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac2] cp pvrdmarac2.vmx pvrdmarac2.vmx.ORIG
2. Extract and save the multi-writer lines, from any VM .vmx file to another file , say ‘multi-writer.out’
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] grep -i multi pvrdmarac1.vmx
scsi3:0.sharing = “multi-writer”
scsi3:1.sharing = “multi-writer”
scsi3:2.sharing = “multi-writer”
scsi3:3.sharing = “multi-writer”
scsi3:4.sharing = “multi-writer”
scsi3:5.sharing = “multi-writer”
scsi3:6.sharing = “multi-writer”
scsi3:8.sharing = “multi-writer”
scsi1:2.sharing = “multi-writer”
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1]
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] grep -i multi pvrdmarac1.vmx > multi-writer.out
3. Shutdown Oracle RAC Cluster on all RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’
[root@pvrdmarac1 ~]# /u01/app/12.2.0/grid/bin/crsctl stop cluster -all
4. Power ALL RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’ , down
5. Edit VM ‘pvrdmarac2’ settings and remove ALL shared devices from VM ‘pvrdmarac2’. Do not delete ANY of the vmdk/s from storage.
6. Now storage vMotion VM ‘pvrdmarac2’ to the new storage. Remember, pvrdmarac2.vmx.ORIG file will NOT get copied over , so move it manually somewhere safe.
7. Remove multi-writer settings from all the shared disks on VM ‘pvrdmarac1’. This can be done in one of the two ways below.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac1’ , navigate to the shared vmdk/s and choose ‘No sharing’ option from the drop-down box
Do the same step for ALL shared vmdk/s.
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac1’ folder. Using the vi editor , edit pvrdmarac1.vmx file and remove all lines containing the word ‘multi-writer ‘ from vmx file and save the .vmx file.
8. Storage vMotion VM ‘pvrdmarac1’ to new storage
9. Once the storage vMotion of VM ‘pvrdmarac1’ completes successfully , restore multi-writer setting for all common shared vmdk/s to VM ‘pvrdmarac1’. This can be done in one of the two ways below.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac1’ , navigate to the shared vmdk/s and choose ‘Multi-writer’ option from the drop-down box
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac1’ folder. Append the contents of the saved ‘multi-writer.out’ file to the end of ‘pvrdmarac1.vmx’ file
[root@wdc-esx27:/vmfs/volumes/XXXXXXXX/pvrdmarac1] cat multi-writer.out >> pvrdmarac1.vmx
10 . Now add back the shared disks with the multi-writer settings to VM ‘pvrdmarac2’.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac2’ , choose ‘Existing Hard Disk’, navigate to the VM ‘pvrdmarac1’ folder where the vmdk is located and add it to the same SCSI X:Y slot as is present in VM ‘pvrdmarac1’.
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac2’ folder and copy the contents of VM ‘pvrdmarac2.vmx.ORIG’ file to ‘pvrdmarac2.vmx’ file. Next , edit ‘pvrdmarac2.vmx’ file and change storage location from original NFS datastore location to the new datastore location. Then , append the contents of the saved ‘multi-writer.out’ file to the end of ‘pvrdmarac2.vmx’ file as done earlier for ‘pvrdmarac1.vmx’ file
11. Startup ALL the RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’ , Click OK for option ‘I copied it’ when prompted. Bring up all RAC serices as needed.
Oracle RAC with 2 VM’s ‘pvrdmarac1’ and ‘pvrdmarac2’ has now been successfully moved to the new datastore location.
Summary
Storage Migration of an Oracle RAC Cluster on VMware SDDC is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database
- For production Oracle RAC with stringent SLA’s, follow the steps detailed in the blog Around the “Storage World” in no time – Storage vMotion and Oracle Workloads
- For non-production Oracle RAC with no stringent SLA’s, follow the steps detailed in this blog post
Conclusion
Business Critical Applications especially IO intensive Oracle workloads have a need for storage migration, based on the ever-demanding workload profile, from one Tier to another Tier within a storage array or between storage arrays.
VMware storage vMotion technology can be used to migrate non-production Oracle RAC with minimal downtime in a very short time.
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found at Oracle on VMware Collateral – One Stop Shop
