“Around the World in Eighty Days” – A classic adventure novel and one of Jules Verne’s most acclaimed works which describes how Phileas Fogg and his valet Passepartout attempt to circumnavigate the world in 80 days on a wager.
In the world of Business Critical Applications, especially IO intensive Oracle workloads, there is always a need for storage migration, based on the ever demanding workload profile.
For example –
- Migrate Oracle database storage from one Tier to another Tier within a storage array , accessed by the same vSphere Cluster
- Migrate Oracle database storage from one array to another array (within and between data centers) for the same datastore type [ VMFS . NFS, iSCSI , vVOL , vSAN] ,accessed by the same vSphere Cluster
- Migrate Oracle database storage from one array to another array (within and between data centers) across different datastore types [ VMFS . NFS, iSCSI , vVOL , vSAN] , accessed by the same vSphere Cluster
The storage migration is made seamless across the various permutations and combinations, simply by using vmdk’s , the common denominator of VM storage across all above flavors of VMware datastores.
This article describes how storage migration can be performed for Oracle database running on VMware vSphere, for any types of underlying storage. The migration is done using a combination of two key technologies depending on the database deployment model
- VMware Storage vMotion
- Oracle Automatic Storage Management (ASM).
Before describing this technique, we first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion
This blog focused on how we can storage vMotion an Oracle RAC Cluster , from one datastore to another datastore within a storage array OR between 2 storage arrays, accessed by the same vSphere Cluster , without ANY downtime.
Key Takeaways
- Storage Migration of an Oracle RAC Cluster from one datastore to another datastore , within an storage array OR between 2 storage arrays, on a VMware SDDC , is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database. and VMware storage vMotion alone cannot be used as explained below.
- VMware storage vMotion will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but will not storage vMotion clustered vmdk’s, it can ONLY be used for non-shared vmdk’s [KB 1034165]
- Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk BUT will NOT migrate the Operating System or Oracle binaries vmdk.
- This blog focused on how we can storage vMotion an Oracle RAC Cluster without ANY downtime.
- We deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
- Using Storage vMotion technology to migrate OS and Oracle non-clustered vmdk’s of RAC VM’s – The OS and Oracle binaries vmdk’s for the RAC VM’s are now migrated from one datastore to new datastore on an existing array or between 2 arrays
- Using Oracle ASM technology to migrate cluster database data – RAC database can been migrated from one datastore to new datastore on an existing array or between 2 arrays
Before describing this technique, we first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion.
Oracle layout on vSphere
There are 3 main Oracle database deployment models, all of which are supported and can be deployed on the VMware SDDC stack.
- Oracle Single Instance
- Oracle Real Application Cluster
- Oracle RAC one node (uses the same concept of sharing vmdk’s using multi-writer flag)
There are 3 values of vmdk sharing mode
- Unspecified
- No sharing
- Multi-writer – used for Clustering
An Oracle RAC Cluster will have multiple VM’s , each VM with 2 genres of vmdk’s :
- non-database vmdk’s
- Operating System disks which house the root file system (/)
- Oracle binaries vmdk which houses the Oracle binaries (Grid and RDBMS binaries) /u01)
- shared database vmdk’s using multi-writer attribute
VMware Storage vMotion
With VMware Storage vMotion technology, we can migrate VM’s to relocate the configuration file of a virtual machine and virtual disks, while the virtual machine is powered on, from one datastore to another.
The vSphere documentation details Storage vMotion Requirements and Limitations
Caveats with Multi-writer disks and Storage vMotion
The multi-writer capability for sharing vmdk’s between VM’s play an important role in Oracle RAC deployments on VMware SDDC.
VMFS is a clustered file system that disables (by default) multiple VM’s from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file.
The multi-writer option allows VMFS-backed disks to be shared by multiple VM’s. This option can be used to disable the protection for certain cluster-aware applications e.g. Oracle Clusterware , where the applications ensure that writes originating from two or more different VM’s does not cause data loss and ensures data consistency and concurrency.
Among the unsupported actions for shared vmdk’s using multi-writer capability includes
- Storage vMotion
- Snapshots
- Changed Block Tracking (CBT)
Attempting a storage vMotion of an RAC VM will result in an error:
The supported and unsupported actions / features using multi-writer attribute can be found in the KB 1034165.
Refer to the blog Add shared vmdk online without downtime for Oracle RAC ASM / OCFS2 for adding shared vmdk’s with multi-writer flag online to an Oracle RAC Cluster.
Understanding Oracle Storage migration methods
There are a couple of ways we can migrate Oracle workload storage either from one datastore to another, within in an existing array or between 2 separate arrays.
- Oracle Automatic Storage Management (ASM) technology to migrate oracle blocks from one vmdk on a datastore to a new vmdk on same / different datastore
The process is to add new disks to the existing ASM disk group and remove old disks from the same ASM disk group , while the database continues to access files from the disk group. When you add or remove disks from a disk group, Oracle ASM automatically redistributes the file contents and eliminates the need for downtime when redistributing the content. When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group.
More information can be found at Dropping Disks from Disk Groups.
The power setting parameter ASM_POWER_LIMIT determines the speed with which rebalancing operations occur. The range of values is 0 to 1024. The default value is 1. A value of 0 disables rebalancing. Higher numeric values enable the rebalancing operation to complete more quickly, but might result in higher I/O overhead and more rebalancing processes. With this parameter, the DBA has the power to control the migration throughput hence avoiding causing potential performance during peak production hours
Oracle ASM migration method uses Oracle ASM technology to migrate oracle blocks between the ASM disks and
-
- will ONLY migrate Oracle database ASM disk
- will NOT migrate the Operating System vmdk or Oracle binaries vmdk
The steps for using Oracle ASM technology for storage migrating Oracle database storage from one datastore to new datastore on either an existing array or between 2 arrays are
-
- add new vmdk/s from new datastore (either existing or new array) to VM
- partition the new device with appropriate partitioning offset
- needed if ASMLIB is used, not needed for Linux udev
- create ASM device on new disk/s
- add new ASM disks to existing ASM disk group
- drop the old ASM disk from existing ASM disk group
- When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group
- delete the old vmdk/s from the VM
At the end of this step
-
- database vmdk’s are on the new datastore
- however , operating system and oracle binaries vmdk are still on the old datastore
2. Use VMware storage vMotion method of moving vmdk’s from one datastore to new datastore
VMware storage vMotion
-
- will migrates ALL vmdk’s including Oracle database ASM disk, Operating System vmdk and Oracle binaries vmdk
- can be used ONLY for non-shared vmdk’s [KB 1034165]
Any storage-based migration, be it storage vMotion or array based migration, is faster than Oracle ASM method of add, drop and rebalancing disks. However, the speed of migration is not regulated or controlled. This may cause potential performance during peak production hours.
For migration from existing to a new array, datastores from both arrays must be mounted on the vSphere cluster
There are also other methods for migrating storage which will require time to setup and cut over as well and are out of scope for this blog
- Standing up a target Oracle RAC as Physical standby for the source Oracle RAC cluster
- Using Data Pump / RMAN / Golden Gate / Third party Replication products to move data between source and new created target RAC database
Test Setup of an Oracle RAC prac19c
Below is a setup of a 2 node Oracle RAC prac19c with 2 RAC Instances prac19c1 and prac19c2. Th Operating system is OEL 7.9 with Grid Infrastructure and RDBMS 19.8 with Oracle ASM with ASMLIB.
Both RAC VM’s prac19c1 and prac19c2 storage is on a FC datastore on Pure Storage.
Details of Oracle RAC VMs prac19c1 and prac19c2 are as follows:
- 12 vCPUs with 128GB RAM
- Oracle SGA set to 96GB with traditional HugePages and PGA set to 6GB
- VM hosts both Oracle Grid and RDBMS 19.8 multi-tenant production database vvol19c with a pluggable database pdb1
- For sake of simplicity and illustration, one ASM disk group was created called DATA_DG which houses all the data files, control files, redo log files , archive log files, crs and vote disks. Recommendation is to create separate ASM disk groups for the RAC and Database components as a best practice. Refer to Oracle VMware Hybrid Cloud High Availability Guide for more information
- VM prac19c1 public network adapter is connected to port group APPS-1614 and assigned an IP address 172.16.14.191. The private network adapter is connected to port group APPS-1605 and assigned an IP address 192.168.14.191
- VM prac19c2 public network adapter is connected to port group APPS-1614 and assigned an IP address 172.16.14.192. The private network adapter is connected to port group APPS-1605 and assigned an IP address 192.168.14.192
Oracle RAC prac19c VM’s VMDKs are shown below. All SCSI controllers are set to VMware Paravirtual SCSI Controller type:
- Two non-shared VMDKs
- Hard Disk 1 80GB for Operating System with disk mode Dependent
- Hard Disk 1 80GB for Oracle Grid Infrastructure and RDBMS binaries with disk mode Dependent
- One shared VMDK (500 GB) with multi-writer attribute and disk mode Independent-Persistent for RAC cluster
The shared vmdk’s are created on 1 RAC VM and all other RAC VM’s simply refer/point to it
Details of the shared VMDK with multi-writer flag and disk mode Independent-Persistent are shown below:
Lets look at the .vmx files of RAC prac19c1 and prac19c2 VM’s and see how the shared vmdk’s are attached , for example, to SCSI 2:0 slot.
[root@sc2esx09:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/prac19c1] cat prac19c1.vmx | grep -i vmdk
scsi0:0.fileName = “prac19c1.vmdk”
scsi0:1.fileName = “prac19c1_1.vmdk”
scsi2:0.fileName = “prac19c1_3.vmdk”
[root@sc2esx09:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/prac19c1
[root@sc2esx09:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/prac19c2] cat prac19c2.vmx | grep -i vmdk
scsi0:0.fileName = “prac19c2.vmdk”
scsi0:1.fileName = “prac19c2_1.vmdk”
scsi2:0.fileName = “/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/prac19c1/prac19c1_3.vmdk” <– refer to VM prac19c1 vmdk
[root@sc2esx09:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/prac19c2]
Storage Migration of an Oracle RAC Cluster
Question , how do we now migrate an RAC cluster from one datastore to another , accessed to the same vSphere Cluster , without any downtime?
Remember , Oracle RAC VM’s has shared vmdk/s for RAC database. As stated above
- Oracle Automatic Storage Management (ASM) technology
- will ONLY migrate Oracle database ASM disk
- will NOT migrate the Operating System vmdk or Oracle binaries vmdk
- VMware storage vMotion
- will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk
- can be used ONLY for non-shared vmdk’s [KB 1034165]
Do we have a solution?
Solution: A two-step process which combines both Oracle ASM and storage vMotion technology for a RAC Cluster.
High level Steps of Storage Migration of an Oracle RAC Cluster without Downtime
- The source datastore is Pure Storage FC datastore OraPure and target datastore is Pure Storage FC datastore SC2-Pure-Templates
- Perform Storage vMotion RAC VM’s ‘prac19c1’ and ‘prac19c2 non-shared vmdk’s only from source datastore OraPure to the target datastore SC2-Pure-Templates without migrating the shared vmdk’s.
- Add new shared vmdk’s with multi-writer attribute from target datastore SC2-Pure-Templates to RAC VM’s ‘prac19c1’ and ‘prac19c2.
- Perform steps to add the new shared vmdk’s from target datastore SC2-Pure-Templates to the ASM disk groups DATA_DG
- Drop the old ASM disk for DATA_DG from Oracle ASM
- Then delete the old vmdk from RAC VM’s ‘rac19c1’ and ‘rac19c2
- Now the RAC database shared vmdk’s for DATA_DG are on the target datastore SC2-Pure-Templates
Step 1 – Storage vMotion to migrate OS and Oracle vmdks of RAC VM’s prac19c1 and prac19c2
The steps for migrating the non-clustered OS and Oracle vmdk’s for RAC VM’s from one datastore to new datastore on either an existing array or between 2 arrays are the same as in the case of storage vMotioning any VM.
The steps are as shown below. The source datastore is Pure Storage FC datastore OraPure and target datastore is Pure Storage FC datastore SC2-Pure-Templates
As part of the Storage vMotion , we have a choice to pick the datastore we want to place the vmdk’s for individual vmdk’s ,we choose this option
At the end of this step, the OS and Oracle binaries vmdk’s for the RAC VM’s prac19c1 and prac19c2 are now on the new datastore
The 500GB shared vmdk is still on source datastore OraPure.
Step 2 – Using Oracle ASM technology to migrate cluster database data
The steps for using Oracle ASM technology for storage migrating Oracle RAC database storage from one datastore to new datastore on either an existing array or between 2 arrays are
- On RAC VM prac19c1 – Create a new Eager Zero Thick (EZT) vmdk with multi-writer attribute for sharing from target datastore SC2-Pure-Templates
- On RAC VM prac19c2 – Add the same newly added vmdk on RAC VM’s prac19c1 to RAC VM’s prac19c2 on the SAME SCSI position with multi-writer attribute for sharing (Add an existing hard disk option)
- Both RAC VM’s prac19c1 and prac19c2 now can see the same new vmdk 500GB carved from target datastore SC2-Pure-Templates
- On RAC VM prac19c1 only – partition the new vmdk device with appropriate partitioning offset , needed if ASMLIB is used, not needed for Linux udev
- On RAC VM prac19c2 – scan the SCSI bus to see the new disk partition
- On RAC VM prac19c1 – create new ASM device on new disk
- On RAC VM prac19c2 – run ‘oracleasm scandisks’ command
- On RAC VM prac19c1 – add new ASM disks to existing ASM disk group
- On RAC VM prac19c1 – drop the old ASM disk from existing ASM disk group and perform ASM rebalance with/without POWER option. When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group
- On RAC VM prac19c1 – run ‘oracleasm deletedisk XXX’ where XXX is the old ASM disk.
- On RAC VM prac19vc2 – run ‘oracleasm scandisks’ command
- Delete old vmdk from both RAC VM’s prac19c1 and prac19c2
- Now the RAC database shared vmdk’s for DATA_DG are on the target datastore SC2-Pure-Templates
Thee detailed steps to add and drop ASM disks with ASM rebalance can be found in the blog No downtime Storage vMotion of Oracle RAC Cluster using shared vmdk’s with multi-writer attribute from one vSAN to another vSAN Cluster using VMware HCI Mesh
At the end of this step , on Oracle RAC VM’s prac19c1 and prac19c2 , both shared and non-shared vmdk’s have been Storage vMotioned from one datastore to another datastore.
Summary
- Storage Migration of an Oracle RAC Cluster from one datastore to another datastore , within an storage array OR between 2 storage arrays, on a VMware SDDC , is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database. and VMware storage vMotion alone cannot be used as explained below.
- VMware storage vMotion will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but will not storage vMotion clustered vmdk’s, it can ONLY be used for non-shared vmdk’s [KB 1034165]
- Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk BUT will NOT migrate the Operating System or Oracle binaries vmdk.
- This blog focused on how we can storage vMotion an Oracle RAC Cluster without ANY downtime.
- We deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
- Using Storage vMotion technology to migrate OS and Oracle non-clustered vmdk’s of RAC VM’s – The OS and Oracle binaries vmdk’s for the RAC VM’s are now migrated from one datastore to new datastore on an existing array or between 2 arrays
- Using Oracle ASM technology to migrate cluster database data – RAC database can been migrated from one datastore to new datastore on an existing array or between 2 arrays
Conclusion
Business Critical Applications especially IO intensive Oracle workloads have a need for storage migration, based on the ever-demanding workload profile, from one Tier to another Tier within a storage array or between storage arrays.
By combining both technologies, Oracle ASM technology and VMware storage vMotion, we are able to successfully migrate storage of an n-node Oracle RAC on VMware SDDC without any downtime or breech of SLA which is the fastest way to migrate RAC cluster on VMware SDDC.
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
