posted

0 Comments

Around the World in Eighty Days” – A classic adventure novel and one of Jules Verne’s most acclaimed works which describes how Phileas Fogg and his valet Passepartout attempt to circumnavigate the world in 80 days on a wager.

 

 

In the world of Business Critical Applications, especially IO intensive Oracle workloads, there is always a need for storage migration, based on the ever demanding workload profile. For example,

  • Migrate storage from one Tier to another Tier within a storage array
  • Migrate storage from one array to another array (within and between datacenters)

This approach is independent of whether those workloads are virtualized or not. The end goal is to ensure that the underlying storage architecture can provide and sustain the demanding needs of the Application.

This article describes how storage migration can be performed for Oracle database running on VMware vSphere, for any types of underlying storage.  The migration is done using a combination of two key technologies, VMware Storage vMotion and Oracle Automatic Storage Management (ASM) depending on the database deployment model.

Before describing this technique, we first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion.

 

Oracle layout on vSphere – Single Instance and RAC

There are 3 main Oracle database deployment models, all of which are supported and can be deployed on the VMware SDDC stack.

The primary difference between Oracle Single instance and Oracle RAC vmdk/s setup is the vmdk sharing mode.

There are 3 values of disk sharing mode

  • Unspecified
  • No sharing
  • Multi-writer – used for Clustering

 

 

Both Single Instance VM and RAC VM/s will have 2 genres of vmdk’s :

  • non-database vmdk’s
    • Operating System disks which houses the root file system (/)
    • Oracle binaries vmdk which houses the Oracle binaries (Grid and RDBMS binaries) (/u01)
  • database vmdk’s
    • Database vmdk/s
      • non-shared in case of Single Instance VM
      • shared in case of RAC VM/s

 

Example of Disk layout of Oracle Single Instance VM

‘SB_oracle-RHEL’ VM houses a single instance Oracle database 12.2 running on Red Hat 7.4 operating system.

The vmdk’s for the VM is on ‘TSA_TNTR_Cap_10TB_01’ datastore on a Tintri array.

 

 

The VM ‘SB_oracle-RHEL’  has 3 vmdk’s-

  • Hard Disk 1 of 50 GB capacity which houses the OS (/)
  • Hard Disk 2 of 50 GB capacity which houses the Oracle binaries (/u01)
  • Hard Disk 1 of 50 GB capacity which houses the Oracle database (ASM disk) – No sharing mode

 

 

Example of Disk layout of Oracle RAC VM’s

 Below is a setup of a 4 node RAC Cluster.

VM’s ‘rac1221’, ‘rac1222’, ‘rac1223’ and ‘rac1224’ form the 4 Node 12.2 Oracle RAC cluster (Flex Cluster and Flex ASM) on Oracle Enterprise Linux 7.4.

All 4 RAC VM’s are in the same datastore ‘TSA_TNTR_Cap_10TB_01’.

 

Looking at VM ‘rac1221’ disk layout, we can see, ‘rac1221’ has 11 hard disks

  • Hard Disk 1 of 60 GB capacity which is the OS disk (/)
  • Hard Disk 2 of 60 GB capacity which is the Oracle binaries (/u01) disk
  • Hard Disk 3 – Hard Disk 11 which are the shared Oracle RAC vmdk’s (ASM disks)

 

 

Observation

  • Sharing attribute of vmdk’s Hard Disk [ 3 – 11] are set to ‘Multi-writer’.
  • Disks with multi-writer flag attribute have to be Eager Zero thick
  • They need not be independent persistent KB 1034165

 

 

Let’s look at VM ‘rac1222’ disk layout. We can see ‘rac1222’ has 11 hard disks

  • Hard Disk 1 of 60 GB capacity which is the OS disk
  • Hard Disk 2 of 60 GB capacity which is the Oracle binaries (/u01) disk
  • Hard Disk 3 – Hard Disk 11 which are the shared Oracle RAC vmdk’s (ASM disks)

 

 

 

Observation

  • Sharing attribute of Hard Disk [ 3 – 11] for are also set to ‘Multi-writer’.
  • Hard Disks [ 3 – 11] of ‘rac1222’ refer to / point to ‘rac1221’ vmdk’s.

RAC VM’s ‘rac1223’ and ‘rac1224’ disk layout matches ‘rac1222’.

 

Process of allocating disks to Oracle RAC VM’s

The process for adding new vmdk/s to a RAC cluster using the Multi-writer flag can be found in the article KB 1034165 .

High level steps for adding new vmdk’s to a RAC cluster:

  • new vmdk/s are added to one RAC VM with ‘New Hard Disk’ option to a SCSI x:y controller slot

 

  • then the new vmdk/s are then added to the rest of the RAC VM’s using the ‘Existing Hard Disk’ option, they need to be added to the same SCSI x:y controller slot as allocated above

 

 

After allocating all disk, the vmdk layout for the RAC VM’s would look as below:

 

 

A look at the vmx file of the Oracle RAC VM’s

Lets look at the .vmx files of RAC VM’s ‘rac1221’ , ‘rac1222’ , ‘rac1223’ and ‘rac1224’ VM’s and see how the shared vmdk’s are attached , for example, to SCSI 1:0 slot.

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1221] cat rac1221.vmx
….
scsi1:0.deviceType = “scsi-hardDisk”
scsi1:0.fileName = “rac1221_2.vmdk” <- rac1221 vmdk at 1:0
scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
….
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1221]

 

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1222] cat rac1222.vmx
….
scsi1:0.deviceType = “scsi-hardDisk”
scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1222]

 

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1223] cat rac1223.vmx
……
scsi1:0.deviceType = “scsi-hardDisk”

scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
……
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1223]

 

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1224] cat rac1224.vmx
……
scsi1:0.deviceType = “scsi-hardDisk”

scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
….
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1224]

 

VMware Storage vMotion

 With VMware Storage vMotion technology, we can migrate VM’s to relocate the configuration file of a virtual machine and virtual disks, while the virtual machine is powered on, from one datastore to another.

The vSphere documentation details Storage vMotion Requirements and Limitations

 

Caveats with Multi-writer disks and Storage vMotion

 The multi-writer capability for sharing vmdk’s between VM’s play an important role in Oracle RAC deployments on VMware SDDC.

VMFS is a clustered file system that disables (by default) multiple VM’s from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file..

The multi-writer option allows VMFS-backed disks to be shared by multiple VM’s. This option can be used to disable the protection for certain cluster-aware applications e.g Oracle Clusterware , where the applications ensure that writes originating from two or more different VM’s does not cause data loss and ensures data consistency and concurrency.

Among the unsupported actions for shared vmdk’s using multi-writer capability includes

  • Storage vMotion
  • Snapshots
  • Changed Block Tracking (CBT)

Attempting a storage vMotion of an RAC VM will result in an error:

 

 

The supported and unsupported actions / features using multi-writer attribute can be found in the KB 1034165.

 

 

 

Understanding Oracle Storage migration methods

There are a couple of ways we can migrate Oracle workload storage either from one datastore to another, within in an existing array or between 2 separate arrays.

  1. Use Oracle Automatic Storage Management (ASM) technology to migrate oracle blocks from one vmdk on a datastore to a new vmdk on same / different datastore

The process is to add new disks to the existing ASM disk group and remove old disks from the same ASM disk group , while the database continues to access files from the disk group.  When you add or remove disks from a disk group, Oracle ASM automatically redistributes the file contents and eliminates the need for downtime when redistributing the content. When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group.

More information can be found at Dropping Disks from Disk Groups.

The power setting parameter ASM_POWER_LIMIT determines the speed with which rebalancing operations occur.  The range of values is 0 to 1024. The default value is 1. A value of 0 disables rebalancing. Higher numeric values enable the rebalancing operation to complete more quickly, but might result in higher I/O overhead and more rebalancing processes.  With this parameter, the DBA has the power to control the migration throughput hence avoiding causing potential performance during peak production hours

This migration method uses Oracle ASM technology to migrate oracle blocks between the ASM disks and

  • will ONLY migrate Oracle database ASM disk
  • will NOT migrate the Operating System vmdk or Oracle binaries vmdk

 

  1. Use VMware storage vMotion method of moving vmdk’s from one datastore to new datastore

 This migration method

  • will migrates ALL vmdk’s including Oracle database ASM disk, Operating System vmdk and Oracle binaries vmdk
  • can be used ONLY for non-shared vmdk’s [KB 1034165]

Any storage based migration, be it storage vMotion or array based migration, is faster than Oracle ASM method of add, drop and rebalancing disks.  However, the speed of migration is not regulated or controlled. This may cause potential performance during peak production hours.

For migration from existing to a new array, datastores from both arrays must be mounted on the vSphere cluster

There are also other methods for migrating storage which will require time to setup and cut over as well and are out of scope for this blog

  • Standing up a target Oracle RAC as Physical standby for the source Oracle RAC cluster
  • Using Data Pump / RMAN / Golden Gate / Third party Replication products to move data between source and new created target RAC database

 

Storage Migration of an Oracle RAC Cluster

Question , How do we now migrate an RAC cluster from one datastore to another without any downtime?

Remember , Oracle RAC VM’s has shared vmdk/s for RAC database. As stated above

  • Oracle Automatic Storage Management (ASM) technology
    • will ONLY migrate Oracle database ASM disk
    • will NOT migrate the Operating System vmdk or Oracle binaries vmdk
  • VMware storage vMotion
    • will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk
    • can be used ONLY for non-shared vmdk’s [KB 1034165]

Do we have a solution?

 

Solution: A two-step process which combines both Oracle ASM and storage vMotion technology for a RAC Cluster.

 

Storage Migration of an Oracle Single Instance database

Using Oracle ASM technology

The steps for using Oracle ASM technology for storage migrating Oracle database storage from one datastore to new datastore on either an existing array or between 2 arrays are

  • add new vmdk/s from new datastore (either existing or new array) to VM
  • partition the new device with appropriate partitioning offset
    • needed if ASMLIB is used, not needed for Linux udev
  • create ASM device on new disk/s
  • add new ASM disks to existing ASM disk group
  • drop the old ASM disk from existing ASM disk group
    • When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group
  • delete the old vmdk/s from the VM

At the end of this step

  • database vmdk’s are on the new datastore
  • operating system and oracle binaries vmdk are still on the old datastore

As stated earlier, this migration method uses Oracle ASM technology to migrate oracle blocks between the ASM disks and

  1. will ONLY migrate Oracle database ASM disk
  2. will NOT migrate the Operating System or Oracle binaries vmdk

 

Using Storage vMotion technology

Process of storage vMotion of an Oracle single instance deployment on VMware SDDC is the same as storage vMotioning any other workload on VMware SDDC.

Lets storage vMotion the VM ‘SB_oracle-RHEL’ from ‘TSA_TNTR_Cap_10TB_01’ datastore ‘TSA_TNTR_Cap_10TB_02’ on the same array.

Right click on VM ‘SB_oracle-RHEL’ on web client and choose the storage migration option

 

Choose the destination datastore

 

We have a choice to pick the datastore we want to place the vmdk’s, in this case we picked the same datastore for all the VM vmdk’s.

 

Confirm operation

 

 

Storage vMotion operation starts and completes successfully

 

We have successfully storage vMotioned the VM ‘SB_oracle-RHEL’ from ‘TSA_TNTR_Cap_10TB_01’ datastore ‘TSA_TNTR_Cap_10TB_02’ on the same array.

At the end of this step

  • database vmdk’s are on the new datastore
  • operating system and oracle binaries vmdk are on the new datastore

 

Storage Migration of an Oracle RAC Cluster

 This will involve a 2-step process

  • Using Oracle ASM technology to migrate cluster database data
  • Using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s

 

Using Oracle ASM technology to migrate cluster database data

 The steps for using Oracle ASM technology for storage migrating Oracle RAC database storage from one datastore to new datastore on either an existing array or between 2 arrays are

  • pick a RAC VM, for example ‘rac1221’ , as the VM for disk operation
  • add new vmdk/s from new datastore (either existing or new array) to ‘rac1221’ using ‘New Hard disk’ option
  • add these vmdk/s to the other RAC VM’s [ ‘rac1222’, ‘rac1223’, ‘rac1224’ ] using ‘Existing Hard disk’ option
  • partition the new vmdk/s on the Guest with appropriate partitioning offset
    • needed if ASMLIB is used, not needed for Linux udev
    • perform above operation on RAC VM ‘rac1221’ only
    • scan the SCSI bus on the other RAC VM’s [‘rac1222’, ‘rac1223’, ‘rac1224’] to see the partitions
  • create ASM device on new disk/s
    • perform operation on RAC VM ‘rac11221’
    • run ‘oracleasm scandisks’ command on the other RAC VM’s [ ‘rac1222’, ‘rac1223’, ‘rac1224’ ] to see the new ASM disks
  • add new ASM disks to existing ASM disk group
    • perform operation on RAC VM ‘rac11221’
  • drop the old ASM disk from existing ASM disk group
    • perform operation on RAC VM ‘rac11221’
  • delete the old vmdk/s from all the other RAC VMs [‘rac1222’, ‘rac1223’, ‘rac1224’ ]
    • do not check “Delete files from datastore” option
  • delete the old vmdk/s from the RAC VM ‘rac1221’
    • check “Delete files from datastore” option before confirming delete

At the end of this step

  • RAC database has been migrated from one set of vmdk’s to a new set of vmdk’s
  • old database vmdk’s have been deleted from the RAC VM’s
  • The OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are still on the old datastore
    • they need to be moved to the new datastore as well

 

Using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s

The steps for migrating OS and Oracle vmdk’s for RAC VM’s from one datastore to new datastore on either an existing array or between 2 arrays are

  • relocate / move Oracle user connections from RAC instance ‘rac1222’ to the remaining 3 RAC instances [ ‘rac1221’, ‘rac1223’, ‘rac1224’ ]
  • shutdown RAC instance ‘rac1222’
  • power down RAC VM ‘rac1222’
  • make a copy of RAC VM ‘rac1222’ .vmx file just in case
  • remove all shared vmdk’s from RAC VM ‘rac1222’
    • do not check “Delete files from datastore” option
  • storage vMotion RAC VM ‘rac1222’ OS and Oracle vmdk’s to the new datastore
  • add the shared vmdk/s back to RAC VM ‘rac1222’
    • keep in mind the shared vmdk’s have to be added in the same SCSI x:y controller slot positions as before
  • repeat above steps for other RAC VMs ‘rac1223’ , ‘rac1224’ and ‘rac1221’

At the end of this step

  • OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are now on the new datastore

 

Summary

Storage Migration of an Oracle RAC Cluster on VMware SDDC is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database

  • Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk and will NOT migrate the Operating System or Oracle binaries vmdk.
  • VMware storage vMotion will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but can ONLY be used for non-shared vmdk’s [KB 1034165]
  • We can deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above

 

Conclusion

Business Critical Applications especially IO intensive Oracle workloads have a need for storage migration, based on the ever-demanding workload profile, from one Tier to another Tier within a storage array or between storage arrays.

By combining both technologies, Oracle ASM technology and VMware storage vMotion, we are able to successfully migrate storage of an n-node Oracle RAC on VMware SDDC without any downtime or breech of SLA which is the fastest way to migrate RAC cluster on VMware SDDC.

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop