Home > Blogs > Virtualize Business Critical Applications > Monthly Archives: September 2017

Monthly Archives: September 2017

“RAC” n “RAC” all night – Oracle RAC on vSphere 6.x

“I wanna “RAC” and “RAC” all night and party every day” – mantra of an Oracle RAC DBA.

Much has been written , spoken and probably beaten to senseless 🙂 about the magical “Multi-writer” setting and how it helps multiple VM’s share vmdk’s simultaneously for Clustering and FT used cases.

I still get question from customers interested in running Oracle RAC on vSphere about if we have the ability to add shared vmdk’s to a RAC cluster online without any downtime ? Yes we do. Are the steps of adding shared vmdk’s to an extended RAC cluster online without any downtime the same? Yes.

 

Introduction

By default, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled.

The below table describes the various Virtual Machine Disk Modes:

As we all are aware of , Oracle RAC requires shared disks to be accessed by all nodes of the RAC cluster.

KB Article 1034165 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s. Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is

  • has to set to Eager Zero Thick provisioned
  • need not be set to Independent persistent

While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.

Supported and Unsupported Actions or Features with Multi-Writer Flag:

**** Important ***
•    SCSI bus sharing is left at default and not touched at all in case of using shared vmdk’s
•    It’s only used for RAC’s with RDM (Raw Device Mappings) as shared disks

 

Facts about vmdk and multi-writer

Before version 6.0, we had the ability to add vmdk’s with multi-writer option to an Oracle RAC online , the only caveat was that this ability was not exposed in the vSphere Web/C# Client .We had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online.

Setting Multi Writer Flag for Oracle RAC on vSphere using Power Cli
https://blogs.vmware.com/apps/2013/10/setting-multi-writer-flag-for-oracle-rac-on-vsphere-without-any-downtime.html#more-864

http://www.virtuallyghetto.com/2015/10/new-method-of-enabling-multiwriter-vmdk-flag-in-vsphere-6-0-update-1.html

With vSphere 6.0 and onwards, we can add shared disks to an Oracle RAC Cluster online using the Web Client.

 

Key points to take away from this blog:
•    VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
•    vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
•    Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online

 

Example of an Oracle RAC Setup

As per best practices, the 2 VM’s, ‘rac01-g6’ and ‘rac02-g6’ , part of the 2 node Oracle RAC setup was deployed from a template ‘Template-12crac’.

The template has 10 vCPUs with 64 GB RAM with OEL7.3 as the operating system.

The template has 2 vmdk’s, 50GB each on SCSI 0 controller (Paravirtual SCSI Controller type)
•    Hard disk 1 is on SCSI0:0 and is for root volume (/)
•    Hard disk 2 is on SCSI0:1 and is for oracle binaries (/u01 for Grid and RDBMS binaries)

Hard Disk 1 (OS drive) & Hard Disk 2 (Oracle /u01) vmdk’s are set to
•    Thin Provisioning
•    No Sharing i.e. exclusive to the VM
•    Disk mode is set to ‘Dependent’

Template has 2 network adapters of type VMXNET3.
•    Public adapter
•    Private Interconnect

Public Adapter:

Private Interconnect:

Lets add a shared vmdk of size say 50GB to both the VM’s online without powering down the VM’s.

Add shared vmdk to an Oracle RAC online

1. Adding shared disks can be done online without downtime.

2. Add a PVSCSI Controller (SCSI 1) to RAC VM ‘rac01-g6’. Right Click on ‘rac01-g6’ , ‘Edit Settings’ and add New Controller of Type ‘Paravirtual’

Leave the SCSI Bus Sharing to ‘None’ (default)

3. Next step is to add a 50 GB shared vmdk to VM ‘rac01-g6’  to SCSI1:0 bus slot ( you can add the new vmdk it to any slot on SCSI 1 you want to)

Right Click on VM ‘rac01-g6’ and Choose ‘Edit Settings’. Choose ‘New Hard Disk’ ,  set Sharing to ‘Multi-writer’ , leave Disk mode to ‘Dependent’ and click ‘Add’. Click ‘OK’ and monitor progress.

4. Repeat Step 2 to add new ‘Paravirtual’ Controller SCSI 1 to RAC VM ‘rac02-g6’

5. . The new vmdk (vmdk with multi-writer option) created on VM ‘rac01-g6’ at SCSI1:0 bus slot needs to be shared with ‘rac02-g6’ VM for clustering purpose

6. Right Click on VM ‘rac02-g6’, Choose ‘Edit Settings’. Choose ‘Existing Hard Disk’ and Click ‘Add’.

7. Navigate to your Datastore [Group06], expand the Datastore contents and click on ‘rac01-g6’ folder. Click on the shared vmdk ‘rac01-g6_2.vmdk’ which was created on ‘rac01-g6’. Click ‘OK’

8. Note that the Sharing attribute for this vmdk needs to be set to ‘Multi-Writer’ and the SCSI controller set to the same SCSI controller as we did for ‘rac01-g6’ i.e SCSI1:0. Click ‘OK’ when done.

9. Scan the bus on the OS of both VM’s to see the new disk added and list the devices

[root@rac01-g6 ~]# fdisk -lu

Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM

….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# fdisk -lu
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM
….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac02-g6 ~]#

10. Partition Align the shared disk (/dev/sdc) on ‘rac01-g6’ (do this on one node only) using the fdisk / parted utility / tool of your choice) :

11. After partition alignment:

root@rac01-g6 ~]# fdisk -lu /dev/sdc
……
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048   104857599    52427776   83  Linux
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# fdisk -lu /dev/sdc
…..
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048   104857599    52427776   83  Linux
[root@rac02-g6 ~]#

12. Create ASM disks using ASMLIB

Installing and Configuring Oracle ASMLIB Software
https://docs.oracle.com/database/122/LADBI/installing-and-configuring-oracle-asmlib-software.htm#LADBI-GUID-79F9D58F-E5BB-45BD-A664-260C0502D876

 

root@rac01-g6 ~]# /usr/sbin/oracleasm createdisk DATA_DISK01 /dev/sda1
Writing disk header: done
Instantiating disk: done
[root@rac01-g6 ~]#

[root@rac01-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# /usr/sbin/oracleasm scandisks
[root@rac02-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac02-g6 ~]#

As we can see, we have now added a shared vmdk of size 50 GB to both VM’s without any downtime online and created ASM disk on this shared disk to be used for Oracle RAC ASM disk group.

The rest of the steps to create the Oracle RAC is exactly the same as shown in the Oracle documentation
https://docs.oracle.com/database/122/CWSOL/title.htm

########
Summary
########
Key points to keep in mind:

  • VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
  • vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
  • Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online

Best Practices needs to be followed when configuring Oracle RAC environment  which can be found in the “Oracle Databases on VMware – Best Practices Guide”

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-oracle-databases-on-vmware-best-practices-guide.pdf

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html