posted

0 Comments

In the first part of this series we provided a high level view of the benefits of using Virtual Volumes enabled storage for database operations. In this post, we will examine in more detail how Virtual Volumes can improve the backup and recovery capabilities for business critical databases, specifically Oracle.

The backups for Oracle can be Database consistent or Crash consistent. In this part we will look at Database consistent backup and recovery.

The Setup:

The solution requires VVol enabled storage. We leveraged SANBLaze VirtuaLun as the backend storage for the backup and recovery exercise. We used the VirtuaLun 7.3 emulator from SANBlaze. This emulator is VVol enabled and is one of the first VVol certified storage solutions available.

VVOL2_fig1

Figure 1: SANBlaze Array for VVol Testing

Oracle Database Server:

A Single Instance Oracle 12c Database with Grid Infrastructure with database name VVOL12C was setup in a VMware Virtual Machine named ORACLE-VVOL. The Oracle database was hosted on a 2 vCPU and 8GB RAM VM running Oracle Enterprise Linux 6.6 with space allocated on a SANBlaze LUN.

vCLI Package was installed on the Guest operating system in the VM. The vSphere Command-Line Interface (vSphere CLI on Linux) command set allows you to run common system administration commands against ESX/ESXi systems from any machine with network access to those systems. The virtual machine is hosted in a VMware vSphere 6 environment running on Dell POWEREDGE servers.

VVOL2_fig2

Figure 2: VMware Infrastructure Details

Oracle Database Layout using VVol:

The SANBlaze creates a SubLun ( a VVol in the SANBlaze environment) for every VMDK of the oracle database virtual machine (5 SubLuns for 5 vmdk’s). we have 1 extra SubLun for the configuration file.

When the VM is powered up, in additions to the 6 Subluns, there is 1 extra SubLun created for the VM swap file (.vswp file). The disk layout for the virtual machine is as shown below. There was no storage tiering with the default tier used for the testing.

VVOL2_fig3

Figure 3: Disk Layout for Oracle Virtual Machine

The subtlety lies in the way a VMFS datastore of type VVol is implemented than a regular VMFS datastore. A “df” command on the ESXi server will reveal that VVol datastores are implemented as a datastore of type “vvol”

[root@localhost:~] df

Filesystem         Bytes         Used     Available Use% Mounted on

VMFS-5     3298266447872 2363283734528   934982713344 72% /vmfs/volumes/OEL_Clone3

…..

vfat           299712512     216350720       83361792 72% /vmfs/volumes/54a30a0c-93caccba-8482-ecf4bbc08ee0

…..

vvol       10737418240000             0 10737418240000   0% /vmfs/volumes/BCA_VVOL_GOLD

vvol       21474836480000             0 21474836480000   0% /vmfs/volumes/BCA_VVOL_SILVER

[root@localhost:~]

Unlike any regular VM files in the VM folder on a datastore where we can see the different –flat.vmdk files. With VM’s on VVol , the folder contents looked like below:

[root@localhost:/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000] ls -l

-rw-r–r–   1 root     root           43 Jul 15 04:19 Oracle-VVOL-a59e28cb.hlog

-rw——-   1 root     root         8684 Jul 15 17:09 Oracle-VVOL.nvram

-rw——-   1 root     root           581 Jul 15 04:19 Oracle-VVOL.vmdk

-rw-r–r–   1 root     root             0 Jul 14 05:39 Oracle-VVOL.vmsd

-rwxr-xr-x   1 root     root         3996 Jul 15 17:09 Oracle-VVOL.vmx

-rw——-   1 root     root         3179 Jul 14 19:47 Oracle-VVOL.vmxf

-rw——-   1 root     root           555 Jul 15 05:06 Oracle-VVOL_1.vmdk

-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_2.vmdk

-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_3.vmdk

-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_4.vmdk

-rw-r–r–   1 root     root       379606 Jul 15 04:18 vmware-1.log

-rw-r–r–   1 root     root       273649 Jul 15 17:09 vmware.log

[root@localhost:/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000]

Notice the lack of –flat.vmdk’s as they are exist on the Storage VVol!!

Logging into the SANBlaze console, the same disks when viewed from the VVol capable SANBlaze array shows each of the disks and the additional SubLuns for the configuration and the vswp files.

[root@sanblaze1 ~]# grep SubLun /port0/alias0lun0 ; echo ; grep SubLun /port0/alias0lun0 | grep VMW_VVolType | echo “Number of Subluns = ” wc -l

SubLuns=1024

SubLun0=d28000000000,d2a000000000:600110dac045c0450100800000800000 0:4096 MB (0:8388608 blocks) -1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_VVolName=Oracle-VVOL;VMW_VVolType=Config;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0

SubLun1=d28000010000,d2a000010000:600110dac045c0450100800000800001 4096:30720 MB (8388608:62914560 blocks) -1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4

…………

……….

SubLun13=d280000d0000,d2a0000d0000:600110dac045c045010080000080000d 327680:8192 MB (671088640:16777216 blocks) -1 VMW_VvolProfile=7508ed62-e4a9-4cf1-9ef9-fb1f07b72bc1:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_VVolName=Oracle-VVOL-a59e28cb.vswp;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Swap;VMW_VvolAllocationType=3;VMW_GosType=oracleLinux64Guest;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0

Number of Subluns = 7

Database Backup and Recovery using VVol Snapshots:

Database Consistent Backup:

Oracle defines a database consistent backup as follows: (Source: Oracle Glossary )

A Database consistent backup is a whole database backup that you can open with the RESETLOGS option without performing media recovery. In other words, you do not need to apply redo to datafiles in this backup for it to be consistent. All datafiles in a consistent backup must:

  • Have the same checkpoint system change number (SCN) in their headers, unless they are datafiles in tablespaces that are read-only or offline normal (in which case they will have a clean SCN that is earlier than the checkpoint SCN)
  • Contain no changes past the checkpoint SCN, that is, are not fuzzy
  • Match the data file checkpoint information stored in the control file

You can only take consistent backups after you have made a clean shutdown or by turning on hot backup mode of the database.

This is the most trusted backup by DBAs but is also complex, as one needs to run scripts to put the database in hot backup mode take a snapshot and then remove it from hot backup mode.

Backup and Recovery Testing:

The following steps were used for this exercise:

  1. A Database consistent snapshot was taken by using a script “hot_backup”. This Script places the database in Backup / Quiesced mode using begin / end backup mode for Oracle and takes a VMware snapshot of the VM. Details of the script are shown below:

#!/bin/bash

export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1

echo “BEGIN backup mode : date

sqlplus /nolog <<EOF

conn / as sysdba

alter database begin backup;

exit;

EOF

echo “BEGIN Snapshot : date

/usr/bin/vmware-cmd -H 10.152.100.5 -U “vmlabadministrator” -P VMware1! –vihost 10.152.4.9 /vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000/Oracle-VVOL.vmx createsnapshot Oracle-VVOL-DBC-Snapshot ‘Oracle-VVOL-DBC-Snapshot’ 0 0

echo “END Snapshot : date

echo “END backup mode: date

sqlplus /nolog <<EOF

conn / as sysdba

alter database end backup;

EOF

  1. A VMware snapshot is taken for the database virtual machine after putting the DB in a hot backup mode the picture below shows the DB snapsho.
  2. After snapshot is created, the database is removed from the hot backup mode.

VVOL2_fig4Figure 4: Virtual Machine snapshot for DB Consistent ORacl

4. On completion of the snapshot one can look at the volumes associated with the Oracle virtual machine. When the snapshot is taken each SubLun for the VMDKs have an additional copy that is created at the storage level. The number of SubLuns have now increased from 7 to 12 as explained earlier (7 SubLuns for the original VM plus 5 additional SubLuns for the VM vmdk’s)

[root@sanblaze1 ~]# grep SubLun /port0/alias0lun0 ; echo ; grep SubLun /port0/alias0lun0 | grep VMW_VVolType | echo “Number of Subluns = ” wc -l

SubLuns=1024

……..

SubLun2=d28000020000,d2a000020000:600110dac045c0450100800000800002 165888:30720 MB (339738624:62914560 blocks) 1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415

SubLun7=d28000070000,d2a000070000:600110dac045c0450100800000800007 196608:30720 MB (402653184:62914560 blocks) 3 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL_1.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415

SubLun8=d28000080000,d2a000080000:600110dac045c0450100800000800008 227328:30720 MB (465567744:62914560 blocks) 4 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL_2.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415

……

Number of SubLuns = 12

  1. A cloning script is then run to create a cloned DB Virtual machine from the DB consistent VMware snapshot.
  2. Startup the Cloned Database VM via a script. This script performs a media recovery of the database manually using ‘recover database’ command. The recovery script is shown below:

#!/bin/bash

export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1

echo

echo “BEGIN DB Consistent Recovery : date

sqlplus /nolog <<EOF

conn / as sysdba

startup mount pfile=initVVOL12C.ora;

recover database ;

alter database open;

exit;

EOF

echo “END DB Consistent Recovery : date

  1. The database recovers successfully indicating that VVol based snapshots can be used for database consistent backup and recovery. A snippet of the Alert log of the database shown below displays the media recovery steps taken: 

tail –f alert_VVOL12C.log :

 Tue Jul 21 22:38:05 2015

ALTER DATABASE RECOVER database

Tue Jul 21 22:38:05 2015

Media Recovery Start

Started logmerger process

Tue Jul 21 22:38:06 2015

Parallel Media Recovery started with 2 slaves

Tue Jul 21 22:38:06 2015

NOTE: ASMB mounting group 3 (REDO_DG)

…….

NOTE: grp 3 disk 0: REDO_DISK01 path:ORCL:REDO_DISK01

Tue Jul 21 22:38:06 2015

Recovery of Online Redo Log: Thread 1 Group 1 Seq 53 Reading mem 0

Mem# 0: +REDO_DG/VVOL12C/group01_redo01.log

Mem# 1: +REDO_DG/VVOL12C/group01_redo02.log

Tue Jul 21 22:38:06 2015

Tue Jul 21 22:38:06 2015

Media Recovery Complete (VVOL12C)

Completed: ALTER DATABASE RECOVER database

alter database open

Tue Jul 21 22:38:18 2015

 VVol greatly enhances DB consistent Backup capabilities:

We have seen through the first use case that VVol enables storage level snapshots with VM granularity. This is due to the fact that every VMDK is represented by an independent VVol and snapshots create point in time volumes that can be used for backup and recovery.

Through this exercise we have shown that business critical database platforms such as Oracle RDBMS can be backed up and recovered in a database consistent format with ease by leveraging VVol.

In the next blog post we will examine the second use case where we will demonstrate crash consistent backup/recovery and database cloning, leveraging the capabilities of VVol for an Oracle database.

About the Author

Mohan Potheri

Mohan Potheri is VCDX#98 and has more than 20 years in IT infrastructure, with in depth experience on VMWARE virtualization. He currently focuses on evangelization of "Business Critical Applications (BCA)" and "Big Data” Virtualization on vSphere. He has extensive experience with business critical applications such as SAP, Oracle, SQL and Java across UNIX, Linux and Windows environments. Mohan Potheri is an expert on SAP virtualization and has been a speaker in multiple VMWORLD and PEX events. Prior to VMWARE, Mohan worked at many large enterprises where he has engineered fully virtualized ERP and business critical applications. He has planned, designed, implemented and managed robust highly available, DR compliant SAP on Oracle environments in UNIX and x86 environments.