Virtual Volumes for Database Backup and Recovery

In the first part of this series we provided a high level view of the benefits of using Virtual Volumes enabled storage for database operations. In this post, we will examine in more detail how Virtual Volumes can improve the backup and recovery capabilities for business critical databases, specifically Oracle.

The backups for Oracle can be Database consistent or Crash consistent. In this part we will look at Database consistent backup and recovery.

The Setup:

The solution requires VVol enabled storage. We leveraged SANBLaze VirtuaLun as the backend storage for the backup and recovery exercise. We used the VirtuaLun 7.3 emulator from SANBlaze. This emulator is VVol enabled and is one of the first VVol certified storage solutions available.

Figure 1: SANBlaze Array for VVol Testing

Oracle Database Server:

A Single Instance Oracle 12c Database with Grid Infrastructure with database name VVOL12C was setup in a VMware Virtual Machine named ORACLE-VVOL. The Oracle database was hosted on a 2 vCPU and 8GB RAM VM running Oracle Enterprise Linux 6.6 with space allocated on a SANBlaze LUN.

vCLI Package was installed on the Guest operating system in the VM. The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESX/ESXi systems from any machine with network access to those systems.

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vcli.getstart.doc/cli_install.4.4.html?path=1_0_1_2_0#991712

The virtual machine is hosted in a VMware vSphere 6 environment running on Dell POWEREDGE servers.

 Figure 2: VMware Infrastructure Details

Oracle Database Layout using VVol:

The SANBlaze creates a SubLun (a VVol in the SANBlaze environment) for every VMDK of the oracle database virtual machine (5 SubLuns for 5 vmdk’s). we have 1 extra SubLun for the configuration file.

When the VM is powered up, in additions to the 6 Subluns, there is 1 extra SubLun created for the VM swap file (.vswp file).

The disk layout for the virtual machine is as shown below. There was no storage tiering with the default tier used for the testing.

 Figure 3: Disk Layout for Oracle Virtual Machine

The subtlety lies in the way a VMFS datastore of type VVol is implemented than a regular VMFS datastore

A “df” command on the ESXi server will reveal that VVol datastores are implemented as a datastore of type “vvol”

[root@localhost:~] df
Filesystem         Bytes         Used     Available Use% Mounted on
VMFS-5     3298266447872 2363283734528   934982713344 72% /vmfs/volumes/OEL_Clone3
…..
vfat           299712512     216350720       83361792 72% /vmfs/volumes/54a30a0c-93caccba-8482-ecf4bbc08ee0
…..
vvol       10737418240000            0 10737418240000   0% /vmfs/volumes/BCA_VVOL_GOLD
vvol       21474836480000             0 21474836480000   0% /vmfs/volumes/BCA_VVOL_SILVER
…
[root@localhost:~]

Unlike any regular VM files in the VM folder on a datastore where we can see the different –flat.vmdk files , With VM’s on VVol , the folder contents looked like below:

[root@localhost:/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000] ls -l
-rw-r–r–   1 root     root           43 Jul 15 04:19 Oracle-VVOL-a59e28cb.hlog
-rw——-   1 root     root         8684 Jul 15 17:09 Oracle-VVOL.nvram
-rw——-   1 root     root           581 Jul 15 04:19 Oracle-VVOL.vmdk
-rw-r–r–   1 root     root             0 Jul 14 05:39 Oracle-VVOL.vmsd
-rwxr-xr-x   1 root     root         3996 Jul 15 17:09 Oracle-VVOL.vmx
-rw——-   1 root     root         3179 Jul 14 19:47 Oracle-VVOL.vmxf
-rw——-   1 root     root           555 Jul 15 05:06 Oracle-VVOL_1.vmdk
-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_2.vmdk
-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_3.vmdk
-rw——-   1 root     root           555 Jul 15 04:56 Oracle-VVOL_4.vmdk
-rw-r–r–   1 root     root       379606 Jul 15 04:18 vmware-1.log
-rw-r–r–   1 root     root       273649 Jul 15 17:09 vmware.log
[root@localhost:/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000]

Notice the fact that we do not have any *–flat.vmdk’s in the VM folder as they are exist on the Storage VVol!!

Logging into the SANBlaze console, the same disks when viewed from the VVol capable SANBlaze array shows each of the disks and the additional SubLuns for the configuration and the vswp files.

 [root@sanblaze1 ~]# grep SubLun /port0/alias0lun0 ; echo ; grep SubLun /port0/alias0lun0 | grep VMW_VVolType | echo “Number of Subluns = ” wc -l
SubLuns=1024
SubLun0=d28000000000,d2a000000000:600110dac045c0450100800000800000 0:4096 MB (0:8388608 blocks) -1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_VVolName=Oracle-VVOL;VMW_VVolType=Config;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0
SubLun1=d28000010000,d2a000010000:600110dac045c0450100800000800001 4096:30720 MB (8388608:62914560 blocks) -1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4
…………
……….
SubLun13=d280000d0000,d2a0000d0000:600110dac045c045010080000080000d 327680:8192 MB (671088640:16777216 blocks) -1 VMW_VvolProfile=7508ed62-e4a9-4cf1-9ef9-fb1f07b72bc1:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_VVolName=Oracle-VVOL-a59e28cb.vswp;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Swap;VMW_VvolAllocationType=3;VMW_GosType=oracleLinux64Guest;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0
…
Number of Subluns = 7

Database Backup and Recovery use case Scenarios:

Database Consistent Backup & Recovery using VVol Snapshots:

Database Consistent Backup:

Oracle defines a database consistent backup as follows: (Source: Oracle Glossary )

A Database consistent backup is a whole database backup that you can open with the RESETLOGS option without performing media recovery. In other words, you do not need to apply redo to datafiles in this backup for it to be consistent. All datafiles in a consistent backup must:

• Have the same checkpoint system change number (SCN) in their headers, unless they are datafiles in tablespaces that are read-only or offline normal (in which case they will have a clean SCN that is earlier than the checkpoint SCN)
• Contain no changes past the checkpoint SCN, that is, are not fuzzy
• Match the data file checkpoint information stored in the control file

You can only take consistent backups after you have made a clean shutdown or by turning on hot backup mode of the database.

This is the most trusted backup by DBAs but is also complex, as one needs to run scripts to put the database in hot backup mode take a snapshot and then remove it from hot backup mode.

Backup and Recovery Testing:

The following steps were used for this exercise:

1. A Database consistent snapshot was taken by using a script “hot_backup”. This Script places the database in Backup / Quiesced mode using begin / end backup mode for Oracle and takes a VMware snapshot of the VM. Details of the script are shown below:

#!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
echo “BEGIN backup mode : date”
sqlplus /nolog <<EOF
conn / as sysdba
alter database begin backup;
exit;
EOF

echo “BEGIN Snapshot : date”
/usr/bin/vmware-cmd -H 10.152.100.5 -U “vmlab\administrator” -P VMware1! –vihost 10.152.4.9 /vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000/Oracle-VVOL.vmx createsnapshot Oracle-VVOL-DBC-Snapshot ‘Oracle-VVOL-DBC-Snapshot’ 0 0
echo “END Snapshot : date“

echo “END backup mode: date”
sqlplus /nolog <<EOF
conn / as sysdba
alter database end backup;
exit;
EOF

2. A VMware snapshot is taken for the database virtual machine after putting the DB in a hot backup mode the picture below shows the DB snapshot.

Figure 4: Virtual Machine snapshot for DB Consistent Oracle

3. After snapshot is created, the database is removed from the hot backup mode.
4. On completion of the snapshot one can look at the volumes associated with the Oracle virtual machine. When the snapshot is taken each SubLun for the VMDKs have an additional copy that is created at the storage level. The number of SubLuns have now increased from 7 to 12 as explained earlier (7 SubLuns for the original VM plus 5 additional SubLuns for the VM vmdk’s)

 [root@sanblaze1 ~]# grep SubLun /port0/alias0lun0 ; echo ; grep SubLun /port0/alias0lun0 | grep VMW_VVolType | echo “Number of Subluns = ” wc -l
SubLuns=1024
……..
SubLun2=d28000020000,d2a000020000:600110dac045c0450100800000800002 165888:30720 MB (339738624:62914560 blocks) 1 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415
SubLun7=d28000070000,d2a000070000:600110dac045c0450100800000800007 196608:30720 MB (402653184:62914560 blocks) 3 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL_1.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415
SubLun8=d28000080000,d2a000080000:600110dac045c0450100800000800008 227328:30720 MB (465567744:62914560 blocks) 4 VMW_VvolProfile=f4e5bade-15a2-4805-bf8e-52318c4ce443:0;VMW_ContainerId=600110dac045c04541000000f3b02415;VMW_GosType=oracleLinux64Guest;VMW_VVolName=Oracle-VVOL_2.vmdk;VMW_VVolNamespace=/vmfs/volumes/vvol:600110dac045c045-41000000f3b02415/naa.600110dac045c0450100800000800000;VMW_VVolType=Data;VMW_VmID=501109af-235d-3c2a-c6d6-a11a096364a0;VMW_VvolAllocationType=4;VMW_VVolParentContainer=600110dac045c045-41000000f3b02415
……
Number of SubLuns = 12

5. A cloning script is then run to create a cloned DB Virtual machine from the DB consistent VMware snapshot.
6. Startup the Cloned Database VM via a script. This script performs a media recovery of the database manually using ‘recover database’ command. The recovery script is shown below:

#!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
echo
echo “BEGIN DB Consistent Recovery : date”
sqlplus /nolog <<EOF
conn / as sysdba
startup mount pfile=initVVOL12C.ora;
recover database ;
alter database open;
exit;
EOF
echo “END DB Consistent Recovery : date“

7. The database recovers successfully indicating that VVol based snapshots can be used for database consistent backup and recovery. A snippet of the Alert log of the database shown below displays the media recovery steps taken:

 tail –f alert_VVOL12C.log :

Tue Jul 21 22:38:05 2015
ALTER DATABASE RECOVER database
Tue Jul 21 22:38:05 2015
Media Recovery Start
Started logmerger process
Tue Jul 21 22:38:06 2015
Parallel Media Recovery started with 2 slaves
Tue Jul 21 22:38:06 2015
NOTE: ASMB mounting group 3 (REDO_DG)
…….
NOTE: grp 3 disk 0: REDO_DISK01 path:ORCL:REDO_DISK01
Tue Jul 21 22:38:06 2015
Recovery of Online Redo Log: Thread 1 Group 1 Seq 53 Reading mem 0
Mem# 0: +REDO_DG/VVOL12C/group01_redo01.log
Mem# 1: +REDO_DG/VVOL12C/group01_redo02.log
Tue Jul 21 22:38:06 2015
Tue Jul 21 22:38:06 2015
Media Recovery Complete (VVOL12C)
Completed: ALTER DATABASE RECOVER database
alter database open
Tue Jul 21 22:38:18 2015

VVol greatly enhances DB consistent Backup capabilities

We have seen through the first use case that VVol enables storage level snapshots with VM granularity. This is due to the fact that every VMDK is represented by an independent VVol and snapshots create point in time volumes that can be used for backup and recovery.

Through this exercise we have shown that business critical database platforms such as Oracle RDBMS can be backed up and recovered in a database consistent format with ease by leveraging VVol.

In the next blog post we will examine the second use case where we will demonstrate crash consistent backup/recovery and database cloning, leveraging the capabilities of VVol for an Oracle database.

To learn more about Virtual Volumes and how to plan, architect and administer a business-critical Oracle environment, see the VMware vSphere Virtual Volumes: A Game Changer for Business-Critical Oracle Databases white paper.