Author Archives: Sudhir Balasubramanian

Sudhir Balasubramanian

About Sudhir Balasubramanian

Sudhir Balasubramanian is a Staff Solution Architect specializing in all Oracle Technologies on VMware SDDC stack. Prior to joining VMware in 2012 , Sudhir has worked for close to 20 years as a Principal Oracle Database Administrator (DBA) / Architect in Oracle Technologies for fortune 100 companies including Tata Consultancy Services (TCS) , Sony Electronics, Epsilon Marketing (Aspen/Newgen), Teletech Corp, SAIC, Active Network and Sempra Energy Holdings before joining VMware PSO before moving into the Global Field and Partner Readiness (GFPR). Sudhir is also experienced in EMC SAN Technologies & Unix/Linux Operating Systems along with being a VMware vBCA Specialist , VMware vExpert and VMware VCA – Cloud certified. Sudhir is also a Member of the CTO Ambassador Program , run by the VMware Office of the CTO. Sudhir is a recognized Speaker having presented at Oracle Open World, IOUG, VMworld, VMware Partner Exchange, EMC World, EMC Oracle Summit and various Webinars and is an Industry recognized expert in Oracle Virtualization technologies. Sudhir has also co-authored a book "Virtualizing Oracle Business Critical Databases on VMware SDDC” which is a comprehensive authority for Oracle DBA ’s on the subject of Oracle & Linux on vSphere. https://www.amazon.com/Virtualize-Oracle-Business-Critical-Databases/dp/1500135127/ref=sr_1_1?s=books&ie=UTF8&qid=1493001047&sr=1-1&keywords=Virtualize+Oracle+Business+Critical+Databases Sudhir regularly blogs at the official VMware Application blog site https://blogs.vmware.com/apps Sudhir also blogs on his personal website http://vracdba.com Sudhir holds a degree in Master of Computer Science from San Diego State University (SDSU) graduating in 2011. Sudhir also holds a Bachelor Degree in Computer Science Engineering from Bangalore University. Twitter @vracdba

On Demand Scaling up resources for Oracle production workloads

The crux of this blog’s discussion is “How to stop hoarding much needed infrastructure resources and live wisely ever after by scaling up as needed effectively

Typically Oracle workloads running on bare metal environments , or for that matter any environment, are sized very conservatively, given the nature of the workload , with the premise that , in event of any workload spike, the abundant resources thrown at the workload will be able to sustain this spike, but in reality , we need to ask ourselves these questions

  • How much resource is actually allocated to the workload?
  • How much of that allocated resource is actually consumed by that workload ?
  • How often does the workload experience spikes?
  • If spikes are happening regularly then, has proper capacity planning and forecasting been done for this workload?

Proper plan and design along with capacity planning and forecasting is the key to manage any Business Critical Application (BCA) workload and there is no shortcut around this.

Unfortunately what this means in a physical environment is , for example, static allocation of resources to a BCA workload where the CPU utilization has been flat at 30-40% for 11 months of the year with utilization at 55-60% for the last month of the year.

Pre-allocating resources to a workload , in anticipation of peaks for say 1 month in a whole year, basically results in the resources underutilized for the rest of the year , starving other workloads of much needed resource, an ineffective way of resource allocation , thereby leading to increase in larger footprint of servers resulting in increase in CAPEX and OPEX.

Enter “Hot Plug” – “Hot Plug CPU and Hot Plug Memory” on vSphere Platform – Resource allocation on demand thereby resulting in effective and elastic resource management working on the principle of “Ask and thy shall receive”.

Continue reading

Oracle RAC on VMware Cloud on Amazon AWS

Summary

With the recent launch of the VMware Cloud on AWS Software Defined Data Center (SDDC) from VMware, many Business Critical Application (BCA) workloads that were previously difficult to deploy in the cloud no longer require significant platform modifications.

This post describes a Better Together demonstration VMware and AWS presented at VMworld 2017 using an Oracle RAC Database for high-availability zero-downtime client connection failover, supporting a Django-Python application running in a Native AWS Elastic Beanstalk environment.

Oracle RAC presents two requirements that are difficult to meet on AWS infrastructure:

  • Shared Storage
  • Multicast Layer 2 Networking.

VMware vSAN and NSX deployed into the VMware SDDC cluster meet those requirements succinctly.

The Django-Python application layer’s end-to-end provisioning is fully automated with AWS Elastic Beanstalk, which creates one or more environments containing the necessary Elastic Load Balancer, Auto-Scaling Group, Security Group, and EC2 Instances each complete with all of the python prerequisites needed to dynamically scale based on demand.  From a zip file containing your application code, a running environment can be launched with a single command.

By leveraging the AWS Elastic Beanstalk Service for the application tier, and VMware Cloud on AWS for the database tier, this end-to-end architecture delivers a high-performance, consistently repeatable, and straightforward deployment.  Better Together!

 

Architecture

 

 

In the layout above, on the right, VMware Cloud on AWS is provided by VMware directly.  For each Software Defined Data Center (SDDC) cluster, the ESXi hypervisor is installed on Bare Metal hardware provided by AWS EC2, deployed into a Virtual Private Cloud (VPC) within an AWS account owned by VMware.

Each EC2 physical host contributes 8 internal NVMe high performance flash drives, which are pooled together using VMware vSAN to provide shared storage.  This service requires a minimum number of 4 cluster nodes, which can be scaled online (via portal or REST API) to 16 nodes at initial availability, with 32 and 64-node support to follow shortly thereafter.

VMware NSX provides one or more extensible overlay logical networks for Customer virtual machine workloads, while the underlying AWS VPC CIDR block provides a control plane for maintenance and internal management of the service.

All of the supporting infrastructure deployed into the AWS account on the right side of the diagram is incorporated into a consolidated hourly or annual rate to the Customer from VMware.

In the layout above, on the left, a second AWS account directly owned by the Customer is connected to the VMware owned SDDC account for optionally consuming Native AWS services alongside deployed vSphere resources (right).

When initially deploying the VMware Cloud on AWS SDDC cluster, we need to provide temporary credentials to login to a newly created or existing Customer managed AWS account.  The automation workflow then creates an Identity and Access Management (IAM) role in the Customer AWS account (left), and grants account permissions for the SDDC to assume the role in the Customer AWS account.

This role provides a minimal set of permissions necessary to create Elastic Network Interfaces (ENIs) and route table entries within the Customer AWS account to facilitate East-West routing between the Customer AWS Account’s VPC CIDR block (left), and any NSX overlay logical networks the Customer chooses to create in the SDDC account for VM workloads (right).

The East-West traffic within the same Availability Zone provides extremely low latency free of charge, enabling the Customer to integrate technology from both vSphere and AWS within the same application, choosing the best of both worlds.

Oracle RAC Configuration

Database workloads are typically IO latency sensitive.  Per VMware KB article 2121181, there are a few recommendations to consider for significantly improving disk IO performance.

Below is the disk setup for Oracle RAC Cluster using VMware multi-writer setting which allows disks to be shared between the Oracle RAC nodes.

 

The Oracle Databases on VMware Best Practices Guide provides best practice guidelines for deploying Oracle Single Instance and Oracle RAC cluster on VMware SDDC.

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-oracle-databases-on-vmware-best-practices-guide.pdf

For the VMworld demo, the OCI compliant Oracle Instant Client was wrapped with the cx_Oracle python library, and Oracle’s Database Resident Connection Pooling (DRCP).  Database connections are initially evenly balanced between the ORCL1 and ORCL2 instances serving a custom Database Service named VMWORLD.

By failing the database service on a given node, we demonstrate that only 50% of client connections are affected, all of which can immediately reconnect to the surviving instance.

An often overlooked challenge with Oracle RAC is that client connections do not automatically fail back after repairing the failure.  Those client connections must be recycled at the resource pool level, which might require an application outage if only one pool was included in the design.  Multiplexing requests over two connection pools in your application code allows each pool to be iteratively taken out of service without taking the application down.

Given such application design changes often are not tenable post-deployment, AWS Elastic Beanstalk makes quick work of that limitation by simply deploying a GREEN copy of your application environment, validating it passes health-checks, and then transitioning your Customer workload from BLUE to GREEN stacks.  When the GREEN stack boots, its database connections will be properly balanced between instances as desired, after which the BLUE stack can then be safely terminated.  Similarly, application code changes can be deployed using the same BLUE/GREEN methodology, affording rapid rollback to the original stack if problems are encountered.  As many additional stacks can be deployed with a single command, “eb create GREEN”, or automated via REST-API.

 

At VMworld, we ran a live demo continuously failing each database service iteratively followed by an Elastic Beanstalk environment URL swap between BLUE and GREEN every 60 seconds, while monitoring Oracle’s GV$CPOOL_CC_STATS data dictionary view.  The ClassName consists of the database service name VMWORLD, followed by the Beanstalk environment name, and the application server’s EC2 instance identifier.  The second and subsequent columns of the below table indicate the RAC node servicing queries between refresh cycles.

 

 

Conclusion

 VMware Cloud on AWS affords many Better Together opportunities to not only streamline operational processes by leveraging Native AWS services, but also enable a cloud-first IT transformation without needing to disruptively re-platform your Enterprise Business Critical Applications.

The cloud based SDDC cluster deployment is simply another datacenter and cluster managed in the same way you manage your on-premises VMware environments today, without needing to retool or retrain staff.

Creating and expanding SDDC clusters can be accomplished in minutes, allowing you to drive utilization to a much higher efficiency without concern for 18-24 month capacity planning cycles that must be budgeted for peak usage.  Release burst capacity immediately after it is no longer needed without any CAPEX overhead, as well as the OPEX overhead of running your own datacenters.

 

Demo for the “Oracle RAC on VMware Cloud on Amazon AWS” can be found in the url below
https://www.youtube.com/watch?v=vpU0MW8tkhc

All Oracle on VMware SDDC collaterals can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

More information on VMware Cloud on AWS can be found at the url below
https://blog.cloud.vmware.com/s/services-and-products-vmware-cloud-on-aws

 

The Art of P2V and Oracle ASM

“Come with me if you want to live” – famous words from the Terminator series.

It’s also the very reason IT companies are adopting the ‘Virtualize First’ policy to reap all of the benefits of virtualization and move away from the soon to be legacy bare metal architecture world and ‘save a bunch of money’ , just as the Gecko said.

As part of the Virtualization journey, one of the tools VMware Professional Services (PSO), Partners and Customers use to migrate applications from physical x86 servers (Windows & Linux) to VMware Virtual Machine (VM) is using the VMware Convertor tool, the process known as P2V (Physical to Virtual). It transforms the Windows- and Linux-based physical machines and third-party image formats to VMware virtual machines.

One of the most common question I get talking the VMware field, Partners & Customers as part of my role is ‘Can I use VMware Convertor to migrate Oracle databases from physical x86 running  Linux / Oracle OVM running Linux to VMware vSphere platform ?’ , the answer, famous 2 words , ‘it depends !!’ .

Let me explain why I said that.

 

Database Re-Platforming

Oracle databases, being the sophisticated ‘beasts of burden’, there are many key factors to be kept in mind when we embark on an Oracle database re-platforming exercise, either between same / different system architectures, bare metal to bare metal / physical to virtual architecture, some of them include:

  • source and destination system architecture
    • are we moving between like architectures (x86 to x86)
    • are we moving between from a big endian system to a little endian system (Solaris / AIX / HP-UX to x86)
  • size and operating nature of the database (terabytes / production, pre-prod, dev, test etc)
  • database storage (File system / Oracle ASM)

More information on Handiness can be found in the link below
https://en.wikipedia.org/wiki/Endianness

So, if your use case is moving Oracle databases from a big endian system to a little endian system (Solaris / AIX / HP-UX to x86), Stop Right here, you cannot use the VMware Convertor tool to migrate databases between RISC Unix and Linux x86. You need an Oracle Plan and Design exercise to migrate Oracle databases between these 2 systems.

Keep reading if you are replatforming Oracle database between x86 platforms i.e. Physical server / Virtual machine (VMware vSphere / Oracle OVM) to VMware Virtual Machine (VMware).

Continue reading

“RAC” n “RAC” all night – Oracle RAC on vSphere 6.x

“I wanna “RAC” and “RAC” all night and party every day” – mantra of an Oracle RAC DBA.

Much has been written , spoken and probably beaten to senseless 🙂 about the magical “Multi-writer” setting and how it helps multiple VM’s share vmdk’s simultaneously for Clustering and FT used cases.

I still get question from customers interested in running Oracle RAC on vSphere about if we have the ability to add shared vmdk’s to a RAC cluster online without any downtime ? Yes we do. Are the steps of adding shared vmdk’s to an extended RAC cluster online without any downtime the same? Yes.

 

Introduction

By default, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled.

The below table describes the various Virtual Machine Disk Modes:

As we all are aware of , Oracle RAC requires shared disks to be accessed by all nodes of the RAC cluster.

KB Article 1034165 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s. Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is

  • has to set to Eager Zero Thick provisioned
  • need not be set to Independent persistent

While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.

Supported and Unsupported Actions or Features with Multi-Writer Flag:

**** Important ***
•    SCSI bus sharing is left at default and not touched at all in case of using shared vmdk’s
•    It’s only used for RAC’s with RDM (Raw Device Mappings) as shared disks

 

Facts about vmdk and multi-writer

Before version 6.0, we had the ability to add vmdk’s with multi-writer option to an Oracle RAC online , the only caveat was that this ability was not exposed in the vSphere Web/C# Client .We had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online.

Setting Multi Writer Flag for Oracle RAC on vSphere using Power Cli
https://blogs.vmware.com/apps/2013/10/setting-multi-writer-flag-for-oracle-rac-on-vsphere-without-any-downtime.html#more-864

http://www.virtuallyghetto.com/2015/10/new-method-of-enabling-multiwriter-vmdk-flag-in-vsphere-6-0-update-1.html

With vSphere 6.0 and onwards, we can add shared disks to an Oracle RAC Cluster online using the Web Client.

 

Key points to take away from this blog:
•    VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
•    vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
•    Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online

 

Example of an Oracle RAC Setup

As per best practices, the 2 VM’s, ‘rac01-g6’ and ‘rac02-g6’ , part of the 2 node Oracle RAC setup was deployed from a template ‘Template-12crac’.

The template has 10 vCPUs with 64 GB RAM with OEL7.3 as the operating system.

The template has 2 vmdk’s, 50GB each on SCSI 0 controller (Paravirtual SCSI Controller type)
•    Hard disk 1 is on SCSI0:0 and is for root volume (/)
•    Hard disk 2 is on SCSI0:1 and is for oracle binaries (/u01 for Grid and RDBMS binaries)

Hard Disk 1 (OS drive) & Hard Disk 2 (Oracle /u01) vmdk’s are set to
•    Thin Provisioning
•    No Sharing i.e. exclusive to the VM
•    Disk mode is set to ‘Dependent’

Template has 2 network adapters of type VMXNET3.
•    Public adapter
•    Private Interconnect

Public Adapter:

Private Interconnect:

Lets add a shared vmdk of size say 50GB to both the VM’s online without powering down the VM’s.

Add shared vmdk to an Oracle RAC online

1. Adding shared disks can be done online without downtime.

2. Add a PVSCSI Controller (SCSI 1) to RAC VM ‘rac01-g6’. Right Click on ‘rac01-g6’ , ‘Edit Settings’ and add New Controller of Type ‘Paravirtual’

Leave the SCSI Bus Sharing to ‘None’ (default)

3. Next step is to add a 50 GB shared vmdk to VM ‘rac01-g6’  to SCSI1:0 bus slot ( you can add the new vmdk it to any slot on SCSI 1 you want to)

Right Click on VM ‘rac01-g6’ and Choose ‘Edit Settings’. Choose ‘New Hard Disk’ ,  set Sharing to ‘Multi-writer’ , leave Disk mode to ‘Dependent’ and click ‘Add’. Click ‘OK’ and monitor progress.

4. Repeat Step 2 to add new ‘Paravirtual’ Controller SCSI 1 to RAC VM ‘rac02-g6’

5. . The new vmdk (vmdk with multi-writer option) created on VM ‘rac01-g6’ at SCSI1:0 bus slot needs to be shared with ‘rac02-g6’ VM for clustering purpose

6. Right Click on VM ‘rac02-g6’, Choose ‘Edit Settings’. Choose ‘Existing Hard Disk’ and Click ‘Add’.

7. Navigate to your Datastore [Group06], expand the Datastore contents and click on ‘rac01-g6’ folder. Click on the shared vmdk ‘rac01-g6_2.vmdk’ which was created on ‘rac01-g6’. Click ‘OK’

8. Note that the Sharing attribute for this vmdk needs to be set to ‘Multi-Writer’ and the SCSI controller set to the same SCSI controller as we did for ‘rac01-g6’ i.e SCSI1:0. Click ‘OK’ when done.

9. Scan the bus on the OS of both VM’s to see the new disk added and list the devices

[root@rac01-g6 ~]# fdisk -lu

Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM

….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# fdisk -lu
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM
….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac02-g6 ~]#

10. Partition Align the shared disk (/dev/sdc) on ‘rac01-g6’ (do this on one node only) using the fdisk / parted utility / tool of your choice) :

11. After partition alignment:

root@rac01-g6 ~]# fdisk -lu /dev/sdc
……
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048   104857599    52427776   83  Linux
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# fdisk -lu /dev/sdc
…..
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048   104857599    52427776   83  Linux
[root@rac02-g6 ~]#

12. Create ASM disks using ASMLIB

Installing and Configuring Oracle ASMLIB Software
https://docs.oracle.com/database/122/LADBI/installing-and-configuring-oracle-asmlib-software.htm#LADBI-GUID-79F9D58F-E5BB-45BD-A664-260C0502D876

 

root@rac01-g6 ~]# /usr/sbin/oracleasm createdisk DATA_DISK01 /dev/sda1
Writing disk header: done
Instantiating disk: done
[root@rac01-g6 ~]#

[root@rac01-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac01-g6 ~]#

[root@rac02-g6 ~]# /usr/sbin/oracleasm scandisks
[root@rac02-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac02-g6 ~]#

As we can see, we have now added a shared vmdk of size 50 GB to both VM’s without any downtime online and created ASM disk on this shared disk to be used for Oracle RAC ASM disk group.

The rest of the steps to create the Oracle RAC is exactly the same as shown in the Oracle documentation
https://docs.oracle.com/database/122/CWSOL/title.htm

########
Summary
########
Key points to keep in mind:

  • VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
  • vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
  • Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online

Best Practices needs to be followed when configuring Oracle RAC environment  which can be found in the “Oracle Databases on VMware – Best Practices Guide”

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-oracle-databases-on-vmware-best-practices-guide.pdf

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

To be “RDM for Oracle RAC”, or not to be, that is the question

Famous words from William Shakespeare’s play Hamlet. Act III, Scene I.

This is true even in the Virtualization world for Oracle Business Critical Applications where one wonders which way to go when it comes to provisioning shared disks for Oracle RAC disks, Raw Device Mappings (RDM) or VMDK ?

Much has been written and discussed about RDM and VMDK and this post will focus on the Oracle RAC shared disks use case.

Some common questions I get talking to our customer who are embarking on the virtualization journey for Oracle on vSphere are

  • What is the recommended approach when it comes to provisioning storage for Oracle RAC or Oracle Single instance? Is it VMDK or RDM?
  • What is the use case for each approach?
  • How do I provision shared RDM (s)  in Physical or Virtual Compatibility mode for an Oracle RAC environment?
  • If I use shared RDM (s)  (Physical or Virtual) will I be able to vMotion my RAC VM ’s without any cluster node eviction?

Continue reading

Streamlining Oracle on SDDC – VMworld 2017

Interested to find out how to streamline your Business Critical Applications on VMware Software-Defined Datacenter (SDDC) seamlessly?

Come attend our session at VMworld 2017 Las Vegas on Thursday, Aug 31, 1:30 p.m. – 2:30 p.m. where Amanda Blevins and  Sudhir Balasubramanian will talk about the end to end life cycle of an Application on VMware SDDC.

This includes provisioning, management, monitoring, troubleshooting, and cost transparency with the vRealize Suite. The session will also include best practices for running Oracle databases on the SDDC including sizing and performance tuning. Business continuity requirements and procedures will be addressed in the context of the SDDC. It is a formidable task to ensure the smooth operation of critical applications running on Oracle, and the SDDC simplifies and standardizes the approach across all datacenters and systems.

Signup for our session here
https://my.vmworld.com/scripts/catalog/uscatalog.jsp?search=virt1625bu&showEnrolled=false

Oracle on vSAN HCI – VMworld 2017

Interested to find out how VMware HCI vSAN solution provides high availability, workload balancing, seamless site maintenance, stability, resilience, performance and cost effective hardware required to meet critical business SLA’s for running mission critical workloads?

Come attend our session at VMworld 2017 Las Vegas on Wednesday, Aug 30, 2:30 p.m. – 3:30 p.m. where Sudhir Balasubramanian and Palanivenkatesan Murugan will talk about the VMware HCI vSAN solution for Mission Critical Oracle workloads

This session will showcase deployment of Oracle Clustered and Non Clustered databases along with running IO intensive workloads on vSAN and also talk about seamlessly running database day 2 operations like Backup & Recovery, Database Cloning , Data Refreshes , Database Patching etc using vSAN capability.

Signup for our session here
https://my.vmworld.com/scripts/catalog/uscatalog.jsp?search=STO1167BU&showEnrolled=false

Oracle on VMware vSphere, vSAN & VxRAIL – Asks the Oracles

In the last post, we endeavored to explain how to go about an Oracle Licensing Audit effectively by meticulously collecting all artifacts needed for the audit.

We recommend as artifacts, Proof of Compute Enclosure and Audit Trail entries as part of the lists of artifact to collect and store for at least 2-3 audit cycles.

We also concluded that Oracle Licensing Audit should not be taken lightly just as you would for any other software vendor but not special and one does not have to fear it.

This post endeavors to highlight the typical questions customers might have in their minds after reading articles on internet or talking to other colleagues or questions they might have encountered talking to licensing auditors.

 

Oracle Licensing Journey

During the course of my career as an Oracle DBA and Architect working on Oracle technologies, Oracle licensing was one of the facets of a DBA life I had to go through and really , nothing has changed much.

Working as the Oracle Technologies pre-sales Lead in VMware since 2012 and being the lead Oracle pre-sales field guy, talking to customers and clarifying their questions about Oracle licensing on VMware SDDC is one of my charters.

 

Continue reading

Oracle on VMware vSphere, vSAN & VxRAIL – Preparing for an the Oracle Audit

In the last post , we addressed the Licensing fuds and myths when it comes to addressing Oracle Licensing on VMware vSphere / vSAN / VxRAIL technologies and explained how Oracle licensing DOES NOT change from a licensing perspective, whether you run Oracle workloads on a classic vSphere environment or Hyper-Converged Infrastructure solution like VSAN.

This post endeavors to explain how to go about an Oracle Licensing audit effectively by meticulously collecting all artifacts needed for the audit.

FUD

Googling the word FUD does certainly explains clearly the meaning and intention of this oft used word in the Oracle Licensing space.

 

Oracle License Audit

Having put these myths to rest, let’s talk about the “Oracle License Audit” process. Many horror stories have been echoed in the hallways of IT and around water coolers but the key thing to keep in is “Yes, we need to take that seriously but no reason to be scared about it!!! , it’s just another software audit”.

The key mantra is to be “Fully prepared for it with all relevant artifacts to defend the audit”.

We have well established beyond any reasonable doubt in the previous blog post that Oracle licensing is not Memory, Storage, Cluster, vCenter or Network based, it’s either User based (Named User Plus) or Processor(Socket in case of SE2 or cores in case of EE edition).

 

Successfully defending an Oracle Audit

The primary goal of effectively defending an Oracle Licensing Audit on VMware vSphere/VSAN is to prove that an effective “Compute Segmentation” has been done to ensure that Oracle Virtual Machines runs on dedicated ESXi servers in the datacenter, because again, to re-iterate, Oracle licensing is Compute (SE2/EE)  /  User (NUP) based.

We can achieve the above goal in 2 ways
1)    Create a “Compute Enclosure” to prevent VM’s from leaving the enclosure by any means whatsoever
2)    Establishing an auditing mechanism of documenting  VM movements via vMotion events in the above “Compute Enclosure”

 

Create “Compute Enclosure”

There are 2 ways to create the “Compute Enclosure”:

Option A: Dedicated vSphere Cluster for Oracle VM’s (Recommended). This model is a widely accepted model purely from an Oracle licensing perspective.

Option B: Common vSphere Cluster where we use Affinity rules to bind Oracle VM‘s to a set of ESXi servers dedicated for Oracle workloads

Either of the 2 ways are acceptable as the Oracle OLSA / OMA does not stipulate anything about vSphere Cluster apart from the definition of the Processor as “Processor shall be defined as all processors where the Oracle programs are installed and/or running.”

In case of option B, the process of pinning Oracle VM’s to ESXI hosts have been explained in the previous blog post

https://blogs.vmware.com/apps/2017/01/oracle-vmware-vsan-dispelling-licensing-myths.html

Having created the “Compute Enclosure” i.e. a vSphere Cluster for Oracle Workloads, now we need to establish an auditing mechanism of documenting the Oracle VM movements by tracking the movement of the Oracle VM’s via vMotion events within the above “Compute Enclosure”.

 

Database Options in the Oracle “Compute Enclosure”

We could have  Oracle VM’s with certain database options turned on as part of the Oracle deployment , for ex Advanced Compression , Partitioning, Database Tuning Pack.

https://docs.oracle.com/database/122/DBLIC/Licensing-Information.htm#DBLIC-GUID-B6113390-9586-46D7-9008-DCC9EDA45AB4

In order to ensure that we pay database options only for a particular VM / a set of Oracle VM’s instead of paying for ALL the Oracle VM’s , whether the options are used or not, lets look at a couple of scenarios with the above Cluster deployment models (Option A and B) :

  1. Licensing database options for a set of VM’s

In this scenario , regardless of whether we use a dedicated or shared vSphere Cluster for Oracle VM’s, we use DRS rules to constrict the movement of these set of VM’s which have the database options to a subset of ESXI servers within the vSphere cluster.

Ex : In a dedicated vSphere cluster for Oracle with 5 ESXi servers  and say we have 10 VM’s with database options and assuming that 2 ESXi servers is sufficient to run the workload for these 10 VM’s, we  can choose to constrict the movement of these 10 VM’s , using DRS rules, to these 2 ESXI servers which within the 5 node vSphere cluster

   2. Licensing database options for a VM

In this scenario , regardless of whether we use a dedicated or shared vSphere Cluster for Oracle VM’s, we use CPU Affinity rules to constrict the movement of this VM which has the database options to an ESXI server within the vSphere cluster, that way we only pay licensing for Database Options for that VM only for a set of cores within a socket within an ESXI Server.

Ex : In a dedicated vSphere cluster for Oracle with 5 ESXi servers , each with 2 socket x 10 cores per socket and say we have 1 VM with database options with 10vCPU’s ,  we can then pin the 10 vCPU’s of this VM to cores 0-9 of socket 0 of ESXI 1 in the 5 node vSphere cluster.

 

Establishing Audit Mechanisms

Audit Information about VM Power on/off event

In the previous blog post, we showed how the VM Power On operations audit information is recorded in the vmware.log file.
https://blogs.vmware.com/apps/2017/01/oracle-vmware-vsan-dispelling-licensing-myths.html

Let’s see the contents of the vmware.log file for “testoravm” when we power it up on a vSAN Cluster

[root@w2-pe-vsan-esx-029:/vmfs/volumes/vsan:52803547e520f694-1f6104395ada7b7c/05735458-cc86-e1e9-ca71-0025b501004e] cat vmware.log
2016-12-27T21:09:09.124Z| vmx| I125: Log for VMware ESX pid=2597049 version=6.5.0 build=build-4564106 option=Release
2016-12-27T21:09:09.124Z| vmx| I125: The process is 64-bit.
2016-12-27T21:09:09.124Z| vmx| I125: Host codepage=UTF-8 encoding=UTF-8
2016-12-27T21:09:09.124Z| vmx| I125: Host is VMkernel 6.5.0
2016-12-27T21:09:09.091Z| vmx| I125: VTHREAD initialize main thread 0 “vmx” tid 2597049
2016-12-27T21:09:09.092Z| vmx| I125: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
……….
……….
2016-12-27T21:09:09.124Z| vmx| I125: Hostname=w2-pe-vsan-esx-029
2016-12-27T21:09:09.124Z| vmx| I125: IP=127.0.0.1 (lo0)
…..
[root@w2-pe-vsan-esx-029:/vmfs/volumes/vsan:52803547e520f694-1f6104395ada7b7c/05735458-cc86-e1e9-ca71-0025b501004e]

The Power On process of an Oracle VM on a classic vSphere Cluster also records the information of the host it powers on, no different than the o/p we see above on a vSAN Cluster.

[root@wdc-esx10:/vmfs/volumes/56bce95e-eb1c7670-1464-0025b3b1b790/Template_OEL70] more vmware.log
2016-11-02T04:36:09.871Z| vmx| I120: Log for VMware ESX pid=3165445 version=6.0.0 build=build-3029758 option=Release
2016-11-02T04:36:09.871Z| vmx| I120: The process is 64-bit.
2016-11-02T04:36:09.871Z| vmx| I120: Host codepage=UTF-8 encoding=UTF-8
2016-11-02T04:36:09.871Z| vmx| I120: Host is VMkernel 6.0.0
2016-11-02T04:36:09.854Z| vmx| I120: VTHREAD initialize main thread 0 “vmx” pid 3165445
2016-11-02T04:36:09.854Z| vmx| I120: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
….
2016-11-02T04:36:09.856Z| vmx| I120: DictionaryLoad: Cannot open file “//.vmware/config”: No such file or directory.
……..
2016-11-02T04:36:09.859Z| vmx| I120: PREF Failed to load user preferences.
2016-11-02T04:36:09.872Z| vmx| I120: Hostname=wdc-esx10.tsalab.local

 

Audit Information about VM vMotion event

Let’s see the contents of the vmware.log file of an Oracle VM when we vMotion it from one ESXi server to another ESXi server within a vSphere Cluster

[root@wdc-esx10:/vmfs/volumes/56bce95e-eb1c7670-1464-0025b3b1b790/Template_OEL70] more vmware.log
2016-11-02T04:36:09.871Z| vmx| I120: Log for VMware ESX pid=3165445 version=6.0.0 build=build-3029758 option=Release
2016-11-02T04:36:09.871Z| vmx| I120: The process is 64-bit.
2016-11-02T04:36:09.871Z| vmx| I120: Host codepage=UTF-8 encoding=UTF-8
2016-11-02T04:36:09.871Z| vmx| I120: Host is VMkernel 6.0.0
2016-11-02T04:36:09.854Z| vmx| I120: VTHREAD initialize main thread 0 “vmx” pid 3165445
2016-11-02T04:36:09.854Z| vmx| I120: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
….
2016-11-02T04:36:09.856Z| vmx| I120: DictionaryLoad: Cannot open file “//.vmware/config”: No such file or directory.
……..
2016-11-02T04:36:09.859Z| vmx| I120: PREF Failed to load user preferences.
2016-11-02T04:36:09.872Z| vmx| I120: Hostname=wdc-esx10.tsalab.local

The VM was initially powered on wdc-esx10.tsalab.local server.

When the Oracle VM vMotion to another ESXI server either done manually or through DRS events the vMotion entries along with the source and target ESXI servers are recorded in the vmware.log file.

In the above case the Oracle VM vMotioned from wdc-esx10.tsalab.local server to wdc-esx09.tsalab.local server

root@wdc-esx10:/vmfs/volumes/56bce95e-eb1c7670-1464-0025b3b1b790/Template_OEL70] more vmware.log
…..
2016-11-02T04:44:38.156Z| vmx| I120: MigrateVMXdrToSpec: type: 1 srcIp=<10.128.136.110> dstIp=<10.128.136.109> mid=5404a192575ee uuid=38383135-3735-5355-4530-343132465936 priority=yes checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>>
….

2016-11-02T04:44:38.156Z| vmx| I120: Received migrate ‘from’ request for mid id 1478061877196270, src ip <10.128.136.110>.
….
…..
2016-11-02T04:44:38.156Z| vmx| I120:    OpType: vmotion
…..
2016-11-02T04:44:38.200Z| vmx| I120: UNAME VMkernel wdc-esx09 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 (uwglibc release: vmware, version: 2.12.2)

The above audit trail entries are able to correctly report on the below events
•    VM Power on / off
•    VM vMotion to / from

The same Audit entries can also be captured from the vCenter database by mining the database for VM Power on / off and VM vMotion to / from events. We need to be mindful of the purge retention settings for Oracle/SQL Server vCenter database in order to ensure that we have audit trail entries for at least 2-3 audit cycles.

As we can see by creating a “Compute Enclosure” and establishing a “Effective Audit Mechanism”, we can conclusively day without any doubt that the Oracle VM’ always lived and migrated within the “Compute Enclosure” and never wandered outside !!!.

Tools to help gather audit trail

Another product from VMware which helps for purpose of Oracle Auditing is the VMware vRealize Log Insight which delivers heterogeneous and highly scalable log management with intuitive, actionable dashboards, sophisticated analytics and broad third-party extensibility. It provides deep operational visibility and faster troubleshooting across physical, virtual and cloud environments.

VMware LogInsight dashboard can help customers gather by means of audit trail records which can then be presented to Oracle LMS team as proof of Oracle workload footprint within a vSphere Cluster or a vSAN cluster.

The video below demonstrates the capabilities of VMware vRealize LogInsight for Oracle License Compliance
https://www.youtube.com/watch?v=EHcT4xDyONc

Also keep in mind the below listed controls demanded by licensing zealots is completely un-necessary and non-contractual.

-Not needed to create Network Segmentation to separate and dedicate a network segment for the vSphere Cluster for Oracle workloads

-Not needed to create Storage Segmentation to zone, map and mask Oracle specific storage LUNS to only the ESXI servers  in the dedicated vSphere Cluster for Oracle

-Do not run PowerCLI scripts / commands against the vCenter database which shows all the ESXI servers connected to the vCenter regardless of whether they are part of the vSphere dedicated cluster for Oracle or not.

If you have to run it to gather information about the ESXi servers in the Oracle vSphere Cluster, login as the user who has access to only the Oracle cluster so that way it reduces the scope of discovery to only the Oracle Cluster

This is the document which is handed out to Customers which has information how to gather information about the ESXi servers connected to the Virtual Center , it does not specify running the script against the Oracle vSphere Cluster.

 

 

A key point to keep in mind is if this document is really contractual , why is this NOT public facing ?

-Do not give access to any auditor the keys of the kingdom i.e. vCenter username and password

Really, what’s next? Separate the vSphere Cluster for Oracle in its own cage in the data center and ensure no one goes near it!! Throw a black cloth around the cage so that no one can see what’s in it?

Both of the above steps are completely un-necessary as we have well established beyond any reasonable doubt in the previous blog post that Oracle licensing is not Memory, Storage, Cluster, vCenter or Network based, it’s either User based (Named User Plus) or Processor(Socket in case of SE2 or cores in case of EE edition).

 

Artifacts helpful for an Oracle Licensing Audit defense

Here are some of the important artifacts which are useful for an Oracle Licensing audit defense

1)    Proof of Compute Enclosure
a.    Screenshot of the vSphere dedicated cluster for Oracle Workloads

b.    Screenshot of one of the ESXI servers in the cluster which clearly shows Processor Family, number of Socket and number of Cores

The Effective number of cores calculation can be found in the previous blog post
https://blogs.vmware.com/apps/2017/01/oracle-vmware-vsan-dispelling-licensing-myths.html

2. Audit Trail entries which are log file entries for every Oracle VM which shows the Power on /off and vMotion to / from operations.

VMware LogInsight can be used to extract these entries and the video below demonstrates the capabilities of VMware vRealize LogInsight for Oracle License Compliance:
https://www.youtube.com/watch?v=EHcT4xDyONc

The same Audit entries can also be captured from the vCenter database by mining the database for VM Power on / off and VM vMotion to / from events. We need to be mindful of the purge retention settings for Oracle/SQL Server vCenter database in order to ensure that we have audit trail entries for at least 2-3 audit cycles.

Conclusion
In conclusion, Oracle Licensing Audit should not be taken lightly just as you would for any other software vendor but not special and one does not have to fear it.

Be prepared with all the audit artifacts as detailed above.

 

Need Help?
For any additional Oracle Licensing on VMware clarification or help, please reach out to your respective VMware Account teams who can get our team involved in a discussion (Internal VMware folks can reach directly to us at the Tier1-Apps-Sales-Support team mailing list) and we can definitely help guide you and connect you to some of our Premier specialist partners for further discussions.

Oracle on VMware SDDC Collateral
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, and workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop [Customer]
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

VMworld 2017 Oracle Customer Bootcamps

VMworld 2017 Oracle Customer Bootcamps

On a mission to arm yourself with the latest knowledge and skills needed to master application virtualization?

reading book

VMworld Customer bootcamps can get you in shape to lead the virtualization charge in your organization, with Instructor-led demos and In-depth course work designed to put you in the ranks of the IT elite.

Oracle on vSphere
The Oracle on VMware vSphere Bootcamp will provide the attendee the opportunity to learn the essential skills necessary to run Oracle implementations on VMware vSphere. The best practices and optimal approaches to deployment, operation and management of Oracle database and application software will be presented by VMware expert Sudhir Balasubramanian who will be joined by other VMware and Industry Experts.

This technical workshop will exceed the standard breakout session format by delivering “real-life,” instructor-led, live training and incorporating the recommended design and configuration practices for architecting Business Critical Databases on VMware vSphere infrastructure. Subjects such as Real Applications Clusters, Automatic Storage Management, vSAN and NSX will be covered in depth.

Learn More

https://www.vmworld.com/en/us/learning/sessions.html?mid=9592&eid=CVMW2000001358867&elqTrackId=ac4f78fd201d4b8ea8c06c94903ec64e&elq=a30d659ad2934a969e912b357d9624d2&elqaid=9592&elqat=1&elqCampaignId=4153

Details

Cost: $725 / seat

Schedule:
Saturday August 26, 2017
8:00am to 5:00pm
(registration opens at 6:30am)

Location:
Mandalay Bay, South Convention Centre
3rd Floor Jasmine Rooms

Registration

Be sure to add the Bootcamp in step 4 of your VMworld conference registration, under Educational Offerings, after you’ve selected your conferences pass.

Registration is open, seating is limited! Lunch and breaks provided.

Looking forward to seeing you all there!