Oracle on VMware Hybrid Multi-Clouds – Preparing for an the Oracle Audit

In the last post , we addressed the Licensing fuds and myths when it comes to addressing Oracle Licensing on VMware vSphere , vSAN & VMware Cloud on AWS technologies and explained how –

Oracle licensing DOES NOT change , from a licensing perspective, whether you run Oracle workloads on a

Classic vSphere environment connected to a NAS / SAN / iSCSI / NVMe
Hyper-Converged Infrastructure solution vSAN
Oracle workloads on VMware Cloud e.g. VMware Cloud on AWS

This post endeavors to explain how to go about an Oracle Licensing audit effectively by meticulously collecting all artifacts needed for the audit.

FUD

Googling the word FUD does certainly explains clearly the meaning and intention of this oft used word in the Oracle Licensing space.

Purpose of this Blog

The purpose of this blog is to provide customers with information along with GUIDANCE when it comes to certification, support, and licensing of Oracle on VMware vSphere
This information along with GUIDANCE is based on the experience and knowledge that VMware and proponents of its technology have acquired from over a decade during which VMware has successfully virtualized the majority of Oracle workloads on vSphere
This blog does not provide ANY legal advice concerning a customer’s license or support agreement with Oracle or any other third party. Rather, this blog is intended to help customers understand the issues and be better prepared for optimal licensing interaction with Oracle and third-party vendors.

In addition to the blog, please refer to the Oracle on vSphere Certification, Support and Licensing Guide 2017

Oracle License Audit

Having put these myths to rest, let’s talk about the “Oracle License Audit” process. Many horror stories have been echoed in the hallways of IT and around water coolers but the key thing to keep in is “Yes, we need to take that seriously but no reason to be scared about it!!! , it’s just another software audit”.

The key mantra is to be “Fully prepared for it with all relevant artifacts to defend the audit”.

We have well established beyond any reasonable doubt in the previous blog post that Oracle licensing is not Memory, Storage, Cluster, vCenter or Network based, it’s either User based (Named User Plus) or Processor(Socket in case of SE2 or cores in case of EE edition).

Oracle licensing DOES NOT change , from a licensing perspective, whether you run Oracle workloads on a

Classic vSphere environment connected to a NAS / SAN / iSCSI / NVMe
Hyper-Converged Infrastructure solution vSAN
Oracle workloads on VMware Cloud e.g. VMware Cloud on AWS

Successfully defending an Oracle Audit

The primary goal of effectively defending an Oracle Licensing Audit on VMware vSphere/vSAN is to prove that an effective “Compute Segmentation” has been done to ensure that Oracle Virtual Machines runs on dedicated ESXi servers in the datacenter, because again, to re-iterate, Oracle licensing is Compute (SE2/EE) / User (NUP) based.

On a high level, we can achieve the above goal in 3 steps

1) Create a “Compute Enclosure” to prevent VM (s) from leaving the enclosure by any means whatsoever

Option A: Dedicated vSphere Cluster for Oracle VM (s)
Option B: Common vSphere Cluster with VM-Host MUST Affinity rule to bind Oracle VM (s) to a set of ESXi servers dedicated for Oracle workloads , wirhin the vSphere cluster
Option C: VM vCPU Affinity to bind the VM to a set number of physical core (s) within a physical socket (S) in an ESXi server

2) Establishing an auditing mechanism of documenting VM (s) movements via vMotion events in the above “Compute Enclosure”. The 4 VM events critical to Oracle licensing are

VM Power ON
VM Power OFF
VM vMotion TO
VM vMotion FROM

3) Tying the above findings of the audit to the Oracle License and Service Agreement (OLSA) / Oracle Master Agreement(OMA) , specifically to the line which clearly states “Processor: shall be defined as all processors where the Oracle programs are installed and/or running.”

Create “Compute Enclosure”

There are 3 ways to create the “Compute Enclosure” , one of the way being the most restrictive form of enclosure.

Example of Option A – Dedicated vSphere Cluster for Oracle VM’s

This model is a widely accepted model purely from an Oracle licensing perspective. There is NO need to have VM-Host MUST Affinity rule as DRS will NOT migrate the VM’s outside of the vSphere cluster.

For example, Consider a dedicated cluster of 4 servers with each ESXi server with 2 socket x 16 cores.

In this example , to run Oracle workloads in this scenario –

Total no of effective cores for licensing Oracle workloads on 4 ESXi servers in the vSphere Cluster using Enterprise Edition (EE)

= Absolute number of cores in cluster * Processor Core Factor
= 4 servers * 2 sockets * 16 cores/socket * Processor Core Factor
= 128 * 0.5
= 64 Effective cores liable for Oracle licensing

Example of Option B – Common vSphere Cluster with VM-Host MUST Affinity rule

In the example below, we have a vSAN cluster called “vSANCluster” comprising of 4 ESXi servers contributing Compute, Storage & Network resources to the cluster.

The 4 servers are esx-029, esx-030, esx-031 and esx-032.

Each ESXi server has 2 socket x 16 cores as shown below. The processor is Intel Family.

Using VM-Host MUST Affinity, Oracle VM’s can be pinned to a sub-set of ESXi servers in the “vSANCluster” for Oracle licensing reasons. Let’s see how we can go about doing it.

For example, VM ‘testoravm’ needs to be pinned to hosts ‘w2-pe-vsan-esx-029.eng.vmware.com’ and ‘w2-pe-vsan-esx-030.eng.vmware.com’ for Oracle licensing purposes.

Create a VM/Host Group mapping as shown below. In the example below, “OracleVM” is the group created which has a VM “testoravm”. There is a 1:1 mapping between the “OracleVM” group and “testoravm” VM.

Create a new Host Group “OracleVM-Host” as shown below. Hosts w2-pe-vsan-esx-029.eng.vmware.com and w2-pe-vsan-esx-030.eng.vmware.com are part of this host group.

Create new VM/Host rules as shown below which basically will have the VM group “OracleVM” created above (with the “testoravm” VM in the group) mapped to the Host Group “OracleVM-Host” with a “must run on hosts in

group”.

In effect, the Oracle VM “testoravm” is now constricted and contained to run in the Compute Segmentation effectively created above.

In the example above, to run Oracle workloads in the above scenario:

Total no of effective cores for licensing Oracle workloads on 2 ESXi servers in the vSAN Cluster above using Enterprise Edition (EE) = Absolute number of cores in cluster * Processor Core Factor
= 2 servers * 2 sockets * 16 cores/socket * Processor Core Factor
= 64 * 0.5
= 32 Effective cores liable for Oracle licensing

From the example above Oracle VM “testoravm” can run only on hosts w2-pe-vsan-esx-029.eng.vmware.com and w2-pe-vsan-esx-030.eng.vmware.com based on the DRS Affinity Rules.

We ONLY need to license 2 ESXI servers for Oracle workloads as Oracle workloads runs only 2 ESXI servers, regardless of the size of the vSAN Cluster.

Example of Option C – VM vCPU Affinity to bind the VM to a set number of physical core (s) within a physical socket (S) in an ESXi server

In this model , we use VM vCPU Affinity to bind the VM to a set number of physical core (s) within a physical socket (S) in an ESXi server to disable the movement of this VM to this ESXi server only.

Using CPU affinity, you can assign a virtual machine to a specific processor. This allows you to restrict the assignment of virtual machines to a specific available processor in multiprocessor systems.

More information on this can be found here.

For example , consider an ESXi with 2 socket x 14 cores.

VM ‘Oracle122-OL7’ has 7 vCPU’s.

Pin VM ‘Oracle122-OL7’ vCPU’s to as shown below. Under Scheduling Affinity, select physical processor affinity for the virtual machine. Use ‘-‘ for ranges and ‘,’ to separate values. Select the processors where you want the virtual machine to run and click OK.

Either of the 3 ways are acceptable as the Oracle OLSA / OMA does not stipulate anything about vSphere Cluster or Using of CPU pinning apart from the definition of the Processor as “Processor shall be defined as all processors where the Oracle programs are installed and/or running.”

The ultimate proof of the above statement comes from the Oracle OLSA/OMA which defines Processor as “Processor: shall be defined as all processors where the Oracle programs are installed and/or running.”

The above example holds good for a classic vSphere or VxRAIL or VMware Cloud on AWS scenario as well.

Establishing Audit Mechanisms

Having created the “Compute Enclosure” i.e. either a Dedicated vSphere cluster OR a Shared vSphere cluster with VM-Host MUST affinity rules, for Oracle Workloads, now we need to establish an auditing mechanism of documenting the Oracle VM movements by tracking the movement of the Oracle VM’s via vMotion events within the above “Compute Enclosure”.

The important artifact which helps to prove , without any doubt , that the Oracle VM “testoravm” ran on , for example , only 2 ESXI servers as shown in the example above, is via the VM Audit events in the Virtual Machine log file.

By default, ESXi/ESX hosts store virtual machine-specific logging in the same directory as the virtual machine’s configuration files. The default log file name is vmware.log.

The 4 VM events critical to Oracle licensing are

• VM Power ON
• VM Power OFF
• VM vMotion TO
• VM vMotion FROM

Audit Information about VM Power on / off event

Let’s see the contents of the vmware.log file for “testoravm” when we power it up on. Initially its assigned to ‘w2-pe-vsan-esx-029.eng.vmware.com’ when its created.

[root@w2-pe-vsan-esx-029:/vmfs/volumes/vsan:52803547e520f694-1f6104395ada7b7c/05735458-cc86-e1e9-ca71-0025b501004e] cat vmware.log
2016-12-27T21:09:09.124Z| vmx| I125: Log for VMware ESX pid=2597049 version=6.5.0 build=build-4564106 option=Release
2016-12-27T21:09:09.124Z| vmx| I125: The process is 64-bit.
2016-12-27T21:09:09.124Z| vmx| I125: Host codepage=UTF-8 encoding=UTF-8
2016-12-27T21:09:09.124Z| vmx| I125: Host is VMkernel 6.5.0
2016-12-27T21:09:09.091Z| vmx| I125: VTHREAD initialize main thread 0 “vmx” tid 2597049
2016-12-27T21:09:09.092Z| vmx| I125: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
……….
……….
2016-12-27T21:09:09.124Z| vmx| I125: Hostname=w2-pe-vsan-esx-029
2016-12-27T21:09:09.124Z| vmx| I125: IP=127.0.0.1 (lo0)
…..
[root@w2-pe-vsan-esx-029:/vmfs/volumes/vsan:52803547e520f694-1f6104395ada7b7c/05735458-cc86-e1e9-ca71-0025b501004e]

The Power On process of an Oracle VM on a classic vSphere Cluster also records the information of the host it powers on, its no different than the o/p we see above on a vSAN Cluster.

Example of a Power On event of a VM on classic vSphere Cluster :

[root@wdc-esx10:/vmfs/volumes/56bce95e-eb1c7670-1464-0025b3b1b790/Template_OEL70] more vmware.log
2016-11-02T04:36:09.871Z| vmx| I120: Log for VMware ESX pid=3165445 version=6.0.0 build=build-3029758 option=Release
2016-11-02T04:36:09.871Z| vmx| I120: The process is 64-bit.
2016-11-02T04:36:09.871Z| vmx| I120: Host codepage=UTF-8 encoding=UTF-8
2016-11-02T04:36:09.871Z| vmx| I120: Host is VMkernel 6.0.0
2016-11-02T04:36:09.854Z| vmx| I120: VTHREAD initialize main thread 0 “vmx” pid 3165445
2016-11-02T04:36:09.854Z| vmx| I120: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
….
2016-11-02T04:36:09.856Z| vmx| I120: DictionaryLoad: Cannot open file “//.vmware/config”: No such file or directory.
……..
2016-11-02T04:36:09.859Z| vmx| I120: PREF Failed to load user preferences.
2016-11-02T04:36:09.872Z| vmx| I120: Hostname=wdc-esx10.tsalab.local

Audit Information about VM vMotion to / from event

Let’s see the contents of the vmware.log file of an Oracle VM when we vMotion it from one ESXi server to another ESXi server within a vSphere Cluster. The o/p is similar when we do the same operation on a vSAN or VxRAIL environment.

The VM was initially powered on wdc-esx10.tsalab.local server.

When the Oracle VM vMotions to another ESXI server , either manually or through DRS events , the vMotion entries along with the source and target ESXI servers are recorded in the vmware.log file.

In the above case the Oracle VM vMotioned from ‘wdc-esx10.tsalab.local’ server to ‘wdc-esx09.tsalab.local’ server

root@wdc-esx10:/vmfs/volumes/56bce95e-eb1c7670-1464-0025b3b1b790/Template_OEL70] more vmware.log
…..
2016-11-02T04:44:38.156Z| vmx| I120: MigrateVMXdrToSpec: type: 1 srcIp=<10.128.136.110> dstIp=<10.128.136.109> mid=5404a192575ee uuid=38383135-3735-5355-4530-343132465936 priority=yes checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>>
….
…
2016-11-02T04:44:38.156Z| vmx| I120: Received migrate ‘from’ request for mid id 1478061877196270, src ip <10.128.136.110>.
….
…..
2016-11-02T04:44:38.156Z| vmx| I120: OpType: vmotion
…..
2016-11-02T04:44:38.200Z| vmx| I120: UNAME VMkernel wdc-esx09 6.0.0 #1 SMP Release build-3029758 Aug 31 2015 00:54:00 x86_64 (uwglibc release: vmware, version: 2.12.2)

The same Audit entries can also be captured from the vCenter database by mining the database for VM Power on / off and VM vMotion to / from events. We need to be mindful of the purge retention settings for Oracle/SQL Server vCenter database in order to ensure that we have audit trail entries for at least 2-3 audit cycles.

As we can see by creating a “Compute Enclosure” and establishing a “Effective Audit Mechanism”, we can conclusively day without any doubt that the Oracle VM’ always lived and migrated within the “Compute Enclosure” and never wandered outside !!!.

More information about the vmware.log files can be found in the below KB articles:

Locating virtual machine log files on an ESXi/ESX host (1007805)

Location of log files for VMware products (1021806)

Log rotation and logging options for vmware.log (8182749)

Another product from VMware which helps for purpose of Oracle Auditing is the VMware vRealize Log Insight which delivers heterogeneous and highly scalable log management with intuitive, actionable dashboards, sophisticated analytics and broad third-party extensibility. It provides deep operational visibility and faster troubleshooting across physical, virtual and cloud environments.

VMware LogInsight dashboard can help customers gather by means of audit trail records which can then be presented to Oracle LMS team as proof of Oracle workload footprint within a vSphere Cluster or a vSAN cluster.

This video demonstrates the capabilities of VMware vRealize LogInsight for Oracle License Compliance.

For VMware Hybrid Cloud , for example, VMware Cloud on AWS , Azure VMware Solution, Oracle Cloud VMware Solution , LogInsight can be used to extract the above VM entries where direct ssh access is allowed to ESXi server . In case where direct ssh access is not allowed, for example VMware Cloud on AWS, vRealize Log Intelligence product can be used to extract the above VM entries.

Important factor to keep in mind is vSAN’s in kernel design reduces the need for compute sprawl and hence more Oracle licensing as compared to some of the other competitor products found in the market.

Closing the Loop

The final step is to close the loop by tying the results of the above audit findings to the Oracle License and Service Agreement (OLSA) / Oracle Master Agreement(OMA) , specifically to the line which clearly states “Processor: shall be defined as all processors where the Oracle programs are installed and/or running.”

Essentially , the artifacts to be collected are –

1. Compute Enclosure

Dedicated vSphere Cluster – Using Web Client, A screenshot of the cluster showing ESXI servers in that vSphere cluster. Example of a Screenshot of the vSphere dedicated cluster for Oracle Workloads

Shared vSphere Cluster with VM-Host MUST Affinity – Using Web Client, A screenshot of the cluster showing only that subset of ESXi servers in that shared vSphere cluster which is part of the VM-Host MUST affinity rule

Screenshot of one of the ESXi servers in the above cluster which clearly shows Processor Family, number of Socket and number of Cores

The Effective number of cores calculation can be found in the previous blog post
https://blogs.vmware.com/apps/2017/01/oracle-vmware-vsan-dispelling-licensing-myths.html

VM vCPU Affinity to bind the VM to a set number of physical core (s) within a physical socket (S) in an ESXi server – Screenshot of the VM with the CPU Affinity clearly shown

2. Results of the Audit trail

Centralized log file which shows the Power On / Power Off / vMotion To / vMotion From events from all Oracle VM’s in the cluster . VMware LogInsight can be used to extract these entries and the video below demonstrates the capabilities of VMware vRealize LogInsight for Oracle License Compliance:

https://www.youtube.com/watch?v=EHcT4xDyONc

In case of the vCPU pinning example , we can collect the Power On & Power Off events from the Oracle VM.

3. Tie the results of the above audit findings to the Oracle License and Service Agreement (OLSA) by making this statement:

Given the results of the audit trail which clearly shows , for example, Oracle VM’s only powering on/off and vMotioning to and from within a subset of the ESXI servers in this cluster , why do we have to pay for those ESXI servers where Oracle VM’s NEVER powered ON/ OFF and vMotioned TO/FROM, especially as per the OLSA/OMA which very specifically states “Processor: shall be defined as all processors where the Oracle programs are installed and/or running.”. Doing so will be going against the OLSA !!!

Database Options in the Oracle “Compute Enclosure”

We could have Oracle VM’s with certain database options turned on as part of the Oracle deployment , for ex Advanced Compression , Partitioning, Database Tuning Pack.

https://docs.oracle.com/database/122/DBLIC/Licensing-Information.htm#DBLIC-GUID-B6113390-9586-46D7-9008-DCC9EDA45AB4

In order to ensure that we pay database options only for a particular VM / a set of Oracle VM’s instead of paying for ALL the Oracle VM’s , whether the options are used or not, lets look at a couple of scenarios with the above Cluster deployment models (Option A and B) :

Licensing database options for VM (s) in a vSphere cluster

In this scenario , regardless of whether we use a dedicated or shared vSphere Cluster for Oracle workloads , we use the DRS MUST Affinity rules to constrict the movement of the VM (s) , which has the database options , to a subset of ESXi servers within the vSphere cluster.

Ex : In a dedicated vSphere cluster for Oracle with 5 ESXi servers and say we have 10 VM’s with database options and assuming that 2 ESXi servers is sufficient to run the workload for these 10 VM’s, we can choose to constrict the movement of these 10 VM’s , using DRS rules, to these 2 ESXI servers which within the 5 node vSphere cluster

2. Licensing database options for a VM on an ESXi server

In this scenario , regardless of whether we use a dedicated or shared vSphere Cluster for Oracle VM’s, we use CPU Affinity rules to constrict the movement of this VM , which has the database options , to an ESXi server within the vSphere cluster, that way we only pay licensing for Database Options for that VM only for a set of cores within a socket within an ESXI Server.

Ex : In a dedicated vSphere cluster for Oracle with 5 ESXi servers , each with 2 socket x 10 cores per socket and say we have 1 VM with database options with 10vCPU’s , we can then pin the 10 vCPU’s of this VM to cores 0-9 of socket 0 of ESXI 1 in the 5 node vSphere cluster.

Tools to help gather audit trail

Another product from VMware which helps for purpose of Oracle Auditing is the VMware vRealize Log Insight which delivers heterogeneous and highly scalable log management with intuitive, actionable dashboards, sophisticated analytics and broad third-party extensibility. It provides deep operational visibility and faster troubleshooting across physical, virtual and cloud environments.

The video below demonstrates the capabilities of VMware vRealize LogInsight for Oracle License Compliance
https://www.youtube.com/watch?v=EHcT4xDyONc

Conclusion

In conclusion, Oracle Licensing Audit should not be taken lightly just as you would for any other software vendor but not special and one does not have to fear it.

Be prepared with all the audit artifacts as detailed above.

Oracle licensing DOES NOT change , from a licensing perspective, whether you run Oracle workloads on a

Classic vSphere environment connected to a NAS / SAN / iSCSI / NVMe
Hyper-Converged Infrastructure solution vSAN
Oracle workloads on VMware Cloud e.g. VMware Cloud on AWS

Need Help?

For any additional Oracle Licensing on VMware clarification or help, please reach out to your respective VMware Account teams who can get our team involved in a discussion (Internal VMware folks can reach directly to us at the Tier1-Apps-Sales-Support team mailing list) and we can definitely help guide you and connect you to some of our Premier specialist partners for further discussions.

Oracle on VMware SDDC Collateral
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, and workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop [Customer]
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

FUD