Home > Blogs > Virtualize Business Critical Applications


On Demand Scaling up resources for Oracle production workloads

The crux of this blog’s discussion is “How to stop hoarding much needed infrastructure resources and live wisely ever after by scaling up as needed effectively

Typically Oracle workloads running on bare metal environments , or for that matter any environment, are sized very conservatively, given the nature of the workload , with the premise that , in event of any workload spike, the abundant resources thrown at the workload will be able to sustain this spike, but in reality , we need to ask ourselves these questions

  • How much resource is actually allocated to the workload?
  • How much of that allocated resource is actually consumed by that workload ?
  • How often does the workload experience spikes?
  • If spikes are happening regularly then, has proper capacity planning and forecasting been done for this workload?

Proper plan and design along with capacity planning and forecasting is the key to manage any Business Critical Application (BCA) workload and there is no shortcut around this.

Unfortunately what this means in a physical environment is , for example, static allocation of resources to a BCA workload where the CPU utilization has been flat at 30-40% for 11 months of the year with utilization at 55-60% for the last month of the year.

Pre-allocating resources to a workload , in anticipation of peaks for say 1 month in a whole year, basically results in the resources underutilized for the rest of the year , starving other workloads of much needed resource, an ineffective way of resource allocation , thereby leading to increase in larger footprint of servers resulting in increase in CAPEX and OPEX.

Enter “Hot Plug” – “Hot Plug CPU and Hot Plug Memory” on vSphere Platform – Resource allocation on demand thereby resulting in effective and elastic resource management working on the principle of “Ask and thy shall receive”.

The “Hot Plug” CPU and Memory feature was added in vSphere 4.0 release but for best results , use virtual machines that are compatible with ESXi 5.0 and later.

https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc_50%2FGUID-F102B9BD-1B92-4AC5-ADC0-BE4E90473C5F.html

The following conditions had to be met for the “Hot Plug” CPU and Memory to be enabled:

  • VMware Tools is installed. This condition is required for hot plug functionality with Linux guest operating systems
  • The VM has a guest operating system that supports CPU hot plug
  • The VM is using hardware version 7 or later
  • The VM is powered off i.e. by default, virtual machines don’t support Hot Add (add RAM) and Hot Plug (add vCPU)

Caveats with “Hot Plug” CPU

  • If virtual NUMA (vNUMA) is configured with vCPU hot plug settings, the virtual machine will be started without vNUMA and instead it will use Uniform Memory Access (UMA) with interleaved memory access

vNUMA is disabled if vCPU hotplug is enabled (2040375)
https://kb.vmware.com/s/article/2040375

Caveats with “Hot Plug” Memory

  • Prior vSphere 6.0 release, “Hot Plug” memory resulted in additional memory being placed in first NUMA node follow by the placing additional memory , if requested , on 2nd NUMA node and so on in sequence.
  • With vSphere 6.0 release and later , memory allocation is now balance evenly across all the NUMA nodes instead of placing additional memory on 1st NUMA node

 

########
Test Setup
########

VM ‘SB-OL73-ORA122’ has 8 vCPUs with 16 GB RAM running OEL 7.4 with Oracle Database 12.2.0.1.0.

[root@sb-ol73-ora122 ~]# cat /etc/oracle-release
Oracle Linux Server release 7.4
[root@sb-ol73-ora122 ~]#

[root@sb-ol73-ora122 ~]# uname -a
Linux sb-ol73-ora122.corp.localdomain 4.1.12-94.3.9.el7uek.x86_64 #2 SMP Fri Jul 14 20:09:40 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@sb-ol73-ora122 ~]#

Oracle instance ‘ORA12C’ sga_max_size is set to 8GB.

oracle@sb-ol73-ora122:ORA12C:/home/oracle> sqlplus  /nolog
SQL*Plus: Release 12.2.0.1.0 Production on Mon Nov 13 09:28:43 2017
Copyright (c) 1982, 2016, Oracle.  All rights reserved.

SQL> conn / as sysdba
Connected.

SQL> show sga
Total System Global Area 8606711808 bytes
Fixed Size                 12352272 bytes
Variable Size            1862272240 bytes
Database Buffers         6710886400 bytes
Redo Buffers               21200896 bytes
SQL>

SQL> select name, value from v$parameter where name  = ‘sga_max_size’;
NAME                                                                             VALUE
—-                                                                             —-
sga_max_size                                                                     8606711808
SQL>

SQL> select name, value from v$parameter where name  = ‘cpu_count’;
NAME                                                                             VALUE
————————————————-
cpu_count                                                                        8
SQL>

 

#####################
Test Case  – “Hot Add” CPU
#####################

With VM ‘SB-OL73-ORA122’ currently at 8 vCPUs with 16 GB RAM , increase the number of vCPUs from 8 to 16.

  1. Edit VM settings and click on the CPU arrow. Ensure that the ‘CPU Hot Plug’ checkbox is enabled , if not power down VM and enable it and power it back up.

  1. Click on CPU drop down box and select the new vCPU count to 16.

  1. The new vCPU count now is 16 , Click OK to save the new configuration

 

  1. Run the same above SQL statement to get the new value of CPU that the Oracle instance now sees.

SQL> select name, value from v$parameter where name  = ‘cpu_count’;
NAME                                                                             VALUE
————————————————-
cpu_count                                                                        16
SQL>

Oracle is now able to take advantage of the increased number of vCPU’s

 

#########################
Test Case  – “Hot Add” Memory
#########################

With VM ‘SB-OL73-ORA122’ currently at 8 vCPUs with 16 GB RAM, increase the amount of RAM from 16GB to 32GB.

  1. Edit VM Settings. Ensure that the ‘Memory Hot Plug’ setting is enabled else power down the VM , enable it and power the VM back

  1. Increase the VM memory from 16GB to 32GB and Click OK when done

  1. Run the same above SQL statement to get the new value of Memory that the Oracle instance now sees.

SQL> select name, value from v$parameter where name  = ‘sga_max_size’;
NAME                                                                             VALUE
————————————————-
sga_max_size                                                                     8606711808
SQL>

Oracle still sees the old SGA Size of 8GB !!!

Check the operating system to see if the memory has increased

oracle@sb-ol73-ora122:ORA12C:/home/oracle> cat /proc/meminfo
MemTotal:       32928488 kB
MemFree:        17062856 kB
MemAvailable:   20774920 kB
Buffers:          193736 kB
Cached:         11636640 kB
SwapCached:          208 kB
Active:          8872832 kB
Inactive:        5401680 kB
Active(anon):    6992864 kB
Inactive(anon):  4202760 kB
Active(file):    1879968 kB
Inactive(file):  1198920 kB
……
oracle@sb-ol73-ora122:ORA12C:/home/oracle>

The O/S shows that the VM memory is now 32GB , for Oracle instance to see the memory change, we would need to shutdown the instance , change the spfile / init file ‘sga_max_size’ parameter to reflect the change and restart the Oracle instance.

From an O/S perspective ‘Hot Plug’ CPU and Memory works just fine.

 

########
Summary
########
Key points to keep in mind

  • No more need of per-allocating resources to the workload in anticipation of peaks which remains underutilized for a long period of time starving other tenants of much needed resource
  • Scale up by ‘Hot Plug’ CPU & Memory whenever you need
  • “Hot Remove” CPU and Memory is currently not supported even though Red Hat Enterprise Linux version 7.1 (RHEL7.1) and later supports “Hot-Remove” of physical CPU and Memory as long as the underlying hardware supports this function.

https://access.redhat.com/articles/1262753

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

 

Some important KB articles related to “Hot Plug”

CPU Hot Plug Enable and Disable options are grayed out for virtual machines (2020993)
https://kb.vmware.com/s/article/2020993

Support for Hot Add Memory and Hot Add vCPU specifications on different editions of Windows Server 2008 R2 (2051989)
https://kb.vmware.com/s/article/2051989

vCPU hot-add not disabled when multi-core vCPU feature is enabled (2002926)
https://kb.vmware.com/s/article/2002926

vNUMA is disabled if VCPU hotplug is enabled (2040375)
https://kb.vmware.com/s/article/2040375

Virtual Machine vCPU and vNUMA Rightsizing – Rules of Thumb
https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb.html

This entry was posted in ESXi, Oracle, vSphere on by .
Sudhir Balasubramanian

About Sudhir Balasubramanian

Sudhir Balasubramanian is a Staff Solution Architect specializing in all Oracle Technologies on VMware SDDC stack. Prior to joining VMware in 2012 , Sudhir has worked for close to 20 years as a Principal Oracle Database Administrator (DBA) / Architect in Oracle Technologies for fortune 100 companies including Tata Consultancy Services (TCS) , Sony Electronics, Epsilon Marketing (Aspen/Newgen), Teletech Corp, SAIC, Active Network and Sempra Energy Holdings before joining VMware PSO before moving into the Global Field and Partner Readiness (GFPR). Sudhir is also experienced in EMC SAN Technologies & Unix/Linux Operating Systems along with being a VMware vBCA Specialist , VMware vExpert and VMware VCA – Cloud certified. Sudhir is also a Member of the CTO Ambassador Program , run by the VMware Office of the CTO. Sudhir is a recognized Speaker having presented at Oracle Open World, IOUG, VMworld, VMware Partner Exchange, EMC World, EMC Oracle Summit and various Webinars and is an Industry recognized expert in Oracle Virtualization technologies. Sudhir has also co-authored a book "Virtualizing Oracle Business Critical Databases on VMware SDDC” which is a comprehensive authority for Oracle DBA ’s on the subject of Oracle & Linux on vSphere. https://www.amazon.com/Virtualize-Oracle-Business-Critical-Databases/dp/1500135127/ref=sr_1_1?s=books&ie=UTF8&qid=1493001047&sr=1-1&keywords=Virtualize+Oracle+Business+Critical+Databases Sudhir regularly blogs at the official VMware Application blog site https://blogs.vmware.com/apps Sudhir also blogs on his personal website http://vracdba.com Sudhir holds a degree in Master of Computer Science from San Diego State University (SDSU) graduating in 2011. Sudhir also holds a Bachelor Degree in Computer Science Engineering from Bangalore University. Twitter @vracdba

Leave a Reply

Your email address will not be published. Required fields are marked *

*