The crux of this blog’s discussion is “How to stop hoarding much needed infrastructure resources and live wisely ever after by scaling up as needed effectively”
Typically Oracle workloads running on bare metal environments , or for that matter any environment, are sized very conservatively, given the nature of the workload , with the premise that , in event of any workload spike, the abundant resources thrown at the workload will be able to sustain this spike, but in reality , we need to ask ourselves these questions
- How much resource is actually allocated to the workload?
- How much of that allocated resource is actually consumed by that workload ?
- How often does the workload experience spikes?
- If spikes are happening regularly then, has proper capacity planning and forecasting been done for this workload?
Proper plan and design along with capacity planning and forecasting is the key to manage any Business Critical Application (BCA) workload and there is no shortcut around this.
Unfortunately what this means in a physical environment is , for example, static allocation of resources to a BCA workload where the CPU utilization has been flat at 30-40% for 11 months of the year with utilization at 55-60% for the last month of the year.
Pre-allocating resources to a workload , in anticipation of peaks for say 1 month in a whole year, basically results in the resources underutilized for the rest of the year , starving other workloads of much needed resource, an ineffective way of resource allocation , thereby leading to increase in larger footprint of servers resulting in increase in CAPEX and OPEX.
Enter “Hot Plug” – “Hot Plug CPU and Hot Plug Memory” on vSphere Platform – Resource allocation on demand thereby resulting in effective and elastic resource management working on the principle of “Ask and thy shall receive”.
The “Hot Plug” CPU and Memory feature was added in vSphere 4.0 release but for best results , use virtual machines that are compatible with ESXi 5.0 and later.
The following conditions had to be met for the “Hot Plug” CPU and Memory to be enabled:
- VMware Tools is installed. This condition is required for hot plug functionality with Linux guest operating systems
- The VM has a guest operating system (RHEL-5.5 and onwards in a 64 bit operating system) that supports CPU hot plug – RedHat KB Hot adding CPU/RAM/NIC is supported for VMware guest? and Is it possible to “hotplug” a CPU on a running Red Hat Enterprise Linux?
- The VM is using hardware version 7 or later
- The VM is powered off i.e. by default, Powered on virtual machines don’t support Hot Add (add RAM) and Hot Plug (add vCPU)
Caveats with “Hot Plug” CPU
- If virtual NUMA (vNUMA) is configured with vCPU hot plug settings, the virtual machine will be started without vNUMA and instead it will use Uniform Memory Access (UMA) with interleaved memory access . Refer to VMware KB vNUMA is disabled if vCPU hotplug is enabled (2040375)
- This feature can be used for Oracle workloads where NUMA optimization is not needed
- Leaving CPU Hot Add at its default setting of disabled is one of the performance best practices that we have for large VMs. Performance with the default setting of CPU Hot Add disabled performed from 2% to 8% better than when CPU Hot Add was enabled. Learn more about that at CPU Hot Add Performance in vSphere 6.7.
Caveats with “Hot Plug” Memory
- Prior vSphere 6.0 release, “Hot Plug” memory resulted in additional memory being placed in first NUMA node follow by the placing additional memory , if requested , on 2nd NUMA node and so on in sequence.
- With vSphere 6.0 release and later , memory allocation is now balance evenly across all the NUMA nodes instead of placing additional memory on 1st NUMA node
Test Setup
VM ‘SB-OL73-ORA122’ has 8 vCPUs with 16 GB RAM running OEL 7.4 with Oracle Database 12.2.0.1.0.
[root@sb-ol73-ora122 ~]# cat /etc/oracle-release
Oracle Linux Server release 7.4
[root@sb-ol73-ora122 ~]#
[root@sb-ol73-ora122 ~]# uname -a
Linux sb-ol73-ora122.corp.localdomain 4.1.12-94.3.9.el7uek.x86_64 #2 SMP Fri Jul 14 20:09:40 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@sb-ol73-ora122 ~]#
Oracle instance ‘ORA12C’ sga_max_size is set to 8GB.
oracle@sb-ol73-ora122:ORA12C:/home/oracle> sqlplus /nolog
SQL*Plus: Release 12.2.0.1.0 Production on Mon Nov 13 09:28:43 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected.
SQL> show sga
Total System Global Area 8606711808 bytes
Fixed Size 12352272 bytes
Variable Size 1862272240 bytes
Database Buffers 6710886400 bytes
Redo Buffers 21200896 bytes
SQL>
SQL> select name, value from v$parameter where name = ‘sga_max_size’;
NAME VALUE
—- —-
sga_max_size 8606711808
SQL>
SQL> select name, value from v$parameter where name = ‘cpu_count’;
NAME VALUE
————————————————-
cpu_count 8
SQL>
Test Case – “Hot Add” CPU
With VM ‘SB-OL73-ORA122’ currently at 8 vCPUs with 16 GB RAM , increase the number of vCPUs from 8 to 16.
- Edit VM settings and click on the CPU arrow. Ensure that the ‘CPU Hot Plug’ checkbox is enabled , if not power down VM and enable it and power it back up.
- Click on CPU drop down box and select the new vCPU count to 16.
- The new vCPU count now is 16 , Click OK to save the new configuration
- Run the same above SQL statement to get the new value of CPU that the Oracle instance now sees.
SQL> select name, value from v$parameter where name = ‘cpu_count’;
NAME VALUE
————————————————-
cpu_count 16
SQL>
Oracle is now able to take advantage of the increased number of vCPU’s
Test Case – “Hot Add” Memory
With VM ‘SB-OL73-ORA122’ currently at 8 vCPUs with 16 GB RAM, increase the amount of RAM from 16GB to 32GB.
- Edit VM Settings. Ensure that the ‘Memory Hot Plug’ setting is enabled else power down the VM , enable it and power the VM back
- Increase the VM memory from 16GB to 32GB and Click OK when done
- Run the same above SQL statement to get the new value of Memory that the Oracle instance now sees.
SQL> select name, value from v$parameter where name = ‘sga_max_size’;
NAME VALUE
————————————————-
sga_max_size 8606711808
SQL>
Oracle still sees the old SGA Size of 8GB !!!
Check the operating system to see if the memory has increased
oracle@sb-ol73-ora122:ORA12C:/home/oracle> cat /proc/meminfo
MemTotal: 32928488 kB
MemFree: 17062856 kB
MemAvailable: 20774920 kB
Buffers: 193736 kB
Cached: 11636640 kB
SwapCached: 208 kB
Active: 8872832 kB
Inactive: 5401680 kB
Active(anon): 6992864 kB
Inactive(anon): 4202760 kB
Active(file): 1879968 kB
Inactive(file): 1198920 kB
……
oracle@sb-ol73-ora122:ORA12C:/home/oracle>
The O/S shows that the VM memory is now 32GB , for Oracle instance to see the memory change, we would need to shutdown the instance , change the spfile / init file ‘sga_max_size’ parameter to reflect the change and restart the Oracle instance.
For Oracle databases using Oracle system V shared memory (using sga_max_size / sga_target parameters as needed for ASMM) and not /dev/shm (memory_target / memory_max_target as needed for AMM) , on startup, it issues a “shmget()” call for static memory allocation which explains why we need to shutdown an Oracle instance to make changes to the parameter file to take advantage of the increased memory.
For example : init.ora of ‘ORA12C’ :
*.pga_aggregate_target=2G
*.sga_max_size=8G
*.sga_target=8G
Run linux utility ‘strace’ for instance startup:
strace -f -o /tmp/oracle_startup.txt sqlplus / as sysdba
….
53437 shmget(IPC_PRIVATE, 12353536, IPC_CREAT|IPC_EXCL|0600) = 1966083
53437 shmget(IPC_PRIVATE, 8573157376, IPC_CREAT|IPC_EXCL|0600) = 1998852
53437 shmget(IPC_PRIVATE, 21200896, IPC_CREAT|IPC_EXCL|0600) = 2031621
Run ‘ipcs -a’ command to see Oracle shared memory segments:
oracle@sb-ol73-ora122:ORA12C:/home/oracle> ipcs -a
—— Message Queues ——–
key msqid owner perms used-bytes messages
—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status
0x00000000 65536 grid 600 4096 0
0x00000000 32769 grid 600 4096 0
0x4ace20a8 98306 grid 600 24576 24
0x00000000 1966083 oracle 600 12353536 64
0x00000000 1998852 oracle 600 8573157376 32
0x00000000 2031621 oracle 600 21200896 32
0xab08373c 2064390 oracle 600 40960 32
From an O/S perspective ‘Hot Plug’ CPU and Memory works just fine.
Summary
Key points to keep in mind
- No more need of per-allocating resources to the workload in anticipation of peaks which remains underutilized for a long period of time starving other tenants of much needed resource
- Scale up by ‘Hot Plug’ / ‘Hot-Add’ CPU & Memory whenever you need
- Caveats with “Hot Plug” CPU –
- If virtual NUMA (vNUMA) is configured with vCPU hot plug settings, the virtual machine will be started without vNUMA and instead it will use Uniform Memory Access (UMA) with interleaved memory access . Refer to VMware KB vNUMA is disabled if vCPU hotplug is enabled (2040375)
- This feature can be used for Oracle workloads where NUMA optimization is not needed
- Leaving CPU Hot Add at its default setting of disabled is one of the performance best practices that we have for large VMs. Performance with the default setting of CPU Hot Add disabled performed from 2% to 8% better than when CPU Hot Add was enabled. Learn more about that at CPU Hot Add Performance in vSphere 6.7.
- Caveats with “Hot Plug” Memory
- Prior vSphere 6.0 release, “Hot Plug” memory resulted in additional memory being placed in first NUMA node follow by the placing additional memory , if requested , on 2nd NUMA node and so on in sequence.
- With vSphere 6.0 release and later , memory allocation is now balance evenly across all the NUMA nodes instead of placing additional memory on 1st NUMA node
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below
Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html
Some important KB articles related to “Hot Plug”
CPU Hot Add Performance in vSphere 6.7
https://blogs.vmware.com/performance/2019/12/cpu-hot-add-performance-vsphere67.html
CPU Hot Plug Enable and Disable options are grayed out for virtual machines (2020993)
https://kb.vmware.com/s/article/2020993
Support for Hot Add Memory and Hot Add vCPU specifications on different editions of Windows Server 2008 R2 (2051989)
https://kb.vmware.com/s/article/2051989
vCPU hot-add not disabled when multi-core vCPU feature is enabled (2002926)
https://kb.vmware.com/s/article/2002926
vNUMA is disabled if VCPU hotplug is enabled (2040375)
https://kb.vmware.com/s/article/2040375
Virtual Machine vCPU and vNUMA Rightsizing – Rules of Thumb
https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb.html
shmget(2)
https://docs.oracle.com/cd/E19253-01/816-5167/shmget-2/index.html
