Oracle and NUMA Arechitecture
Version 12.1.0.1

General Information
Library Note Morgan's Library Page Footer
The Library is currently in the process of being upgraded from Oracle Database Version 11.2.0.3 to 12.1.0.1. Demos are being upgraded to reflect the new Container paradigm as well as EBR (Edition Based Redefinition) and may contain references to CDBs, PDBs, and other objects you may not be familiar with such as CDB_OBJECTS_AE: Welcome to 12c.
It matters You should expect that any Sun/Solaris server onto which Oracle is being deployed has been configured by the System Admins for NUMA (Non-Uniform Memory Access). You can also make this default assumption with H/P blade appliances such as the Cisco UCS. But no matter the architecture you should check every server on which Oracle is installed.

Why?

Because the Oracle database does not recognized NUMA on installation or any other time unless you tell it the serrver has NUMA architecture and it is enabledin the O/S. And there is almost an equals sign one can write between ORA-04030 and your database is not configured for NUMA. This page demonstrates how you can tell and what you should do about it if you find NUMA.

Also be sure you read the following MyOraclesupport doc because you may want to push back on your System Admins and have NUMA disabled.
Enable Oracle NUMA support with Oracle Server Version 11.2.0.1 (Doc ID 864633.1)
And also read this important work on the Oracle Database and NUMA by Kevin Closson.
You Buy a NUMA System, Oracle Says Disable NUMA! What Gives
Is NUMA a good thing or a bad thing? Like with everything else in Oracle ... it depends.

In a NUMA configured server specific memory is allocated to specific cpu cores so, as you will see below, it is possible for one core to exhaust all of its local memory while there is still the appearance to almost all tools that there is plenty of memory available. Remember the name ... "non-uniform" ... that is what it is referring to.

Since posting this page to the Library I have been involved in several support issues with very large, very sophisticated, organizations that initially had no idea what NUMA was. Oh the UNIX SysAdmins were ... but the DBAs true to the way most IT shops work, were kept ignorant of critical information such as how their servers were configured, what storage they were running on, what switches were used for the fusion interconnect, and whether the interconnect was bare metal or virtualized.

Having never heard of NUMA before the DBAs had no idea what its impact on their databases might be, and were reluctant to touch something they didn't understand even when the documentation at MyOracleSupport clearly identifies the issue and the resolution. Thus the following information is critically important for DBAs to read and understand.

If you know top and sar -B you need to know this too.
 
Detecting NUMA
Not configured [oracle@alpha1 ~]$ numactl --hardware
available: 1 nodes (0)
node 0 size: 4095 MB
node 9 free: 648MB
node distances:
node   0
  0:  10

[oracle@alpha1 ~]$ numactl --show
policy: default
preferred node: current
physcpubind: 0
nodebind: 0
membind: 0
Configured [dmorgan@lxorap1n5 ~]$ numactl --hardware
available: 2 nodes (0-1)
node 0 size: 48457 MB
node 0 free: 269 MB
node 1 size: 48480 MB
node 1 free: 47 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

[dmorgan@lxorap1n5 ~]$ numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpubind: 0 1
nodebind: 0 1
membind: 0 1
 
Database Configuration
Validate database NUMA configuration conn / as sysdba

set linesize 141
col PNAME format a40
col PVAL format a20
col PDESC format a60

SELECT a.ksppinm PNAME, c.ksppstvl PVAL, a.ksppdesc PDESC, b.ksppstdf PDFLT
FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
AND LOWER(a.ksppinm) LIKE '%numa%'
ORDER BY 1;
Configure the database for NUMA conn / as sysdba

ALTER SYSTEM SET "_enable_NUMA_support" = TRUE
COMMENT= 'NUMA Support Enabled 15-Mar-2014'
CONTAINER=ALL
SCOPE=SPFILE
SID='*';

-- then bounce the database
 
Locality Group Hierarchy
lgrpinfo lgrpinfo [-aceGhIlLmrt] [-u unit] [-C|-P] lgrp ...
lgrpinfo -T [-aceGlLmr] [-u unit]
dssva102:root-(/var/stage/Solaris-Lgrp-0.1.4) bin/lgrpinfo
lgroup 0 (root):
Children: 4-7
CPUs: 0-63
Memory: installed 524278 Mb, allocated 169425 Mb, free 354853 Mb
Lgroup resources: 1-4 (CPU); 1-4 (memory)
Latency: 145951
lgroup 1 (leaf):
Children: none, Parent: 5
CPUs: 0-7 32-39
Memory: installed 131062 Mb, allocated 35170 Mb, free 95892 Mb
Lgroup resources: 1 (CPU); 1 (memory)
Load: 5.47
Latency: 128595
lgroup 2 (leaf):
Children: none, Parent: 6
CPUs: 8-15 40-47
Memory: installed 131072 Mb, allocated 18294 Mb, free 112778 Mb
Lgroup resources: 2 (CPU); 2 (memory)
Load: 4.29
Latency: 128595
lgroup 3 (leaf):
Children: none, Parent: 7
CPUs: 16-23 48-55
Memory: installed 131072 Mb, allocated 19369 Mb, free 111703 Mb
Lgroup resources: 3 (CPU); 3 (memory)
Load: 5.02
Latency: 128595
lgroup 4 (leaf):
Children: none, Parent: 0
CPUs: 24-31 56-63
Memory: installed 131072 Mb, allocated 96591 Mb, free 34481 Mb
Lgroup resources: 4 (CPU); 4 (memory)
Load: 6.04
Latency: 128595
lgroup 5 (intermediate):
Children: 1, Parent: 0
CPUs: 0-63
Memory: installed 524278 Mb, allocated 169425 Mb, free 354853 Mb
Lgroup resources: 1-4 (CPU); 1-4 (memory)
Latency: 145951
lgroup 6 (intermediate):
Children: 2, Parent: 0
CPUs: 0-63
Memory: installed 524278 Mb, allocated 169425 Mb, free 354853 Mb
Lgroup resources: 1-4 (CPU); 1-4 (memory)
Latency: 145951
lgroup 7 (intermediate):
Children: 3, Parent: 0
CPUs: 0-63
Memory: installed 524278 Mb, allocated 169425 Mb, free 354853 Mb
Lgroup resources: 1-4 (CPU); 1-4 (memory)
Latency: 145951
kstat kstat
dssva102:root-(/var/stage/Solaris-Lgrp-0.1.4) kstat -m lgrp
module: lgrp instance: 1
name: lgrp1 class: misc
alloc fail 9730
cpus 16
crtime 358.764833472
default policy 0
load average 16436
lwp migrations 4385
next-seg policy 0
next-touch policy 65153808
pages avail 33549740
pages failed to mark 24064
pages failed to migrate from 10719232
pages failed to migrate to 0
pages free 24557932
pages installed 33551789
pages marked for migration 2871296
pages migrated from 37376
pages migrated to 39936
random policy 191729
round robin policy 0
snaptime 19827.067933686
span process policy 0
span psrset policy 0

module: lgrp instance: 2
name: lgrp2 class: misc
alloc fail 0
cpus 16
crtime 359.632538936
default policy 0
load average 18625
lwp migrations 3080
next-seg policy 0
next-touch policy 58716277
pages avail 33554432
pages failed to mark 13312
pages failed to migrate from 10230272
pages failed to migrate to 0
pages free 28880502
pages installed 33554432
pages marked for migration 2841088
pages migrated from 47104
pages migrated to 53760
random policy 190958
round robin policy 0
snaptime 19827.069871957
span process policy 0
span psrset policy 0

module: lgrp instance: 3
name: lgrp3 class: misc
alloc fail 147
cpus 16
crtime 360.490421267
default policy 0
load average 19279
lwp migrations 3350
next-seg policy 0
next-touch policy 65706961
pages avail 33554432
pages failed to mark 21504
pages failed to migrate from 10706944
pages failed to migrate to 0
pages free 28598166
pages installed 33554432
pages marked for migration 2895872
pages migrated from 47104
pages migrated to 38400
random policy 191755
round robin policy 0
snaptime 19827.071554952
span process policy 0
span psrset policy 0

module: lgrp instance: 4
name: lgrp4 class: misc
alloc fail 619059
cpus 16
crtime 361.55153012
default policy 0
load average 17616
lwp migrations 10603
next-seg policy 0
next-touch policy 56934777
pages avail 33554431
pages failed to mark 12800
pages failed to migrate from 10216960
pages failed to migrate to 0
pages free 8826148
pages installed 33554431
pages marked for migration 2846720
pages migrated from 47104
pages migrated to 46592
random policy 192039
round robin policy 0
snaptime 19827.073273311
span process policy 0
span psrset policy 0

Related Topics
lgrpinfo
Troubleshooting

Morgan's Library Page Footer
This site is maintained by Dan Morgan. Last Updated: This site is protected by copyright and trademark laws under U.S. and International law. © 1998-2014 Daniel A. Morgan All Rights Reserved