SHMALL Sizing with Oracle Automatic Memory Management (AMM)

The following might answer your questions or doubts:

Re: Get confused about how to calculate SHMMAX and SHMALL on Linux.

SGA (System Global Area) and PGA (Program Global Area) are assigned and managed by Oracle depending on ASMM or AMM.

Oracle database requires shared memory to share memory between different processes. This can either /dev/shm (POSIX), using standard 4 KB memory pages, or kernel hugepages, which uses 2 MB pages and is more efficient for databases with more than 4 GB of memory use.

SHMALL and SHMMAX are safeguard parameters that apply to System V shared memory (IPC) to prevent a process from occupying all available shared memory. SHMALL sets the total amount of shared memory pages that can be used system wide. SHMMAX sets the maximum shared memory a process can allocate. SHMALL should therefore be SHMMAX/PAGE_SIZE. Page size is usually 4k unless you use kernel Huge Pages (2 MB).

Oracle AMM (memory_target) requires POSIX shared memory (/dev/shm). By default it is 50 % of physical RAM and only allocated as needed. /dev/shm uses standard 4k memory pages and can be swapped to disk. If you have an Oracle database with more than 4 GB of SGA, then performance can drop due to the amount of memory required to manage the memory pages and limited TLB cache.

If you use Oracle Automatic Memory Management (AMM), System V kernel parameters do not apply. Oracle AMM manages SGA and PGA automatically. However, PGA is never allocated out of SGA. Since Oracle 9i, PGA/UGA memory is mapped to /dev zero. You can verify it with the pmap utility.

With this in mind, answering your questions should not be difficult:

1a) System V IPC parameters do not apply to Posix shared memory.

1b) PGA is never allocated out of SGA.

2a) Does not apply to AMM.

2b) AMM uses /dev/shm, which is max. 50 % by default. The SGA under AMM needs to fit into /dev/shm.

3) No, ASM requires AMM and does not use System V IPC parameters.

4) Shared memory can be System V or Posix. SHMALL applies to System V only.

5) Only a few processes use shared memory, Oracle is one of them.

ASM requires AMM and will not run under ASMM.

It is possible to run one database instance using AMM and another using ASMM on the same machine.

The best way to increase /dev/shm is to add more RAM. By default, /dev/shm is max. 50 % of RAM, allocated on demand and can be swapped. The minimum size of /dev/shm needs to be as large as your database SGA. If you have several database, they will share the memory and overall usage will be less than the sum of all SGA's.

I'm not aware of a list that outlines which Linux processes require Posix or other shared memory, but you can use ipcs and ls /dev/shm commands to analyze it's use. Sorry, I have no idea what you mean by Linux shadow processes or PL/SQL collections that exceed some target configuration.

I recommend to use kernel hugepages under Linux for any large oracle SGA in order achive best performance. Exact sizing and limiting of memory use is often not necessary and does not account for the unforeseeable. RAM and disk space are not that expensive and precious anymore. The SHMMAX and SHMALL kernel parameters are only safeguard paramters and do not cause any process to preallocate memory. When using Oracle they are typcially set to the maximum theoretical hardware limit, at least for SHMMAX.

I previously tried to configure an ASM instance using ASMM after reading that AMM and kernel hugepages are incompatible, however, as it turns out, this only applies to the database instance and not the server. You can certainly configure the OS to provide AMM and kernel hugepages, and install one instance using AMM and another on using ASMM on the same server.

https://forums.oracle.com/thread/2409579

I just tried to configure an ASM instance using ASMM again, and it seems to work. Not sure what previously caused the ORA-600 AND ORA-01078 failures. I have attached the protocol below.

But why do you want to configure a ASM instance using ASMM? What's the idea or benefit behind it.

You should configuring a database with 16 GB of SGA using kernel hugepages. It's is not that AMM has a problem with it, but your system will have to manage a lot of memory pages as previously outlined. It will drop performance and waste memory just for managing memory pages.

You can do a simple math with 16 GB, resulting in 4000000 memory pages under AMM and 8192 pages under kernel hugepages.

The following slideshow may help to understand how it works: http://www.slideshare.net/raghusiddarth/transparent-hugepages-in-rhel-6

The following is an example how to configure ASM using ASMM.

[root@ol1 ~]# su - oracle

[oracle@ol1 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM

The Oracle base has been set to /u01/app/oracle

[oracle@ol1 ~]$ cd $ORACLE_HOME/dbs

[oracle@ol1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Thu Aug 29 20:55:13 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

With the Automatic Storage Management option

SQL> sho parameter memory

NAME                     TYPE     VALUE

------------------------------------ ----------- -----

memory_max_target             big integer 272M

memory_target                 big integer 272M

SQL> select name, value from v$pgastat;

NAME                                      VALUE

---------------------------------------------------------------- ----------

aggregate PGA target parameter                       96468992

aggregate PGA auto target                          0

global memory bound                              0

total PGA inuse                            16798720

total PGA allocated                           20529152

maximum PGA allocated                           20529152

SQL> create pfile from spfile;

[oracle@ol1 dbs]$ vi init+ASM.ora

memory_target=0

sga_target=256M

pga_aggregate_target=96M

memory_target=0

[oracle@ol1 dbs]$ sqlplus / as sysasm

SQL> shutdown immediate;

ASM diskgroups dismounted

ASM instance shutdown

SQL> connect / as sysasm

Connected to an idle instance.

SQL> startup pfile=/u01/app/oracle/product/112/grid/dbs/init+ASM.ora

Total System Global Area  283930624 bytes

Fixed Size            2225792 bytes

Variable Size          256539008 bytes

ASM Cache           25165824 bytes

ASM diskgroups mounted

AMM and kernel hugepages are mutually exclusive. It means you cannot use AMM with kernel hugepages. AMM uses posix shared memory, which is allocate on demand, swappable and relies on standard 4 KB kernel memory pages. Hugepages on the other hand is pre-allocated and reserved at system startup, cannot be swapped and uses 2 MB memory pages.

ASM uses it's own database instance (+ASM). You can configure ASM using AMM (default), and install another database instance (ORCL) on the same machine using kernel hugepages and ASMM. The database connecting to the ASM instance to access the ASM diskgroups does not have to use the same memory management.

I cannot explain it because it does not make sense to me. You have to ask Oracle support. It might be a typing error and RDBMS should read ASM instance. Generally, and this seems to be according to the symptoms explained, in order for any Oracle Instance under Linux to use ASMM with kernel hugepages, the parameter memory_target of the database instance must be unset, i.e. not just set to 0, but not defined at all. Unlike for ASM, where memory_target must be set to 0 in order for the ASM instance to use ASMM.

If you use HugePages for RDBMS then I don't believe you can use AMM with ASM.  I recieve the following information from a very reliable source from Oracle (SEE BELOW).

Sorry, I'm afraid your interpretation of the information is incorrect. Did you read the complete thread?

Why should a database instance not be able to use Kernel Hugepages if the ASM instance uses AMM? The RDBMS and ASM instance are separate instances and each can use their own memory management.

And why shouldn't it? You can configure Enterprise Linux to provide Posix (/dev/shm) and Kernel HugePages.

Kernel Hugepages is incompatible with AMM, meaning you cannot use AMM with Kernel Hugepages. You can however run two database instances on one and the same server and use different memory management.

Below is an example, showing ASM (+ASM) configured for AMM (/dev/shm), which is used by a database (orclasm) configured to use Hugepages.

$ grep -i huge /proc/meminfo

HugePages_Total:    1029

HugePages_Free:     1029

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

$ df -Ph

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00   31G   20G  9.8G  67% /

/dev/sda1              99M   43M   52M  46% /boot

tmpfs                 2.0G     0  2.0G   0% /dev/shm

$ ORAENV_ASK=NO

$ ORACLE_SID=+ASM

$ . oraenv

The Oracle base remains unchanged with value /u01/app/oracle

$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Wed Sep 11 22:03:59 2013

SQL> startup

ASM instance started

Total System Global Area  283930624 bytes

Fixed Size            2225792 bytes

Variable Size          256539008 bytes

ASM Cache           25165824 bytes

ASM diskgroups mounted

$ df -Ph

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00   31G   20G  9.8G  67% /

/dev/sda1              99M   43M   52M  46% /boot

tmpfs                 2.0G  176M  1.8G   9% /dev/shm

$ ls -l /dev/shm

total 179352

-rw-r----- 1 oracle oinstall 4194304 Sep 11 22:04 ora_+ASM_2392069_0

$ ORACLE_SID=orclasm

$ . oraenv

The Oracle base remains unchanged with value /u01/app/oracle

$ sqlplus / as sysdba

SQL> startup

ORACLE instance started.

Total System Global Area  546992128 bytes

Fixed Size            2228320 bytes

Variable Size          335548320 bytes

Database Buffers      201326592 bytes

Redo Buffers            7888896 bytes

Database mounted.

Database opened.

SQL> sho parameter use_large_pages

use_large_pages              string     only

SQL>SQL> SELECT NAME FROM V$DATAFILE;

+DATA/orcl/datafile/system.256.741751677

+DATA/orcl/datafile/sysaux.257.741751677

+DATA/orcl/datafile/undotbs1.258.741751679

+DATA/orcl/datafile/users.259.741751679

+DATA/orcl/datafile/example.269.741751827

$ grep -i huge /proc/meminfo

HugePages_Total:    1029

HugePages_Free:      850

HugePages_Rsvd:       84

HugePages_Surp:        0

Hugepagesize:       2048 kB

Interesting conversation. However, I don't know hardly anyone that recommends using AMM for much of anything. Maybe its okay for databases that serve huge swings in workloads on database servers that maybe performing more than just servicing a database. Even then, you must meet peak needs. Thus, to me, AMM is a convenient option for only small workloads.

I don't know exactly what you're trying to accomplish with all your questions. In my honest opinion, you're better off having 2 node RAC and increasing each individual nodes resources.... than having resources split over 3 nodes. The only exception to this, would be if you serving huge amounts of data over extended periods of time. Even then, in a RAC environment, you HAVE to take into consideration tuning your requests to take advantage of the scale of the cluster. Running sequenced batches across each node will not get you anywhere. In small OLTP operations, you're never gone to get more than what one single RAC node will give you. Thus, my recommendation. This may change to some degree was interconnect bandwidth increases. The bandwidth available with technologies such as infiniband will make a difference.