1 2 3 4 Previous Next 53 Replies Latest reply: May 2, 2012 12:39 AM by fuzzydba Go to original post RSS
      • 30. Re: SGA_MAX_SIZE != SGA_TARGET when?
        Hans Forbrich
        Thanks for that first link to the "Administering
        Oracle Database on Solaris" article, Hans
        Something in return for all you've taught me?

        >
        I have been saying 'it depends on your OS and Solaris
        <> Linux in this regard' for a long time, but I had
        You and me both.
        completely forgotten where I got that from. Your link
        reminds me that I am not just making it up!
        <g>

        My thanks also to Bob (bmurching) for bullying me into backing up what I said. I'm still not satisfied with my conclusion, and won't be until I get a chance to verify on a live Solaris box.
        • 31. Re: SGA_MAX_SIZE != SGA_TARGET when?
          412527
          hans,

          No bullying at all... this is how we all learn and I learned something too.

          Bob
          • 32. Re: SGA_MAX_SIZE != SGA_TARGET when?
            Hans Forbrich
            hans,

            No bullying at all... this is how we all learn and I learned something too.
            It was 'bullying'. Nice simple 'prove it' type of bullying. I was all prepared to sit on my fat *&^% and simply have you accept my word.

            Glad you convinced me to follow up. And glad yingkuan followed up with his demos as well. This will serve me well with some of the courses & customers I have coming up.
            • 33. Re: SGA_MAX_SIZE != SGA_TARGET when?
              412527
              As a fellow teacher of Oracle "things" (because it's not just a database anymore!) I have found it helpful to forget everything I know every other year, so that I can go back and learn it all over again. things change, and so must our knowledge and tools. I am reminded of this everytime I meet a 20 year DBA who was working Oracle when I was in elementary school and still thinks the solution to every tuning problem is an undocumented kernel parameter!

              I posted this because this week I was the alleged "expert" troubleshooting a 10.2 cluster and traced some of the problems to a SGA that was too big. I told them to simply drop sga_target whenever they were ready and we'd move on. I was stopped in my tracks by a new DBA who flatly told me that he didn't think OS memory would be released and that we'd need to bounce the cluster nodes to change sga_max_size. It was then I realized that I actually don't know the truth of the matter. Some googling, metalinking and introspecting led me eventually to come here and post.

              From this thread I learned that I need to learn more about DISM and OSs with advanced memory management. And maybe start thinking a tiny bit less about Linux and a little bit more about the OS I started this journey with, Solaris.
              • 34. Re: SGA_MAX_SIZE != SGA_TARGET when?
                Oviwan
                Hey

                I had the same question. this thread clarify me a lot.
                My test:
                # uname -mrs
                HP-UX B.11.31 ia64

                SQL> show sga

                Total System Global Area 1073741824 bytes
                Fixed Size                  2050928 bytes
                Variable Size             721421456 bytes
                Database Buffers          348127232 bytes
                Redo Buffers                2142208 bytes
                SQL> show parameter sga_

                NAME                                 TYPE        VALUE
                ------------------------------------ ----------- ------------------------------
                sga_max_size                         big integer 1G
                sga_target                           big integer 512M

                SQL> !ipcs -ma
                IPC status from /dev/kmem as of Tue Jul 31 17:52:52 2007
                T         ID     KEY        MODE        OWNER     GROUP   CREATOR    CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
                Shared Memory:
                m    1867781 0xa8a97efc rw-r---    oracle  oinstall    oracle  oinstall     22 1077948416  3610  3764 17:50:12 17:50:12 17:49:48
                also here the OS allocates the sga_max_size.

                Regards
                • 35. Re: SGA_MAX_SIZE != SGA_TARGET when?
                  519926
                  After a VERY long summer I've come back to an issue with sga_max_size, memory usage on Solaris 10 and zones. Your postings have helped clarify some points - cheers.

                  However ...

                  We have instances running on a 'non-global' Solaris 10 zone. According to Note 317257.1 “…Oracle should not be configured to use DISM in local Containers.” If I understand correctly, setting sga_max_size makes Oracle start an ora_dism_xxx process, therefore we are in fact using DISM. An SR with Oracle says that this shouldn't be a problem now, but that we need to set the 'proc_lock_memory for the container'. This is with our UNIX team at the moment.

                  Now, your conclusions are that the OS allocates sga_max_size at instance startup. My tests using the same methods that you employed mirror your results. The Reference guide also says that this is the case. With DISM : "The operating system does not have to lock down physical memory for the entire shared memory segment." from http://download.oracle.com/docs/cd/B19306_01/server.102/b15658/appe_sol.htm

                  So ...

                  From an OS perspective, for the life of the instance, the memory used is (approx) : sga_max_size + pga

                  Or should I hang my head in shame because I still haven't understood :-(

                  Note: the you and your in the above message is plural .
                  • 36. Re: SGA_MAX_SIZE != SGA_TARGET when?
                    Hans Forbrich
                    From an OS perspective, for the life of the instance,
                    the memory used is (approx) : sga_max_size + pga

                    Or should I hang my head in shame because I still
                    haven't understood :-(
                    You have understood.

                    The shared memory is defined all the way up to max_size.

                    The OS is free to physically allocate that entire size. If the OS has a trick to physically allocate only a portion then Oracle will send a 'message' saying the target is actually required.

                    Solaris has such a trick. The trick used to have related performance issues - which seem to be resolved per the SR.
                    • 37. Re: SGA_MAX_SIZE != SGA_TARGET when?
                      Tanel Poder
                      Hi,

                      Don't confuse address space set-up with allocating physical memory pages from RAM!

                      Even if ipcs -m shows x GB as the SGA shm segment length, it doesn't mean this memory has actually been initialized and taken from RAM.

                      Decent OS'es do only initialize the pageable meory pages when they're touched the first time, so a shm segment showing 10GB in ipcs -m output may be only 10% "used" really as some pages have never been touched.

                      There are many things which affect when and if the memory is actually allocated, the ones I remember right now are:

                      1) using solaris ISM - means Oracle will be usng non-pageable large pages - the shm seg size you see in ipcs is fully allocated from RAM and locked in RAM.

                      2) using Solaris DISM, the SGA shm segment is pageable (small pages in Solaris 8, large pages from Solaris 9) and may not necessarily be allocated from RAM

                      3) using lock_sga=true -> the SGA shm segment is allocated from RAM and locked in RAM

                      4) using locksga_areas -> some ranges of pages in SGA shm segment are locked to memory, some pages of SGA shm segment may still be uninitialized

                      5) using prepage_sga=true -> all pages of SGA shm segment are touched on startup

                      6) few others like dbcache_pre_warm which affect memory page touching on startup...

                      7) using memory_target on Oracle 11g

                      So, there are many things which affect physical memory allocation, but generally, unless you're using non-pageable pages, not all SGA-size worth of memory is allocated from OS during instance startup.

                      Normally these artificial instance startup errors after setting sga_max_size to xxxGB come from hitting max shm segment size or max RAM + swap size (on Unixes). On linux on the other hand you can overallocate memory as Linux doesn't back anonymous memory mappings with swap space (linux starts killing "random" processes instead when running out-of-memory. nice, huh?)
                      • 38. Re: SGA_MAX_SIZE != SGA_TARGET when?
                        Tanel Poder
                        So, this means that if your SGA_TARGET is lower than SGA_MAX_SIZE during startup then the pages "above" SGA_TARGET will never be touched, thus not allocated!

                        And if you ramp down SGA_TARGET during your instance lifetime, then the pages "above" the new SGA_TARGET won't be touched anymore (after MMAN completes the downsizing), which means these pages will be paged out from physical memory if there's shortage of free physical memory.

                        I wrote an article about Oracle 11g MEMORY_TARGET and /dev/shm usage on Linux a week ago: http://blog.tanelpoder.com
                        • 39. Re: SGA_MAX_SIZE != SGA_TARGET when?
                          Hans Forbrich
                          Thanks for the added information.

                          You are saying 'decent OSs do not allocate'.

                          I made a similar assertion early on in the discussion and had to back track when faced with output from tools that seem to contradict the statement.

                          It seems that some of the discussion is based on the idea that not all OSs are 'decent' - which has been causing the confusion.
                          • 40. Re: SGA_MAX_SIZE != SGA_TARGET when?
                            Tanel Poder
                            Whether a tool lies or not, depends very much on the tool.

                            It's actually not correct to say that the tools lie - it's just we who interpret their output too simplistically.

                            So if we want to find out how much physical memory has been allocated for a SHM segment, the segment size column in ipcs -m output will not tell us the truth.

                            It pretty much only tells us maximum possible size of that vm segment, which (thanks to memory virtualization layer) does not translate directly to physical memory usage nor even allocation of corresponding virtual memory structures.

                            When dealing with processes attached to shm segments (as Oracle processes are), then standard tools like ps, top and prstat also get in trouble reporting how much physical memory really is used by a process.

                            And Windows is no different in the sense of virtualized memory and not immediately physically allocating all memory Oracle asks for SGA.

                            Windows memory allocation is also often misunderstood thanks to Task Managers misleading naming conventions. The task manager's "Memory Used" shows the processes working set size, which includes that processes private memory pages which happen to be paged in at that time + paged in pages of that processes memory mapped files, including Oracle binary and system libraries. The working set size can be limited by (a dynamic) Windows per-process page-count limit.

                            One column which the task manager doesn't show at all is the total size of the virtual address space for the process (task manager calls the private bytes of a process misleadingly VM size). If you use perfmon or process explorer, you'll see that the processes virtual mem addr space there is larger than the number in task manager.

                            Anyway, when you start an Oracle instance on Windows, you will see (in perfmon or procexp) that

                            1) process virtual size is immedately ramped up to SGA_MAX_SIZE + some mem for PGAs & stacks + Oracle binary + libraries
                            2) private bytes jumps up to SGA + some mem for PGAs & stacks - however physical memory is allocated only when the corresponding pages are touched!
                            3) the working set size will initially remain low (working set consists of pages actually in physical memory and paged in to process address space). only if you start generating enough buffer cache & shared pool activity in your database, you will see the working set growing - as now the virtual memory pages are touched first time & paged in. Note that even if some virtual memory mappings are removed from vm translation tables for a process (thus working set size drops) the actual pages may still remain "cached" in physical memory - they'll stay in a standby list if memory pressure permits. So they still occupy memory even though they're not reported in process working set. And you'd need a kernel debugger to report standby pages "belonging" to a process.


                            So the baseline is - "thanks" to memory virtualization and little instrumentation due performance reasons, there is no easy & fast way to get a completely accurate picture of individual processes memory usage.

                            In other words - we usually interpret these memory stats incorrectly.
                            • 41. Re: SGA_MAX_SIZE != SGA_TARGET when?
                              519926
                              I've just spent the morning with another DBA looking into this further. Now I'm not sure what to believe. Excuse the long post, hopefully someone will see something that 'clears the fog' regarding the memory usage of Oracle.

                              To recap : One instance on a Solaris 10 machine running in a non-global zone. 4G RAM. 2 CPUs. Connection to a SAN.

                              The following lines are from the outputs of top, vmstat, prstat, ps and the info given when Oracle starts. Apart from starting and stopping the instance, no modifications were made to the size of the parameters sga_max_size or sga_target (in fact no instance parameters were changed during this data collection).

                              TOP
                              load averages: 0.08, 0.05, 0.04 vxts0099 12:57:01
                              120 processes: 119 sleeping, 1 on cpu
                              CPU states: 84.7% idle, 13.9% user, 1.4% kernel, 0.0% iowait, 0.0% swap
                              Memory: 4.0G real, 2.5G free, 2.9G swap in use, 8.1G swap free

                              load averages: 0.09, 0.06, 0.04 vxts0099 12:57:11
                              120 processes: 118 sleeping, 2 on cpu
                              CPU states: 99.2% idle, 0.2% user, 0.6% kernel, 0.0% iowait, 0.0% swap
                              Memory: 4.0G real, 2.5G free, 2.9G swap in use, 8.1G swap free

                              SHUTDOWN IMMEDIATE

                              load averages: 0.07, 0.06, 0.04 vxts0099 12:57:42
                              104 processes: 102 sleeping, 1 running, 1 on cpu
                              CPU states: 99.4% idle, 0.1% user, 0.5% kernel, 0.0% iowait, 0.0% swap
                              Memory: 4.0G real, 2.8G free, 808M swap in use, 10.2G swap free

                              load averages: 0.07, 0.06, 0.04 vxts0099 12:57:57
                              104 processes: 102 sleeping, 1 running, 1 on cpu
                              CPU states: 97.7% idle, 0.1% user, 2.2% kernel, 0.0% iowait, 0.0% swap
                              Memory: 4.0G real, 2.8G free, 808M swap in use, 10.2G swap free

                              Instance shutdown still
                              prstat -s size -u oracle
                              PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
                              10174 oracle 129M 115M sleep 1 0 0:00:01 0.0% oracle/11
                              2602 oracle 27M 22M sleep 44 0 0:00:00 0.0% tnslsnr/3
                              3433 oracle 26M 22M sleep 1 0 0:00:00 0.0% sqlplus/1
                              24168 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              957 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              15547 oracle 3256K 2992K cpu1 1 0 0:00:00 0.1% prstat/1
                              24170 oracle 1536K 1440K sleep 57 0 0:00:00 0.0% ksh/1
                              959 oracle 1504K 1400K sleep 1 0 0:00:00 0.0% ksh/1

                              Also a couple of secs later to be sure
                              PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
                              10174 oracle 129M 115M sleep 1 0 0:00:01 0.0% oracle/11
                              2602 oracle 27M 22M sleep 44 0 0:00:00 0.0% tnslsnr/3
                              3433 oracle 26M 22M sleep 1 0 0:00:00 0.0% sqlplus/1
                              24168 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              957 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              15547 oracle 3256K 2992K cpu0 1 0 0:00:00 0.1% prstat/1
                              24170 oracle 1536K 1440K sleep 57 0 0:00:00 0.0% ksh/1
                              959 oracle 1504K 1400K sleep 1 0 0:00:00 0.0% ksh/1

                              I've got two sessions open on the server, one is an sqlplus session
                              ps -ef |grep oracle
                              oracle 15548 3433 0 12:58:53 pts/1 0:00 grep oracle
                              oracle 10174 3433 0 11:34:04 ? 0:02 oraclez111ot11 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
                              oracle 15547 959 0 12:58:40 pts/2 0:00 prstat -s size -u oracle
                              oracle 3433 24170 0 09:53:19 pts/1 0:00 sqlplus /as sysdba
                              oracle 2602 11787 0 09:46:02 ? 0:01 /appli/oracle/10.2.0/bin/tnslsnr LISTENER -inherit
                              oracle 959 957 0 09:29:53 pts/2 0:00 -ksh
                              oracle 24170 24168 0 07:40:16 pts/1 0:00 -ksh
                              oracle 957 955 0 09:29:53 ? 0:01 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 24168 24166 0 07:40:16 ? 0:00 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 15549 15548 0 12:58:53 pts/1 0:00 ps -ef

                              STARTUP
                              ORACLE instance started.

                              Total System Global Area 2147483648 bytes
                              Fixed Size 2031520 bytes
                              Variable Size 1577058400 bytes
                              Database Buffers 553648128 bytes
                              Redo Buffers 14745600 bytes
                              Database mounted.
                              Database opened.

                              Repeating the same prstat
                              prstat -s size -u oracle
                              PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
                              15742 oracle 2190M 236M sleep 59 0 0:00:00 0.1% oracle/11
                              15760 oracle 2188M 334M sleep 10 0 0:00:00 0.1% oracle/1
                              15740 oracle 2182M 239M sleep 1 0 0:00:00 0.1% oracle/11
                              15758 oracle 2181M 343M sleep 1 0 0:00:01 1.9% oracle/11
                              15744 oracle 2179M 236M sleep 54 0 0:00:00 0.1% oracle/11
                              15752 oracle 2179M 241M sleep 2 0 0:00:00 1.0% oracle/1
                              15750 oracle 2178M 236M sleep 46 0 0:00:00 0.4% oracle/1
                              15734 oracle 2178M 231M sleep 59 0 0:00:00 0.1% oracle/1
                              15746 oracle 2177M 235M sleep 34 0 0:00:00 0.3% oracle/1
                              15754 oracle 2177M 234M sleep 42 0 0:00:00 0.1% oracle/1
                              15768 oracle 2177M 374M sleep 59 0 0:00:00 0.1% oracle/1
                              15762 oracle 2177M 362M sleep 53 0 0:00:00 0.1% oracle/1
                              15738 oracle 2177M 234M sleep 1 0 0:00:00 0.5% oracle/1
                              15748 oracle 2177M 234M sleep 47 0 0:00:00 0.1% oracle/1
                              15736 oracle 2177M 234M sleep 59 0 0:00:00 0.1% oracle/1
                              2602 oracle 27M 22M sleep 44 0 0:00:00 0.0% tnslsnr/3
                              3433 oracle 26M 22M sleep 57 0 0:00:00 0.0% sqlplus/1
                              24168 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              957 oracle 6112K 1960K sleep 59 0 0:00:00 0.0% sshd/1
                              15547 oracle 3272K 3008K cpu0 1 0 0:00:00 0.1% prstat/1
                              24170 oracle 1536K 1440K sleep 57 0 0:00:00 0.0% ksh/1
                              959 oracle 1504K 1400K sleep 1 0 0:00:00 0.0% ksh/1

                              Verify the processes
                              ps -ef |grep oracle
                              oracle 15750 11787 0 13:00:08 ? 0:00 ora_cjq0_z111ot11
                              oracle 15746 11787 0 13:00:08 ? 0:00 ora_smon_z111ot11
                              oracle 15736 11787 0 13:00:07 ? 0:00 ora_psp0_z111ot11
                              oracle 15779 3433 0 13:00:41 pts/1 0:00 grep oracle
                              oracle 15547 959 0 12:58:40 pts/2 0:00 prstat -s size -u oracle
                              oracle 15754 11787 0 13:00:08 ? 0:00 ora_mmnl_z111ot11
                              oracle 3433 24170 0 09:53:19 pts/1 0:00 sqlplus /as sysdba
                              oracle 15748 11787 0 13:00:08 ? 0:00 ora_reco_z111ot11
                              oracle 15762 11787 0 13:00:13 ? 0:00 ora_qmnc_z111ot11
                              oracle 2602 11787 0 09:46:02 ? 0:01 /appli/oracle/10.2.0/bin/tnslsnr LISTENER -inherit
                              oracle 15760 11787 0 13:00:12 ? 0:00 ora_arc0_z111ot11
                              oracle 959 957 0 09:29:53 pts/2 0:00 -ksh
                              oracle 15742 11787 0 13:00:07 ? 0:00 ora_lgwr_z111ot11
                              oracle 15768 11787 0 13:00:23 ? 0:00 ora_q000_z111ot11
                              oracle 15738 11787 0 13:00:07 ? 0:00 ora_mman_z111ot11
                              oracle 15740 11787 0 13:00:07 ? 0:00 ora_dbw0_z111ot11
                              oracle 24170 24168 0 07:40:16 pts/1 0:00 -ksh
                              oracle 957 955 0 09:29:53 ? 0:01 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 15744 11787 0 13:00:07 ? 0:00 ora_ckpt_z111ot11
                              oracle 15780 15779 0 13:00:41 pts/1 0:00 ps -ef
                              oracle 15752 11787 1 13:00:08 ? 0:01 ora_mmon_z111ot11
                              oracle 24168 24166 0 07:40:16 ? 0:00 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 15758 3433 1 13:00:12 ? 0:01 oraclez111ot11 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
                              oracle 15734 11787 0 13:00:07 ? 0:00 ora_pmon_z111ot11
                              oracle 15750 11787 0 13:00:08 ? 0:00 ora_cjq0_z111ot11
                              oracle 15746 11787 0 13:00:08 ? 0:00 ora_smon_z111ot11
                              oracle 15736 11787 0 13:00:07 ? 0:00 ora_psp0_z111ot11
                              oracle 15779 3433 0 13:00:41 pts/1 0:00 grep oracle
                              oracle 15547 959 0 12:58:40 pts/2 0:00 prstat -s size -u oracle
                              oracle 15754 11787 0 13:00:08 ? 0:00 ora_mmnl_z111ot11
                              oracle 3433 24170 0 09:53:19 pts/1 0:00 sqlplus /as sysdba
                              oracle 15748 11787 0 13:00:08 ? 0:00 ora_reco_z111ot11
                              oracle 15762 11787 0 13:00:13 ? 0:00 ora_qmnc_z111ot11
                              oracle 2602 11787 0 09:46:02 ? 0:01 /appli/oracle/10.2.0/bin/tnslsnr LISTENER -inherit
                              oracle 15760 11787 0 13:00:12 ? 0:00 ora_arc0_z111ot11
                              oracle 959 957 0 09:29:53 pts/2 0:00 -ksh
                              oracle 15742 11787 0 13:00:07 ? 0:00 ora_lgwr_z111ot11
                              oracle 15768 11787 0 13:00:23 ? 0:00 ora_q000_z111ot11
                              oracle 15738 11787 0 13:00:07 ? 0:00 ora_mman_z111ot11
                              oracle 15740 11787 0 13:00:07 ? 0:00 ora_dbw0_z111ot11
                              oracle 24170 24168 0 07:40:16 pts/1 0:00 -ksh
                              oracle 957 955 0 09:29:53 ? 0:01 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 15744 11787 0 13:00:07 ? 0:00 ora_ckpt_z111ot11
                              oracle 15780 15779 0 13:00:41 pts/1 0:00 ps -ef
                              oracle 15752 11787 1 13:00:08 ? 0:01 ora_mmon_z111ot11
                              oracle 24168 24166 0 07:40:16 ? 0:00 /soe3/opt/openssh/sbin/sshd -f /soe3/opt/openssh/etc/sshd_config -R
                              oracle 15758 3433 1 13:00:12 ? 0:01 oraclez111ot11 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
                              oracle 15734 11787 0 13:00:07 ? 0:00 ora_pmon_z111ot11

                              Now I looked at vmstat
                              Instance is running
                              oracle@vxts0099:z111ot11 # vmstat 2 100
                              kthr memory page disk faults cpu
                              r b w swap free re mf pi po fr de sr m0 m1 m3 m4 in sy cs us sy id
                              0 0 26 5887584 390000 33 131 121 33 47 0 642 1 2 1 1 340 712 557 1 1 98
                              0 0 25 8470856 2609984 0 5 0 0 0 0 0 0 0 0 0 318 273 311 0 1 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 320 268 320 0 0 100
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 345 381 358 0 1 98
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 327 237 321 0 4 96
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 318 284 314 0 0 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 314 267 323 0 1 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 4 0 317 205 289 0 0 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 1 319 252 307 0 0 100
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 316 238 312 0 0 100
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 326 358 377 0 0 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 321 295 321 0 0 100
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 323 282 342 0 1 99
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 330 283 336 0 0 100
                              0 0 25 8470856 2609984 0 0 0 0 0 0 0 0 0 0 0 324 217 312 0 1 99
                              0 0 25 8484792 2618472 12 31 0 0 0 0 0 0 0 0 0 366 567 477 1 2 97
                              SHUTDOWN IMMEDIATE HERE
                              0 0 25 8506104 2627952 428 20 0 12 4 0 0 0 0 0 0 572 3135 1025 7 3 90
                              0 0 25 8506104 2623464 0 0 0 0 0 0 0 0 0 0 0 300 186 262 0 0 100
                              kthr memory page disk faults cpu
                              r b w swap free re mf pi po fr de sr m0 m1 m3 m4 in sy cs us sy id
                              0 0 25 8506104 2623464 0 0 0 0 0 0 0 0 0 0 0 316 287 322 0 5 95
                              0 0 25 8520000 2625448 7 10 0 0 0 0 0 0 0 0 0 314 475 336 1 2 97
                              0 0 25 8558760 2648544 11 29 0 4 4 0 0 0 0 0 0 316 455 340 2 4 93
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 305 253 288 0 2 97
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 308 252 306 0 0 100
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 315 310 320 0 0 99
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 311 216 280 0 0 100
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 301 206 285 0 0 100
                              0 0 25 10689920 2968728 0 0 0 0 0 0 0 0 0 0 0 317 243 296 0 0 100
                              0 0 25 10689920 2968728 85 301 0 20 20 0 0 0 0 1 0 311 1278 299 1 1 98
                              0 0 25 10689920 2968768 81 302 0 16 16 0 0 0 0 0 0 323 1343 324 0 1 98
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 308 208 270 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 307 203 283 0 0 99
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 320 231 298 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 299 194 273 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 325 253 305 0 5 95
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 314 210 292 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 319 215 312 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 302 237 282 0 0 100
                              kthr memory page disk faults cpu
                              r b w swap free re mf pi po fr de sr m0 m1 m3 m4 in sy cs us sy id
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 4 0 306 173 271 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 1 316 244 296 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 308 213 281 0 0 100
                              0 0 25 10689920 2968768 0 0 0 0 0 0 0 0 0 0 0 309 212 289 0 0 100
                              0 0 25 10689920 2968760 125 376 0 4 4 0 0 0 0 0 1 317 1251 317 1 1 98
                              0 0 25 10689920 2968760 0 0 0 0 0 0 0 0 0 0 0 312 180 292 0 0 100

                              And a final top, just for the hell of it.
                              load averages: 0.03, 0.05, 0.05 vxts0099 13:04:24
                              104 processes: 103 sleeping, 1 on cpu
                              CPU states: 99.4% idle, 0.2% user, 0.4% kernel, 0.0% iowait, 0.0% swap
                              Memory: 4.0G real, 2.8G free, 808M swap in use, 10.2G swap free

                              The change in memory use given by top shows a 300M difference and for swap a 2.1G difference.

                              The change in memory use given by vmstat shows a 360M difference and for swap a 2.2G difference.

                              So ... to me the initial opinion of Hans is correct. We are not allocating SGA_MAX_SIZE of RAM. It is a combination of memory types that have been allocated. What doesn't help me is when I see something like this from the vmstat man page : "memory Report on usage of virtual and real memory." My understanding was that virtual memory is equal to physical + swap. What exactly is 'real' memory ? Physical ?

                              I am going to pursue this with the instance working a bit to try and understand it more.

                              Or I might try carpentry ;-)
                              • 42. Re: SGA_MAX_SIZE != SGA_TARGET when?
                                Tanel Poder
                                Hi,

                                You are using dynamic ISM which is pageable, that's why you see such behaviour. You didn't mention what are the actual values of your SGA_MAX_SIZE & SGA_TARGET but I'm sure you'r SGA_MAX_SIZE is larger than SGA_TARGET, thus enabling DISM on Solaris 10.

                                You saw a large drop in swapfs usage during shutdown because Solaris has to back ALL requested memory with swapfs space (and swapfs space contains on-disk swap space an in-memory swap cache ;)

                                You saw a smaller drop in physical memory usage just because you had touched & initialized only small amount of SGA pages since last startup, not the whole SGA.

                                If you run your test having SGA_MAX_SIZE = SGA_TARGET you'll experience different behaviour as plain non-dynamic ISM allocates the whole amount of SGA from physical memory and does not allocate any swap space at all (as ISM memory is locked into memory, non-pageable).

                                You can use ipcs -ma and pmap -xs to get detailed info about what kind of shm segments are in use for your instance (my examples are from Solaris 10 x64):

                                See how in the ISM case RSS (resident set size) = Kbytes (shm segment size) = Locked.

                                However in DISM case the RSS may be smaller than Kbytes and the segments are not locked into memory, thus pageable, thus not necessarily allocated from physical memory during the shm segment creation.


                                SGA_MAX_SIZE = SGA_TARGET (ISM)

                                $ pmap -xs 7455 | egrep "Address|ism"
                                Address Kbytes RSS Anon Locked Pgsz Mode Mapped File
                                0000000380000000 464896 464896 - 464896 2M rwxsR [ ism shmid=0x4 ]
                                000000039C600000 47112 47112 - 47112 4K rwxsR [ ism shmid=0x4 ]

                                SGA_MAX_SIZE > SGA_TARGET (DISM)

                                $ pmap -xs `pgrep -f dbw0` | egrep "Address|ism"
                                Address Kbytes RSS Anon Locked Pgsz Mode Mapped File
                                0000000380000000 2048 2048 - - 2M rwxs- [ dism shmid=0x14 ]
                                0000000380200000 436224 333824 - - - rwxs- [ dism shmid=0x14 ]
                                000000039AC00000 6144 6144 - - 2M rwxs- [ dism shmid=0x14 ]
                                000000039B200000 4096 4096 - - - rwxs- [ dism shmid=0x14 ]
                                000000039B600000 2048 2048 - - 2M rwxs- [ dism shmid=0x14 ]
                                000000039B800000 2048 2048 - - - rwxs- [ dism shmid=0x14 ]
                                000000039BA00000 4096 4096 - - 2M rwxs- [ dism shmid=0x14 ]
                                000000039BE00000 18432 18432 - - - rwxs- [ dism shmid=0x14 ]
                                000000039D000000 2048 2048 - - 2M rwxs- [ dism shmid=0x14 ]
                                000000039D200000 2048 2048 - - - rwxs- [ dism shmid=0x14 ]
                                • 43. Re: SGA_MAX_SIZE != SGA_TARGET when?
                                  Tanel Poder
                                  comment: you need to run pmap on any of your oracle server processes (as they are attached to sga)
                                  • 44. Re: SGA_MAX_SIZE != SGA_TARGET when?
                                    519926
                                    Excellent explanation Tanel !!! Thankyou.

                                    One small sentence that I didn't understand : swapfs space contains on-disk swap space an(d)? in-memory swap cache

                                    Could you elaborate on this or point me to a link that does please?

                                    Once again thanks.