3 Replies Latest reply on Aug 30, 2012 5:23 AM by Osama_Mustafa

    Linux ORA-27102 out of memory

    TimWong765
      A few days ago, some processes could not log in and got error message
      ORA-01034: ORACLE not available
      ORA-27102: out of memory
      Linux-x86_64 Error: 12: Cannot allocate memory
      After some time, the problem was suddenly gone. We had such issues already in the past.


      Is there a way to get/monitor actual memory usage ( a)what Oracle thinks it uses and b)what OS thinks Oracle uses)?
      Tools like ps and top seam to show shared memory incorrectly and RAM used may be completely wrong!? We currently only use 30G (memory_target) on a server having 72G. We intended to increase memory_target some time ago but did not implement the change due to ORA-27102 coming up from time.


      OS: SUSE SLES 64bit
      Oracle 11.2.0.2 64bit


      BANNER
      --------------------------------------------------------------------------------
      Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
      PL/SQL Release 11.2.0.2.0 - Production
      CORE 11.2.0.2.0 Production
      TNS for Linux: Version 11.2.0.2.0 - Production
      NLSRTL Version 11.2.0.2.0 - Production



      cat /proc/meminfo
      MemTotal:       74306292 kB
      MemFree:          498240 kB
      Buffers:         5461300 kB
      Cached:         60442304 kB
      SwapCached:       185256 kB
      Active:         33642300 kB
      Inactive:       35066764 kB
      Active(anon):   20735148 kB
      Inactive(anon):  1302912 kB
      Active(file):   12907152 kB
      Inactive(file): 33763852 kB
      Unevictable:        5932 kB
      Mlocked:            5932 kB
      SwapTotal:       8388600 kB
      SwapFree:        7142712 kB
      Dirty:              2700 kB
      Writeback:             0 kB
      AnonPages:       2628540 kB
      Mapped:         19062944 kB
      Shmem:          19229668 kB
      Slab:            1968204 kB
      SReclaimable:    1781108 kB
      SUnreclaim:       187096 kB
      KernelStack:        5744 kB
      PageTables:      2071144 kB
      NFS_Unstable:          0 kB
      Bounce:                0 kB
      WritebackTmp:          0 kB
      CommitLimit:    45541744 kB
      Committed_AS:   24223804 kB
      VmallocTotal:   34359738367 kB
      VmallocUsed:      290600 kB
      VmallocChunk:   34321279204 kB
      HardwareCorrupted:     0 kB
      HugePages_Total:       0
      HugePages_Free:        0
      HugePages_Rsvd:        0
      HugePages_Surp:        0
      Hugepagesize:       2048 kB
      DirectMap4k:        6384 kB
      DirectMap2M:     2080768 kB
      DirectMap1G:    73400320 kB
      Top shows constantly values of high RAM usage.
      Mem:     72564M total,    65263M used,     7301M free,     3797M buffers
      There are no other processes. It's a pure DB server.
      Top shows the following process list
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
       8067 oracle  20   0 30.3g  17g  17g S    0 24.8   4:11.50 oracle
       8075 oracle  20   0 30.3g  16g  16g S    0 22.8  40:26.79 oracle
       8071 oracle  20   0 30.3g  16g  16g S    0 22.8  42:03.30 oracle
       8087 oracle  20   0 30.3g 9.7g 9.7g S    0 13.6  10:35.84 oracle
      16561 oracle  20   0 30.3g 3.0g 3.0g S    0  4.3   0:42.38 oracle
       8095 oracle  20   0 30.3g 2.7g 2.7g S    0  3.8   6:45.29 oracle
       1229 oracle  20   0 30.3g 1.1g 1.1g S    0  1.6   0:32.12 oracle
      16570 oracle  20   0 30.3g 996m 992m S    0  1.4   0:05.02 oracle
      26773 oracle  20   0 30.3g 966m 962m S    0  1.3   0:14.90 oracle
       8099 oracle  20   0 30.3g 907m 906m S    0  1.3  27:40.60 oracle
        624 oracle  20   0 30.4g 737m 647m S    0  1.0   0:04.16 oracle
       8083 oracle  20   0 30.3g 733m 729m D    0  1.0  11:10.35 oracle
       2705 oracle  20   0 30.3g 695m 685m S    0  1.0   0:31.26 oracle
      13357 oracle  20   0 30.3g 692m 687m S    0  1.0   0:22.71 oracle
      SQL> show parameter memory
      
      NAME                                 TYPE                                         VALUE
      ------------------------------------ -------------------------------------------- ------------------------------
      hi_shared_memory_address             integer                                      0
      memory_max_target                    big integer                                  30G
      memory_target                        big integer                                  30G
      shared_memory_address                integer                                      0
      SQL> show parameter sga
      
      NAME                                 TYPE                                         VALUE
      ------------------------------------ -------------------------------------------- ------------------------------
      lock_sga                             boolean                                      FALSE
      pre_page_sga                         boolean                                      FALSE
      sga_max_size                         big integer                                  30G
      sga_target                           big integer                                  15G
      SQL> show parameter pga
      
      NAME                                 TYPE                                         VALUE
      ------------------------------------ -------------------------------------------- ------------------------------
      pga_aggregate_target                 big integer                                  2500M
      ulimit -a
      core file size          (blocks, -c) 1
      data seg size           (kbytes, -d) unlimited
      scheduling priority             (-e) 0
      file size               (blocks, -f) unlimited
      pending signals                 (-i) 580443
      max locked memory       (kbytes, -l) 64
      max memory size         (kbytes, -m) 63160352
      open files                      (-n) 1024
      pipe size            (512 bytes, -p) 8
      POSIX message queues     (bytes, -q) 819200
      real-time priority              (-r) 0
      stack size              (kbytes, -s) 8192
      cpu time               (seconds, -t) unlimited
      max user processes              (-u) 580443
      virtual memory          (kbytes, -v) 66155920
      file locks                      (-x) unlimited
      Edited by: TimWong765 on Aug 29, 2012 9:26 PM
        • 1. Re: Linux ORA-27102 out of memory
          sb92075
          TimWong765 wrote:
          A few days ago, some processes could not log in and got error message
          ORA-01034: ORACLE not available
          ORA-27102: out of memory
          Linux-x86_64 Error: 12: Cannot allocate memory
          http://www.asktherealtom.ch/?tag=ora-27102-out-of-memory
          1 person found this helpful
          • 2. Re: Linux ORA-27102 out of memory
            TimWong765
            Thanks for you reply. We may consider lock_sga=true (events like "latch: shared pool" do actually come up). The referenced thread had the problem during startup though. The problem occurred in our DB while the system was running - but maybe swapping/paging occurred exactly during that time!?

            Edited by: TimWong765 on Aug 29, 2012 9:59 PM
            • 3. Re: Linux ORA-27102 out of memory
              Osama_Mustafa
              Check this MOS note
              Summary of Causes for ORA-27102 on Startup [ID 1080317.1]
              Ora-27102: Out Of Memory: Linux Error: 12: Cannot Allocate Memory with LOCK_SGA=TRUE [ID 401077.1]
              ORA-27102 Received At Startup When LOCK_SGA Is Set Although Enough Memory Is Available. [ID 577898.1]