6 Replies Latest reply: Apr 2, 2013 4:30 PM by 990636 RSS

    ORA-01034,ORA-27102 out of memory

    user13148231
      Oracle 11.2.0.1 on Linux, 3 node RAC. The servers are all 4 CPU, 32gb RAN and all dedicated server used by Oracle only. One of the nodes got error:
      ORA-01034: ORACLE not available
      ORA-27102: out of memory
      Linux-x86_64 Error: 12: Cannot allocate memory
      Additional information: 1
      Additional information: 458763
      Additional information: 8
      Process ID: 0
      Session ID: 0 Serial number: 0
      Look in alert file and saw
      Tue Nov 13 19:30:13 2012
      Starting background process VKRM
      Process VKRM died, see its trace file
      Errors in file /u01/app/oracle/diag/rdbms/prdcch/prdcch2/trace/prdcch2_pmon_18870.trc:
      The trace file did not give more info. Had to restart the server box, and then Oracle start up OK.

      What actually cause the problem. How to use the msg 'Additional information: 458763' and other 2 Additional information.

      Thanks.
        • 1. Re: ORA-01034,ORA-27102 out of memory
          sb92075
          compare OS/kernel parameters between the systems.
          • 2. Re: ORA-01034,ORA-27102 out of memory
            user13148231
            Thanks. Did you mean compare OS/kernel parameters of the 3 nodes?
            • 3. Re: ORA-01034,ORA-27102 out of memory
              nnarimanov
              Please output
              ulimit -a or
              more /etc/security/limits.conf
              • 4. Re: ORA-01034,ORA-27102 out of memory
                user13148231
                Did find some difference in the bad node and a good node
                bad node
                [oracle@cchORdb2 ~]$ ulimit -a
                core file size          (blocks, -c) 0
                data seg size           (kbytes, -d) unlimited
                scheduling priority             (-e) 0
                file size               (blocks, -f) unlimited
                pending signals                 (-i) 278528
                max locked memory       (kbytes, -l) 50000000  
                max memory size         (kbytes, -m) unlimited
                open files                      (-n) 65536
                pipe size            (512 bytes, -p) 8
                POSIX message queues     (bytes, -q) 819200
                real-time priority              (-r) 0
                stack size              (kbytes, -s) 10240
                cpu time               (seconds, -t) unlimited
                max user processes              (-u) 16384
                virtual memory          (kbytes, -v) unlimited
                file locks                      (-x) unlimited
                
                Godd node
                [oracle@cchORdb3 ~]$ ulimit -a
                core file size          (blocks, -c) 0
                data seg size           (kbytes, -d) unlimited
                scheduling priority             (-e) 0
                file size               (blocks, -f) unlimited
                pending signals                 (-i) 278528
                max locked memory       (kbytes, -l) 32
                max memory size         (kbytes, -m) unlimited
                open files                      (-n) 1024
                pipe size            (512 bytes, -p) 8
                POSIX message queues     (bytes, -q) 819200
                real-time priority              (-r) 0
                stack size              (kbytes, -s) 10240
                cpu time               (seconds, -t) unlimited
                max user processes              (-u) 16384
                virtual memory          (kbytes, -v) unlimited
                file locks                      (-x) unlimited
                Two items have different values: max locked memory and open files. The third node, which never show this error has the values as
                max locked memory       (kbytes, -l) 50000000 
                max memory size         (kbytes, -m) unlimited
                open files                      (-n) 131072 
                It is more like the bad node.

                Like to have your comments.

                Thanks
                • 5. Re: ORA-01034,ORA-27102 out of memory
                  user13148231
                  It is the process oraagent.bin used much of the memoryMOS suggest to kill it.
                  • 6. Re: ORA-01034,ORA-27102 out of memory
                    990636
                    ipcs -a

                    check the memory on all nodes using the above command

                    you free up the memory by killing oraagent.bin,but it's not complete solution,it may happen again when a instance goes down it won't come up.

                    check ipcs -a on all nodes and see any of the nodes take more memory than the other instance.