This discussion is archived
6 Replies Latest reply: Apr 2, 2013 2:30 PM by 990636 RSS

ORA-01034,ORA-27102 out of memory

user13148231 Newbie
Currently Being Moderated
Oracle 11.2.0.1 on Linux, 3 node RAC. The servers are all 4 CPU, 32gb RAN and all dedicated server used by Oracle only. One of the nodes got error:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 1
Additional information: 458763
Additional information: 8
Process ID: 0
Session ID: 0 Serial number: 0
Look in alert file and saw
Tue Nov 13 19:30:13 2012
Starting background process VKRM
Process VKRM died, see its trace file
Errors in file /u01/app/oracle/diag/rdbms/prdcch/prdcch2/trace/prdcch2_pmon_18870.trc:
The trace file did not give more info. Had to restart the server box, and then Oracle start up OK.

What actually cause the problem. How to use the msg 'Additional information: 458763' and other 2 Additional information.

Thanks.
  • 1. Re: ORA-01034,ORA-27102 out of memory
    sb92075 Guru
    Currently Being Moderated
    compare OS/kernel parameters between the systems.
  • 2. Re: ORA-01034,ORA-27102 out of memory
    user13148231 Newbie
    Currently Being Moderated
    Thanks. Did you mean compare OS/kernel parameters of the 3 nodes?
  • 3. Re: ORA-01034,ORA-27102 out of memory
    nnarimanov Newbie
    Currently Being Moderated
    Please output
    ulimit -a or
    more /etc/security/limits.conf
  • 4. Re: ORA-01034,ORA-27102 out of memory
    user13148231 Newbie
    Currently Being Moderated
    Did find some difference in the bad node and a good node
    bad node
    [oracle@cchORdb2 ~]$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 278528
    max locked memory       (kbytes, -l) 50000000  
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 65536
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 16384
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    
    Godd node
    [oracle@cchORdb3 ~]$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 278528
    max locked memory       (kbytes, -l) 32
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 16384
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    Two items have different values: max locked memory and open files. The third node, which never show this error has the values as
    max locked memory       (kbytes, -l) 50000000 
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 131072 
    It is more like the bad node.

    Like to have your comments.

    Thanks
  • 5. Re: ORA-01034,ORA-27102 out of memory
    user13148231 Newbie
    Currently Being Moderated
    It is the process oraagent.bin used much of the memoryMOS suggest to kill it.
  • 6. Re: ORA-01034,ORA-27102 out of memory
    990636 Newbie
    Currently Being Moderated
    ipcs -a

    check the memory on all nodes using the above command

    you free up the memory by killing oraagent.bin,but it's not complete solution,it may happen again when a instance goes down it won't come up.

    check ipcs -a on all nodes and see any of the nodes take more memory than the other instance.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points