12 Replies Latest reply: Nov 22, 2010 12:54 AM by CKPT RSS

    ORA-27102: out of memory

    690963
      Database Doesnt start
      SQL> startup
      ORA-27102: out of memory
      IBM AIX RISC System/6000 Error: 12: Not enough space
      OS, db details
      uname -a
      AIX db-server 3 5 00CDD8E14C00
      
      
      Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
      PL/SQL Release 11.1.0.7.0 - Production
      CORE    11.1.0.7.0      Production
      TNS for IBM/AIX RISC System/6000: Version 11.1.0.7.0 - Production
      NLSRTL Version 11.1.0.7.0 - Production
      Alert Log
      Errors in file /oracle/app/oracle/diag/rdbms/enmgr/enmgr/trace/enmgr_ora_3031218.trc  (incident=2099110):
      ORA-00600: internal error code, arguments: [kghfrmrg:prv], [0x700000166F2FE88], [], [], [], [], [], [], [], [], [], []
      Incident details in: /oracle/app/oracle/diag/rdbms/enmgr/enmgr/incident/incdir_2099110/enmgr_ora_3031218_i2099110.trc
      Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4C00012C4800005E] [PC:0x1005A82A0, kocedc()+0020]
      Errors in file /oracle/app/oracle/diag/rdbms/enmgr/enmgr/trace/enmgr_ora_3031218.trc  (incident=2099111):
      ORA-07445: exception encountered: core dump [kocedc()+0020] [SIGSEGV] [ADDR:0x4C00012C4800005E] [PC:0x1005A82A0] [Address not mapped to object] []
      ORA-00600: internal error code, arguments: [kghfrmrg:prv], [0x700000166F2FE88], [], [], [], [], [], [], [], [], [], []
      Incident details in: /oracle/app/oracle/diag/rdbms/enmgr/enmgr/incident/incdir_2099111/enmgr_ora_3031218_i2099111.trc
      Sun Nov 21 00:00:21 2010
      Sweep Incident[2099110]: completed
      Sun Nov 21 00:00:56 2010
      Errors in file /oracle/app/oracle/diag/rdbms/enmgr/enmgr/trace/enmgr_pmon_934134.trc  (incident=2098818):
      ORA-00600: internal error code, arguments: [17112], [0x700000166F306E0], [], [], [], [], [], [], [], [], [], []
      Incident details in: /oracle/app/oracle/diag/rdbms/enmgr/enmgr/incident/incdir_2098818/enmgr_pmon_934134_i2098818.trc
      Errors in file /oracle/app/oracle/diag/rdbms/enmgr/enmgr/trace/enmgr_pmon_934134.trc:
      ORA-00600: internal error code, arguments: [17112], [0x700000166F306E0], [], [], [], [], [], [], [], [], [], []
      Nothing specific on Metalink. Please help
        • 1. Re: ORA-27102: out of memory
          sb92075
          Nothing specific on Metalink. Please help
          most likely the OS kernel parameters are not properly sized.

          The AIX Installation Guide shows correct values.

          When was last time Oracle DB worked on this system?
          What changed?
          • 2. Re: ORA-27102: out of memory
            690963
            Hi SB,

            Thankyou for your reply.

            Db was running fine till this morning. Then it crashed. Nothing changed. I am new to AIX. Not sure how to check SHMMALL (kernel ) parameter.
            • 3. Re: ORA-27102: out of memory
              sb92075
              post results of following 2 OS commands using COPY & PASTE

              id
              env | sort
              • 4. Re: ORA-27102: out of memory
                690963
                oracle@server678: ~ id
                uid=4085(oracle) gid=1140(oinstall) groups=235(dba),739(download)
                
                oracle@server678: ~ env | sort
                AIXTHREAD_SCOPE=S
                A__z=! LOGNAME="*TMOUT
                BOKSPATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/local/bin:/usr/java14/jre/bin:
                BOKS_START_PRG=
                CVSROOT=:ext:server678:/cvs/oracle
                CVS_RSH=ssh
                DEFAULT_BROWSER=firefox
                EDITOR=vi
                G_BROKEN_FILENAMES=1
                HISTSIZE=10000
                HOME=/home/oracle
                HOSTNAME=server678
                IC_DOCUMENT_DIRECTORY=/opt/IBM/aix
                IC_DOCUMENT_SERVER_MACHINE_NAME=standalone
                IC_DOCUMENT_SERVER_PORT=64111
                IC_DOCUMENT_SERVER_TYPE=STANDALONE
                LANG=en_US
                LC__FASTMSG=true
                LOCPATH=/usr/lib/nls/loc
                LOGIN=oracle
                LOGNAME=oracle
                MAIL=/usr/spool/mail/oracle
                MAILMSG=[YOU HAVE NEW MAIL]
                MANPATH=/usr/share/man:/opt/freeware/man:/usr/local/man
                NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
                ODMDIR=/etc/objrepos
                OLDPWD=/oracle/app/oracle/diag/rdbms/enmgr/enmgr/trace
                ORACLE_BASE=/oracle/app/oracle
                ORACLE_HOME=/usr/oracle
                ORACLE_SID=risk
                PATH=/usr/oracle/bin:/usr/oracle/OPatch:/usr/local/bin:/usr/ucb:/bin:/usr/bin:/usr/oracle/bin:/home/oracle/bin:/home/oracle/env:/etc:/opt/boksm/bin
                PS1=\[\033[32m\]\\[\033[00m\]@\[\033[31m\]server789: \[\033[36m\]\w \[\033[00m\]
                PWD=/home/oracle
                SHELL=/bin/ksh
                SHLVL=1
                SSH_CLIENT=179.29.187.45 1934 22
                SSH_CONNECTION=179.29.187.45 1934 169.81.168.77 22
                SSH_TTY=/dev/pts/2
                TERM=xterm
                TMOUT=1800
                TTY=/dev/pts/2
                TZ=EST5EDT
                USER=oracle
                VISUAL=/usr/bin/vi
                _=/bin/env
                {code}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                • 5. Re: ORA-27102: out of memory
                  sb92075
                  ORA-00600/ORA-07445/ORA-03113 = Oracle bug => search on Metalink and/or call Oracle support
                  • 6. Re: ORA-27102: out of memory
                    mbobak
                    It's possible that, when the database crashed, the shared memory segment remained "stuck" in memory. And, now that you're trying to startup the database, you're getting an out of memory error, because the memory is still consumed by that orphaned shared memory segment.

                    I'm not an AIX guy at all, but, most flavors of Unix, hopefully including AIX, will have a command, 'icps'.

                    What do you see if you execute 'ipcs -m'?

                    Does it list any shared memory memory segments owned by 'oracle'? Are there any other Oracle instances running on this server? If there are no other Oracle instances running on this server, and you see shared memory segments owned by 'oracle', then these are orphaned, and need to be removed. You can use the 'ipcrm' command to remove shared memory segments.

                    Hope that helps,

                    -Mark
                    • 7. Re: ORA-27102: out of memory
                      690963
                      ASM instance is already running on this server and here is the output of ipcs -m command.
                      IPC status from /dev/mem as of Sun Nov 21 20:54:35 EST 2010
                      T        ID     KEY        MODE       OWNER    GROUP
                      Shared Memory:
                      m   1048576 0x4d4e5251 --rw-r--r--     root   system
                      m   1048577 0xffffffff --rw-rw----     root   system
                      m   1048578 0x78000005 --rw-rw-rw-     root   system
                      m         3 0x55315352 --rw-rw-rw-     root   system
                      m         4 0x44525354 --rw-r--r--     root   system
                      m         5 0x53494152 --rw-r--r--     root   system
                      m         6 0x00005643 --rw-rw-rw-     root   system
                      m         7 0x00005654 --rw-rw-rw-     root   system
                      m         9 0xffffffff --rw-rw----     root   system
                      m 297796192 0x78000383 --rw-rw-rw-     root   system
                      m 295699246 0xdcd7d2d4 --rw-rw----   oracle oinstall
                      Should i still run ipcrm command?
                      • 8. Re: ORA-27102: out of memory
                        690963
                        I just shutdown the ASM instance. And ran ipcs -m command. This time it wasnt showing anything for oracle. I still ran iprcm command. Database instance wont start.
                        • 9. Re: ORA-27102: out of memory
                          sb92075
                          I just shutdown the ASM instance.
                          Does the DB require ASM to be available?
                          Database instance wont start.
                          What error?
                          • 10. Re: ORA-27102: out of memory
                            Taral
                            What are your OS memory and what is oracle parameters you had set
                            • 11. Re: ORA-27102: out of memory
                              811569
                              Hi,

                              Check out this metalink note - 1080317.1
                              It describes a summary of issues/points that we have to check when we encounter the said error

                              Regards
                              Shamseer
                              • 12. Re: ORA-27102: out of memory
                                CKPT
                                Hi,
                                ORA-07445: exception encountered: core dump [kocedc()+0020] [SIGSEGV] [ADDR:0x4C00012C4800005E] [PC:0x1005A82A0] [Address not mapped to object] []
                                ORA-00600: internal error code, arguments: [kghfrmrg:prv], [0x700000166F2FE88
                                Please check *ORA-600 [kghfrmrg:prv] [ID 370926.1]* but his is already fixed in 11.1.0.6 and your db version is 11.1.0.7..
                                better if you submit alert logfile and generated trace file information to ORACLE support

                                Thanks