4 Replies Latest reply: Jan 28, 2013 1:01 PM by Alan3 RSS

    Coredumps and instance terminated?

    Alan3
      Database is 11.2.0.1 64bit running on physical Oracle Linux R5U8.
      This is a test database that has been running without issue for quite some time.
      Recently moved to new hardware and has been running ok until this past weekend...
      On Friday I enabled archivelog mode for running routine hot backups.
      Early Monday morning we got the first coredump.
      Another Monday afternoon, then again Tuesday & Wednesday early morning.
      I was suspicious of the logs being too small (3 @ 50MB ea) so Wednesday I raised them to 6 @ 100MB each (well more than it should need...)
      Seemed to work ok as we made it through Wednesday & Thursday without incident, but it crashed again Thursday evening... same messages each time.
      Looking at the alert it appears to be tied to one of our test queues, but not sure if that is a culprit or just a result of the crash.
      I have no other alerts or error messages - this is it, so I'm at a bit of a loss where this is coming from.
      Haven't been able to find much anything on the particular error...
      Hoped someone here might have some idea...
      Alert log excerpt follows....
      ----------------------------------------------
      Archived Log entry 40117 added for thread 1 sequence 54083 ID 0x26a060e dest 1:
      Thu Dec 20 17:11:20 2012
      Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9] [PC:0x4375C3D, kwqdlClnupCcoDl()+113] [flags: 0x0, count: 1]
      Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_ora_8278.trc (incident=135758):
      ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
      ORA-25228: timeout or end-of-fetch during message dequeue from HCBPELAQ.IP_IN_QUEUE
      Incident details in: /home/oracle/diag/rdbms/devbpel/devbpel/incident/incdir_135758/devbpel_ora_8278_i135758.trc
      Thu Dec 20 17:11:25 2012
      Trace dumping is performing id=[cdmp_20121220171125]
      Thu Dec 20 17:11:26 2012
      Sweep [inc][135758]: completed
      Sweep [inc2][135758]: completed
      Thu Dec 20 17:12:24 2012
      Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9] [PC:0x4375C3D, kwqdlClnupCcoDl()+113] [flags: 0x0, count: 1]
      Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_pmon_7876.trc (incident=135350):
      ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
      Incident details in: /home/oracle/diag/rdbms/devbpel/devbpel/incident/incdir_135350/devbpel_pmon_7876_i135350.trc
      Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_pmon_7876.trc:
      ORA-00602: internal programming exception
      ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
      PMON (ospid: 7876): terminating the instance due to error 602
      Instance terminated by PMON, pid = 7876
        • 1. Re: Coredumps and instance terminated?
          Alan3
          Trace file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_ora_8278.trc
          Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options ORACLE_HOME = /home/oracle/11g/db
          System name:     Linux
          Node name:     devbpel
          Release:     2.6.32-300.10.1.el5uek
          Version:     #1 SMP Wed Feb 22 17:37:40 EST 2012
          Machine:     x86_64
          Instance name: devbpel
          Redo thread mounted by this instance: 1
          Oracle process number: 53
          Unix process pid: 8278, image: oracle@devbpel

          *** 2012-12-20 13:43:32.279
          *** SESSION ID:(331.9) 2012-12-20 13:43:32.279
          *** CLIENT ID:() 2012-12-20 13:43:32.279
          *** SERVICE NAME:(SYS$USERS) 2012-12-20 13:43:32.279
          *** MODULE NAME:() 2012-12-20 13:43:32.279
          *** ACTION NAME:() 2012-12-20 13:43:32.279

          kwqdlInsDeqLog during stmt prepare: retval -1, errnum 942, errbuf ORA-00942: table or view does not exist

          kwqdlCrtDeqLog during stmt execute: retval 100, errnum 1403, errbuf ORA-01403: no data found
          ORA-06512: at "SYS.DBMS_AQADM_SYS", line 11938
          ORA-06512: at line 1

          *** 2012-12-20 17:11:20.928
          Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9] [PC:0x4375C3D, kwqdlClnupCcoDl()+113] [flags: 0x0, count: 1] Incident 135758 created, dump file: /home/oracle/diag/rdbms/devbpel/devbpel/incident/incdir_135758/devbpel_ora_8278_i135758.trc
          ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
          ORA-25228: timeout or end-of-fetch during message dequeue from HCBPELAQ.IP_IN_QUEUE

          ssexhd: crashing the process...
          Shadow_Core_Dump = partial
          • 2. Re: Coredumps and instance terminated?
            P.Forstmann
            You need to contact Oracle Support because you are hitting some bugs:
            $ oerr ora 7445
            07445, 00000, "exception encountered: core dump [%s] [%s] [%s] [%s] [%s] [%s]"
            // *Cause: An OS exception occurred which should result in the creation of a
            //         core file.  This is an internal error.
            // *Action: Contact your customer support representative.
            $ oerr ora 602
            00602, 00000, "internal programming exception"
            // *Cause:  Internal programming exception
            // *Action: Report as bug
            $
            Note also that Database 11.2.0.1 Error Correction Support has ended Sept. 2011: see https://blogs.oracle.com/stevenChan/entry/reminder_database_11_2_0.
            • 3. Re: Coredumps and instance terminated?
              Alan3
              I finally discovered...

              AQ_TM_PROCESSES parameter must not be zero when using AQ.

              According to the notes, it should be null - which my spfile was set as null, but the running db parameter showed zero...

              I set the parameter to 4 (without a db restart) and no issue since.

              So - moral of the story (which I should know better) - always check the db parameter of the running db - eventhough your init file parameters are set...
              • 4. Re: Coredumps and instance terminated?
                Alan3
                One more update to this...
                While setting the AQ_TM_PROCESSES helped, it still didn't solve the issue as I still encountered the same coredumps and terminations, although not nearly as frequent.
                I ran across several listed bugs with AQ in 11.2.0.1..
                Finally upgraded to 11.2.0.3...
                It's been several days now without another incident...