6 Replies Latest reply: Apr 9, 2014 8:41 AM by Boochi RSS

    we have 4 node RAC, instance 4 crashed for the third time with this error

    1db54947-8f33-44bb-afad-aed2f0ba857d

      WARNING: ASM communication error: op 17 state 0x50 (3113)

      ERROR: slave communication error with ASM

      Wed Apr 02 00:13:24 2014

      NOTE: ASMB terminating

      Errors in file /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_asmb_60751928.trc:

      ORA-15064: communication failure with ASM instance

      ORA-03135: connection lost contact

      Process ID:

      Session ID: 112 Serial number: 207

      Errors in file /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_asmb_60751928.trc:

      ORA-15064: communication failure with ASM instance

      ORA-03135: connection lost contact

      Process ID:

      Session ID: 112 Serial number: 207

      ASMB (ospid: 60751928): terminating the instance due to error 15064

      Wed Apr 02 00:13:25 2014

      System state dump requested by (instance=4, osid=60751928 (ASMB)), summary=[abnormal instance termination].

      System State dumped to trace file /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_diag_45416516.trc

      Wed Apr 02 00:13:26 2014

      ORA-1092 : opitsk aborting process

      Wed Apr 02 00:13:26 2014

      License high water mark = 449

      Instance terminated by ASMB, pid = 60751928

      USER (ospid: 27197736): terminating the instance

      Instance terminated by USER, pid = 27197736

        • 1. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
          hmartinezlopez

          Can you post the ASM instance alert log?

           

           

          Hector

          @hmartinezlopez

          • 2. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
            1db54947-8f33-44bb-afad-aed2f0ba857d

            ADR Home = /opt/oracle/app/grid/diag/asm/+asm/+ASM4:

            *************************************************************************

            2014-04-06 16:05:16.016000 +03:00

            Time drift detected. Please check VKTM trace file for more details.

            2014-04-06 22:09:06.174000 +03:00

            WARNING: client [prod4:prod] not responsive for 207s; state=0x1. killing pid 26870076

            2014-04-06 22:10:07.138000 +03:00

            WARNING: client [prod4:prod] cleanup delayed; waited 268s, pid 26870076 mbr 0x1

            2014-04-06 22:11:07.140000 +03:00

            WARNING: client [prod4:prod] cleanup delayed; waited 328s, pid 26870076 mbr 0x1

            2014-04-06 22:12:07.860000 +03:00

            NOTE: client spdb4:spdb registered, osid 17170736, mbr 0x1

            • 3. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
              1db54947-8f33-44bb-afad-aed2f0ba857d

              this is the error comes from db alert:

               

              2014-04-06 22:11:27.988000 +03:00

              opiodr aborting process unknown ospid (41877680) as a result of ORA-609

              opiodr aborting process unknown ospid (26870140) as a result of ORA-609

              Errors in file /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_asmb_25297090.trc:

              ORA-15064: communication failure with ASM instance

              ORA-03135: connection lost contact

              Process ID:

              Session ID: 115 Serial number: 55382

              Errors in file /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_asmb_25297090.trc:

              ORA-15064: communication failure with ASM instance

              ORA-03135: connection lost contact

              Process ID:

              Session ID: 115 Serial number: 55382

              opiodr aborting process unknown ospid (13041744) as a result of ORA-1092

              opiodr aborting process unknown ospid (15139146) as a result of ORA-1092

              opiodr aborting process unknown ospid (57212956) as a result of ORA-1092

              opiodr aborting process unknown ospid (52887582) as a result of ORA-1092

              ORA-1092 : opitsk aborting process

              opiodr aborting process unknown ospid (9765170) as a result of ORA-1092

              ORA-1092 : opitsk aborting process

              2014-04-06 22:11:29.760000 +03:00

              ORA-1092 : opitsk aborting process

              • 4. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
                hmartinezlopez

                Which version are you using?

                 

                 

                There are some bugs on 11.2 abut this error, there are no more ORA related? what about

                 

                /opt/oracle/app/diag/rdbms/prod/prod4/trace/prod4_asmb_60751928.trc

                 

                 

                 

                 

                 

                Hector

                @hmartinezlopez

                • 5. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
                  1db54947-8f33-44bb-afad-aed2f0ba857d

                  Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                  With the Partitioning, Real Application Clusters and Real Application Testing options
                  ORACLE_HOME = /opt/oracle/app/product/11.2.0/dbhome_1
                  System name:    AIX
                  Node name:      nod4
                  Release:        1
                  Version:        7
                  Machine:        00F829604C00
                  Instance name: prod4
                  Redo thread mounted by this instance: 0 <none>
                  Oracle process number: 25
                  Unix process pid: 60751928, image: oracle@node4 (ASMB)


                  *** 2014-03-30 21:30:29.791
                  *** SESSION ID:(255.1) 2014-03-30 21:30:29.791
                  *** CLIENT ID:() 2014-03-30 21:30:29.791
                  *** SERVICE NAME:() 2014-03-30 21:30:29.791
                  *** MODULE NAME:() 2014-03-30 21:30:29.791
                  *** ACTION NAME:() 2014-03-30 21:30:29.791

                  NOTE: initiating MARK startup

                  *** 2014-04-02 00:13:24.701
                  NOTE: ASMB terminating
                  ORA-15064: communication failure with ASM instance
                  ORA-03135: connection lost contact
                  Process ID:
                  Session ID: 112 Serial number: 207
                  error 15064 detected in background process
                  ORA-15064: communication failure with ASM instance
                  ORA-03135: connection lost contact
                  Process ID:
                  Session ID: 112 Serial number: 207
                  kjzduptcctx: Notifying DIAG for crash event
                  ----- Abridged Call Stack Trace -----
                  ksedsts()+360<-kjzdssdmp()+240<-kjzduptcctx()+228<-kjzdicrshnfy()+100<-ksuitm()+5124<-ksbrdp()+4508<-opirip()+1624<-opidrv()+608<-sou2o()+136<-opimai_real()+188<-ssthrdmain()+268<-main()+204<-__start()+112
                  ----- End of Abridged Call Stack Trace -----

                  *** 2014-04-02 00:13:24.797
                  ASMB (ospid: 60751928): terminating the instance due to error 15064
                  ksuitm: waiting up to [5] seconds before killing DIAG(45416516)

                  • 6. Re: we have 4 node RAC, instance 4 crashed for the third time with this error
                    Boochi

                    See the Note: Database Instance Crashes with ORA-15064 ORA-03135 ORA-00240 on 11.2 (Doc ID 1487108.1)