2 Replies Latest reply: Apr 30, 2013 11:04 AM by Santysharma-Oracle RSS

    WARNING: ASM communication error: op 0 state 0x0

    user12996279
      Hi,

      Oracle 11.2.0.3 Enterprise edition
      Windows 2008R2 64bit

      I'm trying to troubleshoot a random problem that I get in my Oracle RAC (see alert log snippet below).

      Wed Aug 29 12:04:09 2012
      ORA-25253 encountered when generating server alert SMG-3503
      Wed Aug 29 12:05:10 2012
      ORA-25253 encountered when generating server alert SMG-3503
      Wed Aug 29 12:09:38 2012
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (504) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (9856) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (9388) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (7320) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (9360) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (8496) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (4732) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (8936) as a result of ORA-609
      Wed Aug 29 12:12:22 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:22
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (9316) as a result of ORA-609
      Wed Aug 29 12:12:23 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:23
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (9768) as a result of ORA-609
      Wed Aug 29 12:12:23 2012


      ***********************************************************************

      Fatal NI connect error 12537, connecting to:
      (LOCAL=NO)

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
           Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:12:23
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12537

      TNS-12537: TNS:connection closed
      ns secondary err code: 12560
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0
      opiodr aborting process unknown ospid (5848) as a result of ORA-609
      Wed Aug 29 12:24:41 2012
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      WARNING: ASM communication error: op 0 state 0x0 (15055)
      ERROR: direct connection failure with ASM
      Wed Aug 29 12:31:42 2012
      Thread 2 advanced to log sequence 2813 (LGWR switch)
      Current log# 3 seq# 2813 mem# 0: +REDO/qcare00/onlinelog/group_3.259.780247037
      Wed Aug 29 12:31:42 2012
      Archived Log entry 6684 added for thread 2 sequence 2812 ID 0x6cba185b dest 1:
      Wed Aug 29 12:32:18 2012
      ORA-25253 encountered when generating server alert SMG-3503
      Wed Aug 29 12:35:18 2012
      ORA-25253 encountered when generating server alert SMG-3503
      Wed Aug 29 12:38:15 2012


      ***********************************************************************

      Fatal NI connect error 12638, connecting to:
      (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

      VERSION INFORMATION:
           TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
           Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
      Time: 29-AUG-2012 12:38:15
      Tracing not turned on.
      Tns error struct:
      ns main err code: 12638

      TNS-12638: Credential retrieval failed
      ns secondary err code: 0
      nt main err code: 0
      nt secondary err code: 0
      nt OS err code: 0


      When we get the error this is the status of the cluster:

      Microsoft Windows [Version 6.1.7601]
      Copyright (c) 2009 Microsoft Corporation. All rights reserved.

      C:\Windows\system32>crsctl stat res -t
      --------------------------------------------------------------------------------
      NAME TARGET STATE SERVER STATE_DETAILS
      --------------------------------------------------------------------------------
      Local Resources
      --------------------------------------------------------------------------------
      ora.DATA.dg
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.FRA.dg
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.GRID.dg
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.LISTENER.lsnr
      ONLINE ONLINE s0060
      ONLINE INTERMEDIATE s0061 CHECK TIMED OUT
      ora.REDO.dg
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.asm
      ONLINE ONLINE s0060 Started
      ONLINE ONLINE s0061 Started
      ora.gsd
      OFFLINE OFFLINE s0060
      OFFLINE OFFLINE s0061
      ora.net1.network
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.ons
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      ora.registry.acfs
      ONLINE ONLINE s0060
      ONLINE ONLINE s0061
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.LISTENER_SCAN1.lsnr
      1 ONLINE ONLINE s0060
      ora.LISTENER_SCAN2.lsnr
      1 ONLINE INTERMEDIATE s0061 CHECK TIMED OUT
      ora.cvu
      1 ONLINE ONLINE s0061
      ora.s0060.vip
      1 ONLINE ONLINE s0060
      ora.s0061.vip
      1 ONLINE ONLINE s0061
      ora.oc4j
      1 ONLINE OFFLINE
      ora.qc00.db
      1 ONLINE ONLINE s0060 Open
      2 ONLINE ONLINE s0061 Open
      ora.qc00.qsc00.svc
      2 ONLINE ONLINE s0060
      ora.qc01.db
      1 ONLINE ONLINE s0060 Open
      2 ONLINE ONLINE s0061 Open
      ora.qc01.qsc01.svc
      2 ONLINE ONLINE s0061
      ora.scan1.vip
      1 ONLINE ONLINE s0060
      ora.scan2.vip
      1 ONLINE ONLINE s0061

      As soon as we reboot the node with problems the database is working again.

      Thanks in advace.

      Edited by: user12996279 on 29-ago-2012 7:48