6 Replies Latest reply: Nov 27, 2010 11:57 PM by 774334 RSS

    ORA-29701: unable to connect to Cluster Manager

    774334
      Dear all,

      i am new dba,i have 10g RAC db i have found error in alertlog files from my Rac database in my second node, but my db is working, pls give the solution

      pls see below for my error....

      Sat Nov 27 01:00:02 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5671.trc:
      ORA-00206: error in writing (block 956, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 956, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:05 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5682.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:05 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5691.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:05 2010
      ALTER SYSTEM ARCHIVE LOG
      Sat Nov 27 01:00:05 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5671.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:05 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5671.trc:
      ORA-00221: error on write to control file
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:05 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5671.trc:
      ORA-00221: error on write to control file
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:07 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5682.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      SVR4 Error: 1310732408: Unknown system error
      Additional information: 912603192
      Additional information: 758151216
      Additional information: 1698183269
      Sat Nov 27 01:00:07 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5691.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      Sat Nov 27 01:00:08 2010
      Thread 2 advanced to log sequence 2239 (LGWR switch)
      Current log# 3 seq# 2239 mem# 0: +DG_DATA/eregdb/onlinelog/group_3.266.4170352
      41
      Current log# 3 seq# 2239 mem# 1: +DG_FRA/eregdb/onlinelog/group_3.260.41703524
      3
      Sat Nov 27 01:00:09 2010
      Errors in file /u01/app/oracle/admin/eregdb/udump/eregdb2_ora_5682.trc:
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
      ORA-29701: unable to connect to Cluster Manager
      ORA-00206: error in writing (block 41, # blocks 1) of control file
      ORA-00202: control file: '+DG_DATA/eregdb/controlfile/current.260.417035139'
      ORA-29701: unable to connect to Cluster Manager
      SVR4 Error: 1310732408: Unknown system error
      Additional information: 912603192
      Additional information: 758151216
      Additional information: 1698183269
      Sat Nov 27 03:25:30 2010
      Thread 2 advanced to log sequence 2240 (LGWR switch)
      Current log# 4 seq# 2240 mem# 0: +DG_DATA/eregdb/onlinelog/group_4.267.4170352
      43
      Current log# 4 seq# 2240 mem# 1: +DG_FRA/eregdb/onlinelog/group_4.261.41703524
      3
      Sat Nov 27 09:00:17 2010
      Errors in file /u01/app/oracle/admin/eregdb/bdump/eregdb2_ora_11908.trc:
      ORA-01114: IO error writing block to file 7 (block # 106960)
      ORA-29701: unable to connect to Cluster Manager
      ORA-01114: IO error writing block to file 7 (block # 106960)
      ORA-29701: unable to connect to Cluster Manager
      -a $
        • 1. Re: ORA-29701: unable to connect to Cluster Manager
          ora_tech
          Hi,

          Welcome 2 Oracle Forums :)
          ORA-00202: control file: '+DG_FRA/eregdb/controlfile/current.256.417035139'
          make sure that the disk group is mounted (or) you have enogh space available to write on disk group.

          Refer
          OERR: ORA 206 error in writing control file block , # blocks
          Error:  ORA 206  
          Text:   error in writing control file <name> block <num>, # blocks <num> 
          -------------------------------------------------------------------------------
          Cause:  A disk write-failure occurred while attempting to write to the 
                  specified control file.
                  The block location of the failure is given.
          Action: Check that the disk is online.
                  If it is not, bring it online and shut down and restart Oracle.
                  If the disk is online, then look for operating system reasons for 
                  Oracle's inability to write to the disk or control file.
                  See the <Oracle7 Server Administrator's Guide> for information on 
                  recovering from the loss of a control file.
                  See also your operating system-specific Oracle documentation.
          thanks,
          X A H E E R
          • 2. Re: ORA-29701: unable to connect to Cluster Manager
            774334
            yes diskgroup mounted and have enough space
            • 3. Re: ORA-29701: unable to connect to Cluster Manager
              ora_tech
              Hi,
              yes diskgroup mounted and have enough space
              can you post the content of trace file */u01/app/oracle/admin/eregdb/bdump/eregdb2_ora_11908.trc*

              thanks,
              X A H E E R
              • 4. Re: ORA-29701: unable to connect to Cluster Manager
                774334
                -a $cat eregdb2_ora_11908.trc
                /u01/app/oracle/admin/eregdb/bdump/eregdb2_ora_11908.trc
                Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
                With the Partitioning, Real Application Clusters, OLAP, Data Mining
                and Real Application Testing options
                ORACLE_HOME = /u01/app/oracle/product/10
                System name: SunOS
                Node name: -a
                Release: 5.10
                Version: Generic_141414-07
                Machine: sun4u
                Instance name: eregdb2
                Redo thread mounted by this instance: 2
                Oracle process number: 71
                Unix process pid: 11908, image: oracle@-a

                Warning: keltnfy call to ldmInit failed with error 46
                *** ACTION NAME:(Remote-Flush Slave Action) 2010-11-27 09:00:14.291
                *** MODULE NAME:(MMON_SLAVE) 2010-11-27 09:00:14.291
                *** SERVICE NAME:(SYS$BACKGROUND) 2010-11-27 09:00:14.291
                *** SESSION ID:(1072.55468) 2010-11-27 09:00:14.291
                kgxgncin: CLSS init failed with status 21
                kjfmsgr: unable to connect to NM for reg in shared group
                *** KEWROCISTMTEXEC - encountered error: (ORA-01114: IO error writing block to f
                ile 7 (block # 72586)
                ORA-29701: unable to connect to Cluster Manager
                ORA-01114: IO error writing block to file 7 (block # 72586)
                ORA-29701: unable to connect to Cluster Manager
                )
                *** SQLSTR: total-len=1777, dump-len=240,
                STR={INSERT /*+ APPEND */ INTO WRH$_ACTIVE_SESSION_HISTORY ( snap_id, dbid
                , instance_number, sample_id,    sample_time, session_id, session_serial#, user_
                id,    sql_id, sql_child_number,    sql_plan_hash_value, force_matching_signatur
                e, service_}
                kewa_flusher: Exception handler 13509 - Will resignal!
                *** KEWRAFM1: Error=13509 encountered by kewrfteh
                kgxgncin: CLSS init failed with status 21
                kjfmsgr: unable to connect to NM for reg in shared group
                *** 2010-11-27 09:00:17.897
                ksedmp: internal or fatal error
                ORA-01114: IO error writing block to file 7 (block # 106960)
                ORA-29701: unable to connect to Cluster Manager
                ORA-01114: IO error writing block to file 7 (block # 106960)
                ORA-29701: unable to connect to Cluster Manager
                *** KEWROCISTMTEXEC - encountered error: (ORA-01114: IO error writing block to f
                ile 7 (block # 106960)
                ORA-29701: unable to connect to Cluster Manager
                )
                *** SQLSTR: total-len=1887, dump-len=240,
                STR={INSERT INTO wrh$_sql_plan sp         (snap_id, dbid, sql_id, plan_has
                h_value, id,          operation, options, object_node, object#, object_owner,
                       object_name, object_alias, object_type,          optimizer, parent_id, de
                pth, positi}
                *** KEWRAFM1: Error=13509 encountered by kewrfteh
                -a $
                • 5. Re: ORA-29701: unable to connect to Cluster Manager
                  ora_tech
                  Hi,

                  check you have enough spaces available in all tablespaces. If yes then try to restart ASM Instance and CRS services.

                  If this also doesn't help i suggest you to log S.R.

                  thanks,
                  X A H E E R
                  • 6. Re: ORA-29701: unable to connect to Cluster Manager
                    774334
                    Dear mr xaheer


                    if iam checking my crs status

                    it shows css daemon failiure...
                    its only for my seconf node


                    ----------------

                    -a $crsctl check crs
                    Failure 1 contacting CSS daemon
                    CRS appears healthy
                    EVM appears healthy
                    -a $

                    regards

                    syed ibrahim