8 Replies Latest reply: Dec 11, 2012 1:31 AM by Osama_Mustafa RSS

    Error on RAC node

    The Explorer
      Hi All,

      I have Oracle database 10g Release 2 running on Windows server 2003 with a 2-node cluster.

      I have received the following error on one of RAC nodes.

      Error: Controlfile sequence number in file header is different from the one in memory
      > Please check that the correct mount optionsare used if controlfile is located on NFS


      Alert Log shows

      Mon Dec 10 02:32:37 2012
      Errors in file d:\oracle\admin\morc\bdump\morc2_arc0_6112.trc:
      ORA-00600: internal error code, arguments: [kccchb_3], [1579638], [1579642], [1579638], [], [], [], []


      Any help on this is highly appreciated.

      Thanks
        • 1. Re: Error on RAC node
          CKPT
          Hi,

          What is your full 4 digit version?
          use error lookup tool ORA-600/ORA-7445/ORA-700 Error Look-up Tool [ID 153788.1] to find listed bugs if there are no listed bugs then raise an SR with oracle support with alert and trace files, also take a look below note. RAC instance using NFS via Netapps report Ora-600 [kccchb_3] [ID 303986.1]
          • 2. Re: Error on RAC node
            Osama_Mustafa
            Like CKPT said , Follow the below steps :

            -Go to https://support.oracle.com
            -Document number : ORA-600/ORA-7445/ORA-700 Error Look-up Tool [ID 153788.1]
            -Fill database version , and error type .
            -Nothing Found Open SR
            • 3. Re: Error on RAC node
              The Explorer
              I have found the following note on Oracle Knowledge Base :

              _Production Database Is Not Coming Up due to ORA-235 [ID 1266376.1]_

              Suppose when I restart the service on 2nd node, will it recover and get in sync with the node 1 which is running fine.

              Thanks
              • 4. Re: Error on RAC node
                Osama_Mustafa
                The Document Made for 11.1 and later you have 10g , if you don't anything related to your version i believe you should open SR
                • 5. Re: Error on RAC node
                  The Explorer
                  In my case, the second node on RAC usually gets restarted. I understand that there is a heartbeat process, wherein the nodes checks each other's presence.

                  However, the disconnections are so frequent that I am unable to understand whether the heartbeat is failing due to network connection or any other case. Checking alert log and CRS log doesn't reveal any root cause.

                  Any help on this is highly appreciated.

                  Thanks
                  • 6. Re: Error on RAC node
                    Osama_Mustafa
                    did you check CRS logs please , Alertlog in CRS Folder Please post 50 lines
                    • 7. Re: Error on RAC node
                      The Explorer
                      Alert Log
                      Tue Dec 11 08:07:33 2012
                      Reconfiguration started (old inc 12, new inc 14)
                      List of nodes:
                      0
                      Global Resource Directory frozen
                      * dead instance detected - domain 0 invalid = TRUE
                      Communication channels reestablished
                      Master broadcasted resource hash value bitmaps
                      Non-local Process blocks cleaned out
                      Tue Dec 11 08:07:33 2012
                      LMS 1: 6 GCS shadows cancelled, 3 closed
                      Tue Dec 11 08:07:34 2012
                      LMS 0: 3 GCS shadows cancelled, 0 closed
                      Set master node info
                      Submitted all remote-enqueue requests
                      Dwn-cvts replayed, VALBLKs dubious
                      All grantable enqueues granted
                      Post SMON to start 1st pass IR
                      Tue Dec 11 08:07:36 2012
                      LMS 0: 361580 GCS shadows traversed, 0 replayed
                      Tue Dec 11 08:07:36 2012
                      LMS 1: 363922 GCS shadows traversed, 0 replayed
                      Tue Dec 11 08:07:36 2012
                      Submitted all GCS remote-cache requests
                      Post SMON to start 1st pass IR
                      Fix write in gcs resources
                      Tue Dec 11 08:07:36 2012
                      Instance recovery: looking for dead threads
                      Tue Dec 11 08:07:36 2012
                      Beginning instance recovery of 1 threads
                      Reconfiguration complete
                      Tue Dec 11 08:07:38 2012
                      parallel recovery started with 7 processes
                      Tue Dec 11 08:07:38 2012
                      Started redo scan
                      Tue Dec 11 08:07:38 2012
                      Completed redo scan
                      4008 redo blocks read, 766 data blocks need recovery
                      Tue Dec 11 08:07:38 2012
                      Started redo application at
                      Thread 2: logseq 7916, block 48159
                      Tue Dec 11 08:07:39 2012
                      Recovery of Online Redo Log: Thread 2 Group 4 Seq 7916 Reading mem 0
                      Mem# 0 errs 0: M:\ORC\REDO04.LOG
                      Tue Dec 11 08:07:39 2012
                      Completed redo application
                      Tue Dec 11 08:07:39 2012
                      Completed instance recovery at
                      Thread 2: logseq 7916, block 52167, scn 5966110632827
                      746 data blocks read, 785 data blocks written, 4008 redo blocks read
                      Switch log for thread 2 to sequence 7917
                      Tue Dec 11 08:30:03 2012
                      Reconfiguration started (old inc 14, new inc 16)
                      List of nodes:
                      0 1
                      Global Resource Directory frozen
                      Communication channels reestablished
                      Tue Dec 11 08:30:16 2012
                      Master broadcasted resource hash value bitmaps
                      Non-local Process blocks cleaned out
                      Tue Dec 11 08:30:16 2012
                      LMS 0: 0 GCS shadows cancelled, 0 closed
                      Tue Dec 11 08:30:17 2012
                      LMS 1: 0 GCS shadows cancelled, 0 closed
                      Set master node info
                      Submitted all remote-enqueue requests
                      Dwn-cvts replayed, VALBLKs dubious
                      All grantable enqueues granted
                      Tue Dec 11 08:30:18 2012
                      LMS 1: 11204 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:18 2012
                      LMS 0: 11283 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:18 2012
                      LMS 1: 11300 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:18 2012
                      LMS 0: 11328 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:18 2012
                      LMS 1: 11122 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:18 2012
                      LMS 0: 11292 GCS shadows traversed, 4001 replayed
                      LMS 0: 11320 GCS shadows traversed, 4001 replayed
                      LMS 0: 11313 GCS shadows traversed, 4001 replayed
                      LMS 0: 11298 GCS shadows traversed, 4001 replayed
                      LMS 0: 11384 GCS shadows traversed, 4001 replayed
                      LMS 0: 11182 GCS shadows traversed, 4001 replayed
                      LMS 0: 11279 GCS shadows traversed, 4001 replayed
                      LMS 0: 11429 GCS shadows traversed, 4001 replayed
                      LMS 0: 11187 GCS shadows traversed, 4001 replayed
                      LMS 0: 11336 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:19 2012
                      LMS 1: 11031 GCS shadows traversed, 4001 replayed
                      LMS 1: 11207 GCS shadows traversed, 4001 replayed
                      LMS 1: 11170 GCS shadows traversed, 4001 replayed
                      LMS 1: 11293 GCS shadows traversed, 4001 replayed
                      LMS 1: 11236 GCS shadows traversed, 4001 replayed
                      LMS 1: 11202 GCS shadows traversed, 4001 replayed
                      LMS 1: 11154 GCS shadows traversed, 4001 replayed
                      LMS 1: 11200 GCS shadows traversed, 4001 replayed
                      Tue Dec 11 08:30:21 2012
                      LMS 0: 11447 GCS shadows traversed, 4001 replayed
                      LMS 0: 11235 GCS shadows traversed, 4001 replayed
                      LMS 0: 11270 GCS shadows traversed, 4001 replayed
                      LMS 0: 11192 GCS shadows traversed, 4001 replayed
                      LMS 0: 11496 GCS shadows traversed, 4001 replayed
                      LMS 0: 11337 GCS shadows traversed, 4001 replayed
                      LMS 0: 11289 GCS shadows traversed, 4001 replayed
                      LMS 0: 11286 GCS shadows traversed, 4001 replayed
                      LMS 0: 11455 GCS shadows traversed, 4001 replayed
                      LMS 0: 11196 GCS shadows traversed, 4001 replayed
                      LMS 0: 11247 GCS shadows traversed, 4001 replayed
                      LMS 0: 11416 GCS shadows traversed, 4001 replayed
                      LMS 0: 11428 GCS shadows traversed, 4001 replayed
                      LMS 0: 11318 GCS shadows traversed, 4001 replayed
                      LMS 0: 11231 GCS shadows traversed, 4001 replayed
                      LMS 0: 11279 GCS shadows traversed, 4001 replayed
                      LMS 0: 11352 GCS shadows traversed, 4001 replayed
                      LMS 0: 11378 GCS shadows traversed, 4001 replayed
                      LMS 0: 1509 GCS shadows traversed, 552 replayed
                      Tue Dec 11 08:30:22 2012
                      LMS 1: 11139 GCS shadows traversed, 4001 replayed
                      LMS 1: 11352 GCS shadows traversed, 4001 replayed
                      LMS 1: 11199 GCS shadows traversed, 4001 replayed
                      LMS 1: 11181 GCS shadows traversed, 4001 replayed
                      LMS 1: 11148 GCS shadows traversed, 4001 replayed
                      LMS 1: 11100 GCS shadows traversed, 4001 replayed
                      LMS 1: 11345 GCS shadows traversed, 4001 replayed
                      LMS 1: 11101 GCS shadows traversed, 4001 replayed
                      LMS 1: 11100 GCS shadows traversed, 4001 replayed
                      LMS 1: 11277 GCS shadows traversed, 4001 replayed
                      LMS 1: 11215 GCS shadows traversed, 4001 replayed
                      LMS 1: 11203 GCS shadows traversed, 4001 replayed
                      LMS 1: 11216 GCS shadows traversed, 4001 replayed
                      LMS 1: 11264 GCS shadows traversed, 4001 replayed
                      LMS 1: 11226 GCS shadows traversed, 4001 replayed
                      LMS 1: 11203 GCS shadows traversed, 4001 replayed
                      LMS 1: 11125 GCS shadows traversed, 4001 replayed
                      LMS 1: 11286 GCS shadows traversed, 4001 replayed
                      LMS 1: 11195 GCS shadows traversed, 4001 replayed
                      LMS 1: 10937 GCS shadows traversed, 3903 replayed
                      Tue Dec 11 08:30:23 2012
                      Submitted all GCS remote-cache requests
                      Post SMON to start 1st pass IR
                      Fix write in gcs resources
                      Reconfiguration complete
                      Tue Dec 11 08:47:09 2012
                      Thread 1 advanced to log sequence 9420
                      Current log# 1 seq# 9420 mem# 0: M:\ORC\REDO01.LOG
                      Tue Dec 11 09:05:02 2012
                      Reconfiguration started (old inc 16, new inc 18)
                      List of nodes:
                      0
                      Tue Dec 11 09:05:28 2012
                      Global Resource Directory frozen
                      * dead instance detected - domain 0 invalid = TRUE
                      Communication channels reestablished
                      Master broadcasted resource hash value bitmaps
                      Non-local Process blocks cleaned out
                      Tue Dec 11 09:05:28 2012
                      LMS 1: 5 GCS shadows cancelled, 2 closed
                      Tue Dec 11 09:05:29 2012
                      LMS 0: 3 GCS shadows cancelled, 0 closed
                      Set master node info
                      Submitted all remote-enqueue requests
                      Dwn-cvts replayed, VALBLKs dubious
                      All grantable enqueues granted
                      Post SMON to start 1st pass IR
                      Tue Dec 11 09:05:31 2012
                      LMS 0: 386379 GCS shadows traversed, 0 replayed
                      Tue Dec 11 09:05:31 2012
                      LMS 1: 393515 GCS shadows traversed, 0 replayed
                      Tue Dec 11 09:05:31 2012
                      Submitted all GCS remote-cache requests
                      Post SMON to start 1st pass IR
                      Fix write in gcs resources
                      Tue Dec 11 09:05:31 2012
                      Instance recovery: looking for dead threads
                      Tue Dec 11 09:05:31 2012
                      Beginning instance recovery of 1 threads
                      Reconfiguration complete
                      Tue Dec 11 09:05:34 2012
                      parallel recovery started with 7 processes
                      Tue Dec 11 09:05:34 2012
                      Started redo scan
                      Tue Dec 11 09:05:35 2012
                      Completed redo scan
                      6460 redo blocks read, 1499 data blocks need recovery
                      Tue Dec 11 09:05:35 2012
                      Started redo application at
                      Thread 2: logseq 7917, block 16997
                      Tue Dec 11 09:05:36 2012
                      Recovery of Online Redo Log: Thread 2 Group 3 Seq 7917 Reading mem 0
                      Mem# 0 errs 0: M:\ORC\REDO03.LOG
                      Tue Dec 11 09:05:36 2012
                      Completed redo application
                      Tue Dec 11 09:05:36 2012
                      Completed instance recovery at
                      Thread 2: logseq 7917, block 23457, scn 5966110693995
                      1267 data blocks read, 1532 data blocks written, 6460 redo blocks read
                      Switch log for thread 2 to sequence 7918
                      CRS Alert Log
                      2012-12-09 08:04:32.545
                      [crsd(9168)]CRS-1204:Recovering CRS resources for node mdb2.
                      2012-12-09 12:07:11.731
                      [cssd(1932)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
                      2012-12-10 02:31:30.428
                      [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
                      2012-12-10 02:31:30.506
                      [cssd(8708)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
                      2012-12-10 02:31:34.272
                      [crsd(7540)]CRS-1204:Recovering CRS resources for node mdb2.
                      2012-12-10 22:42:03.651
                      [cssd(8868)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
                      2012-12-10 23:58:58.617
                      [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
                      2012-12-10 23:59:00.648
                      [cssd(8512)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
                      2012-12-10 23:59:07.101
                      [crsd(9624)]CRS-1204:Recovering CRS resources for node mdb2.
                      2012-12-11 07:47:41.587
                      [cssd(9328)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
                      2012-12-11 08:07:27.587
                      [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
                      2012-12-11 08:07:29.634
                      [cssd(4552)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
                      2012-12-11 08:07:36.541
                      [crsd(9144)]CRS-1204:Recovering CRS resources for node mdb2.
                      2012-12-11 08:29:35.192
                      [cssd(9656)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
                      2012-12-11 09:04:57.130
                      [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
                      2012-12-11 09:04:59.193
                      [cssd(9468)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
                      2012-12-11 09:05:05.303
                      [crsd(5452)]CRS-1204:Recovering CRS resources for node mdb2.


                      Thanks
                      • 8. Re: Error on RAC node
                        Osama_Mustafa
                        Post crs_stat -t


                        also review this thread
                        10g RAC on AIX node eviction - HELP