This discussion is archived
8 Replies Latest reply: Dec 10, 2012 11:31 PM by Osama_Mustafa RSS

Error on RAC node

The Explorer Explorer
Currently Being Moderated
Hi All,

I have Oracle database 10g Release 2 running on Windows server 2003 with a 2-node cluster.

I have received the following error on one of RAC nodes.

Error: Controlfile sequence number in file header is different from the one in memory
> Please check that the correct mount optionsare used if controlfile is located on NFS


Alert Log shows

Mon Dec 10 02:32:37 2012
Errors in file d:\oracle\admin\morc\bdump\morc2_arc0_6112.trc:
ORA-00600: internal error code, arguments: [kccchb_3], [1579638], [1579642], [1579638], [], [], [], []


Any help on this is highly appreciated.

Thanks
  • 1. Re: Error on RAC node
    CKPT Guru
    Currently Being Moderated
    Hi,

    What is your full 4 digit version?
    use error lookup tool ORA-600/ORA-7445/ORA-700 Error Look-up Tool [ID 153788.1] to find listed bugs if there are no listed bugs then raise an SR with oracle support with alert and trace files, also take a look below note. RAC instance using NFS via Netapps report Ora-600 [kccchb_3] [ID 303986.1]
  • 2. Re: Error on RAC node
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Like CKPT said , Follow the below steps :

    -Go to https://support.oracle.com
    -Document number : ORA-600/ORA-7445/ORA-700 Error Look-up Tool [ID 153788.1]
    -Fill database version , and error type .
    -Nothing Found Open SR
  • 3. Re: Error on RAC node
    The Explorer Explorer
    Currently Being Moderated
    I have found the following note on Oracle Knowledge Base :

    _Production Database Is Not Coming Up due to ORA-235 [ID 1266376.1]_

    Suppose when I restart the service on 2nd node, will it recover and get in sync with the node 1 which is running fine.

    Thanks
  • 4. Re: Error on RAC node
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    The Document Made for 11.1 and later you have 10g , if you don't anything related to your version i believe you should open SR
  • 5. Re: Error on RAC node
    The Explorer Explorer
    Currently Being Moderated
    In my case, the second node on RAC usually gets restarted. I understand that there is a heartbeat process, wherein the nodes checks each other's presence.

    However, the disconnections are so frequent that I am unable to understand whether the heartbeat is failing due to network connection or any other case. Checking alert log and CRS log doesn't reveal any root cause.

    Any help on this is highly appreciated.

    Thanks
  • 6. Re: Error on RAC node
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    did you check CRS logs please , Alertlog in CRS Folder Please post 50 lines
  • 7. Re: Error on RAC node
    The Explorer Explorer
    Currently Being Moderated
    Alert Log
    Tue Dec 11 08:07:33 2012
    Reconfiguration started (old inc 12, new inc 14)
    List of nodes:
    0
    Global Resource Directory frozen
    * dead instance detected - domain 0 invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Tue Dec 11 08:07:33 2012
    LMS 1: 6 GCS shadows cancelled, 3 closed
    Tue Dec 11 08:07:34 2012
    LMS 0: 3 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Tue Dec 11 08:07:36 2012
    LMS 0: 361580 GCS shadows traversed, 0 replayed
    Tue Dec 11 08:07:36 2012
    LMS 1: 363922 GCS shadows traversed, 0 replayed
    Tue Dec 11 08:07:36 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Tue Dec 11 08:07:36 2012
    Instance recovery: looking for dead threads
    Tue Dec 11 08:07:36 2012
    Beginning instance recovery of 1 threads
    Reconfiguration complete
    Tue Dec 11 08:07:38 2012
    parallel recovery started with 7 processes
    Tue Dec 11 08:07:38 2012
    Started redo scan
    Tue Dec 11 08:07:38 2012
    Completed redo scan
    4008 redo blocks read, 766 data blocks need recovery
    Tue Dec 11 08:07:38 2012
    Started redo application at
    Thread 2: logseq 7916, block 48159
    Tue Dec 11 08:07:39 2012
    Recovery of Online Redo Log: Thread 2 Group 4 Seq 7916 Reading mem 0
    Mem# 0 errs 0: M:\ORC\REDO04.LOG
    Tue Dec 11 08:07:39 2012
    Completed redo application
    Tue Dec 11 08:07:39 2012
    Completed instance recovery at
    Thread 2: logseq 7916, block 52167, scn 5966110632827
    746 data blocks read, 785 data blocks written, 4008 redo blocks read
    Switch log for thread 2 to sequence 7917
    Tue Dec 11 08:30:03 2012
    Reconfiguration started (old inc 14, new inc 16)
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    Tue Dec 11 08:30:16 2012
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Tue Dec 11 08:30:16 2012
    LMS 0: 0 GCS shadows cancelled, 0 closed
    Tue Dec 11 08:30:17 2012
    LMS 1: 0 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Tue Dec 11 08:30:18 2012
    LMS 1: 11204 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:18 2012
    LMS 0: 11283 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:18 2012
    LMS 1: 11300 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:18 2012
    LMS 0: 11328 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:18 2012
    LMS 1: 11122 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:18 2012
    LMS 0: 11292 GCS shadows traversed, 4001 replayed
    LMS 0: 11320 GCS shadows traversed, 4001 replayed
    LMS 0: 11313 GCS shadows traversed, 4001 replayed
    LMS 0: 11298 GCS shadows traversed, 4001 replayed
    LMS 0: 11384 GCS shadows traversed, 4001 replayed
    LMS 0: 11182 GCS shadows traversed, 4001 replayed
    LMS 0: 11279 GCS shadows traversed, 4001 replayed
    LMS 0: 11429 GCS shadows traversed, 4001 replayed
    LMS 0: 11187 GCS shadows traversed, 4001 replayed
    LMS 0: 11336 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:19 2012
    LMS 1: 11031 GCS shadows traversed, 4001 replayed
    LMS 1: 11207 GCS shadows traversed, 4001 replayed
    LMS 1: 11170 GCS shadows traversed, 4001 replayed
    LMS 1: 11293 GCS shadows traversed, 4001 replayed
    LMS 1: 11236 GCS shadows traversed, 4001 replayed
    LMS 1: 11202 GCS shadows traversed, 4001 replayed
    LMS 1: 11154 GCS shadows traversed, 4001 replayed
    LMS 1: 11200 GCS shadows traversed, 4001 replayed
    Tue Dec 11 08:30:21 2012
    LMS 0: 11447 GCS shadows traversed, 4001 replayed
    LMS 0: 11235 GCS shadows traversed, 4001 replayed
    LMS 0: 11270 GCS shadows traversed, 4001 replayed
    LMS 0: 11192 GCS shadows traversed, 4001 replayed
    LMS 0: 11496 GCS shadows traversed, 4001 replayed
    LMS 0: 11337 GCS shadows traversed, 4001 replayed
    LMS 0: 11289 GCS shadows traversed, 4001 replayed
    LMS 0: 11286 GCS shadows traversed, 4001 replayed
    LMS 0: 11455 GCS shadows traversed, 4001 replayed
    LMS 0: 11196 GCS shadows traversed, 4001 replayed
    LMS 0: 11247 GCS shadows traversed, 4001 replayed
    LMS 0: 11416 GCS shadows traversed, 4001 replayed
    LMS 0: 11428 GCS shadows traversed, 4001 replayed
    LMS 0: 11318 GCS shadows traversed, 4001 replayed
    LMS 0: 11231 GCS shadows traversed, 4001 replayed
    LMS 0: 11279 GCS shadows traversed, 4001 replayed
    LMS 0: 11352 GCS shadows traversed, 4001 replayed
    LMS 0: 11378 GCS shadows traversed, 4001 replayed
    LMS 0: 1509 GCS shadows traversed, 552 replayed
    Tue Dec 11 08:30:22 2012
    LMS 1: 11139 GCS shadows traversed, 4001 replayed
    LMS 1: 11352 GCS shadows traversed, 4001 replayed
    LMS 1: 11199 GCS shadows traversed, 4001 replayed
    LMS 1: 11181 GCS shadows traversed, 4001 replayed
    LMS 1: 11148 GCS shadows traversed, 4001 replayed
    LMS 1: 11100 GCS shadows traversed, 4001 replayed
    LMS 1: 11345 GCS shadows traversed, 4001 replayed
    LMS 1: 11101 GCS shadows traversed, 4001 replayed
    LMS 1: 11100 GCS shadows traversed, 4001 replayed
    LMS 1: 11277 GCS shadows traversed, 4001 replayed
    LMS 1: 11215 GCS shadows traversed, 4001 replayed
    LMS 1: 11203 GCS shadows traversed, 4001 replayed
    LMS 1: 11216 GCS shadows traversed, 4001 replayed
    LMS 1: 11264 GCS shadows traversed, 4001 replayed
    LMS 1: 11226 GCS shadows traversed, 4001 replayed
    LMS 1: 11203 GCS shadows traversed, 4001 replayed
    LMS 1: 11125 GCS shadows traversed, 4001 replayed
    LMS 1: 11286 GCS shadows traversed, 4001 replayed
    LMS 1: 11195 GCS shadows traversed, 4001 replayed
    LMS 1: 10937 GCS shadows traversed, 3903 replayed
    Tue Dec 11 08:30:23 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Reconfiguration complete
    Tue Dec 11 08:47:09 2012
    Thread 1 advanced to log sequence 9420
    Current log# 1 seq# 9420 mem# 0: M:\ORC\REDO01.LOG
    Tue Dec 11 09:05:02 2012
    Reconfiguration started (old inc 16, new inc 18)
    List of nodes:
    0
    Tue Dec 11 09:05:28 2012
    Global Resource Directory frozen
    * dead instance detected - domain 0 invalid = TRUE
    Communication channels reestablished
    Master broadcasted resource hash value bitmaps
    Non-local Process blocks cleaned out
    Tue Dec 11 09:05:28 2012
    LMS 1: 5 GCS shadows cancelled, 2 closed
    Tue Dec 11 09:05:29 2012
    LMS 0: 3 GCS shadows cancelled, 0 closed
    Set master node info
    Submitted all remote-enqueue requests
    Dwn-cvts replayed, VALBLKs dubious
    All grantable enqueues granted
    Post SMON to start 1st pass IR
    Tue Dec 11 09:05:31 2012
    LMS 0: 386379 GCS shadows traversed, 0 replayed
    Tue Dec 11 09:05:31 2012
    LMS 1: 393515 GCS shadows traversed, 0 replayed
    Tue Dec 11 09:05:31 2012
    Submitted all GCS remote-cache requests
    Post SMON to start 1st pass IR
    Fix write in gcs resources
    Tue Dec 11 09:05:31 2012
    Instance recovery: looking for dead threads
    Tue Dec 11 09:05:31 2012
    Beginning instance recovery of 1 threads
    Reconfiguration complete
    Tue Dec 11 09:05:34 2012
    parallel recovery started with 7 processes
    Tue Dec 11 09:05:34 2012
    Started redo scan
    Tue Dec 11 09:05:35 2012
    Completed redo scan
    6460 redo blocks read, 1499 data blocks need recovery
    Tue Dec 11 09:05:35 2012
    Started redo application at
    Thread 2: logseq 7917, block 16997
    Tue Dec 11 09:05:36 2012
    Recovery of Online Redo Log: Thread 2 Group 3 Seq 7917 Reading mem 0
    Mem# 0 errs 0: M:\ORC\REDO03.LOG
    Tue Dec 11 09:05:36 2012
    Completed redo application
    Tue Dec 11 09:05:36 2012
    Completed instance recovery at
    Thread 2: logseq 7917, block 23457, scn 5966110693995
    1267 data blocks read, 1532 data blocks written, 6460 redo blocks read
    Switch log for thread 2 to sequence 7918
    CRS Alert Log
    2012-12-09 08:04:32.545
    [crsd(9168)]CRS-1204:Recovering CRS resources for node mdb2.
    2012-12-09 12:07:11.731
    [cssd(1932)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
    2012-12-10 02:31:30.428
    [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
    2012-12-10 02:31:30.506
    [cssd(8708)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
    2012-12-10 02:31:34.272
    [crsd(7540)]CRS-1204:Recovering CRS resources for node mdb2.
    2012-12-10 22:42:03.651
    [cssd(8868)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
    2012-12-10 23:58:58.617
    [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
    2012-12-10 23:59:00.648
    [cssd(8512)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
    2012-12-10 23:59:07.101
    [crsd(9624)]CRS-1204:Recovering CRS resources for node mdb2.
    2012-12-11 07:47:41.587
    [cssd(9328)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
    2012-12-11 08:07:27.587
    [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
    2012-12-11 08:07:29.634
    [cssd(4552)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
    2012-12-11 08:07:36.541
    [crsd(9144)]CRS-1204:Recovering CRS resources for node mdb2.
    2012-12-11 08:29:35.192
    [cssd(9656)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 mdb2 .
    2012-12-11 09:04:57.130
    [cssd(5496)]CRS-1607:CSSD evicting node mdb2. Details in d:\oracle\crs\log\mdb1\cssd\ocssd.log.
    2012-12-11 09:04:59.193
    [cssd(9468)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mdb1 .
    2012-12-11 09:05:05.303
    [crsd(5452)]CRS-1204:Recovering CRS resources for node mdb2.


    Thanks
  • 8. Re: Error on RAC node
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Post crs_stat -t


    also review this thread
    10g RAC on AIX node eviction - HELP

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points