6 Replies Latest reply on Jan 17, 2013 8:43 AM by Aman....

    RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064

    user00726
      Hi,

      I have 2 node 11g R2 Cluster on RHEL5.WHen I am trying to start the instance, i am gettiing the below error:
      [root@rac1 bin]# ./srvctl start instance -i dev1 -d dev
      PRCR-1013 : Failed to start resource ora.dev.db
      PRCR-1064 : Failed to start resource ora.dev.db on node rac1
      ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
      CRS-2674: Start of 'ora.dev.db' on 'rac1' failed
      [root@rac1 bin]# ./srvctl status scan_listener
      SCAN Listener LISTENER_SCAN1 is enabled
      SCAN listener LISTENER_SCAN1 is running on node rac1
      SCAN Listener LISTENER_SCAN2 is enabled
      SCAN listener LISTENER_SCAN2 is running on node rac2
      SCAN Listener LISTENER_SCAN3 is enabled
      SCAN listener LISTENER_SCAN3 is running on node rac1
      
      
      [root@rac1 bin]# ./srvctl config nodeapps -a
      VIP exists.:rac1
      VIP exists.: /rac1-vip/10.17.57.114/255.255.255.0/eth0
      VIP exists.:rac2
      VIP exists.: /rac2-vip/10.17.57.115/255.255.255.0/eth0
      [root@rac1 bin]# ./srvctl config nodeapps 
      VIP exists.:rac1
      VIP exists.: /rac1-vip/10.17.57.114/255.255.255.0/eth0
      VIP exists.:rac2
      VIP exists.: /rac2-vip/10.17.57.115/255.255.255.0/eth0
      GSD exists.
      ONS daemon exists. Local port 6100, remote port 6200
      eONS daemon exists. Multicast port 17412, multicast IP address 234.62.213.92, listening port 2016
      [root@rac2 bin]# ./crs_stop -all
      CRS-2500: Cannot stop resource 'ora.gsd' as it is not running
      CRS-2500: Cannot stop resource 'ora.oc4j' as it is not running
      Attempting to stop `ora.eons` on member `rac2`
      Attempting to stop `ora.LISTENER.lsnr` on member `rac1`
      Attempting to stop `ora.eons` on member `rac1`
      Attempting to stop `ora.LISTENER.lsnr` on member `rac2`
      CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac2'
      Attempting to stop `ora.CRS.dg` on member `rac2`
      Attempting to stop `ora.DATA.dg` on member `rac2`
      Attempting to stop `ora.FRA.dg` on member `rac2`
      Attempting to stop `ora.registry.acfs` on member `rac2`
      Attempting to stop `ora.ons` on member `rac2`
      Attempting to stop `ora.CRS.dg` on member `rac1`
      Attempting to stop `ora.DATA.dg` on member `rac1`
      Attempting to stop `ora.FRA.dg` on member `rac1`
      Attempting to stop `ora.registry.acfs` on member `rac1`
      CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac1'
      Stop of `ora.LISTENER.lsnr` on member `rac2` succeeded.
      Stop of `ora.LISTENER.lsnr` on member `rac1` succeeded.
      Attempting to stop `ora.ons` on member `rac1`
      Attempting to stop `ora.rac1.vip` on member `rac1`
      Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
      Stop of `ora.rac1.vip` on member `rac1` succeeded.
      Stop of `ora.ons` on member `rac2` succeeded.
      Attempting to stop `ora.rac2.vip` on member `rac2`
      Attempting to stop `ora.LISTENER_SCAN2.lsnr` on member `rac2`
      Attempting to stop `ora.LISTENER_SCAN3.lsnr` on member `rac2`
      Stop of `ora.rac2.vip` on member `rac2` succeeded.
      Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
      Attempting to stop `ora.scan1.vip` on member `rac1`
      Stop of `ora.scan1.vip` on member `rac1` succeeded.
      Stop of `ora.ons` on member `rac1` succeeded.
      CRS-2789: Cannot stop resource 'ora.ons' as it is not running on server 'rac1'
      Attempting to stop `ora.net1.network` on member `rac1`
      Stop of `ora.net1.network` on member `rac1` succeeded.
      Stop of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
      Attempting to stop `ora.scan2.vip` on member `rac2`
      Stop of `ora.scan2.vip` on member `rac2` succeeded.
      Stop of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
      Attempting to stop `ora.scan3.vip` on member `rac2`
      Stop of `ora.scan3.vip` on member `rac2` succeeded.
      Attempting to stop `ora.net1.network` on member `rac2`
      Stop of `ora.net1.network` on member `rac2` succeeded.
      Stop of `ora.registry.acfs` on member `rac1` succeeded.
      Stop of `ora.registry.acfs` on member `rac2` succeeded.
      Stop of `ora.FRA.dg` on member `rac2` succeeded.
      Stop of `ora.eons` on member `rac2` succeeded.
      Stop of `ora.CRS.dg` on member `rac1` succeeded.
      Stop of `ora.eons` on member `rac1` succeeded.
      Stop of `ora.DATA.dg` on member `rac2` succeeded.
      Stop of `ora.CRS.dg` on member `rac2` succeeded.
      Attempting to stop `ora.asm` on member `rac2`
      Stop of `ora.DATA.dg` on member `rac1` succeeded.
      Stop of `ora.FRA.dg` on member `rac1` succeeded.
      Attempting to stop `ora.asm` on member `rac1`
      CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
      ORA-15097: cannot SHUTDOWN ASM instance with connected client
      CRS-2675: Stop of 'ora.asm' on 'rac1' failed
      
      CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
      ORA-15097: cannot SHUTDOWN ASM instance with connected client
      CRS-2675: Stop of 'ora.asm' on 'rac2' failed
      
      CRS-0216: Could not stop resource 'ora.asm rac1 1'.
      
      CRS-0216: Could not stop resource 'ora.asm rac2 1'.
      
      CRS-0216: Could not stop resource 'ora.gsd'.
      
      CRS-0216: Could not stop resource 'ora.gsd rac1 1'.
      
      CRS-0216: Could not stop resource 'ora.gsd rac2 1'.
      
      CRS-0216: Could not stop resource 'ora.oc4j'.
      
      CRS-0216: Could not stop resource 'ora.ons rac1 1'.
      
      [root@rac2 bin]# ./crs_stat -t
      Name           Type           Target    State     Host        
      ------------------------------------------------------------
      ora.CRS.dg     ora....up.type OFFLINE   OFFLINE               
      ora.DATA.dg    ora....up.type OFFLINE   OFFLINE               
      ora.FRA.dg     ora....up.type OFFLINE   ONLINE    rac2        
      ora....ER.lsnr ora....er.type OFFLINE   OFFLINE               
      ora....N1.lsnr ora....er.type OFFLINE   OFFLINE               
      ora....N2.lsnr ora....er.type OFFLINE   OFFLINE               
      ora....N3.lsnr ora....er.type OFFLINE   OFFLINE               
      ora.asm        ora.asm.type   OFFLINE   ONLINE    rac2        
      ora.dev.db     ora....se.type OFFLINE   OFFLINE               
      ora.eons       ora.eons.type  OFFLINE   OFFLINE               
      ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
      ora....network ora....rk.type OFFLINE   OFFLINE               
      ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
      ora.ons        ora.ons.type   OFFLINE   OFFLINE               
      ora....SM1.asm application    OFFLINE   ONLINE    rac1        
      ora....C1.lsnr application    OFFLINE   OFFLINE               
      ora.rac1.gsd   application    OFFLINE   OFFLINE               
      ora.rac1.ons   application    OFFLINE   OFFLINE               
      ora.rac1.vip   ora....t1.type OFFLINE   OFFLINE               
      ora....SM2.asm application    OFFLINE   ONLINE    rac2        
      ora....C2.lsnr application    OFFLINE   OFFLINE               
      ora.rac2.gsd   application    OFFLINE   OFFLINE               
      ora.rac2.ons   application    OFFLINE   OFFLINE               
      ora.rac2.vip   ora....t1.type OFFLINE   OFFLINE               
      ora....ry.acfs ora....fs.type OFFLINE   OFFLINE               
      ora.scan1.vip  ora....ip.type OFFLINE   OFFLINE               
      ora.scan2.vip  ora....ip.type OFFLINE   OFFLINE               
      ora.scan3.vip  ora....ip.type OFFLINE   OFFLINE               
      [root@rac2 bin]# ./crs_start -all
      CRS-2501: Resource 'ora.gsd' is disabled
      CRS-2501: Resource 'ora.oc4j' is disabled
      CRS-2501: Resource 'ora.gsd' is disabled
      CRS-2501: Resource 'ora.gsd' is disabled
      Attempting to start `ora.registry.acfs` on member `rac2`
      CRS-2664: Resource 'ora.asm' is already running on 'rac2'
      Attempting to start `ora.registry.acfs` on member `rac1`
      Attempting to start `ora.eons` on member `rac1`
      Attempting to start `ora.eons` on member `rac2`
      Attempting to start `ora.DATA.dg` on member `rac1`
      Attempting to start `ora.FRA.dg` on member `rac1`
      Attempting to start `ora.net1.network` on member `rac1`
      Attempting to start `ora.net1.network` on member `rac2`
      Attempting to start `ora.CRS.dg` on member `rac2`
      Attempting to start `ora.CRS.dg` on member `rac1`
      Start of `ora.CRS.dg` on member `rac2` succeeded.
      Start of `ora.net1.network` on member `rac2` succeeded.
      Attempting to start `ora.rac1.vip` on member `rac2`
      Attempting to start `ora.scan2.vip` on member `rac2`
      Attempting to start `ora.scan3.vip` on member `rac2`
      Start of `ora.net1.network` on member `rac1` succeeded.
      Attempting to start `ora.scan1.vip` on member `rac1`
      Start of `ora.registry.acfs` on member `rac1` succeeded.
      Start of `ora.registry.acfs` on member `rac2` succeeded.
      Start of `ora.FRA.dg` on member `rac1` succeeded.
      Start of `ora.scan3.vip` on member `rac2` succeeded.
      Start of `ora.scan2.vip` on member `rac2` succeeded.
      Attempting to start `ora.LISTENER_SCAN2.lsnr` on member `rac2`
      Start of `ora.rac1.vip` on member `rac2` succeeded.
      Start of `ora.scan1.vip` on member `rac1` succeeded.
      Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `rac1`
      Attempting to start `ora.LISTENER_SCAN3.lsnr` on member `rac2`
      Attempting to stop `ora.scan2.vip` on member `rac2`
      Attempting to stop `ora.scan3.vip` on member `rac2`
      Attempting to start `ora.ons` on member `rac1`
      Attempting to start `ora.ons` on member `rac2`
      Stop of `ora.scan2.vip` on member `rac2` succeeded.
      Attempting to start `ora.scan2.vip` on member `rac1`
      Stop of `ora.scan3.vip` on member `rac2` succeeded.
      Attempting to start `ora.scan3.vip` on member `rac1`
      Start of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
      Attempting to stop `ora.rac1.vip` on member `rac2`
      Stop of `ora.rac1.vip` on member `rac2` succeeded.
      Attempting to start `ora.rac1.vip` on member `rac1`
      Start of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
      Start of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
      Attempting to stop `ora.scan1.vip` on member `rac1`
      Stop of `ora.scan1.vip` on member `rac1` succeeded.
      Attempting to start `ora.scan1.vip` on member `rac2`
      Start of `ora.ons` on member `rac1` succeeded.
      Start of `ora.ons` on member `rac2` succeeded.
      Start of `ora.rac1.vip` on member `rac1` succeeded.
      Start of `ora.scan2.vip` on member `rac1` succeeded.
      Attempting to start `ora.LISTENER.lsnr` on member `rac1`
      Start of `ora.scan3.vip` on member `rac1` succeeded.
      Attempting to stop `ora.LISTENER_SCAN2.lsnr` on member `rac2`
      Attempting to stop `ora.LISTENER_SCAN3.lsnr` on member `rac2`
      Stop of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
      Attempting to start `ora.LISTENER_SCAN2.lsnr` on member `rac1`
      Stop of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
      Attempting to start `ora.LISTENER_SCAN3.lsnr` on member `rac1`
      Start of `ora.LISTENER.lsnr` on member `rac1` succeeded.
      Start of `ora.scan1.vip` on member `rac2` succeeded.
      Start of `ora.LISTENER_SCAN2.lsnr` on member `rac1` succeeded.
      Start of `ora.LISTENER_SCAN3.lsnr` on member `rac1` succeeded.
      Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
      Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
      Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `rac2`
      Start of `ora.LISTENER_SCAN1.lsnr` on member `rac2` succeeded.
      Start of `ora.CRS.dg` on member `rac1` succeeded.
      Start of `ora.DATA.dg` on member `rac1` succeeded.
      Start of `ora.eons` on member `rac2` succeeded.
      Start of `ora.eons` on member `rac1` succeeded.
      Attempting to start `ora.DATA.dg` on member `rac2`
      Attempting to start `ora.rac2.vip` on member `rac2`
      Attempting to start `ora.dev.db` on member `rac1`
      Start of `ora.DATA.dg` on member `rac2` succeeded.
      Start of `ora.rac2.vip` on member `rac2` succeeded.
      Attempting to start `ora.LISTENER.lsnr` on member `rac2`
      Start of `ora.LISTENER.lsnr` on member `rac2` succeeded.
      Attempting to start `ora.dev.db` on member `rac2`
      ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
      Start of `ora.dev.db` on member `rac1` failed.
      Attempting to stop `ora.dev.db` on member `rac1`
      Stop of `ora.dev.db` on member `rac1` succeeded.
      CRS-2632: There are no more servers to try to place resource 'ora.dev.db' on that would satisfy its placement policy
      ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
      Start of `ora.dev.db` on member `rac2` failed.
      Attempting to stop `ora.dev.db` on member `rac2`
      Stop of `ora.dev.db` on member `rac2` succeeded.
      Attempting to stop `ora.LISTENER.lsnr` on member `rac2`
      Stop of `ora.LISTENER.lsnr` on member `rac2` succeeded.
      Attempting to stop `ora.rac2.vip` on member `rac2`
      Stop of `ora.rac2.vip` on member `rac2` succeeded.
      Attempting to stop `ora.DATA.dg` on member `rac2`
      Stop of `ora.DATA.dg` on member `rac2` succeeded.
      CRS-0223: Resource 'ora.asm rac2 1' has placement error.
      
      CRS-0223: Resource 'ora.dev.db 1 1' has placement error.
      
      CRS-0215: Could not start resource 'ora.dev.db 2 1'.
      
      CRS-2660: Resource 'ora.gsd' or all of its instances are disabled
      
      CRS-2660: Resource 'ora.oc4j' or all of its instances are disabled
      
      CRS-2660: Resource 'ora.rac1.gsd' or all of its instances are disabled
      
      CRS-2660: Resource 'ora.rac2.gsd' or all of its instances are disabled
      
      [root@rac2 bin]# 
      alert log of rac2
      [root@rac2 rac2]# tail -n 100 alertrac2.log 
      2013-01-16 13:56:10.825
      [ohasd(4816)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 
      2013-01-16 13:56:10.889
      [gpnpd(5269)]CRS-2328:GPNPD started on node rac2. 
      2013-01-16 13:56:15.858
      [cssd(5325)]CRS-1713:CSSD daemon is started in clustered mode
      [client(5480)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
      [client(5502)]CRS-10001:ACFS-9322: done.
      2013-01-16 13:56:25.065
      [cssd(5325)]CRS-1707:Lease acquisition for node rac2 number 2 completed
      2013-01-16 13:56:25.098
      [cssd(5325)]CRS-1605:CSSD voting file is online: ORCL:VOTES1; details in /raczone/grid_home/log/rac2/cssd/ocssd.log.
      2013-01-16 13:56:40.502
      [cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
      2013-01-16 13:56:42.677
      [ctssd(5547)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
      2013-01-16 13:56:42.703
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 13:56:42.980
      [ctssd(5547)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
      2013-01-16 13:56:43.012
      [/raczone/grid_home/bin/oraagent.bin(5201)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/raczone/grid_home/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"
      2013-01-16 13:57:03.995
      [/raczone/grid_home/bin/oraagent.bin(5201)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/raczone/grid_home/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"
      2013-01-16 13:57:15.713
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 13:57:31.719
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 13:57:50.328
      [crsd(5747)]CRS-1012:The OCR service started on node rac2.
      2013-01-16 13:57:50.435
      [evmd(5562)]CRS-1401:EVMD started on node rac2.
      2013-01-16 13:57:53.652
      [crsd(5747)]CRS-1201:CRSD started on node rac2.
      2013-01-16 13:58:01.718
      [crsd(5747)]CRS-2772:Server 'rac2' has been assigned to pool 'Generic'.
      2013-01-16 13:58:01.719
      [crsd(5747)]CRS-2772:Server 'rac2' has been assigned to pool 'ora.dev'.
      2013-01-16 14:00:44.597
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:01:00.604
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:01:21.547
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
      2013-01-16 14:01:21.551
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
      2013-01-16 14:03:25.490
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:03:41.496
      [ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:11:42.321
      [ctssd(5547)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
      2013-01-16 14:12:02.182
      [cssd(5325)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.290 seconds
      2013-01-16 14:12:09.198
      [cssd(5325)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.280 seconds
      2013-01-16 14:12:14.209
      [cssd(5325)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.270 seconds
      2013-01-16 14:12:16.488
      [cssd(5325)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 253469533
      2013-01-16 14:12:16.603
      [cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
      2013-01-16 14:12:16.696
      [crsd(5747)]CRS-5504:Node down event reported for node 'rac1'.
      2013-01-16 14:12:30.571
      [crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'Generic'.
      2013-01-16 14:12:30.571
      [crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'ora.dev'.
      2013-01-16 14:14:55.076
      [cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
      2013-01-16 14:15:28.630
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
      2013-01-16 14:15:28.632
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
      2013-01-16 14:25:22.629
      [cssd(5325)]CRS-1625:Node rac1, number 1, was manually shut down
      2013-01-16 14:25:22.998
      [cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
      2013-01-16 14:25:23.005
      [crsd(5747)]CRS-5504:Node down event reported for node 'rac1'.
      2013-01-16 14:25:23.006
      [crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'Generic'.
      2013-01-16 14:25:23.006
      [crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'ora.dev'.
      2013-01-16 14:26:25.301
      [cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
      2013-01-16 14:27:02.746
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
      2013-01-16 14:27:02.746
      [crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
      2013-01-16 21:03:45.088
      [crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac2'.
      2013-01-16 21:03:54.732
      [crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
      2013-01-17 10:05:25.527
      [crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
      2013-01-17 10:05:25.970
      [crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac2'.
      2013-01-17 10:05:40.587
      [cssd(5325)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.730 seconds
      [root@rac2 rac2]# 
      alert log of rac1
      [root@rac1 rac1]# tail -n 100 alertrac1.log 
      2013-01-16 14:24:58.057
      [/raczone/grid_home/bin/orarootagent.bin(5457)]CRS-5822:Agent '/raczone/grid_home/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /raczone/grid_home/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
      2013-01-16 14:25:01.292
      [ctssd(5171)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user
      [client(7353)]CRS-10001:Waiting for ASM to shutdown.
      2013-01-16 14:25:22.627
      [cssd(5124)]CRS-1603:CSSD on node rac1 shutdown by user.
      2013-01-16 14:25:48.831
      [ohasd(7461)]CRS-2112:The OLR service started on node rac1.
      2013-01-16 14:25:51.112
      [ohasd(7461)]CRS-8011:reboot advisory message from host: rac1, component: ag121731, with time stamp: L-2013-01-14-14:57:16.901
      [ohasd(7461)]CRS-8013:reboot advisory message text: Rebooting after limit 28080 exceeded; disk timeout 28080, network timeout 23640, last heartbeat from CSSD at epoch seconds 1358155608.710, 28170 milliseconds ago based on invariant clock value of 7675224
      2013-01-16 14:25:51.113
      [ohasd(7461)]CRS-8017:location: /etc/oracle/lastgasp has 20 reboot advisory log files, 1 were announced and 0 errors occurred
      2013-01-16 14:25:53.007
      [ohasd(7461)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
      2013-01-16 14:26:00.147
      [ohasd(7461)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 
      2013-01-16 14:26:03.350
      [cssd(7684)]CRS-1713:CSSD daemon is started in clustered mode
      [client(7794)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
      [client(7877)]CRS-10001:ACFS-9322: done.
      2013-01-16 14:26:13.842
      [cssd(7684)]CRS-1707:Lease acquisition for node rac1 number 1 completed
      2013-01-16 14:26:13.885
      [cssd(7684)]CRS-1605:CSSD voting file is online: ORCL:VOTES1; details in /raczone/grid_home/log/rac1/cssd/ocssd.log.
      2013-01-16 14:26:25.258
      [cssd(7684)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
      2013-01-16 14:26:27.458
      [ctssd(7919)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
      2013-01-16 14:26:28.277
      [ctssd(7919)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
      2013-01-16 14:26:51.860
      [crsd(8072)]CRS-1012:The OCR service started on node rac1.
      2013-01-16 14:26:55.017
      [crsd(8072)]CRS-1201:CRSD started on node rac1.
      2013-01-16 14:27:00.522
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:27:16.527
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:43:40.942
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:44:04.951
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:56:13.621
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-16 14:56:29.627
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:05:37.600
      [cssd(7684)]CRS-1612:Network communication with node rac2 (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.390 seconds
      2013-01-17 10:05:37.844
      [cssd(7684)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 10020 milliseconds
      2013-01-17 10:05:41.828
      [cssd(7684)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 6030 milliseconds
      2013-01-17 10:05:44.713
      [cssd(7684)]CRS-1611:Network communication with node rac2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.280 seconds
      2013-01-17 10:06:27.587
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:06:43.591
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:22:51.994
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:45:56.525
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:52:28.656
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:52:44.663
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 10:56:04.766
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:00:30.001
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:00:46.024
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:25:29.077
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:28:25.248
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:28:41.253
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:36:41.512
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:36:57.519
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:40:33.660
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:40:49.670
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:47:38.670
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:47:54.676
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:48:10.768
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:49:15.539
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:50:35.707
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      2013-01-17 11:50:51.713
      [ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
      [root@rac1 rac1]# 
      It seems from the alert log that due to heatbeat/diskbeat, CSSD has evicted the node from cluster.

      Can anynody suggest me for the same....i
        • 1. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
          Osama_Mustafa
          ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
          Indicate as Bug , Go to http://support.oracle.com check this error nothing found Open SR


          Also Refer to
          CSS Timeout Computation in Oracle Clusterware [ID 294430.1]
          • 2. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
            moreajays
            Hi,

            It looks , heartbeat & network issue . Check if voting Disk health/accessibility from all nodes / Private Interface connectivity , clock sync
            [cssd(7684)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 10020 milliseconds
            2013-01-17 10:05:41.828
            [cssd(7684)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 6030 milliseconds
            2013-01-17 10:05:44.713
            [cssd(7684)]CRS-1611:Network communication with node rac2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.280 seconds
            2013-01-17 10:06:27.587
            Thanks,
            Ajay More
            http://moreajays.blogspot.com
            • 3. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
              user00726
              Hi all,

              Solve the issue :
              oracle@rac2 ~]$ . oraenv
              ORACLE_SID = [oracle] ? dev2
              ORACLE_HOME = [/home/oracle] ? /raczone/db_home1
              The Oracle base for ORACLE_HOME=/raczone/db_home1 is /raczone/11.2.0
              [oracle@rac2 ~]$ cd /raczone/db_home1/bin/
              [oracle@rac2 bin]$ ./sqlplus "/as sysdba"
              
              SQL*Plus: Release 11.2.0.1.0 Production on Thu Jan 17 13:38:31 2013
              
              Copyright (c) 1982, 2009, Oracle.  All rights reserved.
              
              Connected to an idle instance.
              
              SQL> startup mount
              ORACLE instance started.
              
              Total System Global Area  790941696 bytes
              Fixed Size                  1339428 bytes
              Variable Size             583012316 bytes
              Database Buffers          192937984 bytes
              Redo Buffers               13651968 bytes
              Database mounted.
              SQL> recover database;
              Media recovery complete.
              SQL> alter database open;
              
              Database altered.
              
              SQL> exit
              Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
              With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
              Data Mining and Real Application Testing options
              [oracle@rac2 bin]$ 
              [root@rac1 bin]# ./srvctl status database -d dev
              Instance dev1 is not running on node rac1
              Instance dev2 is not running on node rac2
              [root@rac1 bin]# ./srvctl start instance -n rac2 -d dev
              [root@rac1 bin]# ./srvctl status database -d dev
              Instance dev1 is not running on node rac1
              Instance dev2 is running on node rac2
              [root@rac1 bin]# ./srvctl start instance -n rac1 -d dev
              [root@rac1 bin]# ./srvctl status database -d dev
              Instance dev1 is running on node rac1
              Instance dev2 is running on node rac2
              [root@rac1 bin]# 
              • 4. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
                Aman....
                You are talking about two different errors. If the node is not a part of the network anymore, you must check the ocssd log and confirm this. If that's the case actually, before doing anything else, fix this issue. The database startup issue is something else and as I said, you should be looking at the network issue first before you touch it. The error for the database is that it's struck while doing the instance recovery and for that, you can mount it and than issue recover database and then open it. It should fix it. I can't access the MOS right now for the supporting doc id for the error. But do give a check on the MOS and I believe you should be able to find the doc id.

                Aman....
                • 5. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
                  user00726
                  Aman, i have crosscheck the same also.But at the OS level one node is easily able to ping the other node or network.and vice versa but still i am not able to understand why this problem has occured.Me too rt now dunt have a MOS access, so really can;t check the same error at MOS.
                  • 6. Re: RAC 11gR2 database not started i.e PRCR-1013 and PRCR-1064
                    Aman....
                    user00726 wrote:
                    Aman, i have crosscheck the same also.But at the OS level one node is easily able to ping the other node or network.and vice versa but still i am not able to understand why this problem has occured.Me too rt now dunt have a MOS access, so really can;t check the same error at MOS.
                    If the database is working on both the nodes, it certainly menas that you don't have the network issue. I doubted that in the first place itself because the errors in the case of the node eviction are not what you had got. This is a case of struck instance recovery . May be due to any transient network glitch your node was not able to ping and was over the heartbeat limit of the control file and the instance was crashed as there is also a instance level heartbeat as well that's maintained by Oracle besides the node level heartbeat, just a guess.

                    Aman....

                    PS: Use proper words on a professional forum like this one. "Rt" , "dunt" etc are best to be used for text messages.