6 Replies Latest reply: Nov 21, 2012 2:16 AM by 975382 RSS

    cluster not comming up.

    user7490278
      $ crsctl check crs
      CRS-4638: Oracle High Availability Services is online
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
      CRS-4533: Event Manager is online
      [oracle@~]$ crsctl check crs
      CRS-4638: Oracle High Availability Services is online
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
      CRS-4533: Event Manager is online

      Edited by: user7490278 on Feb 27, 2012 8:28 AM
        • 1. Re: cluster not comming up.
          CKPT
          user7490278 wrote:
          $ crsctl check crs
          CRS-4638: Oracle High Availability Services is online
          CRS-4535: Cannot communicate with Cluster Ready Services
          CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
          CRS-4533: Event Manager is online
          [oracle@~]$ crsctl check crs
          CRS-4638: Oracle High Availability Services is online
          CRS-4535: Cannot communicate with Cluster Ready Services
          CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
          CRS-4533: Event Manager is online

          Edited by: user7490278 on Feb 27, 2012 8:28 AM
          What is CRS version?
          Have you been referred CRS log files? What they are saying
          • 2. Re: cluster not comming up.
            user7490278
            cluster version is 11gr2

            CRS-4638: Oracle High Availability Services is online
            CRS-4535: Cannot communicate with Cluster Ready Services
            CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
            CRS-4533: Event Manager is online



            LOG INFO:=

            [oracle@racdb01cracdb1]$ cat alertcracdb1.log |tail -400
            2012-02-27 07:28:00.541
            [cssd(9203)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 07:28:05.651
            [ohasd(8179)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 07:28:06.307
            [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 07:28:18.077
            [cssd(10751)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 07:28:26.778
            [ohasd(8179)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 07:28:26.818
            [ohasd(8179)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 07:38:17.066
            [u01/app/grid/11.2.0/bin/cssdagent(10736)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 07:38:17.066
            [cssd(10751)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 07:38:22.187
            [ohasd(8179)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 07:38:22.388
            [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 07:38:22.499
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 08:08:28.170
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 08:38:33.842
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 09:08:39.517
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 09:22:39.192
            [ohasd(8179)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 09:43:47.740
            [ohasd(8179)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 09:54:40.546
            [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 09:55:17.826
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 10:25:24.528
            [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 10:32:28.511
            [u01/app/grid/11.2.0/bin/cssdmonitor(20275)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
            2012-02-27 10:36:36.046
            [ohasd(7636)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 10:36:38.300
            [ohasd(7636)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(7636)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 10:36:39.056
            [ohasd(7636)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 10:36:42.921
            [ohasd(7636)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 10:36:50.313
            [ohasd(7636)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:36:52.217
            [cssd(8524)]CRS-1713:CSSD daemon is started in clustered mode
            [client(8612)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
            [client(8650)]CRS-10001:ACFS-9322: done.
            2012-02-27 10:46:50.824
            [u01/app/grid/11.2.0/bin/cssdagent(8507)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 10:46:50.825
            [cssd(8524)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 10:46:55.944
            [ohasd(7636)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 10:46:56.508
            [ohasd(7636)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 10:47:08.236
            [cssd(10647)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 10:47:17.234
            [ohasd(7636)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 10:47:17.269
            [ohasd(7636)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 10:54:44.707
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:45.714
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:46.722
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:47.729
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:48.736
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:49.743
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:50.750
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:51.757
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:52.764
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:53.771
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:54.778
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:55.785
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:56.792
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:57.799
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:58.806
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:54:59.813
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:00.820
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:01.827
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:02.834
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:03.842
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:04.849
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:05.856
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:06.864
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:07.872
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:08.879
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:09.887
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:10.894
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:11.902
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:12.910
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:13.916
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:14.923
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:15.931
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:16.938
            [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:55:17.707
            [u01/app/grid/11.2.0/bin/cssdmonitor(10617)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
            2012-02-27 10:59:27.293
            [ohasd(7700)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 10:59:30.329
            [ohasd(7700)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(7700)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 10:59:30.673
            [ohasd(7700)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 10:59:39.495
            [ohasd(7700)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 10:59:46.982
            [ohasd(7700)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 10:59:48.952
            [cssd(8736)]CRS-1713:CSSD daemon is started in clustered mode
            [client(8867)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
            [client(8899)]CRS-10001:ACFS-9322: done.
            2012-02-27 11:09:47.529
            [u01/app/grid/11.2.0/bin/cssdagent(8719)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 11:09:47.541
            [cssd(8736)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 11:09:52.656
            [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 11:09:52.907
            [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 11:10:04.650
            [cssd(10870)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 11:10:13.976
            [ohasd(7700)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 11:10:13.991
            [ohasd(7700)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 11:20:03.652
            [u01/app/grid/11.2.0/bin/cssdagent(10855)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 11:20:03.653
            [cssd(10870)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 11:20:08.772
            [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 11:20:09.376
            [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 11:20:09.843
            [ohasd(7700)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 11:20:21.085
            [cssd(11526)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 11:30:20.129
            [u01/app/grid/11.2.0/bin/cssdagent(11505)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 11:30:20.129
            [cssd(11526)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 11:30:25.250
            [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 11:30:25.750
            [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 11:30:46.002
            [ohasd(7700)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 11:30:46.002
            [ohasd(7700)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 11:34:17.647
            [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 12:17:39.626
            [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 12:24:28.387
            [ohasd(7700)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 12:28:07.206
            [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 12:42:24.495
            [u01/app/grid/11.2.0/bin/orarootagent.bin(8547)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
            2012-02-27 12:43:04.550
            [ohasd(16605)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 12:43:05.167
            [ohasd(16605)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(16605)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 12:43:05.204
            [ohasd(16605)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 12:43:05.848
            [ohasd(16605)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 12:43:08.068
            [ohasd(16605)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 12:43:09.529
            [cssd(16802)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 12:45:23.046
            [u01/app/grid/11.2.0/bin/cssdagent(16784)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 12:45:23.046
            [u01/app/grid/11.2.0/bin/cssdmonitor(16767)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
            2012-02-27 12:45:23.046
            [u01/app/grid/11.2.0/bin/orarootagent.bin(16654)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
            2012-02-27 12:45:23.367
            [ohasd(16888)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 12:45:23.378
            [ohasd(16888)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(16888)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 12:45:23.378
            [ohasd(16888)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 12:45:24.057
            [ohasd(16888)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 12:45:28.022
            [cssd(17091)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 12:45:58.840
            [u01/app/grid/11.2.0/bin/cssdmonitor(17050)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
            2012-02-27 12:45:58.841
            [u01/app/grid/11.2.0/bin/orarootagent.bin(16934)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
            2012-02-27 12:45:58.841
            [u01/app/grid/11.2.0/bin/cssdagent(17076)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 12:45:59.143
            [ohasd(17110)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 12:45:59.152
            [ohasd(17110)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(17110)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 12:45:59.153
            [ohasd(17110)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 12:45:59.794
            [ohasd(17110)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 12:46:03.468
            [cssd(17306)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 12:56:02.516
            [u01/app/grid/11.2.0/bin/cssdagent(17288)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 12:56:02.516
            [cssd(17306)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 12:56:07.647
            [ohasd(17110)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 12:56:07.737
            [ohasd(17110)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 13:09:31.212
            [ohasd(17886)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 13:09:31.227
            [ohasd(17886)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(17886)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 13:09:31.227
            [ohasd(17886)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 13:09:31.892
            [ohasd(17886)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            [client(18210)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
            [client(18233)]CRS-10001:ACFS-9322: done.
            2012-02-27 13:09:36.112
            [cssd(18110)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 13:19:35.155
            [u01/app/grid/11.2.0/bin/cssdagent(18084)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 13:19:35.156
            [cssd(18110)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 13:19:40.279
            [ohasd(17886)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 13:19:40.979
            [ohasd(17886)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 16:47:08.768
            [ohasd(7642)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 16:47:12.116
            [ohasd(7642)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(7642)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 16:47:13.020
            [ohasd(7642)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 16:47:16.802
            [ohasd(7642)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 16:47:20.619
            [ohasd(7642)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 16:47:22.453
            [cssd(8493)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 16:57:21.198
            [u01/app/grid/11.2.0/bin/cssdagent(8463)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 16:57:21.206
            [cssd(8493)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 16:57:26.317
            [ohasd(7642)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 16:57:26.671
            [ohasd(7642)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            [client(10085)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
            [client(10129)]CRS-10001:ACFS-9322: done.
            2012-02-27 16:57:38.417
            [cssd(10172)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 16:57:47.784
            [ohasd(7642)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 16:57:47.824
            [ohasd(7642)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 17:07:37.432
            [u01/app/grid/11.2.0/bin/cssdagent(10153)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 17:07:37.433
            [cssd(10172)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 17:07:42.554
            [ohasd(7642)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 17:07:43.157
            [ohasd(7642)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 17:37:54.170
            [ohasd(7642)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 18:07:34.510
            [ohasd(7642)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
            2012-02-27 18:08:05.305
            [u01/app/grid/11.2.0/bin/orarootagent.bin(8497)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
            2012-02-27 18:08:05.305
            [u01/app/grid/11.2.0/bin/cssdmonitor(10879)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
            2012-02-27 18:08:35.368
            [ohasd(14957)]CRS-2112:The OLR service started on node cracdb1.
            2012-02-27 18:08:36.097
            [ohasd(14957)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
            [ohasd(14957)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
            2012-02-27 18:08:36.148
            [ohasd(14957)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
            2012-02-27 18:08:36.647
            [ohasd(14957)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
            2012-02-27 18:08:38.855
            [ohasd(14957)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
            2012-02-27 18:08:40.327
            [cssd(15155)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 18:18:39.373
            [u01/app/grid/11.2.0/bin/cssdagent(15137)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 18:18:39.374
            [cssd(15155)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 18:18:44.492
            [ohasd(14957)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 18:18:44.893
            [ohasd(14957)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 18:18:56.602
            [cssd(15292)]CRS-1713:CSSD daemon is started in clustered mode
            2012-02-27 18:19:05.457
            [ohasd(14957)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
            2012-02-27 18:19:05.458
            [ohasd(14957)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
            2012-02-27 18:28:55.646
            [u01/app/grid/11.2.0/bin/cssdagent(15277)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
            2012-02-27 18:28:55.647
            [cssd(15292)]CRS-1603:CSSD on node racdb01shutdown by user.
            2012-02-27 18:29:00.767
            [ohasd(14957)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
            2012-02-27 18:29:01.671
            [ohasd(14957)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
            2012-02-27 18:29:01.699
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 18:59:07.361
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 19:29:14.018
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 19:59:20.654
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 20:29:27.357
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 20:59:33.034
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 21:29:39.716
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 21:59:46.421
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 22:29:53.107
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 22:59:59.803
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-27 23:30:06.516
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 00:00:13.191
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 00:30:18.876
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 01:00:24.548
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 01:30:31.256
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 02:00:36.928
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 02:30:42.603
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 03:00:49.301
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 03:31:16.664
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 04:01:23.393
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 04:31:30.096
            [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
            2012-02-28 04:58:34.063
            [ohasd(14957)]CRS-2758:Resource 'ora.crsd' is in an unknown state.

            Edited by: user7490278 on Feb 28, 2012 2:21 AM
            • 3. Re: cluster not comming up.
              916154
              HI!

              Why you present all log, you can't select ERRORS?

              You look at

              /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log

              /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
              • 4. Re: cluster not comming up.
                user7490278
                Thanks for you reply:-


                Please have a look to below log...

                error from ohasd.log

                2012-02-29 06:21:11.496: [   CRSPE][1352067392] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
                2012-02-29 06:21:13.426: [ CRSCOMM][1082464576][FFAIL] Listener got clsc error 11 for memNum. 42
                2012-02-29 06:21:16.411: [ CRSCOMM][1082464576][FFAIL] Listener got clsc error 11 for memNum. 40
                2012-02-29 06:56:11.530: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 2
                2012-02-29 06:56:11.548: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 3
                2012-02-29 07:06:17.213: [   CRSPE][1356552512] Start action failed with error code: 3
                2012-02-29 07:06:22.578: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 6

                output from:- oracssdagent_root.log

                2012-02-29 07:37:12.628: [ CSSCLNT][1087777088]clsssInitNative: Init for agent
                2012-02-29 07:37:12.630: [ CSSCLNT][1087777088]clssscConnect: gipc request failed with 29 (0x16)
                2012-02-29 07:37:12.630: [ CSSCLNT][1087777088]clsssInitNative: connect failed, rc 29
                2012-02-29 07:37:13.632: [ CSSCLNT][1087777088]clsssInitNative: Init for agent
                2012-02-29 07:37:13.634: [ CSSCLNT][1087777088]clssscConnect: gipc request failed with 29 (0x16)
                2012-02-29 07:37:13.634: [ CSSCLNT][1087777088]clsssInitNative: connect failed, rc 29
                • 5. Re: cluster not comming up.
                  916154
                  HI!

                  Please try

                  crsctl start crs - under root

                  what you see in alert log?
                  • 6. Re: cluster not comming up.
                    975382
                    Can you tell me how to solve this problem? I
                    I have the same problem