This discussion is archived
6 Replies Latest reply: Nov 21, 2012 12:16 AM by 975382 RSS

cluster not comming up.

user7490278 Newbie
Currently Being Moderated
$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
[oracle@~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online

Edited by: user7490278 on Feb 27, 2012 8:28 AM
  • 1. Re: cluster not comming up.
    CKPT Guru
    Currently Being Moderated
    user7490278 wrote:
    $ crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4533: Event Manager is online
    [oracle@~]$ crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4533: Event Manager is online

    Edited by: user7490278 on Feb 27, 2012 8:28 AM
    What is CRS version?
    Have you been referred CRS log files? What they are saying
  • 2. Re: cluster not comming up.
    user7490278 Newbie
    Currently Being Moderated
    cluster version is 11gr2

    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4533: Event Manager is online



    LOG INFO:=

    [oracle@racdb01cracdb1]$ cat alertcracdb1.log |tail -400
    2012-02-27 07:28:00.541
    [cssd(9203)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 07:28:05.651
    [ohasd(8179)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 07:28:06.307
    [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 07:28:18.077
    [cssd(10751)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 07:28:26.778
    [ohasd(8179)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 07:28:26.818
    [ohasd(8179)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 07:38:17.066
    [u01/app/grid/11.2.0/bin/cssdagent(10736)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 07:38:17.066
    [cssd(10751)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 07:38:22.187
    [ohasd(8179)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 07:38:22.388
    [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 07:38:22.499
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 08:08:28.170
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 08:38:33.842
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 09:08:39.517
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 09:22:39.192
    [ohasd(8179)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 09:43:47.740
    [ohasd(8179)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 09:54:40.546
    [ohasd(8179)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 09:55:17.826
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 10:25:24.528
    [ohasd(8179)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 10:32:28.511
    [u01/app/grid/11.2.0/bin/cssdmonitor(20275)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-02-27 10:36:36.046
    [ohasd(7636)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 10:36:38.300
    [ohasd(7636)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(7636)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 10:36:39.056
    [ohasd(7636)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 10:36:42.921
    [ohasd(7636)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 10:36:50.313
    [ohasd(7636)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:36:52.217
    [cssd(8524)]CRS-1713:CSSD daemon is started in clustered mode
    [client(8612)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(8650)]CRS-10001:ACFS-9322: done.
    2012-02-27 10:46:50.824
    [u01/app/grid/11.2.0/bin/cssdagent(8507)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 10:46:50.825
    [cssd(8524)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 10:46:55.944
    [ohasd(7636)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 10:46:56.508
    [ohasd(7636)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 10:47:08.236
    [cssd(10647)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 10:47:17.234
    [ohasd(7636)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 10:47:17.269
    [ohasd(7636)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 10:54:44.707
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:45.714
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:46.722
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:47.729
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:48.736
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:49.743
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:50.750
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:51.757
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:52.764
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:53.771
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:54.778
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:55.785
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:56.792
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:57.799
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:58.806
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:54:59.813
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:00.820
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:01.827
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:02.834
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:03.842
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:04.849
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:05.856
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:06.864
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:07.872
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:08.879
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:09.887
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:10.894
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:11.902
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:12.910
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:13.916
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:14.923
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:15.931
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:16.938
    [cssd(10647)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:55:17.707
    [u01/app/grid/11.2.0/bin/cssdmonitor(10617)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-02-27 10:59:27.293
    [ohasd(7700)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 10:59:30.329
    [ohasd(7700)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(7700)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 10:59:30.673
    [ohasd(7700)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 10:59:39.495
    [ohasd(7700)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 10:59:46.982
    [ohasd(7700)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 10:59:48.952
    [cssd(8736)]CRS-1713:CSSD daemon is started in clustered mode
    [client(8867)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(8899)]CRS-10001:ACFS-9322: done.
    2012-02-27 11:09:47.529
    [u01/app/grid/11.2.0/bin/cssdagent(8719)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 11:09:47.541
    [cssd(8736)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 11:09:52.656
    [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 11:09:52.907
    [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 11:10:04.650
    [cssd(10870)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 11:10:13.976
    [ohasd(7700)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 11:10:13.991
    [ohasd(7700)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 11:20:03.652
    [u01/app/grid/11.2.0/bin/cssdagent(10855)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 11:20:03.653
    [cssd(10870)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 11:20:08.772
    [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 11:20:09.376
    [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 11:20:09.843
    [ohasd(7700)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 11:20:21.085
    [cssd(11526)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 11:30:20.129
    [u01/app/grid/11.2.0/bin/cssdagent(11505)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 11:30:20.129
    [cssd(11526)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 11:30:25.250
    [ohasd(7700)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 11:30:25.750
    [ohasd(7700)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 11:30:46.002
    [ohasd(7700)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 11:30:46.002
    [ohasd(7700)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 11:34:17.647
    [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 12:17:39.626
    [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 12:24:28.387
    [ohasd(7700)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 12:28:07.206
    [ohasd(7700)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 12:42:24.495
    [u01/app/grid/11.2.0/bin/orarootagent.bin(8547)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
    2012-02-27 12:43:04.550
    [ohasd(16605)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 12:43:05.167
    [ohasd(16605)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(16605)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 12:43:05.204
    [ohasd(16605)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 12:43:05.848
    [ohasd(16605)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 12:43:08.068
    [ohasd(16605)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 12:43:09.529
    [cssd(16802)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 12:45:23.046
    [u01/app/grid/11.2.0/bin/cssdagent(16784)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 12:45:23.046
    [u01/app/grid/11.2.0/bin/cssdmonitor(16767)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-02-27 12:45:23.046
    [u01/app/grid/11.2.0/bin/orarootagent.bin(16654)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
    2012-02-27 12:45:23.367
    [ohasd(16888)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 12:45:23.378
    [ohasd(16888)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(16888)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 12:45:23.378
    [ohasd(16888)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 12:45:24.057
    [ohasd(16888)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 12:45:28.022
    [cssd(17091)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 12:45:58.840
    [u01/app/grid/11.2.0/bin/cssdmonitor(17050)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-02-27 12:45:58.841
    [u01/app/grid/11.2.0/bin/orarootagent.bin(16934)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
    2012-02-27 12:45:58.841
    [u01/app/grid/11.2.0/bin/cssdagent(17076)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 12:45:59.143
    [ohasd(17110)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 12:45:59.152
    [ohasd(17110)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(17110)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 12:45:59.153
    [ohasd(17110)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 12:45:59.794
    [ohasd(17110)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 12:46:03.468
    [cssd(17306)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 12:56:02.516
    [u01/app/grid/11.2.0/bin/cssdagent(17288)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 12:56:02.516
    [cssd(17306)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 12:56:07.647
    [ohasd(17110)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 12:56:07.737
    [ohasd(17110)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 13:09:31.212
    [ohasd(17886)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 13:09:31.227
    [ohasd(17886)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(17886)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 13:09:31.227
    [ohasd(17886)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 13:09:31.892
    [ohasd(17886)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    [client(18210)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(18233)]CRS-10001:ACFS-9322: done.
    2012-02-27 13:09:36.112
    [cssd(18110)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 13:19:35.155
    [u01/app/grid/11.2.0/bin/cssdagent(18084)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 13:19:35.156
    [cssd(18110)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 13:19:40.279
    [ohasd(17886)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 13:19:40.979
    [ohasd(17886)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 16:47:08.768
    [ohasd(7642)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 16:47:12.116
    [ohasd(7642)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(7642)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 16:47:13.020
    [ohasd(7642)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 16:47:16.802
    [ohasd(7642)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 16:47:20.619
    [ohasd(7642)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 16:47:22.453
    [cssd(8493)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 16:57:21.198
    [u01/app/grid/11.2.0/bin/cssdagent(8463)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 16:57:21.206
    [cssd(8493)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 16:57:26.317
    [ohasd(7642)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 16:57:26.671
    [ohasd(7642)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    [client(10085)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(10129)]CRS-10001:ACFS-9322: done.
    2012-02-27 16:57:38.417
    [cssd(10172)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 16:57:47.784
    [ohasd(7642)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 16:57:47.824
    [ohasd(7642)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 17:07:37.432
    [u01/app/grid/11.2.0/bin/cssdagent(10153)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 17:07:37.433
    [cssd(10172)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 17:07:42.554
    [ohasd(7642)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 17:07:43.157
    [ohasd(7642)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 17:37:54.170
    [ohasd(7642)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 18:07:34.510
    [ohasd(7642)]CRS-2758:Resource 'ora.crsd' is in an unknown state.
    2012-02-27 18:08:05.305
    [u01/app/grid/11.2.0/bin/orarootagent.bin(8497)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/orarootagent_root/orarootagent_root.log.
    2012-02-27 18:08:05.305
    [u01/app/grid/11.2.0/bin/cssdmonitor(10879)]CRS-5822:Agent '/u01/app/grid/11.2.0/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
    2012-02-27 18:08:35.368
    [ohasd(14957)]CRS-2112:The OLR service started on node cracdb1.
    2012-02-27 18:08:36.097
    [ohasd(14957)]CRS-8011:reboot advisory message from host: cracdb1, component: mo065050, with time stamp: L-2012-02-27-07:13:42.057
    [ohasd(14957)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    2012-02-27 18:08:36.148
    [ohasd(14957)]CRS-8017:location: /etc/oracle/lastgasp has 84 reboot advisory log files, 1 were announced and 0 errors occurred
    2012-02-27 18:08:36.647
    [ohasd(14957)]CRS-2772:Server 'cracdb1' has been assigned to pool 'Free'.
    2012-02-27 18:08:38.855
    [ohasd(14957)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2012-02-27 18:08:40.327
    [cssd(15155)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 18:18:39.373
    [u01/app/grid/11.2.0/bin/cssdagent(15137)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 18:18:39.374
    [cssd(15155)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 18:18:44.492
    [ohasd(14957)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 18:18:44.893
    [ohasd(14957)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 18:18:56.602
    [cssd(15292)]CRS-1713:CSSD daemon is started in clustered mode
    2012-02-27 18:19:05.457
    [ohasd(14957)]CRS-2765:Resource 'ora.diskmon' has failed on server racdb01
    2012-02-27 18:19:05.458
    [ohasd(14957)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
    2012-02-27 18:28:55.646
    [u01/app/grid/11.2.0/bin/cssdagent(15277)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
    2012-02-27 18:28:55.647
    [cssd(15292)]CRS-1603:CSSD on node racdb01shutdown by user.
    2012-02-27 18:29:00.767
    [ohasd(14957)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log.
    2012-02-27 18:29:01.671
    [ohasd(14957)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server racdb01
    2012-02-27 18:29:01.699
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 18:59:07.361
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 19:29:14.018
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 19:59:20.654
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 20:29:27.357
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 20:59:33.034
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 21:29:39.716
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 21:59:46.421
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 22:29:53.107
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 22:59:59.803
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-27 23:30:06.516
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 00:00:13.191
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 00:30:18.876
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 01:00:24.548
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 01:30:31.256
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 02:00:36.928
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 02:30:42.603
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 03:00:49.301
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 03:31:16.664
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 04:01:23.393
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 04:31:30.096
    [ohasd(14957)]CRS-2765:Resource 'ora.crsd' has failed on server racdb01
    2012-02-28 04:58:34.063
    [ohasd(14957)]CRS-2758:Resource 'ora.crsd' is in an unknown state.

    Edited by: user7490278 on Feb 28, 2012 2:21 AM
  • 3. Re: cluster not comming up.
    916154 Newbie
    Currently Being Moderated
    HI!

    Why you present all log, you can't select ERRORS?

    You look at

    /u01/app/grid/11.2.0/log/cracdb1/ohasd/ohasd.log

    /u01/app/grid/11.2.0/log/cracdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
  • 4. Re: cluster not comming up.
    user7490278 Newbie
    Currently Being Moderated
    Thanks for you reply:-


    Please have a look to below log...

    error from ohasd.log

    2012-02-29 06:21:11.496: [   CRSPE][1352067392] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
    2012-02-29 06:21:13.426: [ CRSCOMM][1082464576][FFAIL] Listener got clsc error 11 for memNum. 42
    2012-02-29 06:21:16.411: [ CRSCOMM][1082464576][FFAIL] Listener got clsc error 11 for memNum. 40
    2012-02-29 06:56:11.530: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 2
    2012-02-29 06:56:11.548: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 3
    2012-02-29 07:06:17.213: [   CRSPE][1356552512] Start action failed with error code: 3
    2012-02-29 07:06:22.578: [ CRSCOMM][1339742528][FFAIL] Listener got clsc error 11 for memNum. 6

    output from:- oracssdagent_root.log

    2012-02-29 07:37:12.628: [ CSSCLNT][1087777088]clsssInitNative: Init for agent
    2012-02-29 07:37:12.630: [ CSSCLNT][1087777088]clssscConnect: gipc request failed with 29 (0x16)
    2012-02-29 07:37:12.630: [ CSSCLNT][1087777088]clsssInitNative: connect failed, rc 29
    2012-02-29 07:37:13.632: [ CSSCLNT][1087777088]clsssInitNative: Init for agent
    2012-02-29 07:37:13.634: [ CSSCLNT][1087777088]clssscConnect: gipc request failed with 29 (0x16)
    2012-02-29 07:37:13.634: [ CSSCLNT][1087777088]clsssInitNative: connect failed, rc 29
  • 5. Re: cluster not comming up.
    916154 Newbie
    Currently Being Moderated
    HI!

    Please try

    crsctl start crs - under root

    what you see in alert log?
  • 6. Re: cluster not comming up.
    975382 Newbie
    Currently Being Moderated
    Can you tell me how to solve this problem? I
    I have the same problem

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points