2 Replies Latest reply: Aug 22, 2012 11:59 PM by 415289 RSS

    CRS-4535: Cannot communicate with Cluster Ready Services

    user11952526
      Hi,

      I have installed 11gR2 two nodes RAC on OEL5.2 64-bit using Oracle VM. Installation went fine. Second Node is working fine.

      After installation in Node# 1 +ASM1 is NOT comming up. I do NOT see the Disk Group visible from Node one. Before installation it was visible.

      I have executed these commands in Node one. Please help me to fix this issue. Please let me know if you need and information...
      FYI...
      ------------------------------------------------------------------------------------------------------------------------------------------------------------
      [root@rac1 bin]# */etc/init.d/oracleasm listdisks*

      [root@rac1 bin]# */etc/init.d/oracleasm scandisks*
      Scanning system for ASM disks: [  OK  ]

      [root@rac1 bin]# /etc/init.d/oracleasm listdisks
      [root@rac1 bin]#


      [root@rac1 bin]# *./crsctl start crs*
      CRS-4640: Oracle High Availability Services is already active
      CRS-4000: Command Start failed, or completed with errors.


      [root@rac1 bin]# *./crsctl start cluster*
      CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
      CRS-2674: Start of 'ora.cssd' on 'rac1' failed
      CRS-2679: Attempting to clean 'ora.cssd' on 'rac1'
      CRS-2681: Clean of 'ora.cssd' on 'rac1' succeeded
      CRS-4000: Command Start failed, or completed with errors.


      [root@rac1 bin]# pwd
      /u01/app/11.2.0/grid/bin


      [root@rac1 bin]# *./crsctl check crs*
      CRS-4638: Oracle High Availability Services is online
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
      CRS-4533: Event Manager is online


      [root@rac1 bin]# *./crsctl stat res -t*
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4000: Command Status failed, or completed with errors.


      [root@rac1 bin]# *./crsctl check cluster -all*
      **************************************************************
      rac1:
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
      CRS-4533: Event Manager is online
      **************************************************************
      rac2:
      CRS-4537: Cluster Ready Services is online
      CRS-4529: Cluster Synchronization Services is online
      CRS-4533: Event Manager is online
      **************************************************************
      [root@rac1 bin]# *./crsctl stat res -t -init*
      --------------------------------------------------------------------------------
      NAME TARGET STATE SERVER STATE_DETAILS
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.asm
      1 ONLINE OFFLINE
      ora.crsd
      1 ONLINE INTERMEDIATE rac1
      ora.cssd
      1 ONLINE OFFLINE
      ora.cssdmonitor
      1 ONLINE ONLINE rac1
      ora.ctssd
      1 ONLINE OFFLINE
      ora.diskmon
      1 ONLINE ONLINE rac1
      ora.drivers.acfs
      1 ONLINE ONLINE rac1
      ora.evmd
      1 ONLINE ONLINE rac1
      ora.gipcd
      1 ONLINE ONLINE rac1
      ora.gpnpd
      1 ONLINE ONLINE rac1
      ora.mdnsd
      1 ONLINE ONLINE rac1

      [root@rac1 bin]# *./crsctl status resource*
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4000: Command Status failed, or completed with errors.


      [root@rac1 bin]# *./crsctl query css votedisk*
      Unable to communicate with the Cluster Synchronization Services daemon.


      [root@rac1 log]# ps -ef|grep -i oracle
      oracle 6066 1 0 10:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
      oracle 6098 1 0 10:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
      oracle 6104 1 0 10:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
      oracle 6118 1 0 10:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
      oracle 6579 1 0 10:12 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
      oracle 18138 1 0 13:33 ? 00:00:00 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f


      [root@rac1 log]# cat /u01/app/11.2.0/grid/log/rac1/ohasd/ohasd.log |grep -i error
      2012-08-11 00:51:18.398: [UiServer][1353296192] S(0x1a9c8dd0): Error sending msg over socket.6
      2012-08-11 00:52:12.697: [ CRSCOMM][1332283712][FFAIL] Listener got clsc error 11 for memNum. 1
      2012-08-11 00:52:13.378: [ CRSCOMM][1332283712][FFAIL] Listener got clsc error 11 for memNum. 4
      2012-08-11 00:52:13.947: [ CRSCOMM][1332283712][FFAIL] Listener got clsc error 11 for memNum. 2
      2012-08-11 00:53:18.195: [UiServer][1353296192] S(0x1aa64d40): Error sending msg over socket.6
      2012-08-11 02:29:49.749: [ CRSCOMM][1332283712][FFAIL] Listener got clsc error 11 for memNum. 6
      2012-08-11 02:30:07.886: [ CRSCOMM][1332283712][FFAIL] Listener got clsc error 11 for memNum. 7
      2012-08-11 21:08:09.398: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 2
      2012-08-11 21:08:10.335: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 1
      2012-08-11 21:08:11.609: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 3
      2012-08-11 21:08:11.610: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 4
      2012-08-11 21:08:24.116: [   CRSPE][1357756736] Start action failed with error code: 2
      2012-08-11 21:08:34.351: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 8
      2012-08-11 21:08:34.372: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 7
      2012-08-11 21:08:34.546: [   CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-11 21:08:36.523: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 9
      2012-08-11 21:08:48.128: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 10
      2012-08-11 21:17:36.861: [   CRSPE][1357756736] Sequencer for [ora.crsd 1 1] has completed with error: CRS-0216: Could not stop resource 'ora.crsd'.
      2012-08-11 21:18:42.401: [   CRSPE][1357756736] Start action failed with error code: 3
      2012-08-11 21:18:47.470: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 11
      2012-08-11 21:18:47.670: [   CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-11 21:18:49.554: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 13
      2012-08-11 21:19:02.319: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 12
      2012-08-11 21:25:54.587: [   CRSPE][1357756736] Sequencer for [ora.crsd 1 1] has completed with error: CRS-0216: Could not stop resource 'ora.crsd'.
      2012-08-12 01:01:24.082: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 2
      2012-08-12 01:01:24.147: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 1
      2012-08-12 01:01:25.951: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 3
      2012-08-12 01:01:25.961: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 4
      2012-08-12 01:01:26.760: [  CRSCCL][1331632448]Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
      2012-08-12 01:01:35.249: [   CRSPE][1354729792] Start action failed with error code: 2
      2012-08-12 01:01:40.362: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 8
      2012-08-12 01:01:40.560: [   CRSPE][1354729792] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 01:01:42.442: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 9
      2012-08-12 01:01:45.094: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 7
      2012-08-12 01:11:51.833: [   CRSPE][1354729792] Start action failed with error code: 3
      2012-08-12 01:11:56.949: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 11
      2012-08-12 01:11:57.150: [   CRSPE][1354729792] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 01:11:59.031: [ CRSCOMM][1084680512][FFAIL] Listener got clsc error 11 for memNum. 12
      2012-08-12 06:17:34.849: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 2
      2012-08-12 06:17:36.144: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 1
      2012-08-12 06:17:38.019: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 3
      2012-08-12 06:17:38.020: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 4
      2012-08-12 06:17:40.661: [  CRSCCL][1322793280]Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
      2012-08-12 06:17:49.049: [   CRSPE][1350093120] Start action failed with error code: 2
      2012-08-12 06:17:50.564: [   CRSPE][1350093120] Sequencer for [ora.diskmon 1 1] has completed with error: CRS-0215: Could not start resource 'ora.diskmon'.
      2012-08-12 06:17:54.215: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 8
      2012-08-12 06:17:54.413: [   CRSPE][1350093120] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 06:17:56.406: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 9
      2012-08-12 06:17:58.730: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 7
      2012-08-12 06:28:05.009: [   CRSPE][1350093120] Start action failed with error code: 3
      2012-08-12 06:28:10.127: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 11
      2012-08-12 06:28:10.342: [   CRSPE][1350093120] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 06:28:12.221: [ CRSCOMM][1091606848][FFAIL] Listener got clsc error 11 for memNum. 12
      2012-08-12 06:33:49.163: [   CRSPE][1350093120] Sequencer for [ora.crsd 1 1] has completed with error: CRS-0216: Could not stop resource 'ora.crsd'.
      2012-08-12 10:02:16.427: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 3
      2012-08-12 10:02:16.879: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 4
      2012-08-12 10:02:16.926: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 1
      2012-08-12 10:02:16.928: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 2
      2012-08-12 10:02:22.435: [  CRSCCL][1318725952]Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
      2012-08-12 10:02:30.895: [   CRSPE][1350228288] Start action failed with error code: 2
      2012-08-12 10:02:41.027: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 8
      2012-08-12 10:02:41.028: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 7
      2012-08-12 10:02:41.211: [   CRSPE][1350228288] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 10:02:43.196: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 9
      2012-08-12 10:02:54.890: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 10
      2012-08-12 10:12:49.158: [   CRSPE][1350228288] Start action failed with error code: 3
      2012-08-12 10:12:54.277: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 11
      2012-08-12 10:12:54.478: [   CRSPE][1350228288] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 10:12:56.470: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 13
      2012-08-12 11:26:04.125: [   CRSPE][1350228288] Sequencer for [ora.crsd 1 1] has completed with error: CRS-0216: Could not stop resource 'ora.crsd'.
      2012-08-12 11:27:09.412: [   CRSPE][1350228288] Start action failed with error code: 3
      2012-08-12 11:27:09.413: [UiServer][1354430784] S(0x1b30c020): Error sending msg over socket.6
      2012-08-12 11:27:14.524: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 14
      2012-08-12 11:27:14.721: [   CRSPE][1350228288] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 11:27:16.657: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 15
      2012-08-12 11:38:56.397: [   CRSPE][1350228288] Start action failed with error code: 3
      2012-08-12 11:39:01.522: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 16
      2012-08-12 11:39:01.735: [   CRSPE][1350228288] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 11:39:03.642: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 17
      2012-08-12 11:49:09.151: [   CRSPE][1350228288] Start action failed with error code: 3
      2012-08-12 11:49:14.263: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 18
      2012-08-12 11:49:14.461: [   CRSPE][1350228288] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
      2012-08-12 11:49:16.345: [ CRSCOMM][1333418304][FFAIL] Listener got clsc error 11 for memNum. 19
      [root@rac1 log]# cat /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log |grep -i error
      [root@rac1 log]# tail -25 /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log |grep -i failed
      2012-08-12 11:49:15.317: [ CSSCLNT][1096005952]clssscConnect: gipc request failed with 29 (0x16)
      2012-08-12 11:49:15.317: [ CSSCLNT][1096005952]clsssInitNative: connect failed, rc 29
      2012-08-12 11:49:16.327: [ CSSCLNT][1096005952]clssscConnect: gipc request failed with 29 (0x16)
      2012-08-12 11:49:16.327: [ CSSCLNT][1096005952]clsssInitNative: connect failed, rc 29

      ------------------------------------------------------------------------------------------------------------------------------------------------------------

      Thanks.
      Joy.

      Edited by: user11952526 on Aug 12, 2012 1:43 PM