1 2 Previous Next 19 Replies Latest reply: May 14, 2014 12:20 AM by hitgon RSS

    CRS-4535: Cannot communicate with Cluster Ready Services

    hitgon

      /u01/app/11.2.0/grid/bin/crsctl check crs

      CRS-4638: Oracle High Availability Services is online

      CRS-4535: Cannot communicate with Cluster Ready Services

      CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

      CRS-4534: Cannot communicate with Event Manager

       

      cluster alert logs w

      [/u01/app/11.2.0/grid/bin/oraagent.bin(17624)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log"

       

      cat /u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log | grep CLSN00006

      2014-04-15 18:38:51.450: [ora.asm][2691127040] {0:18:3} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0

      2014-04-15 18:38:59.163: [ora.asm][2688231168] {0:5:48} [clean] (:CLSN00006:)AsmAgent::check failed gimh state 0

      2014-04-15 18:38:59.356: [ora.asm][2688231168] {0:5:48} [clean] (:CLSN00006:)InstAgent::check: return unplanned offline

      2014-04-15 18:38:59.554: [ora.asm][2688231168] {0:5:48} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0

      2014-04-15 18:38:59.721: [ora.asm][2688231168] {0:5:48} [check] (:CLSN00006:)InstAgent::check: return unplanned offline

      2014-04-15 19:20:18.880: [ora.asm][1738524416] {0:0:2} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0

      2014-04-15 19:52:05.843: [ora.asm][2696546048] {0:0:2} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0

      2014-04-15 20:12:57.733: [ora.asm][581101312] {0:0:2} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0

        • 1. Re: CRS-4535: Cannot communicate with Cluster Ready Services
          Vandana B -Oracle

          Hi,

           

          Looks like the clusterware stack has not come up. What is the output of following?

           

          ++ crsctl stat res -t -init

          ++ ps -ef | grep d.bin

           

           

          ++ are the underlying voting devices and ocr devices available on the server and are the permissions on the same set correctly?

          ++ Was any change implemented in the environment, post which this issue occurred?

           

          Regards,

          Vandana - Oracle

          • 2. Re: CRS-4535: Cannot communicate with Cluster Ready Services
            hitgon

            Hi Vandana,

             

             

            We are facing same issue, And below is the detail you have asked for.

             

            ls -ltr

            total 0

            brw-rw---- 1 oracle asmdba 8, 80 Apr 15 20:46 B12ORAFRA

            brw-rw---- 1 oracle asmdba 8, 64 Apr 15 20:46 B12ORADATA

            brw-rw---- 1 oracle asmdba 8, 48 Apr 15 20:46 B12ORASYS

            brw-rw---- 1 oracle asmdba 8,  0 Apr 15 23:18 B12OCRVOTE3

            brw-rw---- 1 oracle asmdba 8, 16 Apr 15 23:18 B12OCRVOTE2

            brw-rw---- 1 oracle asmdba 8, 32 Apr 15 23:18 B12OCRVOTE1

            disks]$

             

             

             

             

            crsctl stat res -t -init

            --------------------------------------------------------------------------------

            NAME           TARGET  STATE        SERVER                   STATE_DETAILS    

            --------------------------------------------------------------------------------

            Cluster Resources

            --------------------------------------------------------------------------------

            ora.asm

                  1        ONLINE  OFFLINE                                                

            ora.cluster_interconnect.haip

                  1        ONLINE  OFFLINE                                                

            ora.crf

                  1        ONLINE  ONLINE       nodes01                               

            ora.crsd

                  1        ONLINE  OFFLINE                                                

            ora.cssd

                  1        ONLINE  OFFLINE                               STARTING         

            ora.cssdmonitor

                  1        ONLINE  ONLINE       nodes01                               

            ora.ctssd

                  1        ONLINE  OFFLINE                                                

            ora.diskmon

                  1        OFFLINE OFFLINE                                                

            ora.evmd

                  1        ONLINE  OFFLINE                                                

            ora.gipcd

                  1        ONLINE  ONLINE       nodes01                               

            ora.gpnpd

                  1        ONLINE  ONLINE       nodes01                               

            ora.mdnsd

                  1        ONLINE  ONLINE       nodes01

             

             

             

            [oracle@nodes01 ~]$ ps -ef | grep d.bin

            root     13964     1  0 22:49 ?        00:00:06 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

            oracle   14078     1  0 22:49 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin

            oracle   14090     1  0 22:49 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin

            oracle   14100     1  0 22:49 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin

            root     14110     1  0 22:49 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin

            oracle   14113     1  0 22:49 ?        00:00:02 /u01/app/11.2.0/grid/bin/gipcd.bin

            root     14127     1  2 22:49 ?        00:00:46 /u01/app/11.2.0/grid/bin/osysmond.bin

            root     14203     1  0 22:49 ?        00:00:01 /u01/app/11.2.0/grid/bin/ologgerd -m ecchpcrfp01 -r -d /u01/app/11.2.0/grid/crf/db/nodes01

            root     16315     1  0 23:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor

            root     16340     1  0 23:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdagent

            oracle   16352     1  0 23:20 ?        00:00:00 /u01/app/11.2.0/grid/bin/ocssd.bin

            oracle   16449 14911  0 23:21 pts/0    00:00:00 grep d.bin

             

            please guide us

            • 3. Re: CRS-4535: Cannot communicate with Cluster Ready Services
              Vandana B -Oracle

              Hi,

               

              Looks like cssd daemon is not coming up. What do you see in the ocssd.log?

               

              Regards,

              Vandana - Oracle

              • 4. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                hitgon

                Hi Please find the ocssd log

                 

                2014-04-16 00:17:09.717: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                2014-04-16 00:17:09.717: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(10/0x7f335001e180)

                2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001e180)

                2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x570

                2014-04-16 00:17:09.904: [    CLSF][1492350720]Allocated CLSF context

                2014-04-16 00:17:09.905: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:09.905: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:10.718: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                2014-04-16 00:17:10.718: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(11/0x7f335001ed40)

                2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001ed40)

                2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x586

                2014-04-16 00:17:10.906: [    CLSF][1492350720]Allocated CLSF context

                2014-04-16 00:17:10.907: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:10.907: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:11.719: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                2014-04-16 00:17:11.719: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                2014-04-16 00:17:11.719: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(12/0x7f335001ed40)

                2014-04-16 00:17:11.720: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001ed40)

                2014-04-16 00:17:11.720: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x59c

                2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x7f3350051b20) client((nil))

                2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDeadProc: proc 0x7f3350051b20

                2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDestroyProc: cleaning up proc(0x7f3350051b20) con(0x51c) skgpid  ospid 4740 with 0 clients, refcount 0

                2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x51c

                2014-04-16 00:17:11.878: [    CSSD][1501222656]clssscSelect: cookie accept request 0x126dd80

                2014-04-16 00:17:11.878: [    CSSD][1501222656]clssgmAllocProc: (0x7f3350075a10) allocated

                2014-04-16 00:17:11.878: [    CSSD][1501222656]clssgmClientConnectMsg: properties of cmProc 0x7f3350075a10 - 1,2,3,4,5

                2014-04-16 00:17:11.879: [    CSSD][1501222656]clssgmClientConnectMsg: Connect from con(0x5f0) proc(0x7f3350075a10) pid(4740) version 11:2:1:4, properties: 1,2,3,4,5

                2014-04-16 00:17:11.879: [    CSSD][1501222656]clssgmClientConnectMsg: msg flags 0x0000

                2014-04-16 00:17:11.908: [    CLSF][1492350720]Allocated CLSF context

                2014-04-16 00:17:11.909: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:11.909: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:12.720: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                2014-04-16 00:17:12.720: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                2014-04-16 00:17:12.720: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(13/0x7f335004f700)

                2014-04-16 00:17:12.721: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335004f700)

                2014-04-16 00:17:12.721: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x616

                2014-04-16 00:17:12.909: [    CSSD][1516943104]clssnmlfmtlease: uniqueness 1397587617, gipc addr gipcha://ecchpcrfs01:nm2_ecchpcrf

                2014-04-16 00:17:12.914: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                2014-04-16 00:17:12.916: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                2014-04-16 00:17:12.917: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                2014-04-16 00:17:12.959: [    CSSD][1516943104]clssnmlgetslot:lease acquisition for node ecchpcrfs01/slot 2 completed in 15050 msecs

                2014-04-16 00:17:13.002: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 285466078, wrtcnt, 22645623, LATS 967144, lastSeqNo 0, uniqueness 1394482854, timestamp 1397587632/3103401744

                2014-04-16 00:17:13.002: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 285466077, wrtcnt, 22551574, LATS 967144, lastSeqNo 0, uniqueness 1397587617, timestamp 1397587605/939384

                2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNcopy: Saving DHB uniqueness of 1394482854 for node ecchpcrfp01, number 1

                2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 1, ecchpcrfp01, has a disk HB, but no network HB, DHB has rcfg 285466078, wrtcnt, 22645624, LATS 967224, lastSeqNo 0, uniqueness 1394482854, timestamp 1397587632/3103401754

                2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x16855f0 for disk :/dev/oracleasm/disks/B12OCRVOTE1:

                 

                2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x14f1bb0 for disk :/dev/oracleasm/disks/B12OCRVOTE2:

                 

                2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x14f1fd0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                 

                2014-04-16 00:17:13.091: [    CSSD][1516943104]clssnmInitNodeDB: Initializing with OCR id 0

                2014-04-16 00:17:13.721: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                2014-04-16 00:17:13.721: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                2014-04-16 00:17:13.721: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(14/0x7f3350072680)

                2014-04-16 00:17:13.721: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f3350072680)

                2014-04-16 00:17:13.722: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x642

                • 5. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                  hitgon

                  Please help on below issue 

                   

                  Node-1 is Up and Running 

                  Node-2 is not able to start

                   

                  /u01/app/11.2.0/grid/bin/crsctl check crs

                  CRS-4638: Oracle High Availability Services is online

                  CRS-4535: Cannot communicate with Cluster Ready Services

                  CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

                  CRS-4534: Cannot communicate with Event Manager

                  • 6. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                    teits

                    post relevant output in

                    ORACLE_GI_HOME/log/node2/alertnode2.log

                     

                    check the time between node1 and node2. $date. hope no time differences.

                     

                    can you reboot node2? if not, stop OHAS and start OHAS/CRS. #./crsctl stop has

                    #crsctl start crs

                     

                    HTH

                    Tobi

                    • 7. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                      hitgon

                      Hi,

                       

                      ORACLE_GI_HOME/log/node2/alertnode2.log


                      [/u01/app/11.2.0/grid/bin/oraagent.bin(17624)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log"


                      no hope...


                      We tried to stop and start ohasd and crsctl as well three or four times

                      We also rebooted the server node2 three time but still node2 services are not able to up.

                      • 8. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                        Shivendra Narain Nirala

                        hitgon wrote:

                         

                        Hi Please find the ocssd log

                         

                         

                        2014-04-16 00:17:13.002: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 285466077, wrtcnt, 22551574, LATS 967144, lastSeqNo 0, uniqueness 1397587617, timestamp 1397587605/939384

                        2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNcopy: Saving DHB uniqueness of 1394482854 for node ecchpcrfp01, number 1

                        2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 1, ecchpcrfp01, has a disk HB, but no network HB, DHB has rcfg 285466078, wrtcnt, 22645624, LATS 967224, lastSeqNo 0, uniqueness 1394482854, timestamp 1397587632/3103401754

                        2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x16855f0 for disk :/dev/oracleasm/disks/B12OCRVOTE1:

                         

                         

                        Hi Hitesh

                         

                        Please check your network configuration .

                         

                        Take a look on log file contents node 2, , has a disk HB, but no network HB,

                         

                        it clearly indicates that your network confirmation has been reset, and due to this node 2 is not able to start .

                         

                        Besides . please share the crssd.log then i can provide more input .

                         

                        Regards

                         

                        Shivendra Narain Nirala

                        • 9. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                          Vandana B -Oracle

                          Hi,

                           

                          As mentioned by Shivendra, verify your network setup. To test the same follow the below mentioned document

                           

                          Ref: How to Validate Network and Name Resolution Setup for the Clusterware and RAC (Doc ID 1054902.1)

                           

                          Regards,

                          Vandana - Oracle

                          • 10. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                            hitgon

                            Thanks...

                             

                             

                            tail -f crsd.log

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: SuiteTes  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: UiServer  0

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Checking the OCR device

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Sync-up with OCR

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Connecting to the CSS Daemon

                            2014-04-15 18:38:51.045: [  CRSRTI][354567968] CSS is not ready. Received status 3

                            2014-04-15 18:38:51.051: [ CRSMAIN][354567968] Created alert : (:CRSD00109:) :  Could not init the CSS context, error: 3

                            2014-04-15 18:38:51.051: [    CRSD][354567968][PANIC] CRSD exiting: Could not init the CSS context, error: 3

                            2014-04-15 18:38:51.051: [    CRSD][354567968] Done.

                             

                             

                            ^C

                            [oracle@ecchpcrfs01 crsd]$ tail -100f crsd.log

                            2014-04-15 18:38:42.222: [UiServer][2591119104] {2:57358:3160} Container [ Name: FENCESERVER

                                    API_HDR_VER:

                                    TextMessage[2]

                                    CLIENT:

                                    TextMessage[]

                                    CLIENT_NAME:

                                    TextMessage[ocssd.bin]

                                    CLIENT_PID:

                                    TextMessage[32189]

                                    CLIENT_PRIMARY_GROUP:

                                    TextMessage[oinstall]

                                    LOCALE:

                                    TextMessage[AMERICAN_AMERICA.AL32UTF8]

                            ]

                            2014-04-15 18:38:42.222: [UiServer][2591119104] {2:57358:3160} Sending message to AGFW. ctx= 0x7f1538008f70, Client PID: 32189

                            2014-04-15 18:38:42.222: [UiServer][2591119104] {2:57358:3160} Force-disconnecting [0]  existing PE clients...

                            2014-04-15 18:38:42.222: [UiServer][2591119104] {2:57358:3160} Sending message: 327514 to AGFW proxy server.

                            2014-04-15 18:38:42.357: [    AGFW][2603726592] {2:57358:3160} Agfw Proxy Server received the message: UnKnown[Proxy] ID 20489:327514

                            2014-04-15 18:38:42.700: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.B12ORADATA.dg

                            2014-04-15 18:38:42.700: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.B12ORAFRA.dg

                            2014-04-15 18:38:42.700: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.B12ORASYS.dg

                            2014-04-15 18:38:42.752: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.DATA.dg

                            2014-04-15 18:38:42.986: [    AGFW][2603726592] {2:57358:3160} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER.lsnr ecchpcrfs01 1] ID 4100:327515 to the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:42.986: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.asm

                            2014-04-15 18:38:42.987: [    AGFW][2603726592] {2:57358:3160} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.ecchpcrfs01.vip 1 1] ID 4100:327516 to the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:42.987: [    AGFW][2603726592] {2:57358:3160} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.net1.network ecchpcrfs01 1] ID 4100:327517 to the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:42.987: [   CRSPE][2603726592] {2:57358:3160} Skipping Fence of : ora.nvtxch.db

                            2014-04-15 18:38:42.987: [    AGFW][2603726592] {2:57358:3160} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.ons ecchpcrfs01 1] ID 4100:327518 to the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:43.506: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.ecchpcrfs01.vip 1 1] ID 4100:327516 from the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:43.507: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network ecchpcrfs01 1] ID 4100:327517 from the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:43.509: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network ecchpcrfs01 1] ID 4100:327517 from the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:43.525: [    AGFW][2603726592] {2:57358:3160} Fenced off the resource [ora.net1.network]

                            2014-04-15 18:38:43.525: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.ecchpcrfs01.vip 1 1] ID 4100:327516 from the agent /u01/app/11.2.0/grid/bin/orarootagent_root

                            2014-04-15 18:38:43.525: [    AGFW][2603726592] {2:57358:3160} Fenced off the resource [ora.ecchpcrfs01.vip]

                            2014-04-15 18:38:43.572: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr ecchpcrfs01 1] ID 4100:327515 from the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:44.466: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.ons ecchpcrfs01 1] ID 4100:327518 from the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:44.730: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.ons ecchpcrfs01 1] ID 4100:327518 from the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:44.730: [    AGFW][2603726592] {2:57358:3160} Fenced off the resource [ora.ons]

                            2014-04-15 18:38:45.278: [    AGFW][2603726592] {2:57358:3160} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr ecchpcrfs01 1] ID 4100:327515 from the agent /u01/app/11.2.0/grid/bin/oraagent_oracle

                            2014-04-15 18:38:45.278: [    AGFW][2603726592] {2:57358:3160} Fenced off the resource [ora.LISTENER.lsnr]

                            2014-04-15 18:38:45.290: [   CRSPE][2603726592] {2:57358:3160} Fence command status: Container [ Name: UI_DATA

                                    RESULT:

                                    TextMessage[0]

                            ]

                            2014-04-15 18:38:45.290: [    AGFW][2603726592] {2:57358:3160} Fence command completed!

                            2014-04-15 18:38:45.290: [UiServer][2591119104] {2:57358:3160} UI server recvd reply from Agfw Proxy Server: 3144

                            2014-04-15 18:38:45.290: [UiServer][2591119104] {2:57358:3160} Container [ Name: UI_DATA

                                    RESULT:

                                    TextMessage[0]

                            ]

                            2014-04-15 18:38:45.291: [UiServer][2591119104] {2:57358:3160} Done for ctx=0x7f1538008f70

                            2014-04-15 18:38:50.542: [ default][354567968] CSS is not ready. Received status 3

                            2014-04-15 18:38:50.542: [ default][354567968] First attempt: init CSS context failed. Error = 3

                            [  clsdmt][348116736]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ecchpcrfs01DBG_CRSD))

                            2014-04-15 18:38:50.621: [  clsdmt][348116736]PID for the Process [8785], connkey 1

                            2014-04-15 18:38:50.627: [  clsdmt][348116736]Creating PID [8785] file for home /u01/app/11.2.0/grid host ecchpcrfs01 bin crs to /u01/app/11.2.0/grid/crs/init/

                            2014-04-15 18:38:50.627: [  clsdmt][348116736]Writing PID [8785] to the file [/u01/app/11.2.0/grid/crs/init/ecchpcrfs01.pid]

                            2014-04-15 18:38:51.038: [ default][354567968] CRS Daemon Starting

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: AGENT  1

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: AGFW  0

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: CLSFRAME  0

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: CLSVER  0

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: CLUCLS  0

                            2014-04-15 18:38:51.039: [ default][354567968] ENV Logging level for Module: CRSAPP  0

                            2014-04-15 18:38:51.040: [ default][354567968] ENV Logging level for Module: CRSCCL  0

                            2014-04-15 18:38:51.040: [ default][354567968] ENV Logging level for Module: CRSCEVT  0

                            2014-04-15 18:38:51.040: [ default][354567968] ENV Logging level for Module: CRSCOMM  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSD  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSEVT  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSMAIN  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSOCR  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSPE  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSPLACE  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSRES  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSRPT  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSRTI  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSSE  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSSEC  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSTIMER  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: CRSUI  0

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: OCRAPI  1

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: OCRASM  1

                            2014-04-15 18:38:51.040: [    CRSD][354567968] ENV Logging level for Module: OCRCAC  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRCLI  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRMAS  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRMSG  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCROSD  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRRAW  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRSRV  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: OCRUTL  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: SuiteTes  1

                            2014-04-15 18:38:51.041: [    CRSD][354567968] ENV Logging level for Module: UiServer  0

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Checking the OCR device

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Sync-up with OCR

                            2014-04-15 18:38:51.041: [ CRSMAIN][354567968] Connecting to the CSS Daemon

                            2014-04-15 18:38:51.045: [  CRSRTI][354567968] CSS is not ready. Received status 3

                            2014-04-15 18:38:51.051: [ CRSMAIN][354567968] Created alert : (:CRSD00109:) :  Could not init the CSS context, error: 3

                            2014-04-15 18:38:51.051: [    CRSD][354567968][PANIC] CRSD exiting: Could not init the CSS context, error: 3

                            2014-04-15 18:38:51.051: [    CRSD][354567968] Done.

                             

                             

                            tail -f ocssd.log

                            2014-04-16 00:17:09.717: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                            2014-04-16 00:17:09.717: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                            2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(10/0x7f335001e180)

                            2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001e180)

                            2014-04-16 00:17:09.717: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x570

                            2014-04-16 00:17:09.904: [    CLSF][1492350720]Allocated CLSF context

                            2014-04-16 00:17:09.905: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:09.905: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:10.718: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                            2014-04-16 00:17:10.718: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                            2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(11/0x7f335001ed40)

                            2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001ed40)

                            2014-04-16 00:17:10.718: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x586

                            2014-04-16 00:17:10.906: [    CLSF][1492350720]Allocated CLSF context

                            2014-04-16 00:17:10.907: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:10.907: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:11.719: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                            2014-04-16 00:17:11.719: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                            2014-04-16 00:17:11.719: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(12/0x7f335001ed40)

                            2014-04-16 00:17:11.720: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335001ed40)

                            2014-04-16 00:17:11.720: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x59c

                            2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x7f3350051b20) client((nil))

                            2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDeadProc: proc 0x7f3350051b20

                            2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDestroyProc: cleaning up proc(0x7f3350051b20) con(0x51c) skgpid  ospid 4740 with 0 clients, refcount 0

                            2014-04-16 00:17:11.875: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x51c

                            2014-04-16 00:17:11.878: [    CSSD][1501222656]clssscSelect: cookie accept request 0x126dd80

                            2014-04-16 00:17:11.878: [    CSSD][1501222656]clssgmAllocProc: (0x7f3350075a10) allocated

                            2014-04-16 00:17:11.878: [    CSSD][1501222656]clssgmClientConnectMsg: properties of cmProc 0x7f3350075a10 - 1,2,3,4,5

                            2014-04-16 00:17:11.879: [    CSSD][1501222656]clssgmClientConnectMsg: Connect from con(0x5f0) proc(0x7f3350075a10) pid(4740) version 11:2:1:4, properties: 1,2,3,4,5

                            2014-04-16 00:17:11.879: [    CSSD][1501222656]clssgmClientConnectMsg: msg flags 0x0000

                            2014-04-16 00:17:11.908: [    CLSF][1492350720]Allocated CLSF context

                            2014-04-16 00:17:11.909: [   SKGFD][1492350720]Handle 0x7f333c0864a0 from lib :UFS:: for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:11.909: [   SKGFD][1492350720]Lib :UFS:: closing handle 0x7f333c0864a0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:12.720: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                            2014-04-16 00:17:12.720: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                            2014-04-16 00:17:12.720: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(13/0x7f335004f700)

                            2014-04-16 00:17:12.721: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f335004f700)

                            2014-04-16 00:17:12.721: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x616

                            2014-04-16 00:17:12.909: [    CSSD][1516943104]clssnmlfmtlease: uniqueness 1397587617, gipc addr gipcha://ecchpcrfs01:nm2_ecchpcrf

                            2014-04-16 00:17:12.914: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                            2014-04-16 00:17:12.916: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                            2014-04-16 00:17:12.917: [    CSSD][1516943104]clssnmvStatusBlkInit: myinfo nodename ecchpcrfs01, uniqueness 1397587617

                            2014-04-16 00:17:12.959: [    CSSD][1516943104]clssnmlgetslot:lease acquisition for node ecchpcrfs01/slot 2 completed in 15050 msecs

                            2014-04-16 00:17:13.002: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 285466078, wrtcnt, 22645623, LATS 967144, lastSeqNo 0, uniqueness 1394482854, timestamp 1397587632/3103401744

                            2014-04-16 00:17:13.002: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 285466077, wrtcnt, 22551574, LATS 967144, lastSeqNo 0, uniqueness 1397587617, timestamp 1397587605/939384

                            2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNcopy: Saving DHB uniqueness of 1394482854 for node ecchpcrfp01, number 1

                            2014-04-16 00:17:13.084: [    CSSD][1516943104]clssnmvDHBValidateNCopy: node 1, ecchpcrfp01, has a disk HB, but no network HB, DHB has rcfg 285466078, wrtcnt, 22645624, LATS 967224, lastSeqNo 0, uniqueness 1394482854, timestamp 1397587632/3103401754

                            2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x16855f0 for disk :/dev/oracleasm/disks/B12OCRVOTE1:

                             

                             

                            2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x14f1bb0 for disk :/dev/oracleasm/disks/B12OCRVOTE2:

                             

                             

                            2014-04-16 00:17:13.091: [   SKGFD][1516943104]Lib :UFS:: closing handle 0x14f1fd0 for disk :/dev/oracleasm/disks/B12OCRVOTE3:

                             

                             

                            2014-04-16 00:17:13.091: [    CSSD][1516943104]clssnmInitNodeDB: Initializing with OCR id 0

                            2014-04-16 00:17:13.721: [    CSSD][1501222656]clssscSelect: cookie accept request 0x7f335001f400

                            2014-04-16 00:17:13.721: [    CSSD][1501222656]clssscevtypSHRCON: getting client with cmproc 0x7f335001f400

                            2014-04-16 00:17:13.721: [    CSSD][1501222656]clssgmRegisterClient: proc(3/0x7f335001f400), client(14/0x7f3350072680)

                            2014-04-16 00:17:13.721: [    CSSD][1501222656]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7f335001f400) client(0x7f3350072680)

                            2014-04-16 00:17:13.722: [    CSSD][1501222656]clssgmDiscEndpcl: gipcDestroy 0x642

                             

                             

                            general cluster logs

                             

                            [cssd(32189)]CRS-1612:Network communication with node ecchpcrfp01 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.520 seconds

                            2014-04-15 18:38:36.106

                            [cssd(32189)]CRS-1611:Network communication with node ecchpcrfp01 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 6.520 seconds

                            2014-04-15 18:38:40.107

                            [cssd(32189)]CRS-1610:Network communication with node ecchpcrfp01 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.520 seconds

                            2014-04-15 18:38:42.198

                            [cssd(32189)]CRS-1608:This node was evicted by node 1, ecchpcrfp01; details at (:CSSNM00005:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:38:42.198

                            [cssd(32189)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log

                            2014-04-15 18:38:42.205

                            [cssd(32189)]CRS-1652:Starting clean up of CRSD resources.

                            2014-04-15 18:38:42.205

                            [cssd(32189)]CRS-1608:This node was evicted by node 1, ecchpcrfp01; details at (:CSSNM00005:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:38:42.516

                            [cssd(32189)]CRS-1608:This node was evicted by node 1, ecchpcrfp01; details at (:CSSNM00005:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:38:44.577

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(32588)]CRS-5016:Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

                            2014-04-15 18:38:45.276

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(32588)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

                             

                             

                             

                             

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(32588)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

                            2014-04-15 18:38:45.292

                            [cssd(32189)]CRS-1654:Clean up of CRSD resources finished successfully.

                            2014-04-15 18:38:45.292

                            [cssd(32189)]CRS-1655:CSSD on node ecchpcrfs01 detected a problem and started to shutdown.

                            2014-04-15 18:38:45.459

                            [cssd(32189)]CRS-1660:The CSS daemon shutdown has completed

                            2014-04-15 18:38:45.588

                            [/u01/app/11.2.0/grid/bin/orarootagent.bin(32593)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:2:33591} in /u01/app/11.2.0/grid/log/ecchpcrfs01/agent/crsd/orarootagent_root/orarootagent_root.log.

                            2014-04-15 18:38:45.649

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(32588)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:1:6} in /u01/app/11.2.0/grid/log/ecchpcrfs01/agent/crsd/oraagent_oracle/oraagent_oracle.log.

                            2014-04-15 18:38:46.749

                            [ohasd(4393)]CRS-2765:Resource 'ora.crsd' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:38:47.653

                            [ohasd(4393)]CRS-2765:Resource 'ora.evmd' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:38:47.716

                            [ohasd(4393)]CRS-2765:Resource 'ora.ctssd' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:38:50.003

                             

                             

                            2014-04-15 18:38:50.003

                            [ctssd(8787)]CRS-2402:The Cluster Time Synchronization Service aborted on host ecchpcrfs01. Details at (:ctss_css_init1:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/ctssd/octssd.log.

                            2014-04-15 18:38:51.045

                            [crsd(8785)]CRS-0805:Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error [3]. Details at (:CRSD00109:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/crsd/crsd.log.

                            2014-04-15 18:38:51.457

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(4726)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log"

                            2014-04-15 18:38:51.708

                            [ohasd(4393)]CRS-2765:Resource 'ora.asm' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:38:51.753

                            [ohasd(4393)]CRS-2765:Resource 'ora.crsd' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:38:55.152

                            [ctssd(8847)]CRS-2402:The Cluster Time Synchronization Service aborted on host ecchpcrfs01. Details at (:ctss_css_init1:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/ctssd/octssd.log.

                            2014-04-15 18:38:59.163

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(4726)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log"

                            2014-04-15 18:38:59.555

                            [/u01/app/11.2.0/grid/bin/oraagent.bin(4726)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oraagent_oracle/oraagent_oracle.log"

                            2014-04-15 18:39:00.753

                            [ctssd(8874)]CRS-2402:The Cluster Time Synchronization Service aborted on host ecchpcrfs01. Details at (:ctss_css_init1:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/ctssd/octssd.log.

                            2014-04-15 18:39:06.203

                            [ohasd(4393)]CRS-2765:Resource 'ora.cssd' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:39:06.211

                            [ohasd(4393)]CRS-2765:Resource 'ora.cluster_interconnect.haip' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:39:10.757

                            [ohasd(4393)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ecchpcrfs01'.

                            2014-04-15 18:39:17.278

                            [cssd(8940)]CRS-1713:CSSD daemon is started in clustered mode

                            2014-04-15 18:39:33.029

                            [cssd(8940)]CRS-1707:Lease acquisition for node ecchpcrfs01 number 2 completed

                            2014-04-15 18:39:34.321

                            [cssd(8940)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/B12OCRVOTE1; details in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:39:34.330

                            [cssd(8940)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/B12OCRVOTE2; details in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:39:34.337

                            [cssd(8940)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/B12OCRVOTE3; details in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log.

                            2014-04-15 18:44:51.034

                            [ohasd(4393)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdmonitor_root'. Details at (:CRSAGF00130:) {0:0:28215} in /u01/app/11.2.0/grid/log/ecchpcrfs01/ohasd/ohasd.log.

                            2014-04-15 18:49:16.342

                            [/u01/app/11.2.0/grid/bin/cssdagent(8928)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:19:3} in /u01/app/11.2.0/grid/log/ecchpcrfs01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

                            2014-04-15 18:49:16.342

                            [cssd(8940)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/ecchpcrfs01/cssd/ocssd.log

                            2014-04-15 18:49:16.342

                            [cssd(8940)]CRS-1603:CSSD on node ecchpcrfs01 shutdown by user.

                            2014-04-15 18:50:21.096

                            [ohasd(4393)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdmonitor_root'. Details at (:CRSAGF00130:) {0:20:5} in /u01/app/11.2.0/grid/log/ecchpcrfs01/ohasd/ohasd.log.

                            2014-04-15 18:50:21.098

                            • 11. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                              hitgon

                              Thanks  Vandana

                               

                              We checked network and all ips configurations

                               

                              private network working fine

                              public network working file

                              vip and scan seems fine

                               

                              Thanks...

                              • 12. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                                Vandana B -Oracle

                                Hi,

                                 

                                You can try the following if the same does not work then I would suggest you to raise an SR with Oracle Support so that support can analyze all the logs and provide a solution

                                 

                                From both the nodes as root user perform the following

                                 

                                ++ rm -rf /var/tmp/,oracle /tmp/.oracle

                                 

                                Once the above is done, reboot the nodes

                                 

                                Regards,

                                Vandana - Oracle

                                • 14. Re: CRS-4535: Cannot communicate with Cluster Ready Services
                                  hitgon

                                  Thanks Shivendra Narain Nirala Waiting your response on shared logs

                                  1 2 Previous Next