3 Replies Latest reply on May 23, 2014 7:49 PM by Levi Pereira

    CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac2/crsd/crsd.log.

    1ea7f8d3-1d7b-449e-900a-65d5e6e2e6e4

      I have install grid X86_64 and after installing reboot the machine. Then got error in alert log of node

      CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac2/crsd/crsd.log.

       

      how  solve it.

        • 1. Re: CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac2/crsd/crsd.log.
          1ea7f8d3-1d7b-449e-900a-65d5e6e2e6e4

          Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.

          2014-05-21 19:41:13.377

          [client(15367)]CRS-2106:The OLR location /u01/app/11.2.0/grid/cdata/rac1.olr is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/client/ocrconfig_15367.log.

          2014-05-21 19:41:13.416

          [client(15367)]CRS-2101:The OLR was formatted using version 3.

          2014-05-21 19:42:01.824

          [ohasd(15753)]CRS-2112:The OLR service started on node rac1.

          2014-05-21 19:42:03.661

          [ohasd(15753)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          2014-05-21 19:43:01.019

          [cssd(16859)]CRS-1713:CSSD daemon is started in exclusive mode

          2014-05-21 19:43:05.428

          [cssd(16859)]CRS-1709:Lease acquisition failed for node rac1 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log

          2014-05-21 19:43:23.000

          [cssd(16859)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 19:43:24.511

          [ctssd(16937)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-21 19:43:24.523

          [ctssd(16937)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.

          2014-05-21 19:43:25.325

          [ctssd(16937)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          [client(17134)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          [client(17138)]CRS-10001:ACFS-9322: done.

          2014-05-21 19:44:19.787

          [client(17198)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/client/ocrconfig_17198.log.

          2014-05-21 19:44:20.178

          [client(17198)]CRS-1001:The OCR was formatted using version 3.

          2014-05-21 19:44:26.983

          [crsd(17242)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 19:44:27.410

          [cssd(16859)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:44:27.444

          [cssd(16859)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:44:27.490

          [cssd(16859)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:44:28.118

          [cssd(16859)]CRS-1626:A Configuration change request completed successfully

          2014-05-21 19:44:28.124

          [cssd(16859)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 19:44:52.900

          [ctssd(16937)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user

          2014-05-21 19:45:00.561

          [cssd(16859)]CRS-1603:CSSD on node rac1 shutdown by user.

          2014-05-21 19:45:14.142

          [cssd(17816)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-21 19:46:38.698

          [cssd(17816)]CRS-1707:Lease acquisition for node rac1 number 1 completed

          2014-05-21 19:46:38.812

          [cssd(17816)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:46:38.830

          [cssd(17816)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:46:38.854

          [cssd(17816)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 19:46:56.765

          [cssd(17816)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 19:46:58.352

          [ctssd(18023)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-21 19:46:58.589

          [ctssd(18023)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.

          2014-05-21 19:46:59.220

          [ctssd(18023)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          2014-05-21 19:47:22.995

          [crsd(18159)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 19:47:29.155

          [crsd(18159)]CRS-1201:CRSD started on node rac1.

          2014-05-21 19:47:30.548

          [crsd(18159)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          2014-05-21 19:52:49.508

          [cssd(17816)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .

          2014-05-21 19:53:26.646

          [crsd(18159)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.

          2014-05-21 19:57:25.690

          [cssd(17816)]CRS-1625:Node rac2, number 2, was manually shut down

          2014-05-21 19:57:25.699

          [cssd(17816)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 19:57:25.725

          [crsd(18159)]CRS-5504:Node down event reported for node 'rac2'.

          2014-05-21 19:57:25.727

          [crsd(18159)]CRS-2773:Server 'rac2' has been removed from pool 'Free'.

          2014-05-21 20:01:12.366

          [cssd(17816)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .

          2014-05-21 20:01:27.881

          [crsd(18159)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.

          2014-05-21 20:26:50.396

          [cssd(17816)]CRS-1612:Network communication with node rac2 (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.630 seconds

          2014-05-21 20:28:29.629

          [cssd(17816)]CRS-1611:Network communication with node rac2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.320 seconds

          2014-05-21 20:28:34.641

          [cssd(17816)]CRS-1610:Network communication with node rac2 (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.310 seconds

          2014-05-21 20:28:42.273

          [crsd(18159)]CRS-2765:Resource 'ora.rac1.vip' has failed on server 'rac1'.

          2014-05-21 20:28:42.325

          [crsd(18159)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac1.vip'; will not restart.

          2014-05-21 20:28:42.868

          [crsd(18159)]CRS-2765:Resource 'ora.scan2.vip' has failed on server 'rac1'.

          2014-05-21 20:28:42.869

          [crsd(18159)]CRS-2771:Maximum restart attempts reached for resource 'ora.scan2.vip'; will not restart.

          2014-05-21 20:28:42.996

          [crsd(18159)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.

          2014-05-21 20:28:44.307

          [crsd(18159)]CRS-2765:Resource 'ora.ons' has failed on server 'rac1'.

          2014-05-21 20:28:56.409

          [cssd(17816)]CRS-1632:Node rac2 is being removed from the cluster in cluster incarnation 296163809

          2014-05-21 20:28:59.161

          [cssd(17816)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 20:28:59.212

          [crsd(18159)]CRS-5504:Node down event reported for node 'rac2'.

          2014-05-21 20:29:01.090

          [crsd(18159)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:29:01.091

          [crsd(18159)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:29:06.404

          [cssd(17816)]CRS-1603:CSSD on node rac1 shutdown by user.

          2014-05-21 20:29:06.470

          [/u01/app/11.2.0/grid/bin/cssdagent(17795)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

          2014-05-21 20:29:06.482

          [/u01/app/11.2.0/grid/bin/cssdmonitor(17778)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.

          2014-05-21 20:31:50.021

          [ohasd(7517)]CRS-2112:The OLR service started on node rac1.

          2014-05-21 20:31:51.049

          [ohasd(7517)]CRS-8017:location: /etc/oracle/lastgasp has 4 reboot advisory log files, 0 were announced and 0 errors occurred

          2014-05-21 20:32:12.111

          [ohasd(7517)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          2014-05-21 20:32:17.752

          [ohasd(7517)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

          2014-05-21 20:32:21.981

          [cssd(7751)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-21 20:32:48.068

          [cssd(7751)]CRS-1707:Lease acquisition for node rac1 number 1 completed

          2014-05-21 20:32:48.184

          [cssd(7751)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 20:32:48.692

          [cssd(7751)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 20:32:49.718

          [cssd(7751)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-21 20:33:05.602

          [cssd(7751)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .

          2014-05-21 20:33:08.786

          [ctssd(7857)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-21 20:33:08.954

          [ctssd(7857)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.

          2014-05-21 20:33:09.034

          [ctssd(7857)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-21 20:33:09.212

          [ctssd(7857)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          [client(8074)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          [client(8088)]CRS-10001:ACFS-9322: done.

          2014-05-21 20:34:13.571

          [ctssd(7857)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-21 20:34:15.689

          [crsd(8254)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:34:18.731

          [crsd(8254)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:34:19.505

          [crsd(8254)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:34:35.729

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:34:36.987

          [crsd(8407)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:34:37.534

          [crsd(8407)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:34:37.775

          [crsd(8407)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:34:48.110

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:34:49.392

          [crsd(8481)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:34:49.876

          [crsd(8481)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:34:49.942

          [crsd(8481)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:35:00.492

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:35:01.978

          [crsd(8572)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:35:02.522

          [crsd(8572)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:35:02.591

          [crsd(8572)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:35:18.043

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:35:19.479

          [crsd(8648)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:35:20.039

          [crsd(8648)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:35:20.205

          [crsd(8648)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:35:31.464

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:35:32.947

          [crsd(8741)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:35:33.538

          [crsd(8741)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:35:33.613

          [crsd(8741)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:35:49.021

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:35:50.438

          [crsd(8816)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:35:50.957

          [crsd(8816)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:35:51.025

          [crsd(8816)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:36:06.547

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:36:09.005

          [crsd(8907)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:36:14.115

          [crsd(8907)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:36:15.242

          [crsd(8907)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:36:31.341

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:36:35.326

          [crsd(9001)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:36:42.127

          [crsd(9001)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:36:42.194

          [crsd(9001)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:36:58.282

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:36:59.616

          [crsd(9091)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:37:00.161

          [crsd(9091)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:37:00.242

          [crsd(9091)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:37:15.832

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:37:17.082

          [crsd(9165)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:37:17.608

          [crsd(9165)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:37:17.904

          [crsd(9165)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:37:33.413

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:37:33.414

          [ohasd(7517)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

          2014-05-21 20:44:23.437

          [crsd(9736)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:44:23.965

          [crsd(9736)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:44:24.299

          [crsd(9736)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:44:39.768

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:44:46.579

          [crsd(9835)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:44:53.988

          [crsd(9835)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:44:54.957

          [crsd(9835)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:45:10.809

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:45:14.209

          [crsd(9927)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:45:24.155

          [crsd(9927)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:45:26.030

          [crsd(9927)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:45:36.680

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:45:40.690

          [crsd(10020)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:45:46.011

          [crsd(10020)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:45:46.820

          [crsd(10020)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:46:02.485

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:46:05.647

          [crsd(10109)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:46:09.360

          [crsd(10109)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:46:10.192

          [crsd(10109)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:46:26.254

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:46:31.431

          [crsd(10202)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:46:40.635

          [crsd(10202)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:46:46.481

          [crsd(10202)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:46:57.315

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:47:06.779

          [crsd(10292)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:47:18.613

          [crsd(10292)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:47:22.577

          [crsd(10292)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:47:33.528

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:47:40.413

          [crsd(10385)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:47:41.266

          [crsd(10385)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:47:41.417

          [crsd(10385)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:47:52.171

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:47:54.185

          [crsd(10460)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:47:54.779

          [crsd(10460)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:47:54.870

          [crsd(10460)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:48:10.765

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:48:11.999

          [crsd(10549)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:48:12.566

          [crsd(10549)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:48:12.764

          [crsd(10549)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:48:23.149

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:48:24.403

          [crsd(10623)]CRS-1012:The OCR service started on node rac1.

          2014-05-21 20:48:24.934

          [crsd(10623)]CRS-1201:CRSD started on node rac1.

          2014-05-21 20:48:25.072

          [crsd(10623)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-21 20:48:35.540

          [ohasd(7517)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-21 20:48:35.541

          [ohasd(7517)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

          2014-05-21 21:34:43.584

          [ctssd(7857)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.

          2014-05-21 21:34:47.890

          [cssd(7751)]CRS-1625:Node rac2, number 2, was manually shut down

          2014-05-21 21:34:48.769

          [cssd(7751)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-21 21:35:03.234

          [ctssd(7857)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user

          [client(12450)]CRS-10001:Waiting for ASM to shutdown.

          2014-05-21 21:35:15.690

          [cssd(7751)]CRS-1603:CSSD on node rac1 shutdown by user.

          2014-05-22 00:23:25.910

          [ohasd(21193)]CRS-2112:The OLR service started on node rac1.

          2014-05-22 00:23:25.957

          [ohasd(21193)]CRS-8017:location: /etc/oracle/lastgasp has 6 reboot advisory log files, 0 were announced and 0 errors occurred

          2014-05-22 00:23:27.267

          [ohasd(21193)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          [client(21541)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          [client(21545)]CRS-10001:ACFS-9322: done.

          2014-05-22 00:23:32.008

          [cssd(21609)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-22 00:23:55.207

          [cssd(21609)]CRS-1707:Lease acquisition for node rac1 number 1 completed

          2014-05-22 00:23:55.373

          [cssd(21609)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-22 00:23:55.699

          [cssd(21609)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-22 00:23:56.639

          [cssd(21609)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-22 00:24:00.740

          [ohasd(21193)]CRS-8011:reboot advisory message from host: rac2, component: ag213645, with time stamp: L-2014-05-22-00:23:56.773

          [ohasd(21193)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS

          2014-05-22 00:24:00.740

          [ohasd(21193)]CRS-8011:reboot advisory message from host: rac2, component: mo213645, with time stamp: L-2014-05-22-00:23:56.774

          [ohasd(21193)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS

          2014-05-22 00:27:27.156

          [cssd(21609)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-22 00:27:28.392

          [ctssd(21831)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-22 00:27:28.459

          [ctssd(21831)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.

          2014-05-22 00:27:29.294

          [ctssd(21831)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          2014-05-22 00:28:00.605

          [crsd(21980)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:28:04.537

          [crsd(21980)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:28:05.168

          [crsd(21980)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:28:20.640

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:28:22.819

          [crsd(22129)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:28:24.026

          [crsd(22129)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:28:24.573

          [crsd(22129)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:28:40.576

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:28:42.294

          [crsd(22222)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:28:43.342

          [crsd(22222)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:28:43.589

          [crsd(22222)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:28:59.240

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:29:00.677

          [crsd(22313)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:29:00.996

          [crsd(22313)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:29:01.114

          [crsd(22313)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:29:11.667

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:29:13.196

          [crsd(22389)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:29:13.469

          [crsd(22389)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:29:13.567

          [crsd(22389)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:29:24.068

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:29:25.538

          [crsd(22464)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:29:25.858

          [crsd(22464)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:29:25.940

          [crsd(22464)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:29:36.459

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:29:37.983

          [crsd(22557)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:29:38.268

          [crsd(22557)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:29:38.315

          [crsd(22557)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:29:54.029

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:29:55.493

          [crsd(22633)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:29:55.748

          [crsd(22633)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:29:55.821

          [crsd(22633)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:30:06.443

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:30:07.919

          [crsd(22725)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:30:08.195

          [crsd(22725)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:30:08.355

          [crsd(22725)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:30:18.818

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:30:20.269

          [crsd(22822)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:30:20.538

          [crsd(22822)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:30:20.619

          [crsd(22822)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:30:31.223

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:30:32.740

          [crsd(22915)]CRS-1012:The OCR service started on node rac1.

          2014-05-22 00:30:33.116

          [crsd(22915)]CRS-1201:CRSD started on node rac1.

          2014-05-22 00:30:33.196

          [crsd(22915)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-22 00:30:48.810

          [ohasd(21193)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-22 00:30:48.811

          [ohasd(21193)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

          2014-05-22 21:01:08.176

          [ohasd(7579)]CRS-2112:The OLR service started on node rac1.

          2014-05-22 21:01:08.514

          [ohasd(7579)]CRS-8017:location: /etc/oracle/lastgasp has 8 reboot advisory log files, 0 were announced and 0 errors occurred

          2014-05-22 21:01:13.704

          [ohasd(7579)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          2014-05-22 21:01:16.197

          [ohasd(7579)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

          2014-05-22 21:01:17.838

          [cssd(7818)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-22 21:01:30.067

          [cssd(7818)]CRS-1603:CSSD on node rac1 shutdown by user.

          2014-05-22 21:01:30.072

          [/u01/app/11.2.0/grid/bin/cssdagent(7788)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

          2014-05-22 21:01:30.071

          [/u01/app/11.2.0/grid/bin/cssdmonitor(7768)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.

          2014-05-23 10:01:08.595

          [ohasd(7564)]CRS-2112:The OLR service started on node rac1.

          2014-05-23 10:01:09.132

          [ohasd(7564)]CRS-8017:location: /etc/oracle/lastgasp has 8 reboot advisory log files, 0 were announced and 0 errors occurred

          2014-05-23 10:01:19.590

          [ohasd(7564)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          [client(7912)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          [client(7930)]CRS-10001:ACFS-9322: done.

          2014-05-23 10:01:25.576

          [ohasd(7564)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

          2014-05-23 10:01:27.520

          [cssd(7975)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-23 10:01:48.709

          [cssd(7975)]CRS-1707:Lease acquisition for node rac1 number 1 completed

          2014-05-23 10:01:49.581

          [cssd(7975)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:01:49.793

          [cssd(7975)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:01:49.944

          [cssd(7975)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:02:48.058

          [cssd(7975)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .

          2014-05-23 10:02:49.596

          [ctssd(8343)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-23 10:02:49.617

          [ctssd(8343)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.

          2014-05-23 10:02:50.362

          [ctssd(8343)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          2014-05-23 10:02:59.186

          [cssd(7975)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .

          2014-05-23 10:03:21.924

          [crsd(8542)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:03:24.306

          [crsd(8542)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:03:24.518

          [crsd(8542)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:03:30.829

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:03:32.089

          [crsd(8697)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:03:32.403

          [crsd(8697)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:03:32.486

          [crsd(8697)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:03:39.083

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:03:40.314

          [crsd(8771)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:03:40.605

          [crsd(8771)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:03:40.683

          [crsd(8771)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:03:47.329

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:03:48.548

          [crsd(8845)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:03:48.883

          [crsd(8845)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:03:48.944

          [crsd(8845)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:03:55.567

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:03:57.033

          [crsd(8920)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:03:57.706

          [crsd(8920)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:03:58.651

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:00.082

          [crsd(8987)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:00.703

          [crsd(8987)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:04:01.727

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:03.149

          [crsd(9040)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:03.501

          [crsd(9040)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:04:03.668

          [crsd(9040)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:04:15.141

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:16.592

          [crsd(9115)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:17.009

          [crsd(9115)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:04:17.076

          [crsd(9115)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:04:23.380

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:24.784

          [crsd(9190)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:25.036

          [crsd(9190)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:04:25.103

          [crsd(9190)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:04:35.761

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:37.529

          [crsd(9283)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:37.830

          [crsd(9283)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:04:37.879

          [crsd(9283)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:04:53.334

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:04:54.770

          [crsd(9359)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:04:55.172

          [crsd(9359)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:04:55.229

          [crsd(9359)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:05:10.872

          [ohasd(7564)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:05:10.872

          [ohasd(7564)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

          2014-05-23 10:22:15.088

          [cssd(7975)]CRS-1603:CSSD on node rac1 shutdown by user.

          2014-05-23 10:22:15.098

          [/u01/app/11.2.0/grid/bin/cssdmonitor(7939)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.

          2014-05-23 10:22:15.099

          [/u01/app/11.2.0/grid/bin/cssdagent(7958)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/cssdagent_root' disconnected from server. Details at (:CRSAGF00117:) in /u01/app/11.2.0/grid/log/rac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

          2014-05-23 10:23:36.633

          [ohasd(7494)]CRS-2112:The OLR service started on node rac1.

          2014-05-23 10:23:37.828

          [ohasd(7494)]CRS-8017:location: /etc/oracle/lastgasp has 10 reboot advisory log files, 0 were announced and 0 errors occurred

          2014-05-23 10:23:44.908

          [ohasd(7494)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.

          2014-05-23 10:23:47.244

          [ohasd(7494)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

          2014-05-23 10:23:48.857

          [cssd(7915)]CRS-1713:CSSD daemon is started in clustered mode

          2014-05-23 10:24:10.435

          [cssd(7915)]CRS-1707:Lease acquisition for node rac1 number 1 completed

          2014-05-23 10:24:10.498

          [cssd(7915)]CRS-1605:CSSD voting file is online: ORCL:ASM03; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:24:10.589

          [cssd(7915)]CRS-1605:CSSD voting file is online: ORCL:ASM02; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:24:10.666

          [cssd(7915)]CRS-1605:CSSD voting file is online: ORCL:ASM01; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.

          2014-05-23 10:24:22.044

          [cssd(7915)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .

          2014-05-23 10:24:23.118

          [ctssd(8004)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.

          2014-05-23 10:24:23.178

          [ctssd(8004)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.

          2014-05-23 10:24:23.873

          [ctssd(8004)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.

          [client(8192)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          [client(8207)]CRS-10001:ACFS-9322: done.

          2014-05-23 10:24:43.930

          [crsd(8353)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:24:45.349

          [crsd(8353)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:24:45.378

          [crsd(8353)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:24:46.090

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:24:47.371

          [crsd(8485)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:24:47.843

          [crsd(8485)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:24:47.906

          [crsd(8485)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:24:54.363

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:24:55.652

          [crsd(8559)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:24:56.167

          [crsd(8559)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:24:56.215

          [crsd(8559)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:24:57.470

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:24:58.785

          [crsd(8634)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:24:59.251

          [crsd(8634)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:24:59.363

          [crsd(8634)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:00.593

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:01.832

          [crsd(8708)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:02.393

          [crsd(8708)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:02.462

          [crsd(8708)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:08.852

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:10.118

          [crsd(8797)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:10.653

          [crsd(8797)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:10.702

          [crsd(8797)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:22.276

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:23.510

          [crsd(8872)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:23.978

          [crsd(8872)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:24.036

          [crsd(8872)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:26.408

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:27.618

          [crsd(8949)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:28.079

          [crsd(8949)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:28.237

          [crsd(8949)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:39.821

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:41.291

          [crsd(9040)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:41.772

          [crsd(9040)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:41.821

          [crsd(9040)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:42.909

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:44.328

          [crsd(9115)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:44.794

          [crsd(9115)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:44.846

          [crsd(9115)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:45.991

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:47.403

          [crsd(9190)]CRS-1012:The OCR service started on node rac1.

          2014-05-23 10:25:47.881

          [crsd(9190)]CRS-1201:CRSD started on node rac1.

          2014-05-23 10:25:47.938

          [crsd(9190)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.

          2014-05-23 10:25:49.056

          [ohasd(7494)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

          2014-05-23 10:25:49.056

          [ohasd(7494)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

          2014-05-23 10:29:04.220

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-23 10:29:52.238

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-23 10:36:48.389

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-23 10:40:00.456

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-23 10:42:24.506

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          2014-05-23 10:42:56.519

          [ctssd(8004)]CRS-2409:The clock on host rac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

          ===========================================================================================================================

           

          crsd.log

           

          -----------

          ----------

          ---------

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->num_ndk=1

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->metactx=[0x1d8eceb0]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_uid[0] is NULL

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_uid[1]=root

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_uid[2]=grid

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_num_cached_uid=2

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_num_uid=2

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_gid[0] is NULL

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_gid[1]=root

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_gid[2]=oinstall

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_gid[3]=asmadmin

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_num_cached_gid=3

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_num_gid=3

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_ocrctx=[0x1d92fde0]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_os_min_block_size=512

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_islocal_only=0

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc_proprhandle: propb_ctx->proprhandle_upgrade_state=0

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: end dumping backenctx->propb_ctx

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: backend_ctx->metactx=[0x1d8eceb0]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: backend_ctx->prop_sctx=[0x1d4e3fe0]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: backend_ctx->prop_sltsmx=[0x0]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: backend_ctx->prop_sclsctx=[0x1d9337c8]

          2014-05-23 10:25:48.471: [  OCRRAW][1260935488]proprdc: backend_ctx->prop_ctx_ocrctx=[0x1d92fde0]

          [  OCRAPI][1260935488]procr_ctx_set_invalid_no_abort: ctx set to invalid

          • 2. Re: CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac2/crsd/crsd.log.
            Billy~Verreynne

            Cluster shared storage, and cluster Interconnect, are MANDATORY - cluster cannot start without these components.

             

            From your log file, it seems that both are unavailable. Node 2 is evicted due to an Interconnect failure. Node 1 is unable to start as it cannot access the OCR/voting files on shared storage (+DATA1 diskgroup).

            • 3. Re: CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac2/crsd/crsd.log.
              Levi Pereira

              Check alertlog of all ASM Instances and cssd.log under $GRID_HOME/log/<hostname>/cssd/