11 Replies Latest reply on Sep 1, 2010 4:11 PM by 103231

    Failure at final check of Oracle CRS stack.10  on the second node

    Leo Rajan
      Hi,
      I am trying to install Oracle Clusterware 10.2.0.1.0 in VM machines (2 nodes config) in Linux (OEL5) using VMware Server (2.0). Everything went very well one the first node upto running the root.sh. Running root.sh ended with Failure at final check of Oracle CRS stack 10 error.

      RAC1 root.sh output

      [root@rac1 crs]# ./root.sh
      WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
      WARNING: directory '/u01/crs/oracle/product' is not owned by root
      WARNING: directory '/u01/crs/oracle' is not owned by root
      WARNING: directory '/u01/crs' is not owned by root
      WARNING: directory '/u01' is not owned by root
      Checking to see if Oracle CRS stack is already configured
      /etc/oracle does not exist. Creating it now.

      Setting the permissions on OCR backup directory
      Setting up NS directories
      Oracle Cluster Registry configuration upgraded successfully
      WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
      WARNING: directory '/u01/crs/oracle/product' is not owned by root
      WARNING: directory '/u01/crs/oracle' is not owned by root
      WARNING: directory '/u01/crs' is not owned by root
      WARNING: directory '/u01' is not owned by root
      assigning default hostname rac1 for node 1.
      assigning default hostname rac2 for node 2.
      Successfully accumulated necessary OCR keys.
      Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
      node <nodenumber>: <nodename> <private interconnect name> <hostname>
      node 1: rac1 rac1-priv rac1
      node 2: rac2 rac2-priv rac2
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      Now formatting voting device: /dev/raw/raw2
      Format of 1 voting devices complete.
      Startup will be queued to init within 90 seconds.
      Adding daemons to inittab
      Expecting the CRS daemons to be up within 600 seconds.
      CSS is active on these nodes.
      rac1
      CSS is inactive on these nodes.
      rac2
      Local node checking complete.
      Run root.sh on remaining nodes to start CRS daemons.
      [root@rac1 crs]#


      RAC2 root.sh output

      [root@rac2 crs]# ./root.sh
      WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
      WARNING: directory '/u01/crs/oracle/product' is not owned by root
      WARNING: directory '/u01/crs/oracle' is not owned by root
      WARNING: directory '/u01/crs' is not owned by root
      WARNING: directory '/u01' is not owned by root
      Checking to see if Oracle CRS stack is already configured
      /etc/oracle does not exist. Creating it now.

      Setting the permissions on OCR backup directory
      Setting up NS directories
      Oracle Cluster Registry configuration upgraded successfully
      WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
      WARNING: directory '/u01/crs/oracle/product' is not owned by root
      WARNING: directory '/u01/crs/oracle' is not owned by root
      WARNING: directory '/u01/crs' is not owned by root
      WARNING: directory '/u01' is not owned by root
      assigning default hostname rac1 for node 1.
      assigning default hostname rac2 for node 2.
      Successfully accumulated necessary OCR keys.
      Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
      node <nodenumber>: <nodename> <private interconnect name> <hostname>
      node 1: rac1 rac1-priv rac1
      node 2: rac2 rac2-priv rac2
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      Now formatting voting device: /dev/raw/raw2
      Format of 1 voting devices complete.
      Startup will be queued to init within 90 seconds.
      Adding daemons to inittab
      Expecting the CRS daemons to be up within 600 seconds.
      Failure at final check of Oracle CRS stack.
      10
      [root@rac2 crs]#

      Output of alterrac2.log

      [root@rac2 rac2]# more alertrac2.log
      2009-08-14 23:02:44.699
      [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
      0/crs/log/rac2/client/ocrconfig_5935.log.
      2009-08-14 23:02:44.704
      [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
      0/crs/log/rac2/client/ocrconfig_5935.log.
      2009-08-14 23:02:44.707
      [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
      0/crs/log/rac2/client/ocrconfig_5935.log.
      2009-08-14 23:02:44.864
      [client(5935)]CRS-1001:The OCR was formatted using version 2.
      2009-08-14 23:02:50.339
      [client(6004)]CRS-1801:Cluster crs configured with nodes rac1 rac2 .
      2009-08-14 23:05:07.603
      [cssd(6600)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/crs/oracle/product/10.2.0/crs/l
      og/rac2/cssd/ocssd.log.
      [root@rac2 rac2]#


      Since raw devices are not supported from OEL5, I did do the workaround in *63-oracle-raw.rules file under /etc/udev/rules.d* dir.

      ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
      ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
      ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
      ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
      ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
      KERNEL=="raw[1-2]*", OWNER="root", GROUP="oinstall", MODE="640"
      KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="640"

      One thing I have noticed after running root.sh on both the nodes is the permissons on raw devices changed from

      Before root.sh

      [root@rac2 crs]# ls -ls /dev/raw*
      0 crw------- 1 root root 162, 0 Aug 14 22:42 /dev/rawctl

      /dev/raw:
      total 0
      0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:42 raw1
      0 crw-r----- 1 root oinstall 162, 2 Aug 14 22:42 raw2
      0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:42 raw3
      0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:42 raw4
      0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:42 raw5

      to

      [root@rac2 crs]# ls -ls /dev/raw*
      0 crw------- 1 root root 162, 0 Aug 14 22:31 /dev/rawctl

      /dev/raw:
      total 0
      0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:56 raw1
      0 crw-r--r-- 1 oracle oinstall 162, 2 Aug 14 23:01 raw2
      0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:31 raw3
      0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:31 raw4
      0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:31 raw5
      [root@rac1 crs]#

      My shared disk listing

      [root@www shared]# ls -ltr
      total 8780
      -rw------- 1 root root 640 Aug 14 22:43 votingdisk.vmdk
      -rw------- 1 root root 598 Aug 14 22:43 ocr.vmdk
      -rw------- 1 root root 604 Aug 14 22:43 asm3.vmdk
      -rw------- 1 root root 604 Aug 14 22:43 asm2.vmdk
      -rw------- 1 root root 604 Aug 14 22:43 asm1.vmdk
      -rw------- 1 root root 65536 Aug 14 22:44 votingdisk-s006.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s005.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s004.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s003.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s002.vmdk
      -rw------- 1 root root 393216 Aug 14 22:44 votingdisk-s001.vmdk
      -rw------- 1 root root 65536 Aug 14 22:44 ocr-s006.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 ocr-s005.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 ocr-s004.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 ocr-s003.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 ocr-s002.vmdk
      -rw------- 1 root root 393216 Aug 14 22:44 ocr-s001.vmdk
      -rw------- 1 root root 65536 Aug 14 22:44 asm3-s006.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm3-s005.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm3-s004.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm3-s003.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm3-s002.vmdk
      -rw------- 1 root root 393216 Aug 14 22:44 asm3-s001.vmdk
      -rw------- 1 root root 65536 Aug 14 22:44 asm2-s006.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm2-s005.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm2-s004.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm2-s003.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm2-s002.vmdk
      -rw------- 1 root root 393216 Aug 14 22:44 asm2-s001.vmdk
      -rw------- 1 root root 65536 Aug 14 22:44 asm1-s006.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm1-s005.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm1-s004.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm1-s003.vmdk
      -rw------- 1 root root 327680 Aug 14 22:44 asm1-s002.vmdk
      -rw------- 1 root root 393216 Aug 14 22:44 asm1-s001.vmdk
      [root@www shared]#


      I don't know how to fix this problem. I did go through many docs and metalink notes.

      I am new to RAC world. It took 3 days to come to this stage. Please help me.

      Thanks
      Leo
        • 1. Re: Failure at final check of Oracle CRS stack.10  on the second node
          Surachart Opun
          if you make sure about disks + permission...

          check disk again with ocrcheck command... on 2 nodes

          Example:
          $ ocrcheck
          Status of Oracle Cluster Registry is as follows :
          Version : 2
          Total space (kbytes) : 248784
          Used space (kbytes) : 5400
          Available space (kbytes) : 243384
          ID : 519182535
          Device/File Name : /dev/raw/raw1
          Device/File integrity check succeeded

          Device/File not configured

          Cluster registry integrity check succeeded

          ID : 519182535
          if different between 2 nodes ...
          check disk... because you map them wrong... (you've not used the same disk between nodes to make OCR + VOTE )

          Or
          please check IPTABLES between 2 nodes

          # iptables -L

          # iptables -F

          and run root.sh again.
          http://surachartopun.com/2009/01/failure-at-final-check-of-oracle-crs.html


          Good Luck
          • 2. Re: Failure at final check of Oracle CRS stack.10  on the second node
            Leo Rajan
            Hi Surachart,
            Thanks for your reply. I did go through your blog and ran the suggested check commands. I didn't see any issue. Here are the output logs...

            RAC1 ocrcheck

            [oracle@rac1 bin]$ ./ocrcheck
            Status of Oracle Cluster Registry is as follows :
            Version : 2
            Total space (kbytes) : 10482292
            Used space (kbytes) : 304
            Available space (kbytes) : 10481988
            ID : 749094372
            Device/File Name : /dev/raw/raw1
            Device/File integrity check succeeded

            Device/File not configured

            Cluster registry integrity check succeeded

            RAC1 iptables -L

            [root@rac1 crs]# iptables -L
            Chain INPUT (policy ACCEPT)
            target prot opt source destination

            Chain FORWARD (policy ACCEPT)
            target prot opt source destination

            Chain OUTPUT (policy ACCEPT)
            target prot opt source destination

            [root@rac1 crs]# iptables -F
            [root@rac1 crs]#


            RAC2 ocrcheck

            [oracle@rac2 ~]$ cd /u01/crs/oracle/product/10.2.0/crs/bin
            [oracle@rac2 bin]$ ./ocrcheck
            Status of Oracle Cluster Registry is as follows :
            Version : 2
            Total space (kbytes) : 10482292
            Used space (kbytes) : 304
            Available space (kbytes) : 10481988
            ID : 1155650855
            Device/File Name : /dev/raw/raw1
            Device/File integrity check succeeded

            Device/File not configured

            Cluster registry integrity check succeeded

            #

            RAC2 iptables -L


            [root@rac2 bin]# iptables -L
            Chain INPUT (policy ACCEPT)
            target prot opt source destination

            Chain FORWARD (policy ACCEPT)
            target prot opt source destination

            Chain OUTPUT (policy ACCEPT)
            target prot opt source destination
            [root@rac2 bin]# iptables -F
            [root@rac2 bin]#
            [root@rac2 bin]# iptables -L
            Chain INPUT (policy ACCEPT)
            target prot opt source destination

            Chain FORWARD (policy ACCEPT)
            target prot opt source destination

            Chain OUTPUT (policy ACCEPT)
            target prot opt source destination

            [root@rac2 bin]# iptables -F
            [root@rac2 bin]#

            Also I did check crsctl check boot and I did not see anything in my output log file. During the run, when I tried 'grep cssd' I saw the following two process.

            [oracle@rac2 bin]$ ps -ef|grep cssd
            root 5099 1 0 16:50 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
            root 9992 6792 1 17:09 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
            oracle 9997 7729 0 17:09 pts/2 00:00:00 grep cssd
            [oracle@rac2 bin]$

            Is there any other log files that I need to look into?

            Please Help.
            Thanks
            Leo
            • 3. Re: Failure at final check of Oracle CRS stack.10  on the second node
              Leo Rajan
              Here is my alertrac2.log from RAC2

              [oracle@rac2 rac2]$ more alertrac2.log
              2009-08-20 16:47:58.090
              [client(4292)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.0/crs/log/rac2/client/ocrconfig_4292.log.
              2009-08-20 16:47:58.095
              [client(4292)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.0/crs/log/rac2/client/ocrconfig_4292.log.
              2009-08-20 16:47:58.099
              [client(4292)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.0/crs/log/rac2/client/ocrconfig_4292.log.
              2009-08-20 16:47:58.217
              [client(4292)]CRS-1001:The OCR was formatted using version 2.
              2009-08-20 16:48:03.717
              [client(4348)]CRS-1801:Cluster crs configured with nodes rac1 rac2 .
              2009-08-20 16:50:21.324
              [cssd(4964)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/crs/oracle/product/10.2.0/crs/log/rac2/cssd/ocssd.log.


              The /u01/crs/oracle/product/10.2.0/crs/log/rac2/client/ocrconfig_4292.log

              [root@rac2 ~]# more /u01/crs/oracle/product/10.2.0/crs/log/rac2/client/ocrconfig_4292.log
              Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
              2009-08-20 16:47:58.079: [ OCRCONF][2054352]ocrconfig starts...
              2009-08-20 16:47:58.081: [ OCRCONF][2054352]Upgrading OCR data
              2009-08-20 16:47:58.089: [  OCRRAW][2054352]propriogid:1: INVALID FORMAT
              2009-08-20 16:47:58.090: [  OCRRAW][2054352]ibctx:1:ERROR: INVALID FORMAT
              2009-08-20 16:47:58.090: [  OCRRAW][2054352]proprinit:problem reading the bootblock or superbloc 22

              2009-08-20 16:47:58.090: [ default][2054352]a_init:7!: Backend init unsuccessful : [22]
              2009-08-20 16:47:58.091: [ OCRCONF][2054352]Exporting OCR data to [OCRUPGRADEFILE]
              2009-08-20 16:47:58.091: [  OCRAPI][2054352]a_init:7!: Backend init unsuccessful : [33]
              2009-08-20 16:47:58.091: [ OCRCONF][2054352]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
              2009-08-20 16:47:58.095: [  OCRRAW][2054352]propriogid:1: INVALID FORMAT
              2009-08-20 16:47:58.095: [  OCRRAW][2054352]ibctx:1:ERROR: INVALID FORMAT
              2009-08-20 16:47:58.095: [  OCRRAW][2054352]proprinit:problem reading the bootblock or superbloc 22

              2009-08-20 16:47:58.095: [ default][2054352]a_init:7!: Backend init unsuccessful : [22]
              2009-08-20 16:47:58.099: [  OCRRAW][2054352]propriogid:1: INVALID FORMAT
              2009-08-20 16:47:58.099: [  OCRRAW][2054352]ibctx:1:ERROR: INVALID FORMAT
              2009-08-20 16:47:58.099: [  OCRRAW][2054352]proprinit:problem reading the bootblock or superbloc 22

              2009-08-20 16:47:58.102: [  OCRRAW][2054352]propriogid:1: INVALID FORMAT
              2009-08-20 16:47:58.125: [  OCRRAW][2054352]propriowv: Vote information on disk 0 [dev/raw/raw1] is adjusted from [0/0] to [2/2]
              2009-08-20 16:47:58.217: [  OCRRAW][2054352]propriniconfig:No 92 configuration
              2009-08-20 16:47:58.217: [  OCRAPI][2054352]a_init:6a: Backend init successful
              2009-08-20 16:47:58.303: [ OCRCONF][2054352]Initialized DATABASE keys in OCR
              2009-08-20 16:47:58.351: [ OCRCONF][2054352]Successfully set skgfr block 0
              2009-08-20 16:47:58.351: [ OCRCONF][2054352]Exiting [status=success]...
              [root@rac2 ~]#
              • 4. Re: Failure at final check of Oracle CRS stack.10  on the second node
                Surachart Opun
                The OCR location /dev/raw/raw1 is inaccessible
                after reboot because the permissions on the raw devices for the OCR and voting disks e.g. /dev/raw/raw{x} revert to their default values (root:disk) and are inaccessible to Oracle.
                root 5099 1 0 16:50 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
                root 9992 6792 1 17:09 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
                you found "cssd startcheck" process.
                Did you see some errors in /var/log/messages?
                Or /tmp/crsctl.* (ls -l /tmp/crsctl.*) and check in files
                • 5. Re: Failure at final check of Oracle CRS stack.10  on the second node
                  Leo Rajan
                  Hi Surachart,
                  Here is my messages output..

                  */var/log/messages*

                  Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f9a8 on eth1.
                  Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.1.196 on eth1.
                  Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f99e on eth0.
                  Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.0.196 on eth0.
                  Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering HINFO record with values 'I686'/'LINUX'.
                  Aug 20 14:05:02 rac2 avahi-daemon[3627]: Server startup complete. Host name is rac2.local. Local service cookie is 927471131.
                  Aug 20 14:05:03 rac2 avahi-daemon[3627]: Service "SFTP File Transfer on rac2" (/services/sftp-ssh.service) successfully established.
                  Aug 20 14:05:08 rac2 smartd[3739]: smartd version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
                  Aug 20 14:05:08 rac2 smartd[3739]: Home page is http://smartmontools.sourceforge.net/
                  Aug 20 14:05:08 rac2 smartd[3739]: Opened configuration file /etc/smartd.conf
                  Aug 20 14:05:08 rac2 smartd[3739]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, opened
                  Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: status=0x51 { DriveReady SeekComplete Error }
                  Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: error=0x04 { AbortedCommand }
                  Aug 20 14:05:08 rac2 kernel: ide: failed opcode was: 0xec
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, not ATA, no IDENTIFY DEVICE Structure
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, opened
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, opened
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdb' to turn on SMART features
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, opened
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdc' to turn on SMART features
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, opened
                  Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdd' to turn on SMART features
                  Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, opened
                  Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sde' to turn on SMART features
                  Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, opened
                  Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdf' to turn on SMART features
                  Aug 20 14:05:10 rac2 smartd[3739]: Monitoring 0 ATA and 0 SCSI devices
                  Aug 20 14:05:10 rac2 smartd[3741]: smartd has fork()ed into background mode. New PID=3741.
                  Aug 20 14:05:13 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
                  Aug 20 14:05:13 rac2 last message repeated 3 times
                  Aug 20 14:05:27 rac2 gconfd (root-3967): starting (version 2.14.0), pid 3967 user 'root'
                  Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
                  Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
                  Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
                  Aug 20 14:05:29 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
                  Aug 20 14:05:29 rac2 hald: mounted /dev/hdc on behalf of uid 0
                  Aug 20 14:05:29 rac2 hcid[3311]: Default passkey agent (:1.8, /org/bluez/applet) registered
                  Aug 20 14:05:31 rac2 nm-system-settings: Loaded plugin ifcfg-rh: (c) 2007 - 2008 Red Hat, Inc. To report bugs please use the NetworkManager mailing list.
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth1 ...
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth1'
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-lo ...
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: error: Ignoring loopback device config.
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth0 ...
                  Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth0'
                  Aug 20 14:05:31 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
                  Aug 20 14:05:32 rac2 last message repeated 4 times
                  Aug 20 14:12:51 rac2 kernel: FS-Cache: Loaded
                  Aug 20 14:22:06 rac2 xinetd[3488]: START: shell pid=5193 from=192.168.0.195
                  Aug 20 14:22:06 rac2 xinetd[3488]: EXIT: shell status=0 pid=5193 duration=0(sec)
                  Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5217 from=192.168.0.195
                  Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5217 duration=0(sec)
                  Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5241 from=192.168.0.195
                  Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5241 duration=0(sec)
                  ......
                  ......
                  Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6236 duration=0(sec)
                  Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6265 from=192.168.0.195
                  Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6265 duration=0(sec)
                  Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6291 from=192.168.0.195
                  Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6291 duration=1(sec)
                  Aug 20 14:22:17 rac2 xinetd[3488]: START: shell pid=6317 from=192.168.0.195
                  Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6317 duration=0(sec)
                  [root@rac2 log]#
                  • 6. Re: Failure at final check of Oracle CRS stack.10  on the second node
                    Leo Rajan
                    Hi,
                    I did not see any crsctl files under /tmp or any other directories.

                    Thanks for your helps.
                    Leo
                    • 7. Re: Failure at final check of Oracle CRS stack.10  on the second node
                      Surachart Opun
                      crsctl files under /tmp that's generated while crs starting

                      Could you start crs manual after machine reboot?

                      # vi /etc/inittab

                      and comment(3 last rows):

                      #h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
                      #h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
                      #h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

                      and then reboot machine

                      # reboot

                      after reboot you don't see process about crs

                      and then start them by

                      # /etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null &
                      # /etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null &
                      # /etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null &

                      after run crs to background process while that check

                      - tail -f /var/log/messages

                      - crsctl files under /tmp

                      - check log under CRS_HOME/log/<nodename>/
                      alert*.log
                      cssd/ocssd.log
                      crsd/crsd.log
                      evmd/evmd.log

                      I hope this can help you investigate the problem.

                      another way you may open SR at metalink.

                      Good Luck
                      • 8. Re: Failure at final check of Oracle CRS stack.10  on the second node
                        719855
                        Hello Leo,

                        Did you fix this issue,I am also facing the same problem,Could you please help me, If you have fixed this issue.

                        Thanks
                        Nitya
                        • 9. Re: Failure at final check of Oracle CRS stack.10  on the second node
                          user11977189
                          Hi,
                          I also get same issue.
                          Oracle 10G RAC on Oracle linux 5, VMware server 2.0. Sucess install with ocfs2, no issue. However, when trying using raw device (metalink Note 465001.1), then run root.sh, got below error.

                          2009-10-15 15:15:28.069: [ OCRCONF][2053824]ocrconfig starts...
                          2009-10-15 15:15:28.070: [ OCRCONF][2053824]Upgrading OCR data
                          2009-10-15 15:15:28.327: [  OCRRAW][2053824]propriogid:1: INVALID FORMAT
                          2009-10-15 15:15:28.365: [  OCRRAW][2053824]ibctx:1:ERROR: INVALID FORMAT
                          2009-10-15 15:15:28.365: [  OCRRAW][2053824]proprinit:problem reading the bootblock or superbloc 22

                          2009-10-15 15:15:28.365: [ default][2053824]a_init:7!: Backend init unsuccessful : [22]
                          2009-10-15 15:15:28.365: [ OCRCONF][2053824]Exporting OCR data to [OCRUPGRADEFILE]
                          2009-10-15 15:15:28.365: [  OCRAPI][2053824]a_init:7!: Backend init unsuccessful : [33]
                          2009-10-15 15:15:28.365: [ OCRCONF][2053824]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
                          2009-10-15 15:15:28.382: [  OCRRAW][2053824]propriogid:1: INVALID FORMAT
                          2009-10-15 15:15:28.382: [  OCRRAW][2053824]ibctx:1:ERROR: INVALID FORMAT
                          2009-10-15 15:15:28.382: [  OCRRAW][2053824]proprinit:problem reading the bootblock or superbloc 22

                          2009-10-15 15:15:28.382: [ default][2053824]a_init:7!: Backend init unsuccessful : [22]
                          2009-10-15 15:15:28.399: [  OCRRAW][2053824]propriogid:1: INVALID FORMAT
                          2009-10-15 15:15:28.399: [  OCRRAW][2053824]ibctx:1:ERROR: INVALID FORMAT
                          2009-10-15 15:15:28.399: [  OCRRAW][2053824]proprinit:problem reading the bootblock or superbloc 22

                          2009-10-15 15:15:28.414: [  OCRRAW][2053824]propriogid:1: INVALID FORMAT
                          2009-10-15 15:15:28.468: [  OCRRAW][2053824]propriowv: Vote information on disk 0 [dev/raw/raw1] is adjusted from [0/0] to [2/2]
                          2009-10-15 15:15:28.579: [  OCRRAW][2053824]propriniconfig:No 92 configuration
                          2009-10-15 15:15:28.579: [  OCRAPI][2053824]a_init:6a: Backend init successful
                          2009-10-15 15:15:28.792: [ OCRCONF][2053824]Initialized DATABASE keys in OCR
                          2009-10-15 15:15:28.972: [ OCRCONF][2053824]Successfully set skgfr block 0
                          2009-10-15 15:15:28.981: [ OCRCONF][2053824]Exiting [status=success]...

                          My raw device info

                          1- [root@rac1 log]# ls -ltr /dev/raw
                          total 0
                          crw-r----- 1 root dba 162, 1 Oct 15 15:15 raw1
                          crw-r--r-- 1 oracle dba 162, 2 Oct 15 15:41 raw2

                          2- [root@rac1 log]# raw -q /dev/raw/raw1
                          /dev/raw/raw1: bound to major 8, minor 17

                          [root@rac1 log]# raw -q /dev/raw/raw2
                          /dev/raw/raw2: bound to major 8, minor 18

                          Any help will be appreciate!
                          • 10. Re: Failure at final check of Oracle CRS stack.10  on the second node
                            Surachart Opun
                            How about your problem? I hope you're able to solve it...
                            oracle@rac1 bin$ ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                            Version : 2
                            Total space (kbytes) : 10482292
                            Used space (kbytes) : 304
                            Available space (kbytes) : 10481988
                            ID : 749094372
                            Device/File Name : /dev/raw/raw1
                            Device/File integrity check succeeded

                            Device/File not configured

                            Cluster registry integrity check succeeded


                            RAC2 ocrcheck

                            oracle@rac2 ~$ cd /u01/crs/oracle/product/10.2.0/crs/bin
                            oracle@rac2 bin$ ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                            Version : 2
                            Total space (kbytes) : 10482292
                            Used space (kbytes) : 304
                            Available space (kbytes) : 10481988
                            ID : 1155650855
                            Device/File Name : /dev/raw/raw1
                            Device/File integrity check succeeded

                            Device/File not configured

                            Cluster registry integrity check succeeded

                            use (metalink 344994.1 ) check by "ocrcheck" and remap
                            RAC1 - > ID : 749094372
                            RAC2 -> ID : 1155650855
                            You map wrong path device name, Please check

                            If your system use multi-path device... you should bind the raw devices to the multipath devices name.

                            Good Luck

                            Edited by: Surachart Opun (HunterX) on Nov 3, 2009 6:20 PM
                            • 11. Re: Failure at final check of Oracle CRS stack.10  on the second node
                              103231
                              I am running into the same problem. Has anyone found a solution to this?