1 2 Previous Next 22 Replies Latest reply: Jul 24, 2013 3:35 AM by user8010309 RSS

    root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1

    759836
      Hello all,
      root.sh fails with the errors below. SR with Oracle opened. Will post the resolution when it is available. Any insights in the meantime? Thank you!

      System information:
      OS: AIX 6.1
      Runcluvfy.sh reported no issue
      Permissions on the raw devices set to 660 and ownership is oracle:dba
      Using external redundancy for ASM, ASM instance is online

      Permissions on block and raw device files
      system1:ux460p1> ls -l /dev/hdisk32
      brw-rw---- 1 oracle dba 17, 32 Mar 11 16:50 /dev/hdisk32
      system11:ux460p1> ls -l /dev/rhdisk32
      crw-rw---- 1 oracle dba 17, 32 Mar 12 15:52 /dev/rhdisk32


      ocrconfig.log
      racle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
      2010-03-15 19:17:19.773: [ OCRCONF][1]ocrconfig starts...
      2010-03-15 19:17:19.775: [ OCRCONF][1]Upgrading OCR data
      2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
      2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
      [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]

      2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
      2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
      2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
      2010-03-15 19:17:20.603: [  OCRRAW][1]proprinit: Could not open raw device
      2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
      2010-03-15 19:17:20.604: [ default][1]a_init:7!: Backend init unsuccessful : [26]
      2010-03-15 19:17:20.604: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
      2010-03-15 19:17:20.604: [  OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
      2010-03-15 19:17:20.605: [ OCRCONF][1]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
      2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
      2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
      [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]

      2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
      2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
      2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
      2010-03-15 19:17:20.966: [  OCRRAW][1]proprinit: Could not open raw device
      2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
      2010-03-15 19:17:20.966: [ default][1]a_init:7!: Backend init unsuccessful : [26]
      2010-03-15 19:17:21.412: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
      2010-03-15 19:17:21.412: [  OCRRAW][1]proprior: Header check from OCR device 0 offset 0 failed (26).
      2010-03-15 19:17:21.414: [  OCRRAW][1]ibctx: Failed to read the whole bootblock. Assumes invalid format.
      2010-03-15 19:17:21.414: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22

      2010-03-15 19:17:21.534: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
      2010-03-15 19:17:21.701: [  OCRRAW][1]iniconfig:No 92 configuration
      2010-03-15 19:17:21.701: [  OCRAPI][1]a_init:6a: Backend init successful
      2010-03-15 19:17:21.764: [ OCRCONF][1]Initialized DATABASE keys
      2010-03-15 19:17:21.770: [ OCRCONF][1]Successfully set skgfr block 0
      2010-03-15 19:17:21.771: [ OCRCONF][1]Exiting [status=success]...



      **alert.log**
      Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
      2010-03-15 19:12:00.148
      [client(483478)]CRS-2106:The OLR location /u01/app/grid/cdata/ux460p1.olr is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_483478.log.
      2010-03-15 19:12:00.171
      [client(483478)]CRS-2101:The OLR was formatted using version 3.
      2010-03-15 14:16:18.620
      [ohasd(471204)]CRS-2112:The OLR service started on node ux460p1.
      2010-03-15 14:16:18.720
      [ohasd(471204)]CRS-8017:location: /etc/oracle/lastgasp has 8 reboot advisory log files, 0 were announced and 0 errors occurred
      2010-03-15 14:16:18.847
      [ohasd(471204)]CRS-2772:Server 'ux460p1' has been assigned to pool 'Free'.
      2010-03-15 14:16:54.107
      [ctssd(340174)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
      2010-03-15 14:16:54.123
      [ctssd(340174)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
      2010-03-15 14:16:54.917
      [ctssd(340174)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
      2010-03-15 19:17:21.414
      [client(376968)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_376968.log.
      2010-03-15 19:17:21.701
      [client(376968)]CRS-1001:The OCR was formatted using version 3.
      2010-03-15 14:17:24.888
      [crsd(303252)]CRS-1012:The OCR service started on node ux460p1.
      2010-03-15 14:17:56.344
      [ctssd(340174)]CRS-2405:The Cluster Time Synchronization Service on host ux460p1 is shutdown by user
      2010-03-15 14:19:14.855
      [ctssd(340188)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
      2010-03-15 14:19:14.870
      [ctssd(340188)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
      2010-03-15 14:19:15.638
      [ctssd(340188)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
      2010-03-15 14:19:32.985
      [crsd(417946)]CRS-1012:The OCR service started on node ux460p1.
      2010-03-15 14:19:35.250
      [crsd(417946)]CRS-1201:CRSD started on node ux460p1.
      2010-03-15 14:19:35.698
      [ohasd(471204)]CRS-2765:Resource 'ora.crsd' has failed on server 'ux460p1'.
      2010-03-15 14:19:38.928
        • 1. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
          Babu Baskar
          SR with Oracle opened. Will post the resolution when it is available
          Thanks for sharing the details.

          Just one information to you; We do not have any access to post metalink details. If you want post your SR details. Then ask to oracle make it your SR as PUBILC.

          Babu
          • 3. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
            Boochi
            The root.sh fails on 10gR2 and 11gR1 with VIP issues on AIX6.1 nodes. I'm not sure whether it's fixed in 11gR2 or not. See the metalink Note below.

            VIP CANNOT START ON AIX 6.1 BECAUSE NETSTAT HAS A NEW COLUMN [ID 805536.1]
            • 4. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
              759836
              Hello, I will try to get a response from Oracle on the VIP issue. No response from Oracle yet.

              I keep looking around and this is what I determined: It is not an access-to-disk issue. That was my first suspicion.

              cluvfy comp ssa validated the accessibility of the shared disk.

              Also the diskgroup +DATA was created and mounted. However, it was NOT onlined because ASM client process (I assume that the process is called ora.asm) could not communicate with crsd/ohasd.  But when I tried to connect to the ASM instances, it is down but all processes are running. Since crsd is down, I cannot shut it down by using srvctl. When I connect using sqlplus, I connect to an idle instance. That is peculiar.


              I am looking at the document 1053147.1 which describes dependencies between processes during the Grid infrastructure startup.
              This is the sequence: ohasd starts orarootagent which then starts crsd which starts oraagen then finally starts ora.sm. But maybe that is a regular startup sequence, not the installation sequence startup. So it seems that crsd should be running by the time ASM is running.

              Since crsd is not running, the VIPs could not be onlined. So I do not whether there is still the VIP bug. I assume that they fixed it in 11gR2.

              This is rather an obscure issue.
              • 5. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                fjgaliaga
                we heve the same problem intalling 11gR2 with vios

                have any solution????
                • 6. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                  user12083506
                  I am having this exact same issue on RHEL 5.4 - I get the same messages that the OP got in his logs. And my runcluvfy passes it. Does anyone have an idea? I submitted a SR to Oracle and will see what they say.

                  My interesting infrastructure setup is that I'm running Infiniband for both public and private connections for the cluster. And that they are using the same interface, different subnets to communicate.

                  And after the root.sh fails with:

                  /apps/grid/oracle/product/11.2.0/grid/bin/srvctl start nodeapps -n racnode1 ... failed
                  Configure Oracle Grid Infrastructure for a Cluster ... failed

                  /apps/grid/oracle/product/11.2.0/grid/bin/srvctl status nodeapps
                  VIP RACNODE1-VIP is enabled
                  VIP RACNODE1-VIP is not running
                  VIP RACNODE2-VIP is enabled
                  VIP RACNODE2-VIP is not running
                  Network is enabled
                  Network is not running on node: racnode1
                  Network is not running on node: racnode2
                  GSD is disabled
                  GSD is not running on node: racnode1
                  GSD is not running on node: racnode2
                  ONS is enabled
                  ONS daemon is not running on node: racnode1
                  ONS daemon is running on node: racnode2
                  eONS is enabled
                  eONS daemon is running on node: racnode1
                  eONS daemon is running on node: racnode2


                  Any ideas what I could try, while I wait for a response?
                  • 7. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                    Zekeriya Besiroglu
                    Hi
                    Pls try

                    Connect root, run “$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force” on all nodes, except the last one.

                    Hope this helps
                    Zekeriya Besiroglu
                    Http://zekeriyabesiroglu.blogspot.com
                    • 8. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                      Mithzz
                      Is this issue resolved ? I am getting the same issue , but i have shared disks NFS mounted from a Linux NFS server.

                      --Amith                                                                                                                                                                                                                                                           
                      • 9. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                        60861
                        I have exactly the same error installing on EL5.

                        Originally, I was trying to create a 2 node cluster, and kept running into this problem. Now I am just trying to get it installed on a single node, but same error. I am running in Virtualbox and have started from scratch 3 times and run into the same error every time.

                        Here is "some" of the output of root.sh (everything before this point was successful.


                        ASM created and started successfully.

                        DiskGroup DATA created successfully.

                        clscfg: -install mode specified
                        Successfully accumulated necessary OCR keys.
                        Creating OCR keys for user 'root', privgrp 'root'..
                        Operation successful.
                        CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
                        CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
                        CRS-4256: Updating the profile
                        Successful addition of voting disk 2a7ce7f467c64f0dbfae31cb7fe7ee71.
                        Successfully replaced voting disk group with +DATA.
                        CRS-4256: Updating the profile
                        CRS-4266: Voting file(s) successfully replaced
                        ## STATE File Universal Id File Name Disk group
                        -- ----- ----------------- --------- ---------
                        1. ONLINE 2a7ce7f467c64f0dbfae31cb7fe7ee71 (ORCL:DISK1) [DATA]
                        Located 1 voting disk(s).
                        CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
                        CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
                        CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
                        CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
                        CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
                        CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
                        CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
                        CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
                        CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
                        CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
                        CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
                        CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
                        CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
                        CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
                        CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
                        CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
                        CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
                        CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.asm' on 'rac1'
                        CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
                        CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
                        CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
                        CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
                        CRS-0184: Cannot communicate with the CRS daemon.

                        PRCR-1070 : Failed to check if resource ora.asm is registered
                        Cannot communicate with crsd
                        add asm ... failed
                        clsr_start_dg return error at loc: 70 rc=0
                        CRS-4535: Cannot communicate with Cluster Ready Services
                        CRS-4000: Command Create failed, or completed with errors.
                        create diskgroup DATA ... failed
                        CRS-4535: Cannot communicate with Cluster Ready Services
                        CRS-4000: Command Add failed, or completed with errors.
                        Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl add type ora.registry.acfs.type -basetype ora.local_resource.type -file /u01/app/11.2.0/grid/crs/template/registry.acfs.type
                        add ora.registry.acfs.type ... failed
                        PRCR-1070 : Failed to check if resource ora.net1.network is registered
                        Cannot communicate with crsd
                        add scan=rac-scan ... failed
                        Preparing packages for installation...
                        cvuqdisk-1.0.7-1
                        Configure Oracle Grid Infrastructure for a Cluster ... failed
                        Updating inventory properties for clusterware
                        Starting Oracle Universal Installer...

                        Checking swap space: must be greater than 500 MB. Actual 3807 MB Passed
                        The inventory pointer is located at /etc/oraInst.loc
                        The inventory is located at /u01/app/oraInventory
                        'UpdateNodeList' was successful.


                        Here is the alert log:


                        Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
                        2010-11-09 09:41:44.286
                        [client(7833)]CRS-2106:The OLR location /u01/app/11.2.0/grid/cdata/rac1.olr is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/client/ocrconfig_7833.log.
                        2010-11-09 09:41:44.347
                        [client(7833)]CRS-2101:The OLR was formatted using version 3.
                        2010-11-09 09:42:38.498
                        [ohasd(8192)]CRS-2112:The OLR service started on node rac1.
                        2010-11-09 09:42:39.104
                        [ohasd(8192)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
                        2010-11-09 09:42:39.319
                        [ohasd(8192)]CRS-8017:location: /etc/oracle/lastgasp has 34 reboot advisory log files, 0 were announced and 0 errors occurred
                        2010-11-09 09:43:22.145
                        [ohasd(8192)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                        2010-11-09 09:43:27.158
                        [cssd(9445)]CRS-1713:CSSD daemon is started in exclusive mode
                        2010-11-09 09:43:29.789
                        [cssd(9445)]CRS-1709:Lease acquisition failed for node rac1 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log
                        2010-11-09 09:43:47.609
                        [cssd(9445)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
                        2010-11-09 09:43:49.749
                        [ctssd(9503)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.
                        2010-11-09 09:43:49.873
                        [ctssd(9503)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
                        2010-11-09 09:43:50.431
                        [ctssd(9503)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
                        [client(9697)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
                        [client(9701)]CRS-10001:ACFS-9322: done.
                        2010-11-09 09:44:46.165
                        [client(9730)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/client/ocrconfig_9730.log.
                        2010-11-09 09:44:47.369
                        [client(9730)]CRS-1001:The OCR was formatted using version 3.
                        2010-11-09 09:44:55.086
                        [crsd(9783)]CRS-1012:The OCR service started on node rac1.
                        2010-11-09 09:44:57.652
                        [cssd(9445)]CRS-1605:CSSD voting file is online: ORCL:DISK1; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.
                        2010-11-09 09:44:58.229
                        [cssd(9445)]CRS-1626:A Configuration change request completed successfully
                        2010-11-09 09:44:58.257
                        [cssd(9445)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
                        2010-11-09 09:45:22.736
                        [ctssd(9503)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user
                        2010-11-09 09:45:35.491
                        [cssd(9445)]CRS-1603:CSSD on node rac1 shutdown by user.
                        2010-11-09 09:45:55.966
                        [cssd(10246)]CRS-1713:CSSD daemon is started in clustered mode
                        2010-11-09 09:46:28.938
                        [cssd(10246)]CRS-1707:Lease acquisition for node rac1 number 1 completed
                        2010-11-09 09:46:28.985
                        [cssd(10246)]CRS-1605:CSSD voting file is online: ORCL:DISK1; details in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log.
                        2010-11-09 09:46:46.823
                        [cssd(10246)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
                        2010-11-09 09:46:48.269
                        [ctssd(10322)]CRS-2403:The Cluster Time Synchronization Service on host rac1 is in observer mode.
                        2010-11-09 09:46:48.288
                        [ctssd(10322)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
                        2010-11-09 09:46:49.191
                        [ctssd(10322)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
                        2010-11-09 09:47:12.676
                        [crsd(10446)]CRS-1012:The OCR service started on node rac1.
                        2010-11-09 09:47:18.107
                        [crsd(10446)]CRS-1201:CRSD started on node rac1.
                        2010-11-09 09:47:19.146
                        [crsd(10446)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
                        2010-11-09 09:47:46.181
                        [ohasd(8192)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
                        2010-11-09 09:47:55.266
                        [crsd(10734)]CRS-1012:The OCR service started on node rac1.
                        2010-11-09 09:47:56.658
                        [crsd(10734)]CRS-1201:CRSD started on node rac1.
                        2010-11-09 09:48:01.041
                        [ohasd(8192)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
                        2010-11-09 09:48:07.486
                        [crsd(10834)]CRS-1012:The OCR service started on node rac1.
                        2010-11-09 09:48:08.967
                        [crsd(10834)]CRS-1201:CRSD started on node rac1.
                        2010-11-09 09:48:12.472
                        [ohasd(8192)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.
                        2010-11-09 09:48:19.757
                        [crsd(10933)]CRS-1012:The OCR service started on node rac1.
                        2010-11-09 09:48:21.868
                        [crsd(10933)]CRS-1201:CRSD started on node rac1.
                        2010-11-09 09:48:24.612
                        [ohasd(8192)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.


                        so, it seems to be breaking on ora.crsd
                        • 10. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                          8398219
                          Mrothwel,
                          Did you have a look at /u01/app/11.2.0/grid/log/rac1/client/ocrconfig_7833.log ?

                          Did you find any logs in
                          $GRID_HOME/cfgtoollogs/crsconfig/
                          • 11. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                            user12083506
                            For me, it turned out to be the fact that the public and private connections were using the same interface - even though they were on different subnets. I un-installed and re-installed with different interfaces and it worked fine.
                            • 12. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                              60861
                              Public and Private are on different devices and subnets.

                              No logfile named: ocrconfig_7833.log

                              I do have ocrconfig_7089.log and ocrconfig_8985.log

                              Here is their contents:

                              ocrconfig_7089.log:
                              Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
                              2010-11-09 13:38:32.518: [ OCRCONF][2819644944]ocrconfig starts...
                              2010-11-09 13:38:32.542: [ OCRCONF][2819644944]Upgrading OCR data
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
                              2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
                              2010-11-09 13:38:32.576: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
                              2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprinit: Could not open raw device
                              2010-11-09 13:38:32.576: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
                              2010-11-09 13:38:32.577: [ OCRCONF][2819644944]Exporting OCR data to [OCRUPGRADEFILE]
                              2010-11-09 13:38:32.577: [  OCRAPI][2819644944]a_init:7!: Backend init unsuccessful : [33]
                              2010-11-09 13:38:32.577: [ OCRCONF][2819644944]There was no previous version of OCR. error:[PROCL-33: Oracle Local Registry is not configured]
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
                              2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
                              2010-11-09 13:38:32.578: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
                              2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprinit: Could not open raw device
                              2010-11-09 13:38:32.578: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
                              2010-11-09 13:38:32.579: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
                              2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
                              2010-11-09 13:38:32.591: [  OCRRAW][2819644944]ibctx: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:38:32.591: [  OCRRAW][2819644944]proprinit:problem reading the bootblock or superbloc 22

                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 0
                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 1
                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 2
                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 3
                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 4
                              2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 5
                              2010-11-09 13:38:32.591: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:38:32.681: [  OCRAPI][2819644944]a_init:6a: Backend init successful
                              2010-11-09 13:38:32.699: [ OCRCONF][2819644944]Initialized DATABASE keys
                              2010-11-09 13:38:32.700: [ OCRCONF][2819644944]Exiting [status=success]...



                              ocrconfig_8985.log:
                              Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
                              2010-11-09 13:41:28.169: [ OCRCONF][2281741840]ocrconfig starts...
                              2010-11-09 13:41:28.175: [ OCRCONF][2281741840]Upgrading OCR data
                              2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
                              2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
                              [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]

                              2010-11-09 13:41:31.208: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
                              2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
                              2010-11-09 13:41:31.210: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
                              2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprinit: Could not open raw device
                              2010-11-09 13:41:31.211: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
                              2010-11-09 13:41:31.213: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
                              2010-11-09 13:41:31.214: [ OCRCONF][2281741840]Exporting OCR data to [OCRUPGRADEFILE]
                              2010-11-09 13:41:31.216: [  OCRAPI][2281741840]a_init:7!: Backend init unsuccessful : [33]
                              2010-11-09 13:41:31.216: [ OCRCONF][2281741840]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
                              2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
                              2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
                              [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]

                              2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
                              2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
                              2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
                              2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprinit: Could not open raw device
                              2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
                              2010-11-09 13:41:32.536: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
                              2010-11-09 13:41:35.359: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
                              2010-11-09 13:41:35.361: [  OCRRAW][2281741840]proprior: Header check from OCR device 0 offset 0 failed (26).
                              2010-11-09 13:41:35.363: [  OCRRAW][2281741840]ibctx: Failed to read the whole bootblock. Assumes invalid format.
                              2010-11-09 13:41:35.363: [  OCRRAW][2281741840]proprinit:problem reading the bootblock or superbloc 22

                              2010-11-09 13:41:35.843: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
                              2010-11-09 13:41:36.430: [  OCRRAW][2281741840]iniconfig:No 92 configuration
                              2010-11-09 13:41:36.431: [  OCRAPI][2281741840]a_init:6a: Backend init successful
                              2010-11-09 13:41:36.540: [ OCRCONF][2281741840]Initialized DATABASE keys
                              2010-11-09 13:41:36.545: [ OCRCONF][2281741840]Successfully set skgfr block 0
                              2010-11-09 13:41:36.552: [ OCRCONF][2281741840]Exiting [status=success]...


                              Both of these log files show errors, then they show success??????
                              • 13. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                                60861
                                Here's something else that I found in the <GRID_HOME>/cfgtoollogs/crsconfig/rootcrs_<node1>.log:

                                2010-11-09 13:44:36: check ASM exists done and ASM does not exist
                                2010-11-09 13:44:36: Prior version ASM does not exist , Invoking add asm
                                2010-11-09 13:44:36: Invoking "/u01/app/11.2.0/grid/bin/srvctl add asm"
                                2010-11-09 13:44:36: trace file=/u01/app/11.2.0/grid/cfgtoollogs/crsconfig/srvmcfg1.log
                                2010-11-09 13:44:36: Running as user oracle: /u01/app/11.2.0/grid/bin/srvctl add asm
                                2010-11-09 13:44:36: Invoking "/u01/app/11.2.0/grid/bin/srvctl add asm" as user "oracle"
                                2010-11-09 13:44:44: "/u01/app/11.2.0/grid/bin/srvctl add asm" failed with status 512.
                                2010-11-09 13:44:44: add asm ... failed
                                2010-11-09 13:44:44: Adding ASM diskgroup resource
                                2010-11-09 13:44:44: Running as user oracle: /u01/app/11.2.0/grid/bin/crsctl create diskgroup DATA
                                2010-11-09 13:44:44: Invoking "/u01/app/11.2.0/grid/bin/crsctl create diskgroup DATA" as user "oracle"
                                2010-11-09 13:44:47: create diskgroup DATA ... failed


                                So, the add asm command failed but I can find nothing online with this error - any ideas?
                                • 14. Re: root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1
                                  60861
                                  Here is the srvmcfg2.log. It shows that rac-scan is not a cluster node. This doesn't make sense to me, what would cause that? I'll also include the hosts file.


                                  # cat srvmcfg2.log
                                  [main] [ 2010-11-09 13:44:51.129 MST ] [OPSCTLDriver.setInternalDebugLevel:168] Trace.configure returns true
                                  [main] [ 2010-11-09 13:44:51.157 MST ] [OPSCTLDriver.setInternalDebugLevel:171] tracing is true at level 2 to file /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/srvmcfg2.log
                                  [main] [ 2010-11-09 13:44:51.157 MST ] [OPSCTLDriver.main:115] SRVCTL arguments : args[0]=add args[1]=scan args[2]=-n args[3]=rac-scan
                                  [main] [ 2010-11-09 13:44:51.340 MST ] [Version.isPre:274] version to be checked 11.2.0.1.0 major version to check against10
                                  [main] [ 2010-11-09 13:44:51.342 MST ] [Version.isPre:285] isPre.java: Returning FALSE
                                  [main] [ 2010-11-09 13:44:51.344 MST ] [OCR.loadLibrary:308]
                                  Inside constructor of OCR
                                  [main] [ 2010-11-09 13:44:51.393 MST ] [OCR.loadLibrary:316] Going to load the ocr library
                                  [main] [ 2010-11-09 13:44:51.394 MST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
                                  [main] [ 2010-11-09 13:44:51.395 MST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
                                  [main] [ 2010-11-09 13:44:51.395 MST ] [Library.load:254] Loading library /u01/app/11.2.0/grid/lib/libsrvmocr11.so
                                  [main] [ 2010-11-09 13:44:51.547 MST ] [OCR.loadLibrary:318] loaded ocr libraries
                                  [main] [ 2010-11-09 13:44:51.548 MST ] [OCR.isCluster:939] Calling OCRNative for isCluster()
                                  [main] [ 2010-11-09 13:44:51.584 MST ] [nativesystem.OCRNative.Native] JNI: isCluster() Successfully initialized meta context
                                  [main] [ 2010-11-09 13:44:51.585 MST ] [nativesystem.OCRNative.Native] JNI: procr_get_conf() retValue = 0
                                  [main] [ 2010-11-09 13:44:51.585 MST ] [nativesystem.OCRNative.Native] The OCR file is configured & local_only=false

                                  [main] [ 2010-11-09 13:44:51.586 MST ] [OCR.isCluster:943] OCR Result status = true
                                  [main] [ 2010-11-09 13:44:51.586 MST ] [OCR.isCluster:955] Bolean result = true
                                  [main] [ 2010-11-09 13:44:51.586 MST ] [CommandLineParser.<init>:331] Is this a cluster? : true
                                  [main] [ 2010-11-09 13:44:51.587 MST ] [CommandLineParser.parse:343] parsing cmdline args
                                  [main] [ 2010-11-09 13:44:51.587 MST ] [CommandLineParser.parse2WordCommandOptions:2223] parsing 2-word cmdline, args count = 4
                                  [main] [ 2010-11-09 13:44:51.587 MST ] [CommandLineParser.parse2WordCommandOptions:2231] Checking single valued options
                                  [main] [ 2010-11-09 13:44:51.588 MST ] [CommandLineParser.parse2WordCommandOptions:2236] argInx = 2, option =-n
                                  [main] [ 2010-11-09 13:44:51.588 MST ] [CommandLineParser.parse2WordCommandOptions:2238] Option string: -n
                                  [main] [ 2010-11-09 13:44:51.588 MST ] [CommandLineParser.parse2WordCommandOptions:2318] Checking mandatory options
                                  [main] [ 2010-11-09 13:44:51.589 MST ] [CommandLineParser.parse2WordCommandOptions:2331] brace_begin_index: -1 vertical_bar_index: -1 brace_end_index: -1
                                  [main] [ 2010-11-09 13:44:51.595 MST ] [CommandLineParser.checkIntegersAndSpecificValues:3143] check specificMultiValues
                                  [main] [ 2010-11-09 13:44:51.595 MST ] [CommandLineParser.validateSingleValueOption:3240] Validating single-value option: n , value = rac-scan
                                  [main] [ 2010-11-09 13:44:51.596 MST ] [CommandLineParser.isValidNodeName:3343] Calling isValidNodeName
                                  [main] [ 2010-11-09 13:44:51.644 MST ] [HASContext.<init>:100] moduleInit = 19
                                  [main] [ 2010-11-09 13:44:51.645 MST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
                                  [main] [ 2010-11-09 13:44:51.645 MST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
                                  [main] [ 2010-11-09 13:44:51.645 MST ] [Library.load:254] Loading library /u01/app/11.2.0/grid/lib/libsrvmhas11.so
                                  [main] [ 2010-11-09 13:44:51.658 MST ] [has.HASContextNative.Native] prsr_trace: no lsf ctx, line=Native: allocHASContext

                                  [main] [ 2010-11-09 13:44:51.658 MST ] [has.HASContextNative.Native]
                                  allocHASContext: Came in
                                  [main] [ 2010-11-09 13:44:51.659 MST ] [has.HASContextNative.Native] allocHASContext: module_init = 19
                                  [main] [ 2010-11-09 13:44:51.678 MST ] [has.HASContextNative.Native]
                                  allocHASContext: META context [1]
                                  [main] [ 2010-11-09 13:44:51.695 MST ] [has.HASContextNative.Native]
                                  allocHASContext: LSF context [1]
                                  [main] [ 2010-11-09 13:44:51.727 MST ] [has.HASContextNative.Native] prsr_trace: Native: prsr_initCLSS

                                  [main] [ 2010-11-09 13:44:51.767 MST ] [has.HASContextNative.Native]
                                  allocHASContext: CLSS context [1]
                                  [main] [ 2010-11-09 13:44:51.767 MST ] [has.HASContextNative.Native]
                                  allocHASContext: retval [1]
                                  [main] [ 2010-11-09 13:44:51.768 MST ] [ClusterUtil.<init>:65] ClusterUtil Instance created
                                  [main] [ 2010-11-09 13:44:51.768 MST ] [ClusterUtil.<init>:68] ClusterUtil m_bSKGXN=true
                                  [main] [ 2010-11-09 13:44:51.772 MST ] [has.ClusterUtilNative.Native] prsr_trace: Native: getHostName

                                  [main] [ 2010-11-09 13:44:51.773 MST ] [has.ClusterUtilNative.Native] prsr_trace: Native: getHostName:hostname length: 256

                                  [main] [ 2010-11-09 13:44:51.778 MST ] [has.ClusterUtilNative.Native] prsr_trace: prsr_throwException: oracle/ops/mgmt/has/HASNativeContextException[null--Unexpected CLSS error]

                                  [main] [ 2010-11-09 13:44:51.817 MST ] [CommandLineParser.isValidNodeName:3349] Getting message: PRKC-1056 : Failed to get the hostname for node rac-scan
                                  PRKH-1001 : HASContext Internal Error
                                  [PRKH-1001 : HASContext Internal Error(Unexpected CLSS error:[12])]
                                  [main] [ 2010-11-09 13:44:51.818 MST ] [CommandLineParser.isValidNodeName:3350] Specified name rac-scan is not a cluster node
                                  [main] [ 2010-11-09 13:44:51.818 MST ] [OPSCTLDriver.execute:180] executing srvctl command
                                  [main] [ 2010-11-09 13:44:51.827 MST ] [OPSCTLDriver.execute:189] checking for -H option: print verbose command syntax
                                  [main] [ 2010-11-09 13:44:51.828 MST ] [OPSCTLDriver.execute:235] executing 2-word command verb=7 noun=108
                                  [main] [ 2010-11-09 13:44:51.865 MST ] [AddAction.executeSCAN:1515] Adding SCAN
                                  [main] [ 2010-11-09 13:44:51.898 MST ] [AddAction.executeSCAN:1540] network number = 1
                                  [main] [ 2010-11-09 13:44:51.955 MST ] [has.HASContextNative.Native] prsr_trace: Native: allocHASContext

                                  [main] [ 2010-11-09 13:44:51.957 MST ] [has.HASContextNative.Native]
                                  allocHASContext: Came in
                                  [main] [ 2010-11-09 13:44:51.957 MST ] [has.HASContextNative.Native] allocHASContext: module_init = 64
                                  [main] [ 2010-11-09 13:44:51.957 MST ] [has.HASContextNative.Native] prsr_trace: Native: prsr_initCLSCRS

                                  [main] [ 2010-11-09 13:44:51.961 MST ] [has.HASContextNative.Native] prsr_trace: prsr_initCLSCRS: before calling prcr_init_crs

                                  [main] [ 2010-11-09 13:44:51.966 MST ] [has.HASContextNative.Native]
                                  allocHASContext: CLSCRS context [1]
                                  [main] [ 2010-11-09 13:44:51.967 MST ] [has.HASContextNative.Native]
                                  allocHASContext: retval [1]
                                  [main] [ 2010-11-09 13:44:52.367 MST ] [CRSNativeResult.clscrsCB:147] clscrs message: 'clscconnect failed with clsc ret 9'
                                  [main] [ 2010-11-09 13:44:52.368 MST ] [CRSNativeResult.clscrsCB:147] clscrs message: 'error connecting to CRSD at [(ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))] clsccon 184
                                  '


                                  /etc/hosts:

                                  # cat /etc/hosts
                                  # Do not remove the following line, or various programs
                                  # that require network functionality will fail.
                                  # 127.0.0.1 localhost.localdomain localhost
                                  # ::1 localhost6.localdomain6 localhost6

                                  # Public
                                  10.1.8.210 rac1.localdomain rac1

                                  # Private
                                  192.168.1.210 rac1-priv.localdomain rac1-priv

                                  # Virtual
                                  10.1.8.230 rac1-vip.localdomain rac1-vip

                                  # SCAN
                                  10.1.8.201 rac-scan.localdomain rac-scan
                                  1 2 Previous Next