6 Replies Latest reply on Dec 29, 2010 11:23 PM by Samiksha

    During the installation of grid infra(cluster) for Oracle 11.2 RAC one.

    807872
      Good Day All, and thanks in advance…

      During the installation of grid infrastructure(cluster) for Oracle 11.2 RAC One Node on AIX6.1 ( PROD) , ASM used. I am getting below errors when executing ./root.sh

      Upon investigation ,I managed to get note: 1068212.1 from the support oracle site ( see below for details) . I might be hitting Unpublished bug 8670579. I also logged Severity 2 SR with Oracle support to get the bug/patch fix and no one has attended the call.

      This might be configuration issue or otherwise , if you have experienced the same issue please assist ? ( if you need more logfiles please feel free to request)….




      I ran the Cluster Verify Check – all passed.


      Many Thanks
      Ezekiel Filane

      /u01/app/11.2.0/grid#./root.sh
      Running Oracle 11g root.sh script...

      The following environment variables are set as:
      ORACLE_OWNER= grid
      ORACLE_HOME= /u01/app/11.2.0/grid

      Enter the full pathname of the local bin directory: [usr/local/bin]:
      The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
      The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
      The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:


      Creating /etc/oratab file...
      Entries will be added to the /etc/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root.sh script.
      Now product-specific root actions will be performed.
      2010-10-19 10:33:11: Parsing the host name
      2010-10-19 10:33:11: Checking for super user privileges
      2010-10-19 10:33:11: User has super user privileges
      Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
      Creating trace directory
      User grid has the required capabilities to run CSSD in realtime mode
      LOCAL ADD MODE
      Creating OCR keys for user 'root', privgrp 'system'..
      Operation successful.
      root wallet
      root wallet cert
      root cert export
      peer wallet
      profile reader wallet
      pa wallet
      peer wallet keys
      pa wallet keys
      peer cert request
      pa cert request
      peer cert
      pa cert
      peer root cert TP
      profile reader root cert TP
      pa root cert TP
      peer pa cert TP
      pa peer cert TP
      profile reader pa cert TP
      profile reader peer cert TP
      peer user cert
      pa user cert
      Adding daemon to inittab
      CRS-4123: Oracle High Availability Services has been started.
      ohasd is starting
      CRS-2672: Attempting to start 'ora.gipcd' on 'csgipm'
      CRS-2672: Attempting to start 'ora.mdnsd' on 'csgipm'
      CRS-2676: Start of 'ora.gipcd' on 'csgipm' succeeded
      CRS-2676: Start of 'ora.mdnsd' on 'csgipm' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'csgipm'
      CRS-2676: Start of 'ora.gpnpd' on 'csgipm' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'csgipm'
      CRS-2676: Start of 'ora.cssdmonitor' on 'csgipm' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'csgipm'
      CRS-2672: Attempting to start 'ora.diskmon' on 'csgipm'
      CRS-2676: Start of 'ora.diskmon' on 'csgipm' succeeded
      CRS-2676: Start of 'ora.cssd' on 'csgipm' succeeded
      CRS-2672: Attempting to start 'ora.ctssd' on 'csgipm'
      Start action for daemon aborted
      CRS-2674: Start of 'ora.ctssd' on 'csgipm' failed
      CRS-2679: Attempting to clean 'ora.ctssd' on 'csgipm'
      CRS-2681: Clean of 'ora.ctssd' on 'csgipm' succeeded
      CRS-4000: Command Start failed, or completed with errors.
      Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init
      Start of resource "ora.ctssd -init" failed
      Clusterware exclusive mode start of resource ora.ctssd failed
      CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
      CRS-4000: Command Stop failed, or completed with errors.
      Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
      Stop of resource "ora.crsd -init" failed
      Failed to stop CRSD
      CRS-2500: Cannot stop resource 'ora.asm' as it is not running
      CRS-4000: Command Stop failed, or completed with errors.
      Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
      Stop of resource "ora.asm -init" failed
      Failed to stop ASM
      CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'csgipm'
      CRS-2677: Stop of 'ora.cssdmonitor' on 'csgipm' succeeded
      CRS-2673: Attempting to stop 'ora.cssd' on 'csgipm'
      CRS-2677: Stop of 'ora.cssd' on 'csgipm' succeeded
      CRS-2673: Attempting to stop 'ora.gpnpd' on 'csgipm'
      CRS-2677: Stop of 'ora.gpnpd' on 'csgipm' succeeded
      CRS-2673: Attempting to stop 'ora.gipcd' on 'csgipm'
      CRS-2677: Stop of 'ora.gipcd' on 'csgipm' succeeded
      CRS-2673: Attempting to stop 'ora.mdnsd' on 'csgipm'
      CRS-2677: Stop of 'ora.mdnsd' on 'csgipm' succeeded
      Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_csgipm.log for details
      csgipm:/u01/app/11.2.0/grid#ps -ef | grep pmon
      root 6160492 3932160 0 10:54:13 pts/2 0:00 grep pmon




      more /u01/app/11.2.0/grid/log/csgipm/client/ocrconfig_5767204.log



      csgipm:/usr/sbin#more /u01/app/11.2.0/grid/log/csgipm/client/ocrconfig_5767204.log
      2010-10-19 10:33:14.435: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
      2010-10-19 10:33:14.435: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
      2010-10-19 10:33:14.435: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
      2010-10-19 10:33:14.435: [  OCRRAW][1]proprioini: all disks are not OCR/OLR formatted
      2010-10-19 10:33:14.435: [  OCRRAW][1]proprinit: Could not open raw device
      2010-10-19 10:33:14.442: [ default][1]a_init:7!: Backend init unsuccessful : [26]
      2010-10-19 10:33:14.461: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
      2010-10-19 10:33:14.461: [  OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
      2010-10-19 10:33:14.461: [ OCRCONF][1]There was no previous version of OCR. error:[PROCL-33: Oracle Local Registry is not configured]
      2010-10-19 10:33:14.461: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
      2010-10-19 10:33:14.461: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
      2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
      2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
      2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
      2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
      2010-10-19 10:33:14.462: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
      2010-10-19 10:33:14.462: [  OCRRAW][1]proprioini: all disks are not OCR/OLR formatted
      2010-10-19 10:33:14.462: [  OCRRAW][1]proprinit: Could not open raw device
      2010-10-19 10:33:14.462: [ default][1]a_init:7!: Backend init unsuccessful : [26]
      2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
      2010-10-19 10:33:14.463: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
      2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
      2010-10-19 10:33:14.483: [  OCRRAW][1]ibctx: Failed to read the whole bootblock. Assumes invalid format.
      2010-10-19 10:33:14.483: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22

      2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 0
      2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 1
      2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 2
      2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 3
      2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 4
      2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 5
      2010-10-19 10:33:14.484: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
      2010-10-19 10:33:14.541: [  OCRAPI][1]a_init:6a: Backend init successful
      2010-10-19 10:33:14.646: [ OCRCONF][1]Initialized DATABASE keys
      2010-10-19 10:33:14.650: [ OCRCONF][1]Exiting [status=success]...
        • 1. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
          Samiksha
          Hi,

          We are also trying to install 11.2.0.2 Grid infrastructure for Oracle RAC One Node on AIX 6.1. We did a POC in our lab environment and after much struggle got that working. Now we are building 4 clusters in the production environment and the first cluster installation failed while running root.sh on node2. We already have a Sev1 ticket open with Oracle Support but have not heard anything.

          Here is root.sh output from node2. The two node names are p01dou416 and p01dou417.

          CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node p01dou416, number 1, and is terminating
          An active cluster was found during exclusive startup, restarting to join the cluster
          Failed to start Oracle Clusterware stack
          Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1020.
          /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
          [root@P01DOU417] /u01/app/11.2.0/grid #

          LOG output: /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/ rootcrs_p01dou417.log

          2010-11-13 17:22:14: Successfully started requested Oracle stack daemons
          2010-11-13 17:22:14: Starting CSS in clustered mode
          2010-11-13 17:22:14: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl start resource ora.cssd -init
          2010-11-13 17:32:28: Command output:
          CRS-2672: Attempting to start 'ora.cssdmonitor' on 'p01dou417'
          CRS-2672: Attempting to start 'ora.gipcd' on 'p01dou417'
          CRS-2676: Start of 'ora.cssdmonitor' on 'p01dou417' succeeded
          CRS-2676: Start of 'ora.gipcd' on 'p01dou417' succeeded> CRS-2679: Attempting to clean 'ora.cssd' on 'p01dou417'
          CRS-2681: Clean of 'ora.cssd' on 'p01dou417' succeeded
          CRS-2673: Attempting to stop 'ora.diskmon' on 'p01dou417'
          CRS-2677: Stop of 'ora.diskmon' on 'p01dou417' succeeded
          CRS-2673: Attempting to stop 'ora.gipcd' on 'p01dou417'
          CRS-2677: Stop of 'ora.gipcd' on 'p01dou417' succeeded
          CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'p01dou417'
          CRS-2677: Stop of 'ora.cssdmonitor' on 'p01dou417' succeeded
          CRS-5804: Communication error with agent process
          CRS-4000: Command Start failed, or completed with errors.
          End Command output
          2010-11-13 17:32:28: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl check css
          2010-11-13 17:32:28: Command output:
          CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
          End Command output
          2010-11-13 17:32:28: Checking the status of css
          2010-11-13 17:32:33: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl check css
          2010-11-13 17:32:33: Command output:
          CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
          End Command output
          2010-11-13 17:32:33: Checking the status of css
          2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.gipcd' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2676: Start of 'ora.cssdmonitor' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2676: Start of 'ora.gipcd' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.cssd' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.diskmon' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2676: Start of 'ora.diskmon' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2674: Start of 'ora.cssd' on 'p01dou417' failed
          2010-11-13 17:32:38: CRS-2679: Attempting to clean 'ora.cssd' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2681: Clean of 'ora.cssd' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.diskmon' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2677: Stop of 'ora.diskmon' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.gipcd' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2677: Stop of 'ora.gipcd' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'p01dou417'
          2010-11-13 17:32:38: CRS-2677: Stop of 'ora.cssdmonitor' on 'p01dou417' succeeded
          2010-11-13 17:32:38: CRS-5804: Communication error with agent process
          2010-11-13 17:32:38: CRS-4000: Command Start failed, or completed with errors.
          2010-11-13 17:32:38: Failed to start Oracle Clusterware stack
          2010-11-13 17:32:38: ###### Begin DIE Stack Trace ######
          2010-11-13 17:32:38: Package File Line Calling
          2010-11-13 17:32:38: --------------- -------------------- ---- ----------
          2010-11-13 17:32:38: 1: main rootcrs.pl 324 crsconfig_lib::dietrap
          2010-11-13 17:32:38: 2: crsconfig_lib crsconfig_lib.pm 1020 main::__ANON__
          2010-11-13 17:32:38: 3: crsconfig_lib crsconfig_lib.pm 997 crsconfig_lib::start_cluster
          2010-11-13 17:32:38: 4: main rootcrs.pl 697 crsconfig_lib::perform_start_cluster
          2010-11-13 17:32:38: ####### End DIE Stack Trace #######

          2010-11-13 17:32:38: 'ROOTCRS_STACK' checkpoint has failed

          Any help on this is appreciated.

          Edited by: user12019257 on Nov 17, 2010 1:26 PM
          • 2. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
            Sebastian Solbach -Database Community-Oracle
            Hi,
            have you already checked on
            MOS Note: 11.2.0.2 Grid Infrastructure Install or Upgrade may fail due to Multicasting Requirement (Doc ID 1212703.1)

            https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1212703.1

            Regards
            Sebastian
            • 3. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
              Samiksha
              Yes we have been working with IBM and Oracle for that particular issue. Our SAs believe that the multicast test is not accurate.

              Thanks

              Samiksha
              • 4. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
                Samiksha
                Oracle issued a patch for the problem we were having. Initially we were provided a library file that was supposed to be included before running root.sh. I believe now the problem is a published bug and there is a patch available.
                • 5. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
                  827368
                  Hi Samiksha,
                  do you happen to have any further info on the published bug and the patch number? I am getting this same issue using a raw device on a ds8000 and AIX 7.1
                  • 6. Re: During the installation of grid infra(cluster) for Oracle 11.2 RAC one.
                    Samiksha
                    I donot have the patch information since Oracle provided an updated library file to us. But I have the name of the file - libhasagen11.so