8 Replies Latest reply: Apr 8, 2013 4:02 PM by 1001806 RSS

    root.sh fails on newly added Node --11gR203GA

    756561
      Hello,




      After adding Node in Oracle RAC cluster(11gR2) it succeded but root.sh fails with "Failed to initialize Oracle Cluster Registry for cluster, rc 105 Oracle Grid Infrastructure Repository configuration failed at /oracle/11g/gridhome/crs/install/crsconfig_lib.pm line 6769" Error.

      Below is snip;


      <snip>
      [grid@punlabsx86-15 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={punlabsx86-21}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={punlabvip11.vxindia.veritas.com}"

      Performing pre-checks for node addition

      Checking node reachability...
      Node reachability check passed from node "punlabsx86-15"


      Checking user equivalence...
      User equivalence check passed for user "grid"

      Checking node connectivity...

      Checking hosts config file...

      Verification of the hosts config file successful

      Check: Node connectivity for interface "eth0"
      Node connectivity passed for interface "eth0"
      TCP connectivity check passed for subnet "10.209.92.0"

      Checking subnet mask consistency...
      Subnet mask consistency check passed for subnet "10.209.92.0".
      Subnet mask consistency check passed.

      Node connectivity check passed

      Checking multicast communication...

      Checking subnet "10.209.92.0" for multicast communication with multicast group "230.0.1.0"...
      Check of subnet "10.209.92.0" for multicast communication with multicast group "230.0.1.0" passed.

      Check of multicast communication passed.

      Checking CRS integrity...

      Clusterware version consistency passed

      CRS integrity check passed

      Checking shared resources...

      Checking CRS home location...
      "/oracle/11g/gridhome" is shared
      Shared resources check for node addition passed


      Checking node connectivity...

      Checking hosts config file...

      Verification of the hosts config file successful

      Check: Node connectivity for interface "eth0"
      Node connectivity passed for interface "eth0"
      TCP connectivity check passed for subnet "10.209.92.0"


      Check: Node connectivity for interface "eth1"
      Node connectivity passed for interface "eth1"
      TCP connectivity check passed for subnet "192.168.10.0"

      Checking subnet mask consistency...
      Subnet mask consistency check passed for subnet "10.209.92.0".
      Subnet mask consistency check passed for subnet "192.168.10.0".
      Subnet mask consistency check passed.

      Node connectivity check passed

      Checking multicast communication...

      Checking subnet "10.209.92.0" for multicast communication with multicast group "230.0.1.0"...
      Check of subnet "10.209.92.0" for multicast communication with multicast group "230.0.1.0" passed.

      Checking subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0"...
      Check of subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0" passed.

      Check of multicast communication passed.
      Total memory check passed
      Available memory check passed
      Swap space check passed
      Free disk space check passed for "punlabsx86-21:/oracle/11g/gridhome,punlabsx86-21:/tmp"
      Free disk space check passed for "punlabsx86-15:/oracle/11g/gridhome,punlabsx86-15:/tmp"
      Check for multiple users with UID value 501 passed
      User existence check passed for "grid"
      Run level check passed
      Hard limits check passed for "maximum open file descriptors"
      Soft limits check passed for "maximum open file descriptors"
      Hard limits check passed for "maximum user processes"
      Soft limits check passed for "maximum user processes"
      System architecture check passed
      Kernel version check passed
      Kernel parameter check passed for "semmsl"
      Kernel parameter check passed for "semmns"
      Kernel parameter check passed for "semopm"
      Kernel parameter check passed for "semmni"
      Kernel parameter check passed for "shmmax"
      Kernel parameter check passed for "shmmni"
      Kernel parameter check passed for "shmall"
      Kernel parameter check passed for "file-max"
      Kernel parameter check passed for "ip_local_port_range"
      Kernel parameter check passed for "rmem_default"
      Kernel parameter check passed for "rmem_max"
      Kernel parameter check passed for "wmem_default"
      Kernel parameter check passed for "wmem_max"
      Kernel parameter check passed for "aio-max-nr"
      Package existence check passed for "make"
      Package existence check passed for "binutils"
      Package existence check passed for "gcc(x86_64)"
      Package existence check passed for "libaio(x86_64)"
      Package existence check passed for "glibc(x86_64)"
      Package existence check passed for "compat-libstdc++-33(x86_64)"
      Package existence check passed for "elfutils-libelf(x86_64)"
      Package existence check passed for "elfutils-libelf-devel"
      Package existence check passed for "glibc-common"
      Package existence check passed for "glibc-devel(x86_64)"
      Package existence check passed for "glibc-headers"
      Package existence check passed for "gcc-c++(x86_64)"
      Package existence check passed for "libaio-devel(x86_64)"
      Package existence check passed for "libgcc(x86_64)"
      Package existence check passed for "libstdc++(x86_64)"
      Package existence check passed for "libstdc++-devel(x86_64)"
      Package existence check passed for "sysstat"
      Package existence check passed for "ksh"
      Check for multiple users with UID value 0 passed
      Current group ID check passed

      Starting check for consistency of primary group of root user

      Check for consistency of root user's primary group passed

      Checking OCR integrity...
      Check for compatible storage device for OCR location "/votevol/ocr"...


      OCR integrity check passed

      Checking Oracle Cluster Voting Disk configuration...

      Oracle Cluster Voting Disk configuration check passed
      Time zone consistency check passed

      Starting Clock synchronization checks using Network Time Protocol(NTP)...

      NTP Configuration file check started...
      NTP Configuration file check passed

      Checking daemon liveness...
      Liveness check passed for "ntpd"
      Check for NTP daemon or service alive passed on all nodes

      NTP daemon slewing option check passed

      NTP daemon's boot time configuration check for slewing option passed

      NTP common Time Server Check started...
      Check of common NTP Time Server passed

      Clock time offset check from NTP Time Server started...
      Clock time offset check passed

      Clock synchronization check using Network Time Protocol(NTP) passed


      User "grid" is not part of "root" group. Check passed
      Checking consistency of file "/etc/resolv.conf" across nodes

      File "/etc/resolv.conf" does not have both domain and search entries defined
      domain entry in file "/etc/resolv.conf" is consistent across nodes
      search entry in file "/etc/resolv.conf" is consistent across nodes
      All nodes have one search entry defined in file "/etc/resolv.conf"
      The DNS response time for an unreachable node is within acceptable limit on all nodes

      File "/etc/resolv.conf" is consistent across nodes

      Checking VIP configuration.
      Checking VIP Subnet configuration.
      Check for VIP Subnet configuration passed.
      Checking VIP reachability
      Check for VIP reachability passed.

      Pre-check for node addition was successful.
      Starting Oracle Universal Installer...

      Checking swap space: must be greater than 500 MB. Actual 16002 MB Passed
      Oracle Universal Installer, Version 11.2.0.3.0 Production
      Copyright (C) 1999, 2011, Oracle. All rights reserved.


      Performing tests to see whether nodes punlabsx86-16,punlabsx86-17,punlabsx86-21 are available
      ............................................................... 100% Done.

      .
      -----------------------------------------------------------------------------
      Cluster Node Addition Summary
      Global Settings
      Source: /oracle/11g/gridhome
      New Nodes
      Space Requirements
      New Nodes
      punlabsx86-21
      /: Required 5.03GB : Available 102.72GB
      Installed Products
      Product Names
      Oracle Grid Infrastructure 11.2.0.3.0
      Sun JDK 1.5.0.30.03
      Installer SDK Component 11.2.0.3.0
      Oracle One-Off Patch Installer 11.2.0.1.7
      Oracle Universal Installer 11.2.0.3.0
      Oracle USM Deconfiguration 11.2.0.3.0
      Oracle Configuration Manager Deconfiguration 10.3.1.0.0
      Enterprise Manager Common Core Files 10.2.0.4.4
      Oracle DBCA Deconfiguration 11.2.0.3.0
      Oracle RAC Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Server) 11.2.0.3.0
      Installation Plugin Files 11.2.0.3.0
      Universal Storage Manager Files 11.2.0.3.0
      Oracle Text Required Support Files 11.2.0.3.0
      Automatic Storage Management Assistant 11.2.0.3.0
      Oracle Database 11g Multimedia Files 11.2.0.3.0
      Oracle Multimedia Java Advanced Imaging 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Oracle Multimedia Locator RDBMS Files 11.2.0.3.0
      Oracle Core Required Support Files 11.2.0.3.0
      Bali Share 1.1.18.0.0
      Oracle Database Deconfiguration 11.2.0.3.0
      Oracle Quality of Service Management (Client) 11.2.0.3.0
      Expat libraries 2.0.1.0.1
      Oracle Containers for Java 11.2.0.3.0
      Perl Modules 5.10.0.0.1
      Secure Socket Layer 11.2.0.3.0
      Oracle JDBC/OCI Instant Client 11.2.0.3.0
      Oracle Multimedia Client Option 11.2.0.3.0
      LDAP Required Support Files 11.2.0.3.0
      Character Set Migration Utility 11.2.0.3.0
      Perl Interpreter 5.10.0.0.2
      PL/SQL Embedded Gateway 11.2.0.3.0
      OLAP SQL Scripts 11.2.0.3.0
      Database SQL Scripts 11.2.0.3.0
      Oracle Extended Windowing Toolkit 3.4.47.0.0
      SSL Required Support Files for InstantClient 11.2.0.3.0
      SQL*Plus Files for Instant Client 11.2.0.3.0
      Oracle Net Required Support Files 11.2.0.3.0
      Oracle Database User Interface 2.2.13.0.0
      RDBMS Required Support Files for Instant Client 11.2.0.3.0
      RDBMS Required Support Files Runtime 11.2.0.3.0
      XML Parser for Java 11.2.0.3.0
      Oracle Security Developer Tools 11.2.0.3.0
      Oracle Wallet Manager 11.2.0.3.0
      Enterprise Manager plugin Common Files 11.2.0.3.0
      Platform Required Support Files 11.2.0.3.0
      Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
      RDBMS Required Support Files 11.2.0.3.0
      Oracle Ice Browser 5.2.3.6.0
      Oracle Help For Java 4.2.9.0.0
      Enterprise Manager Common Files 10.2.0.4.3
      Deinstallation Tool 11.2.0.3.0
      Oracle Java Client 11.2.0.3.0
      Cluster Verification Utility Files 11.2.0.3.0
      Oracle Notification Service (eONS) 11.2.0.3.0
      Oracle LDAP administration 11.2.0.3.0
      Cluster Verification Utility Common Files 11.2.0.3.0
      Oracle Clusterware RDBMS Files 11.2.0.3.0
      Oracle Locale Builder 11.2.0.3.0
      Oracle Globalization Support 11.2.0.3.0
      Buildtools Common Files 11.2.0.3.0
      Oracle RAC Required Support Files-HAS 11.2.0.3.0
      SQL*Plus Required Support Files 11.2.0.3.0
      XDK Required Support Files 11.2.0.3.0
      Agent Required Support Files 10.2.0.4.3
      Parser Generator Required Support Files 11.2.0.3.0
      Precompiler Required Support Files 11.2.0.3.0
      Installation Common Files 11.2.0.3.0
      Required Support Files 11.2.0.3.0
      Oracle JDBC/THIN Interfaces 11.2.0.3.0
      Oracle Multimedia Locator 11.2.0.3.0
      Oracle Multimedia 11.2.0.3.0
      HAS Common Files 11.2.0.3.0
      Assistant Common Files 11.2.0.3.0
      PL/SQL 11.2.0.3.0
      HAS Files for DB 11.2.0.3.0
      Oracle Recovery Manager 11.2.0.3.0
      Oracle Database Utilities 11.2.0.3.0
      Oracle Notification Service 11.2.0.3.0
      SQL*Plus 11.2.0.3.0
      Oracle Netca Client 11.2.0.3.0
      Oracle Net 11.2.0.3.0
      Oracle JVM 11.2.0.3.0
      Oracle Internet Directory Client 11.2.0.3.0
      Oracle Net Listener 11.2.0.3.0
      Cluster Ready Services Files 11.2.0.3.0
      Oracle Database 11g 11.2.0.3.0
      -----------------------------------------------------------------------------


      Instantiating scripts for add node (Tuesday, October 25, 2011 4:41:55 PM IST)
      . 1% Done.
      Instantiation of add node scripts complete

      Copying to remote nodes (Tuesday, October 25, 2011 4:41:58 PM IST)
      ............................................................................................... 96% Done.
      Home copied to new nodes

      Saving inventory on nodes (Tuesday, October 25, 2011 4:44:31 PM IST)
      . 100% Done.
      Save inventory complete
      WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
      To register the new inventory please run the script at '/oracle/11g/oraInventory/orainstRoot.sh' with root privileges on nodes 'punlabsx86-21'.
      If you do not register the inventory, you may not be able to update or patch the products you installed.
      The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
      /oracle/11g/oraInventory/orainstRoot.sh #On nodes punlabsx86-21
      /oracle/11g/gridhome/root.sh #On nodes punlabsx86-21
      To execute the configuration scripts:
      1. Open a terminal window
      2. Log in as "root"
      3. Run the scripts in each cluster node

      The Cluster Node Addition of /oracle/11g/gridhome was successful.
      Please check '/tmp/silentInstall.log' for more details.

      </snip>



      <snip>
      [root@punlabsx86-21 /]# /oracle/11g/gridhome/root.sh
      Performing root user operation for Oracle 11g

      The following environment variables are set as:
      ORACLE_OWNER= grid
      ORACLE_HOME= /oracle/11g/gridhome

      Enter the full pathname of the local bin directory: [usr/local/bin]:
      Copying dbhome to /usr/local/bin ...
      Copying oraenv to /usr/local/bin ...
      Copying coraenv to /usr/local/bin ...


      Creating /etc/oratab file...
      Entries will be added to the /etc/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root script.
      Now product-specific root actions will be performed.
      Using configuration parameter file: /oracle/11g/gridhome/crs/install/crsconfig_params
      Creating trace directory
      User ignored Prerequisites during installation
      OLR initialization - successful
      root wallet
      root wallet cert
      root cert export
      peer wallet
      profile reader wallet
      pa wallet
      peer wallet keys
      pa wallet keys
      peer cert request
      pa cert request
      peer cert
      pa cert
      peer root cert TP
      profile reader root cert TP
      pa root cert TP
      peer pa cert TP
      pa peer cert TP
      profile reader pa cert TP
      profile reader peer cert TP
      peer user cert
      pa user cert
      Adding Clusterware entries to inittab
      CRS-2672: Attempting to start 'ora.mdnsd' on 'punlabsx86-21'
      CRS-2676: Start of 'ora.mdnsd' on 'punlabsx86-21' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'punlabsx86-21'
      CRS-2676: Start of 'ora.gpnpd' on 'punlabsx86-21' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'punlabsx86-21'
      CRS-2672: Attempting to start 'ora.gipcd' on 'punlabsx86-21'
      CRS-2676: Start of 'ora.cssdmonitor' on 'punlabsx86-21' succeeded
      CRS-2676: Start of 'ora.gipcd' on 'punlabsx86-21' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'punlabsx86-21'
      CRS-2672: Attempting to start 'ora.diskmon' on 'punlabsx86-21'
      CRS-2676: Start of 'ora.diskmon' on 'punlabsx86-21' succeeded
      CRS-2676: Start of 'ora.cssd' on 'punlabsx86-21' succeeded
      clscfg: -install mode specified
      clscfg: EXISTING configuration version 5 detected.
      clscfg: version 5 is 11g Release 2.
      Successfully accumulated necessary OCR keys.
      clscfg: Arguments check out successfully.

      NO KEYS WERE WRITTEN. Supply -force parameter to override.
      -force is destructive and will destroy any previous cluster
      configuration.
      Failed to initialize Oracle Cluster Registry for cluster, rc 105
      Oracle Grid Infrastructure Repository configuration failed at /oracle/11g/gridhome/crs/install/crsconfig_lib.pm line 6769.
      /oracle/11g/gridhome/perl/bin/perl -I/oracle/11g/gridhome/perl/lib -I/oracle/11g/gridhome/crs/install /oracle/11g/gridhome/crs/install/rootcrs.pl execution failed
      [root@punlabsx86-21 /]#
      </snip>
        • 1. Re: root.sh fails on newly added Node --11gR203GA
          696867
          Check your ssh user equivalence both ways.

          punlabsx86-21> ssh punlabsx86-15 date
          and
          punlabsx86-15> ssh punlabsx86-21 date

          Good Luck
          • 2. Re: root.sh fails on newly added Node --11gR203GA
            756561
            Hi ,

            Difference observed is just 4 minutes, will this matters?

            <snip>

            [grid@punlabsx86-15 bin]$ ssh punlabsx86-21
            Last login: Tue Oct 25 15:54:38 2011 from punlabsx86-15.vxindia.veritas.com
            [grid@punlabsx86-21 ~]$ date
            Fri Oct 28 11:39:03 IST 2011
            [grid@punlabsx86-21 ~]$ ssh punlabsx86-15
            Last login: Tue Oct 25 15:50:45 2011 from punlabsx86-21.vxindia.veritas.com
            [grid@punlabsx86-15 ~]$ date
            Fri Oct 28 11:35:17 IST 2011

            </snip>
            • 3. Re: root.sh fails on newly added Node --11gR203GA
              696867
              Are you using NTP on the existing nodes? If yes, configure NTP on the failed node. If no, then disable NTP on the new node.

              After the root.sh failed on the new node, did you check what is in your GI_HOME/cfgtoollogs/crsconfig/root*.log? Please post any errors or messages in that log and any other agent logs mentioned in the messages/warnings/errors.

              Good Luck!
              • 4. Re: root.sh fails on newly added Node --11gR203GA
                468439
                Could you possibly tell us how you resolved this problem as I'm having a similar issue.
                • 5. Re: root.sh fails on newly added Node --11gR203GA
                  onedbguru
                  user9075978 wrote:
                  Hi ,

                  Difference observed is just 4 minutes, will this matters?
                  I bet you can answer this question. Cluster synchronization depends on time. What do you think is the answer to your question.

                  Use NTP - make sure you use -x (slewing) option.
                  • 6. Re: root.sh fails on newly added Node --11gR203GA
                    468439
                    onedbguru wrote:
                    user9075978 wrote:
                    Hi ,

                    Difference observed is just 4 minutes, will this matters?
                    I bet you can answer this question. Cluster synchronization depends on time. What do you think is the answer to your question.

                    Use NTP - make sure you use -x (slewing) option.
                    The question was not marked answered and I did not spot the ntp issue in the post.

                    I am getting "Failed to initialize Oracle Cluster Registry for cluster, rc 105 Oracle Grid Infrastructure Repository configuration failed at /oracle/11g/gridhome/crs/install/crsconfig_lib.pm line 6769" Error. I've defintely not got an ntp issue. My times are synched perfectly. I hoped there was a different answer.
                    • 7. Re: root.sh fails on newly added Node --11gR203GA
                      944107
                      Thanks useful information in this post i'm looking for it

                      _______________________
                      American Grown by Michelle Obama
                      SEO Fight Back Review
                      • 8. root.sh fails on newly added Node --11gR203GA.......
                        1001806
                        Dear All,


                        I had successfully added third node to the existing 2 node cluster. After adding new node I need to run the root.sh scripts, but it was faling with the below error.
                        Please help me with the below issue:

                        Instantiating scripts for add node (Monday, April 8, 2013 3:23:14 PM EDT)
                        . 1% Done.
                        Instantiation of add node scripts complete

                        Copying to remote nodes (Monday, April 8, 2013 3:23:16 PM EDT)
                        ............................................................................................... 96% Done.
                        Home copied to new nodes

                        Saving inventory on nodes (Monday, April 8, 2013 3:31:40 PM EDT)
                        . 100% Done.
                        Save inventory complete
                        WARNING:
                        The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
                        /u01/app/11.2.0/grid/root.sh #On nodes svphxwgdbprd06
                        To execute the configuration scripts:
                        1. Open a terminal window
                        2. Log in as "root"
                        3. Run the scripts in each cluster node

                        The Cluster Node Addition of /u01/app/11.2.0/grid was successful.

                        Root.SH Script Log:_


                        [root@svphxwgdbprd06 ~]# /u01/app/11.2.0/grid/root.sh
                        Performing root user operation for Oracle 11g

                        The following environment variables are set as:
                        ORACLE_OWNER= oracle
                        ORACLE_HOME= /u01/app/11.2.0/grid

                        Enter the full pathname of the local bin directory: [usr/local/bin]:
                        The contents of "dbhome" have not changed. No need to overwrite.
                        The contents of "oraenv" have not changed. No need to overwrite.
                        The contents of "coraenv" have not changed. No need to overwrite.


                        Creating /etc/oratab file...
                        Entries will be added to the /etc/oratab file as needed by
                        Database Configuration Assistant when a database is created
                        Finished running generic part of root script.
                        Now product-specific root actions will be performed.
                        Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
                        Creating trace directory
                        User ignored Prerequisites during installation
                        OLR initialization - successful
                        Adding Clusterware entries to inittab
                        CRS-2672: Attempting to start 'ora.mdnsd' on 'svphxwgdbprd06'
                        CRS-2676: Start of 'ora.mdnsd' on 'svphxwgdbprd06' succeeded
                        CRS-2672: Attempting to start 'ora.gpnpd' on 'svphxwgdbprd06'
                        CRS-2676: Start of 'ora.gpnpd' on 'svphxwgdbprd06' succeeded
                        CRS-2672: Attempting to start 'ora.cssdmonitor' on 'svphxwgdbprd06'
                        CRS-2672: Attempting to start 'ora.gipcd' on 'svphxwgdbprd06'
                        CRS-2676: Start of 'ora.cssdmonitor' on 'svphxwgdbprd06' succeeded
                        CRS-2676: Start of 'ora.gipcd' on 'svphxwgdbprd06' succeeded
                        CRS-2672: Attempting to start 'ora.cssd' on 'svphxwgdbprd06'
                        CRS-2672: Attempting to start 'ora.diskmon' on 'svphxwgdbprd06'
                        CRS-2676: Start of 'ora.diskmon' on 'svphxwgdbprd06' succeeded
                        CRS-2676: Start of 'ora.cssd' on 'svphxwgdbprd06' succeeded
                        ASM created and started successfully.

                        Disk Group DATA created successfully.

                        clscfg: -install mode specified
                        clscfg: EXISTING configuration version 5 detected.
                        clscfg: version 5 is 11g Release 2.
                        Successfully accumulated necessary OCR keys.
                        clscfg: Arguments check out successfully.

                        NO KEYS WERE WRITTEN. Supply -force parameter to override.
                        -force is destructive and will destroy any previous cluster
                        configuration.
                        Failed to initialize Oracle Cluster Registry for cluster, rc 105
                        Oracle Grid Infrastructure Repository configuration failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6818.
                        /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed