6 Replies Latest reply: Aug 5, 2011 1:13 AM by 869506 RSS

    How to add new node in 11gR2 RAC

    cgswong
      I thought I'd share my recent use case findings with all concerning the steps to add a new RAC node to a 11gR2 cluster. The documentation has a step missing, though it is likely this is only for a role-separated setup (step 10 - 11 is the critical part):

      Phase I - Extending Oracle Clusterware to a new cluster node
      1. Make physical connections, install the OS. Follow article “11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC [ID 1059847.1]” to ensure successful completion of root.sh!!

      2. Create Oracle accounts, and setup SSH among the new node and the existing cluster nodes. Follow article “How To Configure SSH for a RAC Installation [ID 300548.1]" for correct procedure for SSH setup! EACH NODE MUST BE VISITED TO ENSURE THEY ARE ADDED TO KNOWN_HOSTS FILE!!

      3. Verify the requirements for cluster node addition using the Cluster Verification Utility (CVU). From an existing cluster node:

      $> $GI_HOME/bin/cluvfy stage -post hwos -n <existing and new nodes> -verbose

      4. Compare an existing node with the new node(s) to be added:

      $> $GI_HOME/bin/cluvfy comp peer -refnode <existing node> -n <new node> -orainv oinstall -osdba dba -verbose

      5. Verify the integrity of the cluster and new node by running from an existing cluster node:

      $GI_HOME/bin/cluvfy stage -pre nodeadd -n <new node> -fixup -verbose

      6. Add the new node by running the following from an existing cluster node:

      a. Not using GNS

      $GI_HOME/oui/bin/addNode.sh -silent “CLUSTER_NEW_NODES={<new node>}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={<new node VIP>}”

      b. Using GNS

      $GI_HOME/oui/bin/addNode.sh -silent “CLUSTER_NEW_NODES={<new node>}”

      Run the root scripts when prompted.

      7. Verify that the new node has been added to the cluster:

      $GI_HOME/bin/cluvfy stage -post nodeadd -n <new node> -verbose

      Phase II - Extending Oracle Database RAC to new cluster node
      8a. Using Clone Process

      i. Use ‘tar’ to archive an existing DB home, and extract to the same location on the new node

      ii. On the new node run:

      perl $ORACLE_HOME/clone/bin/clone.pl '-O"CLUSTER_NODES={<existing node>,<new node>}"' '-O"LOCAL_NODE=<new node>"' ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDb11g_home1 '-O-noConfig'

      iii. On the existing node where the DB home was cloned, run:

      $ORACLE_HOME/oui/bin/runInstaller –updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={<existing node>,<new node>}"

      OR

      8b.Using addNode.sh process (RECOMMENDED)

      i. From an existing node in the cluster as the ‘oracle’ user:

      $> $ORACLE_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={<new node>}"

      9. On the new node run the root.sh as prompted.

      10. Set ORACLE_HOME and ensure you are using the ‘oracle’ account user.

      Note: Ensure permissions for Oracle executable are 6751, if not, then as root user:
      cd $ORACLE_HOME/bin
      chgrp asmadmin oracle
      chmod 6751 oracle
      ls -l oracle

      11. On any existing node, run DBCA ($ORACLE_HOME/bin/dbca) to add the new instance:

      $ORACLE_HOME/bin/dbca -silent -addInstance -nodeList <new node> -gdbName <db name> -instanceName <new instance> -sysDBAUserName sys -sysDBAPassword <sys password>

      NOTES: A. Ensure command is run from existing node with same or less memory, otherwise command will fail due to insufficient memory to support database memory structures. Also ensure that the log file is checked for actual success since it can differ from what is displayed at the screen.
      B. Anytime when a patch is applied to the database ORACLE_HOME, please ensure above ownership and permission is corrected after the patch.

      12. Verify the administrator privileges on the new node by running on existing node:

      $ORACLE_HOME/bin/cluvfy comp admprv -o db_config -d $ORACLE_HOME -n <all nodes list> -verbose

      13. For an Admin-Managed Cluster, add the new instance to services, or create additional services. For a Policy-Managed Cluster, verify the instance has been added to an existing server pool.

      14. Setup OCM in Cloned Homes

      a. Delete all subdirectories to remove previously configured host:

      $> rm -rf $ORACLE_HOME/ccr/hosts/*

      b. Move (no copy) both ‘grid’ and ‘oracle’ homes:

      $> mv $ORACLE_HOME/ccr/inventory/core.jar $ORACLE_HOME/ccr/inventory/pending/core.jar

      c. Configure OCM for the cloned home on the new node:

      $> $ORACLE_HOME/ccr/bin/configCCR -a
        • 1. Re: How to add new node in 11gR2 RAC
          Sebastian Solbach -Dba Community-Oracle
          Hi,

          thx for sharing your findings.
          A small sidenote: Instead of using

          chgrp asmadmin oracle
          chmod 6751 oracle

          in step 10 you could also use

          setasmgidwrap executable as written in

          Note 1084186.1: Database Creation on 11.2 Grid Infracture with Role Separation ( ORA-15025, KFSG-00312, ORA-15081 )

          Sebastian
          • 2. Re: How to add new node in 11gR2 RAC
            cgswong
            Thanks for the note. My reference was: "ORA-15183 Unable to Create Database on Server using 11.2 ASM and Grid Infrastructure [ID 1054033.1]".
            • 3. Re: How to add new node in 11gR2 RAC
              186931
              Hi,

              I am getting following error while adding an instance:


              Adding instance
              DBCA_PROGRESS : 1%
              DBCA_PROGRESS : 2%
              DBCA_PROGRESS : 6%
              DBCA_PROGRESS : 13%
              DBCA_PROGRESS : 20%
              DBCA_PROGRESS : 26%
              DBCA_PROGRESS : 33%
              DBCA_PROGRESS : 40%
              DBCA_PROGRESS : 46%
              DBCA_PROGRESS : 53%
              DBCA_PROGRESS : 66%
              Completing instance management.
              DBCA_PROGRESS : 76%
              PRCR-1013 : Failed to start resource ora.racdb.db
              PRCR-1064 : Failed to start resource ora.racdb.db on node prod1
              ORA-01034: ORACLE not available
              ORA-27101: shared memory realm does not exist
              Linux Error: 2: No such file or directory
              Process ID: 0
              Session ID: 0 Serial number: 0

              ORA-01078: failure in processing system parameters
              CRS-2674: Start of 'ora.racdb.db' on 'prod1' failed
              ORA-01034: ORACLE not available
              ORA-27101: shared memory realm does not exist
              Linux Error: 2: No such file or directory
              Process ID: 0
              Session ID: 0 Serial number: 0


              DBCA_PROGRESS : 100%


              Can someone please help? I was able to extend clusterware and oracle database software using addNode.sh.

              Thanks

              Edited by: Andy DBA on Sep 14, 2010 5:07 PM
              • 4. Re: How to add new node in 11gR2 RAC
                186931
                Never mind.

                I figured it out. It was the ASM DG that was not mounted.

                Thanks
                • 5. Re: How to add new node in 11gR2 RAC
                  756561
                  Hi Thanks for the steps.

                  In my setup I got following error during post-check of addNode with 11gR2 Grid;

                  <<<<<<<<<<<<<<<<<<<<<<

                  grid@punlabsx86-16:/oracle/11g/gridhome/oui/bin> ./addNode.sh -silent "CLUSTER_NEW_NODES=punlabsx86-3" "CLUSTER_NEW_VIRTUAL_HOSTNAMES=10.209.92.217"
                  Starting Oracle Universal Installer...

                  Checking swap space: must be greater than 500 MB. Actual 16284 MB Passed
                  Oracle Universal Installer, Version 11.2.0.1.0 Production
                  Copyright (C) 1999, 2009, Oracle. All rights reserved.

                  SEVERE:Values for the following variables could not be obtained from the command line or response file(s):CLUSTER_NEW_VIRTUAL_HOSTNAMES. Silent install cannot continue.
                  grid@punlabsx86-16:/oracle/11g/gridhome/oui/bin> ./addNode.sh -silent "CLUSTER_NEW_NODES={punlabsx86-3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={10.209.92.217}"
                  Starting Oracle Universal Installer...

                  Checking swap space: must be greater than 500 MB. Actual 16284 MB Passed
                  Oracle Universal Installer, Version 11.2.0.1.0 Production
                  Copyright (C) 1999, 2009, Oracle. All rights reserved.


                  Performing tests to see whether nodes punlabsx86-17,punlabsx86-3 are available
                  ............................................................... 100% Done.

                  .
                  -----------------------------------------------------------------------------
                  Cluster Node Addition Summary
                  Global Settings
                  Source: /oracle/11g/gridhome
                  New Nodes
                  Space Requirements
                  New Nodes
                  punlabsx86-3
                  /: Required 6.69GB : Available 41.04GB
                  Installed Products
                  Product Names
                  Oracle Grid Infrastructure 11.2.0.1.0
                  Sun JDK 1.5.0.17.0
                  Installer SDK Component 11.2.0.1.0
                  Oracle One-Off Patch Installer 11.2.0.0.2
                  Oracle Universal Installer 11.2.0.1.0
                  Oracle Configuration Manager Deconfiguration 10.3.1.0.0
                  Enterprise Manager Common Core Files 10.2.0.4.2
                  Oracle DBCA Deconfiguration 11.2.0.1.0
                  Oracle RAC Deconfiguration 11.2.0.1.0
                  Oracle Quality of Service Management (Server) 11.2.0.1.0
                  Installation Plugin Files 11.2.0.1.0
                  Universal Storage Manager Files 11.2.0.1.0
                  Oracle Text Required Support Files 11.2.0.1.0
                  Automatic Storage Management Assistant 11.2.0.1.0
                  Oracle Database 11g Multimedia Files 11.2.0.1.0
                  Oracle Multimedia Java Advanced Imaging 11.2.0.1.0
                  Oracle Globalization Support 11.2.0.1.0
                  Oracle Multimedia Locator RDBMS Files 11.2.0.1.0
                  Oracle Core Required Support Files 11.2.0.1.0
                  Bali Share 1.1.18.0.0
                  Oracle Database Deconfiguration 11.2.0.1.0
                  Oracle Quality of Service Management (Client) 11.2.0.1.0
                  Expat libraries 2.0.1.0.1
                  Oracle Containers for Java 11.2.0.1.0
                  Perl Modules 5.10.0.0.1
                  Secure Socket Layer 11.2.0.1.0
                  Oracle JDBC/OCI Instant Client 11.2.0.1.0
                  Oracle Multimedia Client Option 11.2.0.1.0
                  LDAP Required Support Files 11.2.0.1.0
                  Character Set Migration Utility 11.2.0.1.0
                  Perl Interpreter 5.10.0.0.1
                  PL/SQL Embedded Gateway 11.2.0.1.0
                  OLAP SQL Scripts 11.2.0.1.0
                  Database SQL Scripts 11.2.0.1.0
                  Oracle Extended Windowing Toolkit 3.4.47.0.0
                  SSL Required Support Files for InstantClient 11.2.0.1.0
                  SQL*Plus Files for Instant Client 11.2.0.1.0
                  Oracle Net Required Support Files 11.2.0.1.0
                  Oracle Database User Interface 2.2.13.0.0
                  RDBMS Required Support Files for Instant Client 11.2.0.1.0
                  Enterprise Manager Minimal Integration 11.2.0.1.0
                  XML Parser for Java 11.2.0.1.0
                  Oracle Security Developer Tools 11.2.0.1.0
                  Oracle Wallet Manager 11.2.0.1.0
                  Enterprise Manager plugin Common Files 11.2.0.1.0
                  Platform Required Support Files 11.2.0.1.0
                  Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
                  RDBMS Required Support Files 11.2.0.1.0
                  Oracle Ice Browser 5.2.3.6.0
                  Oracle Help For Java 4.2.9.0.0
                  Enterprise Manager Common Files 10.2.0.4.2
                  Deinstallation Tool 11.2.0.1.0
                  Oracle Java Client 11.2.0.1.0
                  Cluster Verification Utility Files 11.2.0.1.0
                  Oracle Notification Service (eONS) 11.2.0.1.0
                  Oracle LDAP administration 11.2.0.1.0
                  Cluster Verification Utility Common Files 11.2.0.1.0
                  Oracle Clusterware RDBMS Files 11.2.0.1.0
                  Oracle Locale Builder 11.2.0.1.0
                  Oracle Globalization Support 11.2.0.1.0
                  Buildtools Common Files 11.2.0.1.0
                  Oracle RAC Required Support Files-HAS 11.2.0.1.0
                  SQL*Plus Required Support Files 11.2.0.1.0
                  XDK Required Support Files 11.2.0.1.0
                  Agent Required Support Files 10.2.0.4.2
                  Parser Generator Required Support Files 11.2.0.1.0
                  Precompiler Required Support Files 11.2.0.1.0
                  Installation Common Files 11.2.0.1.0
                  Required Support Files 11.2.0.1.0
                  Oracle JDBC/THIN Interfaces 11.2.0.1.0
                  Oracle Multimedia Locator 11.2.0.1.0
                  Oracle Multimedia 11.2.0.1.0
                  HAS Common Files 11.2.0.1.0
                  Assistant Common Files 11.2.0.1.0
                  PL/SQL 11.2.0.1.0
                  HAS Files for DB 11.2.0.1.0
                  Oracle Recovery Manager 11.2.0.1.0
                  Oracle Database Utilities 11.2.0.1.0
                  Oracle Notification Service 11.2.0.0.0
                  SQL*Plus 11.2.0.1.0
                  Oracle Netca Client 11.2.0.1.0
                  Oracle Net 11.2.0.1.0
                  Oracle JVM 11.2.0.1.0
                  Oracle Internet Directory Client 11.2.0.1.0
                  Oracle Net Listener 11.2.0.1.0
                  Cluster Ready Services Files 11.2.0.1.0
                  Oracle Database 11g 11.2.0.1.0
                  -----------------------------------------------------------------------------


                  Instantiating scripts for add node (Tuesday, September 21, 2010 6:25:16 AM IST)
                  . 1% Done.
                  Instantiation of add node scripts complete

                  Copying to remote nodes (Tuesday, September 21, 2010 6:25:20 AM IST)
                  ............................................................................................... 96% Done.
                  Home copied to new nodes

                  Saving inventory on nodes (Tuesday, September 21, 2010 6:29:14 AM IST)
                  SEVERE:Remote 'AttachHome' failed on nodes: 'punlabsx86-3'. Refer to '/oracle/11g/oraInventory/logs/addNodeActions2010-09-21_06-25-07AM.log' for details.
                  You can manually re-run the following command on the failed nodes after the installation:
                  /oracle/11g/gridhome/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/oracle/11g/gridhome ORACLE_HOME_NAME=Ora11g_gridinfrahome1 CLUSTER_NODES=punlabsx86-16,punlabsx86-17,punlabsx86-3 CRS=true "INVENTORY_LOCATION=/oracle/11g/oraInventory" -invPtrLoc "/oracle/11g/gridhome/oraInst.loc" LOCAL_NODE=<node on which command is to be run>.
                  Please refer 'AttachHome' logs under central inventory of remote nodes where failure occurred for more details.
                  SEVERE:Remote 'UpdateNodeList' failed on nodes: 'punlabsx86-3'. Refer to '/oracle/11g/oraInventory/logs/addNodeActions2010-09-21_06-25-07AM.log' for details.
                  You can manually re-run the following command on the failed nodes after the installation:
                  /oracle/11g/gridhome/oui/bin/runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/oracle/11g/gridhome CLUSTER_NODES=punlabsx86-16,punlabsx86-17,punlabsx86-3 CRS=true "INVENTORY_LOCATION=/oracle/11g/oraInventory" -invPtrLoc "/oracle/11g/gridhome/oraInst.loc" LOCAL_NODE=<node on which command is to be run>.
                  Please refer 'UpdateNodeList' logs under central inventory of remote nodes where failure occurred for more details.
                  . 100% Done.
                  Save inventory complete
                  WARNING:OUI-10234:Failed to copy the root script, /oracle/11g/oraInventory/orainstRoot.sh to the cluster nodes punlabsx86-3.[Error in copying file '/oracle/11g/oraInventory/orainstRoot.sh' present inside directory '/' on nodes 'punlabsx86-3'. [PRKC-1080 : Failed to transfer file "/oracle/11g/oraInventory/orainstRoot.sh" to any of the given nodes "punlabsx86-3 ".
                  Error on node punlabsx86-3:null]]
                  Please copy them manually to these nodes and execute the script.
                  WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
                  To register the new inventory please run the script at '/oracle/11g/oraInventory/orainstRoot.sh' with root privileges on nodes 'punlabsx86-3'.
                  If you do not register the inventory, you may not be able to update or patch the products you installed.
                  The following configuration scripts need to be executed as the "root" user in each cluster node.
                  /oracle/11g/oraInventory/orainstRoot.sh #On nodes punlabsx86-3
                  /oracle/11g/gridhome/root.sh #On nodes punlabsx86-3
                  To execute the configuration scripts:
                  1. Open a terminal window
                  2. Log in as "root"
                  3. Run the scripts in each cluster node

                  The Cluster Node Addition of /oracle/11g/gridhome was successful.
                  Please check '/tmp/silentInstall.log' for more details.
                  grid@punlabsx86-16:/oracle/11g/gridhome/oui/bin>

                  >>>>>>>>>>>>>

                  So I executed above scripts on failed node as below;

                  <<<<<<<<<<<<
                  grid@punlabsx86-3:/oracle/11g/oraInventory> /oracle/11g/gridhome/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/oracle/11g/gridhome ORACLE_HOME_NAME=Ora11g_gridinfrahome1 CLUSTER_NODES=punlabsx86-16,punlabsx86-17,punlabsx86-3 CRS=true "INVENTORY_LOCATION=/oracle/11g/oraInventory" -invPtrLoc "/oracle/11g/gridhome/oraInst.loc" LOCAL_NODE=punlabsx86-3
                  Starting Oracle Universal Installer...

                  Checking swap space: must be greater than 500 MB. Actual 49151 MB Passed
                  Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-09-21_05-45-09PM. Please wait ...grid@punlabsx86-3:/oracle/11g/oraInventory> The inventory pointer is located at /oracle/11g/gridhome/oraInst.loc
                  The inventory is located at /oracle/11g/oraInventory
                  'AttachHome' was successful.

                  grid@punlabsx86-3:/oracle/11g/oraInventory> /oracle/11g/gridhome/oui/bin/runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/oracle/11g/gridhome CLUSTER_NODES=punlabsx86-16,punlabsx86-17,punlabsx86-3 CRS=true "INVENTORY_LOCATION=/oracle/11g/oraInventory" -invPtrLoc "/oracle/11g/gridhome/oraInst.loc" LOCAL_NODE=punlabsx86-3
                  Starting Oracle Universal Installer...

                  Checking swap space: must be greater than 500 MB. Actual 49151 MB Passed
                  The inventory pointer is located at /oracle/11g/gridhome/oraInst.loc
                  The inventory is located at /oracle/11g/oraInventory
                  'UpdateNodeList' was successful.


                  punlabsx86-3:/oracle/11g/oraInventory # /oracle/11g/oraInventory/orainstRoot.sh
                  Creating the Oracle inventory pointer file (/etc/oraInst.loc)
                  Changing permissions of /oracle/11g/oraInventory.
                  Adding read,write permissions for group.
                  Removing read,write,execute permissions for world.
                  Changing groupname of /oracle/11g/oraInventory to oinstall.
                  The execution of the script is complete.
                  >>>>>>>>>>>>

                  after this I performed post check which fails as below;


                  grid@punlabsx86-16:/oracle/11g/gridhome/oui/bin> /oracle/11g/gridhome/bin/cluvfy stage -post nodeadd -n punlabsx86-3 -verbose

                  Performing post-checks for node addition

                  Checking node reachability...

                  Check: Node reachability from node "punlabsx86-16"
                  Destination Node Reachable?
                  ------------------------------------ ------------------------
                  punlabsx86-3 yes
                  Result: Node reachability check passed from node "punlabsx86-16"


                  Checking user equivalence...

                  Check: User equivalence for user "grid"
                  Node Name Comment
                  ------------------------------------ ------------------------
                  punlabsx86-3 passed
                  Result: User equivalence check passed for user "grid"

                  ERROR:
                  Shared resources check for node addition failed
                  The following node is not in cluster: punlabsx86-3
                  Post-check for node addition was unsuccessful on all the nodes.
                  grid@punlabsx86-16:/oracle/11g/gridhome/oui/bin>
                  >>>>>>>>>>>


                  Any help on this will be highly appreciable;

                  Thanks,
                  Dhananjay
                  • 6. Re: How to add new node in 11gR2 RAC
                    869506
                    Dear Dhanjay,
                    Could you please share how did you get rid of that problem. I too am stuck there