9 Replies Latest reply: Mar 15, 2012 12:30 PM by user10674190 RSS

    I got CRS-4402 error when I run root.sh

    user10674190
      Hi all,

      I am installing Grid Infrastructure in my new environment, and I have a problem when I run root.sh.
      In the first node, root.sh was OK, but I got a CRS-4402 error in the anothers two nodes:

      "...CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node
      brloulsrac01, number 1, and is terminating
      An active cluster was found during exclusive startup, restarting to join the cluster..."


      Please, can anybody help me?

      Thanks in advance.

      Leonardo.
        • 1. Re: I got CRS-4402 error when I run root.sh
          912595
          Leonardo,

          I'm hoping you are trying installing grid on virtual machine where host operating system is windows 32-bit with having 4 GB ram?
          • 2. Re: I got CRS-4402 error when I run root.sh
            user10674190
            Thanks for your replay.

            No, my environment is:
            3 nodes
            SO: Linux x86-64
            GI: 11.2.0.3
            and physical machines
            • 3. Re: I got CRS-4402 error when I run root.sh
              912595
              could you please check following docs?
              CRS root.sh Script Failing on Second Node When MTU Larger than 1500 (Doc ID 1085885.1)
              Root.sh Unable To Start CRS On Second Node (Doc ID 369699.1)
              11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network (Doc ID 981357.1)

              and tell us which worked?
              • 4. Re: I got CRS-4402 error when I run root.sh
                user10674190
                Thanks for your awnser, please see below:

                =============================================================================

                CRS root.sh Script Failing on Second Node When MTU Larger than 1500 (Doc ID 1085885.1)

                I am using Linux x86-64, and it is support MTU 1500 and 9000

                [root@brloulsrac01 oraInventory]# ifconfig eth2
                eth2 Link encap:Ethernet HWaddr 44:1E:A1:4A:58:30
                inet addr:7.9.10.215 Bcast:7.9.11.255 Mask:255.255.252.0
                UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
                RX packets:1945186 errors:0 dropped:0 overruns:0 frame:0
                TX packets:4648819 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:1000
                RX bytes:132052948 (125.9 MiB) TX bytes:6628992196 (6.1 GiB)


                eth2 - public network

                [root@brloulsrac01 oraInventory]# ifconfig eth3
                eth3 Link encap:Ethernet HWaddr 44:1E:A1:4A:58:34
                inet addr:172.30.2.2 Bcast:172.30.2.255 Mask:255.255.255.0
                UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
                RX packets:1961 errors:0 dropped:0 overruns:0 frame:0
                TX packets:1953 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:1000
                RX bytes:221565 (216.3 KiB) TX bytes:761773 (743.9 KiB)


                eth3 - private network


                =============================================================================

                Root.sh Unable To Start CRS On Second Node (Doc ID 369699.1)

                I don't see any problem in this setup...

                [root@brloulsrac01 oraInventory]# chkconfig --list|grep iptables
                iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off

                [root@brloulsrac01 oraInventory]# service iptables status
                Firewall is stopped.


                [root@brloulsrac01 oraInventory]# chkconfig iptables off

                [root@brloulsrac01 oraInventory]# chkconfig --list|grep iptables
                iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

                [root@brloulsrac01 oraInventory]# traceroute 172.30.2.2
                traceroute to 172.30.2.2 (172.30.2.2), 30 hops max, 40 byte packets
                1 brloulsrac01-priv.amer.exel-intra.net (172.30.2.2) 0.032 ms 0.012 ms 0.007 ms
                [root@brloulsrac01 oraInventory]# traceroute 172.30.2.3
                traceroute to 172.30.2.3 (172.30.2.3), 30 hops max, 40 byte packets
                1 brloulsrac02-priv.amer.exel-intra.net (172.30.2.3) 1.563 ms 1.568 ms 1.562 ms
                [root@brloulsrac01 oraInventory]# traceroute 172.30.2.4
                traceroute to 172.30.2.4 (172.30.2.4), 30 hops max, 40 byte packets
                1 brloulsrac03-priv.amer.exel-intra.net (172.30.2.4) 0.139 ms 0.112 ms 0.106 ms

                ----------------------------------------------------------------------------------------------------------------------------------

                [root@brloulsrac02 oraInventory]# chkconfig --list|grep iptables
                iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
                [root@brloulsrac02 oraInventory]# service iptables status
                Firewall is stopped.

                [root@brloulsrac02 oraInventory]# traceroute 172.30.2.2
                traceroute to 172.30.2.2 (172.30.2.2), 30 hops max, 40 byte packets
                1 brloulsrac01-priv.amer.exel-intra.net (172.30.2.2) 0.195 ms 0.177 ms 0.169 ms
                [root@brloulsrac02 oraInventory]# traceroute 172.30.2.3
                traceroute to 172.30.2.3 (172.30.2.3), 30 hops max, 40 byte packets
                1 brloulsrac02-priv.amer.exel-intra.net (172.30.2.3) 0.024 ms 0.012 ms 0.007 ms
                [root@brloulsrac02 oraInventory]# traceroute 172.30.2.4
                traceroute to 172.30.2.4 (172.30.2.4), 30 hops max, 40 byte packets
                1 brloulsrac03-priv.amer.exel-intra.net (172.30.2.4) 1.139 ms 1.141 ms 1.133 ms

                ----------------------------------------------------------------------------------------------------------------------------------


                [root@brloulsrac03 oraInventory]# chkconfig --list|grep iptables
                iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off


                [root@brloulsrac03 oraInventory]# service iptables status
                Firewall is stopped.

                [root@brloulsrac03 oraInventory]# chkconfig iptables off


                [root@brloulsrac03 oraInventory]# chkconfig --list|grep iptables
                iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off


                [root@brloulsrac03 oraInventory]# traceroute 172.30.2.2
                traceroute to 172.30.2.2 (172.30.2.2), 30 hops max, 40 byte packets
                1 brloulsrac01-priv.amer.exel-intra.net (172.30.2.2) 0.190 ms 0.172 ms 0.165 ms
                [root@brloulsrac03 oraInventory]# traceroute 172.30.2.3
                traceroute to 172.30.2.3 (172.30.2.3), 30 hops max, 40 byte packets
                1 brloulsrac02-priv.amer.exel-intra.net (172.30.2.3) 0.134 ms 0.115 ms 0.105 ms
                [root@brloulsrac03 oraInventory]# traceroute 172.30.2.4
                traceroute to 172.30.2.4 (172.30.2.4), 30 hops max, 40 byte packets
                1 brloulsrac03-priv.amer.exel-intra.net (172.30.2.4) 0.027 ms 0.012 ms 0.006 ms

                ----------------------------------------------------------------------------------------------------------------------------------



                =============================================================================


                *11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network (Doc ID 981357.1)*


                It is the same case of Doc ID 369699.1

                =============================================================================
                Please, do you have any other idea?
                Thanks in advance.
                Leonardo
                • 5. Re: I got CRS-4402 error when I run root.sh
                  912595
                  Most likely:
                  Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]

                  Less likely:
                  Clusterware Not Starting On Newly Added RAC Node [ID 1270208.1]

                  Also run following to know
                  ./cluvfy.sh stage -pre crsinst -n <node2> -verbose

                  Post the results which you got
                  • 6. Re: I got CRS-4402 error when I run root.sh
                    user10674190
                    *Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]*

                    This note does not apply to 11.2.0.3 GI installations, and I am installing 11.2.0.3 GI .

                    ===================================================================================================




                    *Clusterware Not Starting On Newly Added RAC Node [ID 1270208.1]*


                    I don't see any problem in this setup...


                    [root@brloulsrac01 /]# /u01/11.2.0/grid/bin/oifcfg getif
                    eth2 7.9.8.0 global public
                    eth3 172.30.2.0 global cluster_interconnect
                    [root@brloulsrac01 /]# /u01/11.2.0/grid/bin/oifcfg iflist
                    eth2 7.9.8.0
                    eth3 172.30.2.0
                    eth3 169.254.0.0


                    [root@brloulsrac02 oraInventory]# /u01/11.2.0/grid/bin/oifcfg getif
                    PRIF-10: failed to initialize the cluster registry
                    [root@brloulsrac02 oraInventory]# /u01/11.2.0/grid/bin/oifcfg iflist
                    eth2 7.9.8.0
                    eth3 172.30.2.0


                    [root@brloulsrac03 oraInventory]# /u01/11.2.0/grid/bin/oifcfg getif
                    PRIF-10: failed to initialize the cluster registry
                    [root@brloulsrac03 oraInventory]# /u01/11.2.0/grid/bin/oifcfg iflist
                    eth2 7.9.8.0
                    eth3 172.30.2.0

                    ====================================================================================================


                    [grid@brloulsrac01 ~]$ /u01/11.2.0/grid/bin/cluvfy stage -pre crsinst -n brloulsrac02 -verbose

                    Performing pre-checks for cluster services setup

                    Checking node reachability...

                    Check: Node reachability from node "brloulsrac01"
                    Destination Node Reachable?
                    ------------------------------------ ------------------------
                    brloulsrac02 yes
                    Result: Node reachability check passed from node "brloulsrac01"


                    Checking user equivalence...

                    Check: User equivalence for user "grid"
                    Node Name Status
                    ------------------------------------ ------------------------
                    brloulsrac02 passed
                    Result: User equivalence check passed for user "grid"

                    Checking node connectivity...

                    Checking hosts config file...
                    Node Name Status
                    ------------------------------------ ------------------------
                    brloulsrac02 passed

                    Verification of the hosts config file successful


                    Interface information for node "brloulsrac02"
                    Name IP Address Subnet Gateway Def. Gateway HW Address MTU
                    ------ --------------- --------------- --------------- --------------- ----------------- ------
                    eth2 7.9.10.216 7.9.8.0 0.0.0.0 7.9.8.2 44:1E:A1:4A:75:20 1500
                    eth3 172.30.2.3 172.30.2.0 0.0.0.0 7.9.8.2 44:1E:A1:4A:75:24 9000


                    Check: Node connectivity of subnet "7.9.8.0"
                    Result: Node connectivity passed for subnet "7.9.8.0" with node(s) brloulsrac02


                    Check: TCP connectivity of subnet "7.9.8.0"
                    Source Destination Connected?
                    ------------------------------ ------------------------------ ----------------
                    brloulsrac01:7.9.10.215 brloulsrac02:7.9.10.216 passed
                    Result: TCP connectivity check passed for subnet "7.9.8.0"


                    Check: Node connectivity of subnet "172.30.2.0"
                    Result: Node connectivity passed for subnet "172.30.2.0" with node(s) brloulsrac02


                    Check: TCP connectivity of subnet "172.30.2.0"
                    Source Destination Connected?
                    ------------------------------ ------------------------------ ----------------
                    brloulsrac01:7.9.10.215 brloulsrac02:172.30.2.3 passed
                    Result: TCP connectivity check passed for subnet "172.30.2.0"


                    Interfaces found on subnet "7.9.8.0" that are likely candidates for VIP are:
                    brloulsrac02 eth2:7.9.10.216

                    Interfaces found on subnet "172.30.2.0" that are likely candidates for a private interconnect are:
                    brloulsrac02 eth3:172.30.2.3

                    Result: Node connectivity check passed

                    Checking multicast communication...

                    Checking subnet "7.9.8.0" for multicast communication with multicast group "230.0.1.0"...
                    Check of subnet "7.9.8.0" for multicast communication with multicast group "230.0.1.0" passed.

                    Checking subnet "172.30.2.0" for multicast communication with multicast group "230.0.1.0"...
                    Check of subnet "172.30.2.0" for multicast communication with multicast group "230.0.1.0" passed.

                    Check of multicast communication passed.

                    Checking ASMLib configuration.
                    Node Name Status
                    ------------------------------------ ------------------------
                    brloulsrac02 passed
                    Result: Check for ASMLib configuration passed.

                    Check: Total memory
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 78.6334GB (8.2453084E7KB) 1.5GB (1572864.0KB) passed
                    Result: Total memory check passed

                    Check: Available memory
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 78.0306GB (8.1821012E7KB) 50MB (51200.0KB) passed
                    Result: Available memory check passed

                    Check: Swap space
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 32GB (3.3554424E7KB) 16GB (1.6777216E7KB) passed
                    Result: Swap space check passed

                    Check: Free disk space for "brloulsrac02:/u01/11.2.0/grid"
                    Path Node Name Mount point Available Required Status
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    /u01/11.2.0/grid brloulsrac02 /u01 44.9492GB 5.5GB passed
                    Result: Free disk space check passed for "brloulsrac02:/u01/11.2.0/grid"

                    Check: Free disk space for "brloulsrac02:/tmp"
                    Path Node Name Mount point Available Required Status
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    /tmp brloulsrac02 /tmp 2.293GB 1GB passed
                    Result: Free disk space check passed for "brloulsrac02:/tmp"

                    Check: User existence for "grid"
                    Node Name Status Comment
                    ------------ ------------------------ ------------------------
                    brloulsrac02 passed exists(502)

                    Checking for multiple users with UID value 502
                    Result: Check for multiple users with UID value 502 passed
                    Result: User existence check passed for "grid"

                    Check: Group existence for "oinstall"
                    Node Name Status Comment
                    ------------ ------------------------ ------------------------
                    brloulsrac02 failed does not exist
                    Result: Group existence check failed for "oinstall"

                    Check: Group existence for "dba"
                    Node Name Status Comment
                    ------------ ------------------------ ------------------------
                    brloulsrac02 passed exists
                    Result: Group existence check passed for "dba"

                    Check: Membership of user "grid" in group "oinstall" [as Primary]
                    Node Name User Exists Group Exists User in Group Primary Status
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 yes yes no N/A failed
                    Result: Membership check for user "grid" in group "oinstall" [as Primary] failed

                    Check: Membership of user "grid" in group "dba"
                    Node Name User Exists Group Exists User in Group Status
                    ---------------- ------------ ------------ ------------ ----------------
                    brloulsrac02 yes yes yes passed
                    Result: Membership check for user "grid" in group "dba" passed

                    Check: Run level
                    Node Name run level Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 3 3,5 passed
                    Result: Run level check passed

                    Check: Hard limits for "maximum open file descriptors"
                    Node Name Type Available Required Status
                    ---------------- ------------ ------------ ------------ ----------------
                    brloulsrac02 hard 65536 65536 passed
                    Result: Hard limits check passed for "maximum open file descriptors"

                    Check: Soft limits for "maximum open file descriptors"
                    Node Name Type Available Required Status
                    ---------------- ------------ ------------ ------------ ----------------
                    brloulsrac02 soft 1024 1024 passed
                    Result: Soft limits check passed for "maximum open file descriptors"

                    Check: Hard limits for "maximum user processes"
                    Node Name Type Available Required Status
                    ---------------- ------------ ------------ ------------ ----------------
                    brloulsrac02 hard 16384 16384 passed
                    Result: Hard limits check passed for "maximum user processes"

                    Check: Soft limits for "maximum user processes"
                    Node Name Type Available Required Status
                    ---------------- ------------ ------------ ------------ ----------------
                    brloulsrac02 soft 2047 2047 passed
                    Result: Soft limits check passed for "maximum user processes"

                    Check: System architecture
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 x86_64 x86_64 passed
                    Result: System architecture check passed

                    Check: Kernel version
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 2.6.18-274.el5 2.6.18 passed
                    Result: Kernel version check passed

                    Check: Kernel parameter for "semmsl"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 250 250 250 passed
                    Result: Kernel parameter check passed for "semmsl"

                    Check: Kernel parameter for "semmns"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 32000 32000 32000 passed
                    Result: Kernel parameter check passed for "semmns"

                    Check: Kernel parameter for "semopm"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 100 100 100 passed
                    Result: Kernel parameter check passed for "semopm"

                    Check: Kernel parameter for "semmni"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 142 142 128 passed
                    Result: Kernel parameter check passed for "semmni"

                    Check: Kernel parameter for "shmmax"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 68719476736 68719476736 4294967295 passed
                    Result: Kernel parameter check passed for "shmmax"

                    Check: Kernel parameter for "shmmni"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 4096 4096 4096 passed
                    Result: Kernel parameter check passed for "shmmni"

                    Check: Kernel parameter for "shmall"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 4294967296 4294967296 2097152 passed
                    Result: Kernel parameter check passed for "shmall"

                    Check: Kernel parameter for "file-max"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 6815744 6815744 6815744 passed
                    Result: Kernel parameter check passed for "file-max"

                    Check: Kernel parameter for "ip_local_port_range"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
                    Result: Kernel parameter check passed for "ip_local_port_range"

                    Check: Kernel parameter for "rmem_default"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 262144 262144 262144 passed
                    Result: Kernel parameter check passed for "rmem_default"

                    Check: Kernel parameter for "rmem_max"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 4194304 4194304 4194304 passed
                    Result: Kernel parameter check passed for "rmem_max"

                    Check: Kernel parameter for "wmem_default"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 262144 262144 262144 passed
                    Result: Kernel parameter check passed for "wmem_default"

                    Check: Kernel parameter for "wmem_max"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 1048576 1048576 1048576 passed
                    Result: Kernel parameter check passed for "wmem_max"

                    Check: Kernel parameter for "aio-max-nr"
                    Node Name Current Configured Required Status Comment
                    ---------------- ------------ ------------ ------------ ------------ ------------
                    brloulsrac02 3145728 3145728 1048576 passed
                    Result: Kernel parameter check passed for "aio-max-nr"

                    Check: Package existence for "make"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 make-3.81-3.el5 make-3.81 passed
                    Result: Package existence check passed for "make"

                    Check: Package existence for "binutils"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 binutils-2.17.50.0.6-20.el5 binutils-2.17.50.0.6 passed
                    Result: Package existence check passed for "binutils"

                    Check: Package existence for "gcc(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 gcc(x86_64)-4.1.2-52.el5 gcc(x86_64)-4.1.2 passed
                    Result: Package existence check passed for "gcc(x86_64)"

                    Check: Package existence for "libaio(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 libaio(x86_64)-0.3.106-5 libaio(x86_64)-0.3.106 passed
                    Result: Package existence check passed for "libaio(x86_64)"

                    Check: Package existence for "glibc(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 glibc(x86_64)-2.5-81 glibc(x86_64)-2.5-24 passed
                    Result: Package existence check passed for "glibc(x86_64)"

                    Check: Package existence for "compat-libstdc++-33(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 compat-libstdc++-33(x86_64)-3.2.3-61 compat-libstdc++-33(x86_64)-3.2.3 passed
                    Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

                    Check: Package existence for "elfutils-libelf(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 elfutils-libelf(x86_64)-0.137-3.el5 elfutils-libelf(x86_64)-0.125 passed
                    Result: Package existence check passed for "elfutils-libelf(x86_64)"

                    Check: Package existence for "elfutils-libelf-devel"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed

                    WARNING:
                    PRVF-7584 : Multiple versions of package "elfutils-libelf-devel" found on node brloulsrac02: elfutils-libelf-devel(i386)-0.137-3.el5,elfutils-libelf-devel(x86_64)-0.137-3.el5
                    Result: Package existence check passed for "elfutils-libelf-devel"

                    Check: Package existence for "glibc-common"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 glibc-common-2.5-81 glibc-common-2.5 passed
                    Result: Package existence check passed for "glibc-common"

                    Check: Package existence for "glibc-devel(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 glibc-devel(x86_64)-2.5-81 glibc-devel(x86_64)-2.5 passed
                    Result: Package existence check passed for "glibc-devel(x86_64)"

                    Check: Package existence for "glibc-headers"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 glibc-headers-2.5-81 glibc-headers-2.5 passed
                    Result: Package existence check passed for "glibc-headers"

                    Check: Package existence for "gcc-c++(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 gcc-c++(x86_64)-4.1.2-52.el5 gcc-c++(x86_64)-4.1.2 passed
                    Result: Package existence check passed for "gcc-c++(x86_64)"

                    Check: Package existence for "libaio-devel(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 libaio-devel(x86_64)-0.3.106-5 libaio-devel(x86_64)-0.3.106 passed
                    Result: Package existence check passed for "libaio-devel(x86_64)"

                    Check: Package existence for "libgcc(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 libgcc(x86_64)-4.1.2-52.el5 libgcc(x86_64)-4.1.2 passed
                    Result: Package existence check passed for "libgcc(x86_64)"

                    Check: Package existence for "libstdc++(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 libstdc++(x86_64)-4.1.2-52.el5 libstdc++(x86_64)-4.1.2 passed
                    Result: Package existence check passed for "libstdc++(x86_64)"

                    Check: Package existence for "libstdc++-devel(x86_64)"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 libstdc++-devel(x86_64)-4.1.2-52.el5 libstdc++-devel(x86_64)-4.1.2 passed
                    Result: Package existence check passed for "libstdc++-devel(x86_64)"

                    Check: Package existence for "sysstat"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 sysstat-7.0.2-11.el5 sysstat-7.0.2 passed
                    Result: Package existence check passed for "sysstat"

                    Check: Package existence for "ksh"
                    Node Name Available Required Status
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 ksh-20100621-5.el5 ksh-20060214 passed
                    Result: Package existence check passed for "ksh"

                    Checking for multiple users with UID value 0
                    Result: Check for multiple users with UID value 0 passed

                    Check: Current group ID
                    Result: Current group ID check passed

                    Starting check for consistency of primary group of root user
                    Node Name Status
                    ------------------------------------ ------------------------
                    brloulsrac02 passed

                    Check for consistency of root user's primary group passed

                    Starting Clock synchronization checks using Network Time Protocol(NTP)...

                    NTP Configuration file check started...
                    The NTP configuration file "/etc/ntp.conf" is available on all nodes
                    NTP Configuration file check passed

                    Checking daemon liveness...

                    Check: Liveness for "ntpd"
                    Node Name Running?
                    ------------------------------------ ------------------------
                    brloulsrac02 yes
                    Result: Liveness check passed for "ntpd"
                    Check for NTP daemon or service alive passed on all nodes

                    Checking NTP daemon command line for slewing option "-x"
                    Check: NTP daemon command line
                    Node Name Slewing Option Set?
                    ------------------------------------ ------------------------
                    brloulsrac02 yes
                    Result:
                    NTP daemon slewing option check passed

                    Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

                    Check: NTP daemon's boot time configuration
                    Node Name Slewing Option Set?
                    ------------------------------------ ------------------------
                    brloulsrac02 yes
                    Result:
                    NTP daemon's boot time configuration check for slewing option passed

                    Checking whether NTP daemon or service is using UDP port 123 on all nodes

                    Check for NTP daemon or service using UDP port 123
                    Node Name Port Open?
                    ------------------------------------ ------------------------
                    brloulsrac02 yes

                    NTP common Time Server Check started...
                    NTP Time Server "10.35.100.100" is common to all nodes on which the NTP daemon is running
                    NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
                    Check of common NTP Time Server passed

                    Clock time offset check from NTP Time Server started...
                    Checking on nodes "[brloulsrac02]"...
                    Check: Clock time offset from NTP Time Server

                    Time Server: 10.35.100.100
                    Time Offset Limit: 1000.0 msecs
                    Node Name Time Offset Status
                    ------------ ------------------------ ------------------------
                    brloulsrac02 -0.569 passed
                    Time Server "10.35.100.100" has time offsets that are within permissible limits for nodes "[brloulsrac02]".

                    Time Server: .LOCL.
                    Time Offset Limit: 1000.0 msecs
                    Node Name Time Offset Status
                    ------------ ------------------------ ------------------------
                    brloulsrac02 0.0 passed
                    Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[brloulsrac02]".
                    Clock time offset check passed

                    Result: Clock synchronization check using Network Time Protocol(NTP) passed

                    Checking Core file name pattern consistency...
                    Core file name pattern consistency check passed.

                    Checking to make sure user "grid" is not in "root" group
                    Node Name Status Comment
                    ------------ ------------------------ ------------------------
                    brloulsrac02 passed does not exist
                    Result: User "grid" is not part of "root" group. Check passed

                    Check default user file creation mask
                    Node Name Available Required Comment
                    ------------ ------------------------ ------------------------ ----------
                    brloulsrac02 0022 0022 passed
                    Result: Default user file creation mask check passed
                    Checking consistency of file "/etc/resolv.conf" across nodes

                    Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
                    File "/etc/resolv.conf" does not have both domain and search entries defined
                    Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
                    domain entry in file "/etc/resolv.conf" is consistent across nodes
                    Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
                    search entry in file "/etc/resolv.conf" is consistent across nodes
                    Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
                    All nodes have one search entry defined in file "/etc/resolv.conf"
                    Checking DNS response time for an unreachable node
                    Node Name Status
                    ------------------------------------ ------------------------
                    brloulsrac02 passed
                    The DNS response time for an unreachable node is within acceptable limit on all nodes

                    File "/etc/resolv.conf" is consistent across nodes

                    Check: Time zone consistency
                    Result: Time zone consistency check passed

                    Pre-check for cluster services setup was unsuccessful on all the nodes.
                    • 7. Re: I got CRS-4402 error when I run root.sh
                      912595
                      check permissions on "/var/tmp/.oracle" and also check followoing
                      Clusterware Not Starting On Newly Added RAC Node [ID 1270208.1]

                      also check what message you get in alert.log,ocssd.log,crsd.log etc in grid home
                      • 8. Re: I got CRS-4402 error when I run root.sh
                        user10674190
                        ===================================================================================================



                        *Clusterware Not Starting On Newly Added RAC Node [ID 1270208.1]*



                        [root@brloulsrac01 /]# /u01/11.2.0/grid/bin/oifcfg getif
                        eth2 7.9.8.0 global public
                        eth3 172.30.2.0 global cluster_interconnect                              
                        [root@brloulsrac01 /]# /u01/11.2.0/grid/bin/oifcfg iflist
                        eth2 7.9.8.0
                        eth3 172.30.2.0
                        eth3 169.254.0.0


                        [root@brloulsrac02 oraInventory]# /u01/11.2.0/grid/bin/oifcfg getif
                        PRIF-10: failed to initialize the cluster registry          *<-------- here I got this error, is it a problem?*

                        [root@brloulsrac02 oraInventory]# /u01/11.2.0/grid/bin/oifcfg iflist
                        eth2 7.9.8.0
                        eth3 172.30.2.0


                        [root@brloulsrac03 oraInventory]# /u01/11.2.0/grid/bin/oifcfg getif
                        PRIF-10: failed to initialize the cluster registry     *<-------- here I got this error, is it a problem?*

                        [root@brloulsrac03 oraInventory]# /u01/11.2.0/grid/bin/oifcfg iflist
                        eth2 7.9.8.0
                        eth3 172.30.2.0

                        ====================================================================================================


                        About the result command line below, do you see any problem?

                        /u01/11.2.0/grid/bin/cluvfy stage -pre crsinst -n brloulsrac02 -verbose

                        ====================================================================================================


                        check permissions on "/var/tmp/.oracle"
                        node1 (brloulsrac01):

                        [root@brloulsrac01 tmp]# pwd
                        /var/tmp

                        [root@brloulsrac01 tmp]# ls -lart
                        total 24
                        drwxr-xr-x 7 root root 4096 Mar 12 14:37 OFED_topdir
                        drwxr-xr-x 23 root root 4096 Mar 12 17:53 ..
                        drwxrwxrwt 4 root root 4096 Mar 15 08:46 .
                        drwxrwxrwt 2 root dba 4096 Mar 15 08:52 .oracle

                        node2 (brloulsrac02):

                        [root@brloulsrac02 ~]# cd /var/tmp

                        [root@brloulsrac02 tmp]# ls -lart
                        total 20
                        drwxr-xr-x 23 root root 4096 Mar 12 18:22 ..
                        drwxrwxrwt 3 root root 4096 Mar 15 08:55 .
                        drwxrwxrwt 2 root dba 4096 Mar 15 08:59 .oracle

                        node3 (brloulsrac03):


                        [root@brloulsrac03 ~]# cd /var/tmp

                        [root@brloulsrac03 tmp]# ls -lart
                        total 20
                        drwxr-xr-x 23 root root 4096 Mar 12 17:52 ..
                        drwxrwxrwt 3 root root 4096 Mar 15 09:04 .
                        drwxrwxrwt 2 root dba 4096 Mar 15 09:08 .oracle


                        Is it correct?
                        ====================================================================================================

                        *and also check followoing Clusterware Not Starting On Newly Added RAC Node [ID 1270208.1]*

                        *I checked the CSS log on node2 (brloulsrac02), and I didn't see any thing about "dynamic endp gipc://"
                        I don't if it is a problem...
                        But, please, see result command line below, I am using the same subnet network:*

                        [root@brloulsrac01 tmp]# /u01/11.2.0/grid/bin/oifcfg getif
                        eth2 7.9.8.0 global public
                        eth3 172.30.2.0 global cluster_interconnect


                        [root@brloulsrac02 tmp]# /u01/11.2.0/grid/bin/oifcfg iflist
                        eth2 7.9.8.0
                        eth3 172.30.2.0

                        What do you think?
                        • 9. Re: I got CRS-4402 error when I run root.sh
                          user10674190
                          I disabled Jumbo Frames (MTU from 9000 to 1500) and root.sh passed on the nodes 2 and 3, it is OK now!!!!

                          [root@brloulsrac01 ~]# [$GRID_HOME]/bin/crsctl stop crs -f

                          [root@brloulsrac01 ~]# ifconfig eth3 mtu 1500
                          [root@brloulsrac02 ~]# ifconfig eth3 mtu 1500
                          [root@brloulsrac02 ~]# ifconfig eth3 mtu 1500


                          [root@brloulsrac01 ~]# [$GRID_HOME]/bin/crsctl start crs

                          I ran root.sh on node 2 and 3.

                          I will talk with Network team about Jumbo Frames, I think it is disable on Switch/VLAN.

                          Thanks for your help.