1 2 Previous Next 26 Replies Latest reply: Apr 26, 2013 7:17 AM by FreddieEssex RSS

    Second Node seen offline even though its online in RAC

    1005155
      Hi,
      I have 3 node RAC setup on RHEL 6.Everything was working fine before I rebooted the second node.
      After the second node came up, the CRS doesnt start on this node.
      If I run the nodeapps from the first node, it shows that the second node is offline.

      ===================================================
      From Node 2 :
      bash-4.1$ ./crs_start -all
      CRS-0184: Cannot communicate with the CRS daemon.

      From Node 1 :
      [root@NikhilRac1 bin]# ./srvctl start nodeapps -n nikhilrac2
      PRCR-1013 : Failed to start resource ora.net1.network
      PRCR-1064 : Failed to start resource ora.net1.network on node nikhilrac2
      CRS-2546: Server 'nikhilrac2' is not online
      ====================================================
      I looked into all the logs but didnt find any clue.

      Can somebody please help me ?


      Thanks in advance,
      Nikhil.
        • 2. Re: Second Node seen offline even though its online in RAC
          1005155
          Hi,
          The version is 11gR2



          Thanks,
          Nikhil
          • 3. Re: Second Node seen offline even though its online in RAC
            FreddieEssex
            So there is no error message in log files in $GRID_HOME/log/`hostname` ????

            What errors when you run the following:
            crsctl start crs
            • 4. Re: Second Node seen offline even though its online in RAC
              1005155
              Hi,
              Please find below the output of the start crs command :

              [root@NikhilRac2 bin]# ./crsctl start crs
              CRS-4124: Oracle High Availability Services startup failed.
              CRS-4000: Command Start failed, or completed with errors.



              Thanks,
              Nikhil.
              • 5. Re: Second Node seen offline even though its online in RAC
                FreddieEssex
                There should be additional error messages in the log file I mentioned.

                Also check Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1].
                • 6. Re: Second Node seen offline even though its online in RAC
                  1005155
                  Hi,
                  I checked the alertlogfile again and found the below error :
                  The OCR location in an ASM disk group is inaccessible.

                  -bash-4.1$ ./ocrcheck -config
                  Oracle Cluster Registry configuration is :
                  Device/File Name : +DATA


                  Can you please suggest what I am misisng ?


                  Thanks,
                  Nikhil.
                  • 7. Re: Second Node seen offline even though its online in RAC
                    FreddieEssex
                    Sounds like permissions of the disks.

                    You need to ensure that permissions of the ASM disks persist across a reboot:

                    Maybe this will help:

                    http://www.oracle-base.com/articles/10g/asm-using-asmlib-and-raw-devices.php
                    • 8. Re: Second Node seen offline even though its online in RAC
                      1005155
                      Hi,
                      Thanks for the reply.
                      I checked the permissions and it looks fine :

                      [root@NikhilRac2 dev]# ls -l | grep grid
                      brw-rw---- 1 grid oinstall 8, 16 Apr 24 07:36 sdb
                      brw-rw---- 1 grid oinstall 8, 32 Apr 24 07:36 sdc
                      brw-rw---- 1 grid oinstall 8, 48 Apr 24 07:36 sdd
                      brw-rw---- 1 grid oinstall 8, 64 Apr 24 07:36 sde
                      brw-rw---- 1 grid oinstall 8, 80 Apr 24 07:36 sdf
                      brw-rw---- 1 grid oinstall 8, 96 Apr 24 07:36 sdg
                      brw-rw---- 1 grid oinstall 8, 112 Apr 24 07:36 sdh
                      brw-rw---- 1 grid oinstall 8, 128 Apr 24 07:36 sdi

                      Anything I am missing ?

                      Thanks,
                      Nikhil.
                      • 9. Re: Second Node seen offline even though its online in RAC
                        FreddieEssex
                        Can you post the full error message from your logs please. A few lines before and after the error would also be handy.

                        Does your alert log mention any other logfile? If so, post the error from that logfile as well.
                        • 10. Re: Second Node seen offline even though its online in RAC
                          1005155
                          Hi,
                          From the alertlog file :

                          [client(3677)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 07:45:57.436
                          [client(3461)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:08.124
                          [client(5547)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:08.127
                          [client(5547)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log.
                          2013-04-24 09:26:13.251
                          [client(5570)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:13.254
                          [client(5570)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log.
                          2013-04-24 09:26:18.527
                          [client(5605)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:18.530
                          [client(5605)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log.
                          2013-04-24 09:26:23.440
                          [client(5613)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:23.442
                          [client(5613)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log.
                          2013-04-24 09:26:28.792
                          [client(5622)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
                          2013-04-24 09:26:28.794
                          [client(5622)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log.

                          ==========================================================
                          From /u03/app/11.2.0/grid/log/nikhilrac2/client/crsctl.log :

                          Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
                          2013-04-24 09:26:13.328: [ CSSCLNT][2145265440]clssscConnect: gipc request failed with 29 (0x13)
                          2013-04-24 09:26:13.328: [ CSSCLNT][2145265440]clsssInitNative: connect failed, rc 29
                          2013-04-24 12:10:26.784: [ CSSCLNT][3119134496]clssscConnect: gipc request failed with 29 (0x13)
                          2013-04-24 12:10:26.784: [ CSSCLNT][3119134496]clsssInitNative: connect failed, rc 29




                          Thanks,
                          Nikhil.
                          • 11. Re: Second Node seen offline even though its online in RAC
                            FreddieEssex
                            Please check $GRID_HOME/log/`hostname/cssd/ocssd.log

                            Any errors in here for the time when you tried to start CRS?
                            • 12. Re: Second Node seen offline even though its online in RAC
                              FreddieEssex
                              Can you also run the following:
                              cluvfy stage -post crsinst -n all -verbose
                              • 13. Re: Second Node seen offline even though its online in RAC
                                1005155
                                Hi,
                                Thanks for the reply.
                                Both the servers are rebooted now.After the servers are up, I ran the cluvfy :
                                ================================================================
                                -bash-4.1$ ./cluvfy stage -post crsinst -n nikhilrac1,nikhilrac2

                                Performing post-checks for cluster services setup

                                Checking node reachability...
                                Node reachability check passed from node "NikhilRac3"


                                Checking user equivalence...
                                User equivalence check passed for user "grid"

                                ERROR:
                                /u03/app/oraInventory/ContentsXML/inventory.xml (No such file or directory)

                                ERROR:
                                CRS is not installed on any of the nodes
                                Verification cannot proceed


                                Post-check for cluster services setup was unsuccessful on all the nodes.
                                ================================================================

                                Thanks,
                                Nikhil.
                                • 14. Re: Second Node seen offline even though its online in RAC
                                  1005155
                                  Hi,
                                  Thanks for the reply.
                                  Both the servers are rebooted now.After the servers are up, I ran the cluvfy :
                                  ================================================================
                                  -bash-4.1$ ./cluvfy stage -post crsinst -n nikhilrac1,nikhilrac2

                                  Performing post-checks for cluster services setup

                                  Checking node reachability...
                                  Node reachability check passed from node "NikhilRac3"


                                  Checking user equivalence...
                                  User equivalence check passed for user "grid"

                                  ERROR:
                                  /u03/app/oraInventory/ContentsXML/inventory.xml (No such file or directory)

                                  ERROR:
                                  CRS is not installed on any of the nodes
                                  Verification cannot proceed


                                  Post-check for cluster services setup was unsuccessful on all the nodes.
                                  ================================================================

                                  Thanks,
                                  Nikhil.
                                  1 2 Previous Next