8 Replies Latest reply: May 29, 2014 8:43 PM by 2683578 RSS

    NOt able to start CRS

    974882

      Hi Support ,

       

      We are having two node RAC, due to some hardware issue both servers goes down. Both servers came up now but RAC Services/Cluster has bee started in only one node where as in other node we are facing issue.

      OS: Solaris.

      version Oracle: 10.2.0

       

      Refer the logs:

      -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

      oracle$ crsctl check crs

      Failure 1 contacting CSS daemon

      Cannot communicate with CRS

      Cannot communicate with EVM

       

      oracle$ crsctl start crs

      Insufficient user privileges.   {Using Oracle user}

       

      root$  crsctl start crs

      crsctl: not found                    {Using root user}

       

      root$/etc/init.d/init.crs start

      Startup will be queued to init within 30 seconds.

       

      CRS logs:

      -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

      2014-04-04 18:20:02.741: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

      2014-04-04 18:20:03.748: [ COMMCRS][2058]clsc_connect: (100a513d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ch1d5bl0_localhost))

      2014-04-04 18:20:03.748: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

      -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

      CSS logs: Are not getting generated after march 2013.

       

      I have verified the permissions and ownership of crs/bin, it seems ok. /etc/inittab all enteries are availble. OCR and voting disk are in sync/same with correct permissions.

       

      Permissions:

      -------------------------------------------------------------------------------------

      drwxr-xr-x   2 root     oinstall    3072 Mar 12  2013 bin

      $ id

      uid=102(oracle) gid=200(oinstall)

      root$ id

      uid=0(root) gid=0(root)

      ------------------------------------------------------------------------------------

       

      thanks....

        • 1. Re: NOt able to start CRS
          Tom321

          Hi,

           

          on the failing node as root user go to grid bin directory and execute

           

          ./crsctl stat res -t -init

           

          That will show you the status of your crs component, if any are running shut them down to clean up the crs stack:

          ./crsctl stop crs -f

           

          Then try to startup:

          ./crsctl start crs

           

          If that does not work check the different logfiles for crs. Beside the logfile in .../grid/log/your_node_name/alertrac_your_node_name.log you should find directories for cssd, crsd, ...

          Some files should hold an error entry, to get an idea where the error is. Otherwise it is pure guessing.

           

          Regards

          Thomas

          • 2. Re: NOt able to start CRS
            974882

            Hi Thomas,

             

            Thanks for the revert .

             

            I have executed the sharable points :

            root# cd /u01/app/oracle/product/10.2.0/crs_1/bin

            root # ./crsctl stat res -t -init

            Unknown parameter: stat

             

            root # ./crsctl stop crs -f

            Stopping CSSD.

            Unable to communicate with the CSS daemon.

             

            root # ./crsctl start crs

            Attempting to start CRS stack

            The CRS stack will be started shortly

             

            I have check the logs available in CRSD & whereas CSSD logs are not generated:

            --------------------------------------------------------------------------------------------------------------------------------------------------------------

            2014-04-09 10:45:31.898: [ COMMCRS][586]clsc_connect: (100a513d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ch1d5bl0_localhost))

            2014-04-09 10:45:31.898: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

            2014-04-09 10:45:31.899: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

            --------------------------------------------------------------------------------------------------------------------------------------------------------------

            • 3. Re: NOt able to start CRS
              Vandana B -Oracle

              Hi,

               

              Appears to be some socket file issue. Try the following and let us know if the same resolves your issue

               

              On all nodes

               

              As root user

              ++ rm -rf /var/tmp/.oracle /tmp/.oracle

               

              Once the above is done, reboot the nodes

               

              Regards,

              Vandana - Oracle

              • 4. Re: NOt able to start CRS
                Tom321

                "Unknown parameter: stat"

                Right... 10g hasn't integrated it in crsctl

                ./crs_stat -t  -> that will show the resources

                ./crsctl check crs -> will show which of the 3 demons runs

                 

                Regarding the logfiles check these locations:

                $crs_home/crs/log $crs_home/css/log ... <- here are the logfiles AFTER successfull startup of a crs stack component

                $crs_home/crs/init  <- here are the logfiles during startup of the crs stack. If anything goes wrong you should find some info here

                • 5. Re: NOt able to start CRS
                  Shivendra Narain Nirala

                  Hi,

                   

                  It seems to me this issue related to voting disk.Please check the ocssd.log at the location mentionetd by TOM .

                   

                  Apart from that , Please run the following commands and share the output :-

                   

                  ./ocrcheck

                   

                  ./crsctl query css votedisk

                   

                  warm regards

                   

                  Shivendra Narain Nirala

                  • 6. Re: NOt able to start CRS
                    Kfekete-Oracle

                    Hi Harish,

                     

                    crsd.log indicates that cssd is down

                    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                    2014-04-04 18:20:02.741: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

                    2014-04-04 18:20:03.748: [ COMMCRS][2058]clsc_connect: (100a513d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ch1d5bl0_localhost))

                    2014-04-04 18:20:03.748: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

                     

                    Action plan

                    ----------

                    1) stop clusterware with crsctl stop crs -f (if that fails reboot the node)

                    2) remove socket

                    rm -rf /tmp/.oracle/*

                    rm -rf /var/tmp/.oracle/*

                    3) startup clusterware with crsctl start crs

                    4) check whether ocssd.bin is running at OS level (ps -ef|grep ocssd.bin)

                    5) review ocssd.log under $CRS_HOME/log/<nodename>/cssd

                     

                    Cheers,

                    Krisz

                    • 7. Re: NOt able to start CRS
                      Levi Pereira

                      While this message "[  CRSRTI][1] CSS is not ready"  is written on crsd.log.

                       

                      The error with a root cause is logged on CSSD.log and Alertlog of cluster.

                      • 8. Re: NOt able to start CRS
                        2683578

                        you have to do it as root."crsctl: not found",because you not add $crs_home/bin to Environment Variables.or you can execute "$crs_home/bin/crsctl start crs",it can be OK.