11 Replies Latest reply: Jun 11, 2014 1:05 PM by 4youall RSS

    Not able to stop crs

    KarthikSinghDBA

      Dear Legends,

       

      As I was trying to decommision a database as I did

      - Shutdown 2 DB's

      - Shutdown 2 Listeners

      - Shutdown agents

      - Shutdown ASM Instance

       

      But I forget to stop the crs when I checked with

       

      ps -ef|grep has

      grid      2280  2104  0 07:18 pts/3    00:00:00 grep has

      grid      3030     1  0 Apr26 ?        00:06:27 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

      root      3338     1  0 Apr26 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run

       

      Error or Email Alert receiving due to this as follows

      Corrective action=USER_DEFINED_ASM_USAGE_ERRORS_CORRECTIVE_ACTION

      Corrective action owner=SYSMAN

      Corrective action status=Succeeded

      Corrective action output=

      Command:Output Log

      SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 4 00:07:15 2014

      Copyright (c) 1982, 2009, Oracle. All rights reserved.

      SQL> SQL> SQL> SQL> Connected.

      SQL> SQL> SQL> SQL> SQL> SQL> 2 3

      AUE_HOST AUE_DATAB

      ---------------------------------------------------------------- ---------

      AUE_TIMES

      ---------

      AUE_ERROR_TEXT

      --------------------------------------------------------------------------------

      Beldon.host.com RMCLDDMO

      04-JUN-14

      ORA-12541: TNS:no listener

      SQL> SQL> 2 3

      1 row updated.

      SQL> SQL>

      Commit complete.

      SQL> SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production

      With the Partitioning, Automatic Storage Management, OLAP, Data Mining

      and Real Application Testing options

      ~~End Step Output/Error Log~~

      Target Name=GOCPRD1.host.com

      Target type=

      Host=

      Occurred At=Jun 4, 2014 12:07:16 AM EDT

      Message=

      Metric=User-Defined Numeric Metric

      Metric value=1

      Script=User_Defined_ASM_USAGE_ERRORS

      Severity=Warning

      Acknowledged=

      Notification Rule Name=ASM Usage Error Notification rule (User Defined)

      Notification Rule Owner=SYSMAN

       

      When I tried to stop using below command and it errored out

      ./crsctl stop has

      CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'beldon'

      CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'beldon'

      CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'beldon' failed

      CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'beldon' has failed

      CRS-4687: Shutdown command has completed with error(s).

      CRS-4000: Command Stop failed, or completed with errors.

       

      Let me know what steps should I follow as I am new to the Environment need to stick and learn more. But I agree I didn't follow a standard procedure to decommission.

       

      Thanks,

      Karthik

        • 1. Re: Not able to stop crs
          FreddieEssex

          You just need to stop HAS:

           

          crsctl stop has
          

           

          Check if it's stopped:

           

          crsctl check has
          

           

          However it doesn't really matter if you stop this or not if you're then decommissioning the server.  Even though HAS is down the daemons sill still be running.

          • 2. Re: Not able to stop crs
            KarthikSinghDBA

            Thanks Freddie,

             

            I tried and the output as follows

             

            [grid@host bin]$ ./crsctl stop has

            CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'host'

            CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'

            CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed

            CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host' has failed

            CRS-4687: Shutdown command has completed with error(s).

            CRS-4000: Command Stop failed, or completed with errors.

            [grid@host bin]$ ./crsctl check has

            CRS-4638: Oracle High Availability Services is online

             

            Also tried

             

            [grid@host ~]$ crsctl stop res ora.LISTENER_RMCLDDG1.lsnr -f

            CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'

            CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed

            CRS-5802: Unable to start the agent process

            CRS-4000: Command Stop failed, or completed with errors.

             

             

            /etc/init.d/ohasd stop

            Stopping Oracle Clusterware stackCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'host'

            CRS-2679: Attempting to clean 'ora.LISTENER_RMCLDDG1.lsnr' on 'host'

            CRS-2680: Clean of 'ora.LISTENER_RMCLDDG1.lsnr' on 'host' failed

            CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host' has failed

            CRS-4687: Shutdown command has completed with error(s).

            CRS-4000: Command Stop failed, or completed with errors.

             

            Thanks,

            Karthik

            • 3. Re: Not able to stop crs
              KarthikSinghDBA

              Also while checking the status the listener which is problematic has went to UNKNOWN state

               

              NAME=ora.LISTENER_RMCLDDG1.lsnr

              TYPE=ora.listener.type

              TARGET=OFFLINE

              STATE=UNKNOWN on host

               

              Not sure still the process is running

              grid      3030     1  0 Apr26 ?        00:06:28 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

              root      3338     1  0 Apr26 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run

               

              Even we can kill the process but I would like to know the standard procedure and If I kill it later while starting this instance it may lead to another problem.

              • 4. Re: Not able to stop crs
                FreddieEssex

                Can you try the following:

                 

                crscts stat res -t
                crsctl disable has
                crsctl stop has
                crsctl check has
                

                 

                Sorry...I have no test environment where I can check out Oracle restart as they are all in use....

                 

                The "stop has" would have stopped the database and listener if they were both registered.

                 

                I can't tell you for certain the state of the daemons as I can't remember of the top of my head, but I think the dameons will still run in the background.

                • 5. Re: Not able to stop crs
                  KarthikSinghDBA

                  Oh Thanks Freddie,

                   

                  As it is a decommissioning machine we just did a kill for 3030

                  grid      3030     1  0 Apr26 ?        00:06:28 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

                  root      3338     1  0 Apr26 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run

                   

                  but still init.ohasd is still running with a different process id

                  root      3358     1  0 Apr26 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run


                  ./crsctl stat res -t

                  CRS-4639: Could not contact Oracle High Availability Services

                  CRS-4000: Command Status failed, or completed with errors.

                   

                  I think now there is no harm at the current state of the server.

                  Regards,

                  Karthik

                  • 6. Re: Not able to stop crs
                    FreddieEssex

                    Presumably the date pof the new process is not Apr 26?

                     

                    Yes....if the daemons are killed they are just respawned by init which is what you are seeing.

                     

                    On AIX the file is /etc/inittab.  I think on Linux it's the same file and you'll see a line with the ohasd process.

                    • 7. Re: Not able to stop crs
                      BPeaslandDBA

                      If you are decomissioning this server, then you will want to remove it from the cluster. And you will want to remove the database instances from the RAC configuration. First, delete Oracle RAC from the nodes. Then, delete the node from the cluster. All documented with the links I provided.

                       

                      HTH,

                      Brian

                      • 8. Re: Not able to stop crs
                        KarthikSinghDBA

                        Presumably the date pof the new process is not Apr 26?

                        Yes now it changed to Yesterday's date.

                         

                        Yes....if the daemons are killed they are just respawned by init which is what you are seeing.

                         

                        On AIX the file is /etc/inittab.  I think on Linux it's the same file and you'll see a line with the ohasd process.

                        So It means it will be running/respawning even we kill it, only thing is we can disable the restart ?

                        • 9. Re: Not able to stop crs
                          KarthikSinghDBA

                          Thanks Brian For your valuable response,

                           

                          Since I'm new to the organisation getting into new things quickly.

                          If you are decomissioning this server, then you will want to remove it from the cluster. And you will want to remove the database instances from the RAC configuration. First, delete Oracle RAC from the nodes. Then, delete the node from the cluster. All documented with the links I provided.

                          Had read the referred doc very useful and I think the machine is a Single Node RAC having 2 Databases.

                           

                          FYI - There are several Hosts/Machines controlled by GRID/EM with ASM. Out of those I'm decommissioning 1 Host/Machine which has 2 databases.

                           

                          Thanks,

                          Karthik

                          • 10. Re: Not able to stop crs
                            FreddieEssex

                            Removing the line from the inittab and then killing it will stop it respawing.

                             

                            However, presumably if the server will be decommisioned and rebuilt I probably wouldn't worry about it.

                             

                            From the commands you supply it seems like it's Oracle Restart you are using and not RAC One Node??

                             

                            If it's Oracle Restart then decommisioning should be straight forward as it's essentially just an independant machine running some databases.

                             

                            http://docs.oracle.com/cd/E18283_01/server.112/e17120/restart001.htm

                             

                            If Its RAC One Node then there might be more to it as you will have other servers in the cluster.

                             

                            http://docs.oracle.com/cd/E11882_01/rac.112/e41960/onenode.htm#RACAD7894

                            • 11. Re: Not able to stop crs
                              4youall

                              Can you try  stopping with force option?