11 Replies Latest reply: Mar 25, 2014 8:37 PM by 892776 RSS

    RAC database

    892776

      hello all

      my RAC DB is not starting.

       

      [oracle@node1 ~]$ lsnrctl status

       

      LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 24-MAR-2014 16:39:20

       

      Copyright (c) 1991, 2009, Oracle.  All rights reserved.

       

      Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))

      STATUS of the LISTENER

      ------------------------

      Alias                     LISTENER

      Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production

      Start Date                24-MAR-2014 16:28:59

      Uptime                    0 days 0 hr. 10 min. 20 sec

      Trace Level               off

      Security                  ON: Local OS Authentication

      SNMP                      OFF

      Listener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.ora

      Listener Log File         /u01/app/grid/diag/tnslsnr/node1/listener/alert/log.xml

      Listening Endpoints Summary...

        (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.80.7.27)(PORT=1521)))

        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.80.7.29)(PORT=1521)))

      Services Summary...

      Service "+ASM" has 2 instance(s).

        Instance "+ASM1", status UNKNOWN, has 1 handler(s) for this service...

        Instance "+ASM1", status READY, has 1 handler(s) for this service...

      Service "PLSExtProc" has 1 instance(s).

        Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...

      The command completed successfully

      [oracle@node1 ~]$

       

       

      [oracle@node1 ~]$ srvctl status database -d racdb

      Instance racdb1 is not running on node node1

      Instance racdb2 is not running on node node2

       

       

       

      how to resolve ?

      help me

        • 1. Re: RAC database
          Hemant K Chitale

          Was there are server reboot ?  Is that why you expect a database startup ?

           

          So what happens if you try to startup the database instance (maybe only one the one instance on node1 first) using srvctl ?   Does it succeed ?  Does the alert.log show any errors ?

           

           

          Hemant K Chitale

          • 2. Re: RAC database
            892776

            i am used crs_start -all

             

             

            [grid@node2 ~]$ crs_start -all

            CRS-5702: Resource 'ora.CRS.dg' is already running on 'node1'

            CRS-5702: Resource 'ora.ETAX.dg' is already running on 'node1'

            CRS-5702: Resource 'ora.FRA.dg' is already running on 'node1'

            CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'node1'

            CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'node2'

            CRS-5702: Resource 'ora.asm' is already running on 'node1'

            CRS-5702: Resource 'ora.eons' is already running on 'node1'

            CRS-5702: Resource 'ora.gsd' is already running on 'node1'

            CRS-5702: Resource 'ora.net1.network' is already running on 'node1'

            CRS-5702: Resource 'ora.asm' is already running on 'node1'

            CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'node1'

            CRS-5702: Resource 'ora.gsd' is already running on 'node1'

            CRS-5702: Resource 'ora.ons' is already running on 'node1'

            CRS-5702: Resource 'ora.node1.vip' is already running on 'node1'

            CRS-5702: Resource 'ora.asm' is already running on 'node2'

            CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'node2'

            CRS-5702: Resource 'ora.gsd' is already running on 'node2'

            CRS-5702: Resource 'ora.ons' is already running on 'node2'

            CRS-5702: Resource 'ora.node2.vip' is already running on 'node2'

            CRS-5702: Resource 'ora.oc4j' is already running on 'node1'

            CRS-5702: Resource 'ora.ons' is already running on 'node1'

            CRS-5702: Resource 'ora.scan1.vip' is already running on 'node2'

            Attempting to start `ora.racdb.db` on member `node1`

            Attempting to start `ora.racdb.db` on member `node2`

            ORA-01034: ORACLE not available

            ORA-27101: shared memory realm does not exist

            Linux-x86_64 Error: 2: No such file or directory

            Process ID: 0

            Session ID: 0 Serial number: 0

             

            ORA-01078: failure in processing system parameters

            Start of `ora.racdb.db` on member `node2` failed.

            Attempting to stop `ora.racdb.db` on member `node2`

            ORA-01034: ORACLE not available

            ORA-27101: shared memory realm does not exist

            Linux-x86_64 Error: 2: No such file or directory

            Process ID: 0

            Session ID: 0 Serial number: 0

             

            ORA-01034: ORACLE not available

            ORA-27101: shared memory realm does not exist

            Linux-x86_64 Error: 2: No such file or directory

            Process ID: 0

            Session ID: 0 Serial number: 0

             

            Stop of `ora.racdb.db` on member `node2` succeeded.

            CRS-2632: There are no more servers to try to place resource 'ora.racdb.db' on that would satisfy its placement policy

            ORA-01078: failure in processing system parameters

            Start of `ora.racdb.db` on member `node1` failed.

            Attempting to stop `ora.racdb.db` on member `node1`

            ORA-01034: ORACLE not available

            ORA-27101: shared memory realm does not exist

            Linux-x86_64 Error: 2: No such file or directory

            Process ID: 0

            Session ID: 0 Serial number: 0

             

            Stop of `ora.racdb.db` on member `node1` succeeded.

            CRS-0223: Resource 'ora.CRS.dg' has placement error.

             

            CRS-0223: Resource 'ora.ETAX.dg' has placement error.

             

            CRS-0223: Resource 'ora.FRA.dg' has placement error.

             

            CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.

             

            CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.

             

            CRS-0223: Resource 'ora.asm' has placement error.

             

            CRS-0223: Resource 'ora.eons' has placement error.

             

            CRS-0223: Resource 'ora.gsd' has placement error.

             

            CRS-0223: Resource 'ora.net1.network' has placement error.

             

            CRS-0223: Resource 'ora.node1.ASM1.asm' has placement error.

             

            CRS-0223: Resource 'ora.node1.LISTENER_NODE1.lsnr' has placement error.

             

            CRS-0223: Resource 'ora.node1.gsd' has placement error.

             

            CRS-0223: Resource 'ora.node1.ons' has placement error.

             

            CRS-0223: Resource 'ora.node1.vip' has placement error.

             

            CRS-0223: Resource 'ora.node2.ASM2.asm' has placement error.

             

            CRS-0223: Resource 'ora.node2.LISTENER_NODE2.lsnr' has placement error.

             

            CRS-0223: Resource 'ora.node2.gsd' has placement error.

             

            CRS-0223: Resource 'ora.node2.ons' has placement error.

             

            CRS-0223: Resource 'ora.node2.vip' has placement error.

             

            CRS-0223: Resource 'ora.oc4j' has placement error.

             

            CRS-0223: Resource 'ora.ons' has placement error.

             

            CRS-0215: Could not start resource 'ora.racdb.db 1 1'.

             

            CRS-0223: Resource 'ora.racdb.db 2 1' has placement error.

             

            CRS-0223: Resource 'ora.scan1.vip' has placement error.

            • 3. Re: RAC database
              Simo Kemppinen

              It is easier to debug if you try to start these one thing at the time.

              So first stop everything and then try to first start CRS in one node:

              crsctl check crs

              crsctl start crs

              crsctl check crs

               

              If it is starting ok (all 4 "is online" this will take some time) then change to Oracle user (or user that you have used as oracle owner) and try to start database in this same node:

              srvctl status database -d <your_db_name>

              srvctl start database -d <your_db_name> -i <your_db_local_instance_name>

              example:

              srvctl start database -d testdb -i testdb1

               

              srvctl status database -d <your_db_name>

               

              If you get errors in any step report them over here.

              If you don't get errors then try same on other node.

               

              Get also list of all resource information after you have tried starting everything with this (so you know which services wont start and which will start:

              crs_stat -t

              • 4. Re: RAC database
                892776

                Hi  

                [grid@node1 ~]$ crsctl check crs

                CRS-4638: Oracle High Availability Services is online

                CRS-4537: Cluster Ready Services is online

                CRS-4529: Cluster Synchronization Services is online

                CRS-4533: Event Manager is online

                 

                [grid@node1 ~]$ crsctl start crs

                CRS-4563: Insufficient user privileges.

                 

                CRS-4000: Command Start failed, or completed with errors.

                • 5. Re: RAC database
                  892776

                  [grid@node1 ~]$ crs_stat -t

                  Name           Type           Target    State     Host

                  ------------------------------------------------------------

                  ora.CRS.dg     ora....up.type ONLINE    ONLINE    node1

                  ora.ETAX.dg    ora....up.type ONLINE    ONLINE    node1

                  ora.FRA.dg     ora....up.type ONLINE    ONLINE    node1

                  ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1

                  ora....N1.lsnr ora....er.type ONLINE    ONLINE    node2

                  ora.asm        ora.asm.type   ONLINE    ONLINE    node1

                  ora.eons       ora.eons.type  ONLINE    ONLINE    node1

                  ora.gsd        ora.gsd.type   ONLINE    ONLINE    node1

                  ora....network ora....rk.type ONLINE    ONLINE    node1

                  ora....SM1.asm application    ONLINE    ONLINE    node1

                  ora....E1.lsnr application    ONLINE    ONLINE    node1

                  ora.node1.gsd  application    ONLINE    ONLINE    node1

                  ora.node1.ons  application    ONLINE    ONLINE    node1

                  ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1

                  ora....SM2.asm application    ONLINE    ONLINE    node2

                  ora....E2.lsnr application    ONLINE    ONLINE    node2

                  ora.node2.gsd  application    ONLINE    ONLINE    node2

                  ora.node2.ons  application    ONLINE    ONLINE    node2

                  ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2

                  ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    node2

                  ora.ons        ora.ons.type   ONLINE    ONLINE    node1

                  ora.racdb.db   ora....se.type ONLINE    OFFLINE

                  ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node2

                  • 6. Re: RAC database
                    892776

                    [root@node1 bin]# /u01/app/11.2.0/grid/bin/srvctl start database -d  racdb

                    PRCR-1079 : Failed to start resource ora.racdb.db

                    ORA-01034: ORACLE not available

                    ORA-27101: shared memory realm does not exist

                    Linux-x86_64 Error: 2: No such file or directory

                    Process ID: 0

                    Session ID: 0 Serial number: 0

                     

                    ORA-01078: failure in processing system parameters

                    CRS-2674: Start of 'ora.racdb.db' on 'node2' failed

                    ORA-01034: ORACLE not available

                    ORA-27101: shared memory realm does not exist

                    Linux-x86_64 Error: 2: No such file or directory

                    Process ID: 0

                    Session ID: 0 Serial number: 0

                     

                    ORA-01078: failure in processing system parameters

                    ORA-01034: ORACLE not available

                    ORA-27101: shared memory realm does not exist

                    Linux-x86_64 Error: 2: No such file or directory

                    Process ID: 0

                    Session ID: 0 Serial number: 0

                     

                    CRS-2674: Start of 'ora.racdb.db' on 'node1' failed

                    CRS-2632: There are no more servers to try to place resource 'ora.racdb.db' on that would satisfy its placement policy

                    ORA-01034: ORACLE not available

                    ORA-27101: shared memory realm does not exist

                    Linux-x86_64 Error: 2: No such file or directory

                    Process ID: 0

                    Session ID: 0 Serial number: 0

                    • 7. Re: RAC database
                      Simo Kemppinen

                      This means that your clusterware (CRS) is already up and running (everything is online):

                      --

                      [grid@node1 ~]$ crsctl check crs

                      CRS-4638: Oracle High Availability Services is online

                      CRS-4537: Cluster Ready Services is online

                      CRS-4529: Cluster Synchronization Services is online

                      CRS-4533: Event Manager is online

                      --

                       

                      But this error means that you are missing ENV settings or you are trying to run command as wrong user

                      (usually this needs to be run as root and there needs to be Grid Home path in your PATH ENV or go to grid home bin dir and run it from there  ).):

                      --

                      [grid@node1 ~]$ crsctl start crs

                      CRS-4563: Insufficient user privileges.

                       

                      CRS-4000: Command Start failed, or completed with errors.

                      --

                       

                      So probably the CRS is up and running after the server boot.

                       

                      And

                       

                      This means that only thing is not up is database:

                      --

                      [grid@node1 ~]$ crs_stat -t

                      Name           Type           Target    State     Host

                      ------------------------------------------------------------

                      ora.CRS.dg     ora....up.type ONLINE    ONLINE    node1

                      ora.ETAX.dg    ora....up.type ONLINE    ONLINE    node1

                      ora.FRA.dg     ora....up.type ONLINE    ONLINE    node1

                      ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1

                      ora....N1.lsnr ora....er.type ONLINE    ONLINE    node2

                      ora.asm        ora.asm.type   ONLINE    ONLINE    node1

                      ora.eons       ora.eons.type  ONLINE    ONLINE    node1

                      ora.gsd        ora.gsd.type   ONLINE    ONLINE    node1

                      ora....network ora....rk.type ONLINE    ONLINE    node1

                      ora....SM1.asm application    ONLINE    ONLINE    node1

                      ora....E1.lsnr application    ONLINE    ONLINE    node1

                      ora.node1.gsd  application    ONLINE    ONLINE    node1

                      ora.node1.ons  application    ONLINE    ONLINE    node1

                      ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1

                      ora....SM2.asm application    ONLINE    ONLINE    node2

                      ora....E2.lsnr application    ONLINE    ONLINE    node2

                      ora.node2.gsd  application    ONLINE    ONLINE    node2

                      ora.node2.ons  application    ONLINE    ONLINE    node2

                      ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2

                      ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    node2

                      ora.ons        ora.ons.type   ONLINE    ONLINE    node1

                      ora.racdb.db   ora....se.type ONLINE    OFFLINE

                      ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node2

                      --

                       

                      srvctl command to start database is usually run as oracle user (user that owns oracle binaries) not root.

                      (and srvctl command can be find from Grid Home (Grid Infra (clusterware) home path) and ORACLE_HOME (database home path) usually you run it from ORACLE_HOME as oracle user.)

                      (also if you have your ORACLE_HOME path in your PATH ENV then you don't need full path in command just command and it's options.)

                       

                      So run this as oracle user (or change path (/u01/app/11.2.0/grid/bin/) to ORACLE_HOME path:

                      /u01/app/11.2.0/grid/bin/srvctl stop database -d  racdb

                      # This will try to start only first node instance:

                      /u01/app/11.2.0/grid/bin/srvctl start database -d  racdb -i racdb1

                       

                      /u01/app/11.2.0/grid/bin/srvctl status database -d  racdb

                       

                      # This will try to start only second node instance:

                      /u01/app/11.2.0/grid/bin/srvctl start database -d  racdb -i racdb2

                      • 8. Re: RAC database
                        BPeaslandDBA

                        Have you looked in the Alert Log to see why the instances are not starting? Any relevant error messages?

                         

                        Cheers,
                        Brian

                        • 9. Re: RAC database
                          r_g_financials_dba

                          how about running the following commands

                           

                          ps -ef|grep smon

                           

                          if the asm is up

                          try to connect to asm with the following commands

                           

                          first source the correct environment of asm

                          sqlplus / as sysasm

                          select name,path,state from v$asm_disk;

                          make sure that all the disks are online in the asm

                           

                          next source the database env

                          sqlplus / as sysdba

                          select count(*) from dba_objects;

                           

                          at any stage in the above if you find an issue...you will have to resolve it.

                          • 10. Re: RAC database
                            r_g_financials_dba

                            I think you have version 11 of the grid control

                            How about using

                            crsctl stat res -t

                            • 11. Re: RAC database
                              892776

                              THNAKS ALL.

                               

                              My error resolved via this configuration "permission"

                              $cd /u01/app/11.2.0/grid/bin

                              $chmod 6755 oracle

                              Thank you all