4 Replies Latest reply: Aug 6, 2013 1:43 AM by Abufazal RSS

    Unable to start all 3 instances in RAC

    User395622

      Hello All,

       

      I got the following error while trying to restart a 3 node RAC database. OEM reported two out of three nodes to be down.

       

      We also noticed that the cluster_database parameter was set to "False" and the cluster_database_instances integer was set to 1.We changed the parameters to 'True' and '3' respectively and while trying to restart the database we get the error message as shown below. Any suggestions/help to resolve this would be appreciated.

       


      srvctl start database -d abc123

      PRCR-1079 : Failed to start resource ora.abc123.db

      CRS-5017: The resource action "ora.abc123.db start" encountered the following error:

      ORA-01102: cannot mount database in EXCLUSIVE mode

      . For details refer to "(:CLSN00107:)" in "/oradba/app/grid/11.2.0.3_a/log/abcde104/agent/crsd/oraagent_oradba/oraagent_oradba.log".

       

      CRS-2674: Start of 'ora.abc123.db' on 'abcde104' failed

      CRS-2632: There are no more servers to try to place resource 'ora.abc123.db' on that would satisfy its placement policy

      CRS-5017: The resource action "ora.abc123.db start" encountered the following error:

      ORA-01102: cannot mount database in EXCLUSIVE mode

      . For details refer to "(:CLSN00107:)" in "/oradba/app/grid/11.2.0.3_a/log/abcde103/agent/crsd/oraagent_oradba/oraagent_oradba.log".

       

      CRS-2674: Start of 'ora.abc123.db' on 'abcde103' failed

        • 1. Re: Unable to start all 3 instances in RAC
          user8363520

          Post the output of the below log, also what OS are you running? Also, is your cluster resource(crs,asm,scan,vip etc etc) are they up?

           

          For details refer to "(:CLSN00107:)" in "/oradba/app/grid/11.2.0.3_a/log/abcde104/agent/crsd/oraagent_oradba/oraagent_oradba.log".

          • 2. Re: Unable to start all 3 instances in RAC
            srsatya

            Hi,

             

              If my guess is correct, you have not changed cluster init parameters properly. Please follow below procedure to change cluster init parameters.

             

            1. Login to the database and run "create pfile from spfile" , that will generate pfile in dbs folder on the server where you executed the command.
            2. Shutdown all cluster database instance.
            3. From the node where you create pfile, open the pfile and edit cluster parameters
            4. startup to nomount  with new modified pfile
            5. Run create spfile <existing path and file name> from pfile='Give new pfile path & file name';
            6. shutdown the database
            7. run srvctl to start all database.
            • 3. Re: Unable to start all 3 instances in RAC
              User395622

              Hi srsatya, thanks for your reply!

               

              I used the alter system set cluster_database=true scope=spfile SID='*' command to change my parameter from node1.After that the restart failed. The DB is now running from node 3.

               

               

              My spfile is in a shared location.

               

              SQL> show parameter spfile;

               

              NAME                                 TYPE        VALUE

              ------------------------------------ ----------- ------------------------------

              spfile                               string      +DATA/abc123/spfileabc123.ora

                                                             

               

              1.Login to the database and run "create pfile from spfile" , that will generate pfile in dbs folder on the server where you executed the command.

               

              2.Shutdown all cluster database instance.Can I do this using SQLPLUS individually on each node? Or Srvctl?

              3.From the node where you create pfile, open the pfile and edit cluster parameters

              4.startup to nomount  with new modified pfile

              5.Run create spfile <+DATA/abc123/spfileabc123.ora> from pfile='Give new pfile path & file name';

               

              My current init.ora(pfile) file points to the shared location specified above.This will overwrite my spfile with the new cluster parameters. After that can I remove the new pfile that was created?

               

              6.shutdown the database

              7.run srvctl to start all database.

               

                .

              • 4. Re: Unable to start all 3 instances in RAC
                Abufazal

                Hi srsatya, thanks for your reply!

                 

                I used the alter system set cluster_database=true scope=spfile SID='*' command to change my parameter from node1.After that the restart failed. The DB is now running from node 3.

                 

                 

                My spfile is in a shared location.

                 

                SQL> show parameter spfile;

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                spfile                               string      +DATA/abc123/spfileabc123.ora

                 

                As per this output, cluster_database is already set to TRUE.

                 

                I would suggest to shutdown the running instance (node 3) and then start the database using srvctl start database -d  abc123