10 Replies Latest reply: May 17, 2014 3:20 AM by Jyoti Verma -Oracle RSS

    The asm instance cannot start on the nodes not running the ASM installation.

    liaison1

      This is 10.2.01 ASM install over the CRS 10.2.0.5 of Solaris 10 Sparc 64bit.  For 3 node RAC

       

      Got the error message when run the runInstaller to install ASM home on the runInstaller with a pop up window:
       
      PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
       
      PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [CRS-0215: Could not start resource 'ora.lsssdb40p.ASM3.asm'.]]
       
      I then, let it continue and run the root.sh
       
      After that, I try to start the asm with the command:
      srvctl start asm -n lsssdb39p
       
      Got the same error message as above.
      PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
       
      PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [CRS-0215: Could not start resource 'ora.lsssdb40p.ASM3.asm'.]]

      Let me know what to do to fix the issue. Thanks.

        • 1. Re: The asm instance cannot start on the nodes not running the ASM installation.
          oscrub

          What does it say in the log files?

          • 2. Re: The asm instance cannot start on the nodes not running the ASM installation.
            liaison1

            Only one instance up for ASM, which is the one where the install started.   There is no error from the alert log file of the ASM.     But here is the error from the ASM instances that cannot be started:

             

            Cluster communication is configured to use the following interface(s) for this instance

              10.144.2.202

            Thu May 15 18:51:45 2014

            cluster interconnect IPC version:Oracle UDP/IP

            IPC Vendor 1 proto 2

            Process PMON died, see its trace file

            USER: terminating instance due to error 443

            Instance terminated by USER, pid = 19239

             

            Here is the message from the trace file:

            /oracle/admin/+ASM/udump/+asm2_ora_19239.trc

            Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production

            With the Partitioning, Real Application Clusters, OLAP and Data Mining options

            ORACLE_HOME = /oracle/product/asm

            System name:    SunOS

            Node name:      lsssdb39p.liaison.prod

            Release:        5.10

            Version:        Generic_150400-10

            Machine:        sun4v

            Instance name: +ASM2

            Redo thread mounted by this instance: 0 <none>

            Oracle process number: 0

            Unix process pid: 19239, image: oracle@lsssdb39p.liaison.prod

             

            Number of resource hash buckets is 525

            2014-05-15 18:51:45.080: [ default]prom_init:1:ould not initialize clsc 2

            2014-05-15 18:51:45.081: [  OCRAPI]a_init:11!: Messaging init unsuccessful : [24]

            Number of resource hash buckets is 525

            * kjfcnfy: kjinumbuckets = 512

            KCCDEBUG_LEVEL = 0

            *** 2014-05-15 18:51:46.223

            Process PMON is dead (pid=19241, state=3):

            *** 2014-05-15 18:51:46.223

            KSTDUMP: In-memory trace dump

            TIME:SEQ#        ORAPID   SID EVENT  OP DATA

            ========================================================================

            41DCB9F1:00000003    39     0 10254  34 KSBS1P: process USER trying to start background PMON

            41EBE948:00000008    39     0 10254  51 KSBS1P: out of loop: process did not start

            KSTDUMP: End of in-memory trace dump

            *** 2014-05-15 18:51:46.223

            • 3. Re: The asm instance cannot start on the nodes not running the ASM installation.
              Salman Qureshi

              Hi,

              On failing node, can you set oracle home to ASM home and oracle_sid to ASM and try starting ASM from sqlplus and let us know what is the result?

               

              Salman

              • 4. Re: The asm instance cannot start on the nodes not running the ASM installation.
                liaison1

                It looks that I can start the asm by using sqlplus 
                 
                Connected to an idle instance.
                 
                SQL> startup
                ASM instance started
                 
                Total System Global Area 130023424 bytes
                Fixed Size 2075224 bytes
                Variable Size 102782376 bytes
                ASM Cache 25165824 bytes
                ASM diskgroups mounted
                SQL> exit



                And I can stop the ASM by srvctl. But I cannot start the ASM by srvctl:
                 
                lsssdb39p|+ASM2|/oracle/admin/+ASM/udump $ srvctl start asm -n lsssdb39p
                PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [lsssdb39p:ora.lsssdb39p.ASM2.asm:
                lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Fri May 16 00:03:53 2014
                lsssdb39p:ora.lsssdb39p.ASM2.asm:
                lsssdb39p:ora.lsssdb39p.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
                lsssdb39p:ora.lsssdb39p.ASM2.asm:
                lsssdb39p:ora.lsssdb39p.ASM2.asm:Enter user-name: Connected to an idle instance.
                lsssdb39p:ora.lsssdb39p.ASM2.asm:
                lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL> ORA-00443: background process "PMON" did not start
                lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL> Disconnected
                lsssdb39p:ora.lsssdb39p.ASM2.asm:
                CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
                 
                This is the same message from alert log file:
                Cluster communication is configured to use the following interface(s) for this instance
                   10.144.2.202
                Fri May 16 00:03:53 2014
                cluster interconnect IPC version:Oracle UDP/IP
                IPC Vendor 1 proto 2
                Process PMON died, see its trace file
                USER: terminating instance due to error 443
                Instance terminated by USER, pid = 13863

                • 5. Re: The asm instance cannot start on the nodes not running the ASM installation.
                  Billy~Verreynne

                  Seems like you are running 10.2.0.1 - if so, consider applying the latest patchset for 10.2 as that could very well resolve the problem.

                   

                  What does the PMON trace file say?

                   

                  The first thing that comes to my mind is some kind of environment issue that is resulting in this behaviour. Your manually starting ASM results in ASM being forked/executed from your (interactive/tty) shell process - and inheriting the environment of that shell process.

                   

                  You starting it via server control means that ASM is forked/executed from the background srvctl process (itself a child process of a CRS process) - and this time around the ASM inherits the srcvtl process's environment. Something could be problem in this environment (ulimit, security, variable, etc) and this then could result in PMON's failure.

                   

                  Besides the environment issue (i.e. what exactly the ASM process inherits from the parent process creating it), there are not much differences between the two startup attempts.

                  • 6. Re: The asm instance cannot start on the nodes not running the ASM installation.
                    liaison1

                    Ok, maybe I will need to apply the upgrade patch set to make it to 10.2.0.5 from 10.2.0.1.    But I still do not know why the issue occurred.

                    • 7. Re: The asm instance cannot start on the nodes not running the ASM installation.
                      liaison1

                      Great news.   I applied the 10.2.0.5 patch set.   The issue seems disappeared.   Everything looks good now.   But still do not know why the issue happened.    Thanks!

                      • 8. Re: The asm instance cannot start on the nodes not running the ASM installation.
                        liaison1

                        I would like to apply some later version CPU or PSU on top of 10.2.0.5.  What is the patch # for it?   Thanks!

                        • 9. Re: The asm instance cannot start on the nodes not running the ASM installation.
                          liaison1

                          I find the patches I need.  The patch 16619894 or 14727319.   However, I cannot download it from the metalink as the download button is grey.    Anybody know how to download the patches (for 10.2.0.5.xx).?    Thanks!

                          • 10. Re: The asm instance cannot start on the nodes not running the ASM installation.
                            Jyoti Verma -Oracle

                            10.2.0.5.x is now will need extended support now. Please check: Release Schedule of Current Database Releases(Doc ID 742060.1)