10 Replies Latest reply on May 17, 2014 8:20 AM by Jyoti Verma -Oracle

    The asm instance cannot start on the nodes not running the ASM installation.

    liaison1

      This is 10.2.01 ASM install over the CRS 10.2.0.5 of Solaris 10 Sparc 64bit.  For 3 node RAC

       

      Got the error message when run the runInstaller to install ASM home on the runInstaller with a pop up window:
       
      PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
       
      PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [CRS-0215: Could not start resource 'ora.lsssdb40p.ASM3.asm'.]]
       
      I then, let it continue and run the root.sh
       
      After that, I try to start the asm with the command:
      srvctl start asm -n lsssdb39p
       
      Got the same error message as above.
      PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
       
      PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [PRKS-1009 : Failed to start ASM instance "+ASM3" on node "lsssdb40p", [CRS-0215: Could not start resource 'ora.lsssdb40p.ASM3.asm'.]]

      Let me know what to do to fix the issue. Thanks.

        • 2. Re: The asm instance cannot start on the nodes not running the ASM installation.
          liaison1

          Only one instance up for ASM, which is the one where the install started.   There is no error from the alert log file of the ASM.     But here is the error from the ASM instances that cannot be started:

           

          Cluster communication is configured to use the following interface(s) for this instance

            10.144.2.202

          Thu May 15 18:51:45 2014

          cluster interconnect IPC version:Oracle UDP/IP

          IPC Vendor 1 proto 2

          Process PMON died, see its trace file

          USER: terminating instance due to error 443

          Instance terminated by USER, pid = 19239

           

          Here is the message from the trace file:

          /oracle/admin/+ASM/udump/+asm2_ora_19239.trc

          Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production

          With the Partitioning, Real Application Clusters, OLAP and Data Mining options

          ORACLE_HOME = /oracle/product/asm

          System name:    SunOS

          Node name:      lsssdb39p.liaison.prod

          Release:        5.10

          Version:        Generic_150400-10

          Machine:        sun4v

          Instance name: +ASM2

          Redo thread mounted by this instance: 0 <none>

          Oracle process number: 0

          Unix process pid: 19239, image: oracle@lsssdb39p.liaison.prod

           

          Number of resource hash buckets is 525

          2014-05-15 18:51:45.080: [ default]prom_init:1:ould not initialize clsc 2

          2014-05-15 18:51:45.081: [  OCRAPI]a_init:11!: Messaging init unsuccessful : [24]

          Number of resource hash buckets is 525

          * kjfcnfy: kjinumbuckets = 512

          KCCDEBUG_LEVEL = 0

          *** 2014-05-15 18:51:46.223

          Process PMON is dead (pid=19241, state=3):

          *** 2014-05-15 18:51:46.223

          KSTDUMP: In-memory trace dump

          TIME:SEQ#        ORAPID   SID EVENT  OP DATA

          ========================================================================

          41DCB9F1:00000003    39     0 10254  34 KSBS1P: process USER trying to start background PMON

          41EBE948:00000008    39     0 10254  51 KSBS1P: out of loop: process did not start

          KSTDUMP: End of in-memory trace dump

          *** 2014-05-15 18:51:46.223

          • 3. Re: The asm instance cannot start on the nodes not running the ASM installation.
            Salman Qureshi

            Hi,

            On failing node, can you set oracle home to ASM home and oracle_sid to ASM and try starting ASM from sqlplus and let us know what is the result?

             

            Salman

            • 4. Re: The asm instance cannot start on the nodes not running the ASM installation.
              liaison1

              It looks that I can start the asm by using sqlplus 
               
              Connected to an idle instance.
               
              SQL> startup
              ASM instance started
               
              Total System Global Area 130023424 bytes
              Fixed Size 2075224 bytes
              Variable Size 102782376 bytes
              ASM Cache 25165824 bytes
              ASM diskgroups mounted
              SQL> exit



              And I can stop the ASM by srvctl. But I cannot start the ASM by srvctl:
               
              lsssdb39p|+ASM2|/oracle/admin/+ASM/udump $ srvctl start asm -n lsssdb39p
              PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "lsssdb39p", [lsssdb39p:ora.lsssdb39p.ASM2.asm:
              lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Fri May 16 00:03:53 2014
              lsssdb39p:ora.lsssdb39p.ASM2.asm:
              lsssdb39p:ora.lsssdb39p.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
              lsssdb39p:ora.lsssdb39p.ASM2.asm:
              lsssdb39p:ora.lsssdb39p.ASM2.asm:Enter user-name: Connected to an idle instance.
              lsssdb39p:ora.lsssdb39p.ASM2.asm:
              lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL> ORA-00443: background process "PMON" did not start
              lsssdb39p:ora.lsssdb39p.ASM2.asm:SQL> Disconnected
              lsssdb39p:ora.lsssdb39p.ASM2.asm:
              CRS-0215: Could not start resource 'ora.lsssdb39p.ASM2.asm'.]]
               
              This is the same message from alert log file:
              Cluster communication is configured to use the following interface(s) for this instance
                 10.144.2.202
              Fri May 16 00:03:53 2014
              cluster interconnect IPC version:Oracle UDP/IP
              IPC Vendor 1 proto 2
              Process PMON died, see its trace file
              USER: terminating instance due to error 443
              Instance terminated by USER, pid = 13863

              • 5. Re: The asm instance cannot start on the nodes not running the ASM installation.
                Billy~Verreynne

                Seems like you are running 10.2.0.1 - if so, consider applying the latest patchset for 10.2 as that could very well resolve the problem.

                 

                What does the PMON trace file say?

                 

                The first thing that comes to my mind is some kind of environment issue that is resulting in this behaviour. Your manually starting ASM results in ASM being forked/executed from your (interactive/tty) shell process - and inheriting the environment of that shell process.

                 

                You starting it via server control means that ASM is forked/executed from the background srvctl process (itself a child process of a CRS process) - and this time around the ASM inherits the srcvtl process's environment. Something could be problem in this environment (ulimit, security, variable, etc) and this then could result in PMON's failure.

                 

                Besides the environment issue (i.e. what exactly the ASM process inherits from the parent process creating it), there are not much differences between the two startup attempts.

                • 6. Re: The asm instance cannot start on the nodes not running the ASM installation.
                  liaison1

                  Ok, maybe I will need to apply the upgrade patch set to make it to 10.2.0.5 from 10.2.0.1.    But I still do not know why the issue occurred.

                  • 7. Re: The asm instance cannot start on the nodes not running the ASM installation.
                    liaison1

                    Great news.   I applied the 10.2.0.5 patch set.   The issue seems disappeared.   Everything looks good now.   But still do not know why the issue happened.    Thanks!

                    • 8. Re: The asm instance cannot start on the nodes not running the ASM installation.
                      liaison1

                      I would like to apply some later version CPU or PSU on top of 10.2.0.5.  What is the patch # for it?   Thanks!

                      • 9. Re: The asm instance cannot start on the nodes not running the ASM installation.
                        liaison1

                        I find the patches I need.  The patch 16619894 or 14727319.   However, I cannot download it from the metalink as the download button is grey.    Anybody know how to download the patches (for 10.2.0.5.xx).?    Thanks!

                        • 10. Re: The asm instance cannot start on the nodes not running the ASM installation.
                          Jyoti Verma -Oracle

                          10.2.0.5.x is now will need extended support now. Please check: Release Schedule of Current Database Releases(Doc ID 742060.1)