1 2 Previous Next 17 Replies Latest reply: Feb 14, 2013 11:39 AM by Zoltan Kecskemethy RSS

    The db cannot startup after host crash

    hao
      the db cannot startup after host crash.



      please see the following logs. the strange thing is "lkinstrdmetdev" is the wrong name(sid is rdmetdev), and the file does not exist in the FS too. the correct file name should be "lkrdmetdev" and i've deleted this file from the directory. I did not do any changes. and no any oralce processes running while i started up the oracle instance. Please help



      Thu Feb 14 09:25:20 EST 2013
      sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
      Thu Feb 14 09:25:20 EST 2013
      sculkget: lock held by PID: 272600512
      Thu Feb 14 09:25:20 EST 2013
      Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
      Thu Feb 14 09:25:20 EST 2013
      Failed to acquire instance startup/shutdown serialization primitive
      Thu Feb 14 09:35:33 EST 2013
      sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
      Thu Feb 14 09:35:33 EST 2013
      sculkget: lock held by PID: 125960160
      Thu Feb 14 09:35:33 EST 2013
      Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
      Thu Feb 14 09:35:33 EST 2013
      Failed to acquire instance startup/shutdown serialization primitive


      Thu Feb 14 09:34:57 EST 2013
      Errors in file /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc:
      ORA-27050: function called with invalid FIB/IOV structure
      Additional information: 2
      ORA-27086: unable to lock file - already in use
      Linux-x86_64 Error: 37: No locks available
      Additional information: 10

      rdmetdev_ora_6321.trc
      Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
      With the Partitioning, OLAP, Data Mining and Real Application Testing options
      ORACLE_HOME = /u01/app/oracle/product/10.2
      System name: Linux
      Node name: ecamolx1813
      Release: 2.6.18-194.el5
      Version: #1 SMP Tue Mar 16 21:52:39 EDT 2010
      Machine: x86_64
      Instance name: rdmetdev
      Redo thread mounted by this instance: 0 <none>
      Oracle process number: 0
      Unix process pid: 6321, image: oracle@ecamolx1813

      *** 2013-02-14 09:34:57.630
      ORA-27050: function called with invalid FIB/IOV structure
      Additional information: 2
      ORA-27086: unable to lock file - already in use
      Linux-x86_64 Error: 37: No locks available
      Additional information: 10
      *** 2013-02-14 09:35:33.039
      Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
      *** 2013-02-14 09:35:33.051
      Failed to acquire instance startup/shutdown serialization primitive
        • 1. Re: The db cannot startup after host crash
          sb92075
          hao wrote:
          the db cannot startup after host crash.



          please see the following logs. the strange thing is "lkinstrdmetdev" is the wrong name(sid is rdmetdev), and the file does not exist in the FS too. the correct file name should be "lkrdmetdev" and i've deleted this file from the directory. I did not do any changes. and no any oralce processes running while i started up the oracle instance. Please help



          Thu Feb 14 09:25:20 EST 2013
          sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
          Thu Feb 14 09:25:20 EST 2013
          sculkget: lock held by PID: 272600512
          Thu Feb 14 09:25:20 EST 2013
          Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
          Thu Feb 14 09:25:20 EST 2013
          Failed to acquire instance startup/shutdown serialization primitive
          Thu Feb 14 09:35:33 EST 2013
          sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
          Thu Feb 14 09:35:33 EST 2013
          sculkget: lock held by PID: 125960160
          Thu Feb 14 09:35:33 EST 2013
          Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
          Thu Feb 14 09:35:33 EST 2013
          Failed to acquire instance startup/shutdown serialization primitive


          Thu Feb 14 09:34:57 EST 2013
          Errors in file /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc:
          ORA-27050: function called with invalid FIB/IOV structure
          Additional information: 2
          ORA-27086: unable to lock file - already in use
          Linux-x86_64 Error: 37: No locks available
          Additional information: 10
          OS problem
          Oracle is victim; not culprit.
          Root cause & fix are external to Oracle.
          • 2. Re: The db cannot startup after host crash
            Pavan Kumar
            Hi,
            the strange thing is "lkinstrdmetdev" is the wrong name(sid is rdmetdev), 
            lkinst rdmetdev -- stands for "lock instance"

            Thu Feb 14 09:34:57 EST 2013
            Errors in file /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc:
            ORA-27050: function called with invalid FIB/IOV structure
            Additional information: 2
            ORA-27086: unable to lock file - already in use
            Linux-x86_64 Error: 37: No locks available
            Additional information: 10
            Before starting to re-attempt once again, make you sure check shared memory segments got released at OS level, sysresv
            Further verify the what is the information in trace file, verify OS kernel parameters once.

            - Pavan Kumar N

            Edited by: Pavan Kumar on Feb 14, 2013 9:24 PM
            • 3. Re: The db cannot startup after host crash
              hao
              The server was rebooted twice after the crash. Still the same error message. I guess the share mem seg should be released after the reboot, right?
              • 4. Re: The db cannot startup after host crash
                vlethakula
                Looks like Bug 5722600 : FREQUENT ORA-27077 IN ALERTLOGS
                Check also,
                http://dbaspot.com/oracle-server/363699-what-main-problem-ora-27050-a.html
                • 5. Re: The db cannot startup after host crash
                  hao
                  It's a standalone server.
                  • 6. Re: The db cannot startup after host crash
                    Pavan Kumar
                    Hi,

                    Can you provide what is their in trace file (Information)

                    - Pavan Kumar N
                    • 7. Re: The db cannot startup after host crash
                      Zoltan Kecskemethy
                      I would remove/disable all automatic startup scripts and do restart server again and issue a clean manual startup.
                      And see errors again; compare the to first ones.

                      Also would do a force file system check where db files are. What file system do you use for the oradata location? internal/external? Double check external disk/fs validity/health.

                      Go through DB software install prerequisite checks (required packages, kernel settings, other os settings)

                      Edited by: Zoltan Kecskemethy on Feb 14, 2013 5:14 PM
                      added fs check

                      Edited by: Zoltan Kecskemethy on Feb 14, 2013 5:17 PM
                      added prereq checks

                      Edited by: Zoltan Kecskemethy on Feb 14, 2013 5:19 PM
                      • 8. Re: The db cannot startup after host crash
                        hao
                        which trace file? I've pasted all the relevant traces in the original post.
                        • 9. Re: The db cannot startup after host crash
                          Pavan Kumar
                          Hi,

                          Errors in file /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc

                          - Pavan Kumar N
                          • 10. Re: The db cannot startup after host crash
                            hao
                            bash-3.2$ more /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc
                            /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc

                            Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
                            With the Partitioning, OLAP, Data Mining and Real Application Testing options
                            ORACLE_HOME = /u01/app/oracle/product/10.2
                            System name: Linux
                            Node name: ecamolx1813
                            Release: 2.6.18-194.el5
                            Version: #1 SMP Tue Mar 16 21:52:39 EDT 2010
                            Machine: x86_64
                            Instance name: rdmetdev
                            Redo thread mounted by this instance: 0 <none>
                            Oracle process number: 0
                            Unix process pid: 6321, image: oracle@ecamolx1813

                            *** 2013-02-14 09:34:57.630
                            ORA-27050: function called with invalid FIB/IOV structure
                            Additional information: 2
                            ORA-27086: unable to lock file - already in use
                            Linux-x86_64 Error: 37: No locks available
                            Additional information: 10
                            *** 2013-02-14 09:35:33.039
                            Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
                            *** 2013-02-14 09:35:33.051
                            Failed to acquire instance startup/shutdown serialization primitive


                            ----end
                            • 11. Re: The db cannot startup after host crash
                              Zoltan Kecskemethy
                              Stinks like a file system OR permission issue to me! Did you check db file ownership? All files: db, log, control, spfile, pwd etc
                              • 12. Re: The db cannot startup after host crash
                                Osama_Mustafa
                                Refer to
                                Archiving Redo Logs Fails With ORA-19510, ORA-27086 Linux-x86_64 Error: 37: No locks available [ID 1418233.1]
                                • 13. Re: The db cannot startup after host crash
                                  hao
                                  I've checked, they all have correct ownership setup. I have the full backup, do you think if it is good to do the restore before finding out the real root cause? and I am doubting if the db restore could bring up the db or not
                                  • 14. Re: The db cannot startup after host crash
                                    Osama_Mustafa
                                    Check the MOS document that I posted.
                                    1 2 Previous Next