10 Replies Latest reply: Jul 29, 2012 6:44 PM by sb92075 RSS

    ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!

    FredC
      My 10.2 database crashed, and when it came back I got the following error:

      ORA-00202: control file: '/local/opt/oracle/product/10.2.0/dbs/lkFOOBAR'
      ORA-27086: unable to lock file - already in use
      Linux-x86_64 Error: 11: Resource temporarily unavailable

      This is a classic symptom of a Netapp problem, which likes to hold file locks open on NFS mounts. There is a standard procedure for clearing those locks; see, for instance, document 429912.1 on Metalink.

      Unfortunately, my files are mounted on an Isilon, one of Netapp's twisted cousins. I can find no references to "isilon" on Metalink, and we are at a loss how to resolve this.

      My sysadmin assures me that "there are no locks on the Isilon". But I know this cannot be the case, because if I do the following:

      1. delete the lockfile /local/opt/oracle/product/10.2.0/dbs/lkFOOBAR, and then
      2. move my controlfiles aside, and then copy them back into place,

      then the database will mount. However, it will not open, because now all the datafiles have locks.

      Is there anyone with experience in clearing NFS locks? I know this is more of a SA task than DBA, but I am sure my SA has overlooked something.

      Thanks
        • 1. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
          FredC
          New information:

          As stated above, I moved the controlfiles aside and then copied them back into place, like this:
          mv control01.ctl control01-bak.ctl
          cp control01-bak.ctl control01.ctl
          Did that for each controlfile, and then the database mounted.

          But, after rebooting the machine, we discovered that all locks were back in place-- it looks like the system is locking the files on boot-up, and not letting them go. The lock is held by PID 1, which is init.
          sculkget: lock held by PID: 1
          This is definitely looking like a major system issue, and not a DBA issue, and hence I have little right to expect assistance in this forum. But nonetheless I lay my situation out here before you all in case someone else recognizes my problem before the server bursts into flames!

          The system is CentOS 4.5-- not my choice, but that's the way it is.
          • 2. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
            247514
            ORA-00202: control file: '/local/opt/oracle/product/10.2.0/dbs/lkFOOBAR'
            lk<SID> is a placeholder file while instance is running. If your database is down while you still having this file that means your instance processes were killed instead of proper shutdown.

            check your alert.log see if there's any error messages.

            What's the mount option you had on the NFS mount point?

            BTW, is your ORACLE_HOME /local/opt/oracle/product/10.2.0/ on a NFS mount point?
            • 3. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
              FredC
              Yingkuan,

              We can bypass the lock on the lk<SID> file simply by deleting that file. Then, upon restarting the instance, we encounter locks on the controlfiles. By performing the copying technique I describe above, we can get rid of the locks on the controlfiles and mount the instance. But at that point, we discover there are locks on all the datafiles.
              ORACLE_HOME /local/opt/oracle/product/10.2.0/ on a NFS mount point?
              Yes, it is.

              New information:

              Because all oracle files (including ORACLE_HOME) are on NFS mounts, we unmounted these directories and mounted them on a new host. When we attempted to start the database, the lock errors persisted. Thus we know that the locks are on the level of the Isilon storage device rather than the DB host itself.

              Thanks,
              • 4. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                26741
                Even if it is a proprietary NFS server, it most likely (certainly ?) has locking.

                Wonder if lsilon is used in RAC ? How would it handle RAC ?

                You (or your Storage Administrator) should go back to the vendor and ask them
                about locking. Also ask them about Oracle implementations.
                • 5. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                  FredC
                  I thought of RAC in the midst of this, and said to myself, "I'm sure glad we're not on RAC".

                  We opened a ticket with the vendor. They recommended we restart nfsd and lockd on the Isilon and use a different NFS node, which we did, and that solved our problem. We are not satisfied that we have resolved the issue in the long term, but at least our DB is back up, and we can go to sleep knowing it was not the DB's fault.
                  • 6. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                    247514
                    Well looks like you temporary fixed the problem. but there are few things you might want to take a note to look for.

                    1. the mount options of your NFS mount point. Make sure it's follow the Oracle metalink doc recommendation.

                    2. make sure you umount and release the NFS from old node before mount on new host.

                    3. there should be similar release lock command in Isilon as Netapp. Check with Isilon support.
                    • 7. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                      833410
                      You can check the NFS locks in Isilon using below command,

                      isilon1-1#isi nfs locks list
                      svid@Hostname Lock Type Range Path
                      802@myhost exclusive [1, 18446744073709551615] /ifs/data/mydata1/database/pracle/oracle.changes.record


                      This will list the current NFS locks.

                      isilon1-1# isi nfs locks waiters
                      No waiters found.

                      This will list the waiting locks if any.


                      To release the locks do the following

                      isilon1-1# isi nfs clients remove myhost

                      Client myhost removed


                      isilon1-1#isi nfs locks list
                      No locks found



                      Let me know how it goes.
                      • 8. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                        847447
                        Restart the Nfs Server
                        /etc/init.d/nfs restart
                        All lock will be freed.
                        • 9. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                          Pranilesh Chand.
                          Try to restart the 'nfslock' service as well. It should resolve your issue.
                          http://pranilesh-chand.blogspot.co.nz/2012/07/nfs-mount-casues-ora-27086-unable-to.html
                          • 10. Re: ORA-27086: unable to lock file over NFS -- but it's NOT Netapp!
                            sb92075
                            >

                            Why do you resurrect a FOUR Year old thread?