4 Replies Latest reply: Jun 3, 2014 6:43 PM by Cindys-Oracle RSS

    How to prevent “unexpected free inode" message?

    1410717

      Hello All,

      We have some issue with Solaris10 server and FC disk array. From time to time “unexpected free inode” error occurs on disks from disk array.

      “unexpected free inode 7057804, run fsck(1M) -o f”

      Solaris kernel crashes due to "unexpected free inode".

      The configuration is:

      Sun Fire E4900 with Solaris10u10 and SAN SE6140

      Is any solutions to prevent appearing such kind of errors?

      Is it depends of Solaris kernel version, server’s FW version, DA FW version, etc?

      Yes I know that “fsck” will fix this issue, but we need a solution of how to prevent this in the future.

        • 1. Re: How to prevent “unexpected free inode" message?
          Walter-Oracle

          Hi,

           

          you might want to take a look on

           

          Troubleshooting the Cause of Solaris File System Corruption and Preventing Future Corruption (Doc ID 1009218.1)

           

          and as well installing  the latest recommended Patchset /  Firmware.

           

          regards


          Walter

          • 2. Re: How to prevent “unexpected free inode" message?
            Cindys-Oracle

            Does FMA report any ongoing issues with the SAN devices?

             

            # fmadm faulty

            # fmdump

             

            Are device errors accumulating:

             

            # iostat -en

             

            I agree with keeping the firmware up-to-date.

             

            Thanks, Cindy

            • 3. Re: How to prevent “unexpected free inode" message?
              1410717

              Thanks guys

              I have only an explorer from the corrupted server.

              All files fmadm-faulty.out are blank.

               

              But in the  file "fmdump-e.out" I found some records:

              Nov 15 07:39:01.2142 ereport.cpu.ultraSPARC-IVplus.wdc

              Nov 16 02:02:02.8883 ereport.cpu.ultraSPARC-IVplus.l3-wdc

              Nov 19 01:11:23.4186 ereport.cpu.ultraSPARC-IVplus.l3-wdc

              Nov 19 01:11:23.4186 ereport.cpu.ultraSPARC-IVplus.l3-wdc

              Dec 12 23:32:42.9544 ereport.cpu.ultraSPARC-IVplus.l3-wdc

              Jan 29 01:52:06.5819 ereport.cpu.ultraSPARC-IVplus.l3-wdc

               

              I don't understand, is it critical CPU errors?

              In the fmdump-eV.out there is detailed report.

              Example:

              Jan 29 2014 01:52:06.581931800 ereport.cpu.ultraSPARC-IVplus.l3-wdc

              nvlist version: 0

                class = ereport.cpu.ultraSPARC-IVplus.l3-wdc

                ena = 0xc9c330b8ee080c01

                detector = (embedded nvlist)

                nvlist version: 0

                version = 0x1

                scheme = cpu

                cpuid = 0x203

                cpumask = 0x24

                serial = 80020230C86C4629

                (end detector)


                afsr = 0xc7

                afsr-ext = 0x2

                afar-status = 0x1

                afar = 0xf6874b90

                pc = 0xfd3a1094

                tl = 0x0

                tt = 0x63

                privileged = 0

                multiple = 0

                syndrome-status = 0x1

                syndrome = 0xc7

                l3-cache-ways = 0x4

                l3-cache-data = 0xec0106f1a6 0x74b80 0x0 0x82ad002194 0xd5002194 0xbfc6644310aa6f80 0xbfd6da2fdb541ac0 0x3fcc883085034c80 0xbfb4d9cafd8e9300 0x257cb 0x3fdc2df7b4489440 0xbfda26b0a323dec0 0x3fb1634ddf1c4900 0xbf5a4f722748c000 0x2911e 0xec0106f1a6 0x874b80 0x1 0xd5002194 0xd5002194 0x0 0x0 0x600547fad00 0x600451088b0 0xc6 0x600407de220 0x60068440b80 0x81b4000100640064 0x65e9e 0x1ca1a 0xec0106f1a6 0x1074b80 0x2 0x806c002194 0x101002194 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xec0106f1a6 0x1874b80 0x3 0x101002194 0x101002194 0x30060527d50 0x30060527d50 0x0 0x60045074b98 0x9b81 0x60045074b98 0x0 0x60045074bb0 0x60045074bb0 0x46bb

                l2-cache-ways = 0x4

                l2-cache-data = 0xec0106f1a6 0x74b80 0x0 0x4036105e24 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xec0106f1a6 0xf4b80 0x1 0x4036205e24 0x0 0x2243544e5f455645 0x4e545f4e4241505f 0x524144494f5f4c49 0x4e4b5f5345545550 0x3f388 0x22206368616e6765 0x54696d653d223532 0x2e30303022207665 0x6e646f7253706563 0x1798e 0xec0106f1a6 0x174b80 0x2 0x4036085e24 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xec0106f1a6 0x1f4b80 0x3 0x4036185e24 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

                dcache-ways = 0x0

                icache-ways = 0x0

                resource = (embedded nvlist)

                nvlist version: 0

                version = 0x0

                scheme = mem

                unum = /N0/SB0/P3/E0 J7400

                (end resource)

               

                __ttl = 0x1

                __tod = 0x52e8a516 0x22af9318

               

              And iostat_-E.out:

              ...

              st3       Soft Errors: 0 Hard Errors: 3 Transport Errors: 0

              Vendor: HP       Product: C7438A           Revision: ZP76 Serial No:    9

              st18      Soft Errors: 0 Hard Errors: 11 Transport Errors: 0

              Vendor: HP       Product: C7438A           Revision: ZP76 Serial No:    9

              • 4. Re: How to prevent “unexpected free inode" message?
                Cindys-Oracle

                I think this issue is unrelated to your unexpected free inode problem.

                 

                What patch level is this?

                 

                The above fma errors look like bug 15465424 fixed in this patch 138052-01.

                The problem described in this bug is that the cpumem-diagnosis module gets turned off

                and you need to get it running again.

                 

                thanks, Cindy