2 Replies Latest reply on Nov 14, 2012 5:46 AM by jdp.unix

    ZFS capped cache still too large.

      On my 256GB T4-2 server, I have capped my ZFS ARC cache to 96GB via /etc/system:
      set zfs:zfs_arc_max = 103079215104
      And this works fine, as seen via kstat:
      # kstat zfs:0:arcstats:size
      module: zfs                             instance: 0
      name:   arcstats                        class:    misc
              size                            100820375552
      However, my "ZFS File Data" within ::memstat shows that I'm using 160GB:
      # echo ::memstat | mdb -k
      Page Summary                Pages                MB  %Tot
      Kernel                    3115863             24342    9%
      ZFS File Data            20369843            159139   62%
      Anon                      3910762             30552   12%
      Exec and libs               18155               141    0%
      Page cache                  20009               156    0%
      Free (cachelist)          1020393              7971    3%
      Free (freelist)           4459032             34836   14%
      Total                    32914057            257141
      Physical                 32896889            257006
      So obviously "ZFS File Data" reports on more than just the ARC Cache.

      My host not under memory stress yet, but will be soon. It's the latest rev of Solaris 10 (08/11) patched as of August (147440-19).

      My questions:
      1. What is the extra 60GB used for?
      2. Can I cap this like the ARC cache? Is this desirable to do?


        • 1. Re: ZFS capped cache still too large.
          You did reboot after changing /etc/system?

          And if there is going to be significant memory pressure on this server, and your file IO patterns don't require performance-critical multiple reads from the same file(s), you'll do much better to limit the ZFS ARC to something like 1 GB. Or smaller.

          Yes, the kernel will release the ZFS ARC if something else needs the memory.

          But the release will be slow. Much, much slower than getting truly free memory.
          • 2. Re: ZFS capped cache still too large.
            Thanks for the info - the host was rebooted, and I've seen the impact of slow ZFS memory release first hand, in earlier Solaris 10 releases.

            I'm not really concerned about my ARC cache size - I believe it's being enforced correctly.

            The problem I have is that I cannot explain what the additional 60GB (the difference between "ZFS File Data" at 160GB and the ARC cache at 100GB) is used for.

            That's too much memory to dismiss lightly. I'm sure the kernel is putting this 60GB to good use, it's just that I have no idea what it's for and I should be able to explain this.

            Does anyone know?