apps-infra

    Forum Stats

  • 3,874,067 Users
  • 2,266,672 Discussions
  • 7,911,721 Comments

Discussions

Lot of LWPs in vmstat. What it means? How to fix?

User_Y2Y3I
User_Y2Y3I Member Posts: 31 Red Ribbon
edited Jul 19, 2020 6:41PM in Solaris 11

Hello,

I have Solaris 11.2 Sparc server, which is a Oracle DB server. Since some time I have been informed latency and slowness on the database side. I ran memstat command and it shows me Kernel is taking more than 50% and free memory is very less. I understand that ZFS arc memory could have been set, but that would impact ZFS file data (I guess).

vmstat shows me swap out lot of LWPS in w fileld. Probably doesn't looks ideal.

My questions are -

- Is it memory, I would increase or swap ?

- When kernel is showing 55% of total memory, is it DB/app which is consuming this much? DB guys are blaming on us (Admin guys) that your server/OS is having not-ideal setup, our database is not supposed to consume so much. I have doubts about it.

[email protected]:~# echo ::memstat | mdb -kPage Summary Pages Bytes %Tot ----------------- ---------------- ---------------- ---- Kernel 2292714 17.4G 55%Guest 0 0 0% ZFS Metadata 13953 109.0M 0% ZFS File Data 4500 35.1M 0% Anon 1804250 13.7G 43% Exec and libs 28421 222.0M 1% Page cache 11723 91.5M 0% Free (cachelist) 46 368k 0% Free (freelist) 38697 302.3M 1% Total 4194304 32G [email protected]:~#[email protected]:~# vmstat 1 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr vc vc vc vc in sy cs us sy id 0 0 41 29721768 871728 132 695 1 1 1 514488 9 19 19 47 10 10573 600322 14357 5 3 92 0 0 442 23559016 436608 340 900 0 0 0 514488 0 107 20 516 7 23337 13005 41205 3 3 94 0 0 442 23559016 437720 0 91 0 0 0 463040 0 33 10 512 8 22148 20037 38870 3 3 94 0 0 442 23559016 436856 134 347 0 72 72 514488 0 5 20 536 5 21031 38934 37366 3 8 89 0 0 442 23559016 436728 0 64 0 88 88 463040 0 172 25 604 9 6867 10689 7761 2 2 95 [email protected]:~#

Please suggest me, if this data is telling something.

Thanks

Tagged:

Answers

  • Nik
    Nik Blocked Member Posts: 2,879 Bronze Crown
    edited Jul 7, 2020 4:27AM

    Hi.

      It's not normal that kernel use more than 1/2 of RAM  IMHO.

    Need more information for provide any recommendation (  Is this server only database or some application/ file server? Are  You use ASM ?   Do you have separate FS (or pools) for oracle datafiles, index etc. What SGA size configured for Oracle database... More information about database size,  how many swap configured etc).

    Check this docs:

    Memory Management Between ZFS and Applications in Oracle Solaris 11.x (Doc ID 1663862.1)

    https://docs.oracle.com/cd/E36784_01/html/E36845/chapterzfs-db2.html#scrolltoc

    As initial step -   i try set user_reserve_hint_pct to 75 %

    Regards,

      Nik.

  • YTC#1 - Bruce D Porter
    YTC#1 - Bruce D Porter Member Posts: 106 Red Ribbon
    edited Jul 7, 2020 4:29AM

    What this tells us is that you have run an unpatched system since 2015 :-)

    ZFS, *should* release memory why an application requires use, this does not always read well with monitoring apps. And hence often a guilty finger is pointed at ZFS and then admins start messing with the arc cache size.

    For DB ram use all the usual checks can be made;- top, memstat, ps, prstat, ipcs, pmap.

    Since S11.3 we have been pushed towards using "user_reserve_hint_pct" which constrains ZFS in a much more friendly way. Also we can use "-v" with memstat for more detail.

    I might be mis-reading the vmstat snippit, which I think shows the system is paging ? In which case either constrain ZFS (which may mean a perf hit if it is a high ZFS usage system), or add RAM.

    Other opinions may be available.

  • User_Y2Y3I
    User_Y2Y3I Member Posts: 31 Red Ribbon
    edited Jul 7, 2020 5:47AM

    Yes, I agree it was patched long back and it is due.

    It has total of 32gb of swap

    It is just oracle database server. No other app is installed. There are file-systems of /u01 to /u05, which were created separately as per Oracle's recommendations.

    I am in the impression that ZFS arc memory will fix "ZFS File Data". It holds the memory and whenever an application or database needs memory, it will release that. (Please correct me if I am wrong). But I am concerned by Kernel. How will I justify, why it is holding more than 50% of memory.

    I created a case with Oracle, just got a reply. He pointed it to application/DB, but I am not convinced. It doesn't explain, why Kernel is holding more than 50% of memory. If it is explainable and it shows memory is less, I am in capacity to increase memory or swap

    "From the guds the system was have very low free memory and the system was swapping out lot of LWPS to disk-based swap. Vmstat shows there are a number of swapped out lightweight processes (LWPs) that are waiting for processing resources to finish.

    Regarding the kernel memory usage, from the kmastat , there is 13GB in kmem_alloc_xxx and few hundreds Mb in streams_dblk_xxxx and zfs_data_xxx

    The memory foot print from the kmastat is likely due to the workload on the system.

    You need to work with application and if needed, add more memory and swap."

  • YTC#1 - Bruce D Porter
    YTC#1 - Bruce D Porter Member Posts: 106 Red Ribbon
    edited Jul 7, 2020 5:58AM

    There is still too little detail here ?

    What is your Phys RAM ?

    What is your SRU level ? (you may have a bug (in memstat )?)

    What is you rmax zfs arc cache ? echo "::arc" | mdb -k

    Consider setting that lower as per the evil tuning guide (where ever it is these days)

    Or, as Nik says, use the zfs hint setting (I forgot it went that far back). As it sounds like you don;t overly use ZFS on the system.

    ZFS will ramp up it's usage when you copy large datasets, even running find , if it sees the memory as available and will keep it until requested.

  • User_Y2Y3I
    User_Y2Y3I Member Posts: 31 Red Ribbon
    edited Jul 7, 2020 6:16AM

    "ZFS will ramp up it's usage when you copy large datasets" - I understand, if I set ZFS arc mem or user_reserve_hint_pct (if I upgrade to 11.3), it will release memory when required by OS/DB/application. But will it release from "Kernel", which is currently showing over 50% ? or will it release from "ZFS File Data"?

    Here is all I see, what you asked -->

    [email protected]:~# prtdiag -v | grep -i memMemory size: 32768 [email protected]:~# swap -lswapfile             dev    swaplo   blocks     free/dev/zvol/dsk/rpool/swap 293,1        16 67108848 [email protected]:~# zfs get volsize rpool/swapNAME        PROPERTY  VALUE  SOURCErpool/swap  volsize   32G    [email protected]:~#[email protected]:~#[email protected]:~# pkg info entire          Name: entire       Summary: Incorporation to lock all system packages to the same build   Description: This package constrains system package versions to the same                build.  WARNING: Proper system update and correct package                selection depend on the presence of this incorporation.                Removing this package will result in an unsupported system.      Category: Meta Packages/Incorporations         State: Installed     Publisher: solaris       Version: 0.5.11 Build Release: 5.11        Branch: 0.175.2.0.0.42.0Packaging Date: June 24, 2014 07:38:32 PM          Size: 5.46 kB          FMRI: pkg://solaris/[email protected],5.11-0.175.2.0.0.42.0:[email protected]:~#[email protected]:~# echo "::arc" | mdb -khits                      = 119809619698misses                    =  11126969915demand_data_hits          =  62709026524demand_data_misses        =   2429862209demand_metadata_hits      =  54778367086demand_metadata_misses    =   2107427330prefetch_data_hits        =   2053874944prefetch_data_misses      =   6102618636prefetch_metadata_hits    =    268351144prefetch_metadata_misses  =    487061740mru_hits                  =  10686551023mru_ghost_hits            =   1557259544mfu_hits                  = 113643169459mfu_ghost_hits            =   3960164960deleted                   =  10360117180mutex_miss                =    361859102hash_elements             =   5517427234hash_elements_max         =   5517427242hash_collisions           =   1179931870hash_chains               =     25003106hash_chain_max            =            9p                         =           58 MBc                         =           64 MBc_min                     =           64 MBc_max                     =        31139 MBsize                      =           63 MBbuf_size                  =            1 MBdata_size                 =           43 MBother_size                =           18 MBevict_mfu                 =    643965448 MBevict_mru                 =    337800535 MBevict_l2_cached           =            0 MBevict_l2_eligible         =    472921138 MBevict_l2_ineligible       =    508844845 MBl2_hits                   =            0l2_misses                 =   4537289539l2_feeds                  =            0l2_rw_clash               =            0l2_read_bytes             =            0 MBl2_write_bytes            =            0 MBl2_writes_sent            =            0l2_writes_done            =            0l2_writes_error           =            0l2_writes_hdr_miss        =            0l2_evict_lock_retry       =            0l2_evict_reading          =            0l2_abort_lowmem           =            0l2_cksum_bad              =            0l2_io_error               =            0l2_hdr_size               =            0 MBl2_size                   =            0memory_throttle_count     =       530928meta_used                 =           20 MBmeta_max                  =          539 MBmeta_limit                =            0 MBarc_no_grow               =            1arc_tempreserve           =            0 [email protected]:~#[email protected]:~# vmstat 1 10 kthr      memory            page            disk          faults      cpu r b w   swap  free  re  mf pi po fr de sr vc vc vc vc   in   sy   cs us sy id 0 0 41 29720352 871616 132 695 1 1 1 514488 9 19 19 47 10 10572 600186 14357 5 3 92 0 0 416 23737240 419344 4 74 8 0  0 514488 0 1 0 0  0 1731 1106 1985  1  0 99 0 0 416 23737240 418424 0 10 0 0  0 463040 0 0 72 305 136 3812 714 5292 0 1 99 1 0 416 23737240 418520 0 32 0 0  0 416736 0 4 0 59 3 5975 1847 9599  0  0 99 0 0 416 23737240 418848 26 83 0 0 0 514488 0 88 1 2 13 6733 42000 11626 1 2 97 0 0 416 23737240 418840 0 0 0  0  0 463040 0 2 0 5  8 2339 1181 2880  1  0 99 0 0 416 23737240 418656 0 4 32 0  0 416736 0 4 0 83 0 6485  833 11252 1  0 99 1 0 416 23737240 418560 163 461 16 0 0 514488 0 2 4 345 71 9887 1676 17294 1 1 98 1 0 416 23728336 420456 8 209 0 0 0 463040 0 1 16 2 11 3055 1194 4435 1  1 99 1 0 416 23728336 418984 26 155 0 0 0 514488 0 133 0 5 7 5144 20561 8526 1 1 [email protected]:~#

    Thanks

  • Nik
    Nik Blocked Member Posts: 2,879 Bronze Crown
    edited Jul 7, 2020 6:34AM

    Hi.

    > There are file-systems of /u01 to /u05

    Are You  tuning ZFS according https://docs.oracle.com/cd/E36784_01/html/E36845/chapterzfs-db2.html#scrolltoc

    Oracle recommend  switch off cache of data for FS that store data-files.

    You not just create many separate mountpoint, but  separate logs, index, datafiles and  do correct tuning ZFS for every FS.

    ZFS  use as more memory, as available.  In case You do backup by copy datafiles -  ZFS will store all it at the cache.

    >>  It holds the memory and whenever an application or database needs memory, it will release that.

    It can see that You have many  free swap and not see reason for  release it.   DB work with datafiles and ZFS  store it at the cache....

    Again, start  from set user_reserve_hint_pct .

    In case You will change it dynamic:   Make sure to use decrease the value only by small amounts, no greater than 5% at a time.

    Doc ID 1663862.1- provide script for change value dynamic and commands for check current value.

    You can use kstat -n arcstats for check current state of ARC_Cache.

    http://blog.zach.st/2014/08/02/control-the-size-of-the-zfs-arc-cache-dynamically.html

    Some people prefer set static limit for  zfs_arc_max, zfs_arc_min.

    Be attention:

    zfs_arc_max must be greater than zfs_arc_min.

    In case you will  set only zfs_arc_max, it can be lower that default value of zfs_arc_min and limit will not work.  Set both parameters.

    Regards,

       Nik

  • YTC#1 - Bruce D Porter
    YTC#1 - Bruce D Porter Member Posts: 106 Red Ribbon
    edited Jul 7, 2020 6:37AM

    Where do I start ?

    You are using S11.2 GA, Totally unpatched.

    I'm surprised that support did not suggest tthat before you do *anything* else, you patch and upgrade te system.

  • Marcel Hofstetter JomaSoft
    Marcel Hofstetter JomaSoft Member Posts: 279 Silver Badge
    edited Jul 10, 2020 10:06AM

    Hi

    You are nearly out of free memory. Adding more SWAP does not help here.

    ZFS is not using much, only a few MB!

    ZFS File Data 4500 35.1M 0%

    It is recommended to limit using zfs_arc_max, but that is not

    your current issue.

    The DB or maybe Monitoring or something does currently probably

    use all your memory.


    Would recommended to restart the Database and/or reboot

    the server.

    many years uptime?
    Best regards,

    Marcel Hofstetter

    JomaSoft GmbH

  • Oqelu
    Oqelu Member Posts: 69 Red Ribbon
    edited Jul 19, 2020 6:41PM

    How much of disk is free?
    When it is a couple of hundred mega-bytes, ZFS exsessively acts on disk. Maybe this limit is not fixed. Maybe it depends on size of file you operate on. Maybe it does not depend on the amount of data that goes to file each time.

    Unlike other operating systems, Solaris does not return memory from program to system with free(). man free says so. I did not test this.

apps-infra