Discussions
Categories
- 17.9K All Categories
- 3.4K Industry Applications
- 3.4K Intelligent Advisor
- 73 Insurance
- 537.3K On-Premises Infrastructure
- 138.6K Analytics Software
- 38.6K Application Development Software
- 6K Cloud Platform
- 109.6K Database Software
- 17.6K Enterprise Manager
- 8.8K Hardware
- 71.2K Infrastructure Software
- 105.3K Integration
- 41.6K Security Software
Lot of LWPs in vmstat. What it means? How to fix?
Hello,
I have Solaris 11.2 Sparc server, which is a Oracle DB server. Since some time I have been informed latency and slowness on the database side. I ran memstat command and it shows me Kernel is taking more than 50% and free memory is very less. I understand that ZFS arc memory could have been set, but that would impact ZFS file data (I guess).
vmstat shows me swap out lot of LWPS in w fileld. Probably doesn't looks ideal.
My questions are -
- Is it memory, I would increase or swap ?
- When kernel is showing 55% of total memory, is it DB/app which is consuming this much? DB guys are blaming on us (Admin guys) that your server/OS is having not-ideal setup, our database is not supposed to consume so much. I have doubts about it.
[email protected]:~# echo ::memstat | mdb -kPage Summary Pages Bytes %Tot ----------------- ---------------- ---------------- ---- Kernel 2292714 17.4G 55%Guest 0 0 0% ZFS Metadata 13953 109.0M 0% ZFS File Data 4500 35.1M 0% Anon 1804250 13.7G 43% Exec and libs 28421 222.0M 1% Page cache 11723 91.5M 0% Free (cachelist) 46 368k 0% Free (freelist) 38697 302.3M 1% Total 4194304 32G [email protected]:~#[email protected]:~# vmstat 1 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr vc vc vc vc in sy cs us sy id 0 0 41 29721768 871728 132 695 1 1 1 514488 9 19 19 47 10 10573 600322 14357 5 3 92 0 0 442 23559016 436608 340 900 0 0 0 514488 0 107 20 516 7 23337 13005 41205 3 3 94 0 0 442 23559016 437720 0 91 0 0 0 463040 0 33 10 512 8 22148 20037 38870 3 3 94 0 0 442 23559016 436856 134 347 0 72 72 514488 0 5 20 536 5 21031 38934 37366 3 8 89 0 0 442 23559016 436728 0 64 0 88 88 463040 0 172 25 604 9 6867 10689 7761 2 2 95 [email protected]:~#
Please suggest me, if this data is telling something.
Thanks
Answers
-
Hi.
It's not normal that kernel use more than 1/2 of RAM IMHO.
Need more information for provide any recommendation ( Is this server only database or some application/ file server? Are You use ASM ? Do you have separate FS (or pools) for oracle datafiles, index etc. What SGA size configured for Oracle database... More information about database size, how many swap configured etc).
Check this docs:
Memory Management Between ZFS and Applications in Oracle Solaris 11.x (Doc ID 1663862.1)
https://docs.oracle.com/cd/E36784_01/html/E36845/chapterzfs-db2.html#scrolltoc
As initial step - i try set user_reserve_hint_pct to 75 %
Regards,
Nik.
-
What this tells us is that you have run an unpatched system since 2015 :-)
ZFS, *should* release memory why an application requires use, this does not always read well with monitoring apps. And hence often a guilty finger is pointed at ZFS and then admins start messing with the arc cache size.
For DB ram use all the usual checks can be made;- top, memstat, ps, prstat, ipcs, pmap.
Since S11.3 we have been pushed towards using "user_reserve_hint_pct" which constrains ZFS in a much more friendly way. Also we can use "-v" with memstat for more detail.
I might be mis-reading the vmstat snippit, which I think shows the system is paging ? In which case either constrain ZFS (which may mean a perf hit if it is a high ZFS usage system), or add RAM.
Other opinions may be available.
-
Yes, I agree it was patched long back and it is due.
It has total of 32gb of swap
It is just oracle database server. No other app is installed. There are file-systems of /u01 to /u05, which were created separately as per Oracle's recommendations.
I am in the impression that ZFS arc memory will fix "ZFS File Data". It holds the memory and whenever an application or database needs memory, it will release that. (Please correct me if I am wrong). But I am concerned by Kernel. How will I justify, why it is holding more than 50% of memory.
I created a case with Oracle, just got a reply. He pointed it to application/DB, but I am not convinced. It doesn't explain, why Kernel is holding more than 50% of memory. If it is explainable and it shows memory is less, I am in capacity to increase memory or swap
"From the guds the system was have very low free memory and the system was swapping out lot of LWPS to disk-based swap. Vmstat shows there are a number of swapped out lightweight processes (LWPs) that are waiting for processing resources to finish.
Regarding the kernel memory usage, from the kmastat , there is 13GB in kmem_alloc_xxx and few hundreds Mb in streams_dblk_xxxx and zfs_data_xxx
The memory foot print from the kmastat is likely due to the workload on the system.
You need to work with application and if needed, add more memory and swap."
-
There is still too little detail here ?
What is your Phys RAM ?
What is your SRU level ? (you may have a bug (in memstat )?)
What is you rmax zfs arc cache ? echo "::arc" | mdb -k
Consider setting that lower as per the evil tuning guide (where ever it is these days)
Or, as Nik says, use the zfs hint setting (I forgot it went that far back). As it sounds like you don;t overly use ZFS on the system.
ZFS will ramp up it's usage when you copy large datasets, even running find , if it sees the memory as available and will keep it until requested.
-
"ZFS will ramp up it's usage when you copy large datasets" - I understand, if I set ZFS arc mem or user_reserve_hint_pct (if I upgrade to 11.3), it will release memory when required by OS/DB/application. But will it release from "Kernel", which is currently showing over 50% ? or will it release from "ZFS File Data"?
Here is all I see, what you asked -->
[email protected]:~# prtdiag -v | grep -i memMemory size: 32768 [email protected]:~# swap -lswapfile dev swaplo blocks free/dev/zvol/dsk/rpool/swap 293,1 16 67108848 [email protected]:~# zfs get volsize rpool/swapNAME PROPERTY VALUE SOURCErpool/swap volsize 32G [email protected]:~#[email protected]:~#[email protected]:~# pkg info entire Name: entire Summary: Incorporation to lock all system packages to the same build Description: This package constrains system package versions to the same build. WARNING: Proper system update and correct package selection depend on the presence of this incorporation. Removing this package will result in an unsupported system. Category: Meta Packages/Incorporations State: Installed Publisher: solaris Version: 0.5.11 Build Release: 5.11 Branch: 0.175.2.0.0.42.0Packaging Date: June 24, 2014 07:38:32 PM Size: 5.46 kB FMRI: pkg://solaris/[email protected],5.11-0.175.2.0.0.42.0:[email protected]:~#[email protected]:~# echo "::arc" | mdb -khits = 119809619698misses = 11126969915demand_data_hits = 62709026524demand_data_misses = 2429862209demand_metadata_hits = 54778367086demand_metadata_misses = 2107427330prefetch_data_hits = 2053874944prefetch_data_misses = 6102618636prefetch_metadata_hits = 268351144prefetch_metadata_misses = 487061740mru_hits = 10686551023mru_ghost_hits = 1557259544mfu_hits = 113643169459mfu_ghost_hits = 3960164960deleted = 10360117180mutex_miss = 361859102hash_elements = 5517427234hash_elements_max = 5517427242hash_collisions = 1179931870hash_chains = 25003106hash_chain_max = 9p = 58 MBc = 64 MBc_min = 64 MBc_max = 31139 MBsize = 63 MBbuf_size = 1 MBdata_size = 43 MBother_size = 18 MBevict_mfu = 643965448 MBevict_mru = 337800535 MBevict_l2_cached = 0 MBevict_l2_eligible = 472921138 MBevict_l2_ineligible = 508844845 MBl2_hits = 0l2_misses = 4537289539l2_feeds = 0l2_rw_clash = 0l2_read_bytes = 0 MBl2_write_bytes = 0 MBl2_writes_sent = 0l2_writes_done = 0l2_writes_error = 0l2_writes_hdr_miss = 0l2_evict_lock_retry = 0l2_evict_reading = 0l2_abort_lowmem = 0l2_cksum_bad = 0l2_io_error = 0l2_hdr_size = 0 MBl2_size = 0memory_throttle_count = 530928meta_used = 20 MBmeta_max = 539 MBmeta_limit = 0 MBarc_no_grow = 1arc_tempreserve = 0 [email protected]:~#[email protected]:~# vmstat 1 10 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr vc vc vc vc in sy cs us sy id 0 0 41 29720352 871616 132 695 1 1 1 514488 9 19 19 47 10 10572 600186 14357 5 3 92 0 0 416 23737240 419344 4 74 8 0 0 514488 0 1 0 0 0 1731 1106 1985 1 0 99 0 0 416 23737240 418424 0 10 0 0 0 463040 0 0 72 305 136 3812 714 5292 0 1 99 1 0 416 23737240 418520 0 32 0 0 0 416736 0 4 0 59 3 5975 1847 9599 0 0 99 0 0 416 23737240 418848 26 83 0 0 0 514488 0 88 1 2 13 6733 42000 11626 1 2 97 0 0 416 23737240 418840 0 0 0 0 0 463040 0 2 0 5 8 2339 1181 2880 1 0 99 0 0 416 23737240 418656 0 4 32 0 0 416736 0 4 0 83 0 6485 833 11252 1 0 99 1 0 416 23737240 418560 163 461 16 0 0 514488 0 2 4 345 71 9887 1676 17294 1 1 98 1 0 416 23728336 420456 8 209 0 0 0 463040 0 1 16 2 11 3055 1194 4435 1 1 99 1 0 416 23728336 418984 26 155 0 0 0 514488 0 133 0 5 7 5144 20561 8526 1 1 [email protected]:~#
Thanks
-
Hi.
> There are file-systems of /u01 to /u05
Are You tuning ZFS according https://docs.oracle.com/cd/E36784_01/html/E36845/chapterzfs-db2.html#scrolltoc ?
Oracle recommend switch off cache of data for FS that store data-files.
You not just create many separate mountpoint, but separate logs, index, datafiles and do correct tuning ZFS for every FS.
ZFS use as more memory, as available. In case You do backup by copy datafiles - ZFS will store all it at the cache.
>> It holds the memory and whenever an application or database needs memory, it will release that.
It can see that You have many free swap and not see reason for release it. DB work with datafiles and ZFS store it at the cache....
Again, start from set user_reserve_hint_pct .
In case You will change it dynamic: Make sure to use decrease the value only by small amounts, no greater than 5% at a time.
Doc ID 1663862.1- provide script for change value dynamic and commands for check current value.
You can use kstat -n arcstats for check current state of ARC_Cache.
http://blog.zach.st/2014/08/02/control-the-size-of-the-zfs-arc-cache-dynamically.html
Some people prefer set static limit for zfs_arc_max, zfs_arc_min.
Be attention:
zfs_arc_max must be greater than zfs_arc_min.
In case you will set only zfs_arc_max, it can be lower that default value of zfs_arc_min and limit will not work. Set both parameters.
Regards,
Nik
-
Where do I start ?
You are using S11.2 GA, Totally unpatched.
I'm surprised that support did not suggest tthat before you do *anything* else, you patch and upgrade te system.
-
Hi
You are nearly out of free memory. Adding more SWAP does not help here.
ZFS is not using much, only a few MB!
ZFS File Data 4500 35.1M 0%
It is recommended to limit using zfs_arc_max, but that is notyour current issue.
The DB or maybe Monitoring or something does currently probablyuse all your memory.
Would recommended to restart the Database and/or rebootthe server.
many years uptime?
Best regards,Marcel Hofstetter
JomaSoft GmbH
-
How much of disk is free?
When it is a couple of hundred mega-bytes, ZFS exsessively acts on disk. Maybe this limit is not fixed. Maybe it depends on size of file you operate on. Maybe it does not depend on the amount of data that goes to file each time.Unlike other operating systems, Solaris does not return memory from program to system with
free().man freesays so. I did not test this.
