This discussion is archived
5 Replies Latest reply: Oct 22, 2012 2:30 PM by Avi Miller RSS

how to diagnose OOM killer

Kirill.Boyko Newbie
Currently Being Moderated
Hi!

Oct 16 11:05:44 lnxru0240vg081 kernel: active_anon:20668812 inactive_anon:711680 isolated_anon:0
Oct 16 11:05:44 lnxru0240vg081 kernel: active_file:920 inactive_file:1620 isolated_file:0
Oct 16 11:05:44 lnxru0240vg081 kernel: unevictable:57458 dirty:0 writeback:21 unstable:0
Oct 16 11:05:44 lnxru0240vg081 kernel: free:135636 slab_reclaimable:38069 slab_unreclaimable:28955
Oct 16 11:05:44 lnxru0240vg081 kernel: mapped:12103894 shmem:12916294 pagetables:1923988 bounce:0

Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 DMA free:8264kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8040kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 16 11:05:44 lnxru0240vg081 kernel: lowmem_reserve[]: 0 4024 126242 126242
Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 DMA32 free:490308kB min:1448kB low:1808kB high:2172kB active_anon:2426488kB inactive_anon:485436kB active_file:324kB inactive_file:456kB unevictable:8kB isolated(anon):0kB isolated(file):0kB present:4120800kB mlocked:8kB dirty:0kB writeback:36kB mapped:17512kB shmem:17616kB slab_reclaimable:568kB slab_unreclaimable:4992kB kernel_stack:312kB pagetables:247448kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1573 all_unreclaimable? yes

Oct 16 11:05:44 lnxru0240vg081 kernel: lowmem_reserve[]: 0 0 122217 122217
Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 Normal free:43972kB min:44024kB low:55028kB high:66036kB active_anon:80248760kB inactive_anon:2361284kB active_file:3356kB inactive_file:6024kB unevictable:229824kB isolated(anon):0kB isolated(file):0kB present:125151120kB mlocked:229824kB dirty:0kB writeback:48kB mapped:48398064kB shmem:51647560kB slab_reclaimable:151708kB slab_unreclaimable:110828kB kernel_stack:7616kB pagetables:7448504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:18063 all_unreclaimable? yes

Oct 16 11:05:44 lnxru0240vg081 kernel: lowmem_reserve[]: 0 0 0 0
Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8264kB
Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 DMA32: 519*4kB 565*8kB 392*16kB 240*32kB 230*64kB 241*128kB 63*256kB 145*512kB 52*1024kB 15*2048kB 61*4096kB = 490308kB
Oct 16 11:05:44 lnxru0240vg081 kernel: Node 0 Normal: 8176*4kB 599*8kB 80*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 42872kB
Oct 16 11:05:44 lnxru0240vg081 kernel: 13079384 total pagecache pages
Oct 16 11:05:44 lnxru0240vg081 kernel: 144850 pages in swap cache
Oct 16 11:05:44 lnxru0240vg081 kernel: Swap cache stats: add 1806819, delete 1661969, find 80597/110312
Oct 16 11:05:44 lnxru0240vg081 kernel: Free swap = 0kB
Oct 16 11:05:44 lnxru0240vg081 kernel: Total swap = 6111228kB
Oct 16 11:05:44 lnxru0240vg081 kernel: 32770032 pages RAM
Oct 16 11:05:44 lnxru0240vg081 kernel: 8743335 pages reserved
Oct 16 11:05:44 lnxru0240vg081 kernel: 101922782 pages shared
Oct 16 11:05:44 lnxru0240vg081 kernel: 11762572 pages non-shared


Am I right that my system is out of Lowmem?
What is "Node 0 Normal"? Where could I read more about this?
Am I right that it could be a problem because of large Pagetables?

Thank you for help!

Regards,
Kirill

Edited by: Kirill.Boyko on Oct 16, 2012 11:41 AM
  • 1. Re: how to diagnose OOM killer
    Dude! Guru
    Currently Being Moderated
    LOWMEM is required to manage the virtual memory page table. Apparently you have been running out of swap space and RAM and the OOM kill sacrificed one or more processes in order to free up memory for the system when all else fails.

    I suggest to examine the reason for the memory exhaustion. Perhaps you have a bad process. It might also be a good idea to switch to kernel hugepages, which use a page size of 2m instead of the 4k default, and may increase performance due to a much smaller pagetable.
  • 2. Re: how to diagnose OOM killer
    Kirill.Boyko Newbie
    Currently Being Moderated
    Hello!

    Thanks for help Dude. We are considering Hugepages and have an SR on that. The problem is that we are running this Linux as PV VM in Oracle VM 3.0.3.

    I remeber Avi had told on OOW 2012 :) that there is a Metalink note about enabling Hugepages support in OVM. Unfortunately I am not able to find it. Adding allowhugepage switch to OVS grub.conf and vm.nr_hugepages to linux machine seems does not help.

    Are there any official information on MOS about hugepages in PV Linux on OVM?

    Thx for help in advance!

    Regards, Kirill
  • 3. Re: how to diagnose OOM killer
    Dude! Guru
    Currently Being Moderated
    According to some vague info I can find you need to switch from PVM to PVHVM. Perhaps the following links are useful: Converting Guest to PVM from HVM
    Which kernel to use for paravirtualized guest OS - XEN or UEK?

    The Oracle UEK kernel will automatically detect and switch. You can download appropriate templates from the Oracle edelivery cloud. There is actually a separate forum that discusses OVM issues, which should be more suitable for this topic.
  • 4. Re: how to diagnose OOM killer
    Kirill.Boyko Newbie
    Currently Being Moderated
    Ok!

    Thank you for help Dude! Sure, I will switch to another forum. Seems this problem is somewhere between these forums :).

    Regards,
    Kirill
  • 5. Re: how to diagnose OOM killer
    Avi Miller Guru
    Currently Being Moderated
    Kirill.Boyko wrote:
    I remeber Avi had told on OOW 2012 :) that there is a Metalink note about enabling Hugepages support in OVM. Unfortunately I am not able to find it. Adding allowhugepage switch to OVS grub.conf and vm.nr_hugepages to linux machine seems does not help.
    The note was unpublished as the DB and OVM teams want to do more testing before supporting PV Hugepages in production.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points