This discussion is archived
2 Replies Latest reply: Sep 2, 2012 9:30 PM by 952774 RSS

Dadenie virtual machine based on Linux 5.7

952774 Newbie
Currently Being Moderated
Linux Operating System is installed on a virtual machine that is deployed on VSphere 4.0. On the virtual machine deployed database Oracle 11.2.0.1.
The other day there was the fall of the virtual machine with the database. First, the process was killed DBW0 respectively, and stopped the database.
After that, after some time the process has caused vmmemctl oom-killer, and the operating system to hang.
Tell me because of what might fall in a virtual machine and in what direction to dig. Download the database is minimal. In this night it did not loaded.

Cutting from log file Linux:

Aug 3 23:27:54 dbhyperion kernel: vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Aug 3 23:27:54 dbhyperion kernel: vmtoolsd cpuset=/ mems_allowed=0
Aug 3 23:27:54 dbhyperion kernel: Pid: 2510, comm: vmtoolsd Not tainted 2.6.32-200.13.1.el5uek #1
Aug 3 23:27:54 dbhyperion kernel: Call Trace:
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff811088da>] alloc_pages_current+0x9b/0xa4
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d774f>] __page_cache_alloc+0x5c/0x61
Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810df800>] __do_page_cache_readahead+0x81/0x196
Aug 3 23:27:55 dbhyperion kernel: [<ffffffff81076ded>] ? wake_bit_function+0x0/0x2f
Aug 3 23:27:55 dbhyperion kernel: [<ffffffff810df936>] ra_submit+0x21/0x25
Aug 3 23:27:56 dbhyperion kernel: [<ffffffff810d8d1e>] filemap_fault+0x15e/0x353
Aug 3 23:27:56 dbhyperion kernel: [<ffffffff81115a0a>] ? lookup_page_cgroup+0x32/0x48
Aug 3 23:27:56 dbhyperion kernel: [<ffffffff810efde5>] __do_fault+0x59/0x37c
Aug 3 23:27:57 dbhyperion kernel: [<ffffffff811159d6>] ? __pfn_to_section+0x12/0x14
Aug 3 23:27:57 dbhyperion kernel: [<ffffffff81115a0a>] ? lookup_page_cgroup+0x32/0x48
Aug 3 23:27:57 dbhyperion kernel: [<ffffffff810f1ba3>] handle_mm_fault+0x14b/0x80f
Aug 3 23:27:58 dbhyperion kernel: [<ffffffff8104367e>] ? need_resched+0x23/0x2d
Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81043696>] ? should_resched+0xe/0x2f
Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
Aug 3 23:27:58 dbhyperion kernel: Mem-Info:
Aug 3 23:27:58 dbhyperion kernel: Node 0 DMA per-cpu:
Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 3 23:27:58 dbhyperion kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 3 23:27:58 dbhyperion kernel: Node 0 DMA32 per-cpu:
Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 0
Aug 3 23:27:58 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 0
Aug 3 23:27:58 dbhyperion kernel: Node 0 Normal per-cpu:
Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 27
Aug 3 23:27:59 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 29
Aug 3 23:27:59 dbhyperion kernel: active_anon:735 inactive_anon:325 isolated_anon:0
Aug 3 23:27:59 dbhyperion kernel: active_file:53 inactive_file:140 isolated_file:0
Aug 3 23:27:59 dbhyperion kernel: unevictable:1112 dirty:9 writeback:6 unstable:0
Aug 3 23:28:00 dbhyperion kernel: free:9583 slab_reclaimable:3158 slab_unreclaimable:30263
Aug 3 23:28:00 dbhyperion kernel: mapped:1039 shmem:202 pagetables:24186 bounce:0
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA free:15776kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15292kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 3000 6030 6030
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 free:17308kB min:4936kB low:6168kB high:7404kB active_anon:348kB inactive_anon:372kB active_file:40kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:124kB shmem:116kB slab_reclaimable:780kB slab_unreclaimable:1060kB kernel_stack:216kB pagetables:21868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:320 all_unreclaimable? no
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 3030 3030
Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal free:5248kB min:4988kB low:6232kB high:7480kB active_anon:2592kB inactive_anon:928kB active_file:172kB inactive_file:496kB unevictable:4448kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:4448kB dirty:36kB writeback:44kB mapped:4032kB shmem:692kB slab_reclaimable:11852kB slab_unreclaimable:119992kB kernel_stack:1904kB pagetables:74876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1504 all_unreclaimable? no
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 0 0
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 1*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15776kB
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32: 257*4kB 199*8kB 120*16kB 87*32kB 58*64kB 15*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17308kB
Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal: 367*4kB 7*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5124kB
Aug 3 23:28:01 dbhyperion kernel: 1748 total pagecache pages
Aug 3 23:28:01 dbhyperion kernel: 562 pages in swap cache
Aug 3 23:28:01 dbhyperion kernel: Swap cache stats: add 2091542, delete 2090980, find 1801328/1932208
Aug 3 23:28:01 dbhyperion kernel: Free swap = 9532960kB
Aug 3 23:28:01 dbhyperion kernel: Total swap = 11711376kB
Aug 3 23:28:01 dbhyperion kernel: 1572848 pages RAM
Aug 3 23:28:01 dbhyperion kernel: 42579 pages reserved
Aug 3 23:28:01 dbhyperion kernel: 21230 pages shared
Aug 3 23:28:01 dbhyperion kernel: 1493889 pages non-shared
Aug 3 23:28:01 dbhyperion kernel: Out of memory: kill process 7916 (perl) score 1724610 or a child
Aug 3 23:28:01 dbhyperion kernel: Killed process 7936 (java)
Aug 3 23:28:01 dbhyperion kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Aug 3 23:28:01 dbhyperion kernel: java cpuset=/ mems_allowed=0
Aug 3 23:28:01 dbhyperion kernel: Pid: 8046, comm: java Not tainted 2.6.32-200.13.1.el5uek #1
Aug 3 23:28:01 dbhyperion kernel: Call Trace:
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d725a>] ? find_get_page+0x22/0x69
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079530>] ? __remove_hrtimer+0x7b/0x89
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079c8f>] ? unlock_hrtimer_base+0x18/0x1a
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079d65>] ? hrtimer_try_to_cancel+0x40/0x4b
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81239ddc>] ? __get_user_8+0x1c/0x23
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8108462d>] ? fetch_robust_entry+0x28/0x51
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81085131>] exit_robust_list+0x3d/0xfd
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105cd6f>] ? jiffies_to_timespec+0x26/0x35
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81057396>] mm_release+0x30/0x106
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81059f8d>] exit_mm+0x26/0x127
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105b581>] do_exit+0x209/0x696
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81068988>] ? freezing+0x13/0x15
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105baae>] sys_exit_group+0x0/0x1b
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8106bdff>] get_signal_to_deliver+0x303/0x328
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810111f6>] do_notify_resume+0x90/0x6d7
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810450ca>] ? __dequeue_entity+0x33/0x38
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8107e2b4>] ? timekeeping_get_ns+0x1b/0x3d
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81123593>] ? path_put+0x22/0x27
Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 3 23:28:01 dbhyperion kernel: Mem-Info:
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA per-cpu:
Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 per-cpu:
Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 0
Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 5
Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal per-cpu:
Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 56
Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 70
Aug 3 23:28:01 dbhyperion kernel: active_anon:660 inactive_anon:309 isolated_anon:0
Aug 3 23:28:01 dbhyperion kernel: active_file:0 inactive_file:10 isolated_file:0
Aug 3 23:28:01 dbhyperion kernel: unevictable:1112 dirty:0 writeback:0 unstable:0
Aug 3 23:28:01 dbhyperion kernel: free:9672 slab_reclaimable:3148 slab_unreclaimable:30262
Aug 3 23:28:01 dbhyperion kernel: mapped:997 shmem:197 pagetables:24186 bounce:0
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA free:15776kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15292kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 3000 6030 6030
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 free:17224kB min:4936kB low:6168kB high:7404kB active_anon:112kB inactive_anon:280kB active_file:0kB inactive_file:40kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:80kB slab_reclaimable:752kB slab_unreclaimable:1056kB kernel_stack:216kB pagetables:21868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1068 all_unreclaimable? no
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 3030 3030
Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal free:5688kB min:4988kB low:6232kB high:7480kB active_anon:2528kB inactive_anon:956kB active_file:0kB inactive_file:0kB unevictable:4448kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:4448kB dirty:0kB writeback:0kB mapped:3908kB shmem:708kB slab_reclaimable:11840kB slab_unreclaimable:119992kB kernel_stack:1904kB pagetables:74876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 0 0
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 1*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15776kB
Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32: 267*4kB 189*8kB 120*16kB 88*32kB 58*64kB 15*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17300kB
Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal: 486*4kB 19*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5696kB
Aug 3 23:28:01 dbhyperion kernel: 1535 total pagecache pages
Aug 3 23:28:01 dbhyperion kernel: 559 pages in swap cache
Aug 3 23:28:01 dbhyperion kernel: Swap cache stats: add 2091961, delete 2091402, find 1801378/1932333
Aug 3 23:28:01 dbhyperion kernel: Free swap = 9532516kB
Aug 3 23:28:01 dbhyperion kernel: Total swap = 11711376kB
Aug 3 23:28:01 dbhyperion kernel: 1572848 pages RAM
Aug 3 23:28:01 dbhyperion kernel: 42579 pages reserved
Aug 3 23:28:01 dbhyperion kernel: 21194 pages shared
Aug 3 23:28:01 dbhyperion kernel: 1493703 pages non-shared
Aug 3 23:28:01 dbhyperion kernel: Out of memory: kill process 7916 (perl) score 1349957 or a child
Aug 3 23:28:01 dbhyperion kernel: Killed process 7938 (java)
  • 1. Re: Dadenie virtual machine based on Linux 5.7
    Dude! Guru
    Currently Being Moderated
    It looks like you are experiencing low memory exhaustion. The kernel uses low memory to track allocations of all memory. When low memory is exhausted, it doesn't matter how much high memory is available, the oom-killer will begin killing processes to keep the server alive.

    I suggest you check your host and guest memory usage and allocation in VSphere, and make sure it supports your guest systems. Chances are probably that you do not have enough physical RAM to perform the tasks, or do not have enough memory allocated to your virtual machine, or have a process in your guest system that causes memory starvation because it is not configured correctly, e.g. Oracle SGA.
  • 2. Re: Dadenie virtual machine based on Linux 5.7
    952774 Newbie
    Currently Being Moderated
    Thanks for the reply.
    Will drop further.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points