2 Replies Latest reply: Sep 2, 2012 11:30 PM by 952774 RSS

    Dadenie virtual machine based on Linux 5.7

    952774
      Linux Operating System is installed on a virtual machine that is deployed on VSphere 4.0. On the virtual machine deployed database Oracle 11.2.0.1.
      The other day there was the fall of the virtual machine with the database. First, the process was killed DBW0 respectively, and stopped the database.
      After that, after some time the process has caused vmmemctl oom-killer, and the operating system to hang.
      Tell me because of what might fall in a virtual machine and in what direction to dig. Download the database is minimal. In this night it did not loaded.

      Cutting from log file Linux:

      Aug 3 23:27:54 dbhyperion kernel: vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
      Aug 3 23:27:54 dbhyperion kernel: vmtoolsd cpuset=/ mems_allowed=0
      Aug 3 23:27:54 dbhyperion kernel: Pid: 2510, comm: vmtoolsd Not tainted 2.6.32-200.13.1.el5uek #1
      Aug 3 23:27:54 dbhyperion kernel: Call Trace:
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff811088da>] alloc_pages_current+0x9b/0xa4
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810d774f>] __page_cache_alloc+0x5c/0x61
      Aug 3 23:27:54 dbhyperion kernel: [<ffffffff810df800>] __do_page_cache_readahead+0x81/0x196
      Aug 3 23:27:55 dbhyperion kernel: [<ffffffff81076ded>] ? wake_bit_function+0x0/0x2f
      Aug 3 23:27:55 dbhyperion kernel: [<ffffffff810df936>] ra_submit+0x21/0x25
      Aug 3 23:27:56 dbhyperion kernel: [<ffffffff810d8d1e>] filemap_fault+0x15e/0x353
      Aug 3 23:27:56 dbhyperion kernel: [<ffffffff81115a0a>] ? lookup_page_cgroup+0x32/0x48
      Aug 3 23:27:56 dbhyperion kernel: [<ffffffff810efde5>] __do_fault+0x59/0x37c
      Aug 3 23:27:57 dbhyperion kernel: [<ffffffff811159d6>] ? __pfn_to_section+0x12/0x14
      Aug 3 23:27:57 dbhyperion kernel: [<ffffffff81115a0a>] ? lookup_page_cgroup+0x32/0x48
      Aug 3 23:27:57 dbhyperion kernel: [<ffffffff810f1ba3>] handle_mm_fault+0x14b/0x80f
      Aug 3 23:27:58 dbhyperion kernel: [<ffffffff8104367e>] ? need_resched+0x23/0x2d
      Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81043696>] ? should_resched+0xe/0x2f
      Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
      Aug 3 23:27:58 dbhyperion kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
      Aug 3 23:27:58 dbhyperion kernel: Mem-Info:
      Aug 3 23:27:58 dbhyperion kernel: Node 0 DMA per-cpu:
      Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 0, btch: 1 usd: 0
      Aug 3 23:27:58 dbhyperion kernel: CPU 1: hi: 0, btch: 1 usd: 0
      Aug 3 23:27:58 dbhyperion kernel: Node 0 DMA32 per-cpu:
      Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 0
      Aug 3 23:27:58 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 0
      Aug 3 23:27:58 dbhyperion kernel: Node 0 Normal per-cpu:
      Aug 3 23:27:58 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 27
      Aug 3 23:27:59 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 29
      Aug 3 23:27:59 dbhyperion kernel: active_anon:735 inactive_anon:325 isolated_anon:0
      Aug 3 23:27:59 dbhyperion kernel: active_file:53 inactive_file:140 isolated_file:0
      Aug 3 23:27:59 dbhyperion kernel: unevictable:1112 dirty:9 writeback:6 unstable:0
      Aug 3 23:28:00 dbhyperion kernel: free:9583 slab_reclaimable:3158 slab_unreclaimable:30263
      Aug 3 23:28:00 dbhyperion kernel: mapped:1039 shmem:202 pagetables:24186 bounce:0
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA free:15776kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15292kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 3000 6030 6030
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 free:17308kB min:4936kB low:6168kB high:7404kB active_anon:348kB inactive_anon:372kB active_file:40kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:124kB shmem:116kB slab_reclaimable:780kB slab_unreclaimable:1060kB kernel_stack:216kB pagetables:21868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:320 all_unreclaimable? no
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 3030 3030
      Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal free:5248kB min:4988kB low:6232kB high:7480kB active_anon:2592kB inactive_anon:928kB active_file:172kB inactive_file:496kB unevictable:4448kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:4448kB dirty:36kB writeback:44kB mapped:4032kB shmem:692kB slab_reclaimable:11852kB slab_unreclaimable:119992kB kernel_stack:1904kB pagetables:74876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1504 all_unreclaimable? no
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 0 0
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 1*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15776kB
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32: 257*4kB 199*8kB 120*16kB 87*32kB 58*64kB 15*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17308kB
      Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal: 367*4kB 7*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5124kB
      Aug 3 23:28:01 dbhyperion kernel: 1748 total pagecache pages
      Aug 3 23:28:01 dbhyperion kernel: 562 pages in swap cache
      Aug 3 23:28:01 dbhyperion kernel: Swap cache stats: add 2091542, delete 2090980, find 1801328/1932208
      Aug 3 23:28:01 dbhyperion kernel: Free swap = 9532960kB
      Aug 3 23:28:01 dbhyperion kernel: Total swap = 11711376kB
      Aug 3 23:28:01 dbhyperion kernel: 1572848 pages RAM
      Aug 3 23:28:01 dbhyperion kernel: 42579 pages reserved
      Aug 3 23:28:01 dbhyperion kernel: 21230 pages shared
      Aug 3 23:28:01 dbhyperion kernel: 1493889 pages non-shared
      Aug 3 23:28:01 dbhyperion kernel: Out of memory: kill process 7916 (perl) score 1724610 or a child
      Aug 3 23:28:01 dbhyperion kernel: Killed process 7936 (java)
      Aug 3 23:28:01 dbhyperion kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
      Aug 3 23:28:01 dbhyperion kernel: java cpuset=/ mems_allowed=0
      Aug 3 23:28:01 dbhyperion kernel: Pid: 8046, comm: java Not tainted 2.6.32-200.13.1.el5uek #1
      Aug 3 23:28:01 dbhyperion kernel: Call Trace:
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810d725a>] ? find_get_page+0x22/0x69
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079530>] ? __remove_hrtimer+0x7b/0x89
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079c8f>] ? unlock_hrtimer_base+0x18/0x1a
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81079d65>] ? hrtimer_try_to_cancel+0x40/0x4b
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81239ddc>] ? __get_user_8+0x1c/0x23
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8108462d>] ? fetch_robust_entry+0x28/0x51
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81085131>] exit_robust_list+0x3d/0xfd
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105cd6f>] ? jiffies_to_timespec+0x26/0x35
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81057396>] mm_release+0x30/0x106
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81059f8d>] exit_mm+0x26/0x127
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105b581>] do_exit+0x209/0x696
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81068988>] ? freezing+0x13/0x15
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8105baae>] sys_exit_group+0x0/0x1b
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8106bdff>] get_signal_to_deliver+0x303/0x328
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810111f6>] do_notify_resume+0x90/0x6d7
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff810450ca>] ? __dequeue_entity+0x33/0x38
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8107e2b4>] ? timekeeping_get_ns+0x1b/0x3d
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff81123593>] ? path_put+0x22/0x27
      Aug 3 23:28:01 dbhyperion kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
      Aug 3 23:28:01 dbhyperion kernel: Mem-Info:
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA per-cpu:
      Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 0, btch: 1 usd: 0
      Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 0, btch: 1 usd: 0
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 per-cpu:
      Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 0
      Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 5
      Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal per-cpu:
      Aug 3 23:28:01 dbhyperion kernel: CPU 0: hi: 186, btch: 31 usd: 56
      Aug 3 23:28:01 dbhyperion kernel: CPU 1: hi: 186, btch: 31 usd: 70
      Aug 3 23:28:01 dbhyperion kernel: active_anon:660 inactive_anon:309 isolated_anon:0
      Aug 3 23:28:01 dbhyperion kernel: active_file:0 inactive_file:10 isolated_file:0
      Aug 3 23:28:01 dbhyperion kernel: unevictable:1112 dirty:0 writeback:0 unstable:0
      Aug 3 23:28:01 dbhyperion kernel: free:9672 slab_reclaimable:3148 slab_unreclaimable:30262
      Aug 3 23:28:01 dbhyperion kernel: mapped:997 shmem:197 pagetables:24186 bounce:0
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA free:15776kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15292kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 3000 6030 6030
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32 free:17224kB min:4936kB low:6168kB high:7404kB active_anon:112kB inactive_anon:280kB active_file:0kB inactive_file:40kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:80kB slab_reclaimable:752kB slab_unreclaimable:1056kB kernel_stack:216kB pagetables:21868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1068 all_unreclaimable? no
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 3030 3030
      Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal free:5688kB min:4988kB low:6232kB high:7480kB active_anon:2528kB inactive_anon:956kB active_file:0kB inactive_file:0kB unevictable:4448kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:4448kB dirty:0kB writeback:0kB mapped:3908kB shmem:708kB slab_reclaimable:11840kB slab_unreclaimable:119992kB kernel_stack:1904kB pagetables:74876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
      Aug 3 23:28:01 dbhyperion kernel: lowmem_reserve[]: 0 0 0 0
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 1*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15776kB
      Aug 3 23:28:01 dbhyperion kernel: Node 0 DMA32: 267*4kB 189*8kB 120*16kB 88*32kB 58*64kB 15*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17300kB
      Aug 3 23:28:01 dbhyperion kernel: Node 0 Normal: 486*4kB 19*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5696kB
      Aug 3 23:28:01 dbhyperion kernel: 1535 total pagecache pages
      Aug 3 23:28:01 dbhyperion kernel: 559 pages in swap cache
      Aug 3 23:28:01 dbhyperion kernel: Swap cache stats: add 2091961, delete 2091402, find 1801378/1932333
      Aug 3 23:28:01 dbhyperion kernel: Free swap = 9532516kB
      Aug 3 23:28:01 dbhyperion kernel: Total swap = 11711376kB
      Aug 3 23:28:01 dbhyperion kernel: 1572848 pages RAM
      Aug 3 23:28:01 dbhyperion kernel: 42579 pages reserved
      Aug 3 23:28:01 dbhyperion kernel: 21194 pages shared
      Aug 3 23:28:01 dbhyperion kernel: 1493703 pages non-shared
      Aug 3 23:28:01 dbhyperion kernel: Out of memory: kill process 7916 (perl) score 1349957 or a child
      Aug 3 23:28:01 dbhyperion kernel: Killed process 7938 (java)
        • 1. Re: Dadenie virtual machine based on Linux 5.7
          Catch~22
          It looks like you are experiencing low memory exhaustion. The kernel uses low memory to track allocations of all memory. When low memory is exhausted, it doesn't matter how much high memory is available, the oom-killer will begin killing processes to keep the server alive.

          I suggest you check your host and guest memory usage and allocation in VSphere, and make sure it supports your guest systems. Chances are probably that you do not have enough physical RAM to perform the tasks, or do not have enough memory allocated to your virtual machine, or have a process in your guest system that causes memory starvation because it is not configured correctly, e.g. Oracle SGA.
          • 2. Re: Dadenie virtual machine based on Linux 5.7
            952774
            Thanks for the reply.
            Will drop further.