Restart OVM Server Unknown

User_GR5ND

Observing the messages, it is found that after an "INFO: task dlm_thread: 4495 blocked for more than 120 seconds" ending with "system_call_fastpath + 0x18 / 0xd7"

the server is restarted affecting the VMs.

Additionally, the VM must be started manually since it does not start automatically.

ug 12 16:56:59 ESF-PORA01-TTO kernel: [5780010.626618] o2net: Connection to node EFS-PORA02-TTO (num 0) at 10.158.16.107:7777 has been idle for 90.273 secs.

Aug 12 16:58:30 ESF-PORA01-TTO kernel: [5780100.739882] o2net: Connection to node EFS-PORA02-TTO (num 0) at 10.158.16.107:7777 has been idle for 90.112 secs.

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053421] INFO: task dlm_thread:4495 blocked for more than 120 seconds.

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053636]       Not tainted 4.1.12-124.21.1.el6uek.x86_64 #2

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053833] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054212] dlm_thread      D ffff880094a18480     0  4495      2 0x00000080

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054221]  ffff88008a32fad8 0000000000000246 ffff88008a32fbb8 ffff880005861c00

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054227]  ffff88008a32faf8 ffff88008a330000 ffff88008a32fc18 ffff88008a32fbf0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054232]  ffffffffa05ca9e0 ffff88008a32fc08 ffff88008a32faf8 ffffffff816ec4e7

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054238] Call Trace:

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054262]  [<ffffffff816ec4e7>] schedule+0x37/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054284]  [<ffffffffa05bfcbd>] o2net_send_message_vec+0x64d/0xb20 [ocfs2_nodemanager]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054323]  [<ffffffff810bfb6a>] ? update_curr+0xea/0x190

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054329]  [<ffffffff810bc56e>] ? account_entity_dequeue+0xae/0xd0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054336]  [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054346]  [<ffffffff810f07ab>] ? lock_timer_base.isra.35+0x2b/0x50

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054356]  [<ffffffffa05c01b9>] o2net_send_message+0x29/0x30 [ocfs2_nodemanager]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054369]  [<ffffffffa06154d3>] dlm_drop_lockres_ref+0xc3/0x260 [ocfs2_dlm]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054375]  [<ffffffff816f167b>] ? _raw_spin_unlock_irqrestore+0x2b/0x60

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054380]  [<ffffffff810cc034>] ? remove_wait_queue+0x34/0x40

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054390]  [<ffffffffa0603ddf>] dlm_thread+0x61f/0x1140 [ocfs2_dlm]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054408]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054421]  [<ffffffff816ebede>] ? __schedule+0x23e/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054436]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054452]  [<ffffffff816ebede>] ? __schedule+0x23e/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054467]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054475]  [<ffffffff816ebede>] ? __schedule+0x23e/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054480]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054485]  [<ffffffff816ebede>] ? __schedule+0x23e/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054490]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054495]  [<ffffffff816ebede>] ? __schedule+0x23e/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054500]  [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054508]  [<ffffffffa06037c0>] ? dlm_kick_thread+0x80/0x80 [ocfs2_dlm]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054515]  [<ffffffff810a68fb>] kthread+0xcb/0xf0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054520]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054525]  [<ffffffff816ebeea>] ? __schedule+0x24a/0x810

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054531]  [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054536]  [<ffffffff816f2081>] ret_from_fork+0x61/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054541]  [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054547] INFO: task python:4500 blocked for more than 120 seconds.

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054745]       Not tainted 4.1.12-124.21.1.el6uek.x86_64 #2

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055341] python          D ffff880094e18480     0  4500      1 0x00000080

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055347]  ffff88008acdf898 0000000000000282 ffff88008acdf978 ffff880004fd2a00

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055353]  ffff88008acdf8b8 ffff88008ace0000 ffff88008acdf9d8 ffff88008acdf9b0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055358]  ffffffffa05ca9e0 ffff88008acdf9c8 ffff88008acdf8b8 ffffffff816ec4e7

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055363] Call Trace:

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055371]  [<ffffffff816ec4e7>] schedule+0x37/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055381]  [<ffffffffa05bfcbd>] o2net_send_message_vec+0x64d/0xb20 [ocfs2_nodemanager]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055390]  [<ffffffff81013310>] ? xen_smp_send_reschedule+0x10/0x20

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055397]  [<ffffffff810b09de>] ? resched_curr+0x7e/0xf0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055417]  [<ffffffffa05ff2d9>] ? __dlm_lookup_lockres_full+0xa9/0x160 [ocfs2_dlm]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055431]  [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055447]  [<ffffffffa05c01b9>] o2net_send_message+0x29/0x30 [ocfs2_nodemanager]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055458]  [<ffffffffa061cb03>] dlmlock+0x7a3/0x1720 [ocfs2_dlm]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055466]  [<ffffffffa04b6341>] o2cb_dlm_lock+0x61/0x90 [ocfs2_stack_o2cb]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055491]  [<ffffffffa04b6000>] ? 0xffffffffa04b6000

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055497]  [<ffffffffa04b6020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055505]  [<ffffffffa05683bb>] ocfs2_dlm_lock+0x2b/0x50 [ocfs2_stackglue]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055550]  [<ffffffffa069c7d8>] ocfs2_lock_create+0xa8/0x2f0 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055586]  [<ffffffffa0699cd2>] ? ocfs2_should_refresh_lock_res+0xe2/0x170 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055600]  [<ffffffff810209e9>] ? read_tsc+0x9/0x10

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055623]  [<ffffffffa069cf04>] ocfs2_file_lock+0x114/0x6a0 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055647]  [<ffffffffa069ab3e>] ? ocfs2_inode_lock_update+0x7e/0x640 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055660]  [<ffffffff816f11f8>] ? _raw_spin_lock_irqsave+0x28/0x110

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055689]  [<ffffffffa06bab6c>] ocfs2_do_flock.isra.4+0xbc/0x1b0 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055702]  [<ffffffff81124152>] ? from_kgid_munged+0x12/0x20

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055709]  [<ffffffff81210e34>] ? cp_new_stat+0x144/0x160

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055719]  [<ffffffff811eb682>] ? kmem_cache_alloc+0x222/0x250

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055751]  [<ffffffffa06bacc0>] ocfs2_flock+0x60/0xe0 [ocfs2]

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055758]  [<ffffffff8125f1d6>] ? locks_alloc_lock+0x66/0x70

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055762]  [<ffffffff81262170>] SyS_flock+0x110/0x1b0

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055767]  [<ffffffff816f1b5f>] ? system_call_after_swapgs+0xe9/0x190

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055772]  [<ffffffff816f1b58>] ? system_call_after_swapgs+0xe2/0x190

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055776]  [<ffffffff816f1b51>] ? system_call_after_swapgs+0xdb/0x190

Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055781]  [<ffffffff816f1c1e>] system_call_fastpath+0x18/0xd7

--More--

Aug 12 17:05:57 ESF-PORA01-TTO rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2902" x-info="http://www.rsyslog.com"] start

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] PAT configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Initializing cgroup subsys cpuset

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Initializing cgroup subsys cpu

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Initializing cgroup subsys cpuacct

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Linux version 4.1.12-124.21.1.el6uek.x86_64 ([email protected]) (gcc version 4.9.2 20150212 (Red Hat 4.9.2-6.2.0.3) (GCC) ) #2

SMP Tue Nov 6 13:31:13 PST 2018

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Command line: placeholder root=UUID=b1d48647-6e83-49eb-b678-2b9411bd9a14 ro rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-

sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Released 0 page(s)

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] e820: BIOS-provided physical RAM map:

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x000000000009d400-0x00000000000fffff] reserved

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x0000000000100000-0x0000000099062fff] usable

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000bddac000-0x00000000bddddfff] ACPI data

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000bddde000-0x00000000cfffffff] reserved

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000eeffe000-0x00000000eeffefff] reserved

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000fbefe000-0x00000000fbefefff] reserved

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000feefffff] reserved

Aug 12 17:05:57 ESF-PORA01-TTO kernel: [    0.000000] Xen: [mem 0x00000000ff800000-0x00000000ffffffff] reserved