Discussions
Categories
- 17.9K All Categories
- 3.4K Industry Applications
- 3.3K Intelligent Advisor
- 63 Insurance
- 535.8K On-Premises Infrastructure
- 138.1K Analytics Software
- 38.6K Application Development Software
- 5.6K Cloud Platform
- 109.3K Database Software
- 17.5K Enterprise Manager
- 8.8K Hardware
- 71K Infrastructure Software
- 105.2K Integration
- 41.5K Security Software
Restart OVM Server Unknown

Observing the messages, it is found that after an "INFO: task dlm_thread: 4495 blocked for more than 120 seconds" ending with "system_call_fastpath + 0x18 / 0xd7"
the server is restarted affecting the VMs.
Additionally, the VM must be started manually since it does not start automatically.
ug 12 16:56:59 ESF-PORA01-TTO kernel: [5780010.626618] o2net: Connection to node EFS-PORA02-TTO (num 0) at 10.158.16.107:7777 has been idle for 90.273 secs.
Aug 12 16:58:30 ESF-PORA01-TTO kernel: [5780100.739882] o2net: Connection to node EFS-PORA02-TTO (num 0) at 10.158.16.107:7777 has been idle for 90.112 secs.
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053421] INFO: task dlm_thread:4495 blocked for more than 120 seconds.
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053636] Not tainted 4.1.12-124.21.1.el6uek.x86_64 #2
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.053833] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054212] dlm_thread D ffff880094a18480 0 4495 2 0x00000080
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054221] ffff88008a32fad8 0000000000000246 ffff88008a32fbb8 ffff880005861c00
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054227] ffff88008a32faf8 ffff88008a330000 ffff88008a32fc18 ffff88008a32fbf0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054232] ffffffffa05ca9e0 ffff88008a32fc08 ffff88008a32faf8 ffffffff816ec4e7
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054238] Call Trace:
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054262] [<ffffffff816ec4e7>] schedule+0x37/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054284] [<ffffffffa05bfcbd>] o2net_send_message_vec+0x64d/0xb20 [ocfs2_nodemanager]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054323] [<ffffffff810bfb6a>] ? update_curr+0xea/0x190
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054329] [<ffffffff810bc56e>] ? account_entity_dequeue+0xae/0xd0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054336] [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054346] [<ffffffff810f07ab>] ? lock_timer_base.isra.35+0x2b/0x50
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054356] [<ffffffffa05c01b9>] o2net_send_message+0x29/0x30 [ocfs2_nodemanager]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054369] [<ffffffffa06154d3>] dlm_drop_lockres_ref+0xc3/0x260 [ocfs2_dlm]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054375] [<ffffffff816f167b>] ? _raw_spin_unlock_irqrestore+0x2b/0x60
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054380] [<ffffffff810cc034>] ? remove_wait_queue+0x34/0x40
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054390] [<ffffffffa0603ddf>] dlm_thread+0x61f/0x1140 [ocfs2_dlm]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054408] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054421] [<ffffffff816ebede>] ? __schedule+0x23e/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054436] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054452] [<ffffffff816ebede>] ? __schedule+0x23e/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054467] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054475] [<ffffffff816ebede>] ? __schedule+0x23e/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054480] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054485] [<ffffffff816ebede>] ? __schedule+0x23e/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054490] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054495] [<ffffffff816ebede>] ? __schedule+0x23e/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054500] [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054508] [<ffffffffa06037c0>] ? dlm_kick_thread+0x80/0x80 [ocfs2_dlm]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054515] [<ffffffff810a68fb>] kthread+0xcb/0xf0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054520] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054525] [<ffffffff816ebeea>] ? __schedule+0x24a/0x810
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054531] [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054536] [<ffffffff816f2081>] ret_from_fork+0x61/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054541] [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054547] INFO: task python:4500 blocked for more than 120 seconds.
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054745] Not tainted 4.1.12-124.21.1.el6uek.x86_64 #2
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.054943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055341] python D ffff880094e18480 0 4500 1 0x00000080
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055347] ffff88008acdf898 0000000000000282 ffff88008acdf978 ffff880004fd2a00
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055353] ffff88008acdf8b8 ffff88008ace0000 ffff88008acdf9d8 ffff88008acdf9b0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055358] ffffffffa05ca9e0 ffff88008acdf9c8 ffff88008acdf8b8 ffffffff816ec4e7
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055363] Call Trace:
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055371] [<ffffffff816ec4e7>] schedule+0x37/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055381] [<ffffffffa05bfcbd>] o2net_send_message_vec+0x64d/0xb20 [ocfs2_nodemanager]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055390] [<ffffffff81013310>] ? xen_smp_send_reschedule+0x10/0x20
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055397] [<ffffffff810b09de>] ? resched_curr+0x7e/0xf0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055417] [<ffffffffa05ff2d9>] ? __dlm_lookup_lockres_full+0xa9/0x160 [ocfs2_dlm]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055431] [<ffffffff810cc2a0>] ? wait_woken+0x90/0x90
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055447] [<ffffffffa05c01b9>] o2net_send_message+0x29/0x30 [ocfs2_nodemanager]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055458] [<ffffffffa061cb03>] dlmlock+0x7a3/0x1720 [ocfs2_dlm]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055466] [<ffffffffa04b6341>] o2cb_dlm_lock+0x61/0x90 [ocfs2_stack_o2cb]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055491] [<ffffffffa04b6000>] ? 0xffffffffa04b6000
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055497] [<ffffffffa04b6020>] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055505] [<ffffffffa05683bb>] ocfs2_dlm_lock+0x2b/0x50 [ocfs2_stackglue]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055550] [<ffffffffa069c7d8>] ocfs2_lock_create+0xa8/0x2f0 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055586] [<ffffffffa0699cd2>] ? ocfs2_should_refresh_lock_res+0xe2/0x170 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055600] [<ffffffff810209e9>] ? read_tsc+0x9/0x10
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055623] [<ffffffffa069cf04>] ocfs2_file_lock+0x114/0x6a0 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055647] [<ffffffffa069ab3e>] ? ocfs2_inode_lock_update+0x7e/0x640 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055660] [<ffffffff816f11f8>] ? _raw_spin_lock_irqsave+0x28/0x110
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055689] [<ffffffffa06bab6c>] ocfs2_do_flock.isra.4+0xbc/0x1b0 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055702] [<ffffffff81124152>] ? from_kgid_munged+0x12/0x20
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055709] [<ffffffff81210e34>] ? cp_new_stat+0x144/0x160
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055719] [<ffffffff811eb682>] ? kmem_cache_alloc+0x222/0x250
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055751] [<ffffffffa06bacc0>] ocfs2_flock+0x60/0xe0 [ocfs2]
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055758] [<ffffffff8125f1d6>] ? locks_alloc_lock+0x66/0x70
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055762] [<ffffffff81262170>] SyS_flock+0x110/0x1b0
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055767] [<ffffffff816f1b5f>] ? system_call_after_swapgs+0xe9/0x190
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055772] [<ffffffff816f1b58>] ? system_call_after_swapgs+0xe2/0x190
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055776] [<ffffffff816f1b51>] ? system_call_after_swapgs+0xdb/0x190
Aug 12 16:59:01 ESF-PORA01-TTO kernel: [5780132.055781] [<ffffffff816f1c1e>] system_call_fastpath+0x18/0xd7
--More--
Aug 12 17:05:57 ESF-PORA01-TTO rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2902" x-info="http://www.rsyslog.com"] start
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] PAT configuration [0-7]: WB WT UC- UC WC WP UC UC
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Initializing cgroup subsys cpuset
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Initializing cgroup subsys cpu
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Initializing cgroup subsys cpuacct
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Linux version 4.1.12-124.21.1.el6uek.x86_64 ([email protected]) (gcc version 4.9.2 20150212 (Red Hat 4.9.2-6.2.0.3) (GCC) ) #2
SMP Tue Nov 6 13:31:13 PST 2018
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Command line: placeholder root=UUID=b1d48647-6e83-49eb-b678-2b9411bd9a14 ro rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-
sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Released 0 page(s)
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] e820: BIOS-provided physical RAM map:
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x000000000009d400-0x00000000000fffff] reserved
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x0000000000100000-0x0000000099062fff] usable
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000bddac000-0x00000000bddddfff] ACPI data
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000bddde000-0x00000000cfffffff] reserved
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000eeffe000-0x00000000eeffefff] reserved
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000fbefe000-0x00000000fbefefff] reserved
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000feefffff] reserved
Aug 12 17:05:57 ESF-PORA01-TTO kernel: [ 0.000000] Xen: [mem 0x00000000ff800000-0x00000000ffffffff] reserved