1 2 Previous Next 19 Replies Latest reply on Jul 27, 2020 8:33 AM by Alexander Pavlik

    kernel:watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [swapper/5:0], OEL 7.7 and Oracle Database 12.1.0.2

    Alexander Pavlik

      Hello

       

      I'm trying to run EBS 12.2 with Oracle database 12.1.0.2 on OEL7.7.

      However intermittently but very frequently CPU usage and load average goes up to incredible numbers.

       

      cpu1.png

       

      Sometimes it happens when I just start database. Sometimes, when I run some EBS job connecting to the database, like adop.

      This virtual machine runs on Xen Server, so in management GUI I observe increased usage for one of the CPUs

       

      cpu2.png

       

      After a while error messages appear in the terminal:

       

      cpu3.png

       

      After a reboot systems reports that something went wrong

       

      cpu4.png

       

      I tried to google by error messages in /var/log/messages, but it didn't help me much.

      Update to latest kernel also didn't help.

      Can anyone please share some ideas?

       

      Thank you

       

      Dec 20 07:57:06 demo kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/10:244:14107]

      Dec 20 07:57:06 demo kernel: Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun xenfs xen_privcmd cmac arc4 ecb md4 nls_utf8 cifs ccm dns_resolver fscache scsi_transport_iscsi ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ppdev aesni_intel crypto_simd glue_helper cryptd sg parport_pc parport i2c_piix4 pcspkr binfmt_misc ip_tables xfs libcrc32c

      Dec 20 07:57:06 demo kernel: sr_mod cdrom ata_generic pata_acpi bochs_drm drm_kms_helper xen_netfront syscopyarea xen_blkfront sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crc32c_intel floppy serio_raw dm_mirror dm_region_hash dm_log dm_mod

      Dec 20 07:57:06 demo kernel: CPU: 10 PID: 14107 Comm: kworker/10:244 Tainted: G             L  4.14.35-1902.8.4.el7uek.x86_64 #2

      Dec 20 07:57:06 demo kernel: Hardware name: Xen HVM domU, BIOS 4.11.1-7.3 05/09/2019

      Dec 20 07:57:06 demo kernel: Workqueue: events netstamp_clear

      Dec 20 07:57:06 demo kernel: task: ffff9c97389e1840 task.stack: ffffbf75509e0000

      Dec 20 07:57:06 demo kernel: RIP: 0010:smp_call_function_many+0x21c/0x27d

      Dec 20 07:57:06 demo kernel: RSP: 0018:ffffbf75509e3cc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff0c

      Dec 20 07:57:06 demo kernel: RAX: 0000000000000008 RBX: ffffffffab0366c0 RCX: ffff9c9c77029be0

      Dec 20 07:57:06 demo kernel: RDX: 0000000000000001 RSI: 0000000000000080 RDI: ffff9c988406c700

      Dec 20 07:57:06 demo kernel: RBP: ffffbf75509e3d00 R08: 000000000000fbff R09: 0000000000000000

      Dec 20 07:57:06 demo kernel: R10: ffff9c988406c700 R11: 0000000000000000 R12: 0000000000000000

      Dec 20 07:57:06 demo kernel: R13: ffff9c9c770a3d80 R14: 0000000000000001 R15: 0000000000000080

      Dec 20 07:57:06 demo kernel: FS:  0000000000000000(0000) GS:ffff9c9c77080000(0000) knlGS:0000000000000000

      Dec 20 07:57:06 demo kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

      Dec 20 07:57:06 demo kernel: CR2: 00005644e4887320 CR3: 000000041e40a004 CR4: 00000000003606e0

      Dec 20 07:57:06 demo kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

      Dec 20 07:57:06 demo kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

      Dec 20 07:57:06 demo kernel: Call Trace:

      Dec 20 07:57:06 demo kernel: ? netif_receive_skb_internal+0x20/0x117

      Dec 20 07:57:06 demo kernel: ? cpumask_weight+0x20/0x11

      Dec 20 07:57:06 demo kernel: ? netif_receive_skb_internal+0x21/0x117

      Dec 20 07:57:06 demo kernel: on_each_cpu+0x2d/0x59

      Dec 20 07:57:06 demo kernel: ? netif_receive_skb_internal+0x20/0x117

      Dec 20 07:57:06 demo kernel: text_poke_bp+0x6a/0xe2

      Dec 20 07:57:06 demo kernel: ? netif_receive_skb_internal+0x20/0x117

      Dec 20 07:57:06 demo kernel: __jump_label_transform.isra.0+0x140/0x19b

      Dec 20 07:57:06 demo kernel: arch_jump_label_transform+0x32/0x43

      Dec 20 07:57:06 demo kernel: __jump_label_update+0x77/0x83

      Dec 20 07:57:06 demo kernel: jump_label_update+0x57/0xa9

      Dec 20 07:57:06 demo kernel: static_key_enable_cpuslocked+0x55/0x7a

      Dec 20 07:57:06 demo kernel: static_key_enable+0x1a/0x22

      Dec 20 07:57:06 demo kernel: netstamp_clear+0x2d/0x37

      Dec 20 07:57:06 demo kernel: process_one_work+0x169/0x399

      Dec 20 07:57:06 demo kernel: worker_thread+0x4d/0x3e5

      Dec 20 07:57:06 demo kernel: kthread+0x105/0x138

      Dec 20 07:57:06 demo kernel: ? rescuer_thread+0x380/0x375

      Dec 20 07:57:06 demo kernel: ? kthread_bind+0x20/0x15

      Dec 20 07:57:06 demo kernel: ret_from_fork+0x24/0x49

      Dec 20 07:57:06 demo kernel: Code: 08 89 c7 e8 e7 d0 71 00 3b 05 f5 a1 51 01 0f 83 6b fe ff ff 48 63 d0 49 8b 4d 00 48 03 0c d5 00 97 27 ac 8b 51 18 83 e2 01 74 0a <f3> 90 8b 51 18 83 e2 01 75 f6 eb c6 0f b6 4d cc 4c 89 e2 48 89

      Dec 20 07:57:07 demo sh: abrt-dump-oops: Found oopses: 1

      Dec 20 07:57:07 demo sh: abrt-dump-oops: Creating problem directories

      Dec 20 07:57:07 demo sh: abrt-dump-oops: Not going to make dump directories world readable because PrivateReports is on

      Dec 20 07:57:08 demo abrt-dump-oops: Reported 1 kernel oopses to Abrt

        1 2 Previous Next