1 Reply Latest reply on Feb 14, 2020 5:13 PM by nevgeniev

    collect 12.6 coredumps with onload libs

    nevgeniev

      hi,

       

      just wondering if it will be eventually fixed after all these years (Nikmolchanov-Oracle ?)

       

       

      0. here is an output from jdk crash report:

      Stack: [0x00007fd9047e7000,0x00007fd9048e7000],  sp=0x00007fd9048e1a40,  free space=1002k

      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

      C  [libcollector.so+0x493ae]

      C  [libcollector.so+0x4576d]  __collector_get_frame_info+0x66d

      C  [libcollector.so+0x51ea3]

      C  [libcollector.so+0x410d4]

       

      1. dbx doesn't read core dump file and panics

      Corefile specified executable: "/usr/lib/jvm/java-1.8.0-openjdk/bin/java"

      For information about new features see `help changes'

      To remove this message, put `dbxenv suppress_startup_message 8.2' in your .dbxrc

      Reading java

      core file header read successfully

      Reading ld-linux-x86-64.so.2

      Reading libcollector.so

      Reading libonload.so

      Reading libpthread.so.0

      Reading libz.so.1

      dbx: internal error: dwarf global reference attribute out of bounds

      Reading libjli.so

      dbx: panic: "bad DW_FORM_xxx"(): Assertion (0) failed - "dwr_scan.cc":1607

       

      2. output from gdb:

      Using host libthread_db library "/lib64/libthread_db.so.1".

      Core was generated by `/usr/lib/jvm/java-1.8.0-openjdk/bin/java -XX:+UnlockDiagnosticVMOptions -XX:+Pr'.

      Program terminated with signal 6, Aborted.

      #0  0x00007fda2c0322f7 in ioctl () at ../sysdeps/unix/syscall-template.S:81

      81    T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)

      (gdb) where

      #0  0x00007fda2c0322f7 in ioctl () at ../sysdeps/unix/syscall-template.S:81

      #1  <signal handler called>

      #2  0x00007fda2bf73377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

      #3  0x00007fda2bf74a68 in __GI_abort () at abort.c:90

      #4  0x00007fda27b07f69 in os::abort (dump_core=<optimized out>) at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1572

      #5  0x00007fda27d12136 in VMError::report_and_die (this=this@entry=0x7fd9048e11f0)

          at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1107

      #6  0x00007fda27b11fb5 in JVM_handle_linux_signal (sig=11, info=0x7fd9048e15b0, ucVoid=0x7fd9048e1480, abort_if_unrecognized=<optimized out>)

          at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541

      #7  0x00007fda27b05128 in signalHandler (sig=11, info=0x7fd9048e15b0, uc=0x7fd9048e1480)

          at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4552

      #8  0x00007fda2cbc92b8 in citp_signal_run_app_handler (sig=sig@entry=11, info=info@entry=0x7fd9048e15b0, context=context@entry=0x7fd9048e1480)

          at ../../../../../src/lib/transport/ip/signal.c:106

      #9  0x00007fda2cbc954a in citp_signal_run_now (our_info=<optimized out>, context=0x7fd9048e1480, info=0x7fd9048e15b0, signum=11) at ../../../../../src/lib/transport/ip/signal.c:226

      #10 citp_signal_intercept (signum=11, info=0x7fd9048e15b0, context=0x7fd9048e1480) at ../../../../../src/lib/transport/ip/signal.c:263

      #11 <signal handler called>

      #12 0x00007fda2ce3d3ae in ?? () from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

      #13 0x00007fda2ce3976d in __collector_get_frame_info () from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

      #14 0x00007fda2ce45ea3 in ?? () from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

      #15 0x00007fda2ce350d4 in ?? () from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

      #16 0x00007fda2cbc92b8 in citp_signal_run_app_handler (sig=29, info=0x7fd9048e40a0, context=0x7fd9048e3b80) at ../../../../../src/lib/transport/ip/signal.c:106

      #17 0x00007fda2cbc949d in citp_signal_run_pending (our_info=0x7fd9048e5b30) at ../../../../../src/lib/transport/ip/signal.c:154

      #18 0x00007fda2cb78e64 in __citp_exit_lib (do_errno=0, lib_context=0x7fd9048e42e0) at ../../../../../src/lib/transport/unix/internal.h:1000

      #19 citp_epoll_wait (fdi=fdi@entry=0x7fd878100d30, events=events@entry=0x7fd878101030, ordering=ordering@entry=0x0, maxevents=maxevents@entry=1024, timeout=timeout@entry=0,

          sigmask=sigmask@entry=0x0, lib_context=lib_context@entry=0x7fd9048e42e0) at ../../../../../src/lib/transport/unix/epoll_fd.c:2053

      #20 0x00007fda2cb5e227 in onload_epoll_wait (epfd=672, events=0x7fd878101030, maxevents=1024, timeout=0) at ../../../../../src/lib/transport/unix/sockcall_intercept.c:1474

      #21 0x00007fd941c642c4 in _tibPoller_Wait (e=e@entry=0x7fd870004920, __level=__level@entry=2 '\002', poller=0x7fd878100cf0, events=events@entry=0x7fd9048e4418, maxevents=<optimized out>,

          maxevents@entry=0, timeout=timeout@entry=0) at /src/common/lib/c/tibutil/poller.c:1790

      #22 0x00007fd9422548c9 in _tibTransportReceiver_Receive (e=e@entry=0x7fd870004920, __level=__level@entry=1 '\001', receiver=0x7fd937d43770, max=max@entry=100, timeout=timeout@entry=-1,

          callback=callback@entry=0x7fd94223d8b0 <_processPdus>, context=0x7fd9048e4480) at /src/hydra/lib/public/c/tib/transport/receiver.c:518

      #23 0x00007fd942237bf3 in _pduDistributorThread (arg=0x7fd937fe62c0, arg@entry=<error reading variable: value has been optimized out>) at /src/hydra/lib/public/c/tib/pdudist.c:2101

      #24 0x00007fd9424cad5d in jniThreadHook (func=<optimized out>, funcArg=<optimized out>) at /src/hydra/lib/public/java/hydra_native/com_tibco_ftl_jni_NativeFTL.c:62

      #25 0x00007fd941c81116 in _hookWrapper (arg=0x7fd8781084b0) at /src/common/lib/c/tibutil/tthread.c:472

      #26 0x00007fd941c826e7 in __tibThread_Start (closure=0x7fd8781084d0) at /src/common/lib/c/tibutil/tthread.c:319

      #27 0x00007fda2ce1d504 in ?? () from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

      #28 0x00007fda2c93bea5 in start_thread (arg=0x7fd9048e6700) at pthread_create.c:307

      #29 0x00007fda2c03b8cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

        • 1. Re: collect 12.6 coredumps with onload libs
          nevgeniev

          hi,

           

          just another stack trace on subject (if anyone reading)

           

          #0  0x00007f13d33df2f7 in ioctl () at ../sysdeps/unix/syscall-template.S:81

          #1  <signal handler called>

          #2  __GI_abort () at abort.c:126

          #3  0x00007f13ceeb4f69 in os::abort (dump_core=<optimized out>)

              at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1572

          #4  0x00007f13cf0bf136 in VMError::report_and_die (this=this@entry=0x7f11f986eef0)

              at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1107

          #5  0x00007f13ceebefb5 in JVM_handle_linux_signal (sig=11, info=0x7f11f986f2b0, ucVoid=0x7f11f986f180,

              abort_if_unrecognized=<optimized out>)

              at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541

          #6  0x00007f13ceeb2128 in signalHandler (sig=11, info=0x7f11f986f2b0, uc=0x7f11f986f180)

              at /usr/src/debug/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4552

          #7  0x00007f13d3f762b8 in citp_signal_run_app_handler (sig=sig@entry=11, info=info@entry=0x7f11f986f2b0,

              context=context@entry=0x7f11f986f180) at ../../../../../src/lib/transport/ip/signal.c:106

          #8  0x00007f13d3f7654a in citp_signal_run_now (our_info=<optimized out>, context=0x7f11f986f180,

              info=0x7f11f986f2b0, signum=11) at ../../../../../src/lib/transport/ip/signal.c:226

          #9  citp_signal_intercept (signum=11, info=0x7f11f986f2b0, context=0x7f11f986f180)

              at ../../../../../src/lib/transport/ip/signal.c:263

          #10 <signal handler called>

          #11 0x00007f13d41ef358 in __collector_memcpy ()

            from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

          #12 0x00007f13d41e8e3d in ?? ()

            from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

          #13 0x00007f13d41e676d in __collector_get_frame_info ()

            from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

          #14 0x00007f13d41f2ea3 in ?? ()

            from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

          #15 0x00007f13d41e20d4 in ?? ()

            from /opt/OracleDeveloperStudio12.6-linux-x86-bin/developerstudio12.6/bin/../lib/analyzer/amd64/runtime/libcollector.so

          #16 0x00007f13d3f762b8 in citp_signal_run_app_handler (sig=29, info=0x7f11f98724c0, context=0x7f11f9872740)

              at ../../../../../src/lib/transport/ip/signal.c:106

          #17 0x00007f13d3f7649d in citp_signal_run_pending (our_info=0x7f11f9877b30)

              at ../../../../../src/lib/transport/ip/signal.c:154

          #18 0x00007f13d3f765a1 in citp_signal_run_now (our_info=<optimized out>, context=0x7f11f98725c0,

              info=0x7f11f98726f0, signum=27) at ../../../../../src/lib/transport/ip/signal.c:224

          #19 citp_signal_intercept (signum=27, info=0x7f11f98726f0, context=0x7f11f98725c0)

              at ../../../../../src/lib/transport/ip/signal.c:263

          #20 <signal handler called>

          #21 ef10_pio_set_desc (vi=0x7f12d88f78a8, qs=0x7f12cf09b7a4, dma_id=19572, len=794, offset=0, q=0x7f12d88f7940)

              at ../../../../src/lib/ciul/ef10_vi.c:330

          #22 ef10_ef_vi_transmit_copy_pio (vi=0x7f12d88f78a8, offset=0, src_buf=<optimized out>, len=794, dma_id=19572)

              at ../../../../src/lib/ciul/ef10_vi.c:402

          #23 0x00007f13d3f36208 in ci_netif_send (netif=netif@entry=0x7f12d88f7300, pkt=pkt@entry=0x7f12cc8c4000)

              at ../../../../../src/lib/transport/ip/netif_tx.c:195

          #24 0x00007f13d3f6b595 in ci_udp_sendmsg_send (ni=ni@entry=0x7f12d88f7300, us=us@entry=0x7f12cf1bc000,

              pkt=pkt@entry=0x7f12cc8c4000, flags=0, sinf=sinf@entry=0x0)

              at ../../../../../src/lib/transport/ip/udp_send.c:737

          #25 0x00007f13d3f6bac6 in ci_udp_sendmsg_send_async_q (ni=0x7f12d88f7300, us=0x7f12cf1bc000)

              at ../../../../../src/lib/transport/ip/udp_send.c:877

          #26 0x00007f13d3f4c955 in ci_udp_perform_deferred_socket_work (ni=<optimized out>, us=<optimized out>)

              at ../../../../../src/lib/transport/ip/udp_misc.c:59

          #27 0x00007f13d3f2f68d in citp_waitable_deferred_work (ni=ni@entry=0x7f12d88f7300, w=w@entry=0x7f12cf1bc000)

              at ../../../../../src/lib/transport/ip/netif.c:859

          #28 0x00007f13d3f30de1 in ci_netif_lock_or_defer_work (ni=ni@entry=0x7f12d88f7300, w=w@entry=0x7f12cf1bc000)

              at ../../../../../src/lib/transport/ip/netif.c:936

          #29 0x00007f13d3f6c91f in ci_udp_sendmsg_async_q_enqueue (flags=0, pkt=0x7f12cc8c4000, us=0x7f12cf1bc000,

              ni=0x7f12d88f7300) at ../../../../../src/lib/transport/ip/udp_send.c:898

          #30 ci_udp_sendmsg_onload (sinf=0x7f11f98735c0, flags=0, msg=<optimized out>, us=0x7f12cf1bc000,

              ni=0x7f12d88f7300) at ../../../../../src/lib/transport/ip/udp_send.c:1373

          #31 ci_udp_sendmsg (a=a@entry=0x7f11f9873660, msg=<optimized out>, flags=0)

              at ../../../../../src/lib/transport/ip/udp_send.c:1605

          #32 0x00007f13d3f1f5c4 in citp_udp_send (fdinfo=<optimized out>, msg=<optimized out>, flags=<optimized out>)

              at ../../../../../src/lib/transport/unix/udp_fd.c:422

          #33 0x00007f13d3f0a0d7 in onload_sendto (fd=450, msg=0x7f113083f200, len=len@entry=748, flags=flags@entry=0,

              to=to@entry=0x7f11f9873740, tolen=tolen@entry=16)

              at ../../../../../src/lib/transport/unix/sockcall_intercept.c:982

          #34 0x00007f130bc2c1e6 in _tibSocket_SendTo (e=e@entry=0x7f11f9873be0, __level=__level@entry=9 '\t',

              sock=<optimized out>, dst=<optimized out>, dport=<optimized out>, buf=<optimized out>, len=748)

              at /src/common/lib/c/tibutil/socket.c:1097

          #35 0x00007f12cfde763c in _tibMcastSender_SendTo (sendRetries=1, buff=0x7f113083f170, sender=0x7f12d88f4410,

              __level=8 '\b', e=0x7f11f9873be0) at /src/hydra/lib/private/c/mcast/sender.c:1623

          #36 _tibMcastSender_SendNow (e=0x7f11f9873be0, __level=7 '\a', sender=0x7f12d88f4410, buff=0x7f113083f170,

              sendRetries=1) at /src/hydra/lib/private/c/mcast/sender.c:1919

          #37 0x00007f12cfde8c34 in _tibMcastSender_SendRate (e=0x7f11f9873be0, __level=6 '\006', sender=0x7f12d88f4410,

              buff=0x7f113083f170, sendRetries=<optimized out>) at /src/hydra/lib/private/c/mcast/sender.c:1362

          #38 0x00007f12cfde9ee0 in _tibMcastSender_NoFragment (nPdus=<optimized out>, len=726, iovCnt=6,

              sender=0x7f12d88f4410, __level=<optimized out>, e=0x7f12cfdf7580 <__func__.17684>)

              at /src/hydra/lib/private/c/mcast/sender.c:1108

          #39 _tibMcastSender_Send (e=e@entry=0x7f11f9873be0, __level=<optimized out>, ctx=ctx@entry=0x7f12d885e4a0,

              iovCnt=iovCnt@entry=2, nPdus=nPdus@entry=1, totalLen=totalLen@entry=726)

              at /src/hydra/lib/private/c/mcast/sender.c:1324

          #40 0x00007f12cfdc8aea in _tibTransportMcast_Send (sendSid=false, pduList=<optimized out>, ctx=0x7f12d885e4a0,

              __level=3 '\003', e=<optimized out>) at /src/hydra/lib/private/c/mcast/mcast.c:1294

          #41 _tibTransportMcast_SendDesc (e=<optimized out>, __level=2 '\002', transport=0x7f12d885bba0,

              sendDescs=0x7f12d8c4dcc8) at /src/hydra/lib/private/c/mcast/mcast.c:1328

          #42 0x00007f133c5008a3 in _tibTransport_SendDesc (sendDescs=<optimized out>, transport=0x7f12d885bba0,

              __level=1 '\001', e=0x7f11f9873be0) at /src/hydra/header/private/hydra/transport.h:891

          #43 tibPublisher_Send (e=e@entry=0x7f11f9875c20, id=0x310000002d, msg=<optimized out>)

              at /src/hydra/lib/public/c/tib/publish.c:1337

          #44 0x00007f133c7ad6be in Java_com_tibco_ftl_jni_NativePublisher_nativeSend__Lcom_tibco_ftl_Message_2JJ (

              env=0x7f12dad72260, publisher=<optimized out>, message=<optimized out>, msgPeer=<optimized out>,

              peer=<optimized out>) at /src/hydra/lib/public/java/hydra_native/com_tibco_ftl_jni_NativePublisher.c:60

          #45 0x00007f13bb0e8e4a in ?? ()

          #46 0x00000005ea535358 in ?? ()

          #47 0x00000005e7261388 in ?? ()

          #48 0x0000000000000000 in ?? ()