Forum Stats

  • 3,728,154 Users
  • 2,245,559 Discussions
  • 7,853,357 Comments

Discussions

We are facing jvm hung issue. Please share your observations/root cause/solutions to resolve this is

3548348
3548348 Member Posts: 1

We are facing jvm hung issue. Please share your observations/root cause/solutions to resolve this issue.

Observed that one thread in JVM utilizing CPU 99.9%.

top -H -p <jvm processId>

top - 13:12:59 up 119 days, 19:49,  1 user,  load average: 1.02, 1.18, 1.28

Threads: 3363 total,   1 running, 3362 sleeping,   0 stopped,   0 zombie

%Cpu(s):  2.9 us,  0.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 11741086+total, 51379164 free, 10710248 used, 55321448 buff/cache

KiB Swap:        0 total,        0 free,        0 used. 10082680+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND

12428 root      20   0 28.938g 1.531g  15620 R 99.9  1.4  73:40.88 java

12340 root      20   0 28.938g 1.531g  15620 S  0.0  1.4   0:00.00 java

12346 root      20   0 28.938g 1.531g  15620 S  0.0  1.4   0:02.07 java

If we take core dump for JVM process(when it was in hung state) and using gdb if we observe the stacktrace for high cpu usage thread showing as below.

we are sure that this is not application thread. Suspecting jvm deamon thread causing issue.

#0  0x00007f454a92647e in InstanceKlass::find_instance_method(Symbol*, Symbol*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#1  0x00007f454a7f6ed2 in ClassHierarchyWalker::find_witness_anywhere(Klass*, bool, bool) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#2  0x00007f454a7fa416 in Dependencies::find_unique_concrete_method(Klass*, Method*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#3  0x00007f454a70dd78 in ciMethod::find_monomorphic_target(ciInstanceKlass*, ciInstanceKlass*, ciInstanceKlass*, bool) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#4  0x00007f454a653c45 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#5  0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#6  0x00007f454a65608b in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#7  0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#8  0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#9  0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#10 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#11 0x00007f454a656032 in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#12 0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#13 0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#14 0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#15 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#16 0x00007f454a65608b in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#17 0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#18 0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#19 0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#20 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#21 0x00007f454a656032 in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#22 0x00007f454a657c67 in GraphBuilder::GraphBuilder(Compilation*, IRScope*) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#23 0x00007f454a65ff97 in IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#24 0x00007f454a6600af in IR::IR(Compilation*, ciMethod*, int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#25 0x00007f454a63e64b in Compilation::build_hir() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#26 0x00007f454a63ea0e in Compilation::compile_java_method() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

---Type <return> to continue, or q <return> to quit---

#27 0x00007f454a63eb3e in Compilation::compile_method() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#28 0x00007f454a63eeae in Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#29 0x00007f454a63fc59 in Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#30 0x00007f454a794eaa in CompileBroker::invoke_compiler_on_method(CompileTask*) ()

   from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#31 0x00007f454a795e56 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#32 0x00007f454ad5a543 in JavaThread::thread_main_inner() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#33 0x00007f454ad5a68c in JavaThread::run() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#34 0x00007f454ac0b268 in java_start(Thread*) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#35 0x00007f454babadc5 in start_thread () from /lib64/libpthread.so.0

#36 0x00007f454b3cdced in clone () from /lib64/libc.so.6


We are running jvm as -server flag with below details

Operating System

-----------------------

uname -a

Linux EGSL2C1 3.10.0-327.36.3.el7.x86_64 #1 SMP Thu Oct 20 04:56:07 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[[email protected] kodiak]# rpm -q glibc

glibc-2.17-106.el7_2.8.x86_64

[[email protected] kodiak]# getconf GNU_LIBPTHREAD_VERSION

NPTL 2.17

Command-Line Options

------------------------

$JAVA_HOME/bin/java -server -XX:MaxJavaStackTraceDepth=10 -Xms4096m -Xmx4096m -XX:+UseG1GC  -XX:+DisableExplicitGC -Xnoclassgc -verbose:gc

Environment Variables

------------------------

[[email protected] kodiak]# echo $JAVA_HOME

/usr/java/jdk1.8.0_121

[[email protected] kodiak]# echo $LD_ASSUME_KERNEL

[[email protected] kodiak]# echo $_JAVA_SR_SIGNUM

[[email protected] kodiak]# echo $LD_LIBRARY_PATH

/DG/activeRelease/lib:/opt/ACE_wrappers:/usr/local/BWSIfact/host/linux/lib/release:/usr/local/BWSIgoip/host_libs/lib:/opt/TimesTen/kodiak/lib

[[email protected] kodiak]# echo $LD_PRELOAD

[[email protected] kodiak]# echo $SHELL

/bin/bash

[[email protected] kodiak]# echo $DISPLAY

[[email protected] kodiak]# echo $HOSTTYPE

x86_64

[[email protected] kodiak]# echo $OSTYPE

linux-gnu

[[email protected] kodiak]# echo $ARCH

[[email protected] kodiak]# echo $MACHTYPE

x86_64-redhat-linux-gnu

Tagged:
Sign In or Register to comment.