1 Reply Latest reply: May 5, 2012 11:35 AM by Pierluigi Vernetto RSS

    managed server goes unresponsive-Thread dumps are taken

    630703
      Hi,

      Can someone please help me to read below thread dumps?I am having some serious performance issues and would like to debug using thread dumps...


      Here is an a thread dump from my app:

      ===== FULL THREAD DUMP ===============
      Wed May 2 10:48:14 2012
      Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64

      "Main Thread" id=1 idx=0x4 tid=1521 prio=5 alive, waiting, native_blocked
      -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x96694c98[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:979)
      ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x96694c98[fat lock]
      at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:488)
      at weblogic/Server.main(Server.java:71)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "(Signal Handler)" id=2 idx=0x8 tid=1522 prio=5 alive, native_blocked, daemon

      "(OC Main Thread)" id=3 idx=0xc tid=1523 prio=5 alive, native_waiting, daemon

      "(GC Worker Thread 1)" id=? idx=0x10 tid=1524 prio=5 alive, daemon

      "(GC Worker Thread 2)" id=? idx=0x14 tid=1525 prio=5 alive, daemon

      "(GC Worker Thread 3)" id=? idx=0x18 tid=1526 prio=5 alive, daemon

      "(GC Worker Thread 4)" id=? idx=0x1c tid=1527 prio=5 alive, daemon

      "(Code Generation Thread 1)" id=4 idx=0x20 tid=1528 prio=5 alive, native_waiting, daemon

      "(Code Optimization Thread 1)" id=5 idx=0x24 tid=1529 prio=5 alive, native_waiting, daemon

      "(VM Periodic Task)" id=6 idx=0x28 tid=1530 prio=10 alive, native_blocked, daemon

      "Finalizer" id=7 idx=0x2c tid=1531 prio=8 alive, native_waiting, daemon
      at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
          at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
          at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
          at java/lang/Thread.run(Thread.java:662)
          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
          -- end of trace

      "Reference Handler" id=8 idx=0x30 tid=1532 prio=10 alive, native_waiting, daemon
          at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
          at java/lang/ref/Reference.access$100(Reference.java:11)
          at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
          -- end of trace

      "(Sensor Event Thread)" id=9 idx=0x34 tid=1533 prio=5 alive, native_blocked, daemon

      "VM JFR Buffer Thread" id=10 idx=0x38 tid=1534 prio=5 alive, in native, daemon

      "Timer-0" id=13 idx=0x3c tid=1535 prio=5 alive, waiting, native_blocked, daemon
          -- Waiting for notification on: java/util/TaskQueue@0x95085298[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at java/util/TimerThread.mainLoop(Timer.java:483)
      ^-- Lock released while waiting: java/util/TaskQueue@0x95085298[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "Timer-1" id=14 idx=0x40 tid=1538 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0x95085668[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/util/TimerThread.mainLoop(Timer.java:509)
      ^-- Lock released while waiting: java/util/TaskQueue@0x95085668[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x44 tid=1540 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0x966e2668[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
      at java/lang/Object.wait(Object.java:485)[inlined]
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:160)[optimized]
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x966e2668[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "JFR request timer" id=16 idx=0x48 tid=1541 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: java/util/TaskQueue@0x95dba800[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at java/util/TimerThread.mainLoop(Timer.java:483)
      ^-- Lock released while waiting: java/util/TaskQueue@0x95dba800[fat lock]
      at java/util/TimerThread.run(Timer.java:462)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "weblogic.time.TimeEventGenerator" id=18 idx=0x4c tid=1542 prio=9 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x95610650[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
      ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x95610650[fat lock]
      at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "JMAPI event thread" id=19 idx=0x50 tid=1543 prio=5 alive, in native, daemon

      "weblogic.timers.TimerThread" id=20 idx=0x54 tid=1544 prio=9 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x95ab28b8[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
      ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x95ab28b8[fat lock]
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x58 tid=1545 prio=5 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: weblogic/work/ExecuteThread@0x966e2750[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
      at java/lang/Object.wait(Object.java:485)[inlined]
      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:160)[optimized]
      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x966e2750[fat lock]
      at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x5c tid=1546 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0x95a23108[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[inlined]
      at jrockit/vm/Locks.fatLockReacquire(Locks.java:1649)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1517)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x60 tid=1547 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0x95a23108[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=1548 prio=5 alive, blocked, native_blocked, daemon
      -- Blocked trying to get lock: java/lang/String@0x95a23108[fat lock]
      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
      at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
      at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x68 tid=1549 prio=5 alive, in native, daemon
      at jrockit/ext/epoll/EPoll.epollWait0(ILjava/nio/ByteBuffer;II)I(Native Method)
      at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:115)
      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
      ^-- Holding lock: java/lang/String@0x95a23108[fat lock]
      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      "VDE Transaction Processor Thread" id=30 idx=0x6c tid=1552 prio=2 alive, waiting, native_blocked, daemon
      -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x966e2838[fat lock]
      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
      at java/lang/Object.wait(J)V(Native Method)
      at java/lang/Object.wait(Object.java:485)
      at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
      ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x966e2838[fat lock]
      at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace
        • 1. Re: managed server goes unresponsive-Thread dumps are taken
          Pierluigi Vernetto
          I can't see anything justifying a dead server.... I would strongly recommend to profile your server with yourkit (in the worst case yyou can use jvisualvm, and profile the hot methods to see who is hogging all the CPU). Incidentally, are you experiencing high cpu? how many threads are running ? The thread dump you have posted doesn't seem complete...normally you have at least 35-40 threads running in WLS...