4 Replies Latest reply: Feb 27, 2013 11:24 PM by 993854 RSS

    Weblogic 10mp1: web server holds 1-3 seconds frequently

    993854
      When the server is holding, there is no any log in weblogic access log and application log.
      Use console and found there are many standby thread, ( like total 95, active 5, standby 90).
      All the weblogic configure is default value.

      The dump thread information is the following, please help to the program in the server.
      ----------------------------------------------
      This page displays the current stacks for each thread.





      ===== FULL THREAD DUMP ===============

      Thu Feb 28 10:32:41 2013

      BEA JRockit(R) R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-x86_64

      "Main Thread" id=1 idx=0x4 tid=11258 prio=5 alive, in native, waiting

      -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x14205030[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at java/lang/Object.wait(J)V(Native Method)

      at java/lang/Object.wait(Object.java:474)

      at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:747)

      ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x14205030[fat lock]

      at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:396)

      at weblogic/Server.main(Server.java:67)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "(Signal Handler)" id=2 idx=0x8 tid=11259 prio=5 alive, in native, daemon

      "(GC Main Thread)" id=3 idx=0xc tid=11260 prio=5 alive, in native, native_waiting, daemon

      "(GC Worker Thread 1)" id=? idx=0x10 tid=11261 prio=5 alive, in native, daemon

      "(GC Worker Thread 2)" id=? idx=0x14 tid=11262 prio=5 alive, in native, daemon

      "(GC Worker Thread 3)" id=? idx=0x18 tid=11263 prio=5 alive, in native, daemon

      "(GC Worker Thread 4)" id=? idx=0x1c tid=11264 prio=5 alive, in native, daemon

      "(GC Worker Thread 5)" id=? idx=0x20 tid=11265 prio=5 alive, in native, daemon

      "(GC Worker Thread 6)" id=? idx=0x24 tid=11266 prio=5 alive, in native, daemon

      "(GC Worker Thread 7)" id=? idx=0x28 tid=11267 prio=5 alive, in native, daemon

      "(GC Worker Thread 8)" id=? idx=0x2c tid=11268 prio=5 alive, in native, daemon

      "(GC Worker Thread 9)" id=? idx=0x30 tid=11269 prio=5 alive, in native, daemon

      "(GC Worker Thread 10)" id=? idx=0x34 tid=11270 prio=5 alive, in native, daemon

      "(GC Worker Thread 11)" id=? idx=0x38 tid=11271 prio=5 alive, in native, daemon

      "(GC Worker Thread 12)" id=? idx=0x3c tid=11272 prio=5 alive, in native, daemon

      "(GC Worker Thread 13)" id=? idx=0x40 tid=11273 prio=5 alive, in native, daemon

      "(GC Worker Thread 14)" id=? idx=0x44 tid=11274 prio=5 alive, in native, daemon

      "(GC Worker Thread 15)" id=? idx=0x48 tid=11275 prio=5 alive, in native, daemon

      "(GC Worker Thread 16)" id=? idx=0x4c tid=11276 prio=5 alive, in native, daemon

      "(Code Generation Thread 1)" id=4 idx=0x50 tid=11277 prio=5 alive, in native, native_waiting, daemon

      "(Code Optimization Thread 1)" id=5 idx=0x54 tid=11278 prio=5 alive, in native, native_waiting, daemon

      "(VM Periodic Task)" id=6 idx=0x58 tid=11279 prio=10 alive, in native, native_blocked, daemon

      "Finalizer" id=7 idx=0x5c tid=11280 prio=8 alive, in native, native_waiting, daemon

      at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
               
                      at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
               
                      at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
               
                      at java/lang/Thread.run(Thread.java:595)
               
                      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
               
                      -- end of trace
               
                  "Reference Handler" id=8 idx=0x60 tid=11281 prio=10 alive, in native, native_waiting, daemon
               
                      at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
               
                      at java/lang/ref/Reference.access$100(Reference.java:11)
               
                      at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
               
                      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
               
                      -- end of trace
               
                  "(Sensor Event Thread)" id=9 idx=0x64 tid=11282 prio=5 alive, in native, daemon
               
                  "Timer-0" id=12 idx=0x68 tid=11283 prio=5 alive, in native, waiting, daemon
               
                      -- Waiting for notification on: java/util/TaskQueue@0x114c1140[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at java/lang/Object.wait(J)V(Native Method)

      at java/lang/Object.wait(Object.java:474)

      at java/util/TimerThread.mainLoop(Timer.java:483)

      ^-- Lock released while waiting: java/util/TaskQueue@0x114c1140[fat lock]

      at java/util/TimerThread.run(Timer.java:462)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "Timer-1" id=13 idx=0x6c tid=11284 prio=5 alive, in native, waiting, daemon

      -- Waiting for notification on: java/util/TaskQueue@0x114c34c8[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at java/lang/Object.wait(J)V(Native Method)[optimized]

      at java/util/TimerThread.mainLoop(Timer.java:509)

      ^-- Lock released while waiting: java/util/TaskQueue@0x114c34c8[fat lock]

      at java/util/TimerThread.run(Timer.java:462)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=14 idx=0x70 tid=11285 prio=5 alive, in native, waiting, daemon

      -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b80a0[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

      at java/lang/Object.wait(Object.java:474)[inlined]

      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b80a0[fat lock]

      at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "weblogic.time.TimeEventGenerator" id=15 idx=0x74 tid=11286 prio=9 alive, in native, waiting, daemon

      -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x11c53d78[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at java/lang/Object.wait(J)V(Native Method)[optimized]

      at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)

      ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x11c53d78[fat lock]

      at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)

      at java/lang/Thread.run(Thread.java:595)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "JMAPI event thread" id=16 idx=0x78 tid=11287 prio=5 alive, in native, native_waiting, daemon

      "weblogic.timers.TimerThread" id=17 idx=0x7c tid=11288 prio=9 alive, in native, waiting, daemon

      -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x141b7830[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at java/lang/Object.wait(J)V(Native Method)[optimized]

      at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:265)

      ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x141b7830[fat lock]

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=18 idx=0x80 tid=11289 prio=5 alive, in native, waiting, daemon

      -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b8178[fat lock]

      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

      at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

      at java/lang/Object.wait(Object.java:474)[inlined]

      at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

      ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b8178[fat lock]

      at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "Thread-7" id=21 idx=0x84 tid=11290 prio=5 alive, in native, parked, daemon

      at jrockit/vm/Locks.park0(J)V(Native Method)

      at jrockit/vm/Locks.park(Locks.java:2490)

      at sun/misc/Unsafe.park(ZJ)V(Native Method)

      at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)

      at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)

      at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)

      at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)

      at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)

      at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)

      at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=22 idx=0x88 tid=11291 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x8c tid=11292 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x90 tid=11293 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x94 tid=11294 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x98 tid=11295 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x9c tid=11296 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '6' for queue: 'weblogic.socket.Muxer'" id=28 idx=0xa0 tid=11297 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'" id=29 idx=0xa4 tid=11298 prio=5 alive, in native, blocked, daemon

      -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

      at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

      at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

      at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

      at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

      at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

      at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

      at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

      at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace
        • 1. Re: Weblogic 10mp1: web server holds 1-3 seconds frequently
          993854
          "ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'" id=30 idx=0xa8 tid=11299 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '9' for queue: 'weblogic.socket.Muxer'" id=31 idx=0xac tid=11300 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 prio=5 alive, in native, daemon

          at jrockit/ext/epoll/EPoll.epollWait0(IJII)I(Native Method)

          at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:103)

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)

          ^-- Holding lock: java/lang/String@0x1418e4a8[fat lock]

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '11' for queue: 'weblogic.socket.Muxer'" id=33 idx=0xb4 tid=11302 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '12' for queue: 'weblogic.socket.Muxer'" id=34 idx=0xb8 tid=11303 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '13' for queue: 'weblogic.socket.Muxer'" id=35 idx=0xbc tid=11304 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '14' for queue: 'weblogic.socket.Muxer'" id=36 idx=0xc0 tid=11305 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '15' for queue: 'weblogic.socket.Muxer'" id=37 idx=0xc4 tid=11306 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ExecuteThread: '16' for queue: 'weblogic.socket.Muxer'" id=38 idx=0xc8 tid=11307 prio=5 alive, in native, blocked, daemon

          -- Blocked trying to get lock: java/lang/String@0x1418e4a8[fat lock]

          at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)

          at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized]

          at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]

          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

          at jrockit/vm/Locks.monitorEnter(Locks.java:2439)[optimized]

          at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)

          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "VDE Transaction Processor Thread" id=41 idx=0xcc tid=11308 prio=2 alive, in native, waiting, daemon

          -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x141b82d0[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)

          at java/lang/Object.wait(Object.java:474)

          at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)

          ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x141b82d0[fat lock]

          at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "DoSManager" id=43 idx=0xd4 tid=11310 prio=6 alive, in native, sleeping, native_waiting, daemon

          at java/lang/Thread.sleep(J)V(Native Method)

          at com/octetstring/vde/DoSManager.run(DoSManager.java:433)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "Thread-11" id=44 idx=0xd8 tid=11311 prio=5 alive, in native, parked, daemon

          at jrockit/vm/Locks.park0(J)V(Native Method)

          at jrockit/vm/Locks.park(Locks.java:2490)

          at sun/misc/Unsafe.park(ZJ)V(Native Method)

          at java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)

          at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)

          at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)

          at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)

          at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)

          at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)

          at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=45 idx=0xdc tid=11312 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b84c0[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

          at java/lang/Object.wait(Object.java:474)[inlined]

          at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

          ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b84c0[fat lock]

          at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "Thread-13" id=70 idx=0x140 tid=11340 prio=5 alive, in native, daemon

          at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)

          ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x14007970[recursive]

          at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)

          ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x14007970[thin lock]

          at java/net/DatagramSocket.receive(DatagramSocket.java:712)

          ^-- Holding lock: java/net/DatagramPacket@0x1227b6d8[thin lock]

          ^-- Holding lock: java/net/DatagramSocket@0x14007270[thin lock]

          at monfox/toolkit/snmp/engine/UdpTransportProvider.receive(Lmonfox/toolkit/snmp/engine/SnmpBuffer;Z)Lmonfox/toolkit/snmp/engine/TransportEntity;(UdpTransportProvider.java)

          at monfox/toolkit/snmp/engine/d.a()V(d.java)

          ^-- Holding lock: [B@0x11bcf0e8[thin lock]

          at monfox/toolkit/snmp/engine/d.run()V(d.java)

          at java/lang/Thread.run(Thread.java:595)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "Thread-14" id=71 idx=0x144 tid=11341 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: monfox/toolkit/snmp/util/SimpleQueue@0x141bc150[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)[optimized]

          at monfox/toolkit/snmp/util/SimpleQueue.popFront()Ljava/lang/Object;(SimpleQueue.java)

          ^-- Lock released while waiting: monfox/toolkit/snmp/util/SimpleQueue@0x141bc150[fat lock]

          at monfox/toolkit/snmp/mgr/n.a()V(n.java)

          at monfox/toolkit/snmp/mgr/n.run()V(n.java)

          at java/lang/Thread.run(Thread.java:595)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "TrapInformProcessor0" id=72 idx=0x148 tid=11342 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: monfox/toolkit/snmp/util/Queue@0x1415b520[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)[optimized]

          at monfox/toolkit/snmp/util/Queue.get()Lmonfox/toolkit/snmp/util/WorkItem;(Queue.java)

          ^-- Lock released while waiting: monfox/toolkit/snmp/util/Queue@0x1415b520[fat lock]

          at monfox/toolkit/snmp/util/g.run()V(g.java)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ResponseProcessor0" id=73 idx=0x14c tid=11343 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: monfox/toolkit/snmp/util/Queue@0x1415b5b8[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)[optimized]

          at monfox/toolkit/snmp/util/Queue.get()Lmonfox/toolkit/snmp/util/WorkItem;(Queue.java)

          ^-- Lock released while waiting: monfox/toolkit/snmp/util/Queue@0x1415b5b8[fat lock]

          at monfox/toolkit/snmp/util/g.run()V(g.java)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "TimerProcessor0" id=74 idx=0x150 tid=11344 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: monfox/toolkit/snmp/util/TimerQueue@0x1415b648[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)

          at java/lang/Object.wait(Object.java:474)

          at monfox/toolkit/snmp/util/TimerQueue.get()Lmonfox/toolkit/snmp/util/WorkItem;(TimerQueue.java)

          ^-- Lock released while waiting: monfox/toolkit/snmp/util/TimerQueue@0x1415b648[fat lock]

          at monfox/toolkit/snmp/util/g.run()V(g.java)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "PollingProcessor0" id=75 idx=0x154 tid=11345 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: monfox/toolkit/snmp/util/TimerQueue@0x1415b6e0[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)

          at java/lang/Object.wait(Object.java:474)

          at monfox/toolkit/snmp/util/TimerQueue.get()Lmonfox/toolkit/snmp/util/WorkItem;(TimerQueue.java)

          ^-- Lock released while waiting: monfox/toolkit/snmp/util/TimerQueue@0x1415b6e0[fat lock]

          at monfox/toolkit/snmp/util/g.run()V(g.java)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "MasterAgentXServer: null:7091" id=76 idx=0x158 tid=11346 prio=5 alive, in native, daemon

          at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

          at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)

          ^-- Holding lock: java/net/SocksSocketImpl@0x11563568[thin lock]

          at java/net/ServerSocket.implAccept(ServerSocket.java:450)

          at java/net/ServerSocket.accept(ServerSocket.java:421)

          at monfox/toolkit/snmp/agent/x/connection/MasterAgentXServer$ConnectionProcessor.run()V(MasterAgentXServer.java)

          at java/lang/Thread.run(Thread.java:595)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace

          "ClientNotifForwarder-1" id=77 idx=0x15c tid=11347 prio=5 alive, in native, waiting, daemon

          -- Waiting for notification on: com/sun/jmx/remote/internal/ArrayNotificationBuffer@0x141bc1e0[fat lock]

          at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

          at java/lang/Object.wait(J)V(Native Method)[optimized]

          at com/sun/jmx/remote/internal/ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:344)[optimized]

          ^-- Lock released while waiting: com/sun/jmx/remote/internal/ArrayNotificationBuffer@0x141bc1e0[fat lock]

          at com/sun/jmx/remote/internal/ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:122)[optimized]

          at com/sun/jmx/remote/internal/ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:169)[optimized]

          at javax/management/remote/rmi/RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1215)[optimized]

          at javax/management/remote/rmi/RMIConnectionImpl_WLSkel.invoke(I[Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

          at weblogic/rmi/internal/ServerRequest.sendReceive(ServerRequest.java:174)[optimized]

          at weblogic/rmi/internal/BasicRemoteRef.invoke(BasicRemoteRef.java:223)[optimized]

          at javax/management/remote/rmi/RMIConnectionImpl_1001_WLStub.fetchNotifications(JIJ)Ljavax/management/remote/NotificationResult;(Unknown Source)[optimized]

          at javax/management/remote/rmi/RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1285)[inlined]

          at com/sun/jmx/remote/internal/ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:508)[optimized]

          at com/sun/jmx/remote/internal/ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:399)[optimized]

          at com/sun/jmx/remote/internal/ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:83)

          at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

          -- end of trace
          • 2. Re: Weblogic 10mp1: web server holds 1-3 seconds frequently
            993854
            "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=78 idx=0x160 tid=11348 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9410[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9410[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=79 idx=0x164 tid=11349 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b94e8[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b94e8[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "DynamicListenThread[Default[2]]" id=80 idx=0x168 tid=11350 prio=9 alive, in native, daemon

            at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

            at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)

            ^-- Holding lock: java/net/SocksSocketImpl@0x138d9f70[thin lock]

            at java/net/ServerSocket.implAccept(ServerSocket.java:450)

            at java/net/ServerSocket.accept(ServerSocket.java:421)

            at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:523)

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

            at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:166)

            at java/lang/Thread.run(Thread.java:595)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "DynamicListenThread[Default]" id=81 idx=0x16c tid=11351 prio=9 alive, in native, daemon

            at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

            at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)[inlined]

            at java/net/ServerSocket.implAccept(ServerSocket.java:450)[inlined]

            at java/net/ServerSocket.accept(ServerSocket.java:421)[optimized]

            ^-- Holding lock: java/net/SocksSocketImpl@0x138da1f8[thin lock]

            at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:523)[optimized]

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

            at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:166)

            at java/lang/Thread.run(Thread.java:595)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "DynamicListenThread[Default[1]]" id=82 idx=0x170 tid=11352 prio=9 alive, in native, daemon

            at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)

            at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)[inlined]

            at java/net/ServerSocket.implAccept(ServerSocket.java:450)[inlined]

            at java/net/ServerSocket.accept(ServerSocket.java:421)[optimized]

            ^-- Holding lock: java/net/SocksSocketImpl@0x138da320[thin lock]

            at weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:523)[optimized]

            at weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)

            at weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:166)

            at java/lang/Thread.run(Thread.java:595)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x174 tid=11437 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b95c0[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b95c0[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=84 idx=0x178 tid=11438 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9698[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9698[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" id=85 idx=0x17c tid=11441 prio=5 alive, in native, daemon

            at jrockit/net/SocketNativeIO.writeBytesPinned(Ljava/io/FileDescriptor;[BII)V(Native Method)
                     
                            at jrockit/net/SocketNativeIO.socketWrite(SocketNativeIO.java:73)[optimized]

            at java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(SocketOutputStream.java)[inlined]

            at java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:92)[inlined]

            at java/net/SocketOutputStream.write(SocketOutputStream.java:136)[optimized]

            at weblogic/servlet/internal/ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:548)[optimized]

            at weblogic/servlet/internal/ChunkOutput.writeChunks(ChunkOutput.java:505)[inlined]

            at weblogic/servlet/internal/ChunkOutput.flush(ChunkOutput.java:382)[optimized]

            at weblogic/servlet/internal/ChunkOutput.checkForFlush(ChunkOutput.java:469)[inlined]

            at weblogic/servlet/internal/ChunkOutput.writeStream(ChunkOutput.java:423)[optimized]

            at weblogic/servlet/internal/ChunkOutputWrapper.writeStream(ChunkOutputWrapper.java:168)[inlined]

            at weblogic/servlet/internal/ServletOutputStreamImpl.writeStream(ServletOutputStreamImpl.java:503)[inlined]

            at weblogic/servlet/internal/ServletOutputStreamImpl.writeStream(ServletOutputStreamImpl.java:491)[inlined]

            at weblogic/servlet/FileServlet.sendFile(FileServlet.java:407)[inlined]

            at weblogic/servlet/FileServlet.doGetHeadPost(FileServlet.java:231)[optimized]

            at weblogic/servlet/FileServlet.service(FileServlet.java:170)[optimized]

            at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]

            at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)[optimized]

            at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)[inlined]

            at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[optimized]

            at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)[inlined]

            at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3392)[optimized]

            at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]

            at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)[inlined]

            at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2140)[inlined]

            at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2046)[optimized]

            at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)[optimized]

            at weblogic/work/ExecuteThread.execute(ExecuteThread.java:200)[optimized]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:172)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x180 tid=11442 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9848[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9848[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=87 idx=0x184 tid=11443 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9920[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9920[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" id=88 idx=0x188 tid=11445 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b99f8[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b99f8[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" id=89 idx=0x18c tid=11446 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x11939338[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11939338[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=90 idx=0x190 tid=11447 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x1189bc28[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1189bc28[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=91 idx=0x194 tid=11449 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x119956b0[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x119956b0[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" id=92 idx=0x198 tid=11455 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x11c090e8[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11c090e8[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" id=93 idx=0x19c tid=11457 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x1188f610[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1188f610[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" id=94 idx=0x1a0 tid=11460 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x11a6f608[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11a6f608[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" id=95 idx=0x1a4 tid=11464 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x11c0bab0[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11c0bab0[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" id=96 idx=0x1a8 tid=11465 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x11c0a3d8[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11c0a3d8[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace

            "[STANDBY] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" id=97 idx=0x1ac tid=11475 prio=5 alive, in native, waiting, daemon

            -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9ad0[fat lock]

            at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

            at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

            at java/lang/Object.wait(Object.java:474)[inlined]

            at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

            ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9ad0[fat lock]

            at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

            at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

            -- end of trace
            • 3. Re: Weblogic 10mp1: web server holds 1-3 seconds frequently
              993854
              "[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" id=98 idx=0x1b0 tid=11476 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9ba8[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9ba8[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" id=99 idx=0x1b4 tid=11477 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9c80[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9c80[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" id=100 idx=0x1b8 tid=11478 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9d58[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9d58[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace


              "[STANDBY] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" id=101 idx=0x1bc tid=11479 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x141b9e28[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x141b9e28[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" id=102 idx=0x1c0 tid=11480 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x11c6ecc8[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11c6ecc8[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" id=103 idx=0x1c4 tid=11481 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x11cbd678[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11cbd678[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" id=104 idx=0x1c8 tid=11482 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x11cf1ff0[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x11cf1ff0[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" id=105 idx=0x1cc tid=11483 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x1193a368[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1193a368[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" id=106 idx=0x1d0 tid=11484 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x1189cbd0[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1189cbd0[fat lock]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" id=111 idx=0x1d4 tid=707 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x16691c38[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x16691c38[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" id=112 idx=0x1d8 tid=708 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x166969d8[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x166969d8[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" id=113 idx=0x1dc tid=709 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x166a16c8[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x166a16c8[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" id=114 idx=0x1e0 tid=710 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x166a2ab0[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x166a2ab0[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" id=115 idx=0x1e4 tid=1088 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x1628c6b8[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1628c6b8[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" id=116 idx=0x1e8 tid=1089 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x1628a328[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1628a328[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" id=117 idx=0x1ec tid=1090 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x1628d658[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x1628d658[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" id=118 idx=0x1f0 tid=1091 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x16e80b70[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x16e80b70[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" id=119 idx=0x1f4 tid=1092 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x16e76440[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x16e76440[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" id=120 idx=0x1f8 tid=1093 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x16e7a710[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x16e7a710[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace

              "[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" id=121 idx=0x1fc tid=1094 prio=5 alive, in native, waiting, daemon

              -- Waiting for notification on: weblogic/work/ExecuteThread@0x16e84658[fat lock]

              at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

              at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]

              at java/lang/Object.wait(Object.java:474)[inlined]

              at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]

              at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]

              ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x16e84658[fat lock]

              at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

              -- end of trace
              • 4. Re: Weblogic 10mp1: web server holds 1-3 seconds frequently
                993854
                "Store cacheClientIpResult Spool Thread" id=192 idx=0x2fc tid=18337 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheVip3aIPConfig Spool Thread" id=193 idx=0x300 tid=18338 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheFepLoginname0 Spool Thread" id=194 idx=0x304 tid=18339 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheFepIp0 Spool Thread" id=195 idx=0x308 tid=18340 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheFepLoginname1 Spool Thread" id=196 idx=0x30c tid=18341 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheFepBlacklist1 Spool Thread" id=197 idx=0x310 tid=18342 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheEtvVip3a Spool Thread" id=198 idx=0x314 tid=18343 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheFepIpSection1 Spool Thread" id=199 idx=0x318 tid=18344 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheMode Spool Thread" id=200 idx=0x31c tid=18345 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheVip3aFepIPConfig Spool Thread" id=201 idx=0x320 tid=18346 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "Store cacheMini3a Spool Thread" id=202 idx=0x324 tid=18347 prio=5 alive, in native, sleeping, native_waiting, daemon

                at java/lang/Thread.sleep(J)V(Native Method)

                at net/sf/ehcache/store/DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)

                at net/sf/ehcache/store/DiskStore.access$900(DiskStore.java:74)

                at net/sf/ehcache/store/DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-1" id=203 idx=0x328 tid=18348 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb6910[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb6910[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-2" id=204 idx=0x32c tid=18349 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc96f8[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc96f8[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-3" id=205 idx=0x330 tid=18350 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9848[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9848[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-4" id=206 idx=0x334 tid=18351 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb74e8[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb74e8[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-5" id=207 idx=0x338 tid=18352 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9b98[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9b98[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-6" id=208 idx=0x33c tid=18353 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9db0[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9db0[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-7" id=209 idx=0x340 tid=18354 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9ff0[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acc9ff0[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-8" id=210 idx=0x344 tid=18355 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acca348[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1acca348[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-9" id=211 idx=0x348 tid=18356 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1aac2c98[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1aac2c98[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_Worker-10" id=212 idx=0x34c tid=18357 prio=5 alive, in native, waiting

                -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb7438[fat lock]

                at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                at java/lang/Object.wait(J)V(Native Method)[optimized]

                at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)

                ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x1adb7438[fat lock]

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                "DefaultQuartzScheduler_QuartzSchedulerThread" id=213 idx=0x350 tid=18358 prio=5 alive, in native, sleeping

                at java/lang/Thread.sleep(J)V(Native Method)

                at org/quartz/core/QuartzSchedulerThread.run(QuartzSchedulerThread.java:420)

                at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                -- end of trace

                Blocked lock chains

                ===================

                Chain 2:

                "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x8c tid=11292 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 3:

                "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x90 tid=11293 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 4:

                "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x94 tid=11294 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 5:

                "ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x98 tid=11295 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 6:

                "ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x9c tid=11296 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 7:

                "ExecuteThread: '6' for queue: 'weblogic.socket.Muxer'" id=28 idx=0xa0 tid=11297 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 8:

                "ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'" id=29 idx=0xa4 tid=11298 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 9:

                "ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'" id=30 idx=0xa8 tid=11299 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 10:

                "ExecuteThread: '9' for queue: 'weblogic.socket.Muxer'" id=31 idx=0xac tid=11300 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 11:

                "ExecuteThread: '11' for queue: 'weblogic.socket.Muxer'" id=33 idx=0xb4 tid=11302 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 12:

                "ExecuteThread: '12' for queue: 'weblogic.socket.Muxer'" id=34 idx=0xb8 tid=11303 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 13:

                "ExecuteThread: '13' for queue: 'weblogic.socket.Muxer'" id=35 idx=0xbc tid=11304 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 14:

                "ExecuteThread: '14' for queue: 'weblogic.socket.Muxer'" id=36 idx=0xc0 tid=11305 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 15:

                "ExecuteThread: '15' for queue: 'weblogic.socket.Muxer'" id=37 idx=0xc4 tid=11306 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Chain 16:

                "ExecuteThread: '16' for queue: 'weblogic.socket.Muxer'" id=38 idx=0xc8 tid=11307 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 in chain 1

                Open lock chains

                ================

                Chain 1:

                "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=22 idx=0x88 tid=11291 waiting for java/lang/String@0x1418e4a8 held by:

                "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=32 idx=0xb0 tid=11301 (active)

                ===== END OF THREAD DUMP ===============