0 Replies Latest reply: Jul 24, 2013 7:38 AM by MichalWegrzyn RSS

    WebLogic Thin T3 Client stuck threads during Tomcat shutdown/undeploy

    MichalWegrzyn

      Hi,

       

      We read in our web-application (Tomcat 6) messages from Weblogic (10.3.6.0) JMS queue.

      We use basic WebLogic Thin T3 Client. Everything works fine, except that Tomcat shutdown/application undeploy sometimes do not complete (there are Thin T3 client threads in waiting state).

       

      Why threads are still waiting and are not interrupted during Tomcat shutdown process?

      I do not use T3 client classes directly, javax JMS API is used along with these properties:

       

      JNDI_FACTORY=weblogic.jndi.WLInitialContextFactory

      QUEUE_CONNECTION_FACTORY_NAME=jms/ConnectionFactoryXYZ

      QUEUE_NAME=jms/QueueXYZ

      PROVIDER_URL=t3://localhost:16200

      SECURITY_PRINCIPAL=XYZ

      SECURITY_CREDENTIALS=XYZ

       

      Our JMS client is created in Spring scheduler, but spring threads are shutted down properly when Tomcat is destroying servlet context.

       

      EDIT: exception catched in webapp:

       

      java.lang.NullPointerException: null

          at weblogic.kernel.ThreadLocalStack.popAndPeek(ThreadLocalStack.java:131) ~[wlthint3client-10.3.6.0.jar:10.3.6.0]

          at weblogic.protocol.ProtocolStack.pop(ProtocolStack.java:32) ~[wlthint3client-10.3.6.0.jar:10.3.6.0]

          at weblogic.jndi.internal.WLContextImpl.close(WLContextImpl.java:111) ~[wlthint3client-10.3.6.0.jar:10.3.6.0]

          at javax.naming.InitialContext.close(InitialContext.java:531) ~[na:1.6.0_35]

       

      I see in catalina.out logs different kinds of exceptions at different times:

       

      <Tue Jul 23 16:51:10 CEST 2013> <Error> <RMI> <WL-080020> <Exception occurred in the DGC server sweep

      java.lang.NullPointerException.

          at weblogic.rmi.internal.dgc.DGCServerImpl.timerExpired(DGCServerImpl.java:87)

          at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

          at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

       

      and/or:

       

      <Tue Jul 23 11:59:00 CEST 2013> <Info> <RJVM> <WL-000513> <Failure in heartbeat trigger for RJVM: 1895863031932448351S:127.0.0.1:[16200,-1,-1,-1,-1,-1,-1]:base_domain:UCM_server1

      java.lang.NullPointerException.

          at weblogic.protocol.ProtocolManager.getProtocolByIndex(ProtocolManager.java:188)

          at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1679)

          at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:605)

          at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1597)

          at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

          at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

       

      and/or:

       

      Exception in thread "weblogic.timers.TimerThread" java.lang.NullPointerException

          at weblogic.kernel.Kernel.execute(Kernel.java:179)

          at weblogic.kernel.Kernel.execute(Kernel.java:161)

          at weblogic.work.KernelDelegator.schedule(KernelDelegator.java:72)

          at weblogic.timers.internal.TimerManagerFactoryImpl$WorkManagerExecutor.execute(TimerManagerFactoryImpl.java:132)

          at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:641)

          at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:272)

       

      and/or:

       

      Exception in thread "ExecuteThread: '3' for queue: 'default'" java.lang.NoClassDefFoundError: weblogic/kernel/KernelLogger

              at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:155)

              at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

      Caused by: java.lang.ClassNotFoundException: weblogic.kernel.KernelLogger

              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1645)

              at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)

              ... 2 more

      Exception in thread "ExecuteThread: '4' for queue: 'default'" java.lang.NoClassDefFoundError: weblogic/kernel/KernelLogger

              at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:155)

              at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

      Exception in thread "weblogic.timers.TimerThread" java.lang.NullPointerException

              at weblogic.kernel.Kernel.execute(Kernel.java:179)

              at weblogic.kernel.Kernel.execute(Kernel.java:161)

              at weblogic.work.KernelDelegator.schedule(KernelDelegator.java:72)

              at weblogic.timers.internal.TimerManagerFactoryImpl$WorkManagerExecutor.execute(TimerManagerFactoryImpl.java:132)

              at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:641)

              at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:272)

      Exception in thread "weblogic.transaction.TxTimer: '1'" java.lang.NullPointerException

              at weblogic.transaction.internal.TransactionManagerImpl.getTransactionManager(TransactionManagerImpl.java:1588)

              at weblogic.transaction.internal.GenericTimer.run(GenericTimer.java:17)

              at java.lang.Thread.run(Thread.java:662)

       

       

       

      Thread dump:

       

      2013-07-23 13:12:32

      Full thread dump Java HotSpot(TM) Server VM (20.12-b01 mixed mode):

       

      "RMI TCP Connection(idle)" daemon prio=10 tid=0x0ab81400 nid=0x4f12 waiting on condition [0xb1e5b000]

         java.lang.Thread.State: TIMED_WAITING (parking)

          at sun.misc.Unsafe.park(Native Method)

          - parking to wait for  <0xd4e6c268> (a java.util.concurrent.SynchronousQueue$TransferStack)

          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)

          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)

          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

          at java.lang.Thread.run(Thread.java:662)

       

      "RMI TCP Connection(idle)" daemon prio=10 tid=0x08e49000 nid=0x4f11 waiting on condition [0xb145c000]

         java.lang.Thread.State: TIMED_WAITING (parking)

          at sun.misc.Unsafe.park(Native Method)

          - parking to wait for  <0xd4e6c268> (a java.util.concurrent.SynchronousQueue$TransferStack)

          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)

          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)

          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

          at java.lang.Thread.run(Thread.java:662)

       

      "RMI TCP Connection(idle)" daemon prio=10 tid=0x09800000 nid=0x4f03 waiting on condition [0xb1276000]

         java.lang.Thread.State: TIMED_WAITING (parking)

          at sun.misc.Unsafe.park(Native Method)

          - parking to wait for  <0xd4e6c268> (a java.util.concurrent.SynchronousQueue$TransferStack)

          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

          at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)

          at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

          at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)

          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

          at java.lang.Thread.run(Thread.java:662)

       

      "RMI Scheduler(0)" daemon prio=10 tid=0x092ab000 nid=0x391c waiting on condition [0xb14fe000]

         java.lang.Thread.State: TIMED_WAITING (parking)

          at sun.misc.Unsafe.park(Native Method)

          - parking to wait for  <0xd4edc718> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

          at java.lang.Thread.run(Thread.java:662)

       

      "RMI TCP Accept-0" daemon prio=10 tid=0x08af6800 nid=0x3919 runnable [0xb1183000]

         java.lang.Thread.State: RUNNABLE

          at java.net.PlainSocketImpl.socketAccept(Native Method)

          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)

          - locked <0xd4e6c958> (a java.net.SocksSocketImpl)

          at java.net.ServerSocket.implAccept(ServerSocket.java:462)

          at java.net.ServerSocket.accept(ServerSocket.java:430)

          at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)

          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

          at java.lang.Thread.run(Thread.java:662)

       

      "Attach Listener" daemon prio=10 tid=0x09142800 nid=0x3913 waiting on condition [0x00000000]

         java.lang.Thread.State: RUNNABLE

       

      "http-8888-12" daemon prio=10 tid=0x0913b000 nid=0x712e in Object.wait() [0xaf9ad000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4805de0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd4805de0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-11" daemon prio=10 tid=0x0913a400 nid=0x712d in Object.wait() [0xaf9fe000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4805f78> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd4805f78> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-10" daemon prio=10 tid=0x08e4e800 nid=0x712c in Object.wait() [0xb038b000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd48066b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd48066b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-9" daemon prio=10 tid=0x0b9f3000 nid=0x712b in Object.wait() [0xb03dc000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd48068d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd48068d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-8" daemon prio=10 tid=0x0935f000 nid=0x712a in Object.wait() [0xb042d000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480a000> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480a000> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-7" daemon prio=10 tid=0x08e8d800 nid=0x7129 in Object.wait() [0xb047e000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480a228> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480a228> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-6" daemon prio=10 tid=0x092b5c00 nid=0x6e6a in Object.wait() [0xb04d7000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480a4f0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480a4f0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-5" daemon prio=10 tid=0x0b21f800 nid=0x6e69 in Object.wait() [0xb0528000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480a828> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480a828> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-4" daemon prio=10 tid=0x08eca400 nid=0x6e68 in Object.wait() [0xb0579000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480ab60> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480ab60> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-3" daemon prio=10 tid=0x092b7800 nid=0x6e67 in Object.wait() [0xb05ca000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480bde8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480bde8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-2" daemon prio=10 tid=0x0935f800 nid=0x6e66 in Object.wait() [0xb265c000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480c120> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480c120> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-1" daemon prio=10 tid=0x09b0e400 nid=0x6e65 in Object.wait() [0xb26ad000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd480c498> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)

          - locked <0xd480c498> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor8" daemon prio=10 tid=0x08bb2c00 nid=0x6d00 in Object.wait() [0xb06ad000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd47477d0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd47477d0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor7" daemon prio=10 tid=0x08ecb800 nid=0x6cff in Object.wait() [0xb06fe000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd46b31d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd46b31d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor6" daemon prio=10 tid=0x08ebfc00 nid=0x6cfe in Object.wait() [0xb0c5e000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4747af0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd4747af0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor5" daemon prio=10 tid=0x08eb6c00 nid=0x6cfd in Object.wait() [0xb0caf000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4747358> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd4747358> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Monitor" daemon prio=10 tid=0xb2835000 nid=0x6b58 in Object.wait() [0xb0e0b000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd41490d8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)

          at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)

          - locked <0xd41490d8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor4" daemon prio=10 tid=0xb22b2c00 nid=0x6b57 in Object.wait() [0xb0e5c000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4148780> (a org.apache.jk.common.ChannelSocket)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:306)

          - locked <0xd4148780> (a org.apache.jk.common.ChannelSocket)

          at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)

          at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:877)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor3" daemon prio=10 tid=0xb27fb000 nid=0x6b56 in Object.wait() [0xb0ead000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd41655b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd41655b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor2" daemon prio=10 tid=0xb19fa800 nid=0x6b55 in Object.wait() [0xb24ad000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4166b20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd4166b20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "TP-Processor1" daemon prio=10 tid=0xb31fac00 nid=0x6b54 in Object.wait() [0xb24fe000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd41673f0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Object.wait(Object.java:485)

          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)

          - locked <0xd41673f0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)

          at java.lang.Thread.run(Thread.java:662)

       

      "http-8888-Acceptor-0" daemon prio=10 tid=0xb15e4400 nid=0x6b53 waiting on condition [0xb26fe000]

         java.lang.Thread.State: TIMED_WAITING (sleeping)

          at java.lang.Thread.sleep(Native Method)

          at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:344)

          at java.lang.Thread.run(Thread.java:662)

       

      "ExecuteThread: '0' for queue: 'JmsAsyncQueue'" daemon prio=10 tid=0x0a90d800 nid=0x6b51 in Object.wait() [0xb0efe000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4325930> (a weblogic.kernel.ServerExecuteThread)

          at java.lang.Object.wait(Object.java:485)

          at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:91)

          - locked <0xd4325930> (a weblogic.kernel.ServerExecuteThread)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:115)

       

      "weblogic.transaction.TxTimer: '1'" daemon prio=10 tid=0x0b001c00 nid=0x6b50 in Object.wait() [0xb1090000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4376810> (a weblogic.transaction.internal.GenericTimer)

          at weblogic.transaction.internal.GenericTimer.run(GenericTimer.java:15)

          - locked <0xd4376810> (a weblogic.transaction.internal.GenericTimer)

          at java.lang.Thread.run(Thread.java:662)

       

      "weblogic.timers.TimerThread" daemon prio=10 tid=0xb1f35800 nid=0x6b4f in Object.wait() [0xb10e1000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4373de8> (a weblogic.timers.internal.TimerThread)

          at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)

          - locked <0xd4373de8> (a weblogic.timers.internal.TimerThread)

       

      "ExecuteThread: '4' for queue: 'default'" daemon prio=10 tid=0xb1bfb800 nid=0x6b4e in Object.wait() [0xb16fe000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd437cd40> (a weblogic.socket.utils.DynaQueue)

          at weblogic.socket.utils.DynaQueue.getW(DynaQueue.java:237)

          - locked <0xd437cd40> (a weblogic.socket.utils.DynaQueue)

          at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:332)

          at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

       

      "ExecuteThread: '3' for queue: 'default'" daemon prio=10 tid=0xb191f000 nid=0x6b4d in Object.wait() [0xb185c000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd437cd40> (a weblogic.socket.utils.DynaQueue)

          at weblogic.socket.utils.DynaQueue.getW(DynaQueue.java:237)

          - locked <0xd437cd40> (a weblogic.socket.utils.DynaQueue)

          at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:332)

          at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)

          at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

       

      "ExecuteThread: '2' for queue: 'default'" daemon prio=10 tid=0xb0f37c00 nid=0x6b4c in Object.wait() [0xb18ad000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd4328e40> (a weblogic.kernel.ServerExecuteThread)

          at java.lang.Object.wait(Object.java:485)

          at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:91)

          - locked <0xd4328e40> (a weblogic.kernel.ServerExecuteThread)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:115)

       

      "ExecuteThread: '1' for queue: 'default'" daemon prio=10 tid=0xb0fff800 nid=0x6b4b in Object.wait() [0xb18fe000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd432b220> (a weblogic.kernel.ServerExecuteThread)

          at java.lang.Object.wait(Object.java:485)

          at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:91)

          - locked <0xd432b220> (a weblogic.kernel.ServerExecuteThread)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:115)

       

      "ExecuteThread: '0' for queue: 'default'" daemon prio=10 tid=0xb15bf800 nid=0x6b4a in Object.wait() [0xb1ead000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd42f83c0> (a java.lang.Object)

          at java.lang.Object.wait(Object.java:485)

          at weblogic.security.subject.SubjectManager.getSubjectManager(SubjectManager.java:277)

          - locked <0xd42f83c0> (a java.lang.Object)

          at weblogic.security.subject.SubjectManager$GetKernelIdentityAction.run(SubjectManager.java:440)

          at weblogic.security.service.SecurityManager.getKernelIdentity(SecurityManager.java:164)

          at weblogic.security.service.GetKernelIdentityAction.run(GetKernelIdentityAction.java:25)

          at java.security.AccessController.doPrivileged(Native Method)

          at weblogic.rmi.internal.dgc.DGCClientImpl$HeartBeat.<init>(DGCClientImpl.java:186)

          at weblogic.rmi.internal.dgc.DGCClientImpl.mark(DGCClientImpl.java:171)

          at weblogic.rmi.internal.dgc.DGCClientImpl.timerExpired(DGCClientImpl.java:136)

          at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

          at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

       

      "GC Daemon" daemon prio=10 tid=0x08cc2800 nid=0x6b2c in Object.wait() [0xb2ae9000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd3770128> (a sun.misc.GC$LatencyLock)

          at sun.misc.GC$Daemon.run(GC.java:100)

          - locked <0xd3770128> (a sun.misc.GC$LatencyLock)

       

      "Low Memory Detector" daemon prio=10 tid=0xb319c000 nid=0x6b2a runnable [0x00000000]

         java.lang.Thread.State: RUNNABLE

       

      "C2 CompilerThread1" daemon prio=10 tid=0xb3199400 nid=0x6b29 waiting on condition [0x00000000]

         java.lang.Thread.State: RUNNABLE

       

      "C2 CompilerThread0" daemon prio=10 tid=0xb3197400 nid=0x6b28 waiting on condition [0x00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Signal Dispatcher" daemon prio=10 tid=0xb3195c00 nid=0x6b27 runnable [0x00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Finalizer" daemon prio=10 tid=0xb3187c00 nid=0x6b26 in Object.wait() [0xb2e95000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd3770880> (a java.lang.ref.ReferenceQueue$Lock)

          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

          - locked <0xd3770880> (a java.lang.ref.ReferenceQueue$Lock)

          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

       

      "Reference Handler" daemon prio=10 tid=0xb3186800 nid=0x6b25 in Object.wait() [0xb2ee6000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd3770118> (a java.lang.ref.Reference$Lock)

          at java.lang.Object.wait(Object.java:485)

          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

          - locked <0xd3770118> (a java.lang.ref.Reference$Lock)

       

      "main" prio=10 tid=0x08ad8400 nid=0x6b1f in Object.wait() [0xf6914000]

         java.lang.Thread.State: WAITING (on object monitor)

          at java.lang.Object.wait(Native Method)

          - waiting on <0xd42f83c0> (a java.lang.Object)

          at java.lang.Object.wait(Object.java:485)

          at weblogic.security.subject.SubjectManager.getSubjectManager(SubjectManager.java:277)

          - locked <0xd42f83c0> (a java.lang.Object)

          at weblogic.security.subject.SubjectManager$GetKernelIdentityAction.run(SubjectManager.java:440)

          at weblogic.security.service.SecurityManager.getKernelIdentity(SecurityManager.java:164)

          at weblogic.security.service.GetKernelIdentityAction.run(GetKernelIdentityAction.java:25)

          at java.security.AccessController.doPrivileged(Native Method)

          at weblogic.transaction.internal.TransactionImpl.<clinit>(TransactionImpl.java:76)

          at sun.misc.Unsafe.ensureClassInitialized(Native Method)

          at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:25)

          at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:122)

          at java.lang.reflect.Field.acquireFieldAccessor(Field.java:918)

          at java.lang.reflect.Field.getFieldAccessor(Field.java:899)

          at java.lang.reflect.Field.get(Field.java:358)

          at org.apache.catalina.loader.WebappClassLoader.clearReferencesStaticFinal(WebappClassLoader.java:2031)

          at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1894)

          at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1798)

          at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:738)

          at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4812)

          - locked <0xd38f03a0> (a org.apache.catalina.core.StandardContext)

          at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:924)

          at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1319)

          at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1290)

          at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:323)

          at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)

          at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1086)

          - locked <0xd3785900> (a org.apache.catalina.core.StandardHost)

          at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)

          - locked <0xd3785770> (a org.apache.catalina.core.StandardEngine)

          at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:450)

          at org.apache.catalina.core.StandardService.stop(StandardService.java:587)

          - locked <0xd3785770> (a org.apache.catalina.core.StandardEngine)

          at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)

          at org.apache.catalina.startup.Catalina.stop(Catalina.java:648)

          at org.apache.catalina.startup.Catalina.start(Catalina.java:615)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

          at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

       

      "VM Thread" prio=10 tid=0xb3182800 nid=0x6b24 runnable

       

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x08adf800 nid=0x6b21 runnable

       

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x08ae1000 nid=0x6b22 runnable

       

      "GC task thread#2 (ParallelGC)" prio=10 tid=0x08ae2400 nid=0x6b23 runnable

       

      "VM Periodic Task Thread" prio=10 tid=0xb31ae400 nid=0x6b2b waiting on condition

       

      JNI global references: 1761