0 Replies Latest reply: Oct 28, 2013 4:57 AM by 41e7f3c5-8dd0-4935-be0c-83f0dc839e33 RSS

    ExtendTcpProxyService:TcpInitiator BLOCKED

    41e7f3c5-8dd0-4935-be0c-83f0dc839e33

      Hi

      From the logs it looks like the extend client did not recover due to some reason, the client runs on websphere.

      Following are the logs from the client:

       

      2013-10-22 12:54:55,025 DEBUG [ner : 17] WebEventsController: GetNewEvents : 44483

      2013-10-22 12:54:55,147 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 0

        .... java.lang.Object.wait(Native Method)

        .... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)

        .... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)

        .... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)

        .... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)

        .... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)

        .... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)

        .... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)

        .... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)

        .... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        .... java.lang.reflect.Method.invoke(Method.java:597)

        .... org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)

        .... org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)

        .... org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)

        .... org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)

        .... org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)

        .... org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)

        .... org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)

        .... org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)

        .... org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

        .... org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

        .... javax.servlet.http.HttpServlet.service(HttpServlet.java:738)

        .... javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

        .... com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)

        .... com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)

        .... testerpack.servlet.ajax.filter.SecurityTokenFilter.doFilter(SecurityTokenFilter.java:160)

        .... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)

        .... testerpack.servlet.ajax.filter.PostTrailerFilter.doFilter(PostTrailerFilter.java:63)

        .... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)

        .... testerpack.servlet.ajax.filter.ThreadTimeoutFilter.doFilter(ThreadTimeoutFilter.java:49)

        .... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)

        .... com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)

        .... com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)

        .... com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)

        .... com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)

        .... com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)

        .... com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)

        .... com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)

        .... com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)

        .... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)

        .... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)

        .... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)

        .... com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)

        .... com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)

        .... com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)

        .... com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)

        .... com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)

        .... com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

      2013-10-22 12:54:55,224 DEBUG [ner : 14] WebEventsController: GetNewEvents : 15153

      2013-10-22 12:54:55,310 DEBUG [ner : 27] WebEventsController: GetNewEvents : 28368

      2013-10-22 12:54:55,483 DEBUG [ner : 23] WebEventsController: GetNewEvents : 61628

      2013-10-22 12:54:55,660 DEBUG [ner : 26] WebEventsController: GetNewEvents : 95366

      2013-10-22 12:54:55,684 DEBUG [ner : 22] WebEventsController: GetNewEvents : 10776

      2013-10-22 12:54:56,004 DEBUG [iner : 2] WebEventsController: GetNewEvents : 47801

      2013-10-22 12:54:56,207 DEBUG [ner : 25] WebEventsController: GetNewEvents : 388

      2013-10-22 12:54:56,437 DEBUG [ner : 17] WebEventsController: GetNewEvents : 35509

      2013-10-22 12:54:56,502 DEBUG [ner : 14] WebEventsController: GetNewEvents : 48807

      2013-10-22 12:54:56,738 DEBUG [ner : 27] WebEventsController: GetNewEvents : 97692

      2013-10-22 12:54:56,878 DEBUG [ner : 26] WebEventsController: GetNewEvents : 95317

      2013-10-22 12:54:56,909 DEBUG [ner : 22] WebEventsController: GetNewEvents : 40687

      2013-10-22 12:54:57,080 DEBUG [iner : 2] WebEventsController: GetNewEvents : 40692

      2013-10-22 12:54:57,110 DEBUG [ner : 21] WebEventsController: GetNewEvents : 97487

      2013-10-22 12:54:57,530 DEBUG [ner : 17] WebEventsController: GetNewEvents : 42191

      2013-10-22 12:54:57,620 DEBUG [ner : 14] WebEventsController: GetNewEvents : 104758

      2013-10-22 12:54:58,024 DEBUG [ner : 27] WebEventsController: GetNewEvents : 44483

      2013-10-22 12:54:58,231 DEBUG [ner : 23] WebEventsController: GetNewEvents : 15153

      2013-10-22 12:54:58,245 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 20

        .... java.lang.Object.wait(Native Method)

        .... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)

        .... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)

        .... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)

        .... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)

        .... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)

        .... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)

        .... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)

        .... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)

        .... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      2013-10-22 12:54:58,320 DEBUG [ner : 26] WebEventsController: GetNewEvents : 28368

      2013-10-22 12:54:58,463 DEBUG [ner : 22] WebEventsController: GetNewEvents : 61628

        .... java.lang.Object.wait(Native Method)

        .... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)

        .... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)

        .... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)

        .... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)

        .... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)

        .... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)

        .... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)

        .... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)

        .... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)

        .... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        .... java.lang.reflect.Method.invoke(Method.java:597)

      2013-10-22 12:54:59,226 DEBUG [ner : 27] WebEventsController: GetNewEvents : 388

      2013-10-22 12:54:59,459 DEBUG [ner : 23] WebEventsController: GetNewEvents : 35509

      2013-10-22 12:54:59,523 DEBUG [ner : 26] WebEventsController: GetNewEvents : 48807

      10-22 12:55:58,954 INFO  [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107085,STATUS=VALIDATED] for user 84558

      2013-10-22 12:55:58,954 INFO  [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107121,STATUS=VALIDATED] for user 84558

      2013-10-22 12:55:58,954 INFO  [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107106,STATUS=STALE_PRICE] for user 84558

      2013-10-22 12:55:58,954 INFO  [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107179,STATUS=VALIDATED] for user 84558

      2013-10-22 12:55:58,955 INFO  [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107218,STATUS=VALIDATED] for user 84558

      2013-10-22 12:55:59,194 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 2... java.lang.Object.wait(Native Method)

        .... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)

        .... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:1)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.query(RemoteNamedCache.CDB:36)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.entrySet(RemoteNamedCache.CDB:6)

        .... com.tangosol.util.ConverterCollections$ConverterQueryMap.entrySet(ConverterCollections.java:2508)

        .... com.tangosol.util.ConverterCollections$ConverterNamedCache.entrySet(ConverterCollections.java:2800)

        .... com.tangosol.coherence.component.net.extend.RemoteNamedCache.entrySet(RemoteNamedCache.CDB:22)

        .... com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)

        .... com.tangosol.net.cache.NearCache.entrySet(NearCache.java:399)

        .... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:396)

        .... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)

        .... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)

        .... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)

       

      2013-10-22 12:55:59,206 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "WebContainer : 28" daemon prio=5 Thread id=1537 BLOCKED

        .... WebContainer : 28 testerpack.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.refresh(BlotterEventDispatcher.java:157)

        .... WebContainer : 28 testerpack.set.eventgateway.web.impl.WebSessionManager.refresh(WebSessionManager.java:353)

        .... WebContainer : 28 testerpack.set.eventgateway.web.controller.HomeController.handleRequest(HomeController.java:126)

        .... WebContainer : 28 sun.reflect.GeneratedMethodAccessor651.invoke(Unknown Source)

        .... WebContainer : 28 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        .... WebContainer : 28 java.lang.reflect.Method.invoke(Method.java:597)

       

      2013-10-22 12:55:59,206 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "ExtendTcpProxyService:TcpInitiator" daemon prio=10 Thread id=110 BLOCKED

        .... ExtendTcpProxyService:TcpInitiator java.lang.Object.wait(Native Method)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)

        .... ExtendTcpProxyService:TcpInitiator java.lang.Thread.run(Thread.java:662)

      2013-10-22 12:55:59,420 DEBUG [ner : 32] WebEventsController: GetNewEvents : 47801

      2013-10-22 12:55:59,448 DEBUG [ner : 33] WebEventsController: GetNewEvents : 61628

      2013-10-22 12:55:59,556 DEBUG [ner : 34] WebEventsController: GetNewEvents : 48807

      2013-10-22 12:55:59,598 DEBUG [ner : 35] WebEventsController: GetNewEvents : 388

       

      2013-10-22 12:56:48,720 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 48

        .... sun.misc.Unsafe.park(Native Method)

        .... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)

        .... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)

        .... testerpack.set.eventgateway.web.impl.WebSessionManager.getActiveDispatchersForSession(WebSessionManager.java:367)

        .... testerpack.set.eventgateway.web.impl.MarketWatchSessionManagerImpl.marketWatchSearch(MarketWatchSessionManagerImpl.java:61)

        .... testerpack.set.eventgateway.web.controller.MarketWatchController.loadProducts(MarketWatchController.java:240)

        .... sun.reflect.GeneratedMethodAccessor619.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        .... java.lang.reflect.Method.invoke(Method.java:597)

       

      2013-10-22 12:56:50,252 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "Bus-Remote-__23" daemon prio=5 Thread id=718 BLOCKED

        .... Bus-Remote-__23 java.lang.Object.wait(Native Method)

        .... Bus-Remote-__23 java.lang.Object.wait(Object.java:485)

        .... Bus-Remote-__23 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)

        .... Bus-Remote-__23 com.hsoft.util.HSKeyFifo.popChunk(HSKeyFifo.java:202)

        .... Bus-Remote-__23 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:237)

      2013-10-22 12:56:50,261 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "TF-session-7" daemon prio=5 Thread id=1330 BLOCKED

        .... TF-session-7 java.lang.Object.wait(Native Method)

        .... TF-session-7 java.lang.Object.wait(Object.java:485)

        .... TF-session-7 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)

        .... TF-session-7 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:219)

      2013-10-22 12:56:50,261 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "TF-session-9" daemon prio=5 Thread id=1399 BLOCKED

        .... TF-session-9 java.lang.Object.wait(Native Method)

        .... TF-session-9 java.lang.Object.wait(Object.java:485)

        .... TF-session-9 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)

        .... TF-session-9 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:219)

      2013-10-22 12:56:50,261 INFO  [thTiFiMg] ThreadTimeoutFilterManager: "ExtendTcpProxyService:TcpInitiator" daemon prio=10 Thread id=110 BLOCKED

        .... ExtendTcpProxyService:TcpInitiator java.lang.Object.wait(Native Method)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)

        .... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)

        .... ExtendTcpProxyService:TcpInitiator java.lang.Thread.run(Thread.java:662)

      2013-10-22 12:56:53,941 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 49

        .... sun.misc.Unsafe.park(Native Method)

        .... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)

        .... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)

        .... testerpack.set.eventgateway.web.impl.WebSessionManager.refresh(WebSessionManager.java:345)

        .... testerpack.set.eventgateway.web.controller.HomeController.handleRequest(HomeController.java:126)

        .... sun.reflect.GeneratedMethodAccessor651.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       

      2013-10-22 12:56:54,975 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 50

      .... sun.misc.Unsafe.park(Native Method)

        .... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)

        .... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)

        .... testerpack.set.eventgateway.web.impl.WebSessionManager.getNewEvents(WebSessionManager.java:204)

        .... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:62)

        .... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:76)

        .... sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)

       

      2013-10-22 12:56:55,233 WARN  [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 51

        .... sun.misc.Unsafe.park(Native Method)

        .... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)

        .... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)

        .... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)

        .... testerpack.set.eventgateway.web.impl.WebSessionManager.getNewEvents(WebSessionManager.java:204)

        .... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:62)

        .... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:76)

        .... sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)

        .... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       

       

      The following are the logs from the cache server:

      Around the same time the cache server logs had the following:

      DEBUG [2013-10-22 12:54:17,984] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7580

      DEBUG [2013-10-22 12:54:17,984] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7580

      DEBUG [2013-10-22 12:54:17,985] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing

      DEBUG [2013-10-22 12:54:17,985] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing

      INFO  [2013-10-22 12:54:19,995] Coherence - 2013-10-22 12:54:19.995/98189.315 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 218259 bytes; the target was 2000000 bytes.

      INFO  [2013-10-22 12:54:58,914] Coherence - 2013-10-22 12:54:58.913/98228.233 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 10001 messages (338987 bytes) behind; the threshold is 10000 messages or 10000000 bytes.

      INFO  [2013-10-22 12:55:01,915] Coherence - 2013-10-22 12:55:01.915/98231.235 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 438217 bytes; the target was 2000000 bytes.

      INFO  [2013-10-22 12:55:04,930] Coherence - 2013-10-22 12:55:04.930/98234.250 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 15811 messages (534273 bytes) behind; the threshold is 10000 messages or 10000000 bytes.

      INFO  [2013-10-22 12:55:07,931] Coherence - 2013-10-22 12:55:07.931/98237.251 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 617128 bytes; the target was 2000000 bytes.

      INFO  [2013-10-22 12:55:10,937] Coherence - 2013-10-22 12:55:10.937/98240.257 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 20956 messages (706640 bytes) behind; the threshold is 10000 messages or 10000000 bytes.

      INFO  [2013-10-22 12:55:13,939] Coherence - 2013-10-22 12:55:13.939/98243.259 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 795235 bytes; the target was 2000000 bytes.

      INFO  [2013-10-22 12:55:16,940] Coherence - 2013-10-22 12:55:16.940/98246.260 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 13366 messages (445830 bytes) behind; the threshold is 10000 messages or 10000000 bytes.

      INFO  [2013-10-22 12:55:19,941] Coherence - 2013-10-22 12:55:19.941/98249.261 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 59964 bytes; the target was 2000000 bytes.

      INFO  [2013-10-22 12:55:28,569] Coherence - 2013-10-22 12:55:28.569/98257.889 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 10001 messages (336280 bytes) behind; the threshold is 10000 messages or 10000000 bytes.

      INFO  [2013-10-22 12:55:31,570] Coherence - 2013-10-22 12:55:31.570/98260.890 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 441154 bytes; the target was 2000000 bytes.

      DEBUG [2013-10-22 12:57:39,427] b.s.c.l.ProductTradingHourLoader - Loading the TradingHours for 48453

      DEBUG [2013-10-22 12:57:39,427] b.s.c.l.ProductTradingHourLoader - Loading the TradingHours for 48453

       

       

      Around the same time the cache server slave had the following:

      DEBUG [2013-10-22 12:54:18,210] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing

      DEBUG [2013-10-22 12:54:18,210] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing

      WARN  [2013-10-22 12:56:12,247] Coherence - 2013-10-22 12:56:12.247/98300.867 Oracle Coherence GE 3.7.1.9 <Warning> (thread=PacketPublisher, member=1): Experienced a 1693 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 09:38:40.957, Address=127.0.0.1:8090, MachineId=60314, Location=site:,machine:localhost,process:13982); 37 packets rescheduled, PauseRate=4.0E-4, Threshold=1020

      DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7582

      DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7582

      DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing

       

       

      Coherence version: 3.7.1.9

      Coherence client request-timeout: 20 seconds

      Coherence client heartbeat-interval:2m