2 Replies Latest reply: Mar 6, 2013 5:45 PM by drakitine RSS

    CoherenceExtend hung requesting data

    994868
      Hello,

      We are experiencing an issue using the Coherence Extend client to connect to single-server-mode cache. Very rarely (have seen the issue twice this year), an entrySet request hangs and does not recover, requiring a JVM restart.

      I will upload the full thread dump , has anyone experienced anything similar? What can cause Service/EventDispatcher to hang? We are using v3.7.1.4

      There is nothing interesting in the coherence-side server logs, it shows an opened TCP Connection. Some of the main logs are pasted below.

      *
      *
      *

      Coherence server
      2013/03/05 04:12:16 | [GC 1006185K->664865K(1046144K), 0.0184637 secs]

      2013/03/05 04:12:27 | 2013-03-05 04:12:27.524/141884.155 Oracle Coherence GE 3.7.1.4 <D6> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=1): Opened: TcpConnection(Id=0x0000013D38C01E040A5E50117FE7478C8ECD7B9D85A5EB12CB93B745CEFC83FA, Open=true, Member(Id=0, Timestamp=2013-03-03 12:57:35.078, Address=10.4.178.39:0, MachineId=0, Location=site:,machine:xx,process:8281), LocalAddress=xx:10495, RemoteAddress=xx:39499)

      2013/03/05 04:12:28 | 2013-03-05 04:12:28.418/141885.049 Oracle Coherence GE 3.7.1.4 <D6> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=1): Opened: Channel(Id=1039812181, Open=true, Connection=0x0000013D38C01E040A5E50117FE7478C8ECD7B9D85A5EB12CB93B745CEFC83FA)

      2013/03/05 04:12:30 | 2013-03-05 04:12:30.176/141886.807 Oracle Coherence GE 3.7.1.4 <D6> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=1): Opened: Channel(Id=999856443, Open=true, Connection=0x0000013D38C01E040A5E50117FE7478C8ECD7B9D85A5EB12CB93B745CEFC83FA)

      2013/03/05 04:14:43 | [GC
      2013/03/05 04:14:43 | 1006113K->664945K(1045760K), 0.0187391 secs]
      2013/03/05 04:17:09 | [GC 1006193K->665073K(1046080K), 0.0175332 secs]



      CoherenceExtend client
      2013-03-05 04:12:40,754 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 724

           .... java.lang.Thread.sleep(Native Method)
           .... java.lang.Thread.sleep(Thread.java:851)
           .... com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
           .... com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
           .... com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
           .... com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
           .... com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
           .... com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
           .... com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)
           .... com.tangosol.net.cache.NearCache.entrySet(NearCache.java:389)

      Edited by: 991865 on 05-Mar-2013 06:19
        • 1. Re: CoherenceExtend hung requesting data
          994868
          [3/5/13 4:57:02:610 GMT] 00000016 SystemOut O 2013-03-05 04:57:02.604/144000.934 Oracle Coherence GE 3.7.1.4 <Error> (thread=ExtendTcpProxyService:TcpInitiator:TcpReader, member=n/a): Full Thread Dump
          Thread[ExtendTcpProxyService:TcpInitiator:EventDispatcher,10,main]
               com.set.eventgateway.impl.MapListenerEventDispatcher.propagateEvent(MapListenerEventDispatcher.java:122)
               com.set.eventgateway.impl.MapListenerEventDispatcher$1.entryUpdated(MapListenerEventDispatcher.java:82)
               com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
               com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
               com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:567)
               com.tangosol.coherence.component.util.SafeNamedCache.translateMapEvent(SafeNamedCache.CDB:7)
               com.tangosol.coherence.component.util.SafeNamedCache.entryUpdated(SafeNamedCache.CDB)
               com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
               com.tangosol.coherence.component.net.extend.RemoteNamedCache$ConverterListener.dispatch(RemoteNamedCache.CDB:5)
               com.tangosol.coherence.component.net.extend.RemoteNamedCache$ConverterListener.entryUpdated(RemoteNamedCache.CDB:1)
               com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
               com.tangosol.coherence.component.util.CacheEvent.run(CacheEvent.CDB:18)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onNotify(Service.CDB:26)
               com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
               java.lang.Thread.run(Thread.java:736)


          Thread[WebContainer : 38,5,main]
               java.lang.Thread.sleep(Native Method)
               java.lang.Thread.sleep(Thread.java:851)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
               com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
               com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
               com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
               com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
               com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)
               com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)
               com.set.eventgateway.impl.search.ProductCacheSearchServiceImpl.search(ProductCacheSearchServiceImpl.java:115)
               com.set.eventgateway.web.impl.MarketWatchSessionManagerImpl.marketWatchSearch(MarketWatchSessionManagerImpl.java:66)
               com.set.eventgateway.web.controller.MarketWatchController.loadProducts(MarketWatchController.java:232)
               sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
               sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
               sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
               java.lang.reflect.Method.invoke(Method.java:611)
               org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)
               org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
               org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
               org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
               org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
               org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
               org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
               org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:831)


          Thread[WebContainer : 1,5,main]
               java.lang.Thread.sleep(Native Method)
               java.lang.Thread.sleep(Thread.java:851)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
               com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
               com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
               com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
               com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
               com.tangosol.coherence.component.util.SafeNamedCache.addMapListener(SafeNamedCache.CDB:27)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1467)
               com.tangosol.net.cache.CachingMap.ensureInvalidationStrategy(CachingMap.java:1583)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1488)
               com.tangosol.net.cache.CachingMap.get(CachingMap.java:487)
               com.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.getAllProducts(BlotterEventDispatcher.java:102)
               com.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.dispatch(BlotterEventDispatcher.java:89)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.propagateEvent(WebUserProfileChangeManager.java:152)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.updateUser(WebUserProfileChangeManager.java:145)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.access$200(WebUserProfileChangeManager.java:42)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager$2.userProfileChanged(WebUserProfileChangeManager.java:100)
               com.set.usergateway.subscription.UserEventSubscriptionChannel.subscribe(UserEventSubscriptionChannel.java:62)
               com.set.usergateway.service.impl.UserSubscriptionServiceImpl.subscribeForUser(UserSubscriptionServiceImpl.java:66)
               com.set.usergateway.impl.UserGatewayImpl.subscribeForUserProfile(UserGatewayImpl.java:65)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.registerUserProfileChangeListener(WebUserProfileChangeManager.java:103)
               com.servlet.ajax.CreateSession.storeUserProfileInSession(CreateSession.java:214)
               com.servlet.ajax.CreateSession.doPost(CreateSession.java:141)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:738)


          Thread[ExtendTcpProxyService:TcpInitiator:TcpReader,10,main]
               java.lang.Thread.getStackTraceImpl(Native Method)
               java.lang.Thread.getStackTrace(Thread.java:1042)
               java.lang.Thread.getAllStackTraces(Thread.java:1070)
               com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:811)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflow(Service.CDB:45)
               com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.post(Peer.CDB:9)
               com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.receive(Peer.CDB:26)
               com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.initiator.TcpInitiator$TcpConnection$TcpReader.onNotify(TcpInitiator.CDB:39)
               com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
               java.lang.Thread.run(Thread.java:736)


          Thread[Logger@798240660 3.7.1.4,3,main]
               java.lang.Object.wait(Native Method)
               java.lang.Object.wait(Object.java:196)
               com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
               com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
               java.lang.Thread.run(Thread.java:736)

          Thread[JIT Compilation Thread,10,system]

          Thread[WebContainer : 160,5,main]
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.registerUserProfileChangeListener(WebUserProfileChangeManager.java:74)
               com.servlet.ajax.CreateSession.storeUserProfileInSession(CreateSession.java:214)
               com.servlet.ajax.CreateSession.doPost(CreateSession.java:141)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
               com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
               com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
               com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
               com.servlet.ajax.filter.SecurityTokenFilter.doFilter(SecurityTokenFilter.java:160)
               com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
               com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
               com.servlet.ajax.filter.PostTrailerFilter.doFilter(PostTrailerFilter.java:73)
               com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
               com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
               com.servlet.ajax.filter.ThreadTimeoutFilter.doFilter(ThreadTimeoutFilter.java:49)
               com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
               com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
               com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
               com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
               com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
               com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
               com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
               com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3944)
               com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
               com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
               com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
               com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
               com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
               com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
               com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
               com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
               com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
               com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
               com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
               com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

          Thread[Gc Slave Thread,5,system]

          Thread[WebContainer : 724,5,main]
               java.lang.Thread.sleep(Native Method)
               java.lang.Thread.sleep(Thread.java:851)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
               com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
               com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
               com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
               com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
               com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)
               com.tangosol.net.cache.NearCache.entrySet(NearCache.java:389)
               com.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.getAllEntries(BlotterEventDispatcher.java:127)
               com.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.dispatch(BlotterEventDispatcher.java:88)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.propagateEvent(WebUserProfileChangeManager.java:152)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.updateUser(WebUserProfileChangeManager.java:145)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.access$200(WebUserProfileChangeManager.java:42)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager$2.userProfileChanged(WebUserProfileChangeManager.java:100)
               com.set.usergateway.subscription.UserEventSubscriptionChannel.subscribe(UserEventSubscriptionChannel.java:62)
               com.set.usergateway.service.impl.UserSubscriptionServiceImpl.subscribeForUser(UserSubscriptionServiceImpl.java:66)
               com.set.usergateway.impl.UserGatewayImpl.subscribeForUserProfile(UserGatewayImpl.java:65)
               com.set.eventgateway.web.impl.WebUserProfileChangeManager.registerUserProfileChangeListener(WebUserProfileChangeManager.java:103)
               com.servlet.ajax.CreateSession.storeUserProfileInSession(CreateSession.java:214)
               com.servlet.ajax.CreateSession.doPost(CreateSession.java:141)
               com.servlet.ajax.CreateSession.doGet(CreateSession.java:67)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:718)

          Thread[WebContainer : 10,5,main]
               java.lang.Thread.sleep(Native Method)
               java.lang.Thread.sleep(Thread.java:851)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
               com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
               com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
               com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
               com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
               com.tangosol.coherence.component.util.SafeNamedCache.addMapListener(SafeNamedCache.CDB:27)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1467)
               com.tangosol.net.cache.CachingMap.ensureInvalidationStrategy(CachingMap.java:1583)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1488)
               com.tangosol.net.cache.CachingMap.getAll(CachingMap.java:655)
               com.set.eventgateway.impl.search.ProductCacheSearchServiceImpl.search(ProductCacheSearchServiceImpl.java:128)
               com.set.eventgateway.web.impl.MarketWatchSessionManagerImpl.marketWatchSearch(MarketWatchSessionManagerImpl.java:66)
               com.set.eventgateway.web.controller.MarketWatchController.loadProducts(MarketWatchController.java:232)
               sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
               sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
               sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
               java.lang.reflect.Method.invoke(Method.java:611)
               org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)
               org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
               org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
               org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
               org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
               org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
               org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
               org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:738)


          Thread[ExtendTcpProxyService:TcpInitiator,10,main]
               java.lang.Object.wait(Native Method)
               java.lang.Object.wait(Object.java:196)
               com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB)
               com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
               java.lang.Thread.run(Thread.java:736)


          Thread[WebContainer : 44,5,main]
               java.lang.Thread.sleep(Native Method)
               java.lang.Thread.sleep(Thread.java:851)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainQueue(Service.CDB:14)
               com.tangosol.coherence.component.util.daemon.queueProcessor.Service.drainEvents(Service.CDB:4)
               com.tangosol.coherence.component.net.extend.RemoteService.drainEvents(RemoteService.CDB:7)
               com.tangosol.coherence.component.util.SafeService.drainEvents(SafeService.CDB:12)
               com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:41)
               com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
               com.tangosol.coherence.component.util.SafeNamedCache.addMapListener(SafeNamedCache.CDB:27)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1467)
               com.tangosol.net.cache.CachingMap.ensureInvalidationStrategy(CachingMap.java:1583)
               com.tangosol.net.cache.CachingMap.registerListener(CachingMap.java:1488)
               com.tangosol.net.cache.CachingMap.getAll(CachingMap.java:655)
               com.set.eventgateway.impl.search.ProductCacheSearchServiceImpl.search(ProductCacheSearchServiceImpl.java:128)
               com.set.eventgateway.web.impl.MarketWatchSessionManagerImpl.marketWatchSearch(MarketWatchSessionManagerImpl.java:66)
               com.set.eventgateway.web.controller.MarketWatchController.loadProducts(MarketWatchController.java:232)
               sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
               sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
               sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
               java.lang.reflect.Method.invoke(Method.java:611)
               org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)
               org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
               org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
               org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
               org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
               org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
               org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
               org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
               org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
               javax.servlet.http.HttpServlet.service(HttpServlet.java:738)

          Edited by: 991865 on 05-Mar-2013 07:16
          • 2. Re: CoherenceExtend hung requesting data
            drakitine
            it looks like threads are waiting for events to be processed and event dispatcher is stuck -

            com.set.eventgateway.impl.MapListenerEventDispatcher.propagateEvent(MapListenerEventDispatcher.java:122)
            com.set.eventgateway.impl.MapListenerEventDispatcher$1.entryUpdated(MapListenerEventDispatcher.java:82)
            com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
            com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
            com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:567)
            com.tangosol.coherence.component.util.SafeNamedCache.translateMapEvent(SafeNamedCache.CDB:7)
            com.tangosol.coherence.component.util.SafeNamedCache.entryUpdated(SafeNamedCache.CDB)
            com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
            com.tangosol.coherence.component.net.extend.RemoteNamedCache$ConverterListener.dispatch(RemoteNamedCache.CDB:5)
            com.tangosol.coherence.component.net.extend.RemoteNamedCache$ConverterListener.entryUpdated(RemoteNamedCache.CDB:1)
            com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
            com.tangosol.coherence.component.util.CacheEvent.run(CacheEvent.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onNotify(Service.CDB:26)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:736)