1 Reply Latest reply on Sep 23, 2013 7:01 AM by Jonathan.Knight

    Assertion failed: poll() is a blocking call and cannot be called on the Service thread

    user13507523

      Hi

       

      We are getting a strange issue, the application successfully joins the cluster but after start failing with following exception.

      The cluster have three nodes storage disabled web-logic and two standalone coherence JVM's, we are using distributed cache with Local scheme

       

      <Error> (thread=DistributedCache, member=4): Assertion failed: poll() is a blocking call and cannot be called on the Service thread

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:5)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)

      at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)

      at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)

      at com.thehartford.pi.core.referencedata.dao.cachedaoimpl.ReferenceCacheDAOImpl.getReferenceData(Unknown Source)

      at com.thehartford.pi.core.caching.cachestore.ReferenceCacheStore.load(Unknown Source)

      at com.tangosol.net.cache.ReadWriteBackingMap$CacheLoaderCacheStore.load(ReadWriteBackingMap.java:6132)

      at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:5616)

      at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.load(ReadWriteBackingMap.java:4698)

      at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:717)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.get(PartitionedCache.CDB:10)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:23)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)

      at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)

      at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)

      at java.lang.Thread.run(Thread.java:722)

      ]

      ERROR 2013-09-20 09:06:42,515    :  [2013-09-20 09:06:42.515/8740.228 Oracle Coherence GE 3.7.1.0 <Error> (thread=DistributedCache, member=4): Assertion failed: poll() is a blocking call and cannot be called on the Service thread

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:5)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.get(PartitionedCache.CDB:26)

      at com.tangosol.util.ConverterCollections$ConverterMap.get(ConverterCollections.java:1655)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.get(PartitionedCache.CDB:1)

      at com.tangosol.coherence.component.util.SafeNamedCache.get(SafeNamedCache.CDB:1)

      at com.thehartford.pi.core.referencedata.dao.cachedaoimpl.ReferenceCacheDAOImpl.getReferenceData(Unknown Source)

      at com.thehartford.pi.core.caching.cachestore.ReferenceCacheStore.load(Unknown Source)

      at com.tangosol.net.cache.ReadWriteBackingMap$CacheLoaderCacheStore.load(ReadWriteBackingMap.java:6132)

      at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:5616)

      at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.load(ReadWriteBackingMap.java:4698)

      at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:717)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.get(PartitionedCache.CDB:10)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:23)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)

      at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)

      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)

                at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)

        • 1. Re: Assertion failed: poll() is a blocking call and cannot be called on the Service thread
          Jonathan.Knight

          Hi

           

          The problem is that you are making a re-entrant call back into a cache service from the service thread or worker thread of a cache service. This is a bad thing to do as you risk deadlocking your cluster by consuming all of the threads in the service. From the stack trace it looks like you are doing a get on a cache which is calling through to a cache store which is then doing a get on another cache.

           

          For example, you have done a "get" on a cache, that has now consumed a worker thread (call it Thread-1), that thread is calling the cache store which is doing a get on another cache in the same cache service so will now consume another thread (call it Thread-2) so you now have two threads in use, Thread-1 will not return until Thread-2 completes. Now say you had 2 worker threads on your cache service and two "get" calls came in at the same time, Get-1 and Get-2. Both worker threads are now in use so when Get-1 calls the cache store to do a get on the other cache then it has to wait for a worker thread to become free to process the get. The same applies to Get-2, it is calling the cache store and waiting for a thread to become free. The problem is no threads will become free as they are all waiting. Hopefully that is a clear enough explanation of why you get the warning.

           

          Read this Constraints on Re-entrant Calls - 12c (12.1.2) This is for 12.1.2 but the same applies for any Coherence version.

           

          JK