This discussion is archived
4 Replies Latest reply: Feb 16, 2011 7:50 AM by 735887 RSS

Intermittent problem instantiating PublishingCacheStore

704908 Newbie
Currently Being Moderated
Hi Guys,

I get an intermittent issue starting my cluster with the push replication 3.0.3.20019 and Coherence 3.6.1:
2011-01-05 13:03:22,689 ERROR lccohd2-1 [${build.version}] Log4j [Logger@9240861 n/a] - 306.578 <Error> (thread=DistributedCache:QTG-Distributed-Main, member=7):
(Wrapped: Missing or inaccessible constructor "com.oracle.coherence.patterns.pushreplication.PublishingCacheStore(String)"
<class-scheme>
  <class-name>com.oracle.coherence.patterns.pushreplication.PublishingCacheStore</class-name>
  <init-params>
    <init-param>
      <param-type>java.lang.String</param-type>
      <param-value>publishing-distributed-identifiable-ServiceStatus</param-value>
    </init-param>
  </init-params>
</class-scheme>) java.lang.reflect.InvocationTargetException
        at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
        at com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2542)
        at com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2426)
        at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateAny(DefaultConfigurableCacheFactory.java:3256)
        at com.oracle.coherence.environment.extensible.ExtensibleEnvironment.instantiateAny(ExtensibleEnvironment.java:720)
        at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateCacheStore(DefaultConfigurableCacheFactory.java:3106)
        at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateReadWriteBackingMap(DefaultConfigurableCacheFactory.java:1674)
        at com.tangosol.net.DefaultConfigurableCacheFactory.configureBackingMap(DefaultConfigurableCacheFactory.java:1429)
        at com.tangosol.net.DefaultConfigurableCacheFactory$Manager.instantiateBackingMap(DefaultConfigurableCacheFactory.java:3904)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.instantiateResourceMap(PartitionedCache.CDB:22)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.setCacheName(PartitionedCache.CDB:25)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ServiceConfig$ConfigListener.entryInserted(PartitionedCache.CDB:17)
        at com.tangosol.util.MapEvent.dispatch(MapEvent.java:266)
        at com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
        at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556)
        at com.tangosol.util.ObservableHashMap.dispatchEvent(ObservableHashMap.java:229)
        at com.tangosol.util.ObservableHashMap$Entry.onAdd(ObservableHashMap.java:270)
        at com.tangosol.util.SafeHashMap.put(SafeHashMap.java:244)
        at com.tangosol.coherence.component.util.ServiceConfig$Map.updateInternal(ServiceConfig.CDB:30)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$ConfigUpdate.onReceived(Grid.CDB:42)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:11)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.tangosol.util.ClassHelper.newInstance(ClassHelper.java:662)
        at com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2501)
        ... 24 more
Caused by: (Wrapped) java.lang.InterruptedException: sleep interrupted
        at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
        at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
        at com.tangosol.coherence.component.net.Cluster.ensureStopped(Cluster.CDB:17)
        at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:21)
        at com.tangosol.coherence.component.util.SafeCluster.getRunningCluster(SafeCluster.CDB:1)
        at com.tangosol.coherence.component.util.SafeCluster.getClusterName(SafeCluster.CDB:1)
        at com.oracle.coherence.patterns.pushreplication.PublishingCacheStore.<init>(PublishingCacheStore.java:174)
        ... 30 more
Caused by: java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at com.tangosol.coherence.component.net.Cluster.ensureStopped(Cluster.CDB:12)
        ... 34 more
Here is a filtered list of threads:
2011-01-05 13:03:20,732 ERROR lccohd2-1 [${build.version}] Log4j [Logger@9240861 n/a] - 304.629 <Error> (thread=Cluster, member=7): Detected hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:DistributedCacheForMessages} Service=PartitionedCache{Name=DistributedCacheForMessages, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=12, BackupPartitions=0}}
2011-01-05 13:03:20,748 ERROR lccohd2-1 [${build.version}] Log4j [Logger@9240861 n/a] - 304.643 <Error> (thread=Termination Thread, member=7): Full Thread Dump

Thread[DistributedCacheForSequenceGeneratorsWorker:3,5,DistributedCacheForSequenceGenerators]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForSubscriptionsWorker:3,5,DistributedCacheForSubscriptions]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[Invocation:Management:EventDispatcher,5,Cluster]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForLiveObjectsWorker:7,5,DistributedCacheForLiveObjects]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForSubscriptionsWorker:2,5,DistributedCacheForSubscriptions]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[PublishingService:Thread-6,5,PublishingService]
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:990)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892)
        com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:330)
        com.oracle.coherence.patterns.pushreplication.publishers.supervision.InvocationServiceResourceSupervisor.ensureResource(InvocationServiceResourceSupervisor.java:64)
        com.oracle.coherence.patterns.pushreplication.publishers.supervision.InvocationServiceResourceSupervisor.ensureResource(InvocationServiceResourceSupervisor.java:37)
        com.oracle.coherence.patterns.pushreplication.publishers.supervision.AbstractResourceSupervisor.getResource(AbstractResourceSupervisor.java:78)
        com.oracle.coherence.patterns.pushreplication.publishers.RemoteClusterPublisher.start(RemoteClusterPublisher.java:166)
        com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService.onStart(CoherencePublishingService.java:690)
        com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService$1.run(CoherencePublishingService.java:499)
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        java.util.concurrent.FutureTask.run(FutureTask.java:138)
        java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForSequenceGeneratorsWorker:5,5,DistributedCacheForSequenceGenerators]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[Signal Dispatcher,9,system]

Thread[DistributedCacheForSequenceGeneratorsWorker:8,5,DistributedCacheForSequenceGenerators]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCache:QTG-Distributed-Main,5,Cluster]
        java.lang.Object.wait(Native Method)
        com.tangosol.util.SegmentedConcurrentMap$LockableEntry.waitForNotify(SegmentedConcurrentMap.java:654)
        com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:219)
        com.tangosol.net.internal.ScopedReferenceStore.lock(ScopedReferenceStore.java:268)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:260)
        com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)
        com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
        com.oracle.coherence.patterns.messaging.DefaultMessagingSession.createDestination(DefaultMessagingSession.java:124)
        com.oracle.coherence.patterns.messaging.DefaultMessagingSession.createTopic(DefaultMessagingSession.java:86)
        com.oracle.coherence.patterns.messaging.DefaultMessagingSession.createTopic(DefaultMessagingSession.java:72)
        com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePushReplicationProvider.establishPublishingInfrastructureFor(CoherencePushReplicationProvider.java:120)
        com.oracle.coherence.patterns.pushreplication.PublishingCacheStore.<init>(PublishingCacheStore.java:196)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        com.tangosol.util.ClassHelper.newInstance(ClassHelper.java:615)
        com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2501)
        com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2426)
        com.tangosol.net.DefaultConfigurableCacheFactory.instantiateAny(DefaultConfigurableCacheFactory.java:3256)
        com.oracle.coherence.environment.extensible.ExtensibleEnvironment.instantiateAny(ExtensibleEnvironment.java:720)
        com.tangosol.net.DefaultConfigurableCacheFactory.instantiateCacheStore(DefaultConfigurableCacheFactory.java:3106)
        com.tangosol.net.DefaultConfigurableCacheFactory.instantiateReadWriteBackingMap(DefaultConfigurableCacheFactory.java:1674)
        com.tangosol.net.DefaultConfigurableCacheFactory.configureBackingMap(DefaultConfigurableCacheFactory.java:1429)
        com.tangosol.net.DefaultConfigurableCacheFactory$Manager.instantiateBackingMap(DefaultConfigurableCacheFactory.java:3904)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.instantiateResourceMap(PartitionedCache.CDB:22)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.setCacheName(PartitionedCache.CDB:25)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ServiceConfig$ConfigListener.entryInserted(PartitionedCache.CDB:17)
        com.tangosol.util.MapEvent.dispatch(MapEvent.java:266)
        com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
        com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556)
        com.tangosol.util.ObservableHashMap.dispatchEvent(ObservableHashMap.java:229)
        com.tangosol.util.ObservableHashMap$Entry.onAdd(ObservableHashMap.java:270)
        com.tangosol.util.SafeHashMap.put(SafeHashMap.java:244)
        com.tangosol.coherence.component.util.ServiceConfig$Map.updateInternal(ServiceConfig.CDB:30)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$ConfigUpdate.onReceived(Grid.CDB:42)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:11)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
        java.lang.Thread.run(Thread.java:619)

Thread[Cluster|Member(Id=7, Timestamp=2011-01-05 12:58:19.558, Address=192.168.15.102:8088, MachineId=26470, Location=site:london,machine:lccohd2,process:23548,member:lccohd2-1, Role=LiquidcapitalQtgEmsServer),5,Cluster]
        sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForSequenceGeneratorsWorker:9,5,DistributedCacheForSequenceGenerators]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[QTG-Distributed-MainWorker:0,5,QTG-Distributed-Main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[QTG-Distributed-MainWorker:4,5,QTG-Distributed-Main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCache:DistributedCacheForMessages,5,Cluster]
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:990)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892)
        com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:874)
        com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1231)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290)
        com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)

        com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
        com.oracle.coherence.patterns.messaging.MessageEventManager.<init>(MessageEventManager.java:83)
        com.oracle.coherence.patterns.messaging.MessageEventManager.<clinit>(MessageEventManager.java:70)
        com.oracle.coherence.patterns.messaging.MessagePartitionListener.onEvent(MessagePartitionListener.java:76)
        com.oracle.coherence.patterns.messaging.MessagePartitionListener.onPartitionEvent(MessagePartitionListener.java:56)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.firePartitionEvent(PartitionedService.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.firePartitionEvent(PartitionedService.CDB:13)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onTransferRequest(PartitionedService.CDB:168)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$TransferRequest.onReceived(PartitionedService.CDB:1)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:11)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForLiveObjectsWorker:3,5,DistributedCacheForLiveObjects]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[QTG-Distributed-MainWorker:6,5,QTG-Distributed-Main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForSubscriptionsWorker:8,5,DistributedCacheForSubscriptions]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForMessagesWorker:3,5,DistributedCacheForMessages]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCacheForDestinationsWorker:3,5,DistributedCacheForDestinations]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[QTG-Distributed-MainWorker:7,5,QTG-Distributed-Main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[main,5,main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:18)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:28)
        com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)
        com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
        com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:1057)
        com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892)
        com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497)
        com.tangosol.net.DefaultCacheServer.startServices(DefaultCacheServer.java:81)
        com.tangosol.net.DefaultCacheServer.start(DefaultCacheServer.java:230)
        com.liquidcapital.qtg.core.server.AbstractQTGServer.start(AbstractQTGServer.java:104)
        com.liquidcapital.qtg.EmsServer.main(EmsServer.java:42)

Thread[DistributedCacheForDestinationsWorker:1,5,DistributedCacheForDestinations]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[QTG-Distributed-MainWorker:1,5,QTG-Distributed-Main]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)

Thread[DistributedCache:DistributedCacheForLiveObjects,5,Cluster]
        java.lang.Object.wait(Native Method)
        com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
        com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
        com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
        java.lang.Thread.run(Thread.java:619)
Has anyone else seen this?

Thanks, Paul
  • 1. Re: Intermittent problem instantiating PublishingCacheStore
    704908 Newbie
    Currently Being Moderated
    More of stack:
    Thread[DistributedCacheForDestinationsWorker:2,5,DistributedCacheForDestinations]
            java.lang.Object.wait(Native Method)
            com.tangosol.util.SegmentedConcurrentMap$LockableEntry.waitForNotify(SegmentedConcurrentMap.java:654)
            com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:219)
            com.tangosol.net.internal.ScopedReferenceStore.lock(ScopedReferenceStore.java:268)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:260)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
            com.oracle.coherence.patterns.messaging.Destination.subscribe(Destination.java:265)
            com.oracle.coherence.patterns.messaging.Topic.subscribe(Topic.java:74)
            com.oracle.coherence.patterns.messaging.entryprocessors.TopicSubscribeProcessor.process(TopicSubscribeProcessor.java:117)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.invoke(PartitionedCache.CDB:6)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:48)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)
            com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
            com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    
    Thread[DistributedCache:DistributedCacheForDestinations,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:1,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[QTG-Distributed-MainWorker:8,5,QTG-Distributed-Main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[Invocation:Management,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PublishingService:Thread-7,5,PublishingService]
            com.oracle.coherence.patterns.pushreplication.publishers.supervision.AbstractResourceSupervisor.getResource(AbstractResourceSupervisor.java:73)
            com.oracle.coherence.patterns.pushreplication.publishers.RemoteClusterPublisher.start(RemoteClusterPublisher.java:166)
            com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService.onStart(CoherencePublishingService.java:690)
            com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService$1.run(CoherencePublishingService.java:499)
            java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            java.util.concurrent.FutureTask.run(FutureTask.java:138)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
            java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:4,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:1,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    
    Thread[PacketPublisher,6,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:5,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:9,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForLiveObjects:EventDispatcher,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForMessagesWorker:1,5,DistributedCacheForMessages]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:2,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForSubscriptions,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:1,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:8,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:4,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForSubscriptions:EventDispatcher,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[CacheAppender,5,main]
            com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:995)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:923)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892)
            com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:874)
            com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1231)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
            com.liquidcapital.qtg.common.utils.logging.CacheAppender$EventQueueWorker.run(CacheAppender.java:174)
            java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            java.util.concurrent.FutureTask.run(FutureTask.java:138)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
            java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:0,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PacketListenerN,8,Cluster]
            java.net.PlainDatagramSocketImpl.receive0(Native Method)
            java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            java.net.DatagramSocket.receive(DatagramSocket.java:712)
            com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
            com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForDestinations:EventDispatcher,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:6,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForDestinationsWorker:0,5,DistributedCacheForDestinations]
            java.lang.Object.wait(Native Method)
            com.tangosol.util.SegmentedConcurrentMap$LockableEntry.waitForNotify(SegmentedConcurrentMap.java:654)
            com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:219)
            com.tangosol.net.internal.ScopedReferenceStore.lock(ScopedReferenceStore.java:268)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:260)
    
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
            com.oracle.coherence.patterns.messaging.Destination.subscribe(Destination.java:265)
            com.oracle.coherence.patterns.messaging.Topic.subscribe(Topic.java:74)
            com.oracle.coherence.patterns.messaging.entryprocessors.TopicSubscribeProcessor.process(TopicSubscribeProcessor.java:117)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.invoke(PartitionedCache.CDB:6)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:48)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)
            com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
            com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[IpMonitor,6,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.IpMonitor.onWait(IpMonitor.CDB:4)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
  • 2. Re: Intermittent problem instantiating PublishingCacheStore
    704908 Newbie
    Currently Being Moderated
    More of stack:
    Thread[PacketListener1,8,Cluster]
            java.net.PlainDatagramSocketImpl.receive0(Native Method)
            java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            java.net.DatagramSocket.receive(DatagramSocket.java:712)
            com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
            com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForSequenceGenerators,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:2,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForLiveObjectsWorker:6,5,DistributedCacheForLiveObjects]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForMessagesWorker:2,5,DistributedCacheForMessages]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:5,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:7,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[FIXME:TcpInitiator:EventDispatcher,5,PublishingService]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:6,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[FIXME:TcpInitiator:TcpReader,5,PublishingService]
            java.net.SocketInputStream.socketRead0(Native Method)
            java.net.SocketInputStream.read(SocketInputStream.java:129)
            java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
            java.io.BufferedInputStream.read(BufferedInputStream.java:237)
            java.io.DataInputStream.readUnsignedByte(DataInputStream.java:271)
            com.tangosol.util.ExternalizableHelper.readInt(ExternalizableHelper.java:538)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.initiator.TcpInitiator$TcpConnection$TcpReader.onNotify(TcpInitiator.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PacketSpeaker,8,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
            com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
            com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForSequenceGenerators:EventDispatcher,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[FIXME:TcpInitiator,5,PublishingService]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[Termination Thread,5,Cluster]
            java.lang.Thread.dumpThreads(Native Method)
            java.lang.Thread.getAllStackTraces(Thread.java:1477)
            com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
            com.tangosol.coherence.component.net.Cluster$DefaultFailurePolicy.onGuardableTerminate(Cluster.CDB:4)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.terminate(Grid.CDB:1)
            com.tangosol.net.GuardSupport$Context$2.run(GuardSupport.java:677)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[Reference Handler,10,system]
            java.lang.Object.wait(Native Method)
            java.lang.Object.wait(Object.java:485)
            java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    Thread[DistributedCacheForSubscriptionsWorker:0,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[QTG-Distributed-MainWorker:2,5,QTG-Distributed-Main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:4,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCache:DistributedCacheForMessages:EventDispatcher,5,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[QTG-Distributed-MainWorker:3,5,QTG-Distributed-Main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSequenceGeneratorsWorker:0,5,DistributedCacheForSequenceGenerators]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[EventRouter-QueueStatistics,5,main]
            sun.misc.Unsafe.park(Native Method)
            java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
            java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
            java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
            java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
            java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
            java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForSubscriptionsWorker:7,5,DistributedCacheForSubscriptions]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PacketListener1P,8,Cluster]
            java.net.PlainDatagramSocketImpl.receive0(Native Method)
            java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            java.net.DatagramSocket.receive(DatagramSocket.java:712)
            com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
            com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PacketReceiver,7,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[DistributedCacheForMessagesWorker:0,5,DistributedCacheForMessages]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[QTG-Distributed-MainWorker:5,5,QTG-Distributed-Main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:9)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:619)
    
    Thread[PublishingService:Thread-5,5,PublishingService]
            com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:995)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:923)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:892)
            com.oracle.coherence.environment.extensible.ExtensibleEnvironment.ensureService(ExtensibleEnvironment.java:497)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:874)
            com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1231)
            com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:735)
            com.tangosol.net.CacheFactory.getCache(CacheFactory.java:712)
            com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService.onPublish(CoherencePublishingService.java:947)
            com.oracle.coherence.patterns.pushreplication.providers.coherence.CoherencePublishingService$5.run(CoherencePublishingService.java:702)
            java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            java.util.concurrent.FutureTask.run(FutureTask.java:138)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
            java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            java.lang.Thread.run(Thread.java:619)
  • 3. Re: Intermittent problem instantiating PublishingCacheStore
    648232 Explorer
    Currently Being Moderated
    Hey Paul,

    Reading the stacks off the forums like this is super painful. Is there any chance you could send them to me in an email. My current suspicion is that the PublishingCacheStore is being terminated by the guardian while it's starting up. Are you using the standard push replication as it is provided out of the box, or have you plugged in your own provider implementation, or custom multi-plexing cache store solution? What I'm wondering is if there's anything in your code base that might be taking longer than the guardian is willing to wait.

    -Noah
  • 4. Re: Intermittent problem instantiating PublishingCacheStore
    735887 Newbie
    Currently Being Moderated
    Has this been resolved ? We are experiencing the same problem.
    Thx,
    Steve

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points