Hi,
I'm trying to establish federated caching between two clusters - I had this working a few weeks ago but now I am getting these errors for two of the caches -
2017-06-02 10:27:12,401 [INFO] [Logger@9269610 12.2.1.2.1 hraja:debug] [Coherence] 2017-06-02 10:27:12.400/135.971 Oracle Coherence GE 12.2.1.2.1 hraja:debug <Info> (thread=TicketsCacheService:FederatedCacheControl, member=6): com.tangosol.net.RequestIncompleteException: Partial failure
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeAllRequest$Poll.processAsyncResponse(PartitionedCache$InvokeAllRequest$Poll.java:144)
at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest$Poll.onCompletion(DistributedCacheRequest$Poll.java:194)
at com.tangosol.coherence.component.net.Poll.close(Poll.java:317)
at com.tangosol.coherence.component.net.Poll.onResponded(Poll.java:644)
at com.tangosol.coherence.component.net.Poll.onResponse(Poll.java:655)
at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest$Poll.onResponse(DistributedCacheRequest$Poll.java:255)
at com.tangosol.coherence.component.net.message.requestMessage.distributedCacheRequest.KeySetRequest$Poll.onResponse(KeySetRequest$Poll.java:171)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.java:2966)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.java:3075)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.java:5176)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.java:5362)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.partitionedCache.FederatedCache.onNotify(FederatedCache.java:981)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.java:1216)
at java.lang.Thread.run(Thread.java:745)
Caused by: (Wrapped: Failed request execution for TicketsCacheService service on Member(Id=6, Timestamp=2017-06-02 10:25:23.648, Address=10.134.87.161:22001, MachineId=23133, Location=site:NTCN,machine:NTCN-TD3CNSD01A,process:394,member:NTCN-DEALING-01A-CacheServer-1, Role=CacheServer)) java.lang.IllegalStateException: Exception while deleting envelope PartitionAwareLong{partition=1110, id=5}, journal id JournalRecordKey{partition=1110, partitionVersion=34, EnvId=5}
at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.java:4738)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeAllRequest(PartitionedCache.java:4309)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeAllRequest.run(PartitionedCache$InvokeAllRequest.java:316)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool$WrapperTask.java:408)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool$WrapperTask.java:398)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService$DaemonPool$WrapperTask.java:102)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool$Daemon.java:555)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.java:1216)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Exception while deleting envelope PartitionAwareLong{partition=1110, id=5}, journal id JournalRecordKey{partition=1110, partitionVersion=34, EnvId=5}
at com.tangosol.internal.federation.journal.EnvelopeRemoveProcessor.process(EnvelopeRemoveProcessor.java:93)
at com.tangosol.util.InvocableMap$EntryProcessor.processAll(InvocableMap.java:740)
at com.tangosol.util.processor.AbstractProcessor.processAll(AbstractProcessor.java:34)
at com.tangosol.internal.federation.journal.EnvelopeRemoveProcessor.processAll(EnvelopeRemoveProcessor.java:61)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.invokeAll(PartitionedCache$Storage.java:5174)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeAllRequest(PartitionedCache.java:4264)
... 7 more
Caused by: java.lang.IllegalArgumentException: Attempted to enlist an entry belonging to a partition outside of this transaction: 1453
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvocationContext$StorageContext.getBackingMapEntry(PartitionedCache$InvocationContext$StorageContext.java:133)
at com.tangosol.internal.federation.journal.EnvelopeRemoveProcessor.process(EnvelopeRemoveProcessor.java:84)
... 12 more
Can anyone tell me what sort of thing can cause these errors?
Thanks