We are running coherence version 12.2.1.0.0.
Our coherence cluster runs 48 nodes, and 31 of which saw this below exception stack trace at the same time. By the looks of it, it seems to occur with data persistence. Indeed we were writing some data to the cluster (via REST API) at the time of the error, but it wasn't a unusually big amount. Can someone please advise what this error may suggest? Is this some bug in this Coherence version or do we need some change in our data writing behaviour? Thanks very much.
2017-09-15 19:44:37,914 [Logger@9247854 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74): Terminating PartitionedCache due to unhandled exception: com.tangosol.ut
il.WrapperException
2017-09-15 19:44:37,914 [Logger@9247854 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74):
(Wrapped) java.io.IOException: buffer has reached its max capacity of 2GB
at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:277)
at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:26)
at com.tangosol.coherence.component.net.MessageHandler.post(MessageHandler.CDB:36)
at com.tangosol.coherence.component.net.Message.dispatch(Message.CDB:76)
at com.tangosol.coherence.component.net.Message.post(Message.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.post(Grid.CDB:2)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishToClients(PartitionedCache.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.tryBatchCompletion(PartitionedCache.CDB:10)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.onJobCompleted(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishChanges(PartitionedCache.CDB:196)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:18)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onQueryRequest(PartitionedCache.CDB:55)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: buffer has reached its max capacity of 2GB
at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:360)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.writeObject(Service.CDB:1)
at com.tangosol.coherence.component.net.Message.writeObject(Message.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryResponse.write(PartitionedCache.CDB:27)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.serializeMessage(Grid.CDB:16)
at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:19)
... 18 more
Caused by: java.lang.UnsupportedOperationException: buffer has reached its max capacity of 2GB
at com.tangosol.io.MultiBufferWriteBuffer.addBuffer(MultiBufferWriteBuffer.java:1108)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.advance(MultiBufferWriteBuffer.java:873)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:694)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:671)
at com.tangosol.io.pof.WritingPofHandler.onOctetString(WritingPofHandler.java:637)
at com.tangosol.io.pof.PofBufferWriter.writeBinary(PofBufferWriter.java:810)
at com.tangosol.io.pof.PofBufferWriter.writeObject(PofBufferWriter.java:1744)
at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:354)
... 23 more