Unhandled IOException: buffer capacity of 2G exceeded

Zyd08

We are running coherence version 12.2.1.0.0.

Our coherence cluster runs 48 nodes, and 31 of which saw this below exception stack trace at the same time. By the looks of it, it seems to occur with data persistence. Indeed we were writing some data to the cluster (via REST API) at the time of the error, but it wasn't a unusually big amount. Can someone please advise what this error may suggest? Is this some bug in this Coherence version or do we need some change in our data writing behaviour? Thanks very much.

2017-09-15 19:44:37,914 [[email protected] 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74): Terminating PartitionedCache due to unhandled exception: com.tangosol.ut

il.WrapperException

2017-09-15 19:44:37,914 [[email protected] 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74):

(Wrapped) java.io.IOException: buffer has reached its max capacity of 2GB

        at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)

        at com.tangosol.util.Base.ensureRuntimeException(Base.java:277)

        at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:26)

        at com.tangosol.coherence.component.net.MessageHandler.post(MessageHandler.CDB:36)

        at com.tangosol.coherence.component.net.Message.dispatch(Message.CDB:76)

        at com.tangosol.coherence.component.net.Message.post(Message.CDB:1)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.post(Grid.CDB:2)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishToClients(PartitionedCache.CDB:8)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.tryBatchCompletion(PartitionedCache.CDB:10)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.onJobCompleted(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishChanges(PartitionedCache.CDB:196)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:18)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onQueryRequest(PartitionedCache.CDB:55)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryRequest.run(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)

        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.io.IOException: buffer has reached its max capacity of 2GB

        at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:360)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.writeObject(Service.CDB:1)

        at com.tangosol.coherence.component.net.Message.writeObject(Message.CDB:1)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryResponse.write(PartitionedCache.CDB:27)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.serializeMessage(Grid.CDB:16)

        at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:19)

        ... 18 more

Caused by: java.lang.UnsupportedOperationException: buffer has reached its max capacity of 2GB

        at com.tangosol.io.MultiBufferWriteBuffer.addBuffer(MultiBufferWriteBuffer.java:1108)

        at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.advance(MultiBufferWriteBuffer.java:873)

        at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:694)

        at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:671)

        at com.tangosol.io.pof.WritingPofHandler.onOctetString(WritingPofHandler.java:637)

        at com.tangosol.io.pof.PofBufferWriter.writeBinary(PofBufferWriter.java:810)

        at com.tangosol.io.pof.PofBufferWriter.writeObject(PofBufferWriter.java:1744)

        at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:354)

        ... 23 more

Shyam Radhakrishnan-Oracle

From the exception it looks like a huge object is being serialized, and this is happening on a  query result. I would suggest you raise an SR/bug, and provide all relevant details such as cache configuration, the size of the cluster at that point, all the relevant logs. What is the query that you were using, did you use a generic query such as get all the cache objects etc?

Zyd08

We were running PUT rest calls to persist data to the cluster one at a time (although multithreaded), each less than about 1 MB.

Shyam Radhakrishnan-Oracle

It does seem weird that a query is being done, if as a client, you are doing only put request. I would suggest you raise an SR/bug and we can take it from there.