Discussions
Categories
- 17.9K All Categories
- 3.4K Industry Applications
- 3.3K Intelligent Advisor
- 63 Insurance
- 535.7K On-Premises Infrastructure
- 138.1K Analytics Software
- 38.6K Application Development Software
- 5.6K Cloud Platform
- 109.3K Database Software
- 17.5K Enterprise Manager
- 8.8K Hardware
- 71K Infrastructure Software
- 105.2K Integration
- 41.5K Security Software
Unhandled IOException: buffer capacity of 2G exceeded

We are running coherence version 12.2.1.0.0.
Our coherence cluster runs 48 nodes, and 31 of which saw this below exception stack trace at the same time. By the looks of it, it seems to occur with data persistence. Indeed we were writing some data to the cluster (via REST API) at the time of the error, but it wasn't a unusually big amount. Can someone please advise what this error may suggest? Is this some bug in this Coherence version or do we need some change in our data writing behaviour? Thanks very much.
2017-09-15 19:44:37,914 [[email protected] 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74): Terminating PartitionedCache due to unhandled exception: com.tangosol.ut
il.WrapperException
2017-09-15 19:44:37,914 [[email protected] 12.2.1.0.0] ERROR Coherence - 2017-09-15 19:44:37.914/2102299.989 Oracle Coherence GE 12.2.1.0.0 <Error> (thread=DistributedCacheWorker:0x0000:26, member=74):
(Wrapped) java.io.IOException: buffer has reached its max capacity of 2GB
at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:277)
at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:26)
at com.tangosol.coherence.component.net.MessageHandler.post(MessageHandler.CDB:36)
at com.tangosol.coherence.component.net.Message.dispatch(Message.CDB:76)
at com.tangosol.coherence.component.net.Message.post(Message.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.post(Grid.CDB:2)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishToClients(PartitionedCache.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.tryBatchCompletion(PartitionedCache.CDB:10)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BatchContext.onJobCompleted(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.publishChanges(PartitionedCache.CDB:196)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:18)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onQueryRequest(PartitionedCache.CDB:55)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: buffer has reached its max capacity of 2GB
at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:360)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.writeObject(Service.CDB:1)
at com.tangosol.coherence.component.net.Message.writeObject(Message.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryResponse.write(PartitionedCache.CDB:27)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.serializeMessage(Grid.CDB:16)
at com.tangosol.coherence.component.net.MessageHandler.serializeMessage(MessageHandler.CDB:19)
... 18 more
Caused by: java.lang.UnsupportedOperationException: buffer has reached its max capacity of 2GB
at com.tangosol.io.MultiBufferWriteBuffer.addBuffer(MultiBufferWriteBuffer.java:1108)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.advance(MultiBufferWriteBuffer.java:873)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:694)
at com.tangosol.io.MultiBufferWriteBuffer$MultiBufferOutput.writeBuffer(MultiBufferWriteBuffer.java:671)
at com.tangosol.io.pof.WritingPofHandler.onOctetString(WritingPofHandler.java:637)
at com.tangosol.io.pof.PofBufferWriter.writeBinary(PofBufferWriter.java:810)
at com.tangosol.io.pof.PofBufferWriter.writeObject(PofBufferWriter.java:1744)
at com.tangosol.io.pof.ConfigurablePofContext.serialize(ConfigurablePofContext.java:354)
... 23 more
Answers
-
From the exception it looks like a huge object is being serialized, and this is happening on a query result. I would suggest you raise an SR/bug, and provide all relevant details such as cache configuration, the size of the cluster at that point, all the relevant logs. What is the query that you were using, did you use a generic query such as get all the cache objects etc?
-
We were running PUT rest calls to persist data to the cluster one at a time (although multithreaded), each less than about 1 MB.
-
It does seem weird that a query is being done, if as a client, you are doing only put request. I would suggest you raise an SR/bug and we can take it from there.