0 Replies Latest reply: Jan 17, 2013 5:06 AM by Angel.Ruiz RSS

    -XX:MaxDirectMemorySize setting on Proxy nodes on Coherence 3.7.1

    Angel.Ruiz
      Hi there,

      we are investigating an issue with proxy members, in which these run out of direct memory used by NIO:
      2012-12-17 06:38:58.802/8461192.690 Oracle Coherence GE 3.7.1.4 <Error> (thread=DistributedCache:cache-service, member=33): Error sending MapEvent to Channel(Id=2068895774, Open=true, Connection=0x0000013BA6E960300AD60186C124AEC0CF80EB23778884A2C66EB52319D66C6E): (Wrapped) java.lang.OutOfMemoryError
      at com.tangosol.util.Base.ensureRuntimeException(Base.java:288)
      at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
      at com.tangosol.coherence.component.net.extend.Channel.post(Channel.CDB:37)
      at com.tangosol.coherence.component.net.extend.Channel.send(Channel.CDB:6)
      at com.tangosol.coherence.component.net.extend.proxy.MapListenerProxy.onMapEvent(MapListenerProxy.CDB:9)
      at com.tangosol.coherence.component.net.extend.proxy.MapListenerProxy.entryDeleted(MapListenerProxy.CDB:1)
      at com.tangosol.util.MapEvent.dispatch(MapEvent.java:274)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap$ProxyListener.dispatch(PartitionedCache.CDB:22)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap$ProxyListener.entryDeleted(PartitionedCache.CDB:1)
      at com.tangosol.util.MapListenerSupport$WrapperSynchronousListener.entryDeleted(MapListenerSupport.java:888)
      at com.tangosol.util.MapEvent.dispatch(MapEvent.java:274)
      at com.tangosol.coherence.component.util.CacheEvent.dispatchSafe(CacheEvent.CDB:14)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.dispatch(PartitionedCache.CDB:86)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onMapEvent(PartitionedCache.CDB:55)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$MapEvent.onReceived(PartitionedCache.CDB:3)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
      at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.OutOfMemoryError
      at sun.misc.Unsafe.allocateMemory(Native Method)
      at java.nio.DirectByteBuffer.<init>(Unknown Source)
      at java.nio.ByteBuffer.allocateDirect(Unknown Source)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$BufferPool.instantiateResource(TcpAcceptor.CDB:8)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$BufferPool.acquire(TcpAcceptor.CDB:25)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$BufferPool.allocate(TcpAcceptor.CDB:3)
      at com.tangosol.io.MultiBufferWriteBuffer.addBuffer(MultiBufferWriteBuffer.java:1070)
      at com.tangosol.io.MultiBufferWriteBuffer.<init>(MultiBufferWriteBuffer.java:36)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor.allocateWriteBuffer(TcpAcceptor.CDB:3)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.send(Peer.CDB:12)
      at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.post(Peer.CDB:23)
      at com.tangosol.coherence.component.net.extend.Channel.post(Channel.CDB:25)
      ... 18 more
      Although the amount of NIO space initially allocated may not be the root cause of this issue, we would like to make sure we set direct memory buffer space correctly.
      Is the following recommendation for -XX:MaxDirectMemorySize option still valid on Coherence 3.7.1?

      http://coherence.oracle.com/display/COH35UG/Best+Practices+for+Coherence+Extend#BestPracticesforCoherenceExtend-ConfigureHeapNIOSpacetobeEqualtotheMaxHeapSize

      Thanks on advance.
      Angel