1 Reply Latest reply: Dec 29, 2010 3:06 PM by 601680 RSS

    Stopping cluster due to unhandled exception - OutOfMemoryError

    snidely_whiplash
      Is this the expected behavior or a bug?
      -Andrew

      2010-12-22 08:44:17.402/200767.980 Oracle Coherence GE 3.6.0.0 <Warning> (thread=PacketPublisher, member=2): Experienced a 2041 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer); 84 packets rescheduled, PauseRate=2.0E-4, Threshold=67
      2010-12-22 08:44:17.604/200768.182 Oracle Coherence GE 3.6.0.0 <D6> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer) has failed to respond to 17 packets; declaring this member as paused.
      2010-12-22 08:44:24.648/200775.226 Oracle Coherence GE 3.6.0.0 <Warning> (thread=PacketPublisher, member=2): Experienced a 7044 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer); 88 packets rescheduled, PauseRate=3.0E-4, Threshold=64
      2010-12-22 08:44:24.848/200775.426 Oracle Coherence GE 3.6.0.0 <D6> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer) has failed to respond to 17 packets; declaring this member as paused.
      2010-12-22 08:44:26.616/200777.194 Oracle Coherence GE 3.6.0.0 <Warning> (thread=PacketPublisher, member=2): Experienced a 1767 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer); 65 packets rescheduled, PauseRate=3.0E-4, Threshold=64
      2010-12-22 08:44:26.817/200777.395 Oracle Coherence GE 3.6.0.0 <D6> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer) has failed to respond to 17 packets; declaring this member as paused.
      2010-12-22 08:44:29.780/200780.358 Oracle Coherence GE 3.6.0.0 <Warning> (thread=PacketPublisher, member=2): Experienced a 2963 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2010-12-20 00:58:15.375, Address=192.168.3.20:8089, MachineId=27412, Location=machine:amd4,process:3476,member:Administrator, Role=CoherenceServer); 64 packets rescheduled, PauseRate=3.0E-4, Threshold=64
      2010-12-22 08:45:16.151/200826.729 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster, member=2): Service guardian is 15133ms late, indicating that this JVM may be running slowly or experienced a long GC
      2010-12-22 08:45:35.340/200845.918 Oracle Coherence GE 3.6.0.0 <Error> (thread=DistributedCache:DistributedCacheService, member=2): Terminating PartitionedCache due to unhandled exception: java.lang.OutOfMemoryError
      2010-12-22 08:45:44.277/200854.855 Oracle Coherence GE 3.6.0.0 <Error> (thread=DistributedCache:DistributedStatsCacheService, member=2): Terminating PartitionedCache due to unhandled exception: java.lang.OutOfMemoryError
      2010-12-22 08:45:44.277/200854.855 Oracle Coherence GE 3.6.0.0 <Error> (thread=DistributedCache:DistributedStatsCacheService, member=2): 
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      2010-12-22 08:45:44.277/200854.855 Oracle Coherence GE 3.6.0.0 <D5> (thread=DistributedCache:DistributedCacheService, member=2): Service DistributedCacheService left the cluster
      2010-12-22 08:45:44.277/200854.855 Oracle Coherence GE 3.6.0.0 <D5> (thread=DistributedCache:DistributedStatsCacheService, member=2): Service DistributedStatsCacheService left the cluster
      2010-12-22 08:45:44.277/200854.855 Oracle Coherence GE 3.6.0.0 <Error> (thread=DistributedCache:DistributedExecutionsService, member=2): Terminating PartitionedCache due to unhandled exception: java.lang.OutOfMemoryError
      2010-12-22 08:45:47.283/200857.861 Oracle Coherence GE 3.6.0.0 <Error> (thread=PacketListenerN, member=2): Stopping cluster due to unhandled exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
           at com.tangosol.io.ByteArrayWriteBuffer.<init>(ByteArrayWriteBuffer.java:79)
           at com.tangosol.io.ByteArrayWriteBuffer.<init>(ByteArrayWriteBuffer.java:58)
           at com.tangosol.io.MultiplexingWriteBufferPool$PoolSegment.allocateNonPooledBuffer(MultiplexingWriteBufferPool.java:399)
           at com.tangosol.io.MultiplexingWriteBufferPool$PoolSegment.ensureBuffer(MultiplexingWriteBufferPool.java:364)
           at com.tangosol.io.MultiplexingWriteBufferPool$PoolSegment.acquire(MultiplexingWriteBufferPool.java:243)
           at com.tangosol.io.MultiplexingWriteBufferPool.allocate(MultiplexingWriteBufferPool.java:89)
           at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:11)
           at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
           at java.lang.Thread.run(Thread.java:619)