Forum Stats

  • 3,815,823 Users
  • 2,259,094 Discussions
  • 7,893,259 Comments

Discussions

Coherence Cluster Throws PartitionedfCache$InvocationContext.lockEntry Assertion Failure

d9e2f31b-593b-4cb1-89f5-ccc03321a2d0
edited Jan 2, 2019 6:42PM in Coherence Support

Hello All,

Please share any inputs if this is a known issue and any work arounds.

Only way to fix this currently is restarting the whole cluster.

We are getting the following exception randomly cannot reproduce consistently in production env

Oracle Coherence - Version 12.2.1.0.0

As a result, future get or put (can't tell which) requests using the *same key* are get blocked indefinitely, regardless of the process or thread those requests are being executed from.

Cache server log:

Assertion failed:

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvocationContext.lockEntry(PartitionedCache.CDB:26)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvocationContext.lockEntry(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:54)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)

        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:54)

        at java.lang.Thread.run(Thread.java:748)

Extend logs

        at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:61)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:97)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)

        at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)

        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:54)

        at java.lang.Thread.run(Thread.java:748)

        at <process boundary>

        at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57)

        at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3618)

        at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2889)

        at com.tangosol.io.pof.ConfigurablePofContext.deserialize(ConfigurablePofContext.java:378)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:1)

        at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:1)

        at com.tangosol.coherence.component.net.message.responseMessage.SimpleResponse.read(SimpleResponse.CDB:6)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.deserializeMessage(Grid.CDB:25)

        at com.tangosol.coherence.component.net.MessageHandler$Connection.prepareMessage(MessageHandler.CDB:31)

        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.TransportService$MessageHandler$Connection.prepareMessage(TransportService.CDB:3)

        at com.tangosol.coherence.component.net.MessageHandler.processMessage(MessageHandler.CDB:26)

        at com.tangosol.coherence.component.net.MessageHandler$EventCollector.add(MessageHandler.CDB:51)

        at com.oracle.common.internal.net.socketbus.AbstractSocketBus.emitEvent(AbstractSocketBus.java:665)

        at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.onControlMessage(SocketMessageBus.java:785)

        at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.onReady(SocketMessageBus.java:586)

        at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.read(SocketMessageBus.java:438)

        at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection.processReads(SocketMessageBus.java:176)

        at com.oracle.common.internal.net.socketbus.BufferedSocketBus$BufferedConnection.onReadySafe(BufferedSocketBus.java:545)

        at com.oracle.common.internal.net.socketbus.AbstractSocketBus$Connection.onReady(AbstractSocketBus.java:1945)

        at com.oracle.common.internal.net.RunnableSelectionService.process(RunnableSelectionService.java:401)

        at com.oracle.common.internal.net.RunnableSelectionService.run(RunnableSelectionService.java:274)

        at com.oracle.common.internal.net.ResumableSelectionService.run(ResumableSelectionService.java:133)

        at java.lang.Thread.run(Thread.java:748)

Answers