1 Reply Latest reply on Jan 2, 2019 11:42 PM by EdwardBurns-Oracle

    Coherence Cluster Throws PartitionedfCache$InvocationContext.lockEntry Assertion Failure

    d9e2f31b-593b-4cb1-89f5-ccc03321a2d0

      Hello All,

       

      Please share any inputs if this is a known issue and any work arounds.

       

      Only way to fix this currently is restarting the whole cluster.

       

      We are getting the following exception randomly cannot reproduce consistently in production env

       

      Oracle Coherence - Version 12.2.1.0.0

       

      As a result, future get or put (can't tell which) requests using the *same key* are get blocked indefinitely, regardless of the process or thread those requests are being executed from.

       

      Cache server log:

       

      Assertion failed:

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvocationContext.lockEntry(PartitionedCache.CDB:26)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvocationContext.lockEntry(PartitionedCache.CDB:1)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:54)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)

              at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:54)

              at java.lang.Thread.run(Thread.java:748)

       

       

      Extend logs

       

              at com.tangosol.util.Base.ensureRuntimeException(Base.java:296)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:61)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:97)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)

              at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)

              at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:54)

              at java.lang.Thread.run(Thread.java:748)

              at <process boundary>

              at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57)

              at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3618)

              at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2889)

              at com.tangosol.io.pof.ConfigurablePofContext.deserialize(ConfigurablePofContext.java:378)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:1)

              at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:1)

              at com.tangosol.coherence.component.net.message.responseMessage.SimpleResponse.read(SimpleResponse.CDB:6)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.deserializeMessage(Grid.CDB:25)

              at com.tangosol.coherence.component.net.MessageHandler$Connection.prepareMessage(MessageHandler.CDB:31)

              at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.TransportService$MessageHandler$Connection.prepareMessage(TransportService.CDB:3)

              at com.tangosol.coherence.component.net.MessageHandler.processMessage(MessageHandler.CDB:26)

              at com.tangosol.coherence.component.net.MessageHandler$EventCollector.add(MessageHandler.CDB:51)

              at com.oracle.common.internal.net.socketbus.AbstractSocketBus.emitEvent(AbstractSocketBus.java:665)

              at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.onControlMessage(SocketMessageBus.java:785)

              at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.onReady(SocketMessageBus.java:586)

              at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection$ReadBatch.read(SocketMessageBus.java:438)

              at com.oracle.common.internal.net.socketbus.SocketMessageBus$MessageConnection.processReads(SocketMessageBus.java:176)

              at com.oracle.common.internal.net.socketbus.BufferedSocketBus$BufferedConnection.onReadySafe(BufferedSocketBus.java:545)

              at com.oracle.common.internal.net.socketbus.AbstractSocketBus$Connection.onReady(AbstractSocketBus.java:1945)

              at com.oracle.common.internal.net.RunnableSelectionService.process(RunnableSelectionService.java:401)

              at com.oracle.common.internal.net.RunnableSelectionService.run(RunnableSelectionService.java:274)

              at com.oracle.common.internal.net.ResumableSelectionService.run(ResumableSelectionService.java:133)

              at java.lang.Thread.run(Thread.java:748)