WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
WLS_usr Oct 31, 2019 8:58 PMHi,
We recently moved from WebLogic 10.3.6 to FWM 12.2.1.3 with following set up:
State :
version : FMW domain (12.2.1.3)
OS : RHEL 7.6
1 WL admin running on separate machine
WebLogic Cluster (lets call this JMSCluster) with 3 manage servers running on separate machines
create JMS server and deployed it on JMSCluster
create few distributed queues and "partitioned" distributed topics and deployed then on respective sub deployment module (which is targated to JMS server)
Problem:
We are seeing most of the producers (producing the messages to distributed queues and topics) and consumers (consuming from them) are working as expected, except for one distributed topic and that to a durable subscriber (other consumers to that topic has no issues consuming the messages) which fails with following exception and stops consuming the messages until we restart that consumer app and after a while (no set pattern or time) it again stops consuming the message. The only pattern we are able to see is, when it stops consuming the messages the state of the durable consumer has some messages in pending status.
####<Oct 24, 2019 9:02:01 AM MDT> <Warning> <RMI> <consumerhost> <consumer_Node1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1571929321286> <BEA-080005> <Exception thrown by rmi server: dispatch
java.rmi.UnmarshalException: error unmarshalling arguments, count=13575, lastMethodId15620, lastRequestType9; nested exception is:
java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive..
java.rmi.UnmarshalException: error unmarshalling arguments, count=13575, lastMethodId15620, lastRequestType9; nested exception is:
java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive.
at weblogic.messaging.dispatcher.DispatcherServerRef.invoke(DispatcherServerRef.java:232)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:141)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive.
at weblogic.jms.common.JMSUtilities.versionIOException(JMSUtilities.java:102)
at weblogic.jms.common.TextMessageImpl.readExternal(TextMessageImpl.java:399)
at weblogic.jms.common.JMSPushRequest.readExternal(JMSPushRequest.java:295)
at weblogic.messaging.dispatcher.DispatcherObjectHandler.readRequest(DispatcherObjectHandler.java:48)
at weblogic.messaging.dispatcher.DispatcherServerRef.invoke(DispatcherServerRef.java:227)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:141)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
We didn't see this when durable consumer was connected to JMS on WL 10.3.6.
Troubleshooting that has done so far:
1. turned on JMS related debug (such as JMS common, dispatcher, frontend, backend) however it didn't reveal much except for following on JMS when java.io.StreamCorruptedException reported on consumer :
<Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>
<Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>
<Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>
<Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>
<Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>
<Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>
<Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>
<Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>
<Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>
<Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>
2. We are currently running test with consumer using latest (12.2.1.3) wlfullclient.jar, -Drmi.alt.marshall=false and dms.jar.
Can you please let me know if anyone has seen this issue or point me in some direction?