6 Replies Latest reply on Nov 12, 2019 10:18 PM by Tom B-Oracle

    WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages

    WLS_usr
    WLS_usr Oct 31, 2019 8:58 PM

      Hi,

          We recently moved from WebLogic 10.3.6 to FWM 12.2.1.3 with following set up:

       

      State :

      version : FMW domain (12.2.1.3)

      OS : RHEL 7.6

      1 WL admin running on separate machine

      WebLogic Cluster (lets call this JMSCluster) with 3 manage servers running on separate machines

      create JMS server and deployed it on JMSCluster

      create few distributed queues and "partitioned" distributed topics and deployed then on respective sub deployment module (which is targated to JMS server)

       

       

      Problem:

      We are seeing most of the producers (producing the messages to distributed queues and topics) and consumers (consuming from them) are working as expected, except for one distributed topic and that to a durable subscriber (other consumers to that topic has no issues consuming the messages)  which fails with following exception and stops consuming the messages until we restart that consumer app and after a while (no set pattern or time) it again stops consuming the message. The only pattern we are able to see is, when it stops consuming the messages the state of the durable consumer has some messages in pending status.

       

       

      ####<Oct 24, 2019 9:02:01 AM MDT> <Warning> <RMI> <consumerhost> <consumer_Node1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1571929321286> <BEA-080005> <Exception thrown by rmi server: dispatch

        java.rmi.UnmarshalException: error unmarshalling arguments, count=13575, lastMethodId15620, lastRequestType9; nested exception is: 

        java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive..

      java.rmi.UnmarshalException: error unmarshalling arguments, count=13575, lastMethodId15620, lastRequestType9; nested exception is: 

      java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive.

        at weblogic.messaging.dispatcher.DispatcherServerRef.invoke(DispatcherServerRef.java:232)

        at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:141)

        at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)

        at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      Caused By: java.io.StreamCorruptedException: [Messaging:282000]Unsupported class version 109. Expected a value between 1 and 3 inclusive.

        at weblogic.jms.common.JMSUtilities.versionIOException(JMSUtilities.java:102)

        at weblogic.jms.common.TextMessageImpl.readExternal(TextMessageImpl.java:399)

        at weblogic.jms.common.JMSPushRequest.readExternal(JMSPushRequest.java:295)

        at weblogic.messaging.dispatcher.DispatcherObjectHandler.readRequest(DispatcherObjectHandler.java:48)

        at weblogic.messaging.dispatcher.DispatcherServerRef.invoke(DispatcherServerRef.java:227)

        at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:141)

        at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)

        at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

       

      We didn't see this when durable consumer was connected to JMS on WL 10.3.6.

       

      Troubleshooting that has done so far:

      1. turned on JMS related debug (such as JMS common, dispatcher, frontend, backend) however it didn't reveal much except for following on JMS when java.io.StreamCorruptedException reported on consumer :

      <Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>

      <Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,636 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>

      <Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,637 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <12.2.1 or later output stream.>

      <Oct 28, 2019 2:04:52,641 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Wrote Partition data to the ouput stream>

      <Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition id: 0>

      <Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Partition name: DOMAIN>

      <Oct 28, 2019 2:04:52,642 PM MDT> <Debug> <JMSDispatcherUtilsVerbose> <BEA-000000> <Connection partition name: DOMAIN>

       

      2. We are currently running test with consumer using latest (12.2.1.3) wlfullclient.jar, -Drmi.alt.marshall=false and dms.jar.

       

      Can you please let me know if anyone has seen this issue or point me in some direction?

        • 1. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
          Tom B-Oracle
          Tom B-Oracle Oct 31, 2019 10:07 PM (in response to WLS_usr)

          Thank you for providing a thorough amount of debugging information - much appreciated!

           

          It doesn't look like you're doing anything out of the ordinary unless the system property and dms jar are somehow incompatible with the full client. I've only very rarely seen similar marshaling errors that weren't caused by a bug in Oracle WLS itself, so my main suggestion is to contact Oracle customer support - the marshaling error may already be a known bug with an existing patch.

           

          If you would like to attempt a work-around in the meantime, here are couple of thoughts:

           

          • Potential work-around #1:  I assume the 'dms.jar' has something to do with 'Dynamic Monitoring Service'. Does the 'dms.jar' file need to be in the client's classpath? I'm not familiar with the particulars of DMS, but in my experience it's sometimes been the culprit when it comes to protocol issues.  If I understand correctly, DMS has a flow dyeing capability that can implicitly/secretly inject itself into a protocol as it tracks a particular flow, and this injection sometimes has had broken side effects which manifest as marshaling errors.  So if you have a way to disable it's use on the particular path that you're using, then you might see the problem go away.

           

           

          • Potential work-around #3: 12.2.1.4 was just released a few days ago.  If #1 or #2 don't help, then you might want to give that a try. If you try it, it might be needed only for the client, only for the server, or for both.

           

          • Potential work-around #4:  What's the "-Drmi.alt.marshall=false" about?  I couldn't find any WL documentation for it, nor could I find 'rmi.alt.marshall' string in the WebLogic source code.  The only hit I've found for it so far was for "OC4J" - Oracle's original app server that was used before it switched to WebLogic.  Perhaps it's having an unanticipated side effect.

           

          Hope this helps!

          Tom

          • 2. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
            WLS_usr
            WLS_usr Nov 1, 2019 4:02 AM (in response to Tom B-Oracle)

            Thanks Tom for taking time and providing valuable feedback.

             

            We already have a SR opened, as part of troubleshooting "-Drmi.alt.marshall=false" and dms.jar is suggested by Oracle.. I'll give Potential work-around #2 (using "thin t3" and/or "install client") a try and keep the thread posted with the outcome.

            • 3. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
              Martien van den Akker
              Martien van den Akker Nov 1, 2019 1:18 PM (in response to WLS_usr)

              Hi,

              I see that Tom has a few very worthy remarks and suggestions. I'm just curious about the payload of the topic at hand. Are they just plain XML/Text messages or perhaps java classes?

              It feels a bit like a mismatch between the jar's used by the producer and the consumers. Although I would expect class-cast exceptions, so that counters my idea.

              Still, what are the producers and consumers? Are the producers soa services on that environment and is the consumer a external java application?

               

              Kind regards,
              Martien

              • 4. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
                Tom B-Oracle
                Tom B-Oracle Nov 1, 2019 4:14 PM (in response to Martien van den Akker)

                > It feels a bit like a mismatch between the jar's used by the producer and the consumers.

                 

                Agree.  But if so, then it's more likely the mismatch is in the Oracle jars themselves and not the application classes. Likely something to do with DMS.  Mismatched application classes wouldn't cause an error starting at the 'dispatcher' network layer - they'd manifest differently (and usually with a fairly helpful exception) as their marshaling occurs in a different location in the code.

                • 5. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
                  WLS_usr
                  WLS_usr Nov 12, 2019 9:33 PM (in response to WLS_usr)

                  Hi Tom, Martien..

                              Over the course of days, we ran few tests and here are the results:

                   

                  1. Tests with dms.jar and -Drmi.alt.marshall=false on consumer client and JMS : client consumer stops consuming the messages after a while but this time it doesn't throw "java.rmi.UnmarshalException" exception, however the final result is same.

                  2. Tests with thin client : After some time into test, client throws "java.rmi.UnmarshalException" and stops consuming the messages..

                   

                  Actually, I was thinking of re-establishing the connectivity if code sees this exception as work around (rather than restarting consumer app).

                   

                  Please, let me know if you want us to try something else or any ideas you may have.

                  • 6. Re: WebLogic JMS consumer throws java.rmi.UnmarshalException and stops consuming messages
                    Tom B-Oracle
                    Tom B-Oracle Nov 12, 2019 10:18 PM (in response to WLS_usr)

                     

                    >  Actually, I was thinking of re-establishing the connectivity if code sees this exception as work around (rather than restarting consumer app).

                     

                    Great idea! That'd be a good thing to do regardless. See Client Resiliency Best Practices in the Best Practices chapter of the Administering JMS guide.