10 Replies Latest reply on Jun 24, 2019 11:12 AM by 3161538

    JMS selectors limitations - string property length limitations?

    3161538

      I am experiencing a strange behaviour of selectors that use long string propertie (around 32k chars). I am looking for an explanation of this anomaly. I use Weblogic 11g running on Java 1.6, and clients running on Java 1.8

       

      I have a following set of selectors, X is my property:

       

      1) X is null

      2) X is not null

      3) X like '%somestring%'

      4) X not like '%somestring%'

       

      In a normal situation you would expect one of these cases to happen:

      -> X=null , then selector 1) matches, 2,3,4) don't match

      -> X contains 'somestring', then selector 1),3) match, and 2),4) don't match

      -> X does not contain 'somestring' then selector 1),4) match and 2),3) don't match

       

      So far so good.

      Now if I upload a message with a huge string property (I use javax.jms.QueueSender and javax.jms.Message#setStringProperty for upload) i get this:

      -> selector 2) matches, indicating that property is not empty

      -> both selectors 3),4) does not match as if property was empty.

       

      Another anomaly is that the contents of the property is not displayed in WLS console. When exported to xml from console -> property is null.

      However if I read the message using QueueReceiver, then I am able to read that huge property contents.

       

      This does not occur if I manually prepare a message for import using Weblogic Console and WLS internal xml format for import.

      Then selectors, property display and export work just fine even for 1MB properties.

       

      Are there any limitations on when JMS selectors can be used? Are there any limitations on using QueueSender for big properties? Is this some kind of bug?

      Thanks

      Tomasz

        • 1. Re: JMS selectors limitations - string property length limitations?
          Tom B-Oracle

          Hi,

           

          Your research is helpful - and brings to mind a potential explanation. My educated guess is that the reason your 'queue sender' isn't handling a large message property while an XML export does, is that the former is probably serialized internally using UTF16 or similar, which would explain the 32K upper bound (16 bits ~~ max signed short ~~ 32,767).

           

          A 32K length message property is an unprecedented size, so I suspect you've hit upon an unsupported use case that'd require a relatively involved enhancement to fix. JMS generally expects properties to be small, and expects larger data to be stored in message bodies.

           

          The most common pattern to handle selecting within a large piece of text is to (a) put large data in message bodies, (b) if selectors are elements within the message bodies, have application senders copy important small bits of body data either into message properties or the JMS Correlation ID, and then select based on those properties.

           

          Alternatively, it's possible to select on data in a message body by using an extension.  See JMS_BEA_SELECT in https://docs.oracle.com/middleware/1221/wls/JMSPG/manage_apps.htm#JMSPG305.  This may need you to use an 'XMLMessage' type, which is an extension of TextMessage - but I vaguely recall a plain TextMessage with XML contents might work too.  That said, I usually wouldn't recommend this - your performance may be too low at high message rates or high message backlogs.  The same perf advice applies to selectors in general - they often simply should be avoided - but even when selectors are in play, filtering through large chunks of text is an especially expensive type of selector in of itself, and filtering message bodies adds even more overhead.

           

          HTH,

          Tom

          1 person found this helpful
          • 2. Re: JMS selectors limitations - string property length limitations?
            Martien van den Akker

            Hi,

             

            I don't see the suggestion of Tom that using UTF16 will limit your string-length. Although 16 bits would lead to the number 32767, that is that it would allow you to encode your string in at most 32767 possible different characters. And even that is not correct for UTF-16, because that would suggest that UTF-8 would encode to at most 256 possible different characters (including non-printables). And that is exactly the limitation of ASCII based sets (that is actually 7-bits, so only 127 chars).

             

            JMS properties (either custom or build-in) are meant to identify your messages or help to process or address your messages. By concept I would discourage to use it for large content. Because then you would use the properties as a transport and the message payload would for that.

             

            I even think that the way you define your message selection a bit improper. I think you should preferably base message selectors on a fairly limited set of enumerated values. In my current assignment we have 10's of queues and on each queues multiple 10's of polling adapters each with a message selector.

             

            I think it's best to poll just for every message on a specific queue and then handle it based on the message content. You would need to use message selectors if you need to specifically correlate certain messages to a specific process. But in your case the selection is too 'vague'. Although you can do wildcard selections, I would highly discourage thouse in the context of JMS and then only for small strings. Like, if you have message types like 'starwarsResponse', 'starwarsFault', 'startrekResponse' and 'starttrekFault'.  Then you would want to have the 'starwars%' messages by the StarWars process and the 'starttrek%' messages by the StarTrek process.

             

            This does not explain why your case would fail, or it does not show a specific bug. But it makes me understand that you could easily hit a certain unexpected behavior, not necessarily a bug. For instance because the infrastructure stores large properties differently, seperately from the rest of the message, and therefor not reachable by the message-selector functionality.

             

            Hope this makes sense.

             

            Regards,
            Martien

            1 person found this helpful
            • 3. Re: JMS selectors limitations - string property length limitations?
              Martien van den Akker

              Hi,

               

              I found it a quite interesting question, since my current assignment. I'm creating a MessageDispatcher that would replace most of the current polling adapters. Ideally it would just sit and listen for every message and based on DVMs on the MessageType would dispatch it to the proper message-handler. This investigation would give me ammunition, extra reasons for pushing this through.

               

              I found this: https://it.toolbox.com/question/character-limit-for-a-user-property-value-of-a-jms-message-in-ican-505-051711 which makes the remark that there is no property size limit based on the JMS definition. It just depends on the JMS implementation. It mentions that the Oracle AQ for instance has a max property size of 2000. This has to do with the fact that AQ is implemented in the Oracle database, where JMS based AQ queues are based on types. The userproperty for instance is defined as:

               

              Then I found also the following:

              JMS Best Practices | Enterprise Messaging with the Java Message Service (JMS) | InformIT

              https://stackoverflow.com/questions/587899/jms-messaging-performance-lots-of-topics-queues-vs-extensive-filtering-messag

               

              These state that message selectors have a negative impact on the performance. If you use it on queues, then the message selection is done on the client side. It will have to review each message in the queue to see if the message is eligible for processing. Obviously having wildcard selections on large properties will impact the performance more negatively. Think about that the J in JMS stands for Java, and that means that all the processing has to be done in the Heap. (It impacts the heap of both the client as the server, which in Weblogic may be the same heap).

              Message selecting maybe more suitable for topics, since it will define if a message needs to be retained of can be discarded.

               

              So, bug or no bug, expected or not expected behavior, I think you should review your design and do the message selecting part in a pre-processing service and have that service just poll without message selectors.

               

              Thanks for the question, by the way.

               

              Regards,
              Martien

              • 4. Re: JMS selectors limitations - string property length limitations?
                3161538

                Thanks for quick answer.

                 

                I will implement your suggestion as a workaround - instead of relying on the big property I will put that decision-critical part to a separate, small property (as a boolean or enumeration) so that selectors can work on them easily.

                Fortunately that big property can still be read in my java consumer, it seems my problem only affects selectors. So this 'alternative' property storage is handled in java in both message production and consumption.

                 

                Regards,

                Tomasz

                • 5. Re: JMS selectors limitations - string property length limitations?
                  3161538

                  Thanks for you suggestions, Martien.

                   

                  Funny thing - I have run into this problem because I have to share my messages between polling consumers and listener consumers, to avoid another bug 28637420. I run into deep trouble when reading from multiple queues in a single transaction in standalone java jms client (wlthin3client.jar). So I've added those selectors to pick messages that require polling consumers for correct processing, all other are handled by listener consumers.

                   

                  Be warned if you have similar case.

                  Tomasz

                  • 7. Re: JMS selectors limitations - string property length limitations?
                    Martien van den Akker

                    Hi,

                     

                    I wouldn't see it as 'a workaround', although I see the perspective as hitting a bug. However, I see it as an improvement of the original approach.

                    • 8. Re: JMS selectors limitations - string property length limitations?
                      Martien van den Akker

                      I don't see that the string serialization issues stated in those two threads are necessarily are involved here. It could be that in the underlying infrastructure (Weblogic including the JMS subsystem is a large Java system of course), but that should not be of any interest here: we can't do anything about the JMS implementation.

                       

                      Regards,
                      Martien

                      • 9. Re: JMS selectors limitations - string property length limitations?
                        Tom B-Oracle

                        3161538 wrote:

                         

                        Thanks for quick answer.

                         

                        I will implement your suggestion as a workaround - instead of relying on the big property I will put that decision-critical part to a separate, small property (as a boolean or enumeration) so that selectors can work on them easily.

                        Fortunately that big property can still be read in my java consumer, it seems my problem only affects selectors. So this 'alternative' property storage is handled in java in both message production and consumption.

                         

                        Regards,

                        Tomasz

                        Sounds good, although it'd be better IMO if there were no big properties at all.

                         

                        I'm still a little concerned about one thing:  have you triple-checked that your JMS sender's big properties aren't somehow getting truncated by the time they reach the consumer?  For example, verify that the consumer gets the same length property as what the JMS sender stuffed into the message. 

                         

                        HTH,

                        Tom Barnes

                        WebLogic JMS Developer Team

                        • 10. Re: JMS selectors limitations - string property length limitations?
                          3161538

                          Yes, I have dumped the property contents on the consumer side and it is not truncated even for sizes exceeding 32k characters (for the cases where selector stops working).

                          Tomasz