This discussion is archived
13 Replies Latest reply: Jul 24, 2012 11:58 PM by 934501 RSS

Messaging Pattern performance issue

813474 Newbie
Currently Being Moderated
I found some performance problem when using the Queue from the messaging pattern.

When the queue has little messages (say below 100), the subscriber.getMessage() consumer can return in 3ms. However, when the queue accumulates more messages, say 2000 messages, the return time is degraded to 30ms. And it get worse and worse when the accumulated messages increases.

Do I misconfigure the Coherence or is it an performance issue of the Queue messaging pattern?

Below is my producer and consumer. I make them RTC clients so all of the cache are located in another server clusters.

Producer:
public class QueueProducer {
     
     private static final String TEST_TOPIC_2 = "TESTING.TOPIC.2";
     private MessagingSession messagingSession;
     private Identifier queueId;
     
     public void init() {
          System.setProperty("tangosol.coherence.cacheconfig", "coherence-messagingpattern-cache-config.xml");
          System.setProperty("tangosol.coherence.clusteraddress", "224.3.6.1");
          System.setProperty("tangosol.coherence.clusterport", "37191");
          System.setProperty("tangosol.coherence.edition", "RTC");
          messagingSession = DefaultMessagingSession.getInstance();
          queueId = messagingSession.createQueue(TEST_TOPIC_2);
     }
     
     public void startPublishing() {
          System.out.println("start publishing");
          for (int i=0; ; i++) {
               messagingSession.publishMessage(queueId, new Integer(i));
               
               if (i%1000 == 0) {
                    System.out.println(new Date() + " Published to " + TEST_TOPIC_2 + ". Object: " + i);
               }
          }
     }
     
     public static void main(String ... args) {
         QueueProducer tester = null;
          try {
               tester = new QueueProducer();
               tester.init();
               tester.startPublishing();
          } catch (Exception e) {
              e.printStackTrace();
          }
     }
     
}
Consumer:
public class QueueConsumer {
     
     private static final String TEST_TOPIC_2 = "TESTING.TOPIC.2";
     private MessagingSession messagingSession;
     private Identifier queueId;
     private int recvCnt = 0;
     
     public void init() {
          System.setProperty("tangosol.coherence.cacheconfig", "coherence-messagingpattern-cache-config.xml");
          System.setProperty("tangosol.coherence.clusteraddress", "224.3.6.1");
          System.setProperty("tangosol.coherence.clusterport", "37191");
          System.setProperty("tangosol.coherence.edition", "RTC");
          messagingSession = DefaultMessagingSession.getInstance();
          queueId = messagingSession.createQueue(TEST_TOPIC_2);
     }
     
     public void startListening() {
          Subscriber sub = messagingSession.subscribe(queueId);
          while (true) {
               long startTime = System.currentTimeMillis();
               Object msg = sub.getMessage();
               long waitTime = System.currentTimeMillis() - startTime;
               ++recvCnt;
               System.out.println(new Date() + " Received message: " + msg + ". Totoal received: " +
                    (recvCnt) + ". Wait time:" + waitTime + "ms");
          }
     }
     
     public static void main(String ... args) {
          QueueConsumer tester = null;
          try {
               tester = new QueueConsumer();
               tester.init();
               tester.startListening();
          } catch (Exception e) {
              e.printStackTrace();
          }
     }
}
  • 1. Re: Messaging Pattern performance issue
    pmackin Journeyer
    Currently Being Moderated
    Hi

    The Message Queue maintains a set of all MessageIdentifiers for messages waiting to be delivered. So, when the queue is serialized/deserialized then the MessageIdentifiers are serialized/deserialized also. Every time a request is made by a subscriber, the queue will be updated twice: once to record the subscriber request and once when the message has finally been delivered. This is a possible cause of the performance issue. The only way to know for sure is to profile the code.

    Paul
  • 2. Re: Messaging Pattern performance issue
    765276 Newbie
    Currently Being Moderated
    I am experiencing the exact same issue and can easily show the consumption latency increases depending on the amount of messages in the queue. Currently have an SR open about it.
  • 3. Re: Messaging Pattern performance issue
    813474 Newbie
    Currently Being Moderated
    Do you mean the queue is a single cache entry so the whole queue is serialized/deserialized every time accessing it?

    Can adding an index help on this issue?
  • 4. Re: Messaging Pattern performance issue
    pmackin Journeyer
    Currently Being Moderated
    Yes, a messaging Queue is a single cache entry so the whole queue is serialized/deserialized every time it is accessed. We are currently looking at various options on improving the performance. I will update this thread with the results of that investigation.

    Paul
  • 5. Re: Messaging Pattern performance issue
    pmackin Journeyer
    Currently Being Moderated
    Hi

    I have done some testing and can reproduce the scalability issue. I ran a test where 3000 messages were put into the queue and the average subscriber getMessage time was 7 mills. The same test with 30,000 messages in the Queue had and= average getMessage time of 70 mills.

    One observation is that POF was 3-4x faster than non-POF. So, while we are working on a fix, you may want to use POF if you are not using it already.

    The internal tracking number for this issue is INC-908

    Paul
  • 6. Re: Messaging Pattern performance issue
    813474 Newbie
    Currently Being Moderated
    Thanks Paul.

    Actually I have used POF already.
  • 7. Re: Messaging Pattern performance issue
    515017 Newbie
    Currently Being Moderated
    Has there been any progress on getting this performance issue resolved? We wish to use Push Replication, but with the performance issues plaguing the Queue in the underlying Messaging Pattern, we cannot consider using this at all.
  • 8. Re: Messaging Pattern performance issue
    701681 Explorer
    Currently Being Moderated
    Any update on this?

    This has serious issues for push replication in the result of a remote failure/slow down.

    i.e. the messaging queue backs up (potentially 1000s of messages)

    Scenario

    - Client writes to cache backed by PublishingCacheStore. The cache belongs to a CacheService which has a finite cache service thread pool.
    - If the remote endpoint slows down (eg: slow remote site, slow database, etc), the messaging queue starts to back up.
    - The PublishingCacheStore writes synchronously to messaging using the CacheService's thread pool.

    This means if messaging writes slow down, we start to get thread starvation for the CacheService's thread pool. I have managed to reproduce this.

    Will there be a patch to Messaging 2.7 that will be compatible with Coherence 3.6?

    Cheers,
    Neville.
  • 9. Re: Messaging Pattern performance issue
    Brian Oliver Explorer
    Currently Being Moderated
    Hi All,

    Thanks for reporting the challenges with the Messaging Pattern. It's great to see people pushing it hard and contributing feedback to us. We really appreciate the work and effort.

    While there's been a bit of radio silence over the past few weeks, we've been doing quite a bit of research into the issues and considering the best solutions to move forward. Here are some of our thoughts and conclusions, mostly which concur with what we've seen reported recently:

    (When using Messaging Pattern 2.7) Under load, with consistently high-volumes, when objects tend to increase in size, when individual topic subscriptions fail to consume messages for a long period of time, CPU and network bandwidth are reduced, topic subscriptions (queues) may start "back up" to the point where undesirable latency can be experienced when attempting to publish messages.

    ie: the complete worst case in every dimension for a messaging system that must maintain order in a distributed system.

    From our tests there are two main contributors to the increases in latency.

    1. Serialization Costs (for managing messages and the individual topic subscription information)

    2. Garbage Collection (directly related to #1).

    Fortunately when subscriptions re-commence consuming messages, the latency spikes are resolved. However now that we understand the issue (thanks to everyone's guidance), we believe we can resolve it.

    Although we've considered and prototyped several approaches to solving these challenges, including writing to disk to offload the heap, however we've instead decided to focus on keeping this in-memory and making scalable. It's not easy, but we have a solution that we're working on.

    In terms of compatibility, nothing in what we're doing requires Coherence 3.7. It is designed to work with Coherence 3.6 and suspect we can patch/release a version with compatible interfaces. We would however like to start using Coherence 3.7 (as we have with Incubator 10), so there may be dual versions initially. Ultimately we want to take advantage of the new high-performance journalling capability in Coherence 3.7 and thus completely off-load message storage from the Cluster, so over time we expect to drop Coherence 3.6 support, especially as Coherence 3.8 is progressing quickly.

    Hope this helps. Stay tuned for more updates soon.

    -- Brian
    -----
    Brian Oliver | Architect | Oracle Coherence 
  • 10. Re: Messaging Pattern performance issue
    user10714864 Newbie
    Currently Being Moderated
    Any updates on this issue? Also curious to know when can we expect 3.8 to be released?
  • 11. Re: Messaging Pattern performance issue
    701681 Explorer
    Currently Being Moderated
    Agree, there has been radio silence on this issue for far to long. When will we get an improved messaging implementation?
  • 12. Re: Messaging Pattern performance issue
    695726 Newbie
    Currently Being Moderated
    We are also experiencing the same issue. Is there any improvements about this issue?

    Regards
  • 13. Re: Messaging Pattern performance issue
    934501 Newbie
    Currently Being Moderated
    Is there any updated on this issue ?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points