2 Replies Latest reply: Oct 10, 2012 4:29 PM by robvarga RSS

    Invocation Service Not Scaling

    878816
      Coherence version 3.6.0.4

      I had a synchronous invocation service that compute join and aggregate data in the cache and return client specific object after processing.
      I had set up appropriate thread count for the proxy nodes and invocation service thread count to 20. Also the underneath cache services that are used by the invocation service are set to 4.

      The issue is the time required to process the concurrent request increases linearly with the number of concurrent thread.

      For example single Invocation service finished in 3 sec.

      10 Parallel threads took 11 sec

      15 Parallel threads took 16 sec

      The issues all the threads finished processing almost at the same time. I have enough heap memory and threads to process this request.

      Do we need to tune some buffer size on the proxy to handle the load as all the call return almost 3000+ objects as the result of the single call.

      Please suggest some insight as I am having no clue about the behavior. As per my expectation if system has enough thread count than it should finish processing at least in range of 5 sec for 10-15 threads.

      Please let me know if additional information is needed from my side.

      Regards,
      Ashish

      Edited by: 875813 on Oct 9, 2012 10:19 AM
        • 1. Re: Invocation Service Not Scaling
          user123799
          Hi Ashish,

          Without knowing a lot more detail it is very hard to say why you are experiencing such poor scaling, the number of possible factors is just too great. Such possibilities include, but are not limited too:
          * Infrastructure (CPU flat out, not enough cores, network choked or slow etc)
          * Extend client design, (stopping tasks or results being handled in parallel.
          * Grid configuration
          * Invocable design, (causing deadlock, livelock, starvation or serialisation of request)
          * etc etc

          Andy
          • 2. Re: Invocation Service Not Scaling
            robvarga
            875813 wrote:
            Coherence version 3.6.0.4

            I had a synchronous invocation service that compute join and aggregate data in the cache and return client specific object after processing.
            I had set up appropriate thread count for the proxy nodes and invocation service thread count to 20. Also the underneath cache services that are used by the invocation service are set to 4.

            The issue is the time required to process the concurrent request increases linearly with the number of concurrent thread.

            For example single Invocation service finished in 3 sec.

            10 Parallel threads took 11 sec

            15 Parallel threads took 16 sec

            The issues all the threads finished processing almost at the same time. I have enough heap memory and threads to process this request.

            Do we need to tune some buffer size on the proxy to handle the load as all the call return almost 3000+ objects as the result of the single call.

            Please suggest some insight as I am having no clue about the behavior. As per my expectation if system has enough thread count than it should finish processing at least in range of 5 sec for 10-15 threads.

            Please let me know if additional information is needed from my side.

            Regards,
            Ashish

            Edited by: 875813 on Oct 9, 2012 10:19 AM
            Hi Ashish,

            what do those threads do exactly?

            Also, are you sure that having a thread-count of only 4 on the cache service does not act as a bottleneck? Can you verify that the cache service does not build up a backlog?

            Do you have sufficient cores to run each invocation service and cache service thread on a different core? If not then number of cores may be a bottleneck...

            Best regards,

            Robert