9 Replies Latest reply: Aug 23, 2012 1:35 PM by 878816 RSS

    Invocation Service Thread Count

    878816
      I am trying to load test my Invocation service on Coherence version 3.6.0.4 it seems to bolltenack at 10 threads.

      I had increased the thread count to 50 and can see the thread count is setting to 50 in the JConsole. But when trying to fire the 50 concurrent request its batching in the group of 10.

      Is there some limitation on the thread count number.

      Regards,
      Ashish Garg
        • 1. Re: Invocation Service Thread Count
          user738616
          Hi Ashish,

          Does the client machine has enough capacity to spawn 50 threads concurrently?

          Cheers,
          NJ
          • 2. Re: Invocation Service Thread Count
            878816
            I am able to resolve the issue actually my proxy thread count was set to 10 that why its waiting from proxy to dispatch to Invocation Service.

            But in general how to scale the solution as we have pratcical limit on the thread run on the proxy and extend client not able to pick the member to execute the invocation.

            Is there some load balancer that we can put between client and cluster to scale out the invocation service?
            • 3. Re: Invocation Service Thread Count
              Jonathan.Knight
              Hi Ashish,

              As you have seen, if you use a Remote Invocation Service from an Extend client that invocation executes on the Proxy Service on the Extend Proxy node, not on an Invocation Service on the Extend Proxy node. So to scale it all you can do is increase the thread count of the Extend Proxy.

              All requests from a Client will consume a thread on the Proxy Service of the Extend Proxy node the client is connected to for the duration of that request, so the only way to scale up is to either add more Extend Proxy nodes or up the thread count of the proxy service. Unless you really do have a lot of concurrent requests (not necessarily number of clients) and those concurrent requests all take a long time to execute then you do not need a lot of proxy service threads.

              If you can increase the thread count on the Invocation Service (presumably on the server) to 50, why do you have a limit on the number of threads on the Proxy Service?

              JK
              • 4. Re: Invocation Service Thread Count
                878816
                Thanks JK for your reply.

                What will be the default behavior incase if we doesn't set the thread count. Does it going to allocate till it reaches system limit?

                I have single client which is a long running .net service that fire lot fo conccurrent request based on events.

                Since its a single client and a long running process it connect to one proxy and all request are funnel through it.

                Event though we had 3 proxy nodes they are not utilize in peak spike up though they are available. Is there a way we can make sure that client request are load balanced across multiple proxy?

                I have seen we have BIG IP loadbalancer that gives the load abalacing functionality. Other way is to ask client to reconnect based on request which doesn't seem to be elegant solution.

                Is there a way by which we can load balance the request to all proxy nodes?
                • 5. Re: Invocation Service Thread Count
                  Jonathan.Knight
                  Hi,

                  If you only have one client then it will only connect to a single extend proxy and all requests go over that single connection - that is how Coherence works.

                  Are you saying you single .Net service is very multi-threaded and all these threads are invoking concurrent requests to Coherence?

                  The only way to make a client have multiple load balanced connections is to either define multiple remote invocation services in the clients cache configuration file or instantiate multiple Configurable Cache Factory instances in the client.

                  JK
                  • 6. Re: Invocation Service Thread Count
                    878816
                    There is another issue I had observed. When I am processing the request the time to process concurrent request is growing along with the number of concurrent request.

                    I had proxy thread count set to 50. I am printing the thread ID in my Invocation services and can see its assign to a different thread.

                    For processing 1 request it takes 1 sec
                    For processing 10 requests it takes 10 sec.
                    For processing 20 requests it takes 18 sec.

                    I am wondering why the time is increasing with threads.
                    There is no locking in the code. So if the entire requests are processed in the parallel by different thread than I think all the request should finish in 1 sec.

                    The system has enough capacity to handle more than 50 threads.

                    Edited by: 875813 on Aug 15, 2012 6:55 AM
                    • 7. Re: Invocation Service Thread Count
                      878816
                      Any input on the last queries experts. Its really important for me to understand why the response time is increasing lineraly with the number of concurrent request.
                      • 8. Re: Invocation Service Thread Count
                        user639604
                        Were you running Invocable to all members? Or a particular member?

                        What's the thread count setting on other members?
                        • 9. Re: Invocation Service Thread Count
                          878816
                          I was running on different nodes but one invocable on one node. Also the thread count is set to 50.