Discussions
Categories
- 17.9K All Categories
- 3.4K Industry Applications
- 3.3K Intelligent Advisor
- 62 Insurance
- 536.1K On-Premises Infrastructure
- 138.2K Analytics Software
- 38.6K Application Development Software
- 5.7K Cloud Platform
- 109.4K Database Software
- 17.5K Enterprise Manager
- 8.8K Hardware
- 71.1K Infrastructure Software
- 105.2K Integration
- 41.6K Security Software
Growing Task Backlog
Hi Coherence Community,
During my recent load test I have noticed a growing amount of tasks in the Coherence backlog:
As a result the test ended up with timeouts occurred for all Coherence calls.
Unfortunately, I wasn't able to find any errors or warning messages in the Coherence logs. So, I have no idea what was a reason of the issue. All the checked Coherence-related metrics (memory utilization, network health, system's resources) seem good.
Could you please help me with an advice how can I proceed with my investigation? What logging or reporting should be enabled to debug it?
Some information regarding my current setup
Integration type: Extend Client (proxy model)
Num of storage nodes: 3
Num of proxy nodes: 3
Cache 1 type: Write-Behind
Cache 2 type: Read-Through
Best Answer
-
Hi Viktar.
A few questions.
1) What Coherence version are you using?
2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?
3) Growing task backlog can indicate insufficient server resources to processes the requests.
4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.
5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)
Tim
Answers
-
Hi Viktar.
A few questions.
1) What Coherence version are you using?
2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?
3) Growing task backlog can indicate insufficient server resources to processes the requests.
4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.
5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)
Tim
-
Hi Tim, please find my answers below:
1) What Coherence version are you using?
It's 12.2.1.0
2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?
It's gets and puts for write-behind and gets for read-through caches.
3) Growing task backlog can indicate insufficient server resources to processes the requests.
I have already fixed my proxies' thread pooling settings - I had tread-count set to 5 for my proxy services. And I have already checked the performance, it seems the issue is gone. However, I'm still confused why I had nothing in my logs saying what's going on wrong...
4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.
We have migrated to the 12.2.1.0 recently, probably that's why we had this thread-count parameter remained.
5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)
I hope that misconfiguration on my proxies was the issue of the performance drop.
-
Thats good news.
Regarding the proxy thread-pool, see https://docs.oracle.com/middleware/1212/coherence/COHCG/gs_best.htm#COHCG4936You can set a min and max if you want or leave out.
Cheers
Tim
-
Thanks Tim for your help and reference! Just one thing is still not clear to me. Why might we want to have min and max values specified if we can just remain thread-count value omitted?
-
the min- and max values can still be set and most use of min would be perhaps if you know on startup of a service, (or proxy service) you will have min of say 10 connections. Just avoids very small delays in startup.
But nowadays the recommendation is to have no thread-count and the thread pool should find a happy medium.
Hope this helps.
-
Thank you Tim, it's really helpful!