Forum Stats

  • 3,826,618 Users
  • 2,260,681 Discussions
  • 7,897,033 Comments

Discussions

Growing Task Backlog

Viktar C
Viktar C Member Posts: 31
edited Jul 28, 2017 11:01PM in Coherence Support

Hi Coherence Community,

During my recent load test I have noticed a growing amount of tasks in the Coherence backlog:

taskscount.png

backlog.png

As a result the test ended up with timeouts occurred for all Coherence calls.

Unfortunately, I wasn't able to find any errors or warning messages in the Coherence logs. So, I have no idea what was a reason of the issue. All the checked Coherence-related metrics (memory utilization, network health, system's resources) seem good.

Could you please help me with an advice how can I proceed with my investigation? What logging or reporting should be enabled to debug it?

Some information regarding my current setup

Integration type: Extend Client (proxy model)

Num of storage nodes: 3

Num of proxy nodes: 3

Cache 1 type: Write-Behind

Cache 2 type: Read-Through

Viktar C

Best Answer

  • Tmiddlet-Oracle
    Tmiddlet-Oracle Member Posts: 125
    edited Jul 28, 2017 1:07AM Answer ✓

    Hi Viktar.

    A few questions.

    1) What Coherence version are you using?

    2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?

    3) Growing task backlog can indicate insufficient server resources to processes the requests.

    4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.

    5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)

    Tim

    Viktar C

Answers

  • Tmiddlet-Oracle
    Tmiddlet-Oracle Member Posts: 125
    edited Jul 28, 2017 1:07AM Answer ✓

    Hi Viktar.

    A few questions.

    1) What Coherence version are you using?

    2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?

    3) Growing task backlog can indicate insufficient server resources to processes the requests.

    4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.

    5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)

    Tim

    Viktar C
  • Viktar C
    Viktar C Member Posts: 31
    edited Jul 28, 2017 1:36AM

    Hi Tim, please find my answers below:

    1) What Coherence version are you using?

    It's 12.2.1.0

    2) what sort of workloads are you running? Is it just puts/ gets on the caches with read-write backing maps or are they entry processors?

    It's gets and puts for write-behind and gets for read-through caches.

    3) Growing task backlog can indicate insufficient server resources to processes the requests.

    I have already fixed my proxies' thread pooling settings - I had tread-count set to 5 for my proxy services. And I have already checked the performance, it seems the issue is gone. However, I'm still confused why I had nothing in my logs saying what's going on wrong...

    4) What are the thread-counts you are using on the server? If you are using 12.2.1 + you can leave out thread-count and it will use dynamic thread pool.

    We have migrated to the 12.2.1.0 recently, probably that's why we had this thread-count parameter remained.

    5) What sort of load of clients do you have you could potentially have a bottleneck on proxy if not configured correctly. (also can use dynamic thread pool)

    I hope that misconfiguration on my proxies was the issue of the performance drop.

  • Tmiddlet-Oracle
    Tmiddlet-Oracle Member Posts: 125
    edited Jul 28, 2017 2:26AM

    Thats good news.


    Regarding the proxy thread-pool, see https://docs.oracle.com/middleware/1212/coherence/COHCG/gs_best.htm#COHCG4936

    You can set a min and max if you want or leave out.

    Cheers

    Tim

  • Viktar C
    Viktar C Member Posts: 31
    edited Jul 28, 2017 12:35PM

    Thanks Tim for your help and reference! Just one thing is still not clear to me. Why might we want to have min and max values specified if we can just remain thread-count value omitted?

  • Tmiddlet-Oracle
    Tmiddlet-Oracle Member Posts: 125
    edited Jul 28, 2017 10:57PM

    the min- and max values can still be set and most use of min would be perhaps if you know on startup of a service, (or proxy service) you will have min of say 10 connections. Just avoids very small delays in startup.

    But nowadays the recommendation is to have no thread-count and the thread pool should find a happy medium.

    Hope this helps.

    Viktar C
  • Viktar C
    Viktar C Member Posts: 31
    edited Jul 28, 2017 11:01PM

    Thank you Tim, it's really helpful!

This discussion has been closed.