This discussion is archived
13 Replies Latest reply: Feb 14, 2013 2:23 PM by Chris Muir RSS

AM instance removal from pool

BarbaraGelabert Newbie
Currently Being Moderated
Hi experts.

I need some help with clarifying some application module pool concepts.

What happens with application module instances and their database connections once they are removed from the AM pool, either because of Idle Instance Timeout or Maximum Instance Time to Live Timeout?

1. Aren't reserved DB connections returned back to the JDBC connection pool immediately? As Andrejus stated in his post ( [Optimizing Oracle ADF Application Pool|http://andrejusb.blogspot.com.es/2010/02/optimizing-oracle-adf-application-pool.html] ), even after an application module instance is removed from the pool, it's still alive and keeps DB connection reserved - in order to clean it, maximum time to live is set.
2. Why would be want to preserve DB connection after AM instance removal?

3. Do removed AM instances ever go back into the pool? Or else, wait to be garbage collected?

4. If Idle Instance Timeout (jbo.ampool.maxinactiveage) is set to be less than Maximum Instance Time to Live (jbo.ampool.timetolive). Which role plays the latter in AM instance removal / DB connection release?

Thanks,
Barbara
  • 1. Re: AM instance removal from pool
    Frank Nimphius Employee ACE
    Currently Being Moderated
    +1. Aren't reserved DB connections returned back to the JDBC connection pool immediately? As Andrejus stated in his post ( [Optimizing Oracle ADF Application Pool|http://andrejusb.blogspot.com.es/2010/02/optimizing-oracle-adf-application-pool.html] ), even after an application module instance is removed from the pool, it's still alive and keeps DB connection reserved - in order to clean it, maximum time to live is set.+

    Its configurable. If a user returns in time so the AM he used before could be handed back to him/her then the performance is better if no new database connection needed to be created. However, with the use of JDBC DataSources instead of JDBC URL in ADF BC the connection would be held and pooled by the Java EE container and not ADF Bc. So it is released.

    +2. Why would be want to preserve DB connection after AM instance removal?+

    To the time where ADF BC was configured with JDBC URL it was a performance gain

    +3. Do removed AM instances ever go back into the pool? Or else, wait to be garbage collected?+

    No. If an AM is cleared from the pool, on demand a new instance of the AM is created.

    +4. If Idle Instance Timeout (jbo.ampool.maxinactiveage) is set to be less than Maximum Instance Time to Live (jbo.ampool.timetolive). Which role plays the latter in AM instance removal / DB connection release?+

    I guess the answer is that the first "jbo.ampool.maxinactiveage" would not see light. However, maxinactiveage leads to passivation and so this should not be sacrified

    Frank
  • 2. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Hello, Frank. Thanks for your reply. More comments below:

    (In our case, we are using data source connections)

    +1. Aren't reserved DB connections returned back to the JDBC connection pool immediately? As Andrejus stated in his post ( [Optimizing Oracle ADF Application Pool|http://andrejusb.blogspot.com.es/2010/02/optimizing-oracle-adf-application-pool.html] ), even after an application module instance is removed from the pool, it's still alive and keeps DB connection reserved - in order to clean it, maximum time to live is set.+

    +[Frank] Its configurable. If a user returns in time so the AM he used before could be handed back to him/her then the performance is better if no new database connection needed to be created. However, with the use of JDBC DataSources instead of JDBC URL in ADF BC the connection would be held and pooled by the Java EE container and not ADF Bc. So it is released.+

    I know we can configure AM to disconnect DB connections upon release (Disconnect Application Module Upon Release option under Pooling and Scalability tab). This applies to AM instances being returned back to the pool. However, my question refers to AM instances being removed from the pool. Do they preserve DB connection even after removal?

    +2. Why would be want to preserve DB connection after AM instance removal?+

    +[Frank] To the time where ADF BC was configured with JDBC URL it was a performance gain+

    What about JDBC data source connections?


    +4. If Idle Instance Timeout (jbo.ampool.maxinactiveage) is set to be less than Maximum Instance Time to Live (jbo.ampool.timetolive). Which role plays the latter in AM instance removal / DB connection release?+

    +[Frank] I guess the answer is that the first "jbo.ampool.maxinactiveage" would not see light. However, maxinactiveage leads to passivation and so this should not be sacrified+

    Steve Muench on OTN Forums: Disabling jbo.ampool.timetolive
    +"If you have minavailablesize set to 0 and you set the timetolive to a value in milliseconds that is larger than the max idle time, then you will effectively have disabled the time to live consideration since you'll always have removed the AM from the pool for being idle before you'd ever have to consider removing it for having exceeded its time-to-live setting."+

    So, it's my understanding that when maxinactiveage < timetolive, then maxinactiveage determines AM instance removal while timetolive is a don't care value. Does it care for anything else?

    I also read the following Note in section 40.2.2.2 How passivation Changes When Optional Failover Mode is Enabled of the Fusion Developer's Guide for Oracle ADF: "Passivation can also occur when an application module is timed out [...] (jbo.ampool.timetolive), ...". Therefore, it is timetolive and not maxinactiveage that leads to passivation. Am I right?

    Thanks again,
    Barbara
  • 3. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Ok, I'm going to write my own theory about this. I would appreciate any comments and corrections you may suggest.

    1) Regarding what Andrejus wrote on his blog (http://andrejusb.blogspot.com.es/2010/02/optimizing-oracle-adf-application-pool.html): +"even after an application module instance is *removed* from the pool, it's still alive and keeps DB connection reserved - in order to clean it, maximum time to live is set."+ Perhaps he was referring to application module instances being released back to the pool, not removed from it. As Frank stated, disconnecting AM instances upon release is configurable through Disconnect Application Module Upon Release option under Pooling and Scalability tab. I guess Frank was also referring to the time AM instances are being checked back to (not removed from) the pool. By holding onto the JDBC connection, it allows each AM instance to keep its JDBC PreparedStatement objects open and usable across subsequent accesses by clients, thereby providing the best performance.

    2) AM instances are marked as candidate to be removed from the pool either when its maxinactiveage or timetolive timeout is reached. Here, maxinactiveage applies to Available AM instances, while timetolive applies to all AM instances in the pool.

    3) Once an AM instance is removed from the pool, it is passivated and its JDBC connection is released back to the pool immediately.

    What do you think?

    Edited by: Barbara Gelabert on 11-feb-2013 9:02
    I would really appreciate your feedback.
  • 4. Re: AM instance removal from pool
    Chris Muir Employee ACE
    Currently Being Moderated
    Hi Barbara

    This is my notes on maxinactiveage vs timetolive, your point 2:

    * Both are relevant and used by the jbo.ampool.monitorsleepinterval sweep – they represent two different strategies for removing AMs from the pool.

    * AMs are removed when either
    - the AM instance has been idle/unused greater than jbo.ampool.maxinactiveage
    - or the AM instance has been idle/unused great then jbo.ampool.timetolive and total number of AMs is <= jbo.ampool.minavailablesize
     
    * With the default settings for both these parameters and assuming your AM activity sets the current poolsize above jbo.ampool.minavailablesize, then the jbo.ampool.maxinactiveage parameter is the most important one to consider in terms of reclaiming idle AMs

    * In terms of jbo.ampool.timetolive this should only kick in when we have low activity on a system as it relates to the number of AMs being <= jbo.ampool.minavailablesize.

    For your reference Jobinesh's book has a discussion on AM pooling that is worth the read.


    On point 3, I don't believe passivation will not occur when the AM is removed from the pool, as a removal occurs when the AM is marked for clean up as the AM is idle or timed out. As such there is nothing to passivate.


    Regards,

    CM.
  • 5. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Hi, Chris. Thanks for your reply.

    2) Regarding your comment about jbo.ampool.timetolive: +or the AM instance has been idle/unused great then jbo.ampool.timetolive and total number of AMs is <= jbo.ampool.minavailablesize+

    According to the official docs, jbo.ampool.timetolive = "The number of milliseconds after which to consider an AM instance in the pool as a candidate for removal during the next resource cleanup regardless of whether it would bring the number of instances in the pool below minavailablesize."

    I understand that AM instances are removed after being idle/unused great than jbo.ampool.timetolive, regardless of total number of remaining AMs. Right?

    3) According to section +40.2.2.2 How passivation Changes When Optional Failover Mode is Enabled+ of the Fusion Dev. Guide: "Passivation can also occur when an application module is timed out". This makes sense for referenced AM instances that have been idle for either maxinactiveage or timetolive and then removed from the pool. It they were not passivated, then session data would be lost for subsequent requests of the same user.

    By the way, thanks for the reference to Jobinesh's book, I'll take a look :)
    Barbara
  • 6. Re: AM instance removal from pool
    Chris Muir Employee ACE
    Currently Being Moderated
    Hi Barbara

    Ya, you're right. I've written my note in logical order trying to say even though maxinactiveage pays attention to minavailablesize, timetolive doesn't, but I've written it incorrectly.

    In the end the strategy is maxinactiveage should catch or your normal idle AMs during normal activity and bring down to minavailablesize. But during very low activity, timetolive will go beyond that down to 0.

    On point 3, I'm not sure I agree with that "note". It's seems to be referring to a section that doesn't consider passivations, just time outs. The point is if they're aged out of the pool it's because they have exceeded max age/life expectancy/timeout, we want to kill them, so there would be no point to passivate them. Otherwise if you passivated them eventually your table would fill up with entries in the ps_txn table with dead passivated AMs.

    A final point, its well worth making use of the visual FMW Control diagnostics to experiment with the AM pool to confirm the behaviour.

    Cheers,

    CM.
  • 7. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Thank you, Chris.

    I still have some doubts about "dead" AM passivation. I think there are two scenarios where timed out AMs should be passivated:

    1. A user is filling a very long form (let's say for creating a new row) and takes longer than maxinactiveage to commit changes to the server. Meanwhile, a pool cleanup happens and removes the AM instance from the pool. If it was not passivated, commit action would fail, as there would be no row to commit in the newly assigned AM instance.

    2. A user is working on the same page for more than timetolive time. Then, a pool cleanup happens and removes the AM instance from the pool. If it was not passivated, the next request might also fail.

    I wish Oracle docs were more clear about this, thus not having to experiment by ourselves ;)

    Regards,
    Barbara
  • 8. Re: AM instance removal from pool
    Chris Muir Employee ACE
    Currently Being Moderated
    I understand the requirement but you have to understand the limits of the technology.

    With your solution how possibly could the midtier know that the client hasn't just died instead? It can't, there's nothing to differentiate a dead session from a user who is really slow unless a request goes from the client to the midtier. As such with your solution we'd have to treat even dead sessions as just slow users, and passivate their dead state. End result, in time we continue to collect an unlimited amount of passivated sessions and eventually we run out of space.

    You could argue that we could flush all the passivated sessions after a day or a week or a year. But different customers would want different periods. I've seen terminals in factories not be used for months on end - should we flush their passivated state after a day? Which requirements wins, yours, theres, do we build yet more software and parameters to make this configurable which will make it even harder to understand?

    As such I disagree. The two timeout parameters are just two different strategies for signalling a session is dead. You need an ultimate timeout parameter and that's what these provide.

    So what are your options to provide a longer living session?

    a) Build in some traffic between the client & server (e.g. PPR) on user action
    b) Make your forms smaller forcing more interaction with the server
    c) Lengthen the maxinactiveage/timetolive values
    d) Train the users about timeouts
    e) Or somehow introduce savepoints as part of the workflow (which still requires user interaction)

    As for the documentation, agreed, we continue to work to enhance the documentation (indeed I did so on this topic just last week). Yet we'll never get it perfect and your job as a developer is to interpret the documentation. in turn "learning from books" isn't as effective as experience, and experimenting will help you understand what's going on.

    CM.
  • 9. Re: AM instance removal from pool
    Frank Nimphius Employee ACE
    Currently Being Moderated
    f) set the session timeout warning to a frequency below passivation or whatever time you think suites your case

    docs.oracle.com/cd/E23943_01/web.1111/b31973/ap_config.htm#autoId23

    Frank
  • 10. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Hi, Chris, Frank.

    Thank you both for your replies. Chris, you just mentioned the point where I had lost. I totally agree with you there has to be a ultimate timeout for signalling dead sessions. A "dead session cleanup" approach would clutter the framework unnecessarily. And trying to distinguish actually dead sessions from those active but unused for a long time would lead to the same complexity.

    Please, note that the two scenarios I mentioned are hypothetical, just to expose two cases where passivation would care.
    I don't think the solution would be trying to provide a longer living session by (c) lengthening the maxinactiveage/timetolive values, but preventing undesired behaviours with dead sessions. Now, assuming that AM instances are never passivated on removal:

    1. I agree filling very long forms is not a good practice in web applications. As Chris suggested in options (a), (b) and (e), dividing them into smaller pieces (train steps) or somehow forcing user interaction, would be a solution. Nevertheless, let's assume long wait times between requests. Here, option (f) setting the session timeout warning to a frequency below maxinactiveage would suit. Thanks, Frank.

    2. Please, correct me if I am wrong here, this is the point where I am stuck: timetolive applies to ALL instances in the pool, regardless of whether they have been active or not. If that's right, then let's say a user is interacting with the same page (thus, same root AM) for more than timetolive (1h) time. This might be a common scenario in real life. Eventually, next request might generate an exception because a newly AM instance was assigned. Here, my suggest would be passivating the removed AM instance only if its maxinactiveage was not reached.

    I'll do some monitoring on the latter and tell you my findings.

    Thanks again for your help,
    Barbara
  • 11. Re: AM instance removal from pool
    Chris Muir Employee ACE
    Currently Being Moderated
    My understanding timetolive applies to unused AMs not all AMs. An exert from Jobinesh's book:
    >
    The pool monitor uses the following algorithm to remove unused application module instances. It uses two independent strategies to identify the candidates for removal:

    1) The application module pool monitor will remove all the unused application module instances which have been living in the pool for more than 3600000 milliseconds (one hour) without bothering about the minimum available size for the pool. You can override the maximum time to live for an application module by setting the jbo.ampool.timetolive configuration parameter. The default value is one hour. In real life applications, you may not use this property. To ignore this property at runtime, you can set the value to -1.

    2) The application module pool monitor will remove all the application module instances that are idle for 600000 milliseconds (10 minutes). This clean up will be stopped when the number of instances in the pool reaches the minimum available size. The idle timeout for an application module can be configured using the jbo.ampool.maxinactiveage parameter. The default idle timeout is 10 minutes.
    >
    Can you highlight where you've determined it applies to ALL so I can review please?

    CM.
  • 12. Re: AM instance removal from pool
    BarbaraGelabert Newbie
    Currently Being Moderated
    Hi, Chris.

    I finally got it! timetolive applies only to unused instances in the pool, not all of them, as I misunderstood.
    Thanks a lot for your help.

    I found the definition of timetolive in section *41.2.7.2 Pool Sizing Parameters* of the Fusion Dev. Guide:
    Maximum Instance Time to Live (jbo.ampool.timetolive)
    The number of milliseconds after which to consider an application module instance in the pool as a candidate for removal during the next resource cleanup regardless of whether it would bring the number of instances in the pool below minavailablesize. The default is 3600000 milliseconds of total time to live (which is 3600 seconds, or one hour). The default value is sufficient for most applications.>

    As it doesn't specify unused instances, I thought it referred to any instances, regardless of being idle or not.

    Thanks!
    Barbara
  • 13. Re: AM instance removal from pool
    Chris Muir Employee ACE
    Currently Being Moderated
    Thanks Barbara. I'll make a note for our doc team if they can make that a little clearer.

    CM.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points