10 Replies Latest reply on Jun 27, 2012 9:43 PM by kurtlook

    Apex listener load

    Sue
      Hi,

      I have a couple of questions regarding a standalone Apex listener installation :

      1. How can we measure the load on Apex listener ? Are there any tools available ?
      2. Is there any value in adding another Apex listener instance for the same database ? How can we know if performance will be improved and if yes how can we know when it will be beneficial to add another one ?
      3. We would like to have a process in place to send an alert whenever the Apex listener is trashing. Any advice how to do it and what are the parameters we should watch ?

      thanks,
      Sue

      Edited by: Sue on Jan 26, 2012 1:09 PM
        • 1. Re: Apex listener load
          Udo
          Hi Sue,

          I'm not able to answer all your questions completely, but I'll try to get some light into the areas you named. Perhaps we can evolve the story you are searching for step by step.

          First, I think it would be important to make sure we're talking about the same scenario.
          a standalone Apex listener installation
          How exactly would this look like? Is it a single APEX Listener running in Standalone Mode? Where is it located, where is your database, etc.
          I really recommend to redisgn that kind of deployment architecture, as the Standalone Mode is not supposed to be used for any productive use. On the other hand, you wouldn't think much about load if you just want to have a development/quick-and-dirty testing environment.
          1. How can we measure the load on Apex listener ? Are there any tools available ?
          This depends on what kind of load you want to measure and what kind(s) of application(s) you run behind the APEX Listener.
          Basically, the APEX Listener is a HTTP server, so you can use common HTTP load generators. Most of them are able to measure response times. But this would be a black box test - if response times increase, this can have several reasons that have nothing to do with the APEX Listener itself. So if you want a clean test, you need to have monitors for the complete scenario (including network, database, etc.).
          2. Is there any value in adding another Apex listener instance for the same database ? How can we know if performance will be improved and if yes how can we know when it will be beneficial to add another one ?
          In all cases I have seen so far, the backend (database) stalled much before there was much noticable load on the frontend (APEX Listener).
          But there's no general answer to that question as well. How do your clients access the first instance and how would they access a second one?
          Many productive sites require at least a failover-coverage, which means you need more than one and something to provide the failover mechanism. This could be a hardware loadbalancer or some simple AJP-setup (if the JEE Container that hosts your APEX supports AJP, of course), or some other implementation. In addition to failover, you can, of course, also implement a solution to distribute load to multiple nodes.
          3. We would like to have a process in place to send an alert whenever the Apex listener is trashing. Any advice how to do it and what are the parameters we should watch ?
          You can extract what you need for the APEX Listener part from page "listenerStatus", accessible by a user with "Manager" role (see doc for details).
          There are several parameters you can get there. A very basic one could be "listenerStatus is available". A second one could be "Current Time as reported on the page matches time on monitor (+/- tolerance)", other information like average/median value for processing time could be monitored equally. More complex ones could be "Traffic/Access/Error Count/Download Count/ decreased since last query", availability of database connections in pool, etc.
          What you need really depends on your scenario.
          Of course you should also monitor the JEE container (process load, deployment status, etc.), the host, the network, your backend, ...

          I hope this gives you a first insight. I'm looking forward to read about your scenario, so it'll be possible to give you some more precise answers for relevant aspects.

          -Udo
          • 2. Re: Apex listener load
            Sue
            Hi Udo,

            Thanks for your answer - you gave me a lot of helpful hints. The listener is a standalone one and it's installed on the same box as the database. Our feeling is also that the backend database is usually the culprit for the performance issues rather than the listener. About a month ago we increased the parameter MaxLimit from 10 to 30 and since then we did not have disconnects. Is there any advice on the recommended value for MaxLimit or it all depends on the environment and if so where can I see it ?

            thanks,
            Sue
            • 3. Re: Apex listener load
              Udo
              Hi Sue,
              About a month ago we increased the parameter MaxLimit from 10 to 30 and since then we did not have disconnects.
              It seems you really had an overload somewhere. Probably you have an application with some long-running queries, possibly some running slow because the database was overloaded. Of course, you can increase the number of available connections to the database so all users willing to access will be able to connect. But usually, they won't have fun as response times will be poor, and what can be even more painful - they can make it worse, as their requests will cause the database to get overloaded even more.
              Is there any advice on the recommended value for MaxLimit [...]
              The MaxLimit should be chosen according to the number of concurrent users you want to be able to serve. Usually, this shouldn't be much more than the number of parallel threads your database can handle. For example, I wouldn't recommend 30 sessions for a database instance that has only a few CPU threads to process them. Remember: If the database finishes faster, you'll get the session back to your pool in APEX Listener, so the next user request will be able to use it.
              [...] or it all depends on the environment and if so where can I see it ?
              Somehow, it depends on your applications and your end-user requirements. You could take a look on the following numbers in listenerStatus:
              Database Time in milliseconds (Average and Median) Active Requests Connections in the pool apex (Active and Available)
              The first one is obvious: The APEX Listener measures the time it's requests to the database need until they come back. If the two middle values start to climb fast, your database is probably getting under stress... Of course, it makes sense to give the APEX Listener a fresh start now and then so it will start a fresh calculation.
              In conjunction with the knowledge about the size of your connection pool, the number of active requests and active connections in pool give you a hint of the current usage and whether you may need more connections than the ones you have still available (if any). But again, you should consider the database load first to see whether your database would actually be able to handle additional sessions, or if adding load by allowing more connections would make things worse.

              -Udo
              • 4. Re: Apex listener load
                Sue
                Hi Udo,

                Sorry to bother you again. I'm still not clear about your reply below and therefore I'll paste the values that I see in the Statistics tab of our listener from production maybe you can explain more based on the values that we have.

                your reply :
                =========
                Database Time in milliseconds (Average and Median) Active Requests Connections in the pool apex (Active and Available)
                The first one is obvious: The APEX Listener measures the time it's requests to the database need until they come back. If the two middle values start to climb fast, your database is probably getting under stress... Of course, it makes sense to give the APEX Listener a fresh start now and then so it will start a fresh calculation.
                In conjunction with the knowledge about the size of your connection pool, the number of active requests and active connections in pool give you a hint of the current usage and whether you may need more connections than the ones you have still available (if any). But again, you should consider the database load first to see whether your database would actually be able to handle additional sessions, or if adding load by allowing more connections would make things worse.

                Our Values - can you explain what you mean based on the values below ?
                =============================================
                Current Time:     Wed Feb 01 14:54:36 EST 2012
                Server Uptime:     41 Days 22:37:07.420
                Total Accesses:     120995.0
                Total Traffic:     845.59M
                Total Errors:     281
                Total File Downloads:     3201
                Total Processing Time:     1 Days 05:55:28.798
                Processing Time in milliseconds:     
                Minimum: 3 Average: 890 Median: 37 Maximum: 301366
                Database Time in milliseconds:     
                Minimum: 0 Average: 656 Median: 22 Maximum: 262792
                Active Requests:     55
                Connections in the pool apex:
                Active: 0 Available: 5
                Security:     
                Total Entries: 0 Maximum Entries Allowed: 2000
                Caching Files:     
                Total Entries: 0 Maximum Entries Allowed: 500

                thanks again,
                Sue

                Edited by: Sue on Feb 1, 2012 1:02 PM
                • 5. Re: Apex listener load
                  Sue
                  My last quetion is still not answered - sorry for the inconvenience
                  • 6. Re: Apex listener load
                    Udo
                    Hi Sue,

                    no problem, it actually is a tricky topic.
                    Total Accesses: 120995.0
                    This number says you've had quite a few requests since the last time you've (re)started the APEX Listener. This makes the average and median values hard to interprete. Unfortunately, APEX Listener doesn't offer numbers based on a limited time frame, e.g. average/median of the past hour, or numbers in different categories, e.g. counter of requests answered within 100ms/500ms/1000ms/2000ms/.... The difference you have between your average and median value indicates that you had a lot of requests that were answered much slower than average, but it's hard to tell how much as the total number is too high to make a fair assumption.
                    Active Requests: 55
                    It's unlikely that you actually have 55 active requests - probably these are mainly the "broken connections" you've mentioned earlier. It seems APEX Listener didn't remove them from its counter because these requests didn't come back yet. Unfortunately they won't ever come back, so this number is useless right now as well.

                    I think you have to restart your APEX Listener and monitor the change of these numbers over a usual work day.
                    Connections in the pool apex:
                    Active: 0 Available: 5
                    These values seem to be accurate - no active session at that moment, while the pool has 5 idle connections. Depending on your JDBC parameters this could mean that this value is some kind of average threshold of concurrent users, but it's just a guess as I don't know your actual settings and at what point of activity you've taken that snapshot. Either way, at that very moment, there wouldn't be a need to raise the limit as the pool has enough free connections and has even downsized itself from the allowed maximum (if you still have 30 as MaxLimit) to the 5 it probably used in a certain (configured) interval.

                    I'm not sure what kind of answer you expect. It's definetly hard to give it without knowing the overall server load and how it is partitioned between database, APEX Listener and other resources.

                    -Udo
                    1 person found this helpful
                    • 7. Re: Apex listener load
                      Sue
                      Udo,

                      Thank you very much for your detailed reply. I will restart the listener an dstart monitoring daily to see the changes.

                      thanks again,
                      Sue
                      • 8. Re: Apex listener load
                        kurtlook
                        I really recommend to redisgn that kind of deployment architecture,
                        as the Standalone Mode is not supposed to be used for any productive use.
                        I've seen this stated a couple of times. What does this mean? I've seen statements that the Apex Listener is the future. Then I read Standalone Mode isn't even supposed to be used. I don't need anything more than simple connections to Apex and PL/SQL. I'm not sure why I'd want to install more than the standalone.

                        It is irksome that Oracle would sell something, tell you how to install it, and leave it to their forum support to say that something "wasn't meant for productiv use."

                        I'd like to also make clear that your support is fantastic. My frustration is with this front end stuff not working. I've read a ton of your posts trying to get this working.

                        Kurt
                        • 9. Re: Apex listener load
                          Udo
                          Hello Kurt,

                          please don't resurrect closed threads if you don't have to contribute something to the OPs actual question.
                          There have been a few posts on your question already, but I'll be glad to give you a summary as an answer to a new thread.

                          -Udo
                          • 10. Re: Apex listener load
                            kurtlook
                            Udo,

                            From my perspective, anything is better than EPG. So we started to convert http requests for both Apex and non-Apex (OWA,HTP) through the Apex Listener in Standalone mode. So far, 8,000 accesses without any issue. So I'm very pleased. So I'll re-state my question from a more positive perspective. What is there about Standalone deployment that would be troublesome? It is bone crunching call volume? Or use of certain esoteric features?

                            So far, things are looking good.

                            Kurt

                            Current Time: Wed Jun 27 15:50:17 CDT 2012
                            Server Uptime: 1 Days 21:55:36.106
                            Total Accesses: 8000.0
                            Total Traffic: 110.03M
                            Total Errors: 0
                            Total File Downloads: 106
                            Total Processing Time: 0 Days 00:09:37.640
                            Processing Time in milliseconds:
                            Minimum: 4 Average: 72 Median: 32 Maximum: 6336
                            Database Time in milliseconds:
                            Minimum: 0 Average: 64 Median: 26 Maximum: 6333
                            Active Requests: 0
                            Connections in the pool apex:
                            Active: 0 Available: 2
                            Security:
                            Total Entries: 0 Maximum Entries Allowed: 2000
                            Caching Files:
                            Caching file is disabled.