I need some help pin pointing a really big problem we're having right now regarding performance of ADF applications on a websphere 7 application server. I'm using JDeveloper 220.127.116.11.0.
First, a little background. Our applications used to run well (about late November). Our server team did some consolidation of the servers, and put more applications on our Websphere test servers. The same ADF builds we used back in November now have painfully slow response times. For example: when a user tries to create a new row, there's around a 40 second delay. Previously, it would be almost instantaneous. The delays are exclusive to database transactions (insert/edit row items, along with intermittent delays when scrolling through tables). I've been told the database server is running optimally as well. I trust this assessment, because running it locally from the integrated weblogic servers show original performance speeds.
Now, I've been in contact with the app server admins (I don't have access to the servers). They've repeatedly told me that the server is not overloaded, or out of memory, and that everything should be fine. They've suggested that I look into performance of the applications. I'm skeptical, but I have been looking into performance tuning of the applications. Anything that can help with performance is a good thing.
If our ADF applications were performing well previous to the server shuffle, and now it they are not, is there any clear server settings that we should be looking for? I would like to communicate to the server team some options they can look into that might be causing ADF applications to perform slow, but not show any server overload. In addition, is there any obvious application performance tuning that I might need to implement?
I'm expecting more questions than a clear cut solution at this point. If you need more information, please let me know and I'll do my best to get you what you need.
One thing you should do is to track down where you are spending all the time. For this diagnosticlogging can be used (https://www.youtube.com/watch?v=N-Uo5Uj1uPk). If you can't set it up you can print some time stamps in your application to find out where the time is spend (start bib: find out if the ui or the model spends the time, then dig deeper).
Tuning is always a good thing to to if you know what to tune. Just hanging parameters without knowing if you have a problem in this area won't help you, it just cost time.
Make sure you know the environment (e.g. cluster or not, DB changes anything can be a hint) your application runs in (and have run in before). May be there is a slow router in the network which consumes the time.
Thanks for the responses Timo and Zlatko.
I'm working on narrowing down the issue now (timestamps/logging were a good idea, and I'm verifying that the connection pools look normal), this information puts me in a good direction I think.
Something to note: Sometimes performance is worse and sometimes it's better. I can see a swing anywhere from 5-8 second response time at best, to 40-50 second response times. I'm not ready to pin it completely on the server, but are there any known conditions where the application could cause such irregularity?
As Timo indicated you should be looking at the logging capability to help you pin point the problem.
One thing to check with the server team is their JDBC settings specifically for the JDBC datasources and the pooling parameters they have.
This is beyond the ADF AM pool settings.
Thanks for the replies everyone, I wanted to give an update.
I was able to pour through dynatrace on the requests, and found a lot of the processes were being stalled due to improper logging settings on the application server. (I first suspected this when I enabled logging on my local testing and saw a performance drop).
There's a couple less severe performance hits that I'm hoping to get a little feedback on.
The garbage collecting is running very frequently, and causing a 1-3 second delay in some transactions. Could this be a logging related issue as well? Or is there something I should do to improve the garbage collection performance. I'm in the process having the memory allocated to the JVM from 2gigs to 4gigs to see if this does anything. What else can I try?
Also, I'm getting "application module can't find a database connection" on transactions. It seems like the connection gets terminated frequently, and is having to re-establish. This causes a 1-3 second delay. I can reproduce the delay if I have a table set up, and I click row-by-row on the items. Typically, every 8th row will be the one that has an application module problem in the dynatrace logs. Is this a server setting that I can fix (I'm guessing something in the JDBC)? Or is there any application settings that I can try that will stop it from dropping the connnection to the database so frequently.
Thanks again for all the help!
You mean you run the server on only 2gb?
This is not enough for a production machine. You should at least have 4gb or better more then 4gb, depending on the number of users working with your system.
Tuning the GC is always an option to try on the production machine. I guess you see the gc hit frequently because of the minimal memory of your jvm. Before going into detail on gc tuning get more memory and test again.
Getting a new connection every 8 rows? This should be investigated as it is not normal. May be this too is a memory problem, but check your am pooling settings.
Verify AM pool parameters:
Failover Transaction State Upon Managed Release (jbo.dofailover) = false (default)
Disconnect Application Module Upon Release (jbo.doconnectionpooling) = false (default)
Enable Application Module Pooling (jbo.ampool.doampooling) = true (default)
This was one of our testing server. We've upped to 4gb and saw the same results:
JBO-25200: Application module is not connected to a database.
And lots of garbage collecting causing suspensions in response times.
All the AM pool parameters that Zlatko listed were set to the defaults. Is there any other settings I should be checking? Perhaps on the application server itself?
I've checked the Datasource of the application, all of the settings seem pretty standard.
connection timeout: 180 seconds
Maximum connections: 10
Minimum connections: 1
Reap time: 180 seconds
Unused Timeout: 1800
Aged Timeout: 0
Purge Policy: Entire Pool
Behavior that may help pin this down:
Every 9th request when clicking on rows in a table, cause the JBO-25200 error.
That leads me to believe it has something to do with the max/min connections, but why would changing the selected row cause this? And is there anything I can do to prevent it from consuming a connection for every request?
If you raise the maximum connections on the pool side do you see a difference.
Is the 9th record behavior something you see with any page you create with ADF or a specific page/viewobject?
Can you monitor the connection on the WebSphere side and see if they are dropping/maxing out?