I am using Weblogic 10.3.4 + Oracle DB both on Solaris machines (VM)
Unfortunately last week our app hosted on Weblogic became very very slow not being able to serve new users coming (everything lasted for 1h).
In order to find out the cause is we asked reports from the OS level , DB level.
The reports shown:
a very intesive CPU usage that hour. (clearly was a peak from number of users using the system)
Connection Pool reached the top limit on Weblogic (200 connections !)
AWR stats from Db shown a degradation in response times.
NOW who was first the chicken or the egg ?
A very slow DB can cause WEBlogic to be slow and to reach the max numeber of connections from pool.
However if the CPU is very used , it might be that the threads holding the DB connections to be so busy that they do not return the connection to the pool (because the processor is busy)==> a new client comes a new connection is assgined and so on ...so the connection pool might be exhausted as well in this way ...?
Edited by: csergiu77 on Jan 26, 2013 12:45 AM
what you say makes perfectly sense: each request takes a long time to be processed because of the CPU bottleneck, hence the connection is held for a long time.
I would suggest profiling your application with yourkit or similar tool, to understand where all that CPU is wasted.
Also make sure you have enough heap to handle all that load, monitoring GC time can also be useful.
You might want to throttle requests (eg with a WorkManager) to avoid clogging your system with too much load pushing it beyond the point of maximum throughput.
I identified some methods that have high cpu usage ...and as well i am thinking on putting a work manager until the dev team will fix he issue in the code.
This are some specific methods on a WS.
It is possible to assign work manager (with max constraint) to specific methods (to this level of granularity) ?
If yes , let's say that i put a max thread constraint to 10 what happend to the 11 th concurent request ? it is denied or put on wait ?
(as i remember is denied)