latch: library cache - cause or effect

 Oracle 10.2.0.4Linux 2.6

We experienced an application outage yesterday and I'm in a chicken vs egg situation.

Starting at 13:14 the application servers started experiencing timeouts/disconnects.     The root cause of the disconnects is unknown - the application logging masks the true error (bummer).  As soon as the application experienced the timeouts/disconnects it attempt to reconnect.  This resulted in a bit of a connection flood to the database.

OS watcher reported swapping at 13:17

zzz ***Tue Nov 12 13:17:20 GMT 2013

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

41  4 1206072  54524 148744 42130052    0    0   566   251    0     0  5  1 91  3

39  2 1206092  79724 148748 42130472    0 60844  2285 67812 8827 10767 64 36  0  0

75  3 1206800  43188 148744 42130100    0 75448  2734 81277 7705 10976 45 55  0  0

OS watcher reported the following: