Very High load (80/90) but no CPU contention...

Hello Group,

I have a performance problem and I'm a bit stuck.

DB : 11.2.0.4 (Standard Edition One)

OS : Red Hat Enterprise Linux Server release 7.1 (Maipo)

Server is virtualized with OVM (1 socket 16 cores, 48G of RAM), storage is Oracle as well.

The app running on this database handles cash registers from 250 different clients, with each client stored in its own identical but independent schema. The cash registers connect and disconnect at each transaction on the client side (the shops), no connection pooling. Most of the queries are very similar or identical.

Customers complain from poor performance and we see spikes of CPU usage on the server, with runqueues in the high 80 or 90. No swapping or no feeling of sluggishness from the command line, all the commands issued come back instantly. I ran some statspack reports (see attached), but what I've found in them stumped me.