Large IO contention on powerful storage
On 27 March system (Oracle Linux) has been upgraded to 5.4.17-2136.305.5.3. Also Oracle Database EE to 12.2.0.1.220118. Storage is Pure Storage X20 R3 (40TB)
After over a week of reasonably good performance, starting from 06.04 (Wednesday) large performance decrease has been noticed (but workloads stayed the same/similar). In everyday critical period 07:00 – 09:30 we observe many wait events (latch: cache buffers chains and free buffer waits mainly) and, above all, IO delays. OK, huge workload.
But at the same time strange system behaviour: while we observe OEM “Latency For Synchronous Single Block Reads” growing up (to 10ms from the usual 0,5ms) in the same time we see plenty (thousands!) of kworker system process. Result of Tanel Poder’s Process Snapper :