gc cr multiblock request
2-node RAC, Oracle 19.19.0.0 on RHEL 8.8. Each node has 80 CPUs and 1 TB memory. No underscore parameter set in the database. No optimizer_* parameter set.
On this database, a SQL that normally finishes in a few minutes sometimes runs for hours to finish, with no change in the execution plan (according to plan hash value). When this problem happens, the session always has one symptom, i.e. it starts to wait on the "gc cr multiblock request" event more and more, while during normal run, it shows mostly ASM IO, direct path read, or db file read events.
There's nothing abnormal in other parts of the system, such as OS (CPU and memory usage), network, storage, etc. We thought the interconnect could be slow but that's not the case. There're some "gc blocks lost" stats, but its occasional increase in value has no correlation with increase of the "gc cr multiblock request" waits. (We increased per-socket buffer memory /proc/sys/net/core/[rw]mem* to 4M, with no effect.) We tested by concentrating all user sessions on only one node in the hope of eliminating or reducing cross-node traffic, but the problem persists. In fact, strace on the Oracle server process experiencing severe