oracle instances are terminating on various nodes with errors in exadata
The oracle instances are terminating on various nodes with errors like these but then we are able to start the instance in some cases. It looks like disk access issues.
This is from node 5
ORA-00604: error occurred at recursive SQL level 3
ORA-00376: file 1 cannot be read at this time
ORA-01110: data file 1: '+DATA_PR02/orp722z/datafile/system.1256.861636119'
Errors in file /u01/app/oracle/diag/rdbms/orp722z/orp722z5/trace/orp722z5_rsm0_31902.trc:
Node 7
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_gmon_26811.trc:
ORA-15311: process terminated due to fenced I/O