ORA-01115: IO error reading block

56515

Hi,

We have installed a Oracle Database 10g R2 RAC.
The env are as follows:

OS: SELS 9 64bits
Hardware: two IBM 366 with 4CPUs
File System: OCFS2

[email protected]:~> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 8589934590
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 999999999
virtual memory (kbytes, -v) unlimited

It ran quite well except a daily job.
Every night, when the job runs, we would get the errors randomly anywhere in our store procedure.
The errors are like these.

ORA-01115: IO error reading block from file 4 (block # 8713)
ORA-01110: data file 4: '/opt/oracle/oradata/dvpc/users01.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 8699
Additional information: 49152

ORA-01115: IO error reading block from file 20 (block # 154540)
ORA-01110: data file 20: '/opt/oracle/oradata/dvpc/dvpc2nd_indx02.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 154531
Additional information: 8192

Any suggestions would be appreciated.
Jimmy

8734

Hi

One metalink note describe ORA-01115 as

The main causes for an ORA-01115 are:

1. HARDWARE PROBLEMS

- Disk controller problems: the most common, and usually intermittent.

- Disk problems: these include bad blocks, disk malfunctioning, etc.

2. DATA BLOCK CORRUPTION (AT THE PHYSICAL LEVEL)

Usually caused by previous hardware problems.

3. PROBLEMS HANDLING VERY LARGE DATAFILES

In the same note it suggests to check the objects involved.
"If the datafile belongs to a user or index tablespace, you may also
address the problem as an object recreation issue if the ORA-01115
occurs consistently against the same objects (tables, indexes, etc.).
The following query returns the object in which the bad block is: "

SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
WHERE FILE_ID = 4 and 8713 BETWEEN BLOCK_ID
AND BLOCK_ID + BLOCKS - 1;

SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
WHERE FILE_ID = 20 and 154540 BETWEEN BLOCK_ID
AND BLOCK_ID + BLOCKS - 1;

It will be a good idea to check your alertlog and relevant trace files for more information.

Out of curiousity, are you running any backup job at the same time??

Rgds
Adnan

56515

Hi,

Here are the more detail information we found out.

1) We could shutdown normal/startup successfully without error.
2) We could import database with full option successfully without error.
3) When we shutdown one instance of the RAC, then the job would run without error.
4) The error blocks would be randomly anywhere in the database.
5) We disabled all backup actions and archive.
6) When the error of the job occurred, we couldn't find any log in alertlog. We found this error in our store procedure logging table which would log some info and errors of our programs.
7) We found this error in the alertlog, but it wasn't caused by our job. Here is the trace file detail.

/opt/oracle/admin/dvpc/bdump/dvpc2_j000_27339.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /opt/oracle/product/dvpc
System name: Linux
Node name: rac1
Release: 2.6.5-7.191-smp
Version: #1 SMP Tue Jun 28 14:58:56 UTC 2005
Machine: x86_64
Instance name: dvpc2
Redo thread mounted by this instance: 2
Oracle process number: 37
Unix process pid: 27339, image: [email protected] (J000)

*** ACTION NAME:(AUTO_SPACE_ADVISOR_JOB) 2005-12-14 22:00:12.152
*** MODULE NAME:(DBMS_SCHEDULER) 2005-12-14 22:00:12.152
*** SERVICE NAME:(SYS$USERS) 2005-12-14 22:00:12.152
*** SESSION ID:(126.527) 2005-12-14 22:00:12.152
*** 2005-12-14 22:00:12.152
ORA-12012: error on auto execute of job 8887
ORA-01115: IO error reading block from file ORA-01115: IO error reading block from file 3 (block # 36137)
ORA-01110: data file 3: '/opt/oracle/oradata/dvpc/sysaux01.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 36121
Additional information: 106496
ORA-06512: at "SYS.PRVT_ADVISOR", line 1624
ORA-06512: at "SYS.DBMS_ADVISOR", line 186
ORA-06512: at "SYS.DBMS_SPACE", line 1338
ORA-06512: at "SYS.DBMS_SPACE", line 1554
(block # )

Regards,
Jimmy

56515

Hi,

We applied the SP3 of the SELS 9, it upgraded the OCFS2.
It seems okay now.

Regards,
Jimmy