Forum Stats

  • 3,751,287 Users
  • 2,250,338 Discussions
  • 7,867,372 Comments

Discussions

ORA-01115: IO error reading block

56515
56515 Member Posts: 101
edited Dec 15, 2005 6:25AM in General Database Discussions
Hi,

We have installed a Oracle Database 10g R2 RAC.
The env are as follows:

OS: SELS 9 64bits
Hardware: two IBM 366 with 4CPUs
File System: OCFS2

[email protected]:~> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 8589934590
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 999999999
virtual memory (kbytes, -v) unlimited

It ran quite well except a daily job.
Every night, when the job runs, we would get the errors randomly anywhere in our store procedure.
The errors are like these.

ORA-01115: IO error reading block from file 4 (block # 8713)
ORA-01110: data file 4: '/opt/oracle/oradata/dvpc/users01.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 8699
Additional information: 49152

ORA-01115: IO error reading block from file 20 (block # 154540)
ORA-01110: data file 20: '/opt/oracle/oradata/dvpc/dvpc2nd_indx02.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 154531
Additional information: 8192


Any suggestions would be appreciated.
Jimmy

Comments

  • 8734
    8734 Member Posts: 415
    Hi

    One metalink note describe ORA-01115 as

    The main causes for an ORA-01115 are:

    1. HARDWARE PROBLEMS

    - Disk controller problems: the most common, and usually intermittent.

    - Disk problems: these include bad blocks, disk malfunctioning, etc.

    2. DATA BLOCK CORRUPTION (AT THE PHYSICAL LEVEL)

    Usually caused by previous hardware problems.

    3. PROBLEMS HANDLING VERY LARGE DATAFILES

    In the same note it suggests to check the objects involved.
    "If the datafile belongs to a user or index tablespace, you may also
    address the problem as an object recreation issue if the ORA-01115
    occurs consistently against the same objects (tables, indexes, etc.).
    The following query returns the object in which the bad block is: "

    SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
    WHERE FILE_ID = 4 and 8713 BETWEEN BLOCK_ID
    AND BLOCK_ID + BLOCKS - 1;

    SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
    WHERE FILE_ID = 20 and 154540 BETWEEN BLOCK_ID
    AND BLOCK_ID + BLOCKS - 1;

    It will be a good idea to check your alertlog and relevant trace files for more information.

    Out of curiousity, are you running any backup job at the same time??

    Rgds
    Adnan
  • 56515
    56515 Member Posts: 101
    edited Dec 14, 2005 9:31PM
    Hi,

    Here are the more detail information we found out.

    1) We could shutdown normal/startup successfully without error.
    2) We could import database with full option successfully without error.
    3) When we shutdown one instance of the RAC, then the job would run without error.
    4) The error blocks would be randomly anywhere in the database.
    5) We disabled all backup actions and archive.
    6) When the error of the job occurred, we couldn't find any log in alertlog. We found this error in our store procedure logging table which would log some info and errors of our programs.
    7) We found this error in the alertlog, but it wasn't caused by our job. Here is the trace file detail.

    /opt/oracle/admin/dvpc/bdump/dvpc2_j000_27339.trc
    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
    With the Partitioning, Real Application Clusters, OLAP and Data Mining options
    ORACLE_HOME = /opt/oracle/product/dvpc
    System name: Linux
    Node name: rac1
    Release: 2.6.5-7.191-smp
    Version: #1 SMP Tue Jun 28 14:58:56 UTC 2005
    Machine: x86_64
    Instance name: dvpc2
    Redo thread mounted by this instance: 2
    Oracle process number: 37
    Unix process pid: 27339, image: [email protected] (J000)

    *** ACTION NAME:(AUTO_SPACE_ADVISOR_JOB) 2005-12-14 22:00:12.152
    *** MODULE NAME:(DBMS_SCHEDULER) 2005-12-14 22:00:12.152
    *** SERVICE NAME:(SYS$USERS) 2005-12-14 22:00:12.152
    *** SESSION ID:(126.527) 2005-12-14 22:00:12.152
    *** 2005-12-14 22:00:12.152
    ORA-12012: error on auto execute of job 8887
    ORA-01115: IO error reading block from file ORA-01115: IO error reading block from file 3 (block # 36137)
    ORA-01110: data file 3: '/opt/oracle/oradata/dvpc/sysaux01.dbf'
    ORA-27091: unable to queue I/O
    ORA-27072: File I/O error
    Linux-x86_64 Error: 4: Interrupted system call
    Additional information: 3
    Additional information: 36121
    Additional information: 106496
    ORA-06512: at "SYS.PRVT_ADVISOR", line 1624
    ORA-06512: at "SYS.DBMS_ADVISOR", line 186
    ORA-06512: at "SYS.DBMS_SPACE", line 1338
    ORA-06512: at "SYS.DBMS_SPACE", line 1554
    (block # )


    Regards,
    Jimmy
  • 56515
    56515 Member Posts: 101
    Hi,

    We applied the SP3 of the SELS 9, it upgraded the OCFS2.
    It seems okay now.

    Regards,
    Jimmy
This discussion has been closed.