3 Replies Latest reply on Dec 15, 2005 11:25 AM by 56515

    ORA-01115: IO error reading block

    56515
      Hi,

      We have installed a Oracle Database 10g R2 RAC.
      The env are as follows:

      OS: SELS 9 64bits
      Hardware: two IBM 366 with 4CPUs
      File System: OCFS2

      oracle@rac1:~> ulimit -a
      core file size (blocks, -c) 0
      data seg size (kbytes, -d) unlimited
      file size (blocks, -f) unlimited
      max locked memory (kbytes, -l) 8589934590
      max memory size (kbytes, -m) unlimited
      open files (-n) 999999
      pipe size (512 bytes, -p) 8
      stack size (kbytes, -s) unlimited
      cpu time (seconds, -t) unlimited
      max user processes (-u) 999999999
      virtual memory (kbytes, -v) unlimited

      It ran quite well except a daily job.
      Every night, when the job runs, we would get the errors randomly anywhere in our store procedure.
      The errors are like these.

      ORA-01115: IO error reading block from file 4 (block # 8713)
      ORA-01110: data file 4: '/opt/oracle/oradata/dvpc/users01.dbf'
      ORA-27091: unable to queue I/O
      ORA-27072: File I/O error
      Linux-x86_64 Error: 4: Interrupted system call
      Additional information: 3
      Additional information: 8699
      Additional information: 49152

      ORA-01115: IO error reading block from file 20 (block # 154540)
      ORA-01110: data file 20: '/opt/oracle/oradata/dvpc/dvpc2nd_indx02.dbf'
      ORA-27091: unable to queue I/O
      ORA-27072: File I/O error
      Linux-x86_64 Error: 11: Resource temporarily unavailable
      Additional information: 3
      Additional information: 154531
      Additional information: 8192


      Any suggestions would be appreciated.
      Jimmy
        • 1. Re: ORA-01115: IO error reading block
          8734
          Hi

          One metalink note describe ORA-01115 as

          The main causes for an ORA-01115 are:

          1. HARDWARE PROBLEMS

          - Disk controller problems: the most common, and usually intermittent.

          - Disk problems: these include bad blocks, disk malfunctioning, etc.

          2. DATA BLOCK CORRUPTION (AT THE PHYSICAL LEVEL)

          Usually caused by previous hardware problems.

          3. PROBLEMS HANDLING VERY LARGE DATAFILES

          In the same note it suggests to check the objects involved.
          "If the datafile belongs to a user or index tablespace, you may also
          address the problem as an object recreation issue if the ORA-01115
          occurs consistently against the same objects (tables, indexes, etc.).
          The following query returns the object in which the bad block is: "

          SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
          WHERE FILE_ID = 4 and 8713 BETWEEN BLOCK_ID
          AND BLOCK_ID + BLOCKS - 1;

          SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
          WHERE FILE_ID = 20 and 154540 BETWEEN BLOCK_ID
          AND BLOCK_ID + BLOCKS - 1;

          It will be a good idea to check your alertlog and relevant trace files for more information.

          Out of curiousity, are you running any backup job at the same time??

          Rgds
          Adnan
          • 2. Re: ORA-01115: IO error reading block
            56515
            Hi,

            Here are the more detail information we found out.

            1) We could shutdown normal/startup successfully without error.
            2) We could import database with full option successfully without error.
            3) When we shutdown one instance of the RAC, then the job would run without error.
            4) The error blocks would be randomly anywhere in the database.
            5) We disabled all backup actions and archive.
            6) When the error of the job occurred, we couldn't find any log in alertlog. We found this error in our store procedure logging table which would log some info and errors of our programs.
            7) We found this error in the alertlog, but it wasn't caused by our job. Here is the trace file detail.

            /opt/oracle/admin/dvpc/bdump/dvpc2_j000_27339.trc
            Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
            With the Partitioning, Real Application Clusters, OLAP and Data Mining options
            ORACLE_HOME = /opt/oracle/product/dvpc
            System name: Linux
            Node name: rac1
            Release: 2.6.5-7.191-smp
            Version: #1 SMP Tue Jun 28 14:58:56 UTC 2005
            Machine: x86_64
            Instance name: dvpc2
            Redo thread mounted by this instance: 2
            Oracle process number: 37
            Unix process pid: 27339, image: oracle@rac1 (J000)

            *** ACTION NAME:(AUTO_SPACE_ADVISOR_JOB) 2005-12-14 22:00:12.152
            *** MODULE NAME:(DBMS_SCHEDULER) 2005-12-14 22:00:12.152
            *** SERVICE NAME:(SYS$USERS) 2005-12-14 22:00:12.152
            *** SESSION ID:(126.527) 2005-12-14 22:00:12.152
            *** 2005-12-14 22:00:12.152
            ORA-12012: error on auto execute of job 8887
            ORA-01115: IO error reading block from file ORA-01115: IO error reading block from file 3 (block # 36137)
            ORA-01110: data file 3: '/opt/oracle/oradata/dvpc/sysaux01.dbf'
            ORA-27091: unable to queue I/O
            ORA-27072: File I/O error
            Linux-x86_64 Error: 4: Interrupted system call
            Additional information: 3
            Additional information: 36121
            Additional information: 106496
            ORA-06512: at "SYS.PRVT_ADVISOR", line 1624
            ORA-06512: at "SYS.DBMS_ADVISOR", line 186
            ORA-06512: at "SYS.DBMS_SPACE", line 1338
            ORA-06512: at "SYS.DBMS_SPACE", line 1554
            (block # )


            Regards,
            Jimmy
            • 3. Re: ORA-01115: IO error reading block
              56515
              Hi,

              We applied the SP3 of the SELS 9, it upgraded the OCFS2.
              It seems okay now.

              Regards,
              Jimmy