11 Replies Latest reply on Jan 24, 2011 12:16 PM by 688970

    Instacne Crashed with Linux-x86_64 Error: 5: Input/output error

    826439
      Hi All,
      My instance got crashed while resizing the one of datafiles.
      In the alert file I got the below error:

      ORA-00202: control file: '<PATH>'/control.ctl'
      ORA-27091: unable to queue I/O
      ORA-27072: File I/O error
      Linux-x86_64 Error: 5: Input/output error
      Additional information: 4
      Additional information: 304
      Additional information: -1

      Could you please suggest the reason why it happen?

      After this one of mount point went bad. Server was rebooted.
      And after the starting the database again, it works fine.

      I'm trying to find the cause why Instance got crashed.

      Please Suggest.

      Thanks,
      Ashwani N.
        • 1. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
          g777
          hi

          for more information scan the /var/log/messages*
          or run dmesg
          • 2. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
            826439
            I can't see any /var/log/message for that particular day.

            But I can see some other files @ /var/log
            demsg
            acpid

            Thanks,
            Ashwani N.
            • 3. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
              688970
              Hi

              This error is due to lack of space.

              If you are using RMAN to create backup , then free up some space byy deleting old backup and archive log or one of your tablespace (possibly system) could be full. Add a datafile to it.

              Also check the limit of Oracle user. It should be unlimited(probably it would be set to unlimited).

              Another reason for this error could be bug. We had AIX system which use to give error ORA-27072.The metalink note id Note:390656.1. explains that bug.

              Hope it helps!!
              • 4. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                826439
                I dont think the error is because of the space issue.
                I do have enough space on the disk.

                Initially I was getting the error:

                {
                ORA-00345: redo log write error block 21443 count 2
                ORA-00312: online log 3 thread 1: '<PATH>/redo03.log'
                ORA-27072: File I/O error
                Linux-x86_64 Error: 5: Input/output error
                Additional information: 4
                Additional information: 21443
                Additional information: -1

                After that i was getting
                ORA-00202: control file: '<PATH>/control01.ctl'
                ORA-27091: unable to queue I/O
                ORA-27072: File I/O error
                Linux-x86_64 Error: 5: Input/output error
                Additional information: 4
                Additional information: 35
                Additional information: -1

                finally i got this:

                ORA-00221: error on write to control file
                Wed Jan 19 06:54:45 2011
                ORA-00221: error on write to control file
                Wed Jan 19 06:54:45 2011

                ORA-00221: error on write to control file
                Wed Jan 19 06:54:45 2011
                CKPT: terminating instance due to error 221
                }

                Please advice.

                Thanks,
                Ashwani N.
                • 5. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                  688970
                  Hi Ashwani,


                  It seems your control file is corrupted. If you have got a second control file (if u have multiplexed to another location), then try starting your database with that file.

                  You can also either restore or re-create your control file from a backup or script (I'd use the backup as a first choice).

                  Hope it helps!

                  Edited by: jazz81 on 21-Jan-2011 11:33
                  • 6. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                    826439
                    Thanks Jazz!!

                    The problem occur two days back.
                    There was nothing wrong with the control file.

                    Actually while resizing datafile one of mount point went bad.
                    After rebooting the server, we started the database and it worked fine.

                    Now I'm trying to find the root cause, why it happened?

                    Please suggest.
                    • 7. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                      688970
                      Ashwani,

                      The datafile header and the control file were not syncronized. When u restarted the server the smon process did the syncronization and as a result you did not get the error after you have rebooted the system. Ensure that you take a backup before you do your resizing.

                      Regards
                      J
                      1 person found this helpful
                      • 8. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                        826439
                        Thanks Jazz.

                        The same I could understand from the alert logs.

                        But now the question is: Is it possible that datafile header and the control file are not in sync, if yes Is there any way to check whether datafile header and the control file are in sync or not?

                        As this time My DB got crash while resizing the datafile (Might be because of the same reason). I want to make sure it should not happen in future.

                        Please suggest.


                        Thanks,
                        Ashwani N.
                        • 9. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                          688970
                          Ashwani,

                          If they are out of synchronization, your database would not be working in first place. Any files that are not in sync, oracle offlines them. You will get errors related to any files that are out of sync. Following is the way to find out the syncronization

                          SQL> SELECT checkpoint_change#, controlfile_change# FROM v$database;

                          CHECKPOINT_CHANGE# CONTROLFILE_CHANGE#
                          ------------------ -------------------
                          1147399576 1147408388


                          SQL> select FILE#,checkpoint_change# FROM v$datafile;

                          FILE# CHECKPOINT_CHANGE#
                          ---------- ------------------
                          1 1147399576
                          2 1147399576
                          3 1147399576
                          4 1147399576
                          5 1147399576
                          6 1147399576
                          7 1147399576
                          8 1147399576
                          9 1147399576
                          10 1147399576
                          11 1147399576

                          FILE# CHECKPOINT_CHANGE#
                          ---------- ------------------
                          12 1147399576
                          13 1147399576
                          14 1147399576
                          15 1147399576
                          16 1147399576
                          17 1147399576
                          18 1147399576
                          19 1147399576
                          20 1147399576
                          21 1147399576
                          22 1147399576

                          SQL> SELECT first_change# FROM v$log WHERE status = 'CURRENT';

                          FIRST_CHANGE#
                          -------------
                          1147399576


                          Hope this helps :)

                          Regards
                          J
                          • 10. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                            826439
                            Jazz,

                            As you told in previous note:

                            "The datafile header and the control file were not syncronized. When u restarted the server the smon process did the syncronization and as a result you did not get the error after you have rebooted the system"

                            Actually my DB was running and I was trying to resize the datafile.

                            In what case above scenario may occur.

                            Thanks,
                            Ashwani N.
                            • 11. Re: Instacne Crashed with Linux-x86_64 Error: 5: Input/output error
                              688970
                              Ashwani,

                              There can be many scenarios and a typical scenario is when you are trying to recover a database after a crash where you have an old controlfile checkpoint SCN which is lesser that datafile checkpoint SCN. In this case you have to apply all available archive logs and open the database with resetlogs so all that all file headers and control files get updated with the correct information.

                              The following link is worth reading

                              http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:48052838748707

                              Regards
                              J

                              Edited by: jazz81 on 24-Jan-2011 12:15