9 Replies Latest reply: Apr 30, 2012 4:03 PM by krramnik RSS

    RE: Database was down

    904735
      Hi,

      Our database is down and

      showing Errors in alert log like below

      ORA-221 : Error on write to control file


      ORA-206: Error in writing contril file (block 1, #blocks 1) of control file


      ORA-00202: control file: '/db/app/oracle/Db_1/oradata/orcl/control01.ctl*.core_dump_dest='ORA-27037: unable to obtain file statusLinux Error: 2: No such file or directory

      ORA-27041: unable to open file
      Linux Error: 30: Read - only File system
      Additional Information: 3


      DB version : 11.1.0.6
      OS version: Linux


      Pls help me...

      Thanks,
      Srini ...
        • 1. Re: RE: Database was down
          sb92075
          Srini wrote:
          Hi,

          Our database is down and

          showing Errors in alert log like below

          ORA-221 : Error on write to control file


          ORA-206: Error in writing contril file (block 1, #blocks 1) of control file


          ORA-00202: control file: '/db/app/oracle/Db_1/oradata/orcl/control01.ctl*.core_dump_dest='ORA-27037: unable to obtain file statusLinux Error: 2: No such file or directory

          ORA-27041: unable to open file
          Linux Error: 30: Read - only File system
          Additional Information: 3
          Errors should be expected when trying to WRITE to a READ ONLY file system.

          How did this occur?
          • 2. Re: RE: Database was down
            904735
            Hi,

            We dont the exact root cause, but

            One of our application team member is not able to connect to the database
            Immediatly we check the db status that is down.And we tried to startup the database And getting the above errors.
            • 3. Re: RE: Database was down
              Aman....
              Do you have the details of the mount point where the control file was stored before the db got crashed?

              Aman....
              • 4. Re: RE: Database was down
                sb92075
                Srini wrote:
                Hi,

                We dont the exact root cause, but

                One of our application team member is not able to connect to the database
                Immediatly we check the db status that is down.And we tried to startup the database And getting the above errors.
                Somebody had to have (re)mounted /db volume as READ ONLY (RO)

                Was there a preceding OS system crash?
                Sometimes after OS crash volumes get mounted RO to run fsck.
                • 5. Re: RE: Database was down
                  904735
                  Yes...


                  The details are,


                  [DBHIST]oracle@hicag:/data/u01/DBHIST$ ls -lrt

                  total 121120968

                  -rw-r----- 1 oracle oinstall 34358697984 Apr 28 23:00 temp01.dbf

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 02:08 redo02.log

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 02:15 redo03.log

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 07:50 redo01.log

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 07:55 redo04.log

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 08:03 redo05.log

                  -rw-r----- 1 oracle oinstall 28311560192 Apr 29 08:08 users_temp_tables

                  -rw-r----- 1 oracle oinstall 67117056 Apr 29 08:08 users01.dbf

                  -rw-r----- 1 oracle oinstall 2147491840 Apr 29 08:08 tapdata01.dbf

                  -rw-r----- 1 oracle oinstall 157294592 Apr 29 08:08 example01.dbf

                  -rw-r----- 1 oracle oinstall 104865792 Apr 29 08:08 assetix01.dbf

                  -rw-r----- 1 oracle oinstall 104865792 Apr 29 08:08 asset01.dbf

                  -rw-r----- 1 oracle oinstall 10737426432 Apr 29 09:11 dbhist02.dbf

                  -rw-r----- 1 oracle oinstall 32212262912 Apr 29 10:06 dbhist01.dbf

                  -rw-r----- 1 oracle oinstall 5368717312 Apr 29 14:01 TEMP_1

                  -rw-r----- 1 oracle oinstall 1069555712 Apr 29 17:01 system01.dbf

                  -rw-r----- 1 oracle oinstall 5809119232 Apr 29 17:02 undotbs01.dbf

                  -rw-r----- 1 oracle oinstall 1854414848 Apr 29 17:02 sysaux01.dbf

                  -rw-r----- 1 oracle oinstall 262144512 Apr 29 17:06 redo06.log

                  -rw-r----- 1 oracle oinstall 10764288 Apr 29 19:45 control03.ctl

                  -rw-r----- 1 oracle oinstall 10764288 Apr 29 19:45 control02.ctl

                  -rw-r----- 1 oracle oinstall 10764288 Apr 29 19:45 control01.ctl

                  [DBHIST]oracle@hicag:/data/u01/DBHIST$
                  • 6. Re: RE: Database was down
                    904735
                    HI sb,

                    OS level side we are not sure,

                    According to you are view is that OS LEVEL changes right....

                    Thanks you soo much...
                    • 7. Re: RE: Database was down
                      sb92075
                      Srini wrote:
                      HI sb,

                      OS level side we are not sure,

                      According to you are view is that OS LEVEL changes right....
                      yes, without any question; server side change happened.
                      • 8. Re: RE: Database was down
                        Aman....
                        You have shown us the status of the files. What was asked to check the status of the file system itself? Check it with your system admins that whether they did something when the mounted it , what options they changed because that's for sure that they did indeed change something.

                        Aman....
                        • 9. Re: RE: Database was down
                          krramnik
                          In addition to READ ONLY issue, please take note of the timestamp of files too.

                          The time stamp of the files are different. Generally, the last accessed time of all data files will be same. In the output you have pasted, we see 'few hours' difference in time stamp. They may not be consistent.

                          Probably, a file system restore was done by backup/platform team.

                          Better start checking when was your last good database backup and be prepared to restore from it.