2 Replies Latest reply on Sep 11, 2012 6:08 AM by Billy~Verreynne

    File system went 'readonly'

      OL 5.6 x86-64

      This really is a Linux question, please bear with the set-up.

      This morning I discovered that one of my databases had crashed, reporting tht it couldn't write the control file because it was read-only:
      ORA-00206: error in writing (block 3, # blocks 1) of control file
      ORA-00202: control file: '/oradata/cognosdb/control02.ctl'
      ORA-27072: File I/O error
      Linux-x86_64 Error: 30: Read-only file system
      I checked all the permission from / on down and they all looked good - everything was either rw- or rwx for the owner oracle, and mapped perfectly to all of my other databases as far as file/directory permissions went'.

      /oradata/cognosdb is on a seperate mount point.

      My SA couldn't find anything at the SAN level or in the presentation of the disks to the server, but one of her logs had shown earlier where the server lost a couple paths to each disk on the SAN (there are 4 paths to each disk), then corresponding messages saying it had regained those paths.

      We eventually cleared the problem by dismounting the remouniting the disks.

      While waiting on the SA I did some googling and saw some old messages about linux resetting a file system to readonly if it detected issues with the mount point. I didn't get very far into that.

      Has anyone seen anythign like this before? If it was a matter of linux setting it to readonly due to the loss of the paths, where would we have expected to see confirmation of that? It wasn't showing in the standard file permissions from an 'ls' command.

      All's well that ended well, but I thought I'd see if I could learn a little something extra to put in my toolbox.

        • 1. Re: File system went 'readonly'
          I suggest to check the syslog /var/log/messages for any related issues, e.g. grep "devicename" /var/log/messages
          • 2. Re: File system went 'readonly'
            Unusual in my experience to have a failed mount point remounted as read-only when the path becomes available again.

            What does +/etc/fstab+ say for that mount point? What is the file system used for that mount? Is this SAN LUN seen via fibre channel or FCoE? Does this use SAN s/w (e.g. EMC Powerpath), or standard Linux multipath?

            As Dude said, the SA should look closer at the kernel logs - I would expect something in the config though to have caused this. This is not "standard" behaviour to mount a read-write mount point as read-only.