7 Replies Latest reply: Jan 8, 2014 2:42 AM by User511360-OC RSS

    raw devices problem

    User511360-OC

      Hello, I made a model on Vmware ESX for testing Oracle RAC, so it's two Centos 5.10 machines with shared disks. I'm using raw devices for ocr, voting disks and data, and without ASMLib library.

      After installing clusterware and Oracle (10gR2), everything was OK until now, when after reboot of one node, no more shared disks exist anymore. For every raw disk related comand I got an error:

      #raw -qa

      Cannot open raw device '/dev/rawctl' (No such file or directory)

      #service rawdevices restart

      Assigning raw devices:

      /dev/raw/raw1 --> /dev/sdb1

      Cannot open raw device '/dev/rawctl' (No such file or directory)

      Didn't find anything related to this in /var/log's. Could be glad if someone could explain how to fix it.

        • 1. Re: raw devices problem
          Dude!

          Linux did not initially support the concept of raw devices and the way it was done under RHEL 3 and 4 was not really an efficient solution. The use of raw devices as of RHEL 5 has been depreciated. The device tree under the 2.6 Linux kernel is rebuild at every system restart and you will loose your /dev configuration. You have to use ASMLib or configure UDEV for device name persistence. Perhaps you were following setup instructions for RHEL 4 and earlier. Oracle under Linux opens datafiles with the O_DIRECT flag to bypass kernel buffering, which makes the use of raw devices unnecessary.

          • 2. Re: raw devices problem
            User511360-OC

            I think this is pure OS bug, because /dev/rawctl is kind of system device.

            Raw devices works without problem on RHEL5. For example we have production 2-node RAC. Now we must add more storage to it, and I made this model to test this behavior, beacuse I know that we could loose our /dev configuration. Final purpose was to add disks to UDEV and after this change the raw device bindings, so /dev/raw/raw1 will be binding to /dev/votedisk1 (udev binding) etc... Or is there another way to do the same ?


            • 3. Re: raw devices problem
              Dude!

              You will need to obtain the unique device identifies (/sbin/scsi_id -g -s /block/sdb/sdb1) of your cluster devices and use this info to map it in the UDEV configuration on each node. Since I do not know details of your configuration and what what you have configured I cannot say what is wrong or what else you can do.

               

              If you have access to My Oracle support I suggest the following:

               

              Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 (Doc ID 465001.1)

              Linux 2.6 Kernel Deprecation of Raw Devices (Doc ID 357492.1)

              • 4. Re: raw devices problem
                User511360-OC

                OK, thanks for this links. The main thing I understood, that I can change raw device bindings to block device bindings like this:

                "instead of running '/bin/raw /dev/raw/raw5 /dev/sdc1', run the command '/bin/ln -s /dev/sdc1 /dev/raw/raw5"

                without loosing data, right ? And then make UDEV bindings...

                 

                • 5. Re: raw devices problem
                  Dude!

                  Yes, but keep in mind that the device tree is not only rebuild after a system restart, but also the order of devices can be different, so for instance /dev/sdc1 may not be the same after a system restart if you attach or remove another device or plug in any USB. That's why UDEV should be configured to use the unique device identifier to map storage devices which require persistent device naming.

                   

                  For instance you could use the following entry to map e.g. current /dev/sdc1 to /dev/vote1

                   

                  KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="361a98001686f6959614a453133524171", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"

                   

                  Using raw devices is no longer necessary, not for Oracle, which uses the O_DIRECT flag when opening a device and it does not matter if it is a device or partition. But if you really want to map to raw device names, you could use the following:

                   

                  ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="361a98001686f6959614a453133524171", RUN+="/bin/raw /dev/raw/raw1 %N"

                  • 6. Re: raw devices problem
                    User511360-OC

                    I'd like to ask one more question here. Will it be any benefits, if we will move from raw devices to block devices, or to ASMlib ? By the way, which one is "better" ?

                    • 7. Re: raw devices problem
                      Dude!

                      From what I understand, the primary reason for using raw devices was to bypass kernel buffering, which applies to block devices and file systems. Raw device support is depreciated as of RHEL 5 and Oracle opens all datafiles using the O_DIRECT flag to bypass kernel caching. ASMLib is optional and not required for ASM to function, but depending on your requirements and technical background can simply ASM device administration.