Linux did not initially support the concept of raw devices and the way it was done under RHEL 3 and 4 was not really an efficient solution. The use of raw devices as of RHEL 5 has been depreciated. The device tree under the 2.6 Linux kernel is rebuild at every system restart and you will loose your /dev configuration. You have to use ASMLib or configure UDEV for device name persistence. Perhaps you were following setup instructions for RHEL 4 and earlier. Oracle under Linux opens datafiles with the O_DIRECT flag to bypass kernel buffering, which makes the use of raw devices unnecessary.
I think this is pure OS bug, because /dev/rawctl is kind of system device.
Raw devices works without problem on RHEL5. For example we have production 2-node RAC. Now we must add more storage to it, and I made this model to test this behavior, beacuse I know that we could loose our /dev configuration. Final purpose was to add disks to UDEV and after this change the raw device bindings, so /dev/raw/raw1 will be binding to /dev/votedisk1 (udev binding) etc... Or is there another way to do the same ?
You will need to obtain the unique device identifies (/sbin/scsi_id -g -s /block/sdb/sdb1) of your cluster devices and use this info to map it in the UDEV configuration on each node. Since I do not know details of your configuration and what what you have configured I cannot say what is wrong or what else you can do.
If you have access to My Oracle support I suggest the following:
Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 (Doc ID 465001.1)
Linux 2.6 Kernel Deprecation of Raw Devices (Doc ID 357492.1)
Yes, but keep in mind that the device tree is not only rebuild after a system restart, but also the order of devices can be different, so for instance /dev/sdc1 may not be the same after a system restart if you attach or remove another device or plug in any USB. That's why UDEV should be configured to use the unique device identifier to map storage devices which require persistent device naming.
For instance you could use the following entry to map e.g. current /dev/sdc1 to /dev/vote1
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="361a98001686f6959614a453133524171", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"
Using raw devices is no longer necessary, not for Oracle, which uses the O_DIRECT flag when opening a device and it does not matter if it is a device or partition. But if you really want to map to raw device names, you could use the following:
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="361a98001686f6959614a453133524171", RUN+="/bin/raw /dev/raw/raw1 %N"
From what I understand, the primary reason for using raw devices was to bypass kernel buffering, which applies to block devices and file systems. Raw device support is depreciated as of RHEL 5 and Oracle opens all datafiles using the O_DIRECT flag to bypass kernel caching. ASMLib is optional and not required for ASM to function, but depending on your requirements and technical background can simply ASM device administration.