9 Replies Latest reply: Feb 9, 2011 1:03 PM by Arpman RSS

    OEL5, Multipath, udev issue

    Arpman
      Greetings,

      I have installed OEL5 (2.6.18-194.el5) that requires access to a SAN system.
      I have 2 HBA's installed and configured multipath for the system to use.

      The problem is that I need the mount points to be completed via the fstab for the our oracle application to use the SAN for storage - this does not happen. I expected udev to create the device nodes for the system to mount during startup - no luck - udev never creates the nodes. I can use udevtrigger and udevsettle to complete the action, but that does not fix my problem.

      I have used udevtest on /block/dm-0 and get a good response.

      We have:
      1. edited multipath.conf for the WWID's of the LUN's
      2. rebuilt the initrd image to include dm_multipath module
      3. followed the advice of the 2007 white paper "configuring udev and device mapper for Oracle RAC 10g Release 2 on ReadHat RHEL4 and Oracle EL4". We are not useing clustering or any other technolgies (just OS, Oracle 10g database and SAN for data storage).

      It seems like udev rule (40-multipath.rules) is not being processed for this action (notable: kpartx does not register the partition however I can complete the action manually)

      What is the correct procedure to mount multipath SAN devices during bootup?

      Any troubleshooting advice?
        • 1. Re: OEL5, Multipath, udev issue
          Tommyreynolds-Oracle
          Did you do the original install using:
          boot> linux mpath
          With OEL5, installing with the root file system on SAN should work out-of-the-box.
          • 2. Re: OEL5, Multipath, udev issue
            Arpman
            Tommy - no I did not install OEL as you describe.

            I installed on a machine with a raid 1 system (for the OS and applications) and installed teh HBA for SAN access at a later date, as the SAN will store the application data information. I am not trying to boot from SAN. Booting is accomplished from a local HD. Teh applicatin will not start if the data is not accessable during the boot up process.

            Can I use mpath as a kernel argument on bootup? Would that be helpful?

            Edited by: Arpman on Feb 2, 2011 2:48 PM
            • 3. Re: OEL5, Multipath, udev issue
              Avi Miller-Oracle
              Arpman wrote:
              What is the correct procedure to mount multipath SAN devices during bootup?
              Is multipathd set to start automatically on boot? Run *chkconfig --list multipathd* and make sure it's On. If not, multipath will not trigger udev to create the devices properly.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
              • 4. Re: OEL5, Multipath, udev issue
                Arpman
                Mr Miller,

                Thanks for the reply, and yes multipathd is on. It was one of the things I checked before I got frustrated and requested assitance from the community. Everything I know about the process tells me it should work, but it does not - I am overlooking something or just plain ignorant - hopefully a community memeber will set me straight. Thanks again for your input.

                Arpman
                • 5. Re: OEL5, Multipath, udev issue
                  Avi Miller-Oracle
                  Arpman wrote:
                  Thanks for the reply, and yes multipathd is on. It was one of the things I checked before I got frustrated and requested assitance from the community. Everything I know about the process tells me it should work, but it does not - I am overlooking something or just plain ignorant - hopefully a community memeber will set me straight. Thanks again for your input.
                  Multipath has always just worked out of the box for me on Oracle Linux 5, so I'm not sure what advice to give you. If you have Oracle Unbreakable Linux support, I suggest you open an SR.
                  • 6. Re: OEL5, Multipath, udev issue
                    Tommyreynolds-Oracle
                    I'm missing something: where is your root file system? A local disk? SAN?

                    Does the system boot OK and the problem is you can't get the multipath devices working at all?

                    If your root file system is supposed to be on the SAN:
                    <ol>
                    <li>Please show the command you used to rebuild the initrd?
                    <li>Run these commands to reverse engineer your rebuilt initrd:
                    # mkdir junk
                    # cd junk
                    # zcat < /boot/your-initrd-file | cpio -idv
                    <li>Check that you have the right SAN drivers in the initrd.
                    <li>Also, make sure your multipath.conf file from the initrd is correct.
                    <li>Let us know what you find interesting.
                    </ol>

                    If your system boots fine, but you can't see the multipath LUNs after boot:
                    <ol type="A">
                    <li>Show us your /etc/multipath.conf file.
                    <li>What is the output of:
                    # /sbin/blkid
                    # /sbin/multipath -ll
                    # /sbin/fdisk -l
                    <li>Can you provide us a link to the contents of your /var/log/dmesg file? It's
                    probably too big for an upload and certainly too big to spam the list.
                    </ol>

                    This will at least give us something to work with.
                    • 7. Re: OEL5, Multipath, udev issue
                      Arpman
                      Tommy,

                      Thanks for your assitance. To answer your quesstion: we boot froma local hard drive. Boot up is not the issue.

                      However, my problem is that the device nodes for the multipath SAN devices do not get created, moreover effecting the mount operation of said devices.

                      But - based on your advice I did a little troubleshooting and found out that I was missing my multipath.conf file in my ramdisk image.

                      To correct this problem I had to edit my /sbin/mkinitrd file. I located the use_multipath and use_emc variables and set them to "1".

                      My new ramdisk seems to operate correctly. I tested it several times - with only one hicup where the system failed to created 2 of the 4 device nodes required. Would this be a udev issue? Another reboot seemed to correct the problem.

                      Final solution

                      1. edit /etc/multipath.conf
                      - added WWID
                      - added alias

                      2. edit /sbin/mkinitrd
                      - changed use_multipath=1
                      - changed use_emc=1

                      3. created /etc/sysconfig/mkinitrd/multipath (not sure of purpose)
                      - added MULTIPATH=yes


                      Thanks again,

                      I learned alot and appreciate your assistance!!!
                      • 8. Re: OEL5, Multipath, udev issue
                        Tommyreynolds-Oracle
                        That is great news, on many levels.

                        Do you know about the /sbin/mkinitrd utility? This command-line tool allows you to include additional kernel modules and subsystems into the initrd; by default only drivers sufficient to access the root file system are included. Wrapping the mkinitrd utility with a small shell script can automate the process.

                        If some of your LUNs are not discovered until the next reboot, I suspect a timing problem where the kernel checks for them prior to their discovery. Try adding _netdev* to the mount options for any /etc/fstab entry referencing these LUNs even indirectly via their /dev/mapper names.  Maybe that will allow reproducible reboot behavior.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                        • 9. Re: OEL5, Multipath, udev issue
                          Arpman
                          Tommy,

                          I finally found a really good solution to my problem.

                          Based on an Oracle Community post entiltled "The SAN storage can not be identified during system booting" - I preloaded the scsi modules and hba driver, however I still have errors in my dmesg log, but thats another thread.

                          Thanks again,

                          Arpman

                          Edited by: Arpman on Feb 9, 2011 11:03 AM