8 Replies Latest reply: Jul 25, 2012 11:48 PM by Billy~Verreynne RSS

    OEL 6.0 takes too long to boot after attaching SAN volumes

    903538
      hello all,
      I installed OEL on a blade X6270M2, the installation went smooth without any problem, after I attached 3 volumes from a 6180 SAN storage and rebooted, the server takes too long to start up. whenever I remove the fiber cables and reboot, the server boot fast.I didnt configure multipathing yet since I couldnt find the /etc/multipath.conf evn when I try to execute "mpathconf" command,
      does anyone faced this prob before?
      should I use another driver to enable multipath?
        • 1. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
          Billy~Verreynne
          Define what part of the boot process is slow. And what does +/var/log/messages+ say when discovering SAN devices?

          Multipath is not needed (as yet) - the kernel will see the SAN devices/LUNs as normal scsi devices and create /dev entries for these. If there are multiple paths to the LUNs (dual port fibre channel wired to I/O fabric layer), then the kernel will see the same LUN as multiple scsi devices.

          This is not a problem. Multipath will be used to address that. However, Multipath will not "fix" the slow boot process/discovery of devices on the I/O fabric layer. That is a separate issue and one that should be resolved prior to using Multipath (for configuring a logical device on top of the multiple scsi devices for the same SAN LUN).
          • 2. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
            903538
            these are some outputs from the system:
            dmesg | tail -50
            end_request: I/O error, dev sdl, sector 0
            sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
            sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
            sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
            end_request: I/O error, dev sdk, sector 0
            sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
            sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
            sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
            end_request: I/O error, dev sdk, sector 0
            sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
            sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
            sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 20 00
            end_request: I/O error, dev sdk, sector 0
            sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
            sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
            sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
            end_request: I/O error, dev sdk, sector 0
            __ratelimit: 10 callbacks suppressed


            /var/log/messages:
            Jul 18 18:21:10 MIMIRDBMTNSD kernel: sd 12:0:1:3: [sdl] Sense Key : Illegal Request [current]
            Jul 18 18:21:10 MIMIRDBMTNSD kernel: sd 12:0:1:3: [sdl] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            Jul 18 18:21:10 MIMIRDBMTNSD kernel: sd 12:0:1:3: [sdl] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
            Jul 18 18:21:10 MIMIRDBMTNSD kernel: end_request: I/O error, dev sdl, sector 0
            Jul 18 18:21:11 MIMIRDBMTNSD kernel: sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
            Jul 18 18:21:11 MIMIRDBMTNSD kernel: sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
            Jul 18 18:21:11 MIMIRDBMTNSD kernel: sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
            Jul 18 18:21:11 MIMIRDBMTNSD kernel: sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
            Jul 18 18:21:11 MIMIRDBMTNSD kernel: end_request: I/O error, dev sdk, sector 0


            rpm -q device-mapper-multipath
            device-mapper-multipath-0.4.9-31.el6.x86_64


            also there is something not OK with creating partitions on disk, whenever I type "w" to write the VTOC and exit it gives an error and tell that the new partition will be available after restart!
            • 3. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
              Dude!
              I'm afraid there will be a lot more information necessary about your hardware and configuration to be able to even narrow down the problem to software or hardware.

              Are you using a XEN kernel?
              Are you booting from SAN?
              Do you need RDAC support for Multipathing?

              I guess you do need RDAC according to http://docs.oracle.com/cd/E19373-01/820-4738-13/chapsing.html. Maybe that's what's not enabled or missing.
              • 4. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
                Billy~Verreynne
                900535 wrote:
                these are some outputs from the system:
                dmesg | tail -50
                end_request: I/O error, dev sdl, sector 0
                sd 14:0:1:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
                sd 14:0:1:3: [sdk] Sense Key : Illegal Request [current]
                sd 14:0:1:3: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
                sd 14:0:1:3: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
                What I/O fabric layer do you have? Is smartctl enabled?
                also there is something not OK with creating partitions on disk, whenever I type "w" to write the VTOC and exit it gives an error and tell that the new partition will be available after restart!
                Not an issue. Expected. The kernel needs to be informed of the change. Typically kpartx does this. If multipath is used, it needs to be flushed and devices rediscovered again.
                • 5. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
                  903538
                  Are you using a XEN kernel?
                  I checked the OS and found that I am using the uek kernel and not the Xen kernel
                  Are you booting from SAN?
                  no, the OS is on the internal disks of the blade
                  Do you need RDAC support for Multipathing?
                  can you please provide me with a link to downlaod the specific RDAC for OEL6.0?
                  • 6. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
                    903538
                    What I/O fabric layer do you have? Is smartctl enabled?
                    the volumes that I am talking about are SAN volumes and smartctl doesn'y show any problem on the volumse

                    Not an issue. Expected. The kernel needs to be informed of the change. Typically kpartx does this. If multipath is used, it needs to be flushed and devices rediscovered again.
                    I tried the partprobe but didnt succeed, isn't kpartx same as partprobe?
                    • 7. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
                      Dude!
                      can you please provide me with a link to downlaod the specific RDAC for OEL6.0?
                      sorry, no idea. You might need to contact Oracle support. From what I can gather there a SAN devices that require RDAC Multipath (LSI) instead of DM Multipath to provide proper device path fail-over. I think it has something to do with devices not supporting symmetric multi-pathing, but then again, I don't really have clue. I remember reading that later versions of DM Multipath support RDAC if you have loaded an appropriate scsi RDAC kernel module. You may have to rebuild your initrd boot image.

                      What do you use now or used before when it worked?

                      I found you have a previous thread: RDAC on RHEL5.5 with 6180
                      Re: RDAC on RHEL5.5 with 6180
                      • 8. Re: OEL 6.0 takes too long to boot after attaching SAN volumes
                        Billy~Verreynne
                        900535 wrote:
                        What I/O fabric layer do you have? Is smartctl enabled?
                        the volumes that I am talking about are SAN volumes and smartctl doesn'y show any problem on the volumse
                        With fabric layer I was referring to using fibre channels, Infiniband, Ethernet and the storage protocol used (e.g. SRP, FCoE , etc). This impacts the kernel modules used for SAN access - and older version of a driver, or incorrect driver config, could cause the type of issue you are seeing.

                        For example, using an older SRP driver version on the server-side and a newer version of the storage array, will very likely run into issues.
                        Not an issue. Expected. The kernel needs to be informed of the change. Typically kpartx does this. If multipath is used, it needs to be flushed and devices rediscovered again.
                        I tried the partprobe but didnt succeed, isn't kpartx same as partprobe?
                        As I understand, partprobe notifies the kernel of a device 's partition changes. The device map then needs to be updated. That is what kpartx does.