Forum Stats

  • 3,759,874 Users
  • 2,251,609 Discussions
  • 7,870,852 Comments

Discussions

OL8 UEK and QLogic Driver Issues

user8698986
user8698986 Member Posts: 51 Blue Ribbon
edited Sep 20, 2021 5:57PM in Oracle Linux

I'm working with OL8 using the UEK kernel and the SCST package for managing my fibre channel setup. On CentOS 8, everything works fine. On OL8 with the UEK, there are messages indicative of hardware failure causing the FC ports to be shutdown. I've messaged the SCST developers for assistance and I assume they're looking into it but in digging further in, it appears it may be an issue with the QLogic drivers for the 4/8/16gb fibre channel cards and the UEK kernel. Reverting back to C8 shows that the hardware is fine. I have posted full details showing the dmesg entries in both OL8 UEK and C8 with default kernel.

Any suggestions would be greatly appreciated.

Answers

  • David Gilpin-Oracle
    David Gilpin-Oracle Principal Product Manager, Oracle Linux and Virtualization Frisco, TXPosts: 32 Employee

    Do you have an Oracle Support subscription? If so you can submit an SR on this. Let me know...

  • user8698986
    user8698986 Member Posts: 51 Blue Ribbon

    Sadly, no I don't. I'd hoped that either someone had already come across this and figured it out or OL would like to continue kickin' the crap out of RHEL. I've been quite shocked how superior the UEK has been in supporting Xen virt despite RHEL's effort to corral people into KVM. I've been able to use the UEK as both dom0 and domU with extreme ease over AL8 or RL8. No additional repos other than my own for Xen. No kernel ML or anything. I contribute where I can but this just seems to be one hump I can't get over without a nudge. Any help would be appreciated. I considered testing UEK 6 to see if the problem resolved itself or not.

  • user8698986
    user8698986 Member Posts: 51 Blue Ribbon
    edited Sep 21, 2021 7:46AM

    Just got the following reply on the SCST devel list and did some reading on this. While this SAN does not virtualize anything, if I was to at some point want to use Xen virtualization, I'd have issues with turning off VT-d. This was not an issue previously but seems to only be affected by the UEK. Surely there has to be some other way around this.

    -------------------

    "I ran into this issue after upgrading my Debian boxes from buster (Linux 4.19) to bullseye (Linux 5.10) yesterday. These boxes have been running flawlessly for a year with scst version 3.1.0. After the upgrade the initiator machine would hang on boot and the target machine showed this repeatedly:

    ERROR: qla2x00t(16): CTIO with error status 0x10 received (state 3, scst_cmd 00000000101446c1, op INQUIRY)

    In my case this issue was caused by IOMMU DMAR - DMA remapping. After turning VT-d support OFF in the BIOS of the target machine i was up and running again.

    VT-d on:

    dmesg | grep IOMMU

    [ 0.033346] DMAR: IOMMU enabled

    [ 1.339111] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel [email protected]

    [ 1.339112] AMD-Vi: AMD IOMMUv2 functionality not available on this system

    [email protected]:~$ sudo dmesg | grep DMAR

    [ 0.033346] DMAR: IOMMU enabled

    [ 0.078741] DMAR: Host address width 36

    [ 0.078743] DMAR: DRHD base: 0x000000fe710000 flags: 0x0

    [ 0.078747] DMAR: dmar0: reg_base_addr fe710000 ver 1:0 cap c9008020e30272 ecap 1000

    [ 0.078748] DMAR: DRHD base: 0x000000fe711000 flags: 0x0

    [ 0.078750] DMAR: dmar1: reg_base_addr fe711000 ver 1:0 cap c0000020230272 ecap 1000

    [ 0.078751] DMAR: DRHD base: 0x000000fe713000 flags: 0x1

    [ 0.078753] DMAR: dmar2: reg_base_addr fe713000 ver 1:0 cap c9008020630272 ecap 1000

    [ 0.078754] DMAR: RMRR base: 0x000000bb7d0000 end: 0x000000bb7e7fff

    [ 0.078755] DMAR: RMRR base: 0x000000bdc00000 end: 0x000000bfffffff

    [ 0.078756] DMAR-IR: Not enabling interrupt remapping

    [ 0.415462] pci 0000:00:00.0: DMAR: Disabling batched IOTLB flush on Ironlake

    [ 1.194621] DMAR: No ATSR found

    [ 1.194675] DMAR: dmar1: Using Register based invalidation

    [ 1.194688] DMAR: dmar2: Using Register based invalidation

    [ 1.333478] DMAR: Intel(R) Virtualization Technology for Directed I/O

    [ 1.913905] i915 0000:00:02.0: [drm] DMAR active, disabling use of stolen memory


    VT-d off:

    dmesg | grep DMAR

    [ 0.018007] ACPI: _MAR 0x00000000BB7B367B 000030 (v01 Intel OEMDMAR 00060000 LOHR 00000001)

    [ 0.416977] pci 0000:00:00.0: DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics

    "

  • User_6IRTZ
    User_6IRTZ Member Posts: 27 Employee

    Hi,

    Oracle UEK5 and UEK6 ships with the iommu feature on.

    ls /sys/devices/virtual/iommu/dmar* 
    
    iommu is enabled. 
    
    
    # grubby --update-kernel=ALL --args="iommu=off"
    

    And rebooting.

    ls /sys/devices/virtual/iommu/
    
    iommu is disabled with no dmar.
    


  • user8698986
    user8698986 Member Posts: 51 Blue Ribbon
    edited Sep 21, 2021 4:01PM

    @User_6IRTZ I just checked my CentOS 8 install (where I wiped OL8 because I couldn't make it work) where the FC DOES work to see how it's set in comparison to OL8 UEK. I have no iommu directory. Is that just a difference in the newer kernel or is there something else to look at to explain why it works in C8 and not OL8 UEK?

    # ls /sys/devices/virtual/
    bdi block cpuid dmi graphics input mem misc msr net powercap raw scst_event thermal tty usbmon vc vtconsole workqueue
    

    How would disabling the iommu affect any virtualization?

  • User_6IRTZ
    User_6IRTZ Member Posts: 27 Employee

    Hi

    The iommu feature is only needed for SRIOV device management, usually found on advanced multifunction networking cards that provide SRIOV functionality. As you noticed, the other implementations have it off.