This discussion is archived
9 Replies Latest reply: Oct 27, 2012 8:10 AM by 961426 RSS

No eth0 after install/upgrade to OVM 3.1.1

961426 Newbie
Currently Being Moderated
After install or upgrade to the Oracle VM Server 3.1.1 connected more than 8 multipathed (4 path) volumes of 2* Sun ST2540 disk array
disappears from the hypervisor interface eth0 (all 8 interfaces) (?) in 3.0.3 does not have this problem. Kernel options?

---
Regards
Adam

Edited by: AdamSki on 2012-09-12 11:24
  • 1. Re: No eth0 after install/upgrade to OVM 3.1.1
    budachst Pro
    Currently Being Moderated
    Pardon? You updated OVM server from 3.0.3 to 3.1.1 and lost your NICs? Could you elaborate more on the setup/HW of your host? Otherwise it's nearly impossible to judge what might have been going wrong.
  • 2. Re: No eth0 after install/upgrade to OVM 3.1.1
    961426 Newbie
    Currently Being Moderated
    Unfortunately, but it's possible. As I wrote (regardless whether it's an install, upgrade or update) after adding more than ~8 volumes (4 paths each) after reboot Server 3.1.1 eth0 disappears (bond0 does not work). You can reduce the number of volumes or disconnect hypervisor from the SAN to restore correct operation of eth0 (without altering the configuration).

    Hypervisor without SAN mappded volumes boot time ~2min, with all mapped volumes ~120min!!!

    I didn't have this problem with any previous version (2.2.2, 3.0.3).

    My hardware configuration:
    +4* Dell 910 (8* eth, 4* fc)+
    +2* Broadcom SAN switch (2 fabrick)+
    +2* Storage Sun ST2540+



    Listing dmesg selected items OVM 3.0.3 (working properly)
    ==========================================
    repeated section for each volume
    ...
    [ 4775.537093] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [ 4775.537100] sd 3:0:0:0: [sdc] Sense Key : Illegal Request [current]
    [ 4775.537106] sd 3:0:0:0: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    [ 4775.537114] sd 3:0:0:0: [sdc] CDB: Read(10): 28 00 10 7f fd 48 00 00 08 00
    [ 4775.537127] end_request: I/O error, dev sdc, sector 276823368
    [ 4776.053159] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [ 4776.053166] sd 3:0:0:0: [sdc] Sense Key : Illegal Request [current]
    [ 4776.053172] sd 3:0:0:0: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    [ 4776.053180] sd 3:0:0:0: [sdc] CDB: Read(10): 28 00 10 7f fc 30 00 00 08 00
    [ 4776.053193] end_request: I/O error, dev sdc, sector 276823088
    [ 4776.053201] Buffer I/O error on device sdc, logical block 34602886
    ...
    [ 4787.724387] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [ 4787.724394] sd 3:0:0:0: [sdc] Sense Key : Illegal Request [current]
    [ 4787.724400] sd 3:0:0:0: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    [ 4787.724408] sd 3:0:0:0: [sdc] CDB: Read(10): 28 00 0f ff ff d8 00 00 08 00
    [ 4787.724421] end_request: I/O error, dev sdc, sector 268435416
    [ 4787.724425] __ratelimit: 4 callbacks suppressed
    [ 4787.724429] Buffer I/O error on device sdc, logical block 33554427
    ...
    +<mooore time>+
    ...
    [ 5037.431209] end_request: I/O error, dev sdcr, sector 0
    [ 5037.959783] sd 5:0:1:16: [sddx] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [ 5037.959790] sd 5:0:1:16: [sddx] Sense Key : Illegal Request [current]
    [ 5037.959796] sd 5:0:1:16: [sddx] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    [ 5037.959805] sd 5:0:1:16: [sddx] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
    [ 5037.959817] end_request: I/O error, dev sddx, sector 0
    [ 5038.492597] sd 6:0:1:20: [sdft] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [ 5038.492604] sd 6:0:1:20: [sdft] Sense Key : Illegal Request [current]
    [ 5038.492610] sd 6:0:1:20: [sdft] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    [ 5038.492618] sd 6:0:1:20: [sdft] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
    [ 5038.492630] end_request: I/O error, dev sdft, sector 0
    ...
    ...
    ...
    [ 5038.887094] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.2.14 (Mar 30, 2011)
    [ 5038.887349] bnx2 0000:02:00.0: eth0: Added CNIC device
    [ 5038.887589] bnx2 0000:02:00.1: eth1: Added CNIC device
    [ 5038.887828] bnx2 0000:03:00.0: eth2: Added CNIC device
    [ 5038.888066] bnx2 0000:03:00.1: eth3: Added CNIC device
    [ 5038.888305] bnx2 0000:06:00.0: eth4: Added CNIC device
    [ 5038.888543] bnx2 0000:06:00.1: eth5: Added CNIC device
    [ 5038.888781] bnx2 0000:07:00.0: eth6: Added CNIC device
    [ 5038.889019] bnx2 0000:07:00.1: eth7: Added CNIC device
    ...
    [ 5040.059214] device-mapper: multipath: version 1.1.0 loaded
    [ 5040.075112] sd 3:0:0:0: rdac: LUN 0 (unowned)
    ...
    [ 5040.095794] sd 3:0:0:21: rdac: LUN 21 (owned)
    [ 5040.096720] sd 3:0:1:0: rdac: LUN 0 (unowned)
    ...
    [ 5040.116430] sd 3:0:1:21: rdac: LUN 21 (owned)
    [ 5040.117743] sd 4:0:0:0: rdac: LUN 0 (unowned)
    ...
    [ 5040.136767] sd 4:0:0:21: rdac: LUN 21 (owned)
    [ 5040.137707] sd 4:0:1:0: rdac: LUN 0 (unowned)
    ...
    [ 5040.157887] sd 4:0:1:21: rdac: LUN 21 (owned)
    [ 5040.159901] sd 5:0:0:0: rdac: LUN 0 (owned)
    ...
    [ 5040.187882] sd 5:0:0:21: rdac: LUN 21 (unowned)
    [ 5040.189109] sd 5:0:1:0: rdac: LUN 0 (owned)
    ...
    [ 5040.210051] sd 5:0:1:21: rdac: LUN 21 (unowned)
    [ 5040.210940] sd 6:0:0:0: rdac: LUN 0 (owned)
    ...
    [ 5040.234246] sd 6:0:0:21: rdac: LUN 21 (unowned)
    [ 5040.235543] sd 6:0:1:0: rdac: LUN 0 (owned)
    ...
    [ 5040.256287] sd 6:0:1:21: rdac: LUN 21 (unowned)
    [ 5040.256297] rdac: device handler registered
    [ 5040.262139] device-mapper: multipath round-robin: version 1.0.0 loaded
    [ 5041.092959] Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
    [ 5041.092962] bonding: MII link monitoring set to 250 ms
    [ 5041.143343] bonding: bond0: setting mode to active-backup (1).
    [ 5041.143508] bonding: bond0: Setting MII monitoring interval to 250.
    [ 5041.143511] bonding: bond0: Note: Updating updelay (to 500) since it is a multiple of the miimon value.
    [ 5041.143513] bonding: bond0: Note: Updating downdelay (to 500) since it is a multiple of the miimon value.
    [ 5041.143630] bonding: bond0: Setting use_carrier to 1.
    [ 5041.143743] bonding: bond0: Setting up delay to 500.
    [ 5041.143856] bonding: bond0: Setting down delay to 500.
    [ 5041.143967] bonding: bond0: Unable to set eth0 as primary slave as it is not a slave.
    [ 5041.352854] bonding: bond0: Adding slave eth0.
    ...
    [ 5044.682544] bonding: bond0: first active interface up!
    ...



    Listing dmesg selected items OVM 3.1.1 (does not work the first network interface)
    ===========================================================
    repeated section for each volume
    ...
    sd 3:0:0:0 [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 3:0:0:0 [sdc] Sense Key : Illegal Request [current]
    sd 3:0:0:0 [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 3:0:0:0 [sdc] CDB: Read(10): 28 00 7f ff ff f8 00 00 08 00
    end_request: I/O error, dev sdau, sector 2147483640
    Buffer I/O error on device sdau, logical block 268435455
    sd 3:0:0:0 [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 3:0:0:0 [sdc] Sense Key : Illegal Request [current]
    sd 3:0:0:0 [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 3:0:0:0 [sdc] CDB: Read(10): 28 00 7f ff ff f8 00 00 08 00
    end_request: I/O error, dev sdau, sector 2147483640
    Buffer I/O error on device sdau, logical block 268435455
    ...
    sd 3:0:0:0 [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 3:0:0:0 [sdc] Sense Key : Illegal Request [current]
    sd 3:0:0:0 [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 3:0:0:0 [sdc] CDB: Read(10): 28 00 7f ff ff f0 00 00 08 00
    blk_update_request: 10 callbacks suppressed
    end_request: I/O error, dev sdau, sector 2147483632
    Buffer I/O error on device sdau, logical block 268435454
    ...
    +<mooore time>+
    ...
    sd 6:0:1:21: [sdfu] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 6:0:1:21: [sdfu] Sense Key : Illegal Request [current]
    sd 6:0:1:21: [sdfu] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 6:0:1:21: [sdfu] CDB: Read(10): 28 00 3c ff ff 80 00 00 08 00
    sd 6:0:1:21: [sdfu] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 6:0:1:21: [sdfu] Sense Key : Illegal Request [current]
    sd 6:0:1:21: [sdfu] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 6:0:1:21: [sdfu] CDB: Read(10): 28 00 3c ff ff 80 00 00 08 00
    ...
    sd 6:0:1:21: [sdfu] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 6:0:1:21: [sdfu] Sense Key : Illegal Request [current]
    sd 6:0:1:21: [sdfu] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
    sd 6:0:1:21: [sdfu] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
    blk_update_request: 40 callbacks suppressed
    end_request: I/O error, dev sdfu, sector 0
    Buffer I/O error on device sdfu, logical block 0
    ...
    Buffer I/O error on device sdfu, logical block 8
    ...
    ...
    ...
    cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.7 (July 20, 2011)
    bnx2 0000:02:00.1: eth1: Added CNIC device
    bnx2 0000:03:00.0: eth2: Added CNIC device
    bnx2 0000:03:00.1: eth3: Added CNIC device
    ...
    bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
    bonding: MII link monitoring set to 250 ms
    bonding: bond0: setting mode to active-backup (1).
    bonding: bond0: Setting MII monitoring interval to 250.
    bonding: bond0: Note: Updating updelay (to 500) since it is a multiple of the miimon value.
    bonding: bond0: Note: Updating downdelay (to 500) since it is a multiple of the miimon value.
    bonding: bond0: Setting use_carrier to 1.
    bonding: bond0: Setting up delay to 500.
    bonding: bond0: Setting down delay to 500.
    bonding: bond0: Unable to set eth0 as primary slave.
    device-mapper: multipath: version 1.3.0 loaded
    sd 3:0:0:0: rdac: LUN 0 (RDAC) (unowned)
    ...
    sd 3:0:0:21: rdac: LUN 21 (RDAC) (owned)
    sd 3:0:1:0: rdac: LUN 0 (RDAC) (unowned)
    ...
    sd 3:0:1:21: rdac: LUN 21 (RDAC) (owned)
    sd 4:0:0:0: rdac: LUN 0 (RDAC) (unowned)
    ...
    sd 4:0:0:21: rdac: LUN 21 (RDAC) (owned)
    sd 4:0:1:0: rdac: LUN 0 (RDAC) (unowned)
    ...
    sd 4:0:1:21: rdac: LUN 21 (RDAC) (owned)
    sd 5:0:0:0: rdac: LUN 0 (RDAC) (owned)
    ...
    sd 5:0:0:21: rdac: LUN 21 (RDAC) (unowned)
    sd 5:0:1:0: rdac: LUN 0 (RDAC) (owned)
    ...
    sd 5:0:1:21: rdac: LUN 21 (RDAC) (unowned)
    sd 6:0:0:0: rdac: LUN 0 (RDAC) (owned)
    ...
    sd 6:0:0:21: rdac: LUN 21 (RDAC) (unowned)
    sd 6:0:1:0: rdac: LUN 0 (RDAC) (owned)
    ...
    sd 6:0:1:21: rdac: LUN 21 (RDAC) (unowned)
    rdac: device handler registered
    ...



    I think there are differences in kernel compilation options between OVS versions (?)+



    ---
    Regards
    Adam

    Edited by: AdamSki on 2012-09-13 07:10
  • 3. Re: No eth0 after install/upgrade to OVM 3.1.1
    user12273962 Pro
    Currently Being Moderated
    Sounds like a bug for sure. Probably want to open a ticket. Make sure you have your FC zoning right.

    I had something wierd like this happen on a OEL/Redhat server one time and it was due the fact that the Ethernet connections where actually renumbering themselves after configuring bonding interfaces. Just a thought.
  • 4. Re: No eth0 after install/upgrade to OVM 3.1.1
    ratdude Newbie
    Currently Being Moderated
    Maybe the VNIC got modified somehow or the Hardware Address of the NIC in the ifcfg-eth0 file doesn't now match what is defined for it in VM Manager?
  • 5. Re: No eth0 after install/upgrade to OVM 3.1.1
    Kirill.Boyko Newbie
    Currently Being Moderated
    seems we have similar problems:

    SR 3-5508905681 : bnx2 driver fail to load when disks are attached through lpfc driver

    we experience a problem with bnx2 when apron. more than 4 volumes are mounted on boot time. Unfortunately no response since may on this SR. It is really pity since I was expecting OVS 3.1.1 new kernel would fix it
  • 6. Re: No eth0 after install/upgrade to OVM 3.1.1
    961426 Newbie
    Currently Being Moderated
    Verified that both fabricks are configured correctly.
    There was no renumbering of ethernet ports, only the first port in the running configuration is missing, even though configuration files are correct (with the right MAC address) for the port in ifcfg-eth0.

    Network configuration parameters for the VM Server and VM Manager are definitely properly set (in OVS+OVM-3.0.3 and OVS-3.0.3+OVM-3.1.1 works flawlessly)

    Kirill, I think you're right. my problem is almost the same as observed by you.
    I have the impression that either the problem lies in the newer modules for Broadcom (bnx2i) or Emulex (lpfc) or in the kernel compile options in version 3.1.1.

    ---
    Regards
    Adam
  • 7. Re: No eth0 after install/upgrade to OVM 3.1.1
    Kirill.Boyko Newbie
    Currently Being Moderated
    Hi Adam!

    We use IBM V7000 (svc) to virtualize storage. One possible workaround in our environment for such issue is to map only few volumes on boot time and after attach them in hot mode. It works for us, as we have not a lot of reboots in production, but we definitely expect Oracle to fix this. Just a few days ago I received a response on this SR, so I will update this thread as I will progress


    Regards,
    Kirill

    PS Actually I am really frustrated :) as one of my target for small upgrade project from 3.0.3 to 3.1.1 was to eliminate this problem. I was thinking that it would be fixed.
  • 8. Re: No eth0 after install/upgrade to OVM 3.1.1
    user157995 Explorer
    Currently Being Moderated
    Just FYI we have a OVM cluster using bnx2 and ixgbe Nics, with emulex bfa fiber channel cards, with over 9 FC luns mounted and we don't experience any of these issues.
  • 9. Re: No eth0 after install/upgrade to OVM 3.1.1
    961426 Newbie
    Currently Being Moderated
    Hi Kirill,

    I tested version 3.2.1, works correctly!!!


    ---
    Regards,
    Adam

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points