This discussion is archived
1 2 Previous Next 26 Replies Latest reply: Oct 25, 2012 7:22 AM by user386688 RSS

OMV 3.1.1 10GB nic issues

user157995 Explorer
Currently Being Moderated
Has anyone encountered an issue with OVM nodes issuing tons of pause frames to the switch? We have had instabilities due to this, where if the server issues enough pause frames, the switch will actually shut down the port (Cisco Nexus).

I have a SR1 open to see if Oracle has seen this.

Our setup is 4 HP DL585's 192GB ram each, dual port Intel X520 DA2, one port is for management/heartbeat/livemig and the other is purlely for VM traffic.
  • 1. Re: OMV 3.1.1 10GB nic issues
    user12273962 Pro
    Currently Being Moderated
    I'm running 10Gb on the nexus products. No problems... but I'm not using intel cards. That's probably the issue.
  • 2. Re: OMV 3.1.1 10GB nic issues
    user157995 Explorer
    Currently Being Moderated
    user,

    how large is your OVM environment, and how do you have the 10GB networking configured? Anything special on the ports config?
  • 3. Re: OMV 3.1.1 10GB nic issues
    user12273962 Pro
    Currently Being Moderated
    Not very large in the grand scheme of things.

    I do know there is a bug with running jumbo frames on 10GB with frames over 4000 MTU. That's the reason I've never tried anything other than default on the 10GB. I do have 2 LACP bonds using 10GB for my RAC database host nodes.
  • 4. Re: OMV 3.1.1 10GB nic issues
    user157995 Explorer
    Currently Being Moderated
    Can you do a show int on your OVM interfaces so I could compare settings? (for example, flow control, etc).
  • 5. Re: OMV 3.1.1 10GB nic issues
    user12273962 Pro
    Currently Being Moderated
    Here is the port channel

    interface port-channel.....
    description bond for
    switchport mode trunk
    switchport trunk native vlan #
    switchport trunk allowed vlan #
    spanning-tree port type edge trunk

    Interface settings for each ethernet interface

    description ..........
    switchport mode trunk
    switchport trunk native vlan #
    switchport trunk allowed vlan #
    spanning-tree port type edge trunk
    channel-group # mode active
  • 6. Re: OMV 3.1.1 10GB nic issues
    user12273962 Pro
    Currently Being Moderated
    The forum will not let me post the output of the int command. It says content not allowed.

    I do not show flow control active on the links and I believe by default flow control isn't enabled unless specified. I did post the interface setup and the port channel setup. This is for 2 member bond each being 10GB.
  • 7. Re: OMV 3.1.1 10GB nic issues
    Andrew Simmonds Newbie
    Currently Being Moderated
    Have you got the "Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection" nic?

    Setting this fixed it for all our servers:
    ethtool -C ethX rx-usecs 0
    replace X with the proper number for your card.

    I think Oracle might be working on a fix at the driver level if this workaround fixes your issue.
  • 8. Re: OMV 3.1.1 10GB nic issues
    user157995 Explorer
    Currently Being Moderated
    Yes we have that exact card. What issue were you having that resulted in finding that work around?

    Thanks
    Dave
  • 9. Re: OMV 3.1.1 10GB nic issues
    Andrew Simmonds Newbie
    Currently Being Moderated
    We were experiencing extremely slow transfer speeds (100-200k/s) rather than 4gb/s+
    Issue only appeared when upgrading from 3.0.3 to 3.1.1
    Oracle support suggested that particular workaround although nothing permanent at this stage.
  • 10. Re: OMV 3.1.1 10GB nic issues
    user386688 Newbie
    Currently Being Moderated
    Hi,

    We have a variety of OVM versions running on Cisco Nexus 5596 with the Intel cards, including 2.2.1, 2.2.2 and 3.1.1. We're currently waiting for a backport on the ixgbe drivers for the Dom0 kernel, but have fixed a number of issues that we have seen. We are running vlans over lacp bonded links.

    Make sure LRO is off for the IXGBE, either as ethtool_opts in ifcfg files, or in modprobe.conf. Throughput is adeqaute, but not stellar. We are not running jumbo frames, but tbh you should be able to achieve a pretty good throughput even without and latency improves for smaller IOs without jumbo packets...

    We saw a decrease in performance when setting rx_usecs to 0 rather than 1, and it will cause a big increase in interrupt activity.

    Hope that helps

    Regards

    Julian
  • 11. Re: OMV 3.1.1 10GB nic issues
    user157995 Explorer
    Currently Being Moderated
    Julian,

    In our 3.1.1 environment LRO is already disabled by default. I just checked the ifcfg files and they have ETHTOOL_OFFLOAD_OPTS="lro off" already there (Oracle must have made that happen, I didnt)
  • 12. Re: OMV 3.1.1 10GB nic issues
    user386688 Newbie
    Currently Being Moderated
    We're achieving about 5-6Gb/sec on our OVM servers on 3.1.1 on Nexus. I can post up any port config for nexus, interface setup and tuning we have done if that would help?
  • 13. Re: OMV 3.1.1 10GB nic issues
    user157995 Explorer
    Currently Being Moderated
    Sure that would be helpful to see, as it sounds like we have similar environments...

    Have you had any issues such as nodes fencing themselves, or kernel panicing? What type of hardware too, for arguments sake.

    Thanks for your time and effort
  • 14. Re: OMV 3.1.1 10GB nic issues
    user386688 Newbie
    Currently Being Moderated
    No worries. I see we have turned off gso and lro.

    nexus1:
    vpc domain 1
    role priority 1000
    system-priority 2000
    peer-keepalive destination 10.200.3.2
    auto-recovery

    interface Vlan1

    iinterface port-channel1
    switchport mode trunk
    spanning-tree port type network
    vpc peer-link

    interface port-channel11
    description prodovmsvr01
    switchport mode trunk
    switchport trunk allowed vlan 701-702,723,799
    speed 10000
    duplex full
    vpc 11

    interface Ethernet1/41
    description prodovmsvr01, slot2 left nic
    lacp rate fast
    switchport mode trunk
    switchport trunk allowed vlan 701-702,723,799
    spanning-tree port type edge trunk
    spanning-tree bpduguard enable
    duplex full
    channel-group 11 mode active

    nexus2:
    vpc domain 1
    role priority 4096
    system-priority 2000
    peer-keepalive destination 10.200.3.1
    auto-recovery

    interface Vlan1

    interface port-channel1
    switchport mode trunk
    spanning-tree port type network
    vpc peer-link

    interface port-channel11
    description prodovmsvr01
    switchport mode trunk
    switchport trunk allowed vlan 701-702,723,799
    speed 10000
    duplex full
    vpc 11

    interface Ethernet1/41
    description prodovmsvr01, slot4 left nic
    lacp rate fast
    switchport mode trunk
    switchport trunk allowed vlan 701-702,723,799
    spanning-tree port type edge trunk
    spanning-tree bpduguard enable
    duplex full
    channel-group 11 mode active

    modprobe.conf:
    # Bonding devices used by OVM's networking
    alias bond0 bonding
    options bond0 mode=4 miimon=250 use_carrier=1 updelay=500 downdelay=500

    ifcfg-eth4:
    # Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
    DEVICE=eth4
    BOOTPROTO=none
    HWADDR=90:E2:BA:11:D9:6C
    ONBOOT=yes
    MASTER=bond0
    SLAVE=yes
    ETHTOOL_OFFLOAD_OPTS="lro off gso off"

    ifcfg-eth6:
    #This file was dynamically created by OVM manager. Please Do not edit (oops)
    DEVICE=eth6
    BOOTPROTO=none
    HWADDR=90:E2:BA:11:DB:50
    ONBOOT=yes
    MASTER=bond0
    SLAVE=yes
    ETHTOOL_OFFLOAD_OPTS="lro off gso off"

    ifcfg-bond0:
    DEVICE=bond0
    BOOTPROTO=none
    ONBOOT=yes
    BONDING_OPTS="mode=4 miimon=250 use_carrier=1 updelay=500 downdelay=500 primary=eth4"

    ifcfg-bond0.701:
    #This file was dynamically created by OVM manager. Please Do not edit
    DEVICE=bond0.701
    HWADDR=90:E2:BA:11:D9:6C
    BOOTPROTO=none
    ONBOOT=yes
    VLAN=yes
    ETHTOOL_OFFLOAD_OPTS="lro off" # Would have thought this unnecessary tbh :)
    BRIDGE=0004fb00103790b
    NM_CONTROLLED=no
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points