2 Replies Latest reply: Nov 7, 2011 5:57 PM by 394149 RSS

    Oracle VM 3.0 Bonding

    394149
      We are implementing Oracle VM 3.0 using HP Proliant BL460c G6 with 24 processors and 48 GB RAM.

      Using 2 NICs, we have configured an Active-Backup failover (mode 1) bonding. That network was used for Management, Live Migration and Virtual Machines. When tests are performed, failing the main NIC, the physical machines (Dom 0) pass it, but the virtual machines doesn't recover from failure. We noticed that when the VM request a network action (from console) to a external terminal (ping), the network is recovered from outside. Are there any special configuration needed to support bonding on virtual machines?

      This is the configuration and the output from some commands we have:

      [root@myserver ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
      # Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
      DEVICE=eth0
      BOOTPROTO=none
      HWADDR=xx:xx:xx:xx:xx:xx
      ONBOOT=yes
      MASTER=bond0
      SLAVE=yes


      [root@myserver ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
      # Broadcom Corpoation NetXtreme II BCM57711E 10-Gigabit PCIe
      DEVICE=eth1
      BOOTPROTO=none
      HWADDR=xx:xx:xx:xx:xx:yy
      ONBOOT=yes
      MASTER=bond0
      SLAVE=yes


      [root@myserver ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
      #This file was dynamically created by OVM manager. Please Do not edit
      DEVICE=bond0
      BOOTPROTO=none
      ONBOOT=yes
      BRIDGE=192.168.247.0
      NM_CONTROLLED=no
      BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=200 downdelay=200 primary=eth0"


      [root@myserver ~]# cat /etc/sysconfig/network-scripts/ifcfg-192.168.247.0
      #This file was dynamically created by OVM manager. Please Do not edit
      DEVICE=192.168.247.0
      TYPE=Bridge
      IPADDR=192.168.247.40
      NETMASK=255.255.255.0
      BOOTPROTO=static
      ONBOOT=yes
      DELAY=0


      [root@myserver ~]# cat /etc/modprobe.conf | grep bond
      alias bond0 bonding
      options bond0 mode=1 miimon=100 use_carrier=1 updelay=200 downdelay=200
      alias bond1 bonding
      alias bond2 bonding
      alias bond3 bonding
      alias bond4 bonding
      alias bond5 bonding


      [root@myserver ~]# brctl show
      bridge name bridge id STP enabled interfaces
      192.168.247.0 8000.78e3b50117b0 no bond0

      vif3.0


      [root@myserver ~]# cat /proc/net/bonding/bond0
      Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

      Bonding Mode: fault-tolerance (active-backup)
      Primary Slave: eth0 (primary_reselect always)
      Currently Active Slave: eth0
      MII Status: up
      MII Polling Interval (ms): 100
      Up Delay (ms): 200
      Down Delay (ms): 200

      Slave Interface: eth1
      MII Status: up
      Link Failure Count: 1
      Permanent HW addr: 78:e3:b5:01:17:b4
      Slave queue ID: 0

      Slave Interface: eth0
      MII Status: up
      Link Failure Count: 0
      Permanent HW addr: 78:e3:b5:01:17:b0
      Slave queue ID: 0


      [root@myserver scripts]# /etc/xen/scripts/ovs-bond-if show bondm=bond0
      ============================================================
      Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

      Bonding Mode: fault-tolerance (active-backup)
      Primary Slave: eth0 (primary_reselect always)
      Currently Active Slave: eth0
      MII Status: up
      MII Polling Interval (ms): 100
      Up Delay (ms): 200
      Down Delay (ms): 200

      Slave Interface: eth1
      MII Status: up
      Link Failure Count: 1
      Permanent HW addr: 78:e3:b5:01:17:b4
      Slave queue ID: 0


      Slave Interface: eth0
      MII Status: up
      Link Failure Count: 0
      Permanent HW addr: 78:e3:b5:01:17:b0
      Slave queue ID: 0

      ============================================================


      root@myserver scripts]# ./ovs-local-bridge status bridge=192.168.247.0
      ============================================================
      5: 192.168.247.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      link/ether 78:e3:b5:01:17:b0 brd ff:ff:ff:ff:ff:ff
      inet 192.168.247.40/24 brd 192.168.247.255 scope global 192.168.247.0
      inet 192.168.247.41/24 brd 192.168.247.255 scope global secondary 192.168.247.0:0

      bridge name bridge id STP enabled interfaces
      192.168.247.0 8000.78e3b50117b0 no bond0
      vif3.0


      192.168.247.0/24 dev 192.168.247.0 proto kernel scope link src 192.168.247.40
      169.254.0.0/16 dev 192.168.247.0 scope link
      default via 192.168.247.1 dev 192.168.247.0

      Kernel IP routing table
      Destination Gateway Genmask Flags Metric Ref Use Iface
      192.168.247.0 0.0.0.0 255.255.255.0 U 0 0 0 192.168.247.0
      169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 192.168.247.0
      0.0.0.0 192.168.247.1 0.0.0.0 UG 0 0 0 192.168.247.0
      ============================================================


      [root@myserver scripts]# ./ovs-network-bridge status bridge=192.168.247.0
      ============================================================
      4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 78:e3:b5:01:17:b0 brd ff:ff:ff:ff:ff:ff
      5: 192.168.247.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      link/ether 78:e3:b5:01:17:b0 brd ff:ff:ff:ff:ff:ff
      inet 192.168.247.40/24 brd 192.168.247.255 scope global 192.168.247.0
      inet 192.168.247.41/24 brd 192.168.247.255 scope global secondary 192.168.247.0:0

      bridge name bridge id STP enabled interfaces
      192.168.247.0 8000.78e3b50117b0 no bond0
      vif3.0

      192.168.247.0/24 dev 192.168.247.0 proto kernel scope link src 192.168.247.40
      169.254.0.0/16 dev 192.168.247.0 scope link
      default via 192.168.247.1 dev 192.168.247.0

      Kernel IP routing table
      Destination Gateway Genmask Flags Metric Ref Use Iface
      192.168.247.0 0.0.0.0 255.255.255.0 U 0 0 0 192.168.247.0
      169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 192.168.247.0
      0.0.0.0 192.168.247.1 0.0.0.0 UG 0 0 0 192.168.247.0
      ============================================================
        • 1. Re: Oracle VM 3.0 Bonding
          Avi Miller-Oracle
          ja1979 wrote:
          but the virtual machines doesn't recover from failure. We noticed that when the VM request a network action (from console) to a external terminal (ping), the network is recovered from outside.
          Could you be a little bit clearer on this? Is what your'e saying that the guest VM doesn't recover until it makes an external network connection? Because traffic on the bridge should be unaffected, as far as I know.
          • 2. Re: Oracle VM 3.0 Bonding
            394149
            Hi Avi...

            The switch to the backup NIC works at the Dom 0 level, if you make a ping to the host machine the connection is still working. However, the communication with the virtual machines are broken and 3-5 minutes later it is recovered. We noticed that when a network request is made from the virtual machine (like ping command) the connection is recovered too.