1 2 Previous Next 20 Replies Latest reply: Oct 15, 2013 2:20 PM by Catch_22 RSS

    Bonding configuration

    user7193548


      Hello,

      I have configured Bonding on OEL 6.4 and everything is working fine.

      I wanted to check the active interface under /proc/net/bonding/bond0 and the file does not exist.

      Directory /proc/net/bonding does not exist.

      I have no idea why ?

      Reboot the server give the same...

       

      My configuration is as follow:

      /etc/modprobe.d/bonding.conf:

      alias bond0 bonding

       

      ifcfg-bond0:

      DEVICE=bond0

      ONBOOT=yes

      BOOTPROTO=none

      IPADDR=172.27.2.21

      NETMASK=255.255.255.0

      GATEWAY=172.27.2.1

      NETWORK=172.27.2.0

      BROADCAST=172.27.2.255

      NM_CONTROLLED=no

      USERCTL=no

      BONDING_OPTS="mode=active-backup miimon=100"

      TYPE=Ethernet

      IPV6INIT=no

       

      ifcfg-eth3:

      DEVICE=eth3

      BOOTPROTO=none

      ONBOOT=yes

      USERCTL=no

      NM_CONTROLLED=no

      MASTER=bond0

      SLAVE=yes

      TYPE=Ethernet

      IPV6INIT=no

       

      ifcfg-eth5:

      DEVICE=eth5

      BOOTPROTO=none

      ONBOOT=yes

      USERCTL=no

      NM_CONTROLLED=no

      MASTER=bond0

      SLAVE=yes

      TYPE=Ethernet

      IPV6INIT=no

       

      Regards

        • 1. Re: Bonding configuration
          Catch_22

          You may need to restart the network:

           

          # service network restart

           

          Check the following link for an example:

           

          GUIDE: Oracle VM VirtualBox and Oracle Linux NIC bonding

          • 2. Re: Bonding configuration
            user7193548

            I have restarted the network and it is the same. (Even after a reboot of the server).

            dzi57

            • 3. Re: Bonding configuration
              Catch_22

              Is the bonding working? What your the output of the following:

               

              # ifconfig

              # modinfo bonding | head

              # uname -r

              • 4. Re: Bonding configuration
                user7193548

                Yes, bonding is working.

                # ifconfig

                bond0     Link encap:Ethernet  HWaddr 90:B1:1C:4F:1F:98
                          inet addr:172.27.2.21  Bcast:172.27.2.255  Mask:255.255.255.0
                          inet6 addr: fe80::92b1:1cff:fe4f:1f98/64 Scope:Link
                          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                          RX packets:18623232 errors:0 dropped:0 overruns:0 frame:0
                          TX packets:14032493 errors:0 dropped:0 overruns:0 carrier:0
                          collisions:0 txqueuelen:1000
                          RX bytes:13076608186 (12.1 GiB)  TX bytes:4440275219 (4.1 GiB)
                          Interrupt:38

                lo        Link encap:Local Loopback
                          inet addr:127.0.0.1  Mask:255.0.0.0
                          inet6 addr: ::1/128 Scope:Host
                          UP LOOPBACK RUNNING  MTU:16436  Metric:1
                          RX packets:56324939 errors:0 dropped:0 overruns:0 frame:0
                          TX packets:56324939 errors:0 dropped:0 overruns:0 carrier:0
                          collisions:0 txqueuelen:0
                          RX bytes:31991646914 (29.7 GiB)  TX bytes:31991646914 (29.7 GiB)

                 

                # modinfo bonding | head

                filename:       /lib/modules/2.6.32-358.el6.x86_64/kernel/drivers/net/bonding/bonding.ko

                author:         Thomas Davis, tadavis@lbl.gov and many others

                description:    Ethernet Channel Bonding Driver, v3.6.0

                version:        3.6.0

                license:        GPL

                srcversion:     990DDFEBDBA03F305236885

                depends:        8021q,ipv6

                vermagic:       2.6.32-358.el6.x86_64 SMP mod_unload modversions

                parm:           max_bonds:Max number of bonded devices (int)

                parm:           tx_queues:Max number of transmit queues (default = 16) (int)

                 

                # uname -r

                2.6.32-358.el6.x86_64

                • 5. Re: Bonding configuration
                  Catch_22

                  You are using the RHCK kernel and not Oracle UEK?!

                   

                  Anyway, where are your Ethernet devices? It seems your system has not detected any Ethernet controllers. Missing support/drivers?

                   

                  What is your output of the following (please only post relevant info about your Ethernet controllers, not the whole output):

                   

                  # lspci -v

                  • 6. Re: Bonding configuration
                    user7193548

                    Here is the result

                    01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Dell Device 1f5b
                            Flags: bus master, fast devsel, latency 0, IRQ 35
                            Memory at d51a0000 (64-bit, prefetchable) [size=64K]
                            Memory at d51b0000 (64-bit, prefetchable) [size=64K]
                            Memory at d51c0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d8800000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-90-b1-1c-4f-1f-97
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    01:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Dell Device 1f5b
                            Flags: bus master, fast devsel, latency 0, IRQ 38
                            Memory at d51d0000 (64-bit, prefetchable) [size=64K]
                            Memory at d51e0000 (64-bit, prefetchable) [size=64K]
                            Memory at d51f0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d5100000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable+ Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-90-b1-1c-4f-1f-98
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Dell Device 1f5b
                            Flags: bus master, fast devsel, latency 0, IRQ 34
                            Memory at d50a0000 (64-bit, prefetchable) [size=64K]
                            Memory at d50b0000 (64-bit, prefetchable) [size=64K]
                            Memory at d50c0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d8000000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-90-b1-1c-4f-1f-99
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Dell Device 1f5b
                            Flags: bus master, fast devsel, latency 0, IRQ 36
                            Memory at d50d0000 (64-bit, prefetchable) [size=64K]
                            Memory at d50e0000 (64-bit, prefetchable) [size=64K]
                            Memory at d50f0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d5000000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-90-b1-1c-4f-1f-9a
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    ----

                    04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Broadcom Corporation Device 2003
                            Flags: bus master, fast devsel, latency 0, IRQ 40
                            Memory at d52a0000 (64-bit, prefetchable) [size=64K]
                            Memory at d52b0000 (64-bit, prefetchable) [size=64K]
                            Memory at d52c0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d9000000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-00-10-18-f7-81-86
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    04:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
                            Subsystem: Broadcom Corporation Device 2003
                            Flags: bus master, fast devsel, latency 0, IRQ 44
                            Memory at d52d0000 (64-bit, prefetchable) [size=64K]
                            Memory at d52e0000 (64-bit, prefetchable) [size=64K]
                            Memory at d52f0000 (64-bit, prefetchable) [size=64K]
                            Expansion ROM at d5200000 [disabled] [size=256K]
                            Capabilities: [48] Power Management version 3
                            Capabilities: [50] Vital Product Data
                            Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                            Capabilities: [a0] MSI-X: Enable- Count=17 Masked-
                            Capabilities: [ac] Express Endpoint, MSI 00
                            Capabilities: [100] Advanced Error Reporting
                            Capabilities: [13c] Device Serial Number 00-00-00-10-18-f7-81-87
                            Capabilities: [150] Power Budgeting <?>
                            Capabilities: [160] Virtual Channel
                            Kernel driver in use: tg3
                            Kernel modules: tg3

                    • 7. Re: Bonding configuration
                      Catch_22

                      Does ifconfig -a show your Ethernet interfaces?

                       

                      What is your output of the following:

                       

                      # ls /etc/sysconfig/network-scripts/ifcfg*

                       


                      • 8. Re: Bonding configuration
                        user7193548

                        ifconfig -a shows the interfaces except the one actice in the bond (eth3 in my case)

                        bond0     Link encap:Ethernet  HWaddr 90:B1:1C:4F:22:71
                                  inet addr:172.27.2.27  Bcast:172.27.2.255  Mask:255.255.255.0
                                  inet6 addr: fe80::92b1:1cff:fe4f:2271/64 Scope:Link
                                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                                  RX packets:140971 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:96705 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000
                                  RX bytes:74996011 (71.5 MiB)  TX bytes:26817000 (25.5 MiB)
                                  Interrupt:38

                        eth0      Link encap:Ethernet  HWaddr 90:B1:1C:4F:22:72
                                  BROADCAST MULTICAST  MTU:1500  Metric:1
                                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000
                                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                                  Interrupt:34

                        eth1      Link encap:Ethernet  HWaddr 90:B1:1C:4F:22:73
                                  BROADCAST MULTICAST  MTU:1500  Metric:1
                                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000
                                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                                  Interrupt:36

                        eth2      Link encap:Ethernet  HWaddr 90:B1:1C:4F:22:70
                                  BROADCAST MULTICAST  MTU:1500  Metric:1
                                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000
                                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                                  Interrupt:35

                        eth4      Link encap:Ethernet  HWaddr 00:10:18:F7:80:64
                                  BROADCAST MULTICAST  MTU:1500  Metric:1
                                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000
                                  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                                  Interrupt:40

                        eth5      Link encap:Ethernet  HWaddr 00:10:18:F7:80:65
                                  BROADCAST MULTICAST  MTU:1500  Metric:1
                                  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                  collisions:0 txqueuelen:1000

                        # ls /etc/sysconfig/network-scripts/ifcfg*

                        /etc/sysconfig/network-scripts/ifcfg-bond0  /etc/sysconfig/network-scripts/ifcfg-eth2  /etc/sysconfig/network-scripts/ifcfg-eth5

                        /etc/sysconfig/network-scripts/ifcfg-eth0   /etc/sysconfig/network-scripts/ifcfg-eth3  /etc/sysconfig/network-scripts/ifcfg-lo

                        /etc/sysconfig/network-scripts/ifcfg-eth1   /etc/sysconfig/network-scripts/ifcfg-eth4

                        • 9. Re: Bonding configuration
                          Catch_22

                          Bonding cannot be configured because your NIC's are down and not configured. Perhaps you are missing core script files, such as ifup-eth or your network is not starting for some other reason. Regarding the name of your devices you can check /etc/udev/rules.d/70-persistent-net.rules and see if there are any conflicts with device names and hardware addresses.

                          • 10. Re: Bonding configuration
                            user7193548

                            I have 9 servers with the same config and they all work fine ....

                             

                            # This file was automatically generated by the /lib/udev/write_net_rules
                            # program, run by the persistent-net-generator.rules rules file.
                            #
                            # You can modify it, as long as you keep each rule on a single
                            # line, and change only the value of the NAME= key.

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:b1:1c:4f:22:72", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:10:18:f7:80:65", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:b1:1c:4f:22:73", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:10:18:f7:80:64", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:b1:1c:4f:22:71", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

                            # PCI device 0x14e4:0x165f (tg3)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:b1:1c:4f:22:70", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

                            # PCI device 0x14e4:0x165f (tg3) (custom name provided by external tool)
                            SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:b1:1c:4f:22:71", ATTR{type}=="1", KERNEL=="eth*", NAME="bond0"

                            • 11. Re: Bonding configuration
                              Catch_22

                              There are no miracles about Linux. Obviously some configuration aspect is missing or wrong. I suggest to troubleshoot the network startup to see if there are any issues. You can press F2 during system startup to get more info or even better remove rhgb and quiet from the kernel boot parameter in grub.conf to see why your network devices are not being configured.

                              • 12. Re: Bonding configuration
                                user7193548

                                What I can see in /var/log/messages during a reboot:

                                Oct 11 12:58:07 server1 kernel: udev: renamed network interface eth3 to bond0

                                         => That's why I can't see it with ifconfig -a

                                 

                                Oct 11 12:58:07 server1 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready

                                Oct 11 12:58:07 server1 kernel: tg3 0000:01:00.1: bond0: Link is up at 1000 Mbps, full duplex

                                Oct 11 12:58:07 server1 kernel: tg3 0000:01:00.1: bond0: Flow control is off for TX and off for RX

                                Oct 11 12:58:07 server1 kernel: tg3 0000:01:00.1: bond0: EEE is disabled

                                Oct 11 12:58:07 server1 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready

                                 

                                But stil nothing under /proc/net/bonding ...

                                • 13. Re: Bonding configuration
                                  Catch_22

                                  I suggest to sort out the basic networking first before bonding. If you cannot get the Ethernet cards up and configured there is no reason to try bonding. Your NIC might need a firmware upgrade or a special driver. Have you compared the Linux kernel version, NIC driver and firmware version with any of your working systems?

                                  • 14. Re: Bonding configuration
                                    user7193548

                                    As I said before, bonding is working properly.

                                    The only thing is /proc/net/bonding does not exists.

                                    I have 9 server with fresh install and this is the same for the 9 server.

                                    What I have done is disable the NetworkManager:

                                    chkconfig NetworkManager off

                                    And configure bonding manually

                                    1 2 Previous Next