1 Reply Latest reply: Jan 3, 2011 2:59 PM by 70824 RSS

    not getting 10Gb/s speed on a 10Gb interconnect

    user620488
      Why We are not getting 10Gb/s on 10Gb HP NC522SFP NIC and 10Gb Cisco 4900M switches?

      We have 4 HP Proliant 580 G5 node cluster running Orcle Enterprise Linux 5.4 2.6.18-164.el5 x86_64
      With interconnect of 2 10Gb Cisco 4900M switches and 10Gb HP NC522SFP NICs.
      We are using IP Bonding with mode 1 active-backup mode with jumbo frames (8000 MTU).

      NC522SFP NIC detils:
      [root@dionysus2 network-scripts]# ethtool -i eth0
      driver: netxen_nic
      version: 4.0.11
      firmware-version: 4.0.520
      bus-info: 0000:0d:00.0
      [root@dionysus2 network-scripts]# ethtool -i eth1
      driver: netxen_nic
      version: 4.0.11
      firmware-version: 4.0.520
      bus-info: 0000:0d:00.1


      The max we have seen was around 8Gb/s for TCP stream and around 6Gb/s for UDP.



      [root@dionysus3 ~]# netperf -t TCP_STREAM -D 10 -l 300 -H prvdionysus2
      TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to prvdionysus2.houdiniinc.com (1.1.1.141) port 0 AF_INET : demo
      Interim result: 7948.70 10^6bits/s over 10.00 seconds
      Interim result: 8126.52 10^6bits/s over 10.01 seconds
      Interim result: 8406.64 10^6bits/s over 10.00 seconds
      Interim result: 8441.50 10^6bits/s over 10.00 seconds
      Interim result: 8482.62 10^6bits/s over 10.00 seconds
      Interim result: 8446.60 10^6bits/s over 10.04 seconds
      Interim result: 8447.09 10^6bits/s over 10.00 seconds
      Interim result: 8422.01 10^6bits/s over 10.03 seconds
      Interim result: 8478.59 10^6bits/s over 10.00 seconds
      Interim result: 8407.24 10^6bits/s over 10.08 seconds
      Interim result: 8451.19 10^6bits/s over 10.00 seconds
      Interim result: 8440.60 10^6bits/s over 10.01 seconds
      Interim result: 8429.85 10^6bits/s over 10.01 seconds
      Interim result: 8438.75 10^6bits/s over 10.00 seconds
      Interim result: 8447.06 10^6bits/s over 10.00 seconds
      Interim result: 8434.38 10^6bits/s over 10.02 seconds
      Interim result: 8457.51 10^6bits/s over 10.00 seconds
      Interim result: 8344.26 10^6bits/s over 10.14 seconds
      Interim result: 8308.06 10^6bits/s over 10.04 seconds
      Interim result: 8269.21 10^6bits/s over 10.05 seconds
      Interim result: 8277.53 10^6bits/s over 10.00 seconds
      Interim result: 8259.36 10^6bits/s over 10.02 seconds
      Interim result: 8099.29 10^6bits/s over 10.20 seconds
      Interim result: 8242.95 10^6bits/s over 10.00 seconds
      Interim result: 8263.19 10^6bits/s over 10.00 seconds
      Interim result: 8256.44 10^6bits/s over 10.01 seconds
      Interim result: 8258.27 10^6bits/s over 10.00 seconds
      Interim result: 8262.64 10^6bits/s over 10.00 seconds
      Interim result: 8267.47 10^6bits/s over 10.00 seconds
      Recv Send Send
      Socket Socket Message Elapsed
      Size Size Size Time Throughput
      bytes bytes bytes secs. 10^6bits/sec

      87380 16384 16384 300.01 8337.65


      [root@dionysus3 ~]# netperf -t UDP_STREAM -D 10 -l 300 -H prvdionysus2
      UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to prvdionysus2.houdiniinc.com (1.1.1.141) port 0 AF_INET : demo
      Interim result: 6552.58 10^6bits/s over 10.00 seconds
      Interim result: 6554.02 10^6bits/s over 10.00 seconds
      Interim result: 6547.88 10^6bits/s over 10.01 seconds
      Interim result: 6559.75 10^6bits/s over 10.00 seconds
      Interim result: 6547.93 10^6bits/s over 10.02 seconds
      Interim result: 6536.14 10^6bits/s over 10.02 seconds
      Interim result: 6556.46 10^6bits/s over 10.00 seconds
      Interim result: 6564.40 10^6bits/s over 10.00 seconds
      Interim result: 6556.85 10^6bits/s over 10.01 seconds
      Interim result: 6561.21 10^6bits/s over 10.00 seconds
      Interim result: 6570.83 10^6bits/s over 10.00 seconds
      Interim result: 6581.72 10^6bits/s over 10.00 seconds
      Interim result: 6571.45 10^6bits/s over 10.02 seconds
      Interim result: 6541.24 10^6bits/s over 10.05 seconds
      Interim result: 6531.86 10^6bits/s over 10.01 seconds
      Interim result: 6568.38 10^6bits/s over 10.00 seconds
      Interim result: 6568.17 10^6bits/s over 10.00 seconds
      Interim result: 6546.56 10^6bits/s over 10.03 seconds
      Interim result: 6551.74 10^6bits/s over 10.00 seconds
      Interim result: 6565.21 10^6bits/s over 10.00 seconds
      Interim result: 6561.55 10^6bits/s over 10.01 seconds
      Interim result: 6576.34 10^6bits/s over 10.00 seconds
      Interim result: 6578.83 10^6bits/s over 10.00 seconds
      Interim result: 6567.92 10^6bits/s over 10.02 seconds
      Interim result: 6573.36 10^6bits/s over 10.00 seconds
      Interim result: 6573.26 10^6bits/s over 10.00 seconds
      Interim result: 6573.46 10^6bits/s over 10.00 seconds
      Interim result: 6561.73 10^6bits/s over 10.02 seconds
      Interim result: 6564.56 10^6bits/s over 10.00 seconds
      Socket Message Elapsed Messages
      Size Size Time Okay Errors Throughput
      bytes bytes secs # # 10^6bits/sec

      262144 65507 300.00 3756254 0 6561.58
      262144 300.00 3679169 6426.93


      [root@dionysus2 network-scripts]# netstat -s | egrep -i "udpInOverflows|packet receive errors|fragments dropped|reassembles failed|fragments dropped after timeout"
      154170 packet receive errors