This discussion is archived
1 Reply Latest reply: Jan 3, 2011 12:59 PM by 70824 RSS

not getting 10Gb/s speed on a 10Gb interconnect

623491 Newbie
Currently Being Moderated
Why We are not getting 10Gb/s on 10Gb HP NC522SFP NIC and 10Gb Cisco 4900M switches?

We have 4 HP Proliant 580 G5 node cluster running Orcle Enterprise Linux 5.4 2.6.18-164.el5 x86_64
With interconnect of 2 10Gb Cisco 4900M switches and 10Gb HP NC522SFP NICs.
We are using IP Bonding with mode 1 active-backup mode with jumbo frames (8000 MTU).

NC522SFP NIC detils:
[root@dionysus2 network-scripts]# ethtool -i eth0
driver: netxen_nic
version: 4.0.11
firmware-version: 4.0.520
bus-info: 0000:0d:00.0
[root@dionysus2 network-scripts]# ethtool -i eth1
driver: netxen_nic
version: 4.0.11
firmware-version: 4.0.520
bus-info: 0000:0d:00.1


The max we have seen was around 8Gb/s for TCP stream and around 6Gb/s for UDP.



[root@dionysus3 ~]# netperf -t TCP_STREAM -D 10 -l 300 -H prvdionysus2
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to prvdionysus2.houdiniinc.com (1.1.1.141) port 0 AF_INET : demo
Interim result: 7948.70 10^6bits/s over 10.00 seconds
Interim result: 8126.52 10^6bits/s over 10.01 seconds
Interim result: 8406.64 10^6bits/s over 10.00 seconds
Interim result: 8441.50 10^6bits/s over 10.00 seconds
Interim result: 8482.62 10^6bits/s over 10.00 seconds
Interim result: 8446.60 10^6bits/s over 10.04 seconds
Interim result: 8447.09 10^6bits/s over 10.00 seconds
Interim result: 8422.01 10^6bits/s over 10.03 seconds
Interim result: 8478.59 10^6bits/s over 10.00 seconds
Interim result: 8407.24 10^6bits/s over 10.08 seconds
Interim result: 8451.19 10^6bits/s over 10.00 seconds
Interim result: 8440.60 10^6bits/s over 10.01 seconds
Interim result: 8429.85 10^6bits/s over 10.01 seconds
Interim result: 8438.75 10^6bits/s over 10.00 seconds
Interim result: 8447.06 10^6bits/s over 10.00 seconds
Interim result: 8434.38 10^6bits/s over 10.02 seconds
Interim result: 8457.51 10^6bits/s over 10.00 seconds
Interim result: 8344.26 10^6bits/s over 10.14 seconds
Interim result: 8308.06 10^6bits/s over 10.04 seconds
Interim result: 8269.21 10^6bits/s over 10.05 seconds
Interim result: 8277.53 10^6bits/s over 10.00 seconds
Interim result: 8259.36 10^6bits/s over 10.02 seconds
Interim result: 8099.29 10^6bits/s over 10.20 seconds
Interim result: 8242.95 10^6bits/s over 10.00 seconds
Interim result: 8263.19 10^6bits/s over 10.00 seconds
Interim result: 8256.44 10^6bits/s over 10.01 seconds
Interim result: 8258.27 10^6bits/s over 10.00 seconds
Interim result: 8262.64 10^6bits/s over 10.00 seconds
Interim result: 8267.47 10^6bits/s over 10.00 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 300.01 8337.65


[root@dionysus3 ~]# netperf -t UDP_STREAM -D 10 -l 300 -H prvdionysus2
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to prvdionysus2.houdiniinc.com (1.1.1.141) port 0 AF_INET : demo
Interim result: 6552.58 10^6bits/s over 10.00 seconds
Interim result: 6554.02 10^6bits/s over 10.00 seconds
Interim result: 6547.88 10^6bits/s over 10.01 seconds
Interim result: 6559.75 10^6bits/s over 10.00 seconds
Interim result: 6547.93 10^6bits/s over 10.02 seconds
Interim result: 6536.14 10^6bits/s over 10.02 seconds
Interim result: 6556.46 10^6bits/s over 10.00 seconds
Interim result: 6564.40 10^6bits/s over 10.00 seconds
Interim result: 6556.85 10^6bits/s over 10.01 seconds
Interim result: 6561.21 10^6bits/s over 10.00 seconds
Interim result: 6570.83 10^6bits/s over 10.00 seconds
Interim result: 6581.72 10^6bits/s over 10.00 seconds
Interim result: 6571.45 10^6bits/s over 10.02 seconds
Interim result: 6541.24 10^6bits/s over 10.05 seconds
Interim result: 6531.86 10^6bits/s over 10.01 seconds
Interim result: 6568.38 10^6bits/s over 10.00 seconds
Interim result: 6568.17 10^6bits/s over 10.00 seconds
Interim result: 6546.56 10^6bits/s over 10.03 seconds
Interim result: 6551.74 10^6bits/s over 10.00 seconds
Interim result: 6565.21 10^6bits/s over 10.00 seconds
Interim result: 6561.55 10^6bits/s over 10.01 seconds
Interim result: 6576.34 10^6bits/s over 10.00 seconds
Interim result: 6578.83 10^6bits/s over 10.00 seconds
Interim result: 6567.92 10^6bits/s over 10.02 seconds
Interim result: 6573.36 10^6bits/s over 10.00 seconds
Interim result: 6573.26 10^6bits/s over 10.00 seconds
Interim result: 6573.46 10^6bits/s over 10.00 seconds
Interim result: 6561.73 10^6bits/s over 10.02 seconds
Interim result: 6564.56 10^6bits/s over 10.00 seconds
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

262144 65507 300.00 3756254 0 6561.58
262144 300.00 3679169 6426.93


[root@dionysus2 network-scripts]# netstat -s | egrep -i "udpInOverflows|packet receive errors|fragments dropped|reassembles failed|fragments dropped after timeout"
154170 packet receive errors
  • 1. Re: not getting 10Gb/s speed on a 10Gb interconnect
    70824 Explorer
    Currently Being Moderated
    8GB/s raw data - no processing at all. With 8k Jumbo frames. And a lot of packet receive errors. Not being a network specialist, I'd still recommend to drop the jumbo frames, and return to standard 1500 byte frames. Get rid of these packet errors.

    What does this have to do with the product Oracle Application Interconnect?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points