This discussion is archived
3 Replies Latest reply: Dec 2, 2009 10:01 PM by 807567 RSS

bulk data transfer over WAN

807567 Newbie
Currently Being Moderated
Hello Gurus!

I need some help with improving our data transfer rate between two locations (Midwest to East-Coast).

our setup information:
1. we have a 1Gbit dedicated pipe between locations
2. our RTT = 23 milliseconds.
3. both servers (source, destination) = Sun x4600, Solaris10 Update 6 (x86)
4. we are using FTP to transfer data (fastest method?)


Till now, we tried:
1. 1GB file and 5GB files.
2. tweaked TCP parameters to:

tcp_xmit_hiwat = 4194304
tcp_recv_hiwat = 4194304
tcp_cwnd_max =8388608
tcp_max_buf = 33554432
tcp_wscale_always = 1

3. at one point we got about 61Mbytes (480 Mbits/sec) both directions...

BUT, we are not able to reproduce that again.... not sure what changed.

Question:
1. can someone please advice that best procedure to optimise your data-transfer rate between two locations?
(we have 1Gbit pipe, 23 millisec roundTripTime).

2. also does FTP have different overhead when it does "put" over "get" .. performance is better when "put"?

please advice,
Thank you very much in advance!

Edited by: pveeravalli on Jul 7, 2009 6:19 AM
  • 1. Re: bulk data transfer over WAN
    807567 Newbie
    Currently Being Moderated
    wow if i have that kind of dedicated link and at that speed, i'd do a cpio instead. beats having to put and get.

    btw, i use cpio for bulk data transfers between my server over an nfs network. but my network are geographically close.
  • 2. Re: bulk data transfer over WAN
    807567 Newbie
    Currently Being Moderated
    Hi all!
    I've just visited this forum. Happy to get acquainted with you. Thanks.
    __________________
    http://animewatching.com

    Edited by: ntvinh986 on Nov 24, 2009 3:33 AM
  • 3. Re: bulk data transfer over WAN
    807567 Newbie
    Currently Being Moderated
    Hello.

    First problem I see is the 5GB. Many methods are limited to 2GiB (Gi means 1024^3) or 4GiB. So you first have to find out if the method choosen allows transferring 5GB.

    FTP uses two network connections. One of them for control (e.g. "PUT" command) and the other one only for data. On the data connection there is no byte stuffing or control bytes. This means: On a TCP network FTP is the fastest method for large files (for small files you'd had to think about the overhead in the control connection).

    However:
    - If you use "your" network (not the public Internet) you may try to choose a network protocol that has less data overhead than TCP/IP. (Unfortunately I'm not an expert in this question.)
    - Maybe data compression will also be an option to improve your performance. This will probably not work if your data is already compressed.

    Martin