[TriLUG] Tuning WAN links

[Aaron S. Joyner]

Shawn Hood wrote:
> Hey guys...
>
> I've recently had a dedicated gigabit fiber WAN link that runs between
> Rackspace in Dallas and an office in Bethesda, MD dropped in my lap.  It's
> not often (read: ever) that I'm given a high-bandwidth high-latency link to
> tune.
>
> Here are the basics:
>
> Office in Bethesda
> Catalyst 3560
>     |
> AboveNet POP - Vienna, VA
> Catalyst 6509
>     |
> AboveNet IP/MPLS Backbone
>     |
> AboveNet POP - Dallas, TX
> Catalyst 6509
>     |
> Rackspace - Dallas, TX
> Catalyst 3560
>
>
> I've run iperf between two RHEL4 boxes connected to the 3560s.  The most
> throughput I've been able to get is ~45mbit by increasing the buffer sizes
> in /etc/sysctl and using massive window sizes on iperf.  I was hoping you
> guys could point me in the right direction.  I need to do some reading about
> how to get the most out of this link, and any reference would be greatly
> appreciated.  Will this be a matter of create a Linux router on each end to
> shape the traffic destined for this link?  Is this something better suited
> for proprietary technology that claims to 'auto-tune' this kinds of links.
> I'm fairly fluent when it comes to talking about this stuff 'in theory,' but
> have yet to get any hands on experience.
>
> Questions, comments, suggestions?
>
> Shawn
>   
Let me start by saying that it's not typical for you to have one machine 
on one end transfer at 1gbit to the other end, that's a rather unusual 
use case.  That's not to say it can't be done, but it's just not the 
common way to test such a link.  Typical usage for this type of link is 
to have a lot of computers one one end talking to a lot of computers on 
the other end, and you measure the throughput by looking at the raw 
numbers on one end point or the other, in terms of packets or bits per 
second.  That said, you can certainly saturate it with an appropriate 
setup on both ends.  A good place to start is to read up on TCP tuning, 
but it sounds like you're most of the way there in terms of munging 
/etc/sysctl
http://www.google.com/search?q=tcp+tuning
particularly, this has a pretty good analysis:
http://www-didc.lbl.gov/TCP-tuning/linux.html

Well, this response was somewhat content-free.  If you can provide some 
more details about what you have tried with and with out success in 
terms of TCP settings, perhaps I can suggest additional, more specific 
things to try.

Happy tweaking!
Aaron S. Joyner