[TriLUG] copying files

[Steve Litt]

On Thu, 21 Jun 2012 18:24:53 -0700 (PDT), Joseph Mack NA3T said:
> On Thu, 21 Jun 2012, Steve Litt wrote:

> > Up to 20x faster transfer than normal SSH. This would rock 
> > whether you're doing rsync over ssh, or ssh mount, or scp, 
> > or pretty much anything involving SSH. I haven't tried it 
> > -- I just heard about it a few minutes ago.
> >
> > And of course, if your bottleneck turns out to be disk 
> > access at either end, or CPU at either end, or a slow 
> > wire, this won't help you.
> 
> 
> I'm uploading through a home dsl line where 30kBps up is 
> about it and it's already busy doing other things. This is 
> about the bandwidth I'll get when it's in production.

Hi Joe,

At 30kBps, I'm almost positive your bottleneck is the wire, by a huge
factor. Therefore, just about the only thing you should be worrying
about is how to compress the daylights out of the files before they hit
the wire. That wire is so slow that your cpus and hard disks on the
ends won't even break a sweat, whether you use rsync, or cpio, or
whatever, even if you use an insanely bloaty compression algorithm.

Under those circumstances, I'd use rsync's --compress-level=NUM
parameter to put those files under enough pressure to make diamonds.
I'm pretty sure that, given your situation, nothing else but
compression could make a substantial difference in transfer time,
always assuming you use a technique that correctly deduces which files
need to be transferred.

Thanks

SteveT

Steve Litt                *  http://www.troubleshooters.com/
                          *  http://twitter.com/stevelitt
Troubleshooting Training  *  Human Performance