[TriLUG] Ot: File Transfer

[David Rasch]

On Tue, Jun 21, 2005 at 07:47:10AM -0400, Mark Freeze <mfreeze at gmail.com> wrote:
> Since I moved here, and they have seen a few of the things Linux can
> do, they are trying to integrate some Linux into their environment.
> They want me to set up a dedicated Linux box (or M$ box) to use as a
> repository for their daily backup file, which runs out at about 16GB. 
> The data is already compressed from the mac.  They are willing to
> upgrade my internet service to the 3 Meg service from Wind Channel.
> 
> Before I get 10,000 comments on how they should do things differently,
> please realize that I have no control or input over their systems. 
> I'm just trying to figure out the best way to handle this on my end. 
> I had originally thought that this was a text file from a customer
> until I started asking about specifics.  Their comment to me was
> basically, "We want to use your place as a backup location.  What all
> do we need to do to get that going."
> 
> Given those facts, I'd be glad to hear ideas from anyone.
> 

Comment <?php print(i++); ?> on how to do differently: 
As a manager of corporate data, this seems like a "bad idea(tm)".
Storing backup data at people's houses introduces all kinds of legal
and security questions.  I wouldn't want my corporate data backed up at
my house, much less one of my employees.  

As an alternative, Time Warner Telecom offers a backup service where
they'll host this service for you at far higher bandwidth than 3Mbps and
also probably have a full web-interface for restoring individual files,
or much better semantics than a 16GB file.  

Alternatively, lease a server at a place like Server Beach and sync the
file to there using rsync.  

Both of these will allow you higher bandwidth, reasonable price
(ServerBeach can be had for ~ $100/month) and are hundreds of times more
reliable and secure than any home network in terms of connectivity,
bandwidth, redundant power, etc.  Not to mention, you are no longer
responsible for the file in your home.



To answer your question directly:
Rsync the file from the office to your home.   Depending on the data
type, you might increase the chunk size to speed up the initial
checksumming and transfer.  

If you want to store backup copies efficiently (by using more CPU, and
probably increasing total backup time), you might try something like
"rdiff-backup", though be careful to create some backup checking scripts
as we've recently run into some problems where data we thought was being
backed up was in fact, not.

Altneratively, you can specify a command-line option to plain rsync
"--compare-dest" that will use the old file for comparison and to aid
transfer, but keep the old file intact so you could keep X copies from
the last X days.

-David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://www.trilug.org/pipermail/trilug/attachments/20050621/b18251d8/attachment.pgp>