[TriLUG] copying files

[David Brain]

And while we are talking about neat rsync 'tricks' (i.e. slight
diversion from topic at hand), 'rsync --link-dest'  is nice for making
snapshot like backups, basically rather than doing:

rsync -a source dest

you do

rsync -a --link-dest compare_dir source dest

then rsync will sync over any from source to dest, but any files that
are unchanged between compare_dir and dest are just hardlinked, thus
not using up additional disk space.

I use this for backing up my (kvm) VMs at home, I have the machine
hosting the VMs make LVM snapshots of their file systems, mount them
and do an rsync --link-dest to a daily backup directory, comparing to
yesterday's backup.  Then unmount and remove the snapshot and I'm
done.  Keeping 14 days of easy to access  backups of ~700G of data on
a 1TB drive that way, on an external drive.

There's a project that simplifies setting up some of this http://rsnapshot.org/.

David.

On Fri, Jun 22, 2012 at 9:50 AM, Alan P'orter <porter at trilug.org> wrote:
>
>
>> I'm uploading through a home dsl line where 30kBps up is about it and it's
>> already busy doing other things. This is about the bandwidth I'll get when
>> it's in production.
>
>
> Ah, so if you're not tired of hearing about rsync yet...
>
> One thing you'll notice about a large OUTBOUND file transfer from your home
> is that the rest of your internet usage is crap while the file transfer is
> going on. The problem is not with your inbound connection... you have plenty
> of incoming bandwidth. The problem is that you are saturating your outbound
> connection.
>
> So while you're backing up your data at full-throttle, your wife comes in to
> check her email and it takes FOREVER to retrieve even the tiniest file. She
> can't even play Words With Friends... it keeps timing out. Then you catch
> flack for killing the network.
>
> Her email is slow because her TCP connection needs to send a handful of ACK
> packets to tell the remote end to keep sending. Without the ACK packets, the
> flow stops. But her ACK packets are stuck behind 400 tractor-trailers on the
> information superhighway. You need to open up an HOV lane.
>
> Rsync allows you to cap the bandwidth at a fixed level. Say you want to stay
> just under your 30kBps limit. You simply do this:
>
> rsync -avz --bwlimit=28 /home/joe/pr0n/ joe at trilug.org:pr0n-from-home/
>
> That keeps the goods flowing out, but it also leaves just a small amount of
> outgoing bandwidth to allow other normal traffic on the home network. In
> fact, you could probably even watch a Netflix movie while your files are
> transferring... as long as you don't saturate that uplink.
>
> And here's the cool thing... you can tweak that bwlimit (KBYTES/sec) number
> by CTRL-C and trying again with a new bwlimit. Rsync is smart enough to pick
> up where it left off (OK, it's dumb, it recalculates the differences each
> time, but the end result is the same). So you never end up re-copying files
> that don't need to be re-copied.
>
> --
> # ɹǝʇɹoԀ uɐl∀
>
>
>
>
>
> --
> This message was sent to: dbrain at gmail.com <dbrain at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that
> address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web  :
> http://www.trilug.org/mailman/options/trilug/dbrain%40gmail.com
> TriLUG FAQ          : http://www.trilug.org/wiki/Frequently_Asked_Questions