[TriLUG] copying files

[David Brain]

Another couple of options in the scenario you are describing would be:

http://code.google.com/p/inotify-sync/
http://code.google.com/p/rsync-inotify/

Which trigger sync based on detecting changes in the source dir.

David.

On Tue, Jun 19, 2012 at 10:15 PM, Joseph Mack NA3T <jmack at wm7d.net> wrote:
> On Tue, 19 Jun 2012, Jeff Schornick wrote:
>
>> On Tue, Jun 19, 2012 at 9:39 PM, Joseph Mack NA3T <jmack at wm7d.net> wrote:
>>>
>>> I haven't used rsync. So after the initial phase, both ends know the
>>> files
>>> at each end and when I add a new file at one end, rsync will notice and
>>> just
>>> handle it?
>>
>>
>> Not quite.
>>
>> On each synchronization run, rsync creates a local list from the
>> source directory, while simultaneously creating the analogous list on
>> the remote end.  This means if you have 1000 files, you may be looking
>> at 1000 fstats on each end.  However, these checks are both done
>> locally on the corresponding machines.  As long as the target system's
>> local file I/O isn't significantly slower than the source machine's,
>> you shouldn't be introducing any additional delay.
>>
>> After both lists have been generated, rsync uses a minimal amount of
>> network traffic to compare the lists and generate a final list of
>> which files need to be updated.  As expected, only those files are
>> sent over the network.
>>
>> After the synchronization is complete, the generated lists get tossed
>> out as dirty laundry.  There is no long running daemon which attempts
>> to keep them up-to-date in realtime.  However, I imagine someone has
>> created a slick piece of code using inotify to do just that.
>
>
> OK, so I'd have to invoke rsync every 5 mins. Assembling the list of files
> at each end has to be done anyhow (eg find). Presumably 1000 fstats take the
> same time no matter whether find or rsync then processes the list. The
> problem then is comparing the lists at each end.
>
> cp -auv is really slow
>
> rsync you say is fast (and I believe you).
>
> but I already have my list from `find`, so there's no extra cost if I use
> find.
>
> The copy of the files takes the same time no matter which way I assembled
> the list of files to be copied.
>
> So `find` followed by `cp --parents` or `cpio` seems to be it.
>
> Alan points out the resilience of rsync. This is a good feature, but as it
> turns out (and I didn't say this), I don't mind loosing an occassional file,
> but throughput is high priority. The backup machine is writing files from
> many sources and it only has a few seconds to service a source machine, or
> it will fall over with the load.
>
>
> Joe
>
> --
> Joseph Mack NA3T EME(B,D), FM05lw North Carolina
> jmack (at) wm7d (dot) net - azimuthal equidistant map
> generator at http://www.wm7d.net/azproj.shtml
> Homepage http://www.austintek.com/ It's GNU/Linux!
> --
> This message was sent to: dbrain at gmail.com <dbrain at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that
> address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web  :
> http://www.trilug.org/mailman/options/trilug/dbrain%40gmail.com
> TriLUG FAQ          : http://www.trilug.org/wiki/Frequently_Asked_Questions