[TriLUG] Scheduling file transfers

[John Turner]

On Apr 7, 2005, at 9:33 AM, Mark Freeze wrote:

> A year or so ago I had a problem downloading a file via ftp onto a
> Windows box with WS_FTP.  The file was about 100MB and I started
> downloading the file while my customer was still uploading, so I only
> got about half of the file.  WS_FTP allowed me to do this with no
> error. (Which I thought was kinda crazy.)
>
> Now I have an offsite ftp spot that my customers use to send me files
> at random times during the day. I want to automatically download and
> process these files onto my box as soon as they appear on the site so
> I was thinking that I would scehedule up a cron job to look for these
> files every 10 min. When I do this am I going to have the problem of
> seeing the file and trying to get it as they are uploading?  Some of
> these files are over 100MB and might take my customer a while to
> upload.  Someone told me to make sure that I have exclusive access to
> the file before I download it, but since I have no control over the
> ftp server I'm not sure on how to accomplish that task.
>
>

If you can only use FTP, then you might want to try ncftp or ncftpget 
from your server. ncftp will resume downloads, I haven't checked that 
this also means it will just get the end of a file that has grown, but 
it is worth a try.

John