[TriLUG] Scheduling file transfers

[Scott G. Hall]

This is the classic asynchronous data transfer problem.  I boy, have I
done a lot of these over the years.

The old-fashioned (meaning old 1980's UNIX way) of doing this is to
transfer the data file.  Then when complete, transfer a zero-byte
flag file of the same name with a *.flag extension.  On the receiving
end have a watchdog process wake up every now and then (or run from
cron; an inittab/restart process with a sleep loop works best), and
look for any flag files; if it finds one, process the associated data
file.

More recently (mid-1990's) I worked on a project the was truly
asynchronous in nature.  When the data-source would finish transferring
a data file to the target machine, it would then run a remote command
to process that file, giving the filename as the argument (using rsh or
ssh).  The target machine responsible for displaying the data had no
cron-job or watchdog at all, just an executable (usually a script)
called by the source machine(s) to process data files.

The advantage to the latter is it is more responsive -- processing of
the data file takes place as soon as transferred, and there is the
possibility of processing more than one file at a time, possibly from
multiple sources.

On 7-Apr-2005 15:18 EDT, Mark Freeze wrote:
> The way I need my script to read will not allow for picking up part of
> a file then picking the rest of the file up later, unless by later you
> guys mean later like a couple of ms later.  My script would need to do
> the following things:
> 
> 1. Check for the existance of a file.
> 2. Download the file.
> 3. Run the file through a parsing program.
> 4. Import the parsed file into a database.
> 5. Query the database and email the results to a recipient list.
> (Quantity and dollar totals of the downloaded file.)
> 6. Export the file into a Samba directory so my Windows box can pick
> it up and process it through some canned software then place the
> result file back into the Samba directory.
> 7. Use php to convert the result file into seperate pdf images.
> 8. Place the pdf images into a directory and index the pdf file list
> into a web-enabled database so users can log into my website and view
> customers bills (the pdfs) online.
> 
> On some small files this process would be almost instanateous.  But on
> larger files the download might take a while so I didnt want my script
> getting part of the file and starting the next step before the
> download was complete.

--
Scott G. Hall
Raleigh, NC, USA
ScottGHall at BellSouth.Net