[TriLUG] Data manipulation over Samba

[Steve Litt]

On Tuesday 22 May 2007 07:30, Jim Tuttle wrote:
> So, this has been bothering me. I'm hoping someone has an answer and,
> perhaps, a reference.
>
> Ok, I'm running some python data processing scripts against an
> orthophoto collection residing on a disk array in the basement.  There
> are about 4,300 images each about 76MB.  There are several smaller files
> with each image.  Part of the processing includes copying each file to
> another partition on our 14TB ATABeast.  The question is this: Is any of
> this data moving over the network to my machine?
>
> The processing is taking forever.  215 images in 8 hours.  I wondered if
> the images are being read into memory by my machine then written to the
> other partition on the array.  I have this fantasy that python tells the
> processor on the disk array to do the copying, but I imagine that isn't
> true.  To make matters worse, there are several connections through
> which this data traverses.  The array is mounted via fiber channel to a
> Solaris cluster which offers it to a linux machine in the cube next to
> me via NFS and I'm mounting that via samba on my desktop.
>
> I could have and probably should have run this on the intermediate
> machine, but wasn't thinking last night.  Neither the ATABeast nor the
> Solaris cluster have python installed and that's a non-starter.
>
> Thanks,
> Jim

If the python scripts run on your local client, then the data goes over the 
wire to your local client, and back to the server's other partition.

You're not running the python scrips on the server because the server doesn't 
have python. Assuming for some reason you can't install python on the server, 
then you can have your client python write a shellscript (or batch file if 
the server's Windows based) to the server, and then find a way to execute the 
shellscript/batch file on the server. If the server's Windows based, you can 
create a phony Samba server whose print command= the shellscript. If the 
server is Unix, a cron or some sort of rsh can do it. If it's Unix you could 
also use sockets to make a client-server setup, which wouldn't require python 
on the server.

HTH

SteveT

Steve Litt
Author: Universal Troubleshooting Process books and courseware
http://www.troubleshooters.com/