[Trilug-ontopic] [bash] copying `find`ed files across network
Tom Roche
Tom_Roche at pobox.com
Wed May 23 18:26:11 EDT 2012
http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000315.html
Tom Roche Wed, 23 May 2012 15:36:07 -0400
>>>> I'm trying to copy a buncha files across a network[:]
>>>> me at remote:~ $ find /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | sort | xargs du -ch | tail -n 1
>>>> > 36M total
>>>> and I want to copy only those files.
http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000316.html
Alan Porter Wed May 23 15:45:15 EDT 2012
>>> tar -zcf - $(find /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$') | ssh root at remote "tar -zxvf -"
http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000317.html
Tom Roche Wed, May 23, 2012 at 05:19:04PM -0400
>> making a similar adjustment adds a datapoint (starred below):
...
>> > $ mkdir -p /home/me/code/CMAQ/BLD_ddm_saprc07tc
>> > $ pushd /home/me/code/CMAQ/BLD_ddm_saprc07tc
...
>> # case 1: `raw find`
>> * $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
>> * /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
>> * 266
>> # case 2: `find` inside `tar`
>> * $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar cfvz - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar xvfz -
>> * tar: Cowardly refusing to create an empty archive
>> * Try `tar --help' or `tar --usage' for more information.
...
>> Why is the raw `find` working, but the same `find` inside `tar`
>> draws the 'empty archive' warning (presumably due to *not* finding
>> the files)?
http://www.trilug.org/pipermail/trilug-ontopic/2012-May/000318.html
Brian McCullough Wed, 23 May 2012 17:41:07 -0400
> I suspect that it is because of the order of the parameters.
> "f" requires a value, which it is, I suspect, taking [as] "z".
IIUC, your hypothesis is, 'f' should be the final parameter--no?
Above I used `tar` flags={cfvz, xvfz}. Dropping the 'v', since it
seems to merely complicate this issue, that would seem to equate to
using `tar` flags={cfz, xfz}. So fix that:
# here I use `tar` flags={zcf, zxf} -- should use a bash array :-)
REMOT_PATH='/work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc'
LEAF="$(basename ${REMOT_PATH})"
LOCAL_ROOT="${HOME}/code/CMAQ/${LEAF}"
REMOT_FIND="find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$'"
REMOT_TAR="pushd ${REMOT_PATH} ; tar zcf - \$(${REMOT_FIND})"
for CMD in \
"mkdir -p ${LOCAL_ROOT}" \
"pushd ${LOCAL_ROOT}" \
"ssh t \"pushd ${REMOT_PATH} ; ${REMOT_FIND} | wc -l\"" \
"ssh t \"${REMOT_TAR}\" | tar zxf -" \
"find . -type f | wc -l" \
"popd" \
; do
echo -e "$ ${CMD}"
eval "${CMD}"
done
> $ mkdir -p /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> $ pushd /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> ~/code/CMAQ/BLD_ddm_saprc07tc ~
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
> /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
> 266
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar zcf - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar zxf -
> tar: Cowardly refusing to create an empty archive
> Try `tar --help' or `tar --usage' for more information.
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> $ find . -type f | wc -l
> 0
> $ popd
No change :-(
For completeness, I checked the other permutation ending in 'f':
# here I use `tar` flags={czf, xzf}
REMOT_PATH='/work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc'
LEAF="$(basename ${REMOT_PATH})"
LOCAL_ROOT="${HOME}/code/CMAQ/${LEAF}"
REMOT_FIND="find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$'"
REMOT_TAR="pushd ${REMOT_PATH} ; tar czf - \$(${REMOT_FIND})"
for CMD in \
"mkdir -p ${LOCAL_ROOT}" \
"pushd ${LOCAL_ROOT}" \
"ssh t \"pushd ${REMOT_PATH} ; ${REMOT_FIND} | wc -l\"" \
"ssh t \"${REMOT_TAR}\" | tar xzf -" \
"find . -type f | wc -l" \
"popd" \
; do
echo -e "$ ${CMD}"
eval "${CMD}"
done
> $ mkdir -p /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> $ pushd /home/tlroche/code/CMAQ/BLD_ddm_saprc07tc
> ~/code/CMAQ/BLD_ddm_saprc07tc ~
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$' | wc -l"
> /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ~
> 266
> $ ssh t "pushd /work/MOD3EVAL/nsu/boundary/BLD_ddm_saprc07tc ; tar czf - $(find . -type f | grep -ve 'CCTM\|CVS\|~$\|\.o$')" | tar xzf -
> tar: Cowardly refusing to create an empty archive
> Try `tar --help' or `tar --usage' for more information.
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> $ find . -type f | wc -l
> 0
> $ popd
So it seems the hypothesis fails--no? Thanks anyway.
your assistance is appreciated! Tom Roche <Tom_Roche at pobox.com>
More information about the Trilug-ontopic
mailing list