[TriLUG] file copy flushing memory

[Corey Mutter]

On Sat, Dec 03, 2005 at 06:49:20AM -0800, Joseph Mack NA3T wrote:
> I don't see why a copy need flush all of memory by default. 
> A copy only needs enough memory (which can't be a whole lot) 
> and not ask for any more.

It's probably not using memory in the normal way; I belive cp uses 
mmap() to get a speedup. 

mmap() maps memory space to a file, in exactly the same way that the 
virtual memory subsystem uses disk blocks to back up pages of RAM. cp 
(sometimes? always?) mmap's both the source and destination files, and 
simply copies bytes from the source memory to the destination memory. 
The (highly optimized) virtual memory subsystem takes care of the disk 
reading and writing. 

Maybe this is taxing the VM subsystem so much that its other duties 
(allocating and swapping RAM) are being neglected. Alternatively, maybe 
for huge files it's mapping most/all of the (1GB? 2GB?) address space 
for itself, causing slowness as there's contention for the remaining 
space. 

The only solutions I could offer, though, are ones already presented: 
using dd (you'd probably want bs=1, unless the file is a known size, to 
prevent padding it out to a whole block), or writing a quick program / 
Perl script to read() to a fixed-size buffer, then write() from it. 

Corey