[TriLUG] Failure Copying Large Amounts of Data

[Steve Litt]

Depends on how the software is written. Typically it's a loop that copies the 
source to a memory buffer, and then writes from the memory buffer. If the 
software, for some reason, malloc()ed a file-sized buffer for the read and the 
write, then it would be 28GB of RAM, which is more than my 24GB swapfile. My 
experience tells me that upsizing the buffer improves things only to maybe a 
half a megabyte, but who knows, on a 64bit machine a 14GB buffer is possible.

STeveT

On Friday 16 April 2010 16:36:23 Lance A. Brown wrote:
> This is a boondoggle.  Unless I don't know something about linux swap
> space, it's not used when moving files around.  It's used to store pages
> from memory when a process gets swapped out to run another program.
> 
> BTW,  that's a LOT of swap space for a linux machine. :-)
> 
> Steve Litt wrote:
> > Fascinating! I have 24GB of swap, and the total of the 9 big files is
> > 14GB. So if somehow two different copies, one for src and one for dst, of
> > all 14GB were stored in swap, that would have indeed overloaded swap. The
> > diagnostic test to rule that in or out would be to make a shellscript
> > that copies one file at a time and blocks til the copy is complete.
> > Presumably the swap would be released after completion of a copy.
>