[TriLUG] Failure Copying Large Amounts of Data

[Steve Litt]

Someone suggested it might be too little swap, and so I just explored the 
possibility.

SteveT

On Friday 16 April 2010 22:19:55 Rodney Radford wrote:
> No reasonable file copy program would work like you describe.  Even if it
>  malloc'ed a space for the entire file and read it all in memory, there
>  would only be need for one copy.  However, I have never heard of a file
>  copy that would malloc that much space, putting it out to swap, as that
>  would require just additional disk reads/writes.
> 
> Perhaps you are confusing the idea that many systems will allow the kernel
>  to continue to allocate in memory buffers for the input and output buffers
>  so that portions of the file can be read/written in larger quantities (ie:
>  track at a time, or for RAID an entire stripe at a time, etc). But it
>  would only use 'free' or available memory, not pushing it out to swap, and
>  would never allocate sizes such as you suggest (2x 14GB file size).
> 
> 
> -----Original Message-----
> 
> >From: Steve Litt <slitt at troubleshooters.com>
> >Sent: Apr 16, 2010 7:03 PM
> >To: Triangle Linux Users Group General Discussion <trilug at trilug.org>
> >Subject: Re: [TriLUG] Failure Copying Large Amounts of Data
> >
> >Depends on how the software is written. Typically it's a loop that copies
> > the source to a memory buffer, and then writes from the memory buffer. If
> > the software, for some reason, malloc()ed a file-sized buffer for the
> > read and the write, then it would be 28GB of RAM, which is more than my
> > 24GB swapfile. My experience tells me that upsizing the buffer improves
> > things only to maybe a half a megabyte, but who knows, on a 64bit machine
> > a 14GB buffer is possible.
> >
> >STeveT
> >
> >On Friday 16 April 2010 16:36:23 Lance A. Brown wrote:
> >> This is a boondoggle.  Unless I don't know something about linux swap
> >> space, it's not used when moving files around.  It's used to store pages
> >> from memory when a process gets swapped out to run another program.
> >>
> >> BTW,  that's a LOT of swap space for a linux machine. :-)
> >>
> >> Steve Litt wrote:
> >> > Fascinating! I have 24GB of swap, and the total of the 9 big files is
> >> > 14GB. So if somehow two different copies, one for src and one for dst,
> >> > of all 14GB were stored in swap, that would have indeed overloaded
> >> > swap. The diagnostic test to rule that in or out would be to make a
> >> > shellscript that copies one file at a time and blocks til the copy is
> >> > complete. Presumably the swap would be released after completion of a
> >> > copy.
>