[TriLUG] Building a beowulf (or other) cluster

[Ron Kelley]

For running parallel jobs on the same server, I use the GNU "parallel" utility.  Very, very nice utility that has saved us a bunch of time (compressing thousands of txt files, computing lots of data in parallel, etc).  Especially handy for dual 6-core CPU servers :-)


-Ron




On Mar 28, 2011, at 5:13 PM, Joel J. Adamson wrote:

> Justis Peters <jtrilug at indythinker.com> writes:
> 
>> On 03/28/2011 11:07 AM, Joseph Mack NA3T wrote:
>>> On Mon, 28 Mar 2011, Ron Kelley wrote:
>>> 
>>>> I would like to install some sort of distributed process management
>>>> tool so we can enable N-nodes to run the computations
>>>> simultaneously.
>>> 
>>> You only use a beowulf if the job cannot fit inside a single
>>> machine/node. This usually means that the job needs more memory than
>>> a single node holds. If this is your situation, you then recode the
>>> app to use the nodes in parallel. This usually means using mpi or
>>> omp.
>>> 
>>> If each job can be run in a single node, then you need a job farm
>>> (lots of machines with a job dispatcher).
>> Ron,
>> 
>> I agree with Joe's take on your issue. You said, "Our processing
>> happens in batch jobs and can easily be run on multiple servers at the
>> same time." That sounds like an "embarrassingly parallel workload"
>> (http://en.wikipedia.org/wiki/Embarrassingly_parallel), which is good
>> news.
> 
> Good news, yes: embarrassingly parallel doesn't mean you should be
> embarrassed.  I found for my own purposes that
> 
> make -j
> 
> did the trick.  In other words, parallel /jobs/ was what I needed,
> rather than parallel code (as with OpenMP).
> 
> Joel
> 
> -- 
> Joel J. Adamson
> Servedio Lab
> University of North Carolina at Chapel Hill
> 
> FSF Member #8164
> http://www.unc.edu/~adamsonj
> -- 
> This message was sent to: Ron Kelley <rkelleyrtp at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web	: http://www.trilug.org/mailman/options/trilug/rkelleyrtp%40gmail.com
> TriLUG FAQ          : http://www.trilug.org/wiki/Frequently_Asked_Questions