[TriLUG] System overload issues

[Brian McCullough]

Once again, I come to the oracle for help.  A while ago, I asked about
troubleshooting and tuning MySQL and got some very helpful answers. I
implemented some of those answers, as well as continuing my own
research, and made some changes to the configuration.  However, the
machine is continuing to present very troubling symptoms, and so I am
back.  As you may know, I am much more of a programmer than a high-level
System Administrator, and this seems to be way beyond my pay grade ( mostly zero ).


The story. ( I also have a different issue that I will show in a
different thread. )


I am having some serious issues with this machine, and hoping that you can offer some pointers. 
 
As far as I can see, this is an 8-CPU machine with 8 Gigs of RAM, and 2 RAID 1 arrays. It is running CentOS 5.6, MySQL 5.0.77 and Apache 2.2.3. 
 
The ongoing issue is one of system load and non-responsiveness.

Frequently during the day, the system will become ( or the web sites will become ) non-responsive for periods ranging from one minute to well over an hour. 
 
Through DNS trickery, approximately 1,800 domain names are being served from this web site, all data driven. 

The original symptoms involved MySQL showing a CPU load of approximately 800% in "top," along with a system 1, 5 and 15 minute load of over 200.

Since performing various tuning exercises on MySQL, the load there seems to be more reasonable, occasionally popping up over 80%, but usually showing values close to zero. 
 
Now, other things seem to be showing failure symtoms; for instance, bzip2, which compresses the MySQL database backup seems to take hours instead of minutes; rsync, which is transferring image files from the Production machine to the
 Backup machine, 
 
Again, overall system load is over 150, and the system was non-responsive for 6.5 hours yesterday morning. 


Can you make any suggestions or do you have any questions for me to investigate?


Thank you,
Brian