[TriLUG] System overload issues

[Ron Kelley]

Hi Brian,

The rsync and gzip issues smell like bad drives (or, very high I/O).  I suggest to download the "nmon" binary for CentOS 5.6 (http://nmon.sourceforge.net/pmwiki.php?n=Site.Download and look for the NMON binaries for CentOS 5).  The direct URL is http://docs/MPG_nmon_for_Linux_14a_binaries.zip

Start nmon then type "ncd" to get network, CPU, and disk stats.  Let it run for a while and see exactly which areas are affected.  Depending on your terminal type, you might need to use this command:  "export TERM=xterm && /usr/bin/nmon" (or, specify the right location for the nmon binary.


Also, can you share a portion of the /var/log/messages file?  That should help uncover some potential items...



Thanks,


-Ron

On May 24, 2013, at 9:34 AM, Brian McCullough wrote:

> Once again, I come to the oracle for help.  A while ago, I asked about
> troubleshooting and tuning MySQL and got some very helpful answers. I
> implemented some of those answers, as well as continuing my own
> research, and made some changes to the configuration.  However, the
> machine is continuing to present very troubling symptoms, and so I am
> back.  As you may know, I am much more of a programmer than a high-level
> System Administrator, and this seems to be way beyond my pay grade ( mostly zero ).
> 
> 
> The story. ( I also have a different issue that I will show in a
> different thread. )
> 
> 
> I am having some serious issues with this machine, and hoping that you can offer some pointers. 
> 
> As far as I can see, this is an 8-CPU machine with 8 Gigs of RAM, and 2 RAID 1 arrays. It is running CentOS 5.6, MySQL 5.0.77 and Apache 2.2.3. 
> 
> The ongoing issue is one of system load and non-responsiveness.
> 
> Frequently during the day, the system will become ( or the web sites will become ) non-responsive for periods ranging from one minute to well over an hour. 
> 
> Through DNS trickery, approximately 1,800 domain names are being served from this web site, all data driven. 
> 
> The original symptoms involved MySQL showing a CPU load of approximately 800% in "top," along with a system 1, 5 and 15 minute load of over 200.
> 
> Since performing various tuning exercises on MySQL, the load there seems to be more reasonable, occasionally popping up over 80%, but usually showing values close to zero. 
> 
> Now, other things seem to be showing failure symtoms; for instance, bzip2, which compresses the MySQL database backup seems to take hours instead of minutes; rsync, which is transferring image files from the Production machine to the
> Backup machine, 
> 
> Again, overall system load is over 150, and the system was non-responsive for 6.5 hours yesterday morning. 
> 
> 
> Can you make any suggestions or do you have any questions for me to investigate?
> 
> 
> Thank you,
> Brian
> 
> -- 
> This message was sent to: Ron Kelley <rkelleyrtp at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web	: http://www.trilug.org/mailman/options/trilug/rkelleyrtp%40gmail.com
> TriLUG FAQ          : http://www.trilug.org/wiki/Frequently_Asked_Questions
> TriLUG is dedicated to a harassment-free experience for everyone. Our anti-harassment policy can be found at: http://trilug.org/anti-harassment