[TriLUG] System overload issues
Brian McCullough
bdmc at buadh-brath.com
Fri May 24 10:50:39 EDT 2013
On Fri, May 24, 2013 at 07:35:53AM -0700, John Vaughters wrote:
> Brian,
>
> I highly recommend reading up on sysstat and sar command. This gives you quick system stats that will point you to many problems and when they occur. With that info you can narrow in on your other logs for further information. You can find out about CPU, MEM, SWAP, I/O waits (disk), NIC. Centos usually defaults to 7 days of stats, but yo can increase that if you like.
Yes, I have used sar when I work on AIX machines, and have also tried it
on this machine, back when I first starting trying to diagnose this
issue. ( these issues? )
> My favorite sar command to view yesterday's stats:
>
> sar -f /var/log/sa/sa$(date +%d -d yesterday)
OK. Part of the frustration is that when I do so, things LOOK all
right. For instance:
# sar -f /var/log/sa/sa$(date +%d -d yesterday)
Linux 2.6.18-238.19.1.el5 (hostname) 05/23/2013
12:00:06 AM CPU %user %nice %system %iowait %steal %idle
12:10:05 AM all 55.26 0.00 44.71 0.00 0.00 0.04
12:20:01 AM all 55.73 0.00 44.09 0.06 0.00 0.12
12:30:02 AM all 57.24 0.00 42.32 0.07 0.00 0.38
12:40:30 AM all 48.63 0.00 40.60 6.45 0.00 4.32
01:10:12 AM all 27.86 0.00 69.94 1.58 0.00 0.62
01:20:05 AM all 55.80 0.00 44.08 0.02 0.00 0.10
01:30:04 AM all 55.76 0.00 44.19 0.00 0.00 0.05
01:40:02 AM all 55.79 0.00 44.17 0.00 0.00 0.04
01:50:01 AM all 41.59 0.00 31.48 3.79 0.00 23.14
02:00:02 AM all 39.43 0.00 28.29 10.35 0.00 21.93
02:10:02 AM all 54.48 0.00 45.47 0.01 0.00 0.04
02:30:26 AM all 26.78 0.00 25.28 47.72 0.00 0.22
02:40:24 AM all 16.37 0.00 11.03 71.29 0.00 1.31
02:50:16 AM all 16.46 0.00 10.94 72.60 0.00 0.00
03:02:20 AM all 28.06 0.00 19.90 51.99 0.00 0.04
03:10:28 AM all 52.36 0.00 39.77 7.87 0.00 0.00
03:20:15 AM all 56.13 0.00 43.83 0.01 0.00 0.03
03:30:06 AM all 56.08 0.00 43.88 0.00 0.00 0.04
03:40:02 AM all 56.04 0.00 43.93 0.00 0.00 0.03
03:50:10 AM all 56.13 0.00 43.83 0.00 0.00 0.03
04:00:48 AM all 55.05 0.00 42.90 1.72 0.00 0.33
04:10:03 AM all 13.40 0.07 7.93 74.43 0.00 4.17
04:20:10 AM all 44.25 0.02 38.17 9.07 0.00 8.50
05:01:08 AM all 0.76 0.00 87.15 11.59 0.00 0.50
05:10:49 AM all 1.02 0.00 80.80 17.42 0.00 0.76
05:20:47 AM all 0.92 0.00 77.51 20.34 0.00 1.23
05:41:02 AM all 0.62 0.00 81.72 16.85 0.00 0.81
05:51:17 AM all 0.52 0.00 83.59 15.23 0.00 0.67
06:00:43 AM all 0.60 0.00 76.22 21.63 0.00 1.54
06:10:22 AM all 0.85 0.01 68.45 27.38 0.00 3.31
06:20:05 AM all 2.05 0.01 44.17 32.96 0.00 20.83
06:30:18 AM all 0.20 0.00 1.20 45.94 0.00 52.66
06:41:15 AM all 2.76 0.02 3.88 42.90 0.00 50.45
06:50:11 AM all 5.33 0.00 4.45 66.51 0.00 23.70
07:00:32 AM all 16.85 0.00 13.94 63.53 0.00 5.67
07:10:25 AM all 27.87 0.01 21.85 48.47 0.00 1.80
07:20:02 AM all 50.65 0.04 41.06 7.03 0.00 1.22
07:30:16 AM all 32.75 0.02 24.72 42.49 0.00 0.03
07:40:05 AM all 55.82 0.01 44.13 0.01 0.00 0.03
07:53:59 AM all 14.12 0.01 12.92 72.34 0.00 0.61
08:01:57 AM all 10.90 0.00 9.14 79.76 0.00 0.19
08:10:07 AM all 10.60 0.01 8.74 80.10 0.00 0.55
08:10:07 AM CPU %user %nice %system %iowait %steal %idle
08:20:01 AM all 27.38 0.05 20.27 43.95 0.00 8.35
08:30:01 AM all 44.07 0.11 27.82 3.49 0.00 24.51
08:40:01 AM all 18.56 0.30 4.03 6.41 0.00 70.70
08:50:01 AM all 12.77 0.28 4.43 2.72 0.00 79.79
09:00:01 AM all 12.56 0.15 4.14 2.50 0.00 80.65
09:10:01 AM all 13.37 0.08 4.83 1.36 0.00 80.36
09:20:01 AM all 14.44 0.05 4.96 1.34 0.00 79.21
09:30:01 AM all 12.91 0.07 4.24 1.11 0.00 81.68
09:40:01 AM all 12.35 0.35 4.08 2.16 0.00 81.06
09:50:01 AM all 13.38 0.04 4.60 0.84 0.00 81.15
10:00:01 AM all 13.55 0.04 4.47 1.02 0.00 80.92
10:10:01 AM all 14.18 0.04 4.57 1.07 0.00 80.13
10:20:01 AM all 14.59 0.05 4.93 1.07 0.00 79.36
10:30:02 AM all 16.64 0.04 5.95 1.01 0.00 76.36
10:40:01 AM all 14.47 0.27 5.20 1.07 0.00 78.98
10:50:01 AM all 13.61 0.24 4.61 1.40 0.00 80.13
11:00:01 AM all 17.15 0.27 6.23 2.14 0.00 74.22
11:10:01 AM all 17.21 0.25 6.07 2.21 0.00 74.26
11:20:01 AM all 16.97 0.23 6.06 1.96 0.00 74.79
11:30:01 AM all 18.95 0.21 6.65 1.75 0.00 72.44
11:40:01 AM all 19.58 0.22 7.56 2.06 0.00 70.58
11:50:01 AM all 19.09 0.22 7.71 1.77 0.00 71.21
12:00:01 PM all 18.57 0.26 6.84 2.48 0.00 71.84
12:10:01 PM all 16.12 0.27 5.24 1.93 0.00 76.43
12:20:01 PM all 16.12 0.26 5.27 1.69 0.00 76.66
12:30:01 PM all 24.32 0.17 11.95 0.99 0.00 62.57
12:40:01 PM all 27.39 0.14 16.14 0.88 0.00 55.44
12:50:01 PM all 16.86 0.21 5.70 1.96 0.00 75.27
01:00:01 PM all 16.04 0.23 5.62 1.75 0.00 76.36
01:10:01 PM all 16.41 0.24 5.19 1.75 0.00 76.40
01:20:01 PM all 15.40 0.28 4.84 1.96 0.00 77.52
01:30:01 PM all 17.04 0.24 5.61 1.58 0.00 75.55
01:40:02 PM all 18.67 0.23 6.05 1.65 0.00 73.40
01:50:01 PM all 15.38 0.23 5.03 1.63 0.00 77.73
02:00:01 PM all 14.66 0.21 4.87 1.99 0.00 78.27
02:10:01 PM all 15.56 0.16 4.87 1.78 0.00 77.63
02:20:01 PM all 15.60 0.23 4.93 1.54 0.00 77.70
02:30:01 PM all 17.98 0.24 6.28 1.54 0.00 73.97
02:40:01 PM all 18.10 0.18 6.30 1.14 0.00 74.28
02:50:01 PM all 14.98 0.26 5.26 1.84 0.00 77.67
03:00:01 PM all 13.61 0.24 4.65 1.00 0.00 80.50
03:10:01 PM all 13.93 0.20 4.59 0.94 0.00 80.34
03:10:01 PM CPU %user %nice %system %iowait %steal %idle
03:20:01 PM all 14.10 0.21 4.99 0.84 0.00 79.85
03:30:01 PM all 14.45 0.22 4.95 0.67 0.00 79.71
03:40:01 PM all 14.91 0.21 5.29 0.86 0.00 78.73
03:50:01 PM all 14.12 0.23 4.93 0.84 0.00 79.88
04:00:01 PM all 8.31 0.02 3.47 0.83 0.00 87.37
04:10:01 PM all 4.06 0.00 1.38 0.60 0.00 93.96
04:20:01 PM all 5.50 0.00 2.00 0.57 0.00 91.92
04:30:01 PM all 7.49 0.00 2.65 0.58 0.00 89.28
04:40:01 PM all 10.55 0.00 3.95 0.60 0.00 84.90
04:50:01 PM all 3.65 0.00 1.10 0.54 0.00 94.71
05:00:01 PM all 4.76 0.00 1.71 0.56 0.00 92.97
05:10:01 PM all 8.07 0.00 2.79 0.82 0.00 88.32
05:20:01 PM all 6.52 0.00 2.51 0.63 0.00 90.34
05:30:01 PM all 4.30 0.00 1.32 0.50 0.00 93.89
05:40:01 PM all 11.39 0.00 5.04 0.41 0.00 83.16
05:50:01 PM all 5.57 0.00 1.74 0.60 0.00 92.08
06:00:01 PM all 7.28 0.00 2.20 0.62 0.00 89.89
06:10:01 PM all 3.36 0.00 1.00 0.59 0.00 95.04
06:20:01 PM all 4.01 0.00 1.25 0.60 0.00 94.13
06:30:02 PM all 4.43 0.00 1.39 0.48 0.00 93.69
06:40:02 PM all 5.53 0.00 1.82 0.54 0.00 92.12
06:50:01 PM all 4.16 0.00 1.59 0.55 0.00 93.70
07:00:01 PM all 3.17 0.00 1.03 0.47 0.00 95.33
07:10:01 PM all 3.92 0.00 1.39 0.70 0.00 93.99
07:20:01 PM all 3.31 0.00 0.98 0.55 0.00 95.15
07:30:01 PM all 3.89 0.00 1.25 0.55 0.00 94.31
07:40:01 PM all 7.95 0.00 2.84 0.60 0.00 88.60
07:50:01 PM all 6.17 0.00 2.23 0.47 0.00 91.13
08:00:01 PM all 3.96 0.00 1.18 0.61 0.00 94.25
08:10:01 PM all 5.08 0.00 1.60 0.74 0.00 92.56
08:20:01 PM all 4.05 0.00 1.19 0.62 0.00 94.13
08:30:01 PM all 4.31 0.00 1.52 0.62 0.00 93.54
08:40:01 PM all 13.90 0.00 9.08 0.46 0.00 76.56
08:50:01 PM all 36.79 0.00 29.68 0.24 0.00 33.29
09:00:01 PM all 4.05 0.00 1.33 0.80 0.00 93.83
09:10:01 PM all 5.46 0.00 1.84 0.77 0.00 91.94
09:20:01 PM all 4.31 0.00 1.64 0.64 0.00 93.40
09:30:01 PM all 3.64 0.00 1.26 0.81 0.00 94.29
09:40:01 PM all 5.50 0.00 2.13 0.72 0.00 91.64
09:50:01 PM all 7.55 0.00 2.73 0.74 0.00 88.98
10:00:01 PM all 4.55 0.00 1.63 0.90 0.00 92.91
10:10:01 PM all 4.14 0.00 1.45 0.64 0.00 93.77
10:10:01 PM CPU %user %nice %system %iowait %steal %idle
10:20:01 PM all 5.71 0.00 2.01 0.51 0.00 91.78
10:30:01 PM all 3.50 0.00 1.23 0.56 0.00 94.70
10:40:01 PM all 3.58 0.00 1.27 0.50 0.00 94.65
10:50:01 PM all 4.66 0.00 1.72 0.58 0.00 93.04
11:00:01 PM all 4.09 0.00 1.45 0.46 0.00 94.00
11:10:01 PM all 5.72 0.00 1.88 0.45 0.00 91.95
11:20:01 PM all 2.79 0.00 1.01 0.42 0.00 95.78
11:30:01 PM all 3.66 0.00 1.34 0.34 0.00 94.67
11:40:01 PM all 3.66 0.00 1.53 0.34 0.00 94.47
11:50:01 PM all 4.37 0.00 1.53 0.41 0.00 93.69
Average: all 16.83 0.06 17.93 9.24 0.00 55.93
This is the day where the web sites were basically non-responsive from
some time between 1am and 2am until about 8:30am, and I do know that
both "w" and "top" were showing system loads of 200 or more.
Brian
More information about the TriLUG
mailing list