[TriLUG] Growing OS cache in RAM causing high system CPU usage

[Michael Peters]

On Wed, Jul 9, 2014 at 1:54 PM, leonard boyle <lenboyle at msn.com> wrote:

> Interesting
> The monitoring tool shows system cpu peaking up towards 100%, but vmstat only peaks at 42 percent. So I assume the monitoring tool is using a smaller sampling period. The idle column of vmstat only goes down to 31 percent.

No, vmstat shows the system CPU going up to 99 and stays in the high
90s for a while. I think you're looking at the wrong column.

> Also note the increase in the number of processes under the r column of vmstat at the peak of system cpu usage.
> It would be interesting to know what those processes are.

They are PHP nginx processes. They spike up not because of an increase
in traffic, but because when the CPUs spike up to 99% existing
processes can't finish their requests so php-fpm spins up new ones to
handler incoming requests. It's not the disease but of symptom of what
happens.

> There does not appear to be any blocked processes.

They aren't blocked on io, but yes they are blocked on CPU.

> The big question is not that the cpu goes to 100%, but does it  impact application performance. As long as the processes using the cpu are running with a low priority and will be replaced by more critical system and application processes it is ok.

Yes it definitely impacts performance. Everything grinds to a halt
while the system CPU jumps through the rough for about 3 minutes.

> Have you run the free -m command to gather more info on the cache/free  space?

Yes, but vmstat gives a lot more info than free does.

> Any chance that there are numa effects on this system?

Nope, not NUMA. If you notice in the posting on server-fault it has a
couple of relevant comments:

* And I'm not sure about the zone reclaim problem because it seems
that NUMA is off (grep -i 'numa' /var/log/dmesg gives [0.000000] NUMA
turned off
* Also, numactl --hardware reports just a single node "0", so I doubt
it's NUMA memory balancing problems.

I'm pretty sure it's the system trying to recover OS cache from RAM to
give to the application. I just didn't expect the overhead to be so
drastic (and not just happen in the background before it gets
critical).

>> Date: Tue, 8 Jul 2014 15:02:01 -0400
>> From: sterling.daniel at gmail.com
>> To: trilug at trilug.org
>> Subject: Re: [TriLUG] Growing OS cache in RAM causing high system CPU usage
>>
>> Also check out admin_reserve_kbytes and min_free_kbytes tunables
>> documented at https://www.kernel.org/doc/Documentation/sysctl/vm.txt
>>
>> Possibly other tunables too like vfs_cache_pressure and overcommit
>> related tunables
>>
>> Always be tuning
>>
>> -- Dan
>>
>> On Tue, Jul 8, 2014 at 2:54 PM, Daniel Sterling
>> <sterling.daniel at gmail.com> wrote:
>> > I didn't expect this, but amazingly it appears you can use cgroups to
>> > limit page cache usage.
>> >
>> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/cgroups/memory.txt?id=HEAD
>> >
>> > Try setting up a cgroup and use accounting to limit the RAM available
>> > to the processes
>> >
>> > -- Dan
>> >
>> >
>> >
>> > On Tue, Jul 8, 2014 at 1:40 PM, Michael Peters
>> > <michael00peters at gmail.com> wrote:
>> >> On Tue, Jul 8, 2014 at 1:28 PM, Bill Farrow <bill at arrowsreach.com> wrote:
>> >>
>> >>> This process is "seamless" to your application but there is still
>> >>> going to be a CPU overhead involved here.
>> >>
>> >> I know there is some overhead, but this seems really excessive. 8 CPUs
>> >> pegged 96-99% for 3 minutes?
>> >>
>> >>> The memory pages being used
>> >>> for OS cache need to be flushed from the CPU caches, and then possibly
>> >>> written out to disk if they are dirty (have changes in them).
>> >>
>> >> There's only a tiny amount of dirty memory in the cache (2060 kB0 and
>> >> no increase in disk IO during these CPU spikes.
>> >>
>> >>> Then
>> >>> they need to be marked as unused and the OS table/list/hash etc needs
>> >>> to be updated accordingly.
>> >>>
>> >>> Once the memory is "unused", it probably has to be initialized to zero
>> >>> before being handed to the application to prevent data leakage.
>> >>>
>> >>> So it is not CPU "free" even if it is seamless to your application.
>> >>
>> >> So given that the memory management overhead of the OS using too much
>> >> cache seems to be causing this how can I limit that? I don't see
>> >> anything in the virtual memory settings to control this. And while I
>> >> can force the cache to be cleared with /proc/sys/vm/drop_caches that's
>> >> a pretty blunt tool that clears everything and will impact performance
>> >> as the OS tries to refill the most needed things. Is there a way to
>> >> clear out the older parts of the cache explicitly?
>> >> --
>> >> This message was sent to: Daniel S. Sterling <sterling.daniel at gmail.com>
>> >> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
>> >> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>> >> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/sterling.daniel%40gmail.com
>> >> Welcome to TriLUG: http://trilug.org/welcome
>> --
>> This message was sent to: len <lenboyle at msn.com>
>> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
>> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>> Unsubscribe or edit options on the web        : http://www.trilug.org/mailman/options/trilug/lenboyle%40msn.com
>> Welcome to TriLUG: http://trilug.org/welcome
>
>
> --
> This message was sent to: Michael Peters <michael00peters at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/michael00peters%40gmail.com
> Welcome to TriLUG: http://trilug.org/welcome