[TriLUG] Growing OS cache in RAM causing high system CPU usage

[Daniel Sterling]

Also check out admin_reserve_kbytes and min_free_kbytes tunables
documented at https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Possibly other tunables too like vfs_cache_pressure and overcommit
related tunables

Always be tuning

-- Dan

On Tue, Jul 8, 2014 at 2:54 PM, Daniel Sterling
<sterling.daniel at gmail.com> wrote:
> I didn't expect this, but amazingly it appears you can use cgroups to
> limit page cache usage.
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/cgroups/memory.txt?id=HEAD
>
> Try setting up a cgroup and use accounting to limit the RAM available
> to the processes
>
> -- Dan
>
>
>
> On Tue, Jul 8, 2014 at 1:40 PM, Michael Peters
> <michael00peters at gmail.com> wrote:
>> On Tue, Jul 8, 2014 at 1:28 PM, Bill Farrow <bill at arrowsreach.com> wrote:
>>
>>> This process is "seamless" to your application but there is still
>>> going to be a CPU overhead involved here.
>>
>> I know there is some overhead, but this seems really excessive. 8 CPUs
>> pegged 96-99% for 3 minutes?
>>
>>> The memory pages being used
>>> for OS cache need to be flushed from the CPU caches, and then possibly
>>> written out to disk if they are dirty (have changes in them).
>>
>> There's only a tiny amount of dirty memory in the cache (2060 kB0 and
>> no increase in disk IO during these CPU spikes.
>>
>>> Then
>>> they need to be marked as unused and the OS table/list/hash etc needs
>>> to be updated accordingly.
>>>
>>> Once the memory is "unused", it probably has to be initialized to zero
>>> before being handed to the application to prevent data leakage.
>>>
>>> So it is not CPU "free" even if it is seamless to your application.
>>
>> So given that the memory management overhead of the OS using too much
>> cache seems to be causing this how can I limit that? I don't see
>> anything in the virtual memory settings to control this. And while I
>> can force the cache to be cleared with /proc/sys/vm/drop_caches that's
>> a pretty blunt tool that clears everything and will impact performance
>> as the OS tries to refill the most needed things. Is there a way to
>> clear out the older parts of the cache explicitly?
>> --
>> This message was sent to: Daniel S. Sterling <sterling.daniel at gmail.com>
>> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
>> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/sterling.daniel%40gmail.com
>> Welcome to TriLUG: http://trilug.org/welcome