[TriLUG] Growing OS cache in RAM causing high system CPU usage

[Michael Peters]

Thanks for the pointers. Unfortunately since this is a legacy system
that is scheduled to be replaced I don't get to spend too much time
playing around with the settings.

As a quick workaround I've set cron to just drop the OS cache every 12
hours. This seems to work fine and hasn't added any additional impact
(yet). If that changes then I'll play around with these.

Thanks again,

On Tue, Jul 8, 2014 at 3:02 PM, Daniel Sterling
<sterling.daniel at gmail.com> wrote:
> Also check out admin_reserve_kbytes and min_free_kbytes tunables
> documented at https://www.kernel.org/doc/Documentation/sysctl/vm.txt
>
> Possibly other tunables too like vfs_cache_pressure and overcommit
> related tunables
>
> Always be tuning
>
> -- Dan
>
> On Tue, Jul 8, 2014 at 2:54 PM, Daniel Sterling
> <sterling.daniel at gmail.com> wrote:
>> I didn't expect this, but amazingly it appears you can use cgroups to
>> limit page cache usage.
>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/cgroups/memory.txt?id=HEAD
>>
>> Try setting up a cgroup and use accounting to limit the RAM available
>> to the processes
>>
>> -- Dan
>>
>>
>>
>> On Tue, Jul 8, 2014 at 1:40 PM, Michael Peters
>> <michael00peters at gmail.com> wrote:
>>> On Tue, Jul 8, 2014 at 1:28 PM, Bill Farrow <bill at arrowsreach.com> wrote:
>>>
>>>> This process is "seamless" to your application but there is still
>>>> going to be a CPU overhead involved here.
>>>
>>> I know there is some overhead, but this seems really excessive. 8 CPUs
>>> pegged 96-99% for 3 minutes?
>>>
>>>> The memory pages being used
>>>> for OS cache need to be flushed from the CPU caches, and then possibly
>>>> written out to disk if they are dirty (have changes in them).
>>>
>>> There's only a tiny amount of dirty memory in the cache (2060 kB0 and
>>> no increase in disk IO during these CPU spikes.
>>>
>>>> Then
>>>> they need to be marked as unused and the OS table/list/hash etc needs
>>>> to be updated accordingly.
>>>>
>>>> Once the memory is "unused", it probably has to be initialized to zero
>>>> before being handed to the application to prevent data leakage.
>>>>
>>>> So it is not CPU "free" even if it is seamless to your application.
>>>
>>> So given that the memory management overhead of the OS using too much
>>> cache seems to be causing this how can I limit that? I don't see
>>> anything in the virtual memory settings to control this. And while I
>>> can force the cache to be cleared with /proc/sys/vm/drop_caches that's
>>> a pretty blunt tool that clears everything and will impact performance
>>> as the OS tries to refill the most needed things. Is there a way to
>>> clear out the older parts of the cache explicitly?
>>> --
>>> This message was sent to: Daniel S. Sterling <sterling.daniel at gmail.com>
>>> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
>>> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
>>> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/sterling.daniel%40gmail.com
>>> Welcome to TriLUG: http://trilug.org/welcome
> --
> This message was sent to: Michael Peters <michael00peters at gmail.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/michael00peters%40gmail.com
> Welcome to TriLUG: http://trilug.org/welcome