[TriLUG] Growing OS cache in RAM causing high system CPU usage

[leonard boyle]

Interesting 
The monitoring tool shows system cpu peaking up towards 100%, but vmstat only peaks at 42 percent. So I assume the monitoring tool is using a smaller sampling period. The idle column of vmstat only goes down to 31 percent. 

Also note the increase in the number of processes under the r column of vmstat at the peak of system cpu usage. 
It would be interesting to know what those processes are. 
There does not appear to be any blocked processes.
The big question is not that the cpu goes to 100%, but does it  impact application performance. As long as the processes using the cpu are running with a low priority and will be replaced by more critical system and application processes it is ok. 
Have you run the free -m command to gather more info on the cache/free  space?
Any chance that there are numa effects on this system?




> Date: Tue, 8 Jul 2014 15:02:01 -0400
> From: sterling.daniel at gmail.com
> To: trilug at trilug.org
> Subject: Re: [TriLUG] Growing OS cache in RAM causing high system CPU usage
> 
> Also check out admin_reserve_kbytes and min_free_kbytes tunables
> documented at https://www.kernel.org/doc/Documentation/sysctl/vm.txt
> 
> Possibly other tunables too like vfs_cache_pressure and overcommit
> related tunables
> 
> Always be tuning
> 
> -- Dan
> 
> On Tue, Jul 8, 2014 at 2:54 PM, Daniel Sterling
> <sterling.daniel at gmail.com> wrote:
> > I didn't expect this, but amazingly it appears you can use cgroups to
> > limit page cache usage.
> >
> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/cgroups/memory.txt?id=HEAD
> >
> > Try setting up a cgroup and use accounting to limit the RAM available
> > to the processes
> >
> > -- Dan
> >
> >
> >
> > On Tue, Jul 8, 2014 at 1:40 PM, Michael Peters
> > <michael00peters at gmail.com> wrote:
> >> On Tue, Jul 8, 2014 at 1:28 PM, Bill Farrow <bill at arrowsreach.com> wrote:
> >>
> >>> This process is "seamless" to your application but there is still
> >>> going to be a CPU overhead involved here.
> >>
> >> I know there is some overhead, but this seems really excessive. 8 CPUs
> >> pegged 96-99% for 3 minutes?
> >>
> >>> The memory pages being used
> >>> for OS cache need to be flushed from the CPU caches, and then possibly
> >>> written out to disk if they are dirty (have changes in them).
> >>
> >> There's only a tiny amount of dirty memory in the cache (2060 kB0 and
> >> no increase in disk IO during these CPU spikes.
> >>
> >>> Then
> >>> they need to be marked as unused and the OS table/list/hash etc needs
> >>> to be updated accordingly.
> >>>
> >>> Once the memory is "unused", it probably has to be initialized to zero
> >>> before being handed to the application to prevent data leakage.
> >>>
> >>> So it is not CPU "free" even if it is seamless to your application.
> >>
> >> So given that the memory management overhead of the OS using too much
> >> cache seems to be causing this how can I limit that? I don't see
> >> anything in the virtual memory settings to control this. And while I
> >> can force the cache to be cleared with /proc/sys/vm/drop_caches that's
> >> a pretty blunt tool that clears everything and will impact performance
> >> as the OS tries to refill the most needed things. Is there a way to
> >> clear out the older parts of the cache explicitly?
> >> --
> >> This message was sent to: Daniel S. Sterling <sterling.daniel at gmail.com>
> >> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> >> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> >> Unsubscribe or edit options on the web  : http://www.trilug.org/mailman/options/trilug/sterling.daniel%40gmail.com
> >> Welcome to TriLUG: http://trilug.org/welcome
> -- 
> This message was sent to: len <lenboyle at msn.com>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web	: http://www.trilug.org/mailman/options/trilug/lenboyle%40msn.com
> Welcome to TriLUG: http://trilug.org/welcome