[TriLUG] nodes, zero byte files

[Scott Chilcote via TriLUG]

On 10/26/2015 02:52 PM, James Jones wrote:
> Scott
>
> ***** You mention that someone intruded into your VM and caused this
> problem to occur.
>
>
> The intruder left hacks in a couple of web pages and this is when the
> exceeded quota started. It is hard to suspect otherwise.
>
> I guess there could be a run-away application, but as an amateur in
> linux, would not know if this can happen.
>
> As root, I tried the following command in the major directories of the VM
>
> find ./ -type f -size 0b -delete
>
> A large number of zero bytes files found in /proc but files could not
> be delete. Permission denied.???
>
> Another directory /usr had large amounts. When running the find
> command, result in this directory
>
> find: cannot delete `./share/******.txt': Disk quota exceeded '
>
> plus -- a re-run of df -ih shows:
>
> =============================================
>  df -ih
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/vzfs                16E     16E     49M  100% /
> none                    128K     158    128K    1% /dev
>
>
> =============================================
>
> As I mentioned earlier:
>
> df -ih
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/vzfs               4.8M    4.8M       0  100% /
> none                    128K     158    128K    1% /dev
> ----------------------------------------------------------------------------
>
> Yep! VM about to be destroyed.
>
>

Hi James,

You are not able to delete files in /proc because it is a virtual
filesystem.  It is created by the kernel at boot time.  The files that
it contains report that they have zero bytes, but their purpose is to
provide a way to look at information about the running system.  For
example, you can see what kind of CPU (or virtual CPU) you have by
listing the contents of /proc/cpuinfo.  You can think of it as an
unwriteable virtual ramdisk containing system information. 
There's more about this at
http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html

/usr/share is not likely to be the problem.  It's a repository of
documentation and related application files, many of which are place
holders or empty configuration files (e.g. __init__.py).  If it has
fewer than a hundred of these it's probably ok.

The disk quota exceeded error from find can happen when you use rm and
have no inodes left.

I still don't have a guess as to where the inodes went.  If they came in
through Apache, it's tempting to speculate that they used something like
the Linux/Cdorked attack that hit the news reports a couple of years
ago, e.g.
http://arstechnica.com/security/2013/04/admin-beware-attack-hitting-apache-websites-is-invisible-to-the-naked-eye/
Since that attack used a large chunk of shared memory to create a
virtual filesystem, it would likely have consumed a lot of inodes as a
result.  But that's just one of many possibilities.  I remember it
because it underscored how devious and motivated the system crackers
have become!

So keep those operating systems patched up to the minute, whatever they
are  ;-)

   Scott C.

-- 
Scott Chilcote
scottchilcote at ncrrbiz.com
Cary, NC USA