[TriLUG] OT: Disinfecting a Club's Website

[Scott Chilcote]

Hello LUGers,

I belong to a recreational club in the triangle that has had a
(primarily HTML) web site for going on two decades.  Believe it or not
it is still well used and has quite an archive of past articles of interest.

As you'd expect it has accumulated hundreds of hand edited HTML files,
and after many attempts no one has managed to successfully update it to
use anything more current.  Why?  Unpaid volunteers, who have full time
jobs, and who joined the organization to get away from doing this kind
of work.  Yeah, like me.  The other is that the pile just keeps getting
bigger.

The service provider is equally old, very cheap, and is using FreeBSD. 
The server's kernel was last updated in February 2011.  The version of
apache on this server is just as old:  Apache 2.0.64, built January 2011. 

Starting to seem rather hackable, no?

A couple of weeks ago, the thinkable happened.  Every one of the club's
several dozen web pages in every directory got a javascript script
tacked onto the end of a closing tag (typically < /div >), somewhere. 
It was a packed javascript that, when unpacked, did some harmless
looking stuff but called a PHP script on a site that appears to be
someone's personal webserver in Tokyo.  This triggered a lot of people's
web site trojan detectors, which said stuff like "JS:Iframe-CSU (Trojan)
- infected".

After I found out about it, I logged into the club's account and looked
at the site.  Sure enough, every HTML file in every directory -
including backups and saves, had been updated within a minute or so a
few days before.  The same javascript trojan was inserted into all of
them. 

I did a bunch of grepping around to see if I could find any
script-related breadcrumbs, but had no success.  Only the club's web
page files had been altered.  The account's home directory files were as
old as ever, and appeared unchanged.  I spent a while reviewing the
shell history files, and found nothing unexpected. Most of the recent
stuff they contained was from my previous logins.

The only positive aspect to this attack was that every Javascript
injection occurred between the same two comment markers, which was a
hexidecimal number:  "<!--d04bb5-->".  This made it rather easy to write
a recursive script to edit the HTML files in-place and erase the
infection, which I did.  I backed up the infected files (tarball) first,
and then made another tarball of the cleaned up files.  It seemed likely
that I would need them, as the barn door was still wide open.

Without having anything better than user grade access to this server, I
feel at a loss to do much more in trying to correct this problem. 

Here's a few of the things I did:

- Looked at every script under public_html and removed the stuff no one
is using.  At this point, the most sophisticated script on there
combines strings to create email addresses (a poor-man's effort to keep
them from being scraped).  There's only two or three left.  One of them
shows the date that a page was last edited (humorously enough, the
infection made them all far more current).  There are a couple of forms
that record stuff like items for sale and membership applications, which
gets emailed.

- Looked at the stuff in /tmp, which is one of the few system
directories visible to the club's account.  Nothing in there looks
recent enough to be infection related, other than some application cache
files that seem normal.

Two days ago, a nearly identical attack occurred.  The only thing that
differed was that a different script was injected into every HTML page
file.  This time the script was encoded as a very long string of
integers, so I have no way to tell what it's doing.  The javascript
unpacking sites that I tried can't do anything with it.

Once again it was wedged between hexidecimal numbered comments, so I was
able to use the same script to clean it out.  But short of camping out
on the club's site looking for reoccurrences, this is a temporary fix.

As much as a PITA as it sounds, I don't see much of a solution other
than moving to another ISP.  The club wanted "cheap" though, and that's
what they got.  $10 a month I think, paid a year in advance.  I have
zero time to shepard a web site move, my job's going full tilt for the
next few months.

Does anyone have a better idea?  I want to avoid getting the club's
pages blacklisted.  I'm very curious whether any other users on the
hosting service have gotten hacked the same way, but I don't know of a
way to find them.  Their directories are unreadable from the club's account.

Thanks for any clues!

Scott C  - reluctant volunteer


-- 
Scott Chilcote
Cary, NC USA
scottchilcote at att.net