[Trilug-announce] trilug.org unplanned outage initial report

[Cristóbal Palmer]

I'm writing on behalf of the TriLUG Steering Committee, Systems
Subcommittee, and various volunteers (thanks Tanner!) to inform you of
an incident that first came to our attention Friday evening.

= Executive Summary =

* Unauthorized access of pilot.trilug.org caused an outage of that
hardware starting the evening of Friday 27th Feb 2009
* Out of an abundance of caution, we (Thanks Alan Porter!) have
re-imaged the machine.
* Some services (eg. the drupal-based main trilug.org site) will be
unavailable until volunteers can conduct a full post-mortem and safely
reinstate them.
* Announcement(s) will be sent to this list later this week as we follow up.
* If you need a password reset on your shell account, please email one
of these SC members:
    * alan -at- alanporter.com
    * john -at- ncphotography.com
* See http://trilug.org/ for contact info and updates as we have them.

= Full Initial Announcement =

On Friday evening pilot.trilug.org became unresponsive at about 19:00
EST. Kevin Otte and John Berninger were kind enough to drive out to
the DC to troubleshoot. They discovered that the machine had been
exploited and the exploit process was sending out so much traffic that
normal services (eg. bind, sshd) were unresponsive. Initial
investigation indicates the exploit began around 18:30 EST and was
clumsy, but we have yet to do a full post-mortem. Please hold your
questions and comments about this exploit until after we have managed
to conduct that inquiry and post a follow-up announcement.

We did not find evidence of privilege escalation, but out of an
abundance of caution it was decided by a group of volunteers including
two Steering Committee members (myself and John Berninger) that we
should proceed with a re-image of the machine. Alan Porter was kind
enough to volunteer to do the job. Thanks, Alan! Beyond the base OS,
work by Alan Porter, John Berninger, Kevin Otte, Tanner Lovelace and
others has been done to restore some services (like this list)
already. More will follow soon, including at least one more all-clear
announcement on this list.

If you have an ssh key, you already have access to login.trilug.org.
Please log in and see ~/READMENOW if you do have an ssh key. If you do
not have a key and need a password reset, please email Alan or John
(see emails at the top of this message). Please remember that we are a
group of volunteers. Some of us spent several hours (thanks again,
Alan!) this weekend getting us back up and running. If you contact us,
please understand that our response time might not match what you
would expect from a customer service group. Please keep checking
trilug.org for details.

Cheers,
-- 
Cristóbal M. Palmer
"The fun thing is to try to persuade others to share your opinions
about what rules and what sucks. Nothing is more fun than evangelism."
  --Larry Wall