[TriLUG] Server DEAD!

[Joseph Tate]

rasch at raschnet.com wrote:

>On Thu, Aug 28, 2003 at 09:05:07AM -0700, auto668 at hush.com wrote:
>  
>
>>Serious issue here, I've had a server running for a couple weeks doing
>>some production virtual hosting.  All has been running great, everything
>>was configured and running fine I haven't done ANYTHING other than run
>>uup2date periodically.  Well, today I'm about to do a test on the box
>>after installing the Real Media server and here's what happens...
>>
>>[root at www Helix]# /etc/rc.d/init.d/iptables stop
>>/etc/rc.d/init.d/iptables: line 41: 14950 Done                    /sbin/lsmod
>>2>/dev/null
>>     14951 Segmentation fault      | grep -q ipchains
>>
>>[root at www Helix]# /etc/rc.d/init.d/iptables restart
>>/etc/rc.d/init.d/iptables: line 41: 14966 Done                    /sbin/lsmod
>>2>/dev/null
>>     14967 Segmentation fault      | grep -q ipchains
>>
>>****SO I DECIDE, I'M LOST, LET'S just try rebooting for the sake of reboting**
>>
>>Now it won't even come back up, I can't copy/paste but here is some of
>>what I'm getting
>>
>>45 Segmentation Fault
>>     LC_ALL=C grep -q "Red Hat" /etc/redhat-release  RedHat Linux
>>
>>Mounting proc filesystem                               [FAILED]
>>    /etc/rc.d/rc.sysinit :  Line 98:   Segmentation Fault   LC_ALL=C
>>grep -q 
>>
>>Coninues this for about 3/4 more lines and totally quits after setting
>>hostname.
>>
>>I literally, haven't done anything other than load the updates using
>>up2date form the command line.  Only had ssh/apache running.
>>
>>Any ideas would be greatly appreciate as I said this is a production
>>box and one customer has already called since this happened1
>>    
>>
>
>Last time I started getting Seg. faults in system programs was when our
>office machine had been "root-kitted" via the samba exploit (why wasn't
>it firewalled!?).  I hope this hasn't happened to you!  I ended up
>re-installing as opposed to playing cleanup, because I kept losing
>ground by touching one of the replaced binaries.  Did you change
>anything prior to this crash?
>
>Best of luck,
>    David
>  
>
Seg Faults can also happen when you try to use the wrong version of 
glibc, or kernel.  up2date should handle that properly though.  If you 
can boot off a rescue disk, and then chroot to your system, run rpm -A | 
grep kernel and rpm -A | grep glibc.

Joseph