[TriLUG] bash died?

[Brian Henning]

Hey y'all,
  Now this one is weird to me.  I was trying to troubleshoot a nonfunctional
mgetty/pppd dialin access setup, and something went very strangely awry.

  I was working remotely from home, where I have both broadband and a modem.
That way, I figured, I could be ssh'd in and watch the logs while I also
attempted the dial connections.  So I had PuTTY open on one computer,
running screen as root, split-screened into a view of tail -f
mgetty.log.ttyS1, and a view of tail -f messages (in /var/log of course).

  So I dial in and it fails.  Interesting messages go by on
mgetty.log.ttyS1.  I notice that mgetty doesn't correctly catch the hangup
(again), so I ^A-c a new bash session and killall -SIGHUP mgetty.

Segmentation fault.

Huh?  I hadn't seen that happen before.  So just for curiosity, I ps -e in
that bash instance, and....nothing.  I can't ^C [break] it or ^S [stop] it,
either.  It's just dead.

So I swap back to the tail -f /var/log/messages and what do I behold but a
kernel core dump in /var/log/messages.  And here's where it gets even
weirder...

I can ^C the tail, so I do that and get another bash prompt.  Stupidly I do
another ps -e, with the same effect - bash merrily scampers off into the
bushes, never to be seen again.

Well, I've got one tail -f still going (the one on mgetty.log.ttyS1), so I
^C it and, thinking a little further ahead, type init 6 to reboot.
And...nothing.  Same effect as when I did ps -e.  Bash just dies.  I still
have control over screen, I can still flip between screens, create new
screens (although bash does not spawn correctly in newly created
screens...just a blank cursor; no prompt)..  This is getting quite silly, I
thought.

Well since I've lost control of that whole session now, I'll just ssh in
anew, right?  Well, almost.  Apparently, sshd is still happily plugging
along, accepting connections and checking credentials.  Bash just never does
its thing.

SO...  the finale of this long-winded play-by-play is this question:  Is
there anything at all I can do remotely to try to get the machine back into
some sense?  Maybe somehow force sshd to try launching a different shell
instead of bash (if it's even tied to bash at all)?  I don't really want to
have to visit the machine in person unless I have to, as it means a
30-minute drive and arranging access into the closed building (we IT peons
don't get keys...).

After this snafu is straightened out, then I'll present my mgetty problems..
;-)

Thanks a bunch fellas,
~Brian