[TriLUG] lesson learned: don't upgrade kernel with rpm -F (or -U)

[Jeremy P]

Hi,

I just spent most of the afternoon carefully fixing a mistake I made on a
semi-production server.  I learned a big lesson: don't upgrade your kernel
with the "upgrade" function of rpm !  I was updating a Red Hat Linux 7.2
server, to apply some of the newest patches (zlib, ssh, etc.).  I had
downloaded all of the RPMs, and usually I move the kernel upgrades to
another directory, but I forgot to this time.  So I blindingly and
stupidly ran "rpm -Fvh *.rpm" on the directory with the downloaded
updates, and got myself a brand-spanking new[1] kernel (2.4.9-31).

Okay, well that should be fine, right?  WRONG.  One of the steps in the
kernel package's postinstall script is to run "mkinitrd" to update the
initial ramdisk (needed for this server's SCSI module).  However, the
mkinitrd step fails with a cryptic error message: "All of your loopback
devices are in use!"  That of course wasn't true, but it turns out the
real problem was that the loopback driver (loop.o) couldn't be loaded,
since the /lib/modules directory for the currently running kernel had
already been deleted by rpm !!!  But I couldn't reboot with the new
kernel, since I don't yet have the initrd!  OOPS.  Chicken-egg situation.

Fortunately, I had another computer running RH 7.2 to which I had not yet
removed the chickens and eggs, as it were.   I was able to install the
new kernel package (with rpm -i --force), so there are now two kernel
packages installed.  Then, I copied /etc/modules.conf temporarily from the
damaged server to this computer.  Now, mkinitrd was able to function and
create an initrd (since this computer has a working loopback
module).  I copied the initrd image back up to the server, and
successfully rebooted to the new kernel.

Obviously, all of this is different if you build your own kernel, or if I
had paid attention to the *proper* way to upgrade the kernel on a Red Hat
Linux system.  But I thought this message might prove useful in case
anyone else came across this problem.

--Jeremy

[1]  Well, new to this computer.  I realize 2.4.9 isn't exactly the newest
kernel out there, but it's the newest released officially by Red Hat.