[TriLUG] pilot/trilug.org reboot - guess what I learned?

[Alan Porter]

Hey, guess what I learned about LVM today?

A few months ago, I made a complete backup of /home on pilot.  I did 
this following the guidelines that have been told to me by smart people:

 - make a snapshot LV of /home (called /home-snap)
 - make a new empty partition (called /home-backup)
 - 'dd' from /home-snap to /home-backup
 - remove the snapshot LV

Guess what I learned today?  The ext3 UUID of /home-backup is the same 
as /home.  You can check this with the "blkid" command.

So when pilot rebooted, it mounted the WRONG copy of /home.  It looked 
in /etc/fstab to see what to mount, read the UUID, and started looking 
for that filesystem.  Unfortunately, it found 'home-backup' before it 
found of 'home', and so our users took a step back in time for the 
evening.  All of the files in our home directories looked like they used 
to in May.

So far, no harm, no foul.

We fixed the problem at about 10pm tonight.  I just did a "tune2fs -U 
random /dev/mapper/vg01-home.bak.20090517", and that gave the backup 
filesystem a new UUID.

But that means that any files that you created between 6pm and 10pm were 
written to the 'home-backup' partition and not to your real home 
directory.  Most people were not logged on at that time.  But if you 
received any email in that time period, it might have gotten dumped into 
(home-backup)/trilug/username/Maildir instead of where it SHOULD have put.

Gosh, this is a lot like "Back to the Future", when Marty's mom tries to 
kiss him.  Except the characters involved here are not as good-looking.

If you are concerned about those four hours worth of emails, or about 
the 18 minutes of missing Watergate recordings, you might want to take a 
look in /mnt/home.bak.20090517/trilug/YOURUSERNAME and see if they are 
hiding in there.

If I weren't such a klutz, I would write a shell script that looks for 
files in that May 2009 partition that have September 2009 dates.  But I 
have done enough damage for one night.

Let me know if I can help clarify what happened and how to look for any 
mis-placed files.

Alan