[TriLUG] Debian, LVM, and RAID

[Kevin Otte]

...or how I wasted 5 hours of my weekend.

Scenario:

I have one 80G disk with LVM VG "vg0" on it.  I have a second identical 80G
drive.  I want to set up a RAID1 array but not have to go through the pain
of a backup and restore cycle.

Where I went (an overview):

- Install second drive as sec IDE master
- Booted from a Debian 3.1 disc and hopped to tty2 once installer was
  started
- Mirror partition sizes
- Change LVM (0x8e) to RAID (0xfd) on the new HD
- Created the RAID mirror consisting of elements "missing" and the new hard
  drive
- Did a pvcreate on the new MD device
- Extended the VG into the new PV
- Performed a pvmove off the PV on the original HD to the new PV in the MD
  (about to die from an acronym overdose yet?)
- Did a vgreduce to eliminate the original HD PV from the VG
- Did a pvremove of the original HD PV
- Changed LVM (0x8e) to RAID (0xfd) on the old HD
- Added the old HD to the RAID array
- Watched it rebuild

Now, all of this worked amazingly well, if incredibly slowly.  pvmove, if I
may paraphrase Mr. Aquilina, takes "forfreakingever".  I attempted to reboot
into my new environment.  It didn't work.  This failure, however, was
expected as I had not created a new initrd.  I actually wanted to see what
would happen.  Anyway, I returned to the Debian installer environment to
mount things up and create the initrd.  Here's where the problems really
began:

By default LVM tries to scan all partitions, regardless of whether they are
type 0x8e.  I think this is stupid, but I'm no developer, so what do I know. 
So naturally, when I tried to manually do the pvscan, it found duplicate PVs
on /dev/hda2 and /dev/hdc2.  Without any way to drop LVM nicely, I boldly
used mdadm to start the RAID array.  It started as I would expected to.  I
did a pvscan, and got "No matching physical volumes found."  I tried other
various incantations of pv and vg commands to try and see my beloved volume
group, but alas it appeared lost in the void.

I tried rebooting the installer environ a couple times in the hopes of
starting the RAID array before LVM got its grubby hands on the PVs.  Still
nothing.  It could not seem to find my VG.  *sigh*

By this point, I have now been down for 5 hours (some of that was some other
maintainence), the bulk of that being the pvmove.  I'm frustrated, but have
a failout plan.  I simply unplugged the new drive and restored the partition
type to 0x8e.  Since Linux's RAID seems to only add a superblock(*), I was
back in business.

My question, fair readers, is this: What the heck did I do wrong?!  I have
installed numerous systems at work using an RAID1 as a PV, but I did the
configuration of those directly from the installer.  Apparently it does some
voodoo, as you can see my luck trying to migrate an existing installation to
this configuration.

I'll try to provide any further detail I can as needed, but by the time I
hit hour 4, frustration started setting in.

Side question:
(*) I tried this in testing using a couple of LVs and it seemed to work, but
I wondered if anyone else has tried it:

- lvcreate -n raidt0 -L 512M vg0
- lvcreate -n raidt1 -L 512M vg0
- mke2fs -j /dev/vg0/raidt0
- Mount the FS and drop a test file on it
- Unmount the FS
- mdadm -C /dev/md0 -l 1 -n 2 raidt0 missing
  (This gives a warning about there being an ext filesystem there already)
- Mount /dev/md0
- Note the test file is still there and happy.

I realize this is probably one of those "not recommended" things, but if it
seems safe enough to some other hackers out here, I'll take it.  I really
don't want to sit through that insanely long pvmove again.

Sorry for the novela folks.

-- 
Kevin Otte, N8VNR
nivex at nivex.net
http://www.nivex.net/

-=-

"Those who cannot remember the past are condemned to repeat it." 
-- George Santayana

"It seems no one reads Santayana anymore."
-- Cdr. Susan Ivanova, Babylon 5