Tuesday, June 9, 2009

Name that boot error!

I ran across an irritating boot error yesterday that basically sucked up my entire day resolving. The situation is this. I was testing a bare-metal recovery of a Debian Lenny server using bacula. I had restored a backup of the server onto new hardware and it was time to reboot the new machine and see what was broken. On boot, I received the following output (re-keyed from a camera-phone photo, may not be totally accurate).

Booting the kernel.
Loading, please wait...
mdadm: no devices found for /dev/md0
mdadm: no devices found for /dev/md1
[ ] device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: release ioctl failed: Invalid argument
1 logical volume(s) in volume group "vg-root" now active
kinit: name_to_dev_t(/dev/sdb1) = sdb1(8,17)
kinit: trying to resume from /dev/sdb1
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
mount: mounting /dev/mapper/vg--root-lv--root on /root failed: No such device
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. try passing init= bootarg.
It then dumped me into an initramfs prompt. I spent the day trying to get the broken system to boot, when the simple solution eluded me. The root problem here is that the kernel cannot assemble the md arrays. In my case, this was caused because my raid partitions were not labeled with raid superblocks. I managed to do this by using the --assemble option in mdadm when I built my arrays. The correct option is --create, which writes superblocks properly. Once I figured this out, I "created" the arrays from a Knoppix disk (Ubuntu works well too), rebooted, and voila, the system boots right up.

From what I have read, you can safely "create" an array over top of the existing array and not lose data. mdadm is careful to do the right thing. That said, don't take my word for it. I tried it on an array that was restored from backup. It worked for me, but if it had not I wasn't out more than 10 minutes of my own time.

Interestingly, I was able to get the system to boot from the broken initramfs prompt. Here is the basic procedure. I did this yesterday and I'm going from memory and notes scribbled on a scrap of paper, so YMMV here as well.
  1. Build md arrays using mdadm
  2. Using the lvm shell, do a pvscan (to redetect LVM structures properly)
  3. Using the lvm shell, activate the logical volumes (e.g. lvchange -ay /dev/mapper/vg--root-lv--root)
  4. Mount / on /root. Must be mounted READ-ONLY!
  5. Mount dev, proc, and sys under /root
  6. Copy the contents of /dev to /root/dev
  7. Type exit to escape the initramfs, this should dump you to a shell. You will get a kernel panic if everything isn't set up just right.
  8. Run /etc/init.d/rcS to startup the system.
  9. Run /etc/init.d/rc 2 to bring the system to a somewhat functioning state. I was actually able to get most everything working like this, but it is obviously not a state to leave a server in, since a reboot will bring you back to square one.

No comments:

Post a Comment