Multiple passwords with geli (on root)

As described in the man page, geli supports two slots for passphrases and/or keyfiles that can be used to decrypt the disk. By default, both of these slots are identical, using the passphrase provided at init. However, they can be set separately. We are using this feature to keep a "backup" passphrase on the encrypted zroot of a host. In theory, this allows us to walk remote hands through unlocking the disk, if we're not on-site, without disclosing the master passphrase. The backup passphrase could then be reset, restoring security.

In the case of geli-on-root configurations, the vintage of the installation determines the correct way to (re)set a passphrase. FreeBSD sysinstall on versions prior to 12.0 created an unencrypted boot partition, and utilize a keyfile in addition to a passphrase. Versions 12.0 and later just use a passphrase.

root@host: cat /boot/loader.conf
geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/encryption.key"
geli_ada2p4_keyfile0_load="YES"
geli_ada2p4_keyfile0_type="ada2p4:geli_keyfile0"
geli_ada2p4_keyfile0_name="/boot/encryption.key"

The newer style does not typically have the above parameters. Before changing a passphrase, verify that the disks you are going to operate on are the correct disks. For example, in the case of a zfs-on-root setup:

root@host: zpool status zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.5G in 0 days 00:13:54 with 0 errors on Wed Jul 10 09:57:10 2019
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            ada2p4.eli  ONLINE       0     0     0
            ada1p4.eli  ONLINE       0     0     0

errors: No known data errors

Once you have determined which style the host is configured with, you can reset the password. The only difference between the two is that for the older style you need to provide the keyfile argument (-K):

root@host: geli setkey -n 1 -K /boot/encryption.key ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If you are using a mirrored root, don't forget to update the passphrase on the second disk.

root@host: geli setkey -n 1 -K /boot/encryption.key ada2p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

Newer style setups can simply omit the passphrase directive.

root@host: geli setkey -n 1 ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If using two different passphrases, you can verify this by rebooting the host and trying each one. Disclaimer: I tested this using a throw-away virtual machine. I recommend testing this before trying it on your real data! Don't trust your data to something that you just copy-paste from the Internet!

Hot-swap SATA disks in FreeBSD

If you ever need to hot-swap a disk on a FreeBSD box, atacontrol(8) is your friend. Swap the disk, then use atacontrol list to retrieve the list of ATA channels on the system.

root@neutron:~-> atacontrol list
ATA channel 0:
    Master: acd0  ATA/ATAPI revision 5
    Slave:       no device present
ATA channel 2:
    Master:  ad4  SATA revision 2.x
    Slave:       no device present
ATA channel 3:
    Master:  ad6  SATA revision 2.x
    Slave:       no device present
ATA channel 4:
    Master:  ad8  SATA revision 1.x
    Slave:       no device present
ATA channel 5:
    Master:    no device present
    Slave:       no device present

Find the appropriate channel, in this case ata5. Then simply perform a detach/attach operation on the channel and the disk should be found.

root@neutron:~-> atacontrol detach ata5
root@neutron:~-> atacontrol attach ata5
Master: ad10  SATA revision 2.x
Slave:       no device present

This example was done on FreeBSD 8.1-RELEASE.

Converting Debian from a single disk to RAID1

I've needed this often enough and stumbled through it; here's a link to a guide for converting a Debian install to a software RAID1 install.

http://www.ctunion.com/node/178

I used it for tips after I was already halfway through, so I don't know how accurate the guide is as a whole. The section dealing with creating a new initrd was close enough for me, but they refer to mkinitrd instead of mkinitramfs that is included in the squeeze initramfs-tools package.

Name that boot error!

I ran across an irritating boot error yesterday that basically sucked up my entire day resolving. The situation is this. I was testing a bare-metal recovery of a Debian Lenny server using bacula. I had restored a backup of the server onto new hardware and it was time to reboot the new machine and see what was broken. On boot, I received the following output (re-keyed from a camera-phone photo, may not be totally accurate).

Booting the kernel.
Loading, please wait...
mdadm: no devices found for /dev/md0
mdadm: no devices found for /dev/md1
[       ] device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: release ioctl failed: Invalid argument
1 logical volume(s) in volume group "vg-root" now active
kinit: name_to_dev_t(/dev/sdb1) = sdb1(8,17)
kinit: trying to resume from /dev/sdb1
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
mount: mounting /dev/mapper/vg--root-lv--root on /root failed: No such device
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. try passing init= bootarg.

It then dumped me into an initramfs prompt. I spent the day trying to get the broken system to boot, when the simple solution eluded me. The root problem here is that the kernel cannot assemble the md arrays. In my case, this was caused because my raid partitions were not labeled with raid superblocks. I managed to do this by using the --assemble option in mdadm when I built my arrays. The correct option is --create, which writes superblocks properly. Once I figured this out, I "created" the arrays from a Knoppix disk (Ubuntu works well too), rebooted, and voila, the system boots right up.

From what I have read, you can safely "create" an array over top of the existing array and not lose data. mdadm is careful to do the right thing. That said, don't take my word for it. I tried it on an array that was restored from backup. It worked for me, but if it had not I wasn't out more than 10 minutes of my own time.

Interestingly, I was able to get the system to boot from the broken initramfs prompt. Here is the basic procedure. I did this yesterday and I'm going from memory and notes scribbled on a scrap of paper, so YMMV here as well.

1. Build md arrays using mdadm
2. Using the lvm shell, do a pvscan (to redetect LVM structures properly)
3. Using the lvm shell, activate the logical volumes (e.g. lvchange -ay /dev/mapper/vg--root-lv--root)
4. Mount / on /root. Must be mounted READ-ONLY!
5. Mount dev, proc, and sys under /root
6. Copy the contents of /dev to /root/dev
7. Type exit to escape the initramfs, this should dump you to a shell. You will get a kernel panic if everything isn't set up just right.
   
8. Run /etc/init.d/rcS to startup the system.
9. Run /etc/init.d/rc 2 to bring the system to a somewhat functioning state. I was actually able to get most everything working like this, but it is obviously not a state to leave a server in, since a reboot will bring you back to square one.