Multiple passwords with geli (on root)

As described in the man page, geli supports two slots for passphrases and/or keyfiles that can be used to decrypt the disk. By default, both of these slots are identical, using the passphrase provided at init. However, they can be set separately. We are using this feature to keep a "backup" passphrase on the encrypted zroot of a host. In theory, this allows us to walk remote hands through unlocking the disk, if we're not on-site, without disclosing the master passphrase. The backup passphrase could then be reset, restoring security.

In the case of geli-on-root configurations, the vintage of the installation determines the correct way to (re)set a passphrase. FreeBSD sysinstall on versions prior to 12.0 created an unencrypted boot partition, and utilize a keyfile in addition to a passphrase. Versions 12.0 and later just use a passphrase.

root@host: cat /boot/loader.conf
geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/encryption.key"
geli_ada2p4_keyfile0_load="YES"
geli_ada2p4_keyfile0_type="ada2p4:geli_keyfile0"
geli_ada2p4_keyfile0_name="/boot/encryption.key"

The newer style does not typically have the above parameters. Before changing a passphrase, verify that the disks you are going to operate on are the correct disks. For example, in the case of a zfs-on-root setup:

root@host: zpool status zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.5G in 0 days 00:13:54 with 0 errors on Wed Jul 10 09:57:10 2019
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            ada2p4.eli  ONLINE       0     0     0
            ada1p4.eli  ONLINE       0     0     0

errors: No known data errors

Once you have determined which style the host is configured with, you can reset the password. The only difference between the two is that for the older style you need to provide the keyfile argument (-K):

root@host: geli setkey -n 1 -K /boot/encryption.key ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If you are using a mirrored root, don't forget to update the passphrase on the second disk.

root@host: geli setkey -n 1 -K /boot/encryption.key ada2p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

Newer style setups can simply omit the passphrase directive.

root@host: geli setkey -n 1 ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If using two different passphrases, you can verify this by rebooting the host and trying each one. Disclaimer: I tested this using a throw-away virtual machine. I recommend testing this before trying it on your real data! Don't trust your data to something that you just copy-paste from the Internet!

ZFS Volumes not showing up on reboot?

If you're using ZFS on FreeBSD and your ZFS volumes do not appear after rebooting the system, verify that your rc.conf file has zfs_enable="YES". This allows /etc/rc.d/zvol to run, which executes an undocumented 'zfs volinit' command to create the /dev/zvol/... device entries. The script also adds swap space volumes if the ZFS volume has org.freebsd:swap=on set.

Renaming user-defined ZFS properties

Here is a short script I cooked up to rename the namespace of all the ZFS user-defined properties on a host. Useful if you need to change com.foo:beans to com.bar:beans for more than a couple of properties on a couple of filesystems.

#!/usr/local/bin/perl

#
# This script looks at the properties for every zfs filesystem and snapshot on
# a server and changes every occurance of a property in in namespace $old to
# namespace $old.
#

use strict;

my ($old, $new) = ("com.foo", "com.bar");
my $overwrite = "yes"; # overwrite existing $new properties
my $localonly = "yes"; # do not move inherited properties

for my $fs ( `zfs list -Ho name` ) {
chomp $fs;
&do_rename($fs);
}

for my $snap ( `zfs list -Ho name -t snapshot` ) {
chomp $snap;
&do_rename($snap);
}

sub do_rename {
#print "Renaming $old to $new on $_[0]\n";
# get list of properties with $old
for my $prop ( `zfs get -Ho property,source,value all $_[0] | grep $old` ) {
$prop =~ m/$old:([a-z._:][a-z.\-_:]*)\t([a-z0-9\/\- ]+)\t(.*)\n/;
#print "Examining property $old:$1 on $_[0] with value $3\n";
my ($suffix, $value) = ($1, $3);
# local check, if enabled
next if (( $localonly eq "yes" ) and ( $2 ne "local" ));
next if (( $overwrite eq "no" ) and ( &check_exists($_[0],"$old:$suffix") == 0 ));
print "Setting $new:$suffix=$value and inheriting (erasing) $old:$suffix on $_[0]\n";
( system("zfs set $new:$suffix=$value $_[0]") == 0 ) or die "Error during zfs set operation";
( system("zfs inherit $old:$suffix $_[0]") == 0 ) or die "Error during zfs inherit operation";
}
}

sub check_exists {
# Return 0 if the specified property [1] exists on object [0]
if ( system("zfs get -Ho value $_[1] $_[0] | grep -qE '^-\$'") == 0 ) {
# DNE
return 1;
} else {
print "$_[1] exists on $_[0]!\n";
return 0;
}
}

Undocumented zfs command

In the process of tracking down a ZFS mount problem (btw, a zpool will not mount automatically if the mount point exists), I realized that my ZFS swap volume was not available to FreeBSD. After some poking around, I found the /etc/rc.d/zvol script, which led me to the 'zfs volinit' command. This command is not documented in the man page or command usage, but it seems to be the goo that gets volumes going. Also, set the org.freebsd:swap=on property on your swap volume, to have it started automatically by the rc scripts.

I/O errors on zfs import?

I've been working on freshening up our backup scripts and I noticed that every time I do a ZFS import, the kernel logs I/O errors on RAID volumes, completely separate from the actual volume I want to import, including already mounted ZFS pools and UFS filesystems. For what it's worth, this behavior was occurring on a Dell r610, with an attached MD1000.

Jul 25 13:59:54 leopard kernel: mfi0: I/O error, status= 12 scsi_status= 0
Jul 25 13:59:54 leopard kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Jul 25 13:59:54 leopard kernel: mfid1: hard error cmd=read 0-255

It was suggested that I should update the firmware on the disks, so this morning I went and updated all the disks, and the PERC 6/E. Voila! No more I/O errors on import.

For reference, here is the link to the firmware download I used. It is a windows executable that allows you to generate a bootable USB key that contains the firmware updater for the disks. I also used the underlying DOS environment to apply the firmware update for the PERC 6/E, and our brand new PERC H800 that came from Dell with ancient firmware.