Getting Prometheus Metrics from TrueNAS

 TrueNAS (specifically referring to TrueNAS SCALE, but CORE should be the same) doesn't offer native support for monitoring by Prometheus. Suggestions on the Internets seem to point at using the built-in Graphite exporter to send metrics to graphite_exporter, which can be scraped by Prometheus. This DOES work, with some caveats:

1. Many metrics seem to report 0, incorrectly.
2. Tons of work to parse and create tags in the style of Prometheus

I spent several hours poking at point number 2, only to realize number 1. My solution, just run node_exporter directly on the truenas. It's a Go application, so it Just Works™, and it gets you probably everything that you would otherwise get from the graphite_exporter. To do this, I created a startup script in the truenas to run after boot. The script starts up node_exporter inside of tmux, done deal. Feel free to point out why this is a bad idea.

If you really want to go the graphite_exporter path, here is the beginning of my config file to start parsing the data stream. If anyone has a better one, let me know and I'll happily link to it.

---

mappings:

  - match: 'dragonfish.truenas.disk_ops.*.*'

    name: 'dragonfish_truenas_disk_ops'

    labels:

      device: $1

      operation: $2

  - match: 'dragonfish.truenas.cputemp.temperatures.*'

    name: 'dragonfish_truenas_cputemp'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.cpu.cpufreq.*'

    name: 'dragonfish_truenas_cpufreq'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.cpu.core_throttling.*'

    name: 'dragonfish_truenas_cpu_core_throttling'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.zfspool_state.*.*'

    name: 'dragonfish_truenas_zpool_state'

    labels:

      pool: $1

      state: $2

  # regex matches are performed after regular matches

  - match: 'dragonfish\.truenas\.cpu\.cpu(\d+)_cpuidle\.(.*)'

    match_type: regex

    name: 'dragonfish_truenas_cpuidle'

    labels:

      cpu: $1

      idlestate: $2

  - match: 'dragonfish\.truenas\.cpu\.cpu(\d+)\.(\w+)'

    match_type: regex

    name: 'dragonfish_truenas_cpu_utilization'

    labels:

      cpu: $1

      type: $2

  - match: 'dragonfish\.truenas\.disk_avgsz\.([[:alnum:]]+)\.writes'

    #help: 'Average I/O write operation size.'

    match_type: regex

    name: 'dragonfish_truenas_disk_avgsz_writes'

    labels:

      device: $1

  - match: 'dragonfish\.truenas\.disk_avgsz\.([[:alnum:]]+)\.reads'

    #help: 'Average I/O read operation size.'

    match_type: regex

    name: 'dragonfish_truenas_disk_avgsz_reads'

    labels:

      device: $1

Finding the hardware revision of a raspberry pi on EL platforms

Enterprise Linux platforms on Raspberry Pi (Centos, Oracle, etc.) do not have the /proc/cpuinfo lines referenced by the official docs to determine what hardware version you have. Instead, read the following files to get model and serial numbers.

# cat /proc/device-tree/serial-number

000000003fe78687

# cat /proc/device-tree/model

Raspberry Pi 3 Model B+

Multiple passwords with geli (on root)

As described in the man page, geli supports two slots for passphrases and/or keyfiles that can be used to decrypt the disk. By default, both of these slots are identical, using the passphrase provided at init. However, they can be set separately. We are using this feature to keep a "backup" passphrase on the encrypted zroot of a host. In theory, this allows us to walk remote hands through unlocking the disk, if we're not on-site, without disclosing the master passphrase. The backup passphrase could then be reset, restoring security.

In the case of geli-on-root configurations, the vintage of the installation determines the correct way to (re)set a passphrase. FreeBSD sysinstall on versions prior to 12.0 created an unencrypted boot partition, and utilize a keyfile in addition to a passphrase. Versions 12.0 and later just use a passphrase.

root@host: cat /boot/loader.conf
geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/encryption.key"
geli_ada2p4_keyfile0_load="YES"
geli_ada2p4_keyfile0_type="ada2p4:geli_keyfile0"
geli_ada2p4_keyfile0_name="/boot/encryption.key"

The newer style does not typically have the above parameters. Before changing a passphrase, verify that the disks you are going to operate on are the correct disks. For example, in the case of a zfs-on-root setup:

root@host: zpool status zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.5G in 0 days 00:13:54 with 0 errors on Wed Jul 10 09:57:10 2019
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            ada2p4.eli  ONLINE       0     0     0
            ada1p4.eli  ONLINE       0     0     0

errors: No known data errors

Once you have determined which style the host is configured with, you can reset the password. The only difference between the two is that for the older style you need to provide the keyfile argument (-K):

root@host: geli setkey -n 1 -K /boot/encryption.key ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If you are using a mirrored root, don't forget to update the passphrase on the second disk.

root@host: geli setkey -n 1 -K /boot/encryption.key ada2p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

Newer style setups can simply omit the passphrase directive.

root@host: geli setkey -n 1 ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If using two different passphrases, you can verify this by rebooting the host and trying each one. Disclaimer: I tested this using a throw-away virtual machine. I recommend testing this before trying it on your real data! Don't trust your data to something that you just copy-paste from the Internet!

Kodi Youtube plugin resolution

I recently started playing around with Kodi on a Raspberry Pi (Model 3+). I'm really interested in the Youtube plugin, because I have been binging on EDM shows lately. However, I was disappointed to discover that the plugin was only playing low quality streams (360P).

After some searching around, I learned that I need to get the MPEG-DASH functionality turned on. This can be found in the Youtube plugin settings, but it is grayed out by default. After some more wild flailing about, I finally got it working, something like this.

1. The "kodi-inputstream-adaptive" package needs to be installed from the Raspbian repos. It doesn't get pulled in as a dep of Kodi.
2. The inputstream plugin now needs to be enabled. Start Kodi and go to the settings menu. Select Add-ons, My add-ons, and VideoPlayer InputStream. Here you will find the InputStream Adaptive plugin. Click on it, and select "Enable".
3. You should now be able to go back to Youtube and enable MPEG-DASH. I also installed the Inputstream Helper plugin from that menu, but I'm not sure what effect it has.

I seem to have stumbled on an issue with the ipsec implementation on OpenBSD. While trying to configure manually-keyed transport SAs between hosts on the same subnet, I discovered that NDP appears to fail. Specifically, it appears that one endpoint will briefly learn the L2 address of the other endpoint, while the other host ndp cache never learns the address. This results in behavior where icmp6 traffic is encrypted in one direction (ironically, it is the traffic transmitted by the host that never appears to learn the address of the other), and not in the other. A tcpdump shows the following pattern:

20:48:22.518275 08:00:27:42:35:6a 33:33:ff:00:00:02 ip6 86: fc0a:4600::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has fc0a:4600::2(src lladdr: 08:00:27:42:35:6a) [icmp6 cksum ok] (len 32, hlim 255)
20:48:22.519842 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 801 len 88 (len 88, hlim 255)
20:48:22.521577 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 802 len 88 (len 88, hlim 255)
20:48:23.521836 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 803 len 88 (len 88, hlim 255)

Some searching turns up a conversation on the OpenBSD mailing lists that appears to describe the same behavior. However, it appears that there was never a consensus on the solution, and thus one was never implemented. I did find a hacky workaround that gets IPSec working between the two hosts. By setting static ndp entries for the remote host, there is no need for neighbor discovery to run, and the transport works.

root@net70-3[][20:51:37]:/etc ndp -s fc0a:4600::2  08:00:27:31:0f:52
root@net70-3[][20:51:58]:/etc ndp -an                                
Neighbor                             Linklayer Address   Netif Expire    S Flags
fc0a:4600::2                         08:00:27:31:0f:52    vio1 permanent R 

For reference, the following is my SA configuration. The spi and key values are the same in both directions, as I was working towards trying to make OpenBSD protect OSPFv3 traffic (I am still unsuccessful).

flow esp from fc0a:4600::3 to fc0a:4600::2 
esp transport from fc0a:4600::3 to fc0a:4600::2 spi 0xa753d86c:0xa753d86c \
 authkey $akey1:$akey1 \
 enckey  $ekey1:$ekey1

Proftpd builds broken on FreeBSD? Use gmake

I recently discovered an issue with our automated build of proftpd. I changed the target branch from the ancient 1.3.5 branch to 1.3.6. When I tried building this branch, I got a failure in the mod_sftp directory.

In file included from mod_sftp.c:29:
./mod_sftp.h:29:10: fatal error: 'conf.h' file not found
#include "conf.h"

After a bunch of screwing around and ripping my hair out, I took the time to actually read the INSTALL file. It turns out that GNU Make is required. We have been using BSD Make, so there must have been a recent-ish change that causes make to fail.

Pulling your hair out trying to fix your Mikrotik? Maybe your version of Netinstall is broken.

I spent an afternoon tearing my hair out. First, I managed to kinda-brick my Mikrotik during a firmware upgrade; the OS appeared to boot, but none of the network interfaces were visible, and it was generally disfunctional.

Reading up on the recovery process, I downloaded and installed the latest version of Netinstall, per the Mikrotik wiki. I spent the next four hours hating life, cursing technology, until I figured out the problem...the current version of Netinstall is broken!

If your copy of Netinstall just sits there, and your device never appears in the list, try this.

1. Download and extract Netinstall version 6.38.7 (I was running 6.43).
2. If you're using Windows 10 64-bit like me, open the properties for the executable. Change compatibility mode to Windows XP SP3, and run as Administrator (I'm not positive that either of these are required, but it's what I used).
3. Run Netinstall.
4. Device promptly appears.

Grrr.