Tailscale Gitops Pipeline Failure on test but not apply

 I am in the process of setting up Gitops pipelines for my Tailscale ACL file. I use Gitlab, so I am following the instructions here. The initial pushes onto main resulted in a successful apply pipeline, great! However, when I created a feature branch with some changes and started a Merge Request, I was greeted with failure and the seemingly unhelpful error:

$ gitops-pusher --policy-file ./policy.hujson test
wanted HTTP status code 200 but got 401

Several test branches later, in the process of creating a bug report on the Tailscale Github, when it dawned on me that when I had set up the CI variables on the repo, I mashed all of the buttons about protecting the TS_API_KEY variable. Go back and look at the Gitlab CI variables and sure enough, I had set the Protected flag on the var, which prevents the variable from being used on non-protected branches.

So it turns out that the unhelpful error message was actually telling me exactly what the problem was; I just wasn't paying attention. In this process, other hits on Google noted that this error is also caused by an expired API key. In that case, you'd see both test and apply stages failing. Tailscale expires keys after a maximum 90-days, which means this is probably an annoyingly frequent occurrence. 

Getting Prometheus Metrics from TrueNAS

 TrueNAS (specifically referring to TrueNAS SCALE, but CORE should be the same) doesn't offer native support for monitoring by Prometheus. Suggestions on the Internets seem to point at using the built-in Graphite exporter to send metrics to graphite_exporter, which can be scraped by Prometheus. This DOES work, with some caveats:

1. Many metrics seem to report 0, incorrectly.
2. Tons of work to parse and create tags in the style of Prometheus

I spent several hours poking at point number 2, only to realize number 1. My solution, just run node_exporter directly on the truenas. It's a Go application, so it Just Works™, and it gets you probably everything that you would otherwise get from the graphite_exporter. To do this, I created a startup script in the truenas to run after boot. The script starts up node_exporter inside of tmux, done deal. Feel free to point out why this is a bad idea.

If you really want to go the graphite_exporter path, here is the beginning of my config file to start parsing the data stream. If anyone has a better one, let me know and I'll happily link to it.

---

mappings:

  - match: 'dragonfish.truenas.disk_ops.*.*'

    name: 'dragonfish_truenas_disk_ops'

    labels:

      device: $1

      operation: $2

  - match: 'dragonfish.truenas.cputemp.temperatures.*'

    name: 'dragonfish_truenas_cputemp'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.cpu.cpufreq.*'

    name: 'dragonfish_truenas_cpufreq'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.cpu.core_throttling.*'

    name: 'dragonfish_truenas_cpu_core_throttling'

    labels:

      cpu: $1

  - match: 'dragonfish.truenas.zfspool_state.*.*'

    name: 'dragonfish_truenas_zpool_state'

    labels:

      pool: $1

      state: $2

  # regex matches are performed after regular matches

  - match: 'dragonfish\.truenas\.cpu\.cpu(\d+)_cpuidle\.(.*)'

    match_type: regex

    name: 'dragonfish_truenas_cpuidle'

    labels:

      cpu: $1

      idlestate: $2

  - match: 'dragonfish\.truenas\.cpu\.cpu(\d+)\.(\w+)'

    match_type: regex

    name: 'dragonfish_truenas_cpu_utilization'

    labels:

      cpu: $1

      type: $2

  - match: 'dragonfish\.truenas\.disk_avgsz\.([[:alnum:]]+)\.writes'

    #help: 'Average I/O write operation size.'

    match_type: regex

    name: 'dragonfish_truenas_disk_avgsz_writes'

    labels:

      device: $1

  - match: 'dragonfish\.truenas\.disk_avgsz\.([[:alnum:]]+)\.reads'

    #help: 'Average I/O read operation size.'

    match_type: regex

    name: 'dragonfish_truenas_disk_avgsz_reads'

    labels:

      device: $1

Finding the hardware revision of a raspberry pi on EL platforms

Enterprise Linux platforms on Raspberry Pi (Centos, Oracle, etc.) do not have the /proc/cpuinfo lines referenced by the official docs to determine what hardware version you have. Instead, read the following files to get model and serial numbers.

# cat /proc/device-tree/serial-number

000000003fe78687

# cat /proc/device-tree/model

Raspberry Pi 3 Model B+

Multiple passwords with geli (on root)

As described in the man page, geli supports two slots for passphrases and/or keyfiles that can be used to decrypt the disk. By default, both of these slots are identical, using the passphrase provided at init. However, they can be set separately. We are using this feature to keep a "backup" passphrase on the encrypted zroot of a host. In theory, this allows us to walk remote hands through unlocking the disk, if we're not on-site, without disclosing the master passphrase. The backup passphrase could then be reset, restoring security.

In the case of geli-on-root configurations, the vintage of the installation determines the correct way to (re)set a passphrase. FreeBSD sysinstall on versions prior to 12.0 created an unencrypted boot partition, and utilize a keyfile in addition to a passphrase. Versions 12.0 and later just use a passphrase.

root@host: cat /boot/loader.conf
geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/encryption.key"
geli_ada2p4_keyfile0_load="YES"
geli_ada2p4_keyfile0_type="ada2p4:geli_keyfile0"
geli_ada2p4_keyfile0_name="/boot/encryption.key"

The newer style does not typically have the above parameters. Before changing a passphrase, verify that the disks you are going to operate on are the correct disks. For example, in the case of a zfs-on-root setup:

root@host: zpool status zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.5G in 0 days 00:13:54 with 0 errors on Wed Jul 10 09:57:10 2019
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            ada2p4.eli  ONLINE       0     0     0
            ada1p4.eli  ONLINE       0     0     0

errors: No known data errors

Once you have determined which style the host is configured with, you can reset the password. The only difference between the two is that for the older style you need to provide the keyfile argument (-K):

root@host: geli setkey -n 1 -K /boot/encryption.key ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If you are using a mirrored root, don't forget to update the passphrase on the second disk.

root@host: geli setkey -n 1 -K /boot/encryption.key ada2p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

Newer style setups can simply omit the passphrase directive.

root@host: geli setkey -n 1 ada1p4
Enter new passphrase:
Reenter new passphrase:
Note, that the master key encrypted with old keys and/or passphrase may still exists in a metadata backup file.

If using two different passphrases, you can verify this by rebooting the host and trying each one. Disclaimer: I tested this using a throw-away virtual machine. I recommend testing this before trying it on your real data! Don't trust your data to something that you just copy-paste from the Internet!

Kodi Youtube plugin resolution

I recently started playing around with Kodi on a Raspberry Pi (Model 3+). I'm really interested in the Youtube plugin, because I have been binging on EDM shows lately. However, I was disappointed to discover that the plugin was only playing low quality streams (360P).

After some searching around, I learned that I need to get the MPEG-DASH functionality turned on. This can be found in the Youtube plugin settings, but it is grayed out by default. After some more wild flailing about, I finally got it working, something like this.

1. The "kodi-inputstream-adaptive" package needs to be installed from the Raspbian repos. It doesn't get pulled in as a dep of Kodi.
2. The inputstream plugin now needs to be enabled. Start Kodi and go to the settings menu. Select Add-ons, My add-ons, and VideoPlayer InputStream. Here you will find the InputStream Adaptive plugin. Click on it, and select "Enable".
3. You should now be able to go back to Youtube and enable MPEG-DASH. I also installed the Inputstream Helper plugin from that menu, but I'm not sure what effect it has.

I seem to have stumbled on an issue with the ipsec implementation on OpenBSD. While trying to configure manually-keyed transport SAs between hosts on the same subnet, I discovered that NDP appears to fail. Specifically, it appears that one endpoint will briefly learn the L2 address of the other endpoint, while the other host ndp cache never learns the address. This results in behavior where icmp6 traffic is encrypted in one direction (ironically, it is the traffic transmitted by the host that never appears to learn the address of the other), and not in the other. A tcpdump shows the following pattern:

20:48:22.518275 08:00:27:42:35:6a 33:33:ff:00:00:02 ip6 86: fc0a:4600::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has fc0a:4600::2(src lladdr: 08:00:27:42:35:6a) [icmp6 cksum ok] (len 32, hlim 255)
20:48:22.519842 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 801 len 88 (len 88, hlim 255)
20:48:22.521577 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 802 len 88 (len 88, hlim 255)
20:48:23.521836 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 803 len 88 (len 88, hlim 255)

Some searching turns up a conversation on the OpenBSD mailing lists that appears to describe the same behavior. However, it appears that there was never a consensus on the solution, and thus one was never implemented. I did find a hacky workaround that gets IPSec working between the two hosts. By setting static ndp entries for the remote host, there is no need for neighbor discovery to run, and the transport works.

root@net70-3[][20:51:37]:/etc ndp -s fc0a:4600::2  08:00:27:31:0f:52
root@net70-3[][20:51:58]:/etc ndp -an                                
Neighbor                             Linklayer Address   Netif Expire    S Flags
fc0a:4600::2                         08:00:27:31:0f:52    vio1 permanent R 

For reference, the following is my SA configuration. The spi and key values are the same in both directions, as I was working towards trying to make OpenBSD protect OSPFv3 traffic (I am still unsuccessful).

flow esp from fc0a:4600::3 to fc0a:4600::2 
esp transport from fc0a:4600::3 to fc0a:4600::2 spi 0xa753d86c:0xa753d86c \
 authkey $akey1:$akey1 \
 enckey  $ekey1:$ekey1

Proftpd builds broken on FreeBSD? Use gmake

I recently discovered an issue with our automated build of proftpd. I changed the target branch from the ancient 1.3.5 branch to 1.3.6. When I tried building this branch, I got a failure in the mod_sftp directory.

In file included from mod_sftp.c:29:
./mod_sftp.h:29:10: fatal error: 'conf.h' file not found
#include "conf.h"

After a bunch of screwing around and ripping my hair out, I took the time to actually read the INSTALL file. It turns out that GNU Make is required. We have been using BSD Make, so there must have been a recent-ish change that causes make to fail.