Friday, April 19, 2019

Kodi Youtube plugin resolution

I recently started playing around with Kodi on a Raspberry Pi (Model 3+). I'm really interested in the Youtube plugin, because I have been binging on EDM shows lately. However, I was disappointed to discover that the plugin was only playing low quality streams (360P).

After some searching around, I learned that I need to get the MPEG-DASH functionality turned on. This can be found in the Youtube plugin settings, but it is grayed out by default. After some more wild flailing about, I finally got it working, something like this.

  1. The "kodi-inputstream-adaptive" package needs to be installed from the Raspbian repos. It doesn't get pulled in as a dep of Kodi.
  2. The inputstream plugin now needs to be enabled. Start Kodi and go to the settings menu. Select Add-ons, My add-ons, and VideoPlayer InputStream. Here you will find the InputStream Adaptive plugin. Click on it, and select "Enable".
  3. You should now be able to go back to Youtube and enable MPEG-DASH. I also installed the Inputstream Helper plugin from that menu, but I'm not sure what effect it has.

Friday, March 15, 2019

I seem to have stumbled on an issue with the ipsec implementation on OpenBSD. While trying to configure manually-keyed transport SAs between hosts on the same subnet, I discovered that NDP appears to fail. Specifically, it appears that one endpoint will briefly learn the L2 address of the other endpoint, while the other host ndp cache never learns the address. This results in behavior where icmp6 traffic is encrypted in one direction (ironically, it is the traffic transmitted by the host that never appears to learn the address of the other), and not in the other. A tcpdump shows the following pattern:

20:48:22.518275 08:00:27:42:35:6a 33:33:ff:00:00:02 ip6 86: fc0a:4600::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has fc0a:4600::2(src lladdr: 08:00:27:42:35:6a) [icmp6 cksum ok] (len 32, hlim 255)
20:48:22.519842 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 801 len 88 (len 88, hlim 255)
20:48:22.521577 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 802 len 88 (len 88, hlim 255)
20:48:23.521836 08:00:27:31:0f:52 08:00:27:42:35:6a ip6 142: esp spi 0xa753d86c seq 803 len 88 (len 88, hlim 255)

Some searching turns up a conversation on the OpenBSD mailing lists that appears to describe the same behavior. However, it appears that there was never a consensus on the solution, and thus one was never implemented. I did find a hacky workaround that gets IPSec working between the two hosts. By setting static ndp entries for the remote host, there is no need for neighbor discovery to run, and the transport works.

root@net70-3[][20:51:37]:/etc ndp -s fc0a:4600::2  08:00:27:31:0f:52
root@net70-3[][20:51:58]:/etc ndp -an                                
Neighbor                             Linklayer Address   Netif Expire    S Flags
fc0a:4600::2                         08:00:27:31:0f:52    vio1 permanent R 

For reference, the following is my SA configuration. The spi and key values are the same in both directions, as I was working towards trying to make OpenBSD protect OSPFv3 traffic (I am still unsuccessful).

flow esp from fc0a:4600::3 to fc0a:4600::2 
esp transport from fc0a:4600::3 to fc0a:4600::2 spi 0xa753d86c:0xa753d86c \
 authkey $akey1:$akey1 \
 enckey  $ekey1:$ekey1

Thursday, November 1, 2018

Proftpd builds broken on FreeBSD? Use gmake

I recently discovered an issue with our automated build of proftpd. I changed the target branch from the ancient 1.3.5 branch to 1.3.6. When I tried building this branch, I got a failure in the mod_sftp directory.

In file included from mod_sftp.c:29:
./mod_sftp.h:29:10: fatal error: 'conf.h' file not found
#include "conf.h"

After a bunch of screwing around and ripping my hair out, I took the time to actually read the INSTALL file. It turns out that GNU Make is required. We have been using BSD Make, so there must have been a recent-ish change that causes make to fail.

Tuesday, September 11, 2018

Pulling your hair out trying to fix your Mikrotik? Maybe your version of Netinstall is broken.

I spent an afternoon tearing my hair out. First, I managed to kinda-brick my Mikrotik during a firmware upgrade; the OS appeared to boot, but none of the network interfaces were visible, and it was generally disfunctional.

Reading up on the recovery process, I downloaded and installed the latest version of Netinstall, per the Mikrotik wiki. I spent the next four hours hating life, cursing technology, until I figured out the problem...the current version of Netinstall is broken!

If your copy of Netinstall just sits there, and your device never appears in the list, try this.

  1. Download and extract Netinstall version 6.38.7 (I was running 6.43).
  2. If you're using Windows 10 64-bit like me, open the properties for the executable. Change compatibility mode to Windows XP SP3, and run as Administrator (I'm not positive that either of these are required, but it's what I used).
  3. Run Netinstall.
  4. Device promptly appears.

Grrr.


Wednesday, August 22, 2018

sudo: ldap_start_tls_s(): Connect error

A quick hint for FreeBSD users of sudo that authorize via LDAP. If you're getting the following message when running sudo:

sudo: ldap_start_tls_s(): Connect error

associated with this error message in the logs:

sudo: in openpam_check_error_code(): pam_sm_authenticate(): unexpected return value 27

Check that your ldap.conf TLS parameters are correct! In my case, Ansible pushed a bunch of pending config changes (and an OS update) to a neglected host, one of which included moving the CA certificate file, but failed to update the ldap.conf file. I chased my tail for a bit, thinking the issue was with nslcd.conf.

You may also notice a corresponding error in the log of the LDAP server. In the case of slapd:

slapd[40731]: conn=4892528 fd=219 closed (TLS negotiation failure)

Monday, September 18, 2017

VMFS version confusion and FreeBSD UNMAP

I recently started moving VMWare guests to a new ESXi 6.5 host, when I experienced an unusual problem. Guests would get tied up in knots, endlessly sending the following error messages to the console.

(da0:mpt0:0:0:0): UNMAP. CDB: 42 00 00 00 00 00 00 02 68 00
(da0:mpt0:0:0:0): CAM status: SCSI Status Error
(da0:mpt0:0:0:0): SCSI status: Busy
(da0:mpt0:0:0:0): Retrying command

After a bunch of Google hits (mostly FreeNAS users) that didn't totally add up for me, I have a theory on the actual cause of the issue. In short, I think that this issue is caused by the presence of the following conditions.


  1.  VMWare vSphere 6.5 host, which supports both VMFS5 and VMFS6.
  2. A FreeBSD guest, using...
  3. A virtual disk that is thinly-provisioned, and stored on a VMFS5 filesystem.
VMFS6 supports the use of the UNMAP command, which allows the guest operating system to inform the hypervisor that it is no longer using a block. When the virtual disk is thin-provisioned, the host can reallocate the block to the pool of available disk space. FreeBSD has included support for this SCSI command for some time.

My theory is this. I think ESXi is lying to the guests. I think that when a thin guest is created on a VMFS5 filesystem, the UNMAP command is still exposed/permitted, even though the underlying filesystem doesn't actually support it. When the guest tries to send the UNMAP command, it gets a bogus response. In the case of FreeBSD, it retries the command perpetually, hanging up the system.

The search hits I found (linked above) mention switching the virtual disk to a SATA/IDE bus as a workaround. I suspect that this works because the UNMAP command does not exist on those buses, preventing the issue from occurring. I believe that the following solutions are less hacky. 

  1. When creating virtual disks for FreeBSD on VMFS5 filesystems, they should always be thick provisioned (I use eager zeroing, I haven't tested lazy). This seems to be the one-size-fits-all solution. It's also worth noting that the ESXi installer uses VMFS5 on the system disk, with no apparent way to use VMFS6.
  2. If you must use thin-provisioning, make sure that it is on a VMFS6 filesystem. I have not tested this extensively, but it seems to work.
  3. Thin-provisioned guests also seem to behave normally on NFS-backed storage.
In my testing, I have not had any issues on guests provisioned per #1.

Saturday, August 26, 2017

Display the listening ports on CentOS 7

It's been a while since I was serious about Linux, but the fun new goodies have lured me back towards the fold. A lot of things have changed over the last few years, some for the better, some not, but that's way beyond the scope here.

One thing that has been removed (at least from CentOS) is netstat. I'm going to call that a win, because invoking netstat always required a trip to the man page, aside from the trusty netstat -lnp.  The problem I have is that CentOS (and presumably RHEL) removed netstat, but decades worth of Google indexing has entrenched netstat as the blessed method of pulling a list of listening sockets.

Installing net-tools seems like the wrong approach, there must be a better way...and there is! Buried in a blog post, I found a conversion reference for netstat functionality. From this, I learned that ss is the replacement for the functionality I need. As an added bonus, it appears that the basic syntax is similar to my beloved sockstat.

For example, my oft-used "what's listening on the network":

ss -l46