Monday, September 18, 2017

VMFS version confusion and FreeBSD UNMAP

I recently started moving VMWare guests to a new ESXi 6.5 host, when I experienced an unusual problem. Guests would get tied up in knots, endlessly sending the following error messages to the console.

(da0:mpt0:0:0:0): UNMAP. CDB: 42 00 00 00 00 00 00 02 68 00
(da0:mpt0:0:0:0): CAM status: SCSI Status Error
(da0:mpt0:0:0:0): SCSI status: Busy
(da0:mpt0:0:0:0): Retrying command

After a bunch of Google hits (mostly FreeNAS users) that didn't totally add up for me, I have a theory on the actual cause of the issue. In short, I think that this issue is caused by the presence of the following conditions.

  1.  VMWare vSphere 6.5 host, which supports both VMFS5 and VMFS6.
  2. A FreeBSD guest, using...
  3. A virtual disk that is thinly-provisioned, and stored on a VMFS5 filesystem.
VMFS6 supports the use of the UNMAP command, which allows the guest operating system to inform the hypervisor that it is no longer using a block. When the virtual disk is thin-provisioned, the host can reallocate the block to the pool of available disk space. FreeBSD has included support for this SCSI command for some time.

My theory is this. I think ESXi is lying to the guests. I think that when a thin guest is created on a VMFS5 filesystem, the UNMAP command is still exposed/permitted, even though the underlying filesystem doesn't actually support it. When the guest tries to send the UNMAP command, it gets a bogus response. In the case of FreeBSD, it retries the command perpetually, hanging up the system.

The search hits I found (linked above) mention switching the virtual disk to a SATA/IDE bus as a workaround. I suspect that this works because the UNMAP command does not exist on those buses, preventing the issue from occurring. I believe that the following solutions are less hacky. 

  1. When creating virtual disks for FreeBSD on VMFS5 filesystems, they should always be thick provisioned (I use eager zeroing, I haven't tested lazy). This seems to be the one-size-fits-all solution. It's also worth noting that the ESXi installer uses VMFS5 on the system disk, with no apparent way to use VMFS6.
  2. If you must use thin-provisioning, make sure that it is on a VMFS6 filesystem. I have not tested this extensively, but it seems to work.
  3. Thin-provisioned guests also seem to behave normally on NFS-backed storage.
In my testing, I have not had any issues on guests provisioned per #1.

Saturday, August 26, 2017

Display the listening ports on CentOS 7

It's been a while since I was serious about Linux, but the fun new goodies have lured me back towards the fold. A lot of things have changed over the last few years, some for the better, some not, but that's way beyond the scope here.

One thing that has been removed (at least from CentOS) is netstat. I'm going to call that a win, because invoking netstat always required a trip to the man page, aside from the trusty netstat -lnp.  The problem I have is that CentOS (and presumably RHEL) removed netstat, but decades worth of Google indexing has entrenched netstat as the blessed method of pulling a list of listening sockets.

Installing net-tools seems like the wrong approach, there must be a better way...and there is! Buried in a blog post, I found a conversion reference for netstat functionality. From this, I learned that ss is the replacement for the functionality I need. As an added bonus, it appears that the basic syntax is similar to my beloved sockstat.

For example, my oft-used "what's listening on the network":

ss -l46

Friday, August 12, 2016

Arcane Bourne shell behavior. The colon (:)

I finally learned the purpose of starting a command with the colon utility (:) in bourne shell. It expands any arguments, then exits 0. This is handy for hiding pointless error messages resulting from variable substitution. Consider the following example.

$ ${D:=foo}
foo: not found
$ echo $D

The example above sets variable D to "foo" if it is unset or null. It works fine, but causes the annoying error 'foo: not found'. We can suppress that message with the colon.

$ unset D
$ : ${D:=foo}
$ echo $D


Much prettier!

Also see:
Parameter Expansion (and explanation of the colon utility at the bottom of the page)

Thursday, March 3, 2016

LDAP password hashing in PHP, a better way

Reviewing some PHP code related to user password changes, I noticed that the code was using unsalted SHA-1 hashes to store the password in LDAP. This is potentially very bad, if an attacker were able to gain access to the password hashes, because pre-computed rainbow tables for unsalted SHA-1 are pretty common. This has been the basis for a number of high-profile data breaches in recent memory, and I don't want to be on the news.

Looking for a better solution on Google, I quickly discovered the source of the bad code, the OpenLDAP FAQ-o-matic. In this FAQ entry, there are code examples for a variety of languages, many offering higher security. However, the PHP example included only a SHA-1 variant, and was copy-pasted nearly verbatim into my subject code (I wonder how many other web applications have done this very thing).

OpenLDAP has support for a variety of different hashing algorithms, including an optional SHA-2 module, and passthrough to the OpenSSL crypto(3) library. The strongest native cipher supported by OpenLDAP is salted-SHA1 (SSHA), which allows sufficient strength for this application, when used with a decently-large salt. I wrote the following function to generate such hashes from PHP, provided here in hopes that people will quit using unsalted SHA in their web-apps.

 * This is a helper function, returning a Salted SHA-1 hash, suitable for LDAP.
 * OpenLDAP uses a slightly strange scheme for generating these hashes, but it's
 * far better than unsalted SHA. The only limit on salt length appears to be the
 * maximum length of the userPassword attribute in LDAP (128), allowing us to
 * safely use a 64-byte salt, resulting in a 118-byte SSHA. For the curious,
 * this works out to:
 *   6B ({SSHA} tag) + 28B (20B SHA, B64 encoded) + 88B (64B salt, B64 encoded)

function generate_ssha_hash($cleartext)

         * Generate a unique 64-byte salt value for the salt.
         * Use mcrypt_create_iv to generate some random bytes. PHP 7 has a
         * random_bytes() function, and the OpenSSL extension provides
         * openssl_random_pseudo_bytes(), which may be better, but this
         * should work
        $pw_salt = mcrypt_create_iv(64, MCRYPT_DEV_URANDOM);

        // Concatenate and hash password+salt.
        $hashed = sha1($cleartext . $pw_salt, TRUE);

        // Add the tag and encoded hash+salt.
        $hashed = '{SSHA}' . base64_encode($hashed . $pw_salt);

        // Clean up

        return $hashed;

} // generate_ssha_hash

iTunes: An unknown error has occurred (-39)

I have been using the "consolidate library" function of iTunes to move all of my music from my NAS to my Macbook. Everything went fine for a while, then I started getting an error to the effect of "Error copying files: An unknown error has occurred (-39)". I didn't take a screen capture, and the exact verbage escapes me. Google didn't turn up anything about error -39.

Poking around the XML file, there were no obvious problems, but it appeared that the problem files were grouped around a time period, several years ago. Over time I have switched between AFP and SMB (Samba) as my protocol of choice to access my music. I recalled running into issues between the two, with regard to special character handling in filenames. I also recalled that the time period in question may have been when I was using SMB.

On a hunch, I reconnected to the NAS, using SMB instead of AFP. I re-ran the organize/consolidate function, and the remainder of my library is currently consolidating without problem.

Tuesday, May 20, 2014

Deleting locked IPSec SAs from Fortigates

We have had a borked IPSec Phase1 definition in our configuration since the initial configuration. The delete option was grayed out for it, despite the ref count showing 0. I finally had to call Fortinet about it. The engineer I spoke with said that the ref count of 0 doesn't necessarily mean that there aren't any references (what good is the ref count then?). He grabbed a copy of the configuration, and searched for the name of the Phase1. Sure enough, a policy routing entry turned up that we had long forgotten about. After removing this, I was able to delete the Phase1.

Monday, May 19, 2014

Advertising arbitrary routes via OSPF on Fortigate

To be clear, I'm not sure this is the correct way to inject routes into OSPF. That being said, it yields the desired behavior. The situation goes like this...

I have addressed all of the interfaces of my Fortigate (FGT) with subnets of network Additionally, I have some virtual IPs (VIPs) defined that map addresses from to corresponding addresses in For example: ->

This is not a garden-variety configuration, mapping one public subnet to another. The reasons are complex, involving BGP, portable subnets, multiple data centers. The bottom line, I need the FGT to NAT this traffic.

My initial solution to this problem was static routes. However, this becomes difficult to maintain as the network grows in complexity (we're definitely into that territory). What I want to do is advertise a subnet of via OSPF. In the Fortigate, it's not as easy as saying "inject this subnet into OSPF." My solution, create a loopback interface on the FGT, and redistribute the connected subnet into OSPF.
  1. Create a Loopback network interface, with an address in the subnet you want to advertise. It doesn't seem to make a difference what address you use.
        edit "port-loopback"
            set vdom "root"
            set ip
            set type loopback
            set description "Loopback interface used to provide route for portable addresses."
            set snmp-index 28

  2. Create a prefix-list entry that identifies the loopback subnet.
        edit "connected-to-ospf-v4"
            set comments "Define connected routes to export to OSPF"
                config rule
                    edit 10
                        set prefix
                        unset ge
                        unset le

  3. Create a route-map that uses the prefix list.
        edit "rm-connected-to-ospf"
            set comments "Defines IPv4 connected routes to redistribute to OSPF"
                config rule
                    edit 10
                        set match-ip-address "connected-to-ospf-v4"

  4. Configure OSPF to redistribute connected networks.
            config redistribute "connected"
                set status enable
                set routemap "rm-connected-to-ospf"