Understanding resolvconf behavior on pxe-booted hosts.

I have been learning more than I ever cared to know about resolvconf, and what happens when you use it on a host with a read-only root filesystem. I have been preparing to roll out pxe-boot virtual machines at a second location. The PXE image has the following characteristics.

* NFS-mounted, read-only / filesystem.
* Local writeable disk for swap and /var.
* Unionfs, md-backed /etc (non-persistent, r/w)

I noticed that on the initial boot of a new VM, /etc/resolv.conf would be written correctly. However, all subsequent boots never see the NFS-supplied resolv.conf updated. After a frustrating afternoon of digging, I determined why resolvconf appears to stop working.

Resolvconf stores state data in /var/run/resolvconf. When dhclient is run for an interface, the dhclient-script script calls resolvconf with DNS particulars, resolvconf looks in the interfaces/ sub-directory for an entry named after the interface. If the file does not exist, or does not match the domain/nameserver options received by resolvconf, a new file is written, and appropriate changes are made to /etc/resolv.conf. If the options match what resolvconf already has, no changes are made. The below output shows the contents of the interfaces/ directory on my pxe host.

> ll /var/run/resolvconf/interfaces/
total 8
-rw-r--r--  1 root  wheel  80 Mar 13 18:00 vmx0:dhcp4
-rw-r--r--  1 root  wheel  76 Mar 13 18:00 vmx1

The problem with my pxe hosts lies in the volatile /etc. Every time the host reboots, the modified contents of /etc vanish. /etc/resolv.conf is replaced with the copy from NFS. In my case, this copy reflects the nameservers at the "original" datacenter. Since the state directory for resolvconf exists on the persistent /var, resolvconf sees the old [unchanged] lease data, and assumes everything is peachy with the resolv.conf file.

I don't need the extra features of resolvconf, so I can solve the problem by disabling it. I created a file in the pxe image, /etc/dhclient-enter-hooks, that contains the following.

resolvconf_enable=NO

My initial, more complicated fix, was to create an rc script to re-initialize the resolvconf state directory on every boot. This also worked flawlessly.

#!/bin/sh
#
# Clean out the contents of the resolvconf state directory. Otherwise,
# /etc/resolv.conf never gets updated after the initial boot of a new pxe host.
#
# BEFORE: netif
# AFTER: FILESYSTEMS
# PROVIDE: clean_resolvconf

echo -n "Cleaning out resolvconf state directory: "

/sbin/resolvconf -I

if [ $? ]; then
echo "OK"
else 
echo "FAILED"
fi

dhclient exits chroot on FreeBSD 10.0

I've booted my first pxe FreeBSD 10.0 image, and discovered that dhclient, of all things, doesn't seem to work. Running dhclient from the command-line after boot results in the following output:

WARNING dhclient failed to start
chroot
exiting

Some searching turns up a thread on the FreeBSD forums. A missing /var/empty directory is to blame. It should be owned by root, with permissions of 755.

Using CARP with VMWare ESXi

If you want to use CARP on your VMWare guest VMs, you will probably find that it doesn't work out of the box. This is due to ESXi rejecting promiscuous mode on the virtual switch by default. To enable promiscuous mode, go to the Network configuration section for the host (in vSphere Client), and click properties for the vSwitch. Edit the properties for the vSwitch, and change the setting of "Promiscuous Mode" to Accept under the "Security" tab.

For bonus points, if you are using NIC Teaming on ESXi (even with just a standby adapter), you will find that your CARP interfaces always remain in BACKUP state, and your logs fill with the following messages.

Nov 12 11:25:51  kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Nov 12 11:25:51  kernel: carp0: link state changed to DOWN
Nov 12 11:25:54  kernel: carp0: link state changed to UP

This is because ESXi is rebroadcasting CARP advertisements that come back down the other members of the team. To correct this, you need to dig into the Advanced Settings, under Software. Change Net.ReversePathFwdCheckPromisc to 1. Annoyingly, you will need to reboot the host to affect these changes, but it works.

Using BIRD to route over OpenVPN tunnels.

OpenVPN tunnels, good. BIRD routing daemon, great. OSPF on OpenVPN tunnels, headache. The combination of OpenVPN and BIRD routing daemon for OSPF is nothing new for me. I've been using it pretty happily for over a year. However, it always seemed as though something was a bit precarious. As I have started to scale the implementation, I've realized that I needed a rethink of my strategy.

The original design used a /31 subnet on each tunnel, with each endpoint using an address. Logically, it made perfect sense. The catch is that OSPF does not automatically advertise the IP address of the tunnels (a quirk of OpenVPN, BIRD, or both). Traffic would flow through the routers, but the remote tunnel address would be unreachable. I solved this by adding a stubnet x.x.x.x/31 directive to the OSPF  configuration at one of the endpoints. At the time this worked fine.

As I brought a third site online, with multiple VPN tunnels between each site, the original design quickly broke down. The /31s were no longer consistently routable. I remedied this by changing the /31 stubnet to a /32, with each router advertising its own IP addresses. Life was briefly good, until I realized that restarting the OpenVPN tunnels would fail. OpenVPN would log the error, "router-a openvpn_1199[18976]: FreeBSD ifconfig failed: external program exited with error status: 1" The following message was also logged, "ifconfig: ioctl (SIOCAIFADDR): File exists" From the depths of my memory, I recalled that this error occurs when you attempt to configure an interface with an address that already exists in the routing table. Since BIRD was already advertising the /32 host IP, trying to address the tunnel would fail.

Some creative restarting of OpenVPN and BIRD resolved the problem, but is an unacceptable solution for production. I turned to the BIRD mailing list with my situation, and very quickly received a response from one of the BIRD developers. He suggested a number of ideas, the most promising of which is to dedicate a subnet to each router, as a pool from which to draw addresses for local tunnel endpoints on that specific host. The subnet is then configured as a stubnet by BIRD. The ends of a given tunnel may not be remotely close to each other, but this is fine because a tunnel is a point-to-point link. Tunnels can be restarted at will, without contention from existing routes in the table.

The diagram below shows an example topology. Notice that the endpoint of each tunnel is in a completely different network. For example, on the tunnel between router-a and router-b, router-a uses an IP of 192.168.0.0, and router-b uses 10.0.0.0. It doesn't matter, it works just fine. With the given stubnet configured in the bird.conf for each router, all tunnel endpoints are reachable. The network and broadcast addresses for each router's stubnet are also useable, as shown in the example.

Fixing SSH timeouts on the ASA

I spent a bunch of time getting my head around class maps, policy maps, and service policies, in an effort to correct the issue of idle SSH connections being timed out after an hour (the default idle timeout for TCP connections on the ASA at version 9.1). The documentation is a confusing web of headache, but I found this blog post to be useful reading. My solution was to leave the timeout unchanged, but to enable Dead Connection Detection for SSH connections. Essentially, when an idle SSH connection hits the threshold, the ASA forges a packet to the endpoints to verify that the socket is still open. If there is a response, the idle timer is reset.

access-list ssh_ports remark access list to id ssh traffic for the ssh_ports class map
access-list ssh_ports extended permit tcp any any eq ssh 
access-list ssh_ports extended permit tcp any any eq 2222

class-map ssh_traffic
 description identify SSH traffic, so we can apply policy
 match access-list ssh_ports

policy-map generic_interface_policy
 class ssh_traffic
  set connection timeout dcd 

service-policy generic_interface_policy interface outside
service-policy generic_interface_policy interface inside

Cisco ASA Remote Access configuration for Mac OS X

I spent the day fighting to get a remote access IPSec connection set up as follows:

• ASA 5515-X, running version 9.1.
• ASA network interfaces are already configured.
• IPSec clients are assigned addresses from the range 123.0.0.199-201.
• Client is running OS X 10.8.4 Mountain Lion.
• Client is using the built-in OS X IPSec client.
• Client IP is private, behind NAT, with a DHCP-assigned WAN IP.
• After connecting, client should be able to reach the internal networks 123.0.0.128/26, 123.0.0.192/27.
• All other traffic is not sent across the VPN.

The following configuration should be added to the ASA:

ip local pool REMOTE_ACCESS_POOL 123.0.0.199-123.0.0.201

management-access inside

access-list REMOTE_ACCESS_SPLIT_TUNNEL remark The corporate network behind the ASA.

access-list REMOTE_ACCESS_SPLIT_TUNNEL standard permit 123.0.0.128 255.255.255.192 
access-list REMOTE_ACCESS_SPLIT_TUNNEL standard permit 123.0.0.192 255.255.255.224 

crypto ipsec ikev1 transform-set REMOTE_ACCESS_TS esp-aes-256 esp-sha-hmac 
crypto dynamic-map REMOTE_ACCESS_DYNMAP 1 set ikev1 transform-set REMOTE_ACCESS_TS
crypto map REMOTE_ACCESS_MAP 1 ipsec-isakmp dynamic REMOTE_ACCESS_DYNMAP
crypto map REMOTE_ACCESS_MAP interface outside

crypto ikev1 enable outside
crypto ikev1 policy 1
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 7200

group-policy REMOTE_ACCESS_GP internal
group-policy REMOTE_ACCESS_GP attributes

 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value REMOTE_ACCESS_SPLIT_TUNNEL

username hunter password **** encrypted
tunnel-group REMOTE_ACCESS_TUNNELGRP type remote-access
tunnel-group REMOTE_ACCESS_TUNNELGRP general-attributes
 address-pool REMOTE_ACCESS_POOL
 default-group-policy REMOTE_ACCESS_GP
tunnel-group REMOTE_ACCESS_TUNNELGRP ipsec-attributes
 ikev1 pre-shared-key *****

For explanation of what all this does, I recommend reading the following Cisco docs. It is worth noting that this configuration does not work with Windows 7/8, which use IKEv2 instead of v1.

Remote Access VPN configuration
Split tunnel configuration
Management access over a VPN tunnel


The configuration for the built-in OS X IPSec client is described in the following doc. One gotcha I ran into (which is clearly stated in the document) is that the tunnel-group name must be specified in the 'Group Name' field on the Mac. In the case of the above configuration, the group name is REMOTE_ACCESS_TUNNELGRP.


VPN Clients for Mac OS X FAQ

Using FreeBSD as a DomU on XenServer

I am investigating a move away from VMWare ESXi to $hypervisor, as a part of our new data center build. The primary candidates I am looking at are XenServer, and an XAPI stack on Debian. Citrix doesn't officially support FreeBSD as a DomU; at least not as of 6.2. However, FreeBSD seems to pretty happily install as an HVM DomU, if you specify the "Other install media" template from the XenCenter "New VM" wizard.

I installed XenServer 6.2 on an old Dell 2950, with a pair of dual-core Xeons, 8GB RAM and 6x15k SAS drives in a RAID10 configuration. As an aside, I get garbage output on the boot prompt when I boot the host...a problem happily solved by mashing the Enter key in frustration until XenServer began booting.

As mentioned above, I installed a FreeBSD 9.1-RELEASE amd64 guest without much difficulty. I then wanted to see how the performance stacked up against our existing ESXi 5.0 infrastructure. As a crude benchmark, I ran a time portsnap extract on the XenSever guest, another FreeBSD guest on a similarly spec'd 1950 running ESXi, and on a 2950 with FreeBSD installed natively. The wall times were as follows.

1. XenServer FreeBSD DomU: 12:20
2. ESXi FreeBSD guest: 6:30
3. Raw hardware: 5:28

I was rather disappointed to see XenServer fare so poorly against VMWare. Not all was lost though, because my XenServer guest was running in HVM mode. I expected that I would see some performance improvement by using the Paravirtualized drivers available in FreeBSD. To summarize the FreeBSD Wiki, full PV support is only available on i386, but amd64 can use the PV block device and network interfaces. I tried building an PV image and shipping it over to the XenServer host, without success. I was unable to get XenServer to even attempt to boot my image.

I went back to my HVM DomU and installed the 9.1-p7 XENHVM kernel. On reboot, the guest hangs immediately after detecting the CDROM drive. For several minutes it displays a message about xenbusb_nop_confighook_cb timeout periodically, then nothing. Some googling suggests that this is a known issue, with a workaround of removing the virtual CD device, as indicated in this thread. I removed the CDROM device by following these instructions, and the guest now boots happily. With the PV drivers, your virtual disk device is named "ad0" by FreeBSD. The network interface becomes "xn0". Because of these changes, you will want to update your guest /etc/fstab file, and probably the network configuration in /etc/rc.conf. Running the portsnap benchmark on the updated guest yields a time of 8:37...a 43% improvement over the full HVM DomU, but still lagging behind VMWare.

More experimentation is required to tell if more performance can be squeezed out of XenServer, or whether the live migration features justify the performance drop.