Understanding resolvconf behavior on pxe-booted hosts.

I have been learning more than I ever cared to know about resolvconf, and what happens when you use it on a host with a read-only root filesystem. I have been preparing to roll out pxe-boot virtual machines at a second location. The PXE image has the following characteristics.

* NFS-mounted, read-only / filesystem.
* Local writeable disk for swap and /var.
* Unionfs, md-backed /etc (non-persistent, r/w)

I noticed that on the initial boot of a new VM, /etc/resolv.conf would be written correctly. However, all subsequent boots never see the NFS-supplied resolv.conf updated. After a frustrating afternoon of digging, I determined why resolvconf appears to stop working.

Resolvconf stores state data in /var/run/resolvconf. When dhclient is run for an interface, the dhclient-script script calls resolvconf with DNS particulars, resolvconf looks in the interfaces/ sub-directory for an entry named after the interface. If the file does not exist, or does not match the domain/nameserver options received by resolvconf, a new file is written, and appropriate changes are made to /etc/resolv.conf. If the options match what resolvconf already has, no changes are made. The below output shows the contents of the interfaces/ directory on my pxe host.

> ll /var/run/resolvconf/interfaces/
total 8
-rw-r--r--  1 root  wheel  80 Mar 13 18:00 vmx0:dhcp4
-rw-r--r--  1 root  wheel  76 Mar 13 18:00 vmx1

The problem with my pxe hosts lies in the volatile /etc. Every time the host reboots, the modified contents of /etc vanish. /etc/resolv.conf is replaced with the copy from NFS. In my case, this copy reflects the nameservers at the "original" datacenter. Since the state directory for resolvconf exists on the persistent /var, resolvconf sees the old [unchanged] lease data, and assumes everything is peachy with the resolv.conf file.

I don't need the extra features of resolvconf, so I can solve the problem by disabling it. I created a file in the pxe image, /etc/dhclient-enter-hooks, that contains the following.

resolvconf_enable=NO

My initial, more complicated fix, was to create an rc script to re-initialize the resolvconf state directory on every boot. This also worked flawlessly.

#!/bin/sh
#
# Clean out the contents of the resolvconf state directory. Otherwise,
# /etc/resolv.conf never gets updated after the initial boot of a new pxe host.
#
# BEFORE: netif
# AFTER: FILESYSTEMS
# PROVIDE: clean_resolvconf

echo -n "Cleaning out resolvconf state directory: "

/sbin/resolvconf -I

if [ $? ]; then
echo "OK"
else 
echo "FAILED"
fi

Using CARP with VMWare ESXi

If you want to use CARP on your VMWare guest VMs, you will probably find that it doesn't work out of the box. This is due to ESXi rejecting promiscuous mode on the virtual switch by default. To enable promiscuous mode, go to the Network configuration section for the host (in vSphere Client), and click properties for the vSwitch. Edit the properties for the vSwitch, and change the setting of "Promiscuous Mode" to Accept under the "Security" tab.

For bonus points, if you are using NIC Teaming on ESXi (even with just a standby adapter), you will find that your CARP interfaces always remain in BACKUP state, and your logs fill with the following messages.

Nov 12 11:25:51  kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Nov 12 11:25:51  kernel: carp0: link state changed to DOWN
Nov 12 11:25:54  kernel: carp0: link state changed to UP

This is because ESXi is rebroadcasting CARP advertisements that come back down the other members of the team. To correct this, you need to dig into the Advanced Settings, under Software. Change Net.ReversePathFwdCheckPromisc to 1. Annoyingly, you will need to reboot the host to affect these changes, but it works.

Using FreeBSD as a DomU on XenServer

I am investigating a move away from VMWare ESXi to $hypervisor, as a part of our new data center build. The primary candidates I am looking at are XenServer, and an XAPI stack on Debian. Citrix doesn't officially support FreeBSD as a DomU; at least not as of 6.2. However, FreeBSD seems to pretty happily install as an HVM DomU, if you specify the "Other install media" template from the XenCenter "New VM" wizard.

I installed XenServer 6.2 on an old Dell 2950, with a pair of dual-core Xeons, 8GB RAM and 6x15k SAS drives in a RAID10 configuration. As an aside, I get garbage output on the boot prompt when I boot the host...a problem happily solved by mashing the Enter key in frustration until XenServer began booting.

As mentioned above, I installed a FreeBSD 9.1-RELEASE amd64 guest without much difficulty. I then wanted to see how the performance stacked up against our existing ESXi 5.0 infrastructure. As a crude benchmark, I ran a time portsnap extract on the XenSever guest, another FreeBSD guest on a similarly spec'd 1950 running ESXi, and on a 2950 with FreeBSD installed natively. The wall times were as follows.

1. XenServer FreeBSD DomU: 12:20
2. ESXi FreeBSD guest: 6:30
3. Raw hardware: 5:28

I was rather disappointed to see XenServer fare so poorly against VMWare. Not all was lost though, because my XenServer guest was running in HVM mode. I expected that I would see some performance improvement by using the Paravirtualized drivers available in FreeBSD. To summarize the FreeBSD Wiki, full PV support is only available on i386, but amd64 can use the PV block device and network interfaces. I tried building an PV image and shipping it over to the XenServer host, without success. I was unable to get XenServer to even attempt to boot my image.

I went back to my HVM DomU and installed the 9.1-p7 XENHVM kernel. On reboot, the guest hangs immediately after detecting the CDROM drive. For several minutes it displays a message about xenbusb_nop_confighook_cb timeout periodically, then nothing. Some googling suggests that this is a known issue, with a workaround of removing the virtual CD device, as indicated in this thread. I removed the CDROM device by following these instructions, and the guest now boots happily. With the PV drivers, your virtual disk device is named "ad0" by FreeBSD. The network interface becomes "xn0". Because of these changes, you will want to update your guest /etc/fstab file, and probably the network configuration in /etc/rc.conf. Running the portsnap benchmark on the updated guest yields a time of 8:37...a 43% improvement over the full HVM DomU, but still lagging behind VMWare.

More experimentation is required to tell if more performance can be squeezed out of XenServer, or whether the live migration features justify the performance drop.

Using VLANs with Virtualbox

After a number of hours wasted trying to figure out why my VLANs stopped working, it's probably a good idea to make note of what I found.

I have a lot of different Virtualbox VMs created that I use for testing different things. A while back I created a VM image to act as a PPPoE server (Debian squeeze guest running on a lenny host). For reasons that made sense to me at the time, I created VLANs on the guest operating system rather than on the host system (and attaching virtual network adaptors to the guest VM). I hadn't used this VM in several months until today; when I fired up the VM for some PPPoE testing. Unsurprisingly, the VM didn't have any network connectivity on boot. I figured it would probably take some finagling to get things working exactly the way they were working before. However the topology was correct, the switch set up properly, and still no PPPoE.

After several hours of screwing around with different VirtualBox versions, different combinations of host/guest VLAN configurations, etc. I identified the following behaviors:

1. On a clean reboot of the host, the VLANs on the virtual machine work fine.
2. As soon as I add a VLAN to the ethernet interface on the host, connectivity to the guest fails.
3. If I remove the VLAN from the host, guest connectivity is restored.

The wheels started spinning and I remembered that at some point in the past I had a bridge interface configured on my host machine. I had recently removed the bridge in order to simplify my configuration. Some quick testing confirmed that when my host machine is configured with a virtual bridge interface, VLANs on the host can happily coexist with VLANs on the guest. My network configuration necessary to set this up is as follows.

1. Add a bridge interface br0 to the host.
2. Add ethernet adaptor eth0 to bridge br0.
3. Add the host VLAN interfaces to bridge br0.
   
4. Configure appropriate IP addressing for the host to br0.
5. Attach the VirtualBox VM network interface to eth0.

In this configuration, both the host and the guest are able to create tagged VLAN interfaces without conflicting.

LVM + XFS + VirtualBox - Resizing a guest partition

I've been doing a lot of work with virtualization lately and I am working on what I hope will become the great solution I've always wanted. The current configuration on the Host machine is a RAID0 array, with LVM logical volumes on top of that for each of my virtual machines. Each LV used for a Virtual Machine is formatted in XFS; from what I have read, it offers better performance with large files.

My Logical Volumes created for my VMs are created with a conservative sized disk. This allows plenty of free space on the LVM VG for future VMs, snapshots, scratch volumes, and expansion of existing LVs. That's right, it is possible to increase the size of all the disk components in this mess (Logical Volume, XFS Partition, Virtual Machine VDI, VM filesystem). For the most part this is a pretty painless process. The only real nasty part that involves downtime is resizing the VDI, which isn't really resizing at all. Rather, we will create a new larger VDI and copy the old disk over.

Here is the process:

1. Resize the logical volume on the host machine (if necessary).

lvextent -L+3G /dev/mapper/vg--vm-lv--ftp (extends ftp lv by 3GB)

2. Resize the filesystem on the host machine (if you resized the lv).

xfs_growfs /var/vm/ftp

3. Create a scratch lv on the host machine. This will be used to hold the resized clone, so make it large enough.

lvcreate -L 22G -n scratch vg-vm

4. Create a filesystem on scratch volume.

mkfs.xfs /dev/mapper/vg--vm-scratch

5. Create a mount point and mount the filesystem. Give vboxuser ownership.

mkdir /tmp/scratch

mount /dev/mapper/vg--vm-scratch /tmp/scratch

chown vboxuser /tmp/scratch

6. Change to vboxuser and create a new VDI of the desired size.

su vboxuser

VBoxManage createhd --filename /tmp/scratch/scratch.vdi --size 20480

7. Register scratch disk in virtualbox and verify

VBoxManage openmedium disk /tmp/scratch/scratch.vdi

VBoxManage list hdds

8. Register the gparted iso

VBoxManage openmedium dvd /root/gparted-live-0.4.5-2.iso

9. Shutdown the VM.
10. Add new VDI to VM and mount the gparted disk.

VBoxManage modifyvm "ftp" --hdb /tmp/scratch/scratch.vdi --dvd /root/gparted-live-0.4.5-2.iso

11. Start the VM, gparted should boot. Follow the default prompts to get to the desktop.

VBoxHeadless -startvm "ftp"

12. Open a prompt. Verify that the source (old) disk is hda and the destination disk (new) is hdb. Then use dd to copy hda to hdb. When finished, reread the partition table on the new disk.

sfdisk -l

dd if=/dev/hda of=/dev/hdb

hdparm -z /dev/hdb

13. Open gparted and use it to grow the disk partition you need to be bigger. Apply the changes, then shutdown the VM.
14. Remove the dvd and both drives. Then re-add the new VDI as hda. Startup the VM to verify that it boots cleanly.

VBoxManage modifyvm "ftp" --dvd none --hdb none --hda none

VBoxManage modifyvm "ftp" --hda /tmp/scratch/scratch.vdi

VBoxHeadless -startvm "ftp"

15. Set hda to none on VM. Close the two hard drive images.

VBoxManage modifyvm "ftp" --hda none

VBoxManage closemedium disk /tmp/

VBoxManage closemedium disk /var/vm/ftp/ftp.vdi

16. Move the scratch disk to the resized lv.

mv /tmp/scratch/scratch.vdi /var/vm/ftp/ftp.vdi

17. Re-open the new disk and add back into the VM

VBoxManage openmedium disk /var/vm/ftp/ftp.vdi

VBoxManage modifyvm "ftp" --hda /var/vm/ftp/ftp.vdi

18. Start VM

VBoxHeadless -startvm "ftp"

19. Unmount the scratch partition on the host machine and remove the lv to tidy up.

umount /tmp/scratch/

lvremove /dev/mapper/vg--vm-scratch