Using FreeBSD as a DomU on XenServer

I am investigating a move away from VMWare ESXi to $hypervisor, as a part of our new data center build. The primary candidates I am looking at are XenServer, and an XAPI stack on Debian. Citrix doesn't officially support FreeBSD as a DomU; at least not as of 6.2. However, FreeBSD seems to pretty happily install as an HVM DomU, if you specify the "Other install media" template from the XenCenter "New VM" wizard.

I installed XenServer 6.2 on an old Dell 2950, with a pair of dual-core Xeons, 8GB RAM and 6x15k SAS drives in a RAID10 configuration. As an aside, I get garbage output on the boot prompt when I boot the host...a problem happily solved by mashing the Enter key in frustration until XenServer began booting.

As mentioned above, I installed a FreeBSD 9.1-RELEASE amd64 guest without much difficulty. I then wanted to see how the performance stacked up against our existing ESXi 5.0 infrastructure. As a crude benchmark, I ran a time portsnap extract on the XenSever guest, another FreeBSD guest on a similarly spec'd 1950 running ESXi, and on a 2950 with FreeBSD installed natively. The wall times were as follows.

1. XenServer FreeBSD DomU: 12:20
2. ESXi FreeBSD guest: 6:30
3. Raw hardware: 5:28

I was rather disappointed to see XenServer fare so poorly against VMWare. Not all was lost though, because my XenServer guest was running in HVM mode. I expected that I would see some performance improvement by using the Paravirtualized drivers available in FreeBSD. To summarize the FreeBSD Wiki, full PV support is only available on i386, but amd64 can use the PV block device and network interfaces. I tried building an PV image and shipping it over to the XenServer host, without success. I was unable to get XenServer to even attempt to boot my image.

I went back to my HVM DomU and installed the 9.1-p7 XENHVM kernel. On reboot, the guest hangs immediately after detecting the CDROM drive. For several minutes it displays a message about xenbusb_nop_confighook_cb timeout periodically, then nothing. Some googling suggests that this is a known issue, with a workaround of removing the virtual CD device, as indicated in this thread. I removed the CDROM device by following these instructions, and the guest now boots happily. With the PV drivers, your virtual disk device is named "ad0" by FreeBSD. The network interface becomes "xn0". Because of these changes, you will want to update your guest /etc/fstab file, and probably the network configuration in /etc/rc.conf. Running the portsnap benchmark on the updated guest yields a time of 8:37...a 43% improvement over the full HVM DomU, but still lagging behind VMWare.

More experimentation is required to tell if more performance can be squeezed out of XenServer, or whether the live migration features justify the performance drop.

Using FreeBSD loopback interfaces with BIRD

Why go for the simple solution, when you can first spend hours tearing out your hair in frustration?

I've been working on a new data center deployment, and getting my fingers back into the networking realm; a welcome change. This includes my first OSPFv3 deployment, and we're using BIRD. For the most part, I have been very happy with BIRD for OSPF and BGP; though I have run into some quirks.

The quirk on my mind at this moment is with loopback interfaces. The Cisco way of doing things seems to be to run iBGP sessions between loopback addresses. The rationale is that if you use an interface address, and that interface goes down, your iBGP session goes with it, as that address becomes unreachable. So you use a loopback interface, which is always up. The addresses on the loopback are advertised via an IGP, facilitating the iBGP connection.

For better or worse, I decided to follow the herd, and go with a loopback interface. For IPv4, this was pretty straightforward. Configure the loopback in the OS, add it to bird.conf as a stub interface, and good to go. For reference, here are the bits to do so on FreeBSD.

# /etc/rc.conf

cloned_interfaces="lo1"

ifconfig_lo1="inet W.X.Y.Z/32"

# /usr/local/etc/bird.conf

protocol ospf {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

authentication cryptographic;

password "password";

};

interface "lo1", "vlan1001" { stub; };

};

}

And since the proof is in the pudding (or output)...

bird> show route for W.X.Y.Z

W.X.Y.Z/32 via A.B.C.D on vlan500 [ospf1 09:08] * I (150/5) [W.X.Y.Z]

When I went to configure OSPF for our IPv6 allocation, things didn't go quite so smoothly. I used the following similar configuration for the v6 BIRD configuration.

# /etc/rc.conf

ifconfig_lo1_ipv6="inet6 2620:W:X:Y::Z/128"

# /usr/local/etc/bird6.conf

protocol ospf ospf_v6 {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

# Authentication is not supported by OSPFv3, supposed to be IPSec AH authenticated.

#authentication cryptographic;

#password "password";

};

interface "lo1", "vlan1001" { stub; };

};

}

With this configuration in place, I realized that my IPv6 loopback address was not being advertised. Examining the logs, BIRD was quite happy to tell me that it had filtered out that route. WTF? After a bunch of time wasted searching Google and throwing shit at the wall, my loopback address was still not working. I finally stumbled on this mailing list thread, where I learned that using a loopback is NOT an expected configuration; at least in the eyes of the developers. Furthermore, the fact that it is working in IPv4 was surprising, and perhaps a bug. The reason that BIRD is denying my lo1 IP is that there is no link-local IP on the interface as well. Without starting a discussion on whether Cisco or BIRD is more right, I'll just say I was *((# *grumble* *f'n BIRD*.

I jumped through a few more hoops, and finally discovered that I could use a tap interface in lieu of a loopback. It would generate a link-local address, OSPF would advertise it, and iBGP happiness filled the kingdom. Nevermind that it is about as hacky as you can get. For what it's worth, here is the rc.conf goo to make it happen.

# /etc/rc.conf

# DON'T USE THIS! IT'S HACKY AND EVERYONE WILL LAUGH AT YOU.

cloned_interfaces="tap0"

ifconfig_tap0_ipv6="inet6 2620:W:X:Y::Z/128 -ifdisabled"

I slept on it. When I woke up, I had some OSPF fixes on my mind for the ASAs (that's another raar story). I was poking around a little more when I read and was reminded about the BIRD stubnet configuration directive. In a nutshell, BIRD will always advertise a stubnet route...perfect! Changed the configuration to support this, and life is good again.

# /etc/rc.conf

ifconfig_lo1_ipv6="inet6 2620:W:X:Y::Z/128"

# /usr/local/etc/bird6.conf

protocol ospf ospf_v6 {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

};

interface "vlan1001" { stub; };

stubnet 2620:W:X:Y::Z/128;

};

}

and the proof!

bird> show route for 2620:W:X:Y::Z/128

2620:W:X:Y::Z/128 via fe80::225:90ff:fe6b:f52c on vlan7 [ospf_v6 09:12] * I (150/15) [W.X.Y.Z]

Configuring NFSv4 on FreeBSD

After much monkeying around, I finally have NFSv4 running on FreeBSD. Since there seems to be a lack of documentation specifically for NFSv4, here is my take on it. The nfsv4 man page does tell you what you need to know in order to get a basic client and server running, is was clear to me only after I had it working. In my setup, my server is running FreeBSD 9.1, and my client runs 9.0.

Server Configuration

1. In /etc/rc.conf, add the following lines
   
   

   nfs_server_enable="YES"
   nfsv4_server_enable="YES"
   nfsuserd_enable="YES"

   
   
2. Start the NFS daemons by running
   
   

   /etc/rc.d/nfsd start

   
   
3. Open /etc/exports in your preferred editor. Add a "V4:" line to the file, specifying the root of your NFSv4 tree. There are two choices at this point. Option 1 is to place your NFSv4 root at a location other than the root of the server filesystem. With this option, you can create an arbitrary NFS tree for clients to attach to, independent of how data is actually situated on the filesystem(s). Nullfs mounts may be used to include outside directories in the NFS root. Option 2 is to make the NFSv4 root the actual root of the system. This preserves the behavior of old NFS implementations. Regardless of where you put your V4 root, you must also add export lines, in the same style as NFSv3. The exports man page can be helpful here, and discusses the security implications of using NFSv4.
   
   

   # Option 1
   V4: /nfsv4 -network=10.0.0.0 -mask=255.255.255.192
   /nfsv4/ports -maproot=root: -network=10.0.0.0 -mask=255.255.255.192
   
   # Option 2
   V4: / -network=10.0.0.0 -mask=255.255.255.192
   /usr/pxe/ports -maproot=root: -network=10.0.0.0 -mask=255.255.255.192

   
   
4. Reload the exports file by signalling mountd
   
   

   killall -HUP mountd

Client Configuration

Clients should now be able to mount the exported filesystem using the following commands, corresponding to the NFSv4 root options specified above. Notice that with option 1, the remote path omits the /nfsv4 prefix of the server.

# Option 1
mount -t nfs -o nfsv4 server:/ports /mnt

# Option 2
mount -t nfs -o nfsv4 server:/usr/pxe/ports /mnt

Errors

If you get the following error when trying to mount from the client, don't be fooled:

mount_nfs: /mnt, : No such file or directory

This may indicate that you have misspelled the remote path in your mount command. It may also indicate that you have an error in your exports file, or that your exports file is not configured the way you think it is. Go back and read step 3 of the Server Configuration.

Constant load on R300 running FreeBSD 9.1

An issue cropped up on the R300 hardware with 9.1. The HPET timer selected on this hardware causes a constant load average of approximately .50 (idle). This seems to be corrected by forcing the LAPIC time source. The following sysctl can be set via the command line utility, or in /etc/sysctl.conf:

sysctl kern.eventtimer.timer=LAPIC

More information here.

FreeBSD 9.0 newnfs slow throughput

We recently updated our production systems to FreeBSD 9.0. Almost immediately afterwards, we began getting reports of crumby performance on NFS mounts. After following a lot of false leads, I finally traced the issue to the new NFS code in FreeBSD, appropriately named 'newnfs'. I remounted NFS using the 'oldnfs' filesystem type, and performance immediately went back to its previous levels; a 55% increase by my most conservative testing, with users reporting orders of magnitude improvement for some jobs.

Using old NFS is a band-aid, not a solution. The FreeBSD NFS wizards desire to prune the old code for FreeBSD 10. I set to work, trying to determine the reason for the poor throughput on newnfs. I discovered that newnfs uses a default rsize and wsize value of 65535 (64k). The old NFS implementation uses a 32768 value. Remounting newnfs with the options rsize=32768,wsize=32768 results in performance very similar to that of oldnfs.

Sorting BackupPC log file by file size.

The following quick script can be used to parse a BackupPC transfer log file and sort the list of files transferred in descending order by size.

/backuppc/pc/pc.example.com-> BackupPC_zcat XferLOG.34.z | sed "s/create d/created/" | awk '{print $4,  "\t", $5}' | grep -vE 'bytes$' | sort -rn | less

Migrating to NFSv4 on FreeBSD

I was recently tasked with configuring an existing FreeBSD NFSv3 server to allow NFSv4 mounts. We don't care about v4 security at this point, just that the v4 protocol is being used. The FreeBSD handbook does not yet have any content discussing the configuration of a v4 server, and I was not able to find any good resources on Google. The nfsv4 man pages were informative, but not helpful. Here is the nutshell version of what I finally figured out.

On the NFS server, the following changes need to be made. These notes assume that you already have a working NFSv3 server.

# /etc/rc.conf changes

nfsv4_server_enable="YES"
nfsuserd_enable="YES"

# /etc/exports changes
# The 'V4' line defines the root of your NFSv4 tree. As I
# understand it, the paths you want exported must also be listed
# in the exports file, the same way they are for v3.
V4: /
/home -alldirs -maproot=root: -network=10.0.0.0/24

To mount the filesystem on the client using NFSv4, you must add the nfsv4 option to the mount command.

root@client:~-> mount -t nfs -o nfsv4 server:/home /mnt
root@client:~-> mount
server:/home on /mnt (nfs, nfsv4acls)