Configuring NFSv4 on FreeBSD

After much monkeying around, I finally have NFSv4 running on FreeBSD. Since there seems to be a lack of documentation specifically for NFSv4, here is my take on it. The nfsv4 man page does tell you what you need to know in order to get a basic client and server running, is was clear to me only after I had it working. In my setup, my server is running FreeBSD 9.1, and my client runs 9.0.

Server Configuration

1. In /etc/rc.conf, add the following lines
   
   

   nfs_server_enable="YES"
   nfsv4_server_enable="YES"
   nfsuserd_enable="YES"

   
   
2. Start the NFS daemons by running
   
   

   /etc/rc.d/nfsd start

   
   
3. Open /etc/exports in your preferred editor. Add a "V4:" line to the file, specifying the root of your NFSv4 tree. There are two choices at this point. Option 1 is to place your NFSv4 root at a location other than the root of the server filesystem. With this option, you can create an arbitrary NFS tree for clients to attach to, independent of how data is actually situated on the filesystem(s). Nullfs mounts may be used to include outside directories in the NFS root. Option 2 is to make the NFSv4 root the actual root of the system. This preserves the behavior of old NFS implementations. Regardless of where you put your V4 root, you must also add export lines, in the same style as NFSv3. The exports man page can be helpful here, and discusses the security implications of using NFSv4.
   
   

   # Option 1
   V4: /nfsv4 -network=10.0.0.0 -mask=255.255.255.192
   /nfsv4/ports -maproot=root: -network=10.0.0.0 -mask=255.255.255.192
   
   # Option 2
   V4: / -network=10.0.0.0 -mask=255.255.255.192
   /usr/pxe/ports -maproot=root: -network=10.0.0.0 -mask=255.255.255.192

   
   
4. Reload the exports file by signalling mountd
   
   

   killall -HUP mountd

Client Configuration

Clients should now be able to mount the exported filesystem using the following commands, corresponding to the NFSv4 root options specified above. Notice that with option 1, the remote path omits the /nfsv4 prefix of the server.

# Option 1
mount -t nfs -o nfsv4 server:/ports /mnt

# Option 2
mount -t nfs -o nfsv4 server:/usr/pxe/ports /mnt

Errors

If you get the following error when trying to mount from the client, don't be fooled:

mount_nfs: /mnt, : No such file or directory

This may indicate that you have misspelled the remote path in your mount command. It may also indicate that you have an error in your exports file, or that your exports file is not configured the way you think it is. Go back and read step 3 of the Server Configuration.

Constant load on R300 running FreeBSD 9.1

An issue cropped up on the R300 hardware with 9.1. The HPET timer selected on this hardware causes a constant load average of approximately .50 (idle). This seems to be corrected by forcing the LAPIC time source. The following sysctl can be set via the command line utility, or in /etc/sysctl.conf:

sysctl kern.eventtimer.timer=LAPIC

More information here.

FreeBSD 9.0 newnfs slow throughput

We recently updated our production systems to FreeBSD 9.0. Almost immediately afterwards, we began getting reports of crumby performance on NFS mounts. After following a lot of false leads, I finally traced the issue to the new NFS code in FreeBSD, appropriately named 'newnfs'. I remounted NFS using the 'oldnfs' filesystem type, and performance immediately went back to its previous levels; a 55% increase by my most conservative testing, with users reporting orders of magnitude improvement for some jobs.

Using old NFS is a band-aid, not a solution. The FreeBSD NFS wizards desire to prune the old code for FreeBSD 10. I set to work, trying to determine the reason for the poor throughput on newnfs. I discovered that newnfs uses a default rsize and wsize value of 65535 (64k). The old NFS implementation uses a 32768 value. Remounting newnfs with the options rsize=32768,wsize=32768 results in performance very similar to that of oldnfs.

Sorting BackupPC log file by file size.

The following quick script can be used to parse a BackupPC transfer log file and sort the list of files transferred in descending order by size.

/backuppc/pc/pc.example.com-> BackupPC_zcat XferLOG.34.z | sed "s/create d/created/" | awk '{print $4,  "\t", $5}' | grep -vE 'bytes$' | sort -rn | less

Migrating to NFSv4 on FreeBSD

I was recently tasked with configuring an existing FreeBSD NFSv3 server to allow NFSv4 mounts. We don't care about v4 security at this point, just that the v4 protocol is being used. The FreeBSD handbook does not yet have any content discussing the configuration of a v4 server, and I was not able to find any good resources on Google. The nfsv4 man pages were informative, but not helpful. Here is the nutshell version of what I finally figured out.

On the NFS server, the following changes need to be made. These notes assume that you already have a working NFSv3 server.

# /etc/rc.conf changes

nfsv4_server_enable="YES"
nfsuserd_enable="YES"

# /etc/exports changes
# The 'V4' line defines the root of your NFSv4 tree. As I
# understand it, the paths you want exported must also be listed
# in the exports file, the same way they are for v3.
V4: /
/home -alldirs -maproot=root: -network=10.0.0.0/24

To mount the filesystem on the client using NFSv4, you must add the nfsv4 option to the mount command.

root@client:~-> mount -t nfs -o nfsv4 server:/home /mnt
root@client:~-> mount
server:/home on /mnt (nfs, nfsv4acls)

Parsing tricky whitespace with awk.

I spent a bunch of time trying to make awk properly parse some tricky whitespace in a text file. Essentially, I have a backup log file with a list of files transferred. Given the following example output:


  create d 755       0/0         512 .ccache
  create d 755       0/0         512 .ccache/0
  create d 755       0/0         512 .ccache/0/0
  same     644       0/0        6558 .ccache/0/0/1ebefd822077669fa42316da42e2c4-11870.manifest
  same     644       0/0        8251 .ccache/0/0/5ae5f481490e100d6d85b693aee8df-3158.manifest
  same     644       0/0        8407 .ccache/0/0/70b8ebd6f2cb5085c0f828e9145216-5033.manifest
  same     644       0/0         359 .ccache/0/0/7de2817532c5d1365a70d8dec4e378-30356.manifest

I want to grab the file size from the 4th column, so that I can sort it and find large files. The problem is, if I use a simple awk statement like

awk "{print $4}"

the file size is either the fourth or fifth column, depending on whether the first (logical) column has a space in it. I tried accounting for this with more complex awk, like so

awk 'BEGIN {FS="[ ]{2,}"} { print $4 }'

but to no avail. Admittedly, my awk-fu is weak. I did find a solution, rather a colleague came up with one.

sed "s/create d/created/" | awk '{print $4 }'

A bit outside of the box, but clever enough that I want to remember it.

BackupPC gotchas for PCBSD.

I am getting BackupPC configured to back up a new PCBSD desktop. Every time I do this, I have to relearn what tricks are required, so here is my troubleshooting checklist. Probably applicable to most Linux and Unix-like Operating Systems as well. If you've tried getting this set up before, the useless BackupPC error "Unable to read 4 bytes" has probably become the bane of your existence.

1. Does the client sshd configuration allow root logins? Check /etc/ssh/sshd_config, specifically the the PermitRootLogin parameter. Unless you absolutely need password authentication for root, use the following configuration: PermitRootLogin without-password
2. Make sure that the client has the public ssh for the BackupPC user on the server included in /root/.ssh/authorized_keys. If you haven't generated an ssh key for BackupPC, you should do that.
3. The permissions for /root/.ssh/authorized_keys on the client should be 640. /root/.ssh should be 750. Ownership should be root:wheel.
4. The .ssh/known_hosts file for the server BackupPC user should contain the fingerprint for the client. If you have connected to the client from the server before as another user (root), the fingerprint may already be in that user's known_hosts file. You can copy it into the BackupPC user's file. If you have not already connected to this host, you can verify the client configuration by connecting with the BackupPC ssh keys like so: ssh -i /path/to/bpc/.ssh/id_rsa client.example.org. The hostname you use (and known_hosts contains) MUST contain the exact hostname BackupPC is configured to connect to. If you connect to 'client' (leveraging your domain search list), but BackupPC is configured to connect to 'client.example.org', the ssh connection will fail.