Wednesday, October 31, 2012

FreeBSD 9.0 newnfs slow throughput

We recently updated our production systems to FreeBSD 9.0. Almost immediately afterwards, we began getting reports of crumby performance on NFS mounts. After following a lot of false leads, I finally traced the issue to the new NFS code in FreeBSD, appropriately named 'newnfs'. I remounted NFS using the 'oldnfs' filesystem type, and performance immediately went back to its previous levels; a 55% increase by my most conservative testing, with users reporting orders of magnitude improvement for some jobs.

Using old NFS is a band-aid, not a solution. The FreeBSD NFS wizards desire to prune the old code for FreeBSD 10. I set to work, trying to determine the reason for the poor throughput on newnfs. I discovered that newnfs uses a default rsize and wsize value of 65535 (64k). The old NFS implementation uses a 32768 value. Remounting newnfs with the options rsize=32768,wsize=32768 results in performance very similar to that of oldnfs.

Thursday, September 6, 2012

Sorting BackupPC log file by file size.

The following quick script can be used to parse a BackupPC transfer log file and sort the list of files transferred in descending order by size.

/backuppc/pc/> BackupPC_zcat XferLOG.34.z | sed "s/create d/created/" | awk '{print $4,  "\t", $5}' | grep -vE 'bytes$' | sort -rn | less

Tuesday, September 4, 2012

Migrating to NFSv4 on FreeBSD

I was recently tasked with configuring an existing FreeBSD NFSv3 server to allow NFSv4 mounts. We don't care about v4 security at this point, just that the v4 protocol is being used. The FreeBSD handbook does not yet have any content discussing the configuration of a v4 server, and I was not able to find any good resources on Google. The nfsv4 man pages were informative, but not helpful. Here is the nutshell version of what I finally figured out.

On the NFS server, the following changes need to be made. These notes assume that you already have a working NFSv3 server.

# /etc/rc.conf changes


# /etc/exports changes
# The 'V4' line defines the root of your NFSv4 tree. As I
# understand it, the paths you want exported must also be listed
# in the exports file, the same way they are for v3.
V4: /
/home -alldirs -maproot=root: -network=

To mount the filesystem on the client using NFSv4, you must add the nfsv4 option to the mount command.

root@client:~-> mount -t nfs -o nfsv4 server:/home /mnt
root@client:~-> mount
server:/home on /mnt (nfs, nfsv4acls)

Parsing tricky whitespace with awk.

I spent a bunch of time trying to make awk properly parse some tricky whitespace in a text file. Essentially, I have a backup log file with a list of files transferred. Given the following example output:

  create d 755       0/0         512 .ccache
  create d 755       0/0         512 .ccache/0
  create d 755       0/0         512 .ccache/0/0
  same     644       0/0        6558 .ccache/0/0/1ebefd822077669fa42316da42e2c4-11870.manifest
  same     644       0/0        8251 .ccache/0/0/5ae5f481490e100d6d85b693aee8df-3158.manifest
  same     644       0/0        8407 .ccache/0/0/70b8ebd6f2cb5085c0f828e9145216-5033.manifest
  same     644       0/0         359 .ccache/0/0/7de2817532c5d1365a70d8dec4e378-30356.manifest

I want to grab the file size from the 4th column, so that I can sort it and find large files. The problem is, if I use a simple awk statement like

awk "{print $4}"

the file size is either the fourth or fifth column, depending on whether the first (logical) column has a space in it. I tried accounting for this with more complex awk, like so

awk 'BEGIN {FS="[ ]{2,}"} { print $4 }'

but to no avail. Admittedly, my awk-fu is weak. I did find a solution, rather a colleague came up with one.

sed "s/create d/created/" | awk '{print $4 }'

A bit outside of the box, but clever enough that I want to remember it.

Tuesday, June 12, 2012

BackupPC gotchas for PCBSD.

I am getting BackupPC configured to back up a new PCBSD desktop. Every time I do this, I have to relearn what tricks are required, so here is my troubleshooting checklist. Probably applicable to most Linux and Unix-like Operating Systems as well. If you've tried getting this set up before, the useless BackupPC error "Unable to read 4 bytes" has probably become the bane of your existence.

  1. Does the client sshd configuration allow root logins? Check /etc/ssh/sshd_config, specifically the the PermitRootLogin parameter. Unless you absolutely need password authentication for root, use the following configuration: PermitRootLogin without-password
  2. Make sure that the client has the public ssh for the BackupPC user on the server included in /root/.ssh/authorized_keys. If you haven't generated an ssh key for BackupPC, you should do that.
  3. The permissions for /root/.ssh/authorized_keys on the client should be 640. /root/.ssh should be 750. Ownership should be root:wheel.
  4. The .ssh/known_hosts file for the server BackupPC user should contain the fingerprint for the client. If you have connected to the client from the server before as another user (root), the fingerprint may already be in that user's known_hosts file. You can copy it into the BackupPC user's file. If you have not already connected to this host, you can verify the client configuration by connecting with the BackupPC ssh keys like so: ssh -i /path/to/bpc/.ssh/id_rsa The hostname you use (and known_hosts contains) MUST contain the exact hostname BackupPC is configured to connect to. If you connect to 'client' (leveraging your domain search list), but BackupPC is configured to connect to '', the ssh connection will fail.

Pwning the Spotify client

I've been trying to bend the Spotify client to my will for months. I love the service, but the mandatory P2P network traffic generated by the client is so abusive that I can't do my job and listen to music at the same time. You hear that Spotify, your draconian client prevents me from doing my job. I've tried all sorts of tricks to try and block the network traffic, but it's slippery to try and block without impacting legitimate traffic.

I finally decided to try poking at the storage space available to the client, in hopes that I could cut it off at the knees there. I've learned some interesting things (thanks to this blog post for a point in the _right_ direction).

  • The Spotify client for OSX puts the settings file at ~/Library/Application\ Support/Spotify/settings
  • The cache_location parameter controls where the clients tries to put downloaded data. I don't know if parameter position is important in the Spotify configuration file, but the client puts this parameter (for me) between the listen_port and cache_size params.
  • Spotify does not appear to respect the cache_size parameter when it is running, at least not in the short term. I tried setting a cache_size of 1MB, but it appears that the client continuously caches music you listen to. The cache storage directory is reduced to the configured size on client start, apparently.
  • Because of this, the client cannot be contained by changing the location of the cache to a tiny filesystem. I tried using a 20MB Mac disk image as storage; Spotify happily filled the entire image, then stopped playing, complaining about a full drive.
  • When I reduced the cache_size to 1MB and deleted the existing cache, starting the Spotify client produced a message that offline playback is disabled. It remains to be seen if this also means P2P is disabled. Time will tell.
I really wish that Spotify would allow users to control (or disable) the P2P function of the client. I am not opposed to giving a little bit of bandwidth to P2P, but I need to be able to do my job as well. I'm not optimistic though. Although their product is excellent, Spotify seems to be unresponsive to customer requests, and their customer support seems to be dismal, at least in the United States. If anyone at Spotify cares to prove any of this wrong, I would be happy to update/revise this post.

Tuesday, April 10, 2012

/sbin/shutdown permission denied

I have been working on setting up a vmware-based demo environment to be used for a presentation I will be giving in about a month. As a part of the demonstation, I need a lot of nearly identical virtual machines, so PXE-booting them seems like a great idea.

I set up my test NFS server and built a PXE image, following the lead of the FreeBSD handbook, and a PXE setup that we use at $work. After getting my virtual PXE host to boot, I quickly discovered that I was unable to login as root, despite having set a password. After a lot of screwing around, including compiling 3 different FreeBSD source trees, I finally tried booting with the memory filesystems, as described in section 32.8.2 of the handbook. This allowed me to log in, but was still not ideal, since changing anything in /etc required rebuilding the archive. I was also unable to shut down the PXE VM, because calling shutdown returned "permission denied," as root.

Turning to smarter people on IRC, it was suggested that this behavior is usually seen in FreeBSD jails, a nosuid-mounted filesystem, or incorrect permissions on /sbin/shutdown. The second possibility got me to thinking, which led me to the answer. My NFS export on the server was missing the '-maproot=root:' directive. Adding this directive got everything working as expected.

Tuesday, January 24, 2012

Stupid Cacti tricks

I spent some time this weekend consolidating services I have running at home down to one box. I migrated my Cacti installation to its new home (from one FreeBSD jail to another), and moved the mysql server to its own jail. I quickly realized that there was a problem. My graphs weren't working anymore. Some checking and I determined that there was a problem with the rrd files failing because the timestamp of updates was in the past, by six hours, which is precisely the offset of my time zone (CST). I checked all of the time zone configuration on the Cacti jail, and everything was set correctly. I went to bed, frustrated, but figuring that the data would catch up by morning.

Morning came, and my graphs were indeed logging data again, but offset by six hours. The graph window showed the correct time range, but the end of the data line was six hours behind local time. After too much screwing around, reading through the Cacti php. I finally discovered that Cacti sends its poller results to mysql, before retrieving them to put in the rrd files (and purging the records from mysql). The times I was getting back from the new mysql server were being sent as local time, not UTC.

Long story longer, I had neglected to set the time zone in the jail containing mysql.