Thursday, September 6, 2012

Sorting BackupPC log file by file size.

The following quick script can be used to parse a BackupPC transfer log file and sort the list of files transferred in descending order by size.

/backuppc/pc/> BackupPC_zcat XferLOG.34.z | sed "s/create d/created/" | awk '{print $4,  "\t", $5}' | grep -vE 'bytes$' | sort -rn | less

Tuesday, September 4, 2012

Migrating to NFSv4 on FreeBSD

I was recently tasked with configuring an existing FreeBSD NFSv3 server to allow NFSv4 mounts. We don't care about v4 security at this point, just that the v4 protocol is being used. The FreeBSD handbook does not yet have any content discussing the configuration of a v4 server, and I was not able to find any good resources on Google. The nfsv4 man pages were informative, but not helpful. Here is the nutshell version of what I finally figured out.

On the NFS server, the following changes need to be made. These notes assume that you already have a working NFSv3 server.

# /etc/rc.conf changes


# /etc/exports changes
# The 'V4' line defines the root of your NFSv4 tree. As I
# understand it, the paths you want exported must also be listed
# in the exports file, the same way they are for v3.
V4: /
/home -alldirs -maproot=root: -network=

To mount the filesystem on the client using NFSv4, you must add the nfsv4 option to the mount command.

root@client:~-> mount -t nfs -o nfsv4 server:/home /mnt
root@client:~-> mount
server:/home on /mnt (nfs, nfsv4acls)

Parsing tricky whitespace with awk.

I spent a bunch of time trying to make awk properly parse some tricky whitespace in a text file. Essentially, I have a backup log file with a list of files transferred. Given the following example output:

  create d 755       0/0         512 .ccache
  create d 755       0/0         512 .ccache/0
  create d 755       0/0         512 .ccache/0/0
  same     644       0/0        6558 .ccache/0/0/1ebefd822077669fa42316da42e2c4-11870.manifest
  same     644       0/0        8251 .ccache/0/0/5ae5f481490e100d6d85b693aee8df-3158.manifest
  same     644       0/0        8407 .ccache/0/0/70b8ebd6f2cb5085c0f828e9145216-5033.manifest
  same     644       0/0         359 .ccache/0/0/7de2817532c5d1365a70d8dec4e378-30356.manifest

I want to grab the file size from the 4th column, so that I can sort it and find large files. The problem is, if I use a simple awk statement like

awk "{print $4}"

the file size is either the fourth or fifth column, depending on whether the first (logical) column has a space in it. I tried accounting for this with more complex awk, like so

awk 'BEGIN {FS="[ ]{2,}"} { print $4 }'

but to no avail. Admittedly, my awk-fu is weak. I did find a solution, rather a colleague came up with one.

sed "s/create d/created/" | awk '{print $4 }'

A bit outside of the box, but clever enough that I want to remember it.