Arcane Bourne shell behavior. The colon (:)

I finally learned the purpose of starting a command with the colon utility (:) in bourne shell. It expands any arguments, then exits 0. This is handy for hiding pointless error messages resulting from variable substitution. Consider the following example.

$ ${D:=foo}

foo: not found

$ echo $D

foo

The example above sets variable D to "foo" if it is unset or null. It works fine, but causes the annoying error 'foo: not found'. We can suppress that message with the colon.

$ unset D

$ : ${D:=foo}

$ echo $D

foo

Much prettier!

Also see:

Parameter Expansion (and explanation of the colon utility at the bottom of the page)

Understanding resolvconf behavior on pxe-booted hosts.

I have been learning more than I ever cared to know about resolvconf, and what happens when you use it on a host with a read-only root filesystem. I have been preparing to roll out pxe-boot virtual machines at a second location. The PXE image has the following characteristics.

* NFS-mounted, read-only / filesystem.
* Local writeable disk for swap and /var.
* Unionfs, md-backed /etc (non-persistent, r/w)

I noticed that on the initial boot of a new VM, /etc/resolv.conf would be written correctly. However, all subsequent boots never see the NFS-supplied resolv.conf updated. After a frustrating afternoon of digging, I determined why resolvconf appears to stop working.

Resolvconf stores state data in /var/run/resolvconf. When dhclient is run for an interface, the dhclient-script script calls resolvconf with DNS particulars, resolvconf looks in the interfaces/ sub-directory for an entry named after the interface. If the file does not exist, or does not match the domain/nameserver options received by resolvconf, a new file is written, and appropriate changes are made to /etc/resolv.conf. If the options match what resolvconf already has, no changes are made. The below output shows the contents of the interfaces/ directory on my pxe host.

> ll /var/run/resolvconf/interfaces/
total 8
-rw-r--r--  1 root  wheel  80 Mar 13 18:00 vmx0:dhcp4
-rw-r--r--  1 root  wheel  76 Mar 13 18:00 vmx1

The problem with my pxe hosts lies in the volatile /etc. Every time the host reboots, the modified contents of /etc vanish. /etc/resolv.conf is replaced with the copy from NFS. In my case, this copy reflects the nameservers at the "original" datacenter. Since the state directory for resolvconf exists on the persistent /var, resolvconf sees the old [unchanged] lease data, and assumes everything is peachy with the resolv.conf file.

I don't need the extra features of resolvconf, so I can solve the problem by disabling it. I created a file in the pxe image, /etc/dhclient-enter-hooks, that contains the following.

resolvconf_enable=NO

My initial, more complicated fix, was to create an rc script to re-initialize the resolvconf state directory on every boot. This also worked flawlessly.

#!/bin/sh
#
# Clean out the contents of the resolvconf state directory. Otherwise,
# /etc/resolv.conf never gets updated after the initial boot of a new pxe host.
#
# BEFORE: netif
# AFTER: FILESYSTEMS
# PROVIDE: clean_resolvconf

echo -n "Cleaning out resolvconf state directory: "

/sbin/resolvconf -I

if [ $? ]; then
echo "OK"
else 
echo "FAILED"
fi

Parsing tricky whitespace with awk.

I spent a bunch of time trying to make awk properly parse some tricky whitespace in a text file. Essentially, I have a backup log file with a list of files transferred. Given the following example output:


  create d 755       0/0         512 .ccache
  create d 755       0/0         512 .ccache/0
  create d 755       0/0         512 .ccache/0/0
  same     644       0/0        6558 .ccache/0/0/1ebefd822077669fa42316da42e2c4-11870.manifest
  same     644       0/0        8251 .ccache/0/0/5ae5f481490e100d6d85b693aee8df-3158.manifest
  same     644       0/0        8407 .ccache/0/0/70b8ebd6f2cb5085c0f828e9145216-5033.manifest
  same     644       0/0         359 .ccache/0/0/7de2817532c5d1365a70d8dec4e378-30356.manifest

I want to grab the file size from the 4th column, so that I can sort it and find large files. The problem is, if I use a simple awk statement like

awk "{print $4}"

the file size is either the fourth or fifth column, depending on whether the first (logical) column has a space in it. I tried accounting for this with more complex awk, like so

awk 'BEGIN {FS="[ ]{2,}"} { print $4 }'

but to no avail. Admittedly, my awk-fu is weak. I did find a solution, rather a colleague came up with one.

sed "s/create d/created/" | awk '{print $4 }'

A bit outside of the box, but clever enough that I want to remember it.

Stupid Cacti tricks

I spent some time this weekend consolidating services I have running at home down to one box. I migrated my Cacti installation to its new home (from one FreeBSD jail to another), and moved the mysql server to its own jail. I quickly realized that there was a problem. My graphs weren't working anymore. Some checking and I determined that there was a problem with the rrd files failing because the timestamp of updates was in the past, by six hours, which is precisely the offset of my time zone (CST). I checked all of the time zone configuration on the Cacti jail, and everything was set correctly. I went to bed, frustrated, but figuring that the data would catch up by morning.

Morning came, and my graphs were indeed logging data again, but offset by six hours. The graph window showed the correct time range, but the end of the data line was six hours behind local time. After too much screwing around, reading through the Cacti php. I finally discovered that Cacti sends its poller results to mysql, before retrieving them to put in the rrd files (and purging the records from mysql). The times I was getting back from the new mysql server were being sent as local time, not UTC.

Long story longer, I had neglected to set the time zone in the jail containing mysql.

Apache syslogging on FreeBSD

If you need to use Syslog to send Apache log output there are plenty of examples already on the Internet. The first hit on google was the O'Reilly Sysadmin blog, which is very useful. However, the page is a bit old and the perl script they provide for Syslogging access logs is in need of updating. My modified version is below. To summarize the process..

1. Put the following script in /usr/local/bin/apache_syslog.

   
   #!/usr/bin/perl
   
   # $Id$
   #
   # A wrapper script that logs apache access via syslog. Copied from an example
   # at http://oreilly.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html
   # Script requires sysutils/p5-Sys-Syslog from FreeBSD ports.
   #
   
   use Sys::Syslog qw( :DEFAULT setlogsock );
   
   # Excluded, per the rules of Sys:Syslog
   # http://search.cpan.org/~saper/Sys-Syslog-0.29/Syslog.pm#THE_RULES_OF_SYS::SYSLOG
   #setlogsock('unix');
   openlog('httpd', "cons, pid", 'local2');
   
   while ($log = ) {
   syslog('notice', $log);
   }
   
   closelog;

2. Install sysutils/p5-Sys-Syslog from ports (FreeBSD).
3. In your Apache config replace your ErrorLog directive with "ErrorLog syslog:local1".
4. Replace your CustomLog directive (for access logs) with "CustomLog |/usr/local/bin/apache_syslog combined".
5. Edit /etc/syslog.conf, adding the following lines
   !httpd
   local1.* /var/log/httpd-error.log
   local2.* /var/log/httpd-access.log
   !*
   
6. Create the log files with "touch /var/log/httpd-error.log /var/log/httpd-access.log".
7. Edit /etc/newsyslog.conf, adding the following lines
   /var/log/httpd-error.log 640 14 * @T00 J
   /var/log/httpd-access.log 640 14 * @T00 J
8. (Re)start syslogd and apache.
9. Profit.
   

Clearing my mental hang-ups about Perl

The script below does some simple things to demonstrate to myself how a number of things work in Perl. Simple things, but ones that I sometimes have to stop and think twice about.

use Config::Auto;

# Test Config::Auto output, which should be a reference
my $ca = Config::Auto->new(
source => "test.conf",
format => "equal"
);

my $config = $ca->parse;

print "\$config is a " . ref($config) . " reference\n" if (ref $config);
print "c_one undefined\n" unless (defined $config->{'c_one'});
print "c_two undefined\n" unless (defined $config->{'c_two'});
print "c_three undefined\n" unless (defined $config->{'c_three'});
print "c_one non-existent\n" unless (exists $config->{'c_one'});
print "c_two non-existent\n" unless (exists $config->{'c_two'});
print "c_three non-existent\n" unless (exists $config->{'c_three'});


# Test a non-referenced hash
my %harsh=(
'one' => 1,
'two',
);
print "\$harsh is a " . ref($harsh) . " reference\n" if (ref $harsh);
print "one undefined\n" unless (defined $harsh{'one'});
print "two undefined\n" unless (defined $harsh{'two'});
print "three undefined\n" unless (defined $harsh{'three'});
print "one non-existent\n" unless (exists $harsh{'one'});
print "two non-existent\n" unless (exists $harsh{'two'});
print "three non-existent\n" unless (exists $harsh{'three'});

# Test array interpolation
my @array = ("one", "two");
print "\@array interpolated is: @array\n\@array not interpolated is " . @array . "\n\$\#array is $#array\n";

The test.conf file referred to above contains the following:

c_one = 1
c_two =

The following output is generated by the script:

$config is a HASH reference
c_three undefined
c_three non-existent
two undefined
three undefined
three non-existent
@array interpolated is: one two
@array not interpolated is 2
$#array is 1

Apple DNS cache, and a bourne revelation [to me]

I learned a couple of interesting things today. The first is that Mac OS X, or at least some applications, cache DNS results. This is irritating, but fortunately there is a way to clear out the cache. Run the following command in a terminal to clear out the cache.

dscacheutil -flushcache

On the FreeBSD side of life, I had a minor revelation related to bourne (sh) scripting. I have occasionally wondered if it was possible to build the name of a variable dynamically. When I've wondered aloud, the answer I've received has always been that this is not possible. Today, I was reading through /etc/network.subr and I happened upon this bit of code.

if [ -n "${static_routes}" ]; then
for i in ${static_routes}; do
eval route_args=\$route_${i}
route add ${route_args}
done
fi

Like a beam of light from Heaven, I suddenly realized that this little eval statement is the answer to my hopes. It allows you to build a variable name on the fly!

Renaming files by modification date

I have been working on combining sets of photos that my brother and I shot on a recent vacation. Being the perfectionist that I am, I wanted the two sets of photos to flow chronologically. We both had the clocks on our cameras set pretty close, so I thought I could get away with sorting files by modification dates and using touch to correct the timestamps on any files I modified. Unfortunately, photo viewers seem to only organize photos by name, so I was going to have to find a better way.

I went looking for a quick and easy script to rename all the files in a directory into some common scheme. What I found was a one liner that I used to prepend the modification date onto the name of each file. The command looks like so...

find -type f -name "*jpg" -printf "mv \"%h/%f\" \"%h/%TY%Tm%Td-%TH%TM%TS-%f\"\n" | sh

I recommend initially leaving off the sh pipe at the end of the command. This way you can preview the file name changes and make sure you are getting the result you want. This yields a filename such as 20090201-220505.0000000000-foo.jpg. Leaving the original file name at the end of each file gets around the problem of having non-unique file names if there were multiple photos shot in the same second (definite possibility with a DSLR).

I didn't come up with this trick all on my own. I borrowed from this discussion and modified the command to fit my situation.