Using BIRD to route over OpenVPN tunnels.

OpenVPN tunnels, good. BIRD routing daemon, great. OSPF on OpenVPN tunnels, headache. The combination of OpenVPN and BIRD routing daemon for OSPF is nothing new for me. I've been using it pretty happily for over a year. However, it always seemed as though something was a bit precarious. As I have started to scale the implementation, I've realized that I needed a rethink of my strategy.

The original design used a /31 subnet on each tunnel, with each endpoint using an address. Logically, it made perfect sense. The catch is that OSPF does not automatically advertise the IP address of the tunnels (a quirk of OpenVPN, BIRD, or both). Traffic would flow through the routers, but the remote tunnel address would be unreachable. I solved this by adding a stubnet x.x.x.x/31 directive to the OSPF  configuration at one of the endpoints. At the time this worked fine.

As I brought a third site online, with multiple VPN tunnels between each site, the original design quickly broke down. The /31s were no longer consistently routable. I remedied this by changing the /31 stubnet to a /32, with each router advertising its own IP addresses. Life was briefly good, until I realized that restarting the OpenVPN tunnels would fail. OpenVPN would log the error, "router-a openvpn_1199[18976]: FreeBSD ifconfig failed: external program exited with error status: 1" The following message was also logged, "ifconfig: ioctl (SIOCAIFADDR): File exists" From the depths of my memory, I recalled that this error occurs when you attempt to configure an interface with an address that already exists in the routing table. Since BIRD was already advertising the /32 host IP, trying to address the tunnel would fail.

Some creative restarting of OpenVPN and BIRD resolved the problem, but is an unacceptable solution for production. I turned to the BIRD mailing list with my situation, and very quickly received a response from one of the BIRD developers. He suggested a number of ideas, the most promising of which is to dedicate a subnet to each router, as a pool from which to draw addresses for local tunnel endpoints on that specific host. The subnet is then configured as a stubnet by BIRD. The ends of a given tunnel may not be remotely close to each other, but this is fine because a tunnel is a point-to-point link. Tunnels can be restarted at will, without contention from existing routes in the table.

The diagram below shows an example topology. Notice that the endpoint of each tunnel is in a completely different network. For example, on the tunnel between router-a and router-b, router-a uses an IP of 192.168.0.0, and router-b uses 10.0.0.0. It doesn't matter, it works just fine. With the given stubnet configured in the bird.conf for each router, all tunnel endpoints are reachable. The network and broadcast addresses for each router's stubnet are also useable, as shown in the example.

Using FreeBSD loopback interfaces with BIRD

Why go for the simple solution, when you can first spend hours tearing out your hair in frustration?

I've been working on a new data center deployment, and getting my fingers back into the networking realm; a welcome change. This includes my first OSPFv3 deployment, and we're using BIRD. For the most part, I have been very happy with BIRD for OSPF and BGP; though I have run into some quirks.

The quirk on my mind at this moment is with loopback interfaces. The Cisco way of doing things seems to be to run iBGP sessions between loopback addresses. The rationale is that if you use an interface address, and that interface goes down, your iBGP session goes with it, as that address becomes unreachable. So you use a loopback interface, which is always up. The addresses on the loopback are advertised via an IGP, facilitating the iBGP connection.

For better or worse, I decided to follow the herd, and go with a loopback interface. For IPv4, this was pretty straightforward. Configure the loopback in the OS, add it to bird.conf as a stub interface, and good to go. For reference, here are the bits to do so on FreeBSD.

# /etc/rc.conf

cloned_interfaces="lo1"

ifconfig_lo1="inet W.X.Y.Z/32"

# /usr/local/etc/bird.conf

protocol ospf {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

authentication cryptographic;

password "password";

};

interface "lo1", "vlan1001" { stub; };

};

}

And since the proof is in the pudding (or output)...

bird> show route for W.X.Y.Z

W.X.Y.Z/32 via A.B.C.D on vlan500 [ospf1 09:08] * I (150/5) [W.X.Y.Z]

When I went to configure OSPF for our IPv6 allocation, things didn't go quite so smoothly. I used the following similar configuration for the v6 BIRD configuration.

# /etc/rc.conf

ifconfig_lo1_ipv6="inet6 2620:W:X:Y::Z/128"

# /usr/local/etc/bird6.conf

protocol ospf ospf_v6 {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

# Authentication is not supported by OSPFv3, supposed to be IPSec AH authenticated.

#authentication cryptographic;

#password "password";

};

interface "lo1", "vlan1001" { stub; };

};

}

With this configuration in place, I realized that my IPv6 loopback address was not being advertised. Examining the logs, BIRD was quite happy to tell me that it had filtered out that route. WTF? After a bunch of time wasted searching Google and throwing shit at the wall, my loopback address was still not working. I finally stumbled on this mailing list thread, where I learned that using a loopback is NOT an expected configuration; at least in the eyes of the developers. Furthermore, the fact that it is working in IPv4 was surprising, and perhaps a bug. The reason that BIRD is denying my lo1 IP is that there is no link-local IP on the interface as well. Without starting a discussion on whether Cisco or BIRD is more right, I'll just say I was *((# *grumble* *f'n BIRD*.

I jumped through a few more hoops, and finally discovered that I could use a tap interface in lieu of a loopback. It would generate a link-local address, OSPF would advertise it, and iBGP happiness filled the kingdom. Nevermind that it is about as hacky as you can get. For what it's worth, here is the rc.conf goo to make it happen.

# /etc/rc.conf

# DON'T USE THIS! IT'S HACKY AND EVERYONE WILL LAUGH AT YOU.

cloned_interfaces="tap0"

ifconfig_tap0_ipv6="inet6 2620:W:X:Y::Z/128 -ifdisabled"

I slept on it. When I woke up, I had some OSPF fixes on my mind for the ASAs (that's another raar story). I was poking around a little more when I read and was reminded about the BIRD stubnet configuration directive. In a nutshell, BIRD will always advertise a stubnet route...perfect! Changed the configuration to support this, and life is good again.

# /etc/rc.conf

ifconfig_lo1_ipv6="inet6 2620:W:X:Y::Z/128"

# /usr/local/etc/bird6.conf

protocol ospf ospf_v6 {

tick 2;

area 0 {

stub no;

interface "vlan7", "vlan500" {

cost 5;

hello 2;

dead 10;

};

interface "vlan1001" { stub; };

stubnet 2620:W:X:Y::Z/128;

};

}

and the proof!

bird> show route for 2620:W:X:Y::Z/128

2620:W:X:Y::Z/128 via fe80::225:90ff:fe6b:f52c on vlan7 [ospf_v6 09:12] * I (150/15) [W.X.Y.Z]