FreeBSD CARP+BRIDGE+VLAN=BAD

Bridge, good. VLANs, great. CARP, awesome. BRIDGE+VLAN+CARP, pwned. We decided to purchase a pair of atom-based systems for use as the office firewalls. The only thing we weren't completely pleased with was the single Ethernet port. Given that we have been using vlans elsewhere on the network, we didn't expect that there would be any problems using vlans for all interfaces. It turned out to be a big box of pain. In addition to routing problems, we also appear to have aggravated a bug that hangs the system.

VLAN+CARP is a fairly common configuration on firewalls. Our office and DC LANs use the same subnet and are bridged over an OpenVPN tunnel. Trying to incorporate VLAN+CARP into into a bridge seems to cause problems. This diagram illustrates our logical network setup.

After a lot of trial and error, a number of conclusions were drawn.

• Routing over the bridge doesn't work the same when using vlans. The VPN server pushes a route to our production network when clients connect. When the office firewall was using a physical Ethernet interface for the LAN, this route would refer to the LAN interface as the outgoing interface for this connection. This seems counter-intuitive, but it worked just fine. When the Ethernet LAN interface was replaced with a vlan, the tap (VPN) interface was referenced by the route to production. This seemed to be more logical, except that the Production network became unreachable.

• After some troubleshooting, I figured out that access to the production network could be fixed by adding a static route to the DC firewall (next-hop to production network) pointing out the tap interface. This seemed to allow traffic to flow smoothly to production.

• Adding CARP into the above configuration caused the firewall to hang randomly. There seemed to be no indication of a crash, no excessive resource use or network traffic.

• Routing traffic between tagged vlans and the underlying physical interface may be problematic. This was an earlier configuration I tried, and it seemed to have issues. However, at the time I had not identified CARP as the source of the system hangs, so this may be a non-issue.

The routing issue was reported in a PR that can be found here. The routing tables mentioned above can be found here.