Deleting locked IPSec SAs from Fortigates

We have had a borked IPSec Phase1 definition in our configuration since the initial configuration. The delete option was grayed out for it, despite the ref count showing 0. I finally had to call Fortinet about it. The engineer I spoke with said that the ref count of 0 doesn't necessarily mean that there aren't any references (what good is the ref count then?). He grabbed a copy of the configuration, and searched for the name of the Phase1. Sure enough, a policy routing entry turned up that we had long forgotten about. After removing this, I was able to delete the Phase1.

Using BIRD to route over OpenVPN tunnels.

OpenVPN tunnels, good. BIRD routing daemon, great. OSPF on OpenVPN tunnels, headache. The combination of OpenVPN and BIRD routing daemon for OSPF is nothing new for me. I've been using it pretty happily for over a year. However, it always seemed as though something was a bit precarious. As I have started to scale the implementation, I've realized that I needed a rethink of my strategy.

The original design used a /31 subnet on each tunnel, with each endpoint using an address. Logically, it made perfect sense. The catch is that OSPF does not automatically advertise the IP address of the tunnels (a quirk of OpenVPN, BIRD, or both). Traffic would flow through the routers, but the remote tunnel address would be unreachable. I solved this by adding a stubnet x.x.x.x/31 directive to the OSPF  configuration at one of the endpoints. At the time this worked fine.

As I brought a third site online, with multiple VPN tunnels between each site, the original design quickly broke down. The /31s were no longer consistently routable. I remedied this by changing the /31 stubnet to a /32, with each router advertising its own IP addresses. Life was briefly good, until I realized that restarting the OpenVPN tunnels would fail. OpenVPN would log the error, "router-a openvpn_1199[18976]: FreeBSD ifconfig failed: external program exited with error status: 1" The following message was also logged, "ifconfig: ioctl (SIOCAIFADDR): File exists" From the depths of my memory, I recalled that this error occurs when you attempt to configure an interface with an address that already exists in the routing table. Since BIRD was already advertising the /32 host IP, trying to address the tunnel would fail.

Some creative restarting of OpenVPN and BIRD resolved the problem, but is an unacceptable solution for production. I turned to the BIRD mailing list with my situation, and very quickly received a response from one of the BIRD developers. He suggested a number of ideas, the most promising of which is to dedicate a subnet to each router, as a pool from which to draw addresses for local tunnel endpoints on that specific host. The subnet is then configured as a stubnet by BIRD. The ends of a given tunnel may not be remotely close to each other, but this is fine because a tunnel is a point-to-point link. Tunnels can be restarted at will, without contention from existing routes in the table.

The diagram below shows an example topology. Notice that the endpoint of each tunnel is in a completely different network. For example, on the tunnel between router-a and router-b, router-a uses an IP of 192.168.0.0, and router-b uses 10.0.0.0. It doesn't matter, it works just fine. With the given stubnet configured in the bird.conf for each router, all tunnel endpoints are reachable. The network and broadcast addresses for each router's stubnet are also useable, as shown in the example.

Cisco ASA Remote Access configuration for Mac OS X

I spent the day fighting to get a remote access IPSec connection set up as follows:

• ASA 5515-X, running version 9.1.
• ASA network interfaces are already configured.
• IPSec clients are assigned addresses from the range 123.0.0.199-201.
• Client is running OS X 10.8.4 Mountain Lion.
• Client is using the built-in OS X IPSec client.
• Client IP is private, behind NAT, with a DHCP-assigned WAN IP.
• After connecting, client should be able to reach the internal networks 123.0.0.128/26, 123.0.0.192/27.
• All other traffic is not sent across the VPN.

The following configuration should be added to the ASA:

ip local pool REMOTE_ACCESS_POOL 123.0.0.199-123.0.0.201

management-access inside

access-list REMOTE_ACCESS_SPLIT_TUNNEL remark The corporate network behind the ASA.

access-list REMOTE_ACCESS_SPLIT_TUNNEL standard permit 123.0.0.128 255.255.255.192 
access-list REMOTE_ACCESS_SPLIT_TUNNEL standard permit 123.0.0.192 255.255.255.224 

crypto ipsec ikev1 transform-set REMOTE_ACCESS_TS esp-aes-256 esp-sha-hmac 
crypto dynamic-map REMOTE_ACCESS_DYNMAP 1 set ikev1 transform-set REMOTE_ACCESS_TS
crypto map REMOTE_ACCESS_MAP 1 ipsec-isakmp dynamic REMOTE_ACCESS_DYNMAP
crypto map REMOTE_ACCESS_MAP interface outside

crypto ikev1 enable outside
crypto ikev1 policy 1
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 7200

group-policy REMOTE_ACCESS_GP internal
group-policy REMOTE_ACCESS_GP attributes

 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value REMOTE_ACCESS_SPLIT_TUNNEL

username hunter password **** encrypted
tunnel-group REMOTE_ACCESS_TUNNELGRP type remote-access
tunnel-group REMOTE_ACCESS_TUNNELGRP general-attributes
 address-pool REMOTE_ACCESS_POOL
 default-group-policy REMOTE_ACCESS_GP
tunnel-group REMOTE_ACCESS_TUNNELGRP ipsec-attributes
 ikev1 pre-shared-key *****

For explanation of what all this does, I recommend reading the following Cisco docs. It is worth noting that this configuration does not work with Windows 7/8, which use IKEv2 instead of v1.

Remote Access VPN configuration
Split tunnel configuration
Management access over a VPN tunnel


The configuration for the built-in OS X IPSec client is described in the following doc. One gotcha I ran into (which is clearly stated in the document) is that the tunnel-group name must be specified in the 'Group Name' field on the Mac. In the case of the above configuration, the group name is REMOTE_ACCESS_TUNNELGRP.


VPN Clients for Mac OS X FAQ

Using AES on the Cisco 837

While trying to setup an IPSec SA on a Cisco 837 router I ran across an annoying problem. The solution is not obvious (to me) and google wasn't very helpful, so I figure it is worth mentioning. When trying to configure a transform set using esp-aes I got the following message:

Cisco837(config)#crypto ipsec transform-set my-ts esp-aes 256
Transform ecessa-ts disabled because esp-aes is not supported by encryption hardware

Re-checking the 837 datasheet, it does support AES in software. Setting 128 or 192 for the cipher length also had the same result. I also tried setting up AES as an ISAKMP encryption parameter and also got an error:

Cisco837(config-isakmp)#enc aes 256
May 15 15:38:03.704: %CRYPTO-4-ENC_METHOD_NOT_SUPPORTED: Invalid encryption method for IKE policy 10

Google was oddly unhelpful, until I searched the comp.dcom.sys.cisco usenet group and found this thread. Apparently, to use AES in software on the 837, you need to disable the hardware 3des engine using the following global configuration command:

no crypto engine accelerator

After that, specifying AES works fine.

ISAKMP: reserved not zero on payload 5!

I've been running into this issue at work, trying to connect an OpenSWAN peer to a Cisco PIX 501. I set both OpenSWAN the PIX to use aes and sha for ISAKMP encryption and hashing. When I establish the tunnel SA using the PIX as initiator, everything works fine. When initiating using OpenSWAN, the SA is not established. Checking the debug output on the PIX shows the PIX accepting the proposal from OpenSWAN, then failing later in the ISAKMP negotiation with the following error message.

ISAKMP: reserved not zero on payload 5!
ISAKMP: malformed payload

Googling around for the problem led to a lot of suggestions that this error is caused by an mismatched shared secret. This is certainly not the case for me, since the SA establishes just fine when set to use 3DES for Phase 1 encryption. I think the shared secret solution on the web is actually a mis-read of the Cisco documentation; they say, "This means that the ISAKMP keys do not match." The fact that 3DES works fine led us to discover that the ike parameter in my OpenSWAN configuration file was the problem. My config file looked like this:

conn Site50_common
auto=add
leftsubnet=192.168.20.0/24
leftid=20.20.20.2
right=50.50.50.2
rightsubnet=192.168.50.0/24
rightid=50.50.50.2
keyingtries=2
aggrmode=no
pfs=no
ikelifetime=8h
ike=aes-sha1-modp1536
auth=esp
esp=aes-sha1-96
authby=secret
dpddelay=30
dpdtimeout=60
dpdaction=restart

It appears that using ike=aes-sha1-modp1536 in OpenSWAN does not specify the 128-bit AES encryption expected by the PIX when the ISAKMP policy encryption is set to aes. I changed the OpenSWAN configuration file to specify aes128, and the tunnel now comes up immediately when the OpenSWAN client initiates the SA. Here is my updated OpenSWAn config file with the relevant line highlighted.

conn Site50_common
auto=add
leftsubnet=192.168.20.0/24
leftid=20.20.20.2
right=50.50.50.2
rightsubnet=192.168.50.0/24
rightid=50.50.50.2
keyingtries=2
aggrmode=no
pfs=no
ikelifetime=8h
ike=aes128-sha1-modp1536
auth=esp
esp=aes128-sha1-96
authby=secret
dpddelay=30
dpdtimeout=60
dpdaction=restart

And the [now] successful PIX debug output...

ISAKMP (0): processing SA payload. message ID = 0

ISAKMP (0): Checking ISAKMP transform 0 against priority 1 policy
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 28800
ISAKMP: encryption AES-CBC
ISAKMP: hash MD5
ISAKMP: auth pre-share
ISAKMP: default group 5
ISAKMP: keylength of 128
ISAKMP (0): atts are not acceptable. Next payload is 3
ISAKMP (0): Checking ISAKMP transform 1 against priority 1 policy
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 28800
ISAKMP: encryption AES-CBC
ISAKMP: hash SHA
ISAKMP: auth pre-share
ISAKMP: default group 5
ISAKMP: keylength of 128
ISAKMP (0): atts are acceptable. Next payload is 3
ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0): remote peer supports dead peer detection

ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T
ISAKMP (0): processing vendor id payload

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T
ISAKMP (0): processing vendor id payload

ISAKMP (0): SA is doing pre-shared key authentication using id type ID_IPV4_ADDR
return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_MM exchange
ISAKMP (0): processing KE payload. message ID = 0

ISAKMP (0): processing NONCE payload. message ID = 0

return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_MM exchange
ISAKMP (0): processing ID payload. message ID = 0
ISAKMP (0): processing HASH payload. message ID = 0
ISAKMP (0): SA has been authenticated

ISAKMP (0): ID payload
next-payload : 8
type : 1
protocol : 17
port : 500
length : 8
ISAKMP (0): Total payload length: 12
return status is IKMP_NO_ERROR
ISAKMP (0): sending INITIAL_CONTACT notify
ISAKMP (0): sending NOTIFY message 24578 protocol 1
VPN Peer: ISAKMP: Added new peer: ip:20.20.20.2/500 Total VPN Peers:1
VPN Peer: ISAKMP: Peer ip:20.20.20.2/500 Ref cnt incremented to:1 Total VPN Peers:1
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_QM exchange
oakley_process_quick_mode:
OAK_QM_IDLE
ISAKMP (0): processing SA payload. message ID = 61453885

ISAKMP : Checking IPSec proposal 0

ISAKMP: transform 0, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 256IPSEC(validate_proposal): transform proposal (prot 3, trans 12, hmac_alg 2) not supported

ISAKMP (0): atts not acceptable. Next payload is 3
ISAKMP: transform 1, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 192IPSEC(validate_proposal): transform proposal (prot 3, trans 12, hmac_alg 2) not supported

ISAKMP (0): atts not acceptable. Next payload is 3
ISAKMP: transform 2, ESP_AES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 28800
ISAKMP: authenticator is HMAC-SHA
ISAKMP: key length is 128
ISAKMP (0): atts are acceptable.IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) dest= 50.50.50.2, src= 20.20.20.2,
dest_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
src_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x4

ISAKMP (0): processing NONCE payload. message ID = 61453885

ISAKMP (0): processing ID payload. message ID = 61453885
ISAKMP (0): ID_IPV4_ADDR_SUBNET src 192.168.20.0/255.255.255.0 prot 0 port 0
ISAKMP (0): processing ID payload. message ID = 61453885
ISAKMP (0): ID_IPV4_ADDR_SUBNET dst 192.168.50.0/255.255.255.0 prot 0 port 0IPSEC(key_engine): got a queue event...
IPSEC(spi_response): getting spi 0x3bb5df50(1001774928) for SA
from 20.20.20.2 to 50.50.50.2 for prot 3

return status is IKMP_NO_ERROR
crypto_isakmp_process_block:src:20.20.20.2, dest:50.50.50.2 spt:500 dpt:500
OAK_QM exchange
oakley_process_quick_mode:
OAK_QM_AUTH_AWAIT
ISAKMP (0): Creating IPSec SAs
inbound SA from 20.20.20.2 to 50.50.50.2 (proxy 192.168.20.0 to 192.168.50.0)
has spi 1001774928 and conn_id 3 and flags 4
lifetime of 28800 seconds
outbound SA from 50.50.50.2 to 20.20.20.2 (proxy 192.168.50.0 to 192.168.20.0)
has spi 1662679932 and conn_id 4 and flags 4
lifetime of 28800 secondsIPSEC(key_engine): got a queue event...
IPSEC(initialize_sas): ,
(key eng. msg.) dest= 50.50.50.2, src= 20.20.20.2,
dest_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
src_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 28800s and 0kb,
spi= 0x3bb5df50(1001774928), conn_id= 3, keysize= 128, flags= 0x4
IPSEC(initialize_sas): ,
(key eng. msg.) src= 50.50.50.2, dest= 20.20.20.2,
src_proxy= 192.168.50.0/255.255.255.0/0/0 (type=4),
dest_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= esp-aes esp-sha-hmac ,
lifedur= 28800s and 0kb,
spi= 0x631a7b7c(1662679932), conn_id= 4, keysize= 128, flags= 0x4

VPN Peer: IPSEC: Peer ip:20.20.20.2/500 Ref cnt incremented to:2 Total VPN Peers:1
VPN Peer: IPSEC: Peer ip:20.20.20.2/500 Ref cnt incremented to:3 Total VPN Peers:1
return status is IKMP_NO_ERROR