Thesis VPN Troubleshooting Tlun Final
Thesis VPN Troubleshooting Tlun Final
Thesis
9 May 2017
Abstract
Instructor
Marko Uusitalo, Senior Lecturer
With the vast reliance on the Internet, corporations end up managing not only their intranet
but extranets as well to allow resource sharing with their partner corporations. In order to
connect users or sites to these intra- and extranets over various internets, VPN tunnels are
typically constructed to create such a logical connection over physical infrastructure. In
such environments different networking devices are found with varying degrees of support
for vendor interoperability. This will set constraints and considerations to the VPN tunnel
creation as well in multivendor environments.
This study was fueled by the desire to explore differences in VPN tunnel building and trou-
bleshooting processes between two devices designed by different vendors. As such this
project aims to construct an IPsec site-to-site VPN tunnel between two such gateways and
systematically introduce issues to the connection for troubleshooting analysis. The results
then would be compiled into a guide to aid future troubleshooting. Additionally, any differ-
ences and similarities in the devices’ protocol handling will be documented.
The available hardware for the project was Cisco’s ASA 5505 and Palo Alto Networks’ PA-
200 security gateways. The physical network setup was done in the campus’ laboratory
environment the third-party network included.
The experimentation results were fruitful, providing two points of view to the protocol ex-
changes. While the standardized protocols guided the negotiations along a certain path,
the debugging output certainly differed between the vendors; Palo Alto focused more on
printing out each stage transitioning, while Cisco was more focused on user-friendliness.
In the future when site-to-site IPsec VPNs are implemented, corporations could be seen
gradually switching to IKEv2 protocol as the latest stable software versions have confirmed
the support. As the authentication method is more of a question of scalability, both PSK
and Certificate- based methods will certainly remain in use. Other than the upgrade of se-
curity levels, future studies will more often implement support for IPv6 tunnelling parallel to
IPv4.
Koulutusohjelma Tietotekniikka
Suuntautumisvaihtoehto Tietoverkot
Ohjaaja
Lehtori Marko Uusitalo
Internetistä on kasvanut niin suuri tekijä yrityksille, että intranetin lisäksi ekstranetien
hallinnointi tulee olla huomioitu resurssienjakoa varten. Etäkäyttäjien ja toimipisteiden
yhteenliittämiseksi julkisien verkkojen kautta, VPN-tunneleita yleisesti käytetään näiden
loogisien yhteyksien muodostamisessa. Tällaisissä ympäristöissä tulee vastaan monien eri
laitevalmistajien laitteita, jotka vaihtelevissa määrin ovat yhteensopivia. Eri laitevalmistajien
laitteiden tukeminen vaatii tiettyjen ominaisuuksien huomioimista VPN-tunneleidenkin
kohdalla.
1 Introduction 1
4 Device Setup 11
5 Device Configuration 16
8 Conclusion 56
References 57
Appendices
Appendix 1. ASA 5505 Running Configuration
Appendix 2. PA-200 Running Configuration
Abbreviations
1 Introduction
This paper aims to serve as a guide for troubleshooting Virtual Private Network (VPN)
issues between two networking devices: Cisco’s ASA 5505 and Palo Alto Networks’
PA-200. In this study a Site-to-Site VPN connection is set up using the Internet Proto-
col Security (IPsec) protocol suite.
Starting with the theory section the concept of VPN and the IPsec protocol suite is pre-
sented and the structure of the necessary modifications to Internet Protocol (IP) pack-
ets covered. Next the devices and vendors are briefly introduced following step-by-step
device configurations. Once the VPN tunnels have been established, problem situa-
tions are introduced to the VPN connectivity via systematic misconfigurations. These
issues are documented and identified from both devices’ perspectives via various
methods introduced by their respective vendors. After the issues have been identified,
a summary is drawn in conclusion to report the findings.
The motivation for the study comes from having worked in a multi-vendor environment
where access to only one tunnel endpoint is available. In order to save time and be
able to draw conclusions with proof to show in such cases, I chose this topic.
While this study covers Site-to-Site IPsec VPN connectivity troubleshooting methods, it
is limited to the devices Cisco’s ASA 5505 and Palo Alto Networks’ PA-200. Changes
in software versions and peering with any other network devices are likely to introduce
issues not covered by this troubleshooting guide.
2
The Internet, being a composite of various devices and mediums in-between, is typical-
ly treated as an obscure, virtual cloud. The ownership of the Internet is divided between
numerous Internet Service Providers (ISPs) and other organizations each responsible
for their own infrastructure as part of the Internet. End-users utilizing their ISP’s ser-
vices trust that they receive connectivity to all resources connected to the Internet.
However, in case of confidential data a layer of security should be added to the data
travelling across the Internet. Here we introduce the concept of VPN.
The purpose of a VPN, as its name implies, is the creation of a virtual private network
connection also called a VPN tunnel. The VPN tunnel establishes a logical connection
between two endpoints over a third-party network, such as the Internet, for security and
management purposes. As a security measure data can be delivered over third-party
networks encrypted and then authenticated and integrity checked at the end of the tun-
nel. For management purposes the whole third-party infrastructure can be viewed as a
flexible pipe spanning from one end to another without caring what there is physically
in-between.
VPN types are commonly divided into Site-to-Site and Remote Access VPNs. A Site-to-
Site VPN is established between two gateways connecting for instance a branch site’s
network to the headquarters’ network. A Remote Access VPN on the other hand con-
nects a host to a remote gateway for instance a telecommuter to corporate site. A less
commonly known third type, host-to-host, also exists, though it can be classified as a
restricted Remote Access VPN [1, 245–250]. This study uses Site-to-Site VPN to con-
nect two sites together, allowing their private networks to communicate securely.
IPsec is what commonly provides VPNs with the security they are known for – at least
in Site-to-Site implementations. IPsec is a protocol suite operating on the Network lay-
er, thus it complements VPN well when transmitting data between trusted networks
over an untrusted network. [2, 256–257].
3
Rather than being an actual tunnelling protocol, IPsec, defined by Internet Engineering
Task Force’s (IETF) RFC6071 document among others, is a modular collection of pro-
tocols to provide VPN tunnelling with data confidentiality, message integrity, data origin
authentication and anti-replay protection [3, 3]. IPsec’s ability to support various open
standards also allows for vendor interoperability [2, 256].
The IPsec framework comprises two main protocols: Authentication Header (AH) and
Encapsulating Security Payload (ESP). These two are responsible for providing the
security services of the IPsec framework. Figure 1 below breaks the framework down
and presents its components.
The data integrity ensures that data does not get lost or modified in transmit. This is
done by taking a packet’s content and running it through a cryptographic hash function.
This will result in a theoretically unique, irreversible and compressed bit stream known
as a hash. The hash generated at tunnel’s transmitting end is concatenated to the
packet to be used at the receiving end in verifying the packet’s integrity; if the hash
included in the packet matches with the hash generated at the receiving end the con-
tents can be verified to be unharmed. Cryptographic hash functions as shown in Figure
1 include:
Message Digest 5 (MD5)
Secure Hash Algorithm (SHA). [2, 256–259.]
Additionally utilizing Hashed Message Authentication Code (HMAC) with the chosen
hash function, the message authenticity can be verified along with its integrity as a
shared secret is used in producing the hash [5, 1–2]. Implemented together with a cryp-
tographic hash function such as SHA-1 results in a variant called HMAC-SHA-1 [6].
VPN peer authentication ensures that the VPN tunnel is established between the right
devices and is thus an essential step in forming VPN connectivity. The primary meth-
ods are usage of Pre-Shared Key (PSK), RSA signatures and RSA-encrypted nonces.
PSKs are manually configured and shared between the engineers responsible for the
VPN peers over a different communication channel. RSA signatures are digital certifi-
cates signed by a trusted, third-party Certificate Authority (CA). RSA-encrypted nonces
5
utilize public key cryptography making it more complex than PSK. RSA-encrypted
nonces as an authentication method is also limited to the Cisco Internetwork Operating
System (IOS) software. While RSA signatures possess superior scalability over PSKs
due to saving the trouble of configuring it separately for each individual peer, this study
will use the more easily deployed PSK as the peer authentication method.
Finally the Diffie-Hellman (DH) group selection will determine the security level of the
secure key exchange. DH, being a public-key cryptography protocol, allows the secure
generation of shared secret keys for encryption algorithms over third-party networks.
The higher the DH group number the more secure it is against brute-force attacks.
IPsec can operate either in Transport mode or Tunnel mode. The Transport mode pro-
tects packets up to the Transport layer and the Tunnel mode protects up to the Net-
work layer.
In the Transport mode IPsec places the header (AH, ESP or AH+ESP) between the
original IP header and the IP payload (data). This will leave the original IP header ex-
posed to traffic analysis when transmitted over public networks. [7, 457–459.]
In the Tunnel mode the IPsec header (AH, ESP or AH+ESP) encapsulates both the
original IP header and IP payload. Additionally a new IP header is created on top of the
IPsec header. This new IP header specifies the VPN peers as the source and destina-
tion addresses, thus concealing the actual message sender and recipient. While the
Tunnel mode increases the packet overhead, it provides additional security. [7, 457–
459.]
In practice IPsec security services are applied to traffic via AH and/or ESP header en-
capsulation. The AH and ESP headers contain the information necessary to validate
the IPsec Security Association (SA) and the data within. These headers are not applied
to IKE/IPsec negotiation exchanges and only for the data to be secured once the IKE
and IPsec SAs have been established.
6
The Next Header field contains the protocol number of the next header such as 4 for
IPv4 header. The Payload length specifies the length of the AH without the following
payload. The Reserved functions as padding and is set to zero. The Security Parame-
ter Index (SPI), combined with destination address and protocol type, acts as an identi-
fier for IPsec SAs – receiver should have a matching inbound SA. Sequence Number is
a running number used to protect against replay attacks by preventing retransmission.
Integrity Check Value (ICV) is a signed hash which the recipient uses to verify data
integrity and to authenticate the sender. [7, 465.]
ESP as the other IPsec security protocol provides the following features:
data confidentiality
data integrity
data origin authentication
anti-replay protection.
Figure 3 below displays the fields within an ESP header.
7
Compared to AH and it’s distinct header, ESP is scattered around the payload due to
the encryption scheme. ESP header comprises SPI and Sequence Number fields act-
ing as IPsec SA identifier and anti-replay protection just as with AH header. Following
ESP header is ESP Payload, the encrypted data. Following the ESP Payload is ESP
Trailer which contains Padding, Pad Length and Next Header. ESP Trailer is encrypted
along with ESP Payload and provides necessary padding to adjust the input for the
encryption algorithm. Next Header field in IPsec Tunnel mode has the number 4 for IP
Header inside the ESP Payload. Finally, like AH, ESP contains an ICV, ESP Authenti-
cation Data, to verify data integrity and authenticate the sender. [7, 466–470.]
Figure 4 portraits the AH and ESP encapsulations for both Transport and Tunnel mode.
8
Figure 4. IPsec encapsulation in Transport and Tunnel mode (modified) [2, 261; 8, 17–19.]
As shown in Figure 4 Transport mode leaves the original IP header exposed, whereas
Tunnel mode encapsulates it inside AH or ESP header and creates a new header for
transmission until tunnel endpoint. Figure 4 also highlights which headers are included
in ESP’s encryption and authentication (ICV). It is noteworthy to point out that AH addi-
tionally includes the IP header preceding the AH header to its ICV computation. As a
result AH requires that the topmost IP header remains immutable during transmit. [8,
17–19]. This renders Network Address Translation (NAT) as well as IPsec’s optional
feature Network Address Translation-Traversal (NAT-T) impossible to combine with
AH. Also, the ESP header does not specify Transmission Control Protocol (TCP) / User
Datagram Protocol (UDP) header information. This is only inside the inner, encapsulat-
ed IP header. In case NAT-T is enabled and discovered during IKE Phase 1 MM mes-
sages 3–4, a new UDP header will precede the ESP header with UDP port 4500 as the
source and destination port.
Internet Key Exchange (IKE) is the protocol responsible for the policy negotiations be-
tween peers and establishment of Security Associations (SA) in the IPsec framework.
IKE is in fact a hybrid protocol utilizing the functions of three protocols:
Internet Security Association and Key Management Protocol (ISAKMP)
9
Oakley
SKEME. [9, 2–3.]
ISAKMP provides IKE with the framework for SA negotiation, establishment, updating
and termination. Oakley is a key determination protocol and manages the key ex-
changes using a DH algorithm. Oakley can also provide Perfect Forward Secrecy
(PFS) for keys and peer identities. Similar to Oakley, SKEME defines public key en-
cryption methods for authenticated key exchange and swift re-keying via nonce ex-
change. [9, 2–3.]
IKE currently has two different implementations: IKE version 1 (IKEv1) and IKE version
2 (IKEv2). IKEv2, being the successor for IKEv1, runs most of the same features as
IKEv1 does with improved stability, more efficient exchanges and various other im-
provements. [10, 135–136]. However, as not nearly all networking devices yet fully
support IKEv2, IKEv1 is still widely implemented. This study will only cover configura-
tion steps for IKEv1.
IKE negotiations are divided into two phases: Phase 1 and Phase 2. IKE Phase 1 ne-
gotiates IKE policies, performs DH key exchange and peer authentication in order to
establish a bidirectional IKE SA – also known as ISAKMP SA. IKE Phase 2, utilizing
the existing IKE SA, negotiates IPsec security parameters for tunnelled traffic and es-
tablishes a pair of unidirectional IPsec SAs. [2, 284–288.] An optional IKE Phase 1.5
also exists to provide user authentication via Extended Authentication (Xauth) [2, 263–
266].
IKE Phase 1 can operate in two modes: Main Mode (MM) and Aggressive Mode (AM).
Both MM and AM perform the same tasks; MM does this in 6 messages while AM in 3
messages. MM stages are as follows:
1. The Initiator sends the receiver its IKE policy sets as proposals
2. Receiver compares the policy sets to its own and sends the agreed policy set
back
3. Initiator sends its DH public key and nonce to initiate DH key exchange
4. Receiver sends its DH public key and nonce completing DH key exchange
With the exchanged keys, they both compute a shared secret
5. Initiator sends its identity payload encrypted and hashed
10
6. Receiver decrypts and validates the hashed identity payload in order to authen-
ticate the peer then sends its identity payload encrypted and hashed in return.
In comparison AM stages are as follows:
1. Initiator sends its IKE policy set proposal, DH key materials and identity payload
2. Receiver replies with the selected IKE policy set, its own DH key materials and
identity payload with authentication hash for Initiator to authenticate.
3. Initiator sends its encrypted authentication hash for receiver to authenticate.
While AM employs more efficiency in its exchanges, it does not provide protection for
identity payload and receiver has to agree on initiator’s DH group selection. [9, 7–15.]
IKE Phase 2 utilizes the established IKE SA applying the negotiated security parame-
ters to protect its exchanges via encryption and authentication. IKE Phase 2 operation
mode is called Quick Mode (QM). QM exchanges the following three messages:
1. Initiator sends the IPsec policy set proposal, a nonce and a hash payload
If PFS is enabled, new DH key materials are exchanged for IPsec SAs
2. Receiver sends the agreed IPsec policy set, its own nonce and a hash payload
If PFS is enabled, new DH key materials are exchanged for IPsec SAs
3. Initiator sends a hash payload generated with the nonces completing the ex-
change [9, 17.]
4 Device Setup
For the study two firewalls capable of acting as gateways for a small to mid-size site
were chosen. Among the feature requirements were support for IPsec Site-to-Site VPN
and IPv6 deployment. The devices chosen were Palo Alto Networks’ PA-200 and Cis-
co’s Adaptive Security Appliance (ASA) 5505. The two are the most affordable models
among their series of firewalls.
Palo Alto Networks’ PA-200 is a next-generation security appliance running Palo Alto’s
PAN-OS operating system. The device comes with a 4-port Ethernet switch supporting
Layer 3, Layer 2 and transparent Virtual-wire interface modes as well as separate
management, console and USB ports. PA-200 is capable of applying security policies
not only based on the IP address, zone, port or protocol but also based on User-ID,
App-ID and Content-ID. In addition, the firewall provides Intrusion Prevention System
(IPS) engine for traffic scanning, Secure Sockets Layer (SSL) and IPsec VPN tunnel-
ling and virtualization of system or routing instances for scalability. [11.]
For initial configuration either the console port or the pre-configured Management port
(MGT) should be used. This study will handle device configurations via GUI and use
both GUI and CLI for troubleshooting. The Web-based GUI is illustrated in Figure 5
below.
12
The Dashboard contains selectable widgets to display the overall operating status and
recent events of the device. The ACC constructs illustrious application usage, Uniform
Resource Locator (URL) filtering, data filtering and threat prevention statistics based on
traffic logs. The Monitor allows the viewing and filtering of traffic, threat, URL, data,
configuration, system and alarm logs. Monitor also allows packet capture and reporting.
Policies section enforces the firewall’s security policies, NAT rules, Policy-Based For-
warding (PBF) and Quality of Service (QoS) policies. Objects tab contains all defini-
tions pertaining to entities or entity groups given custom identifiers to be referenced in
places such as security policies. Network contains all configurations regarding interfac-
es, virtual routers, routing protocols, zones, Virtual Local Area Networks (VLANs), IP-
sec tunnels, Dynamic Host Configuration Protocol (DHCP) and QoS. Finally all device
13
management settings such as access rights, MGT port settings, service routes, log
management as well as software and content updates are located under the Device
tab.
The Cisco ASA 5505 is a network security appliance running Cisco Systems’ IOS op-
erating system. The device has a built-in 8-port Layer 2 Ethernet switch, requiring logi-
cal VLAN interfaces for Layer 3 connectivity, as well as a console port and two USB
ports. ASA 5505 integrates a stateful, zone-based firewall with IPsec/SSL VPN concen-
trator and IPS engine.
For initial configuration the console port should be used. While the factory default con-
figuration comes with pre-configured VLAN1 and VLAN2 interfaces, we will start with a
clean start-up configuration. In this case access to the ASDM needs to be explicitly
configured. To allow an internal host 10.0.2.1, connected to Ethernet0/0 port, access
the ASDM GUI, the interface, HTTP server service and an access filter need to be con-
figured to allow HTTP/HTTPS requests. Listing 1 below shows how to allow a host to
access the ASDM.
interface Vlan10
nameif trust
security-level 100
ip address 10.0.2.254 255.255.255.0
!
interface Ethernet0/0
switchport access vlan 10
no shutdown
!
http server enable
http 10.0.2.1 255.255.255.255 trust
This study goes through the ASA device configurations via CLI and uses both ASDM
and CLI for troubleshooting. The ASDM is illustrated in Figure 6 below.
14
The Device Dashboard under Home view displays general device information, system
resource and interface statuses, and traffic levels. The adjacent Firewall Dashboard
draws graphs for session and packet filtering statistics. The Configuration tab allows
the configuration of interfaces, routing protocols, firewall policies, SSL/IPsec VPNs and
Management services. The Monitoring tab displays interface statuses, routing tables,
VPN session details and statistics, and logging data.
The ASDM top panel additionally holds useful features under File, Tools and Wizards
menus. The File drop-down menu lists configuration management functionalities. Tools
menu contains troubleshooting functionalities such as Packet Tracer, Ping and Trac-
eroute tools, updating services and even allows usage of CLI via ASDM. Wizards menu
contains step-by-step setups for device start-up configuration, VPN configuration as
well as a Packet Capture Wizard for capturing specified traffic passing the ASA firewall.
The network topology comprises the two firewalls, PA-200 and ASA 5505, a router rep-
resenting a third-party hop in-between the firewalls and a host behind each firewall.
15
The firewalls act as the gateway for the two sites allowing the sites’ private networks to
connect with the use of a VPN. Figure 7 below illustrates the network topology.
The site behind PA-200 will have 10.0.1.0/24 as its internal Local Area Network (LAN).
PA-200 site’s internal host’s Network Interface Card (NIC) is configured with the IPv4
address 10.0.1.1, subnet mask 255.255.255.0 and default gateway 10.0.1.254. Similar-
ly the site behind ASA 5505 uses 10.0.2.0/24 as its LAN while the site’s host is config-
ured with IPv4 address 10.0.2.1, subnet mask 255.255.255.0 and default gateway.
Although third-party networks, the public network facing interfaces on PA-200 and ASA
5505 have 172.16.1.0/30 and 172.16.2.0/30 respectively as their upstream networks.
All device connections use standard Ethernet cables (RJ-45). Table 1 presents the
whole IP addressing scheme.
16
IP Addressing Table
In addition to the topology view in Figure 7, the router will get its Loopback0 interface
configured when NAT-T is covered later in the Experimenting and Troubleshooting sec-
tion. This will give the router a distinct pool of addresses for NAT.
5 Device Configuration
This section will cover the steps and commands required to configure the devices in
the topology. The host computers are configured with static IPv4 addresses and default
gateways according to Table 1. The third-party router only has its interfaces configured,
thus only routing between its directly connected networks 172.16.1.0/30 and
172.16.2.0/30.
The configuration steps or commands may vary with different software versions. The
software versions used in this study are as follows:
PA-200 software version 6.1.4
ASA 5505 software version 9.2(4)
ASA 5505 ASDM version 7.4(3).
17
In the case of PA-200 the GUI is recommended over CLI when applying configurations.
As the configuration file is structured in a hierarchical manner, dependencies caused
by virtual routers and virtual systems can be confusing. GUI covers the required de-
pendencies to alleviate confusion. In case all necessary dependencies are not handled
before committing the configuration changes, the commit prompt will issue a warning.
Before beginning, due to choosing a Layer 3 deployment, the default virtual-wire con-
figuration should be purged as the Ethernet1/1 and Ethernet1/2 interfaces are initially
bound and allow traffic pass the PA-200 firewall transparently without switching or rout-
ing. Via CLI we execute the following commands under the configuration mode as
shown in Listing 2 below.
While the MGT port can be used access the GUI with its default settings, in this study
the internal LAN interface will be configured to allow access to the device GUI as
shown in Listing 3 below.
With management access allowed through the data port Ethernet1/1, the host config-
ured with IPv4 address 10.0.1.1/24 and connected to the aforementioned port can now
access https://10.0.1.254 via its web browser. On the GUI Network tab we will create
three Layer 3 zones named ‘trust’, ‘untrust’ and ‘vpn’ for policies and traffic monitoring.
By default zones named ‘trust’ and ‘untrust’ existed as virtual-wire type zones.
18
Next we will finish the interface configurations. Interfaces are configured as Layer 3
interfaces with IP addressing according to Table 1. Choose ‘default’ as the virtual rout-
er and assign Ethernet1/1 to ‘trust’ zone and Ethernet1/2 to ‘untrust’ zone.
To finalize the basic configurations, a default static route towards the third-party Router
will be configured to establish routing between the devices. This is set under Network >
Virtual Routers.
19
To start off the policy negotiations, IKE parameters will be configured. Much like with
other features, a default IKE Crypto Profile already exists. Let us create a new one with
a comprehensive name ‘IKE_P1_ASA’ identifying that we are configuring IKE Phase 1
settings to be used in negotiations with ASA. As displayed in Figure 12 below, AES128
20
is used for encryption, SHA-1 for authentication and DH group 5 for key exchange. IKE
SA lifetime will be set to 8 hours.
Continuing onto Phase 2, again a default set of parameters exists. This time the IPsec
Crypto Profile will be named ‘IPsec_P2_ASA’ to distinguish it as Phase 2 parameter
set. Here as well AES128 and SHA-1 will be selected. ESP will be selected as the IP-
sec protocol. IPsec SA lifetime will be set to 1h. No PFS will be required due to the
relatively low SA lifetimes.
21
In order to bind all details about the VPN peer together an IKE Gateway will be defined.
Here we set the physical interface to reach the VPN peer, VPN peer identity and au-
thentication method as PSK. The peer has a static IPv4 address 172.16.2.1 and it will
also be used to identify the peer. In the Advanced Options tab the Exchange Mode is
by default ‘auto’ which allows both MM and AM. Here we also map the IKE Crypto Pro-
file ‘IKE_P1_ASA’ to this IKE Gateway.
The Last step in configuring the VPN tunnel is creating an IPsec Tunnel. This IPsec
Tunnel named ‘Tunnel_to_ASA’ will be bound to the Tunnel Interface ‘tunnel.1’. This is
also where the IKE Gateway and IPsec Crypto Profile are linked together. If the Tunnel
Interfaces had IP addresses assigned, we can enable the Tunnel Monitor option here.
The QM Proxy-IDs or Encryption domains are also defined here, which is an imperative
step when support for policy-based VPNs is required, like when peering with an ASA.
In order to allow and monitor the traffic flow related to the VPN, we create policies to
allow bidirectional traffic between the local network 10.0.1.0/24 and remote network
10.0.2.0/24. We also explicitly allow incoming IKE and IPsec traffic for traffic logging.
Lastly, a static route to the remote network 10.0.2.0/24 is required for traffic flow. These
packets will be routed to the Tunnel Interface. No next-hop IPv4 address is configured,
since the Interface does not use IP addresses.
23
Optionally, we can implement NAT or Port Address Translation (PAT) to allow the in-
ternal LAN to access public networks. NAT operation is performed before IPsec, so it
must be accounted for. However, as we have assigned a separate zone ‘vpn’ for the
VPN tunnel interface, there will be no need to exempt tunnel traffic from address trans-
lation. Figure 18 displays the configured NAT rule on PA-200.
According to Figure 18 the NAT (PAT) rule will use the ‘untrust’ interface’s IP address
for traffic sourced from internal network 10.0.1.0/24 and destined to any destination
network belonging to ‘untrust’ zone.
24
The configuration wizards in ASA 5505’s CLI and ASDM tend to create an unnecessary
number of default objects; thus the configurations will be done manually via CLI.
Before the IPsec VPN tunnel can be configured the interfaces and default routing need
to be configured. As ASA’s switch ports operate at Layer 2, we also configure VLAN
interfaces to which the switch ports are assigned. For the site’s local network
10.0.2.0/24 we assign the highest security level and call it our ‘trust’ zone. The host
computer is connected to the Ethernet0/0 switch port. For the site’s third-party-
connected network 172.16.2.0/30 we assign the lowest security level and call it our
‘untrust’ zone. The third-party router is connected to the Ethernet0/1 switch port. Final-
ly, we tell ASA to route packets towards the third-party upstream router by default. List-
ing 4 below shows how to set up this preliminary configuration.
interface Vlan10
nameif trust
security-level 100
ip address 10.0.2.254 255.255.255.0
!
interface Vlan20
nameif untrust
security-level 0
ip address 172.16.2.1 255.255.255.252
!
interface Ethernet0/0
switchport access vlan 10
no shutdown
!
interface Ethernet0/1
switchport access vlan 20
no shutdown
!
route untrust 0.0.0.0 0.0.0.0 172.16.2.2
Next we will define the traffic we wish to have traverse the VPN tunnel. For this we de-
fine an ACL (Access Control List) stating the source (10.0.2.0/24) and destination
(10.0.1.0/24) networks. Let us call this ACL ‘PROXY-ID-ACL’. For future referencing,
we create network objects for local and remote networks. Listing 5 below shows the
equivalent configuration.
25
The configuration for the actual VPN tunnel and peer is shown in Listing 6 below. Ex-
clamation marks are used to separate the commands into logical sections.
For the IKE negotiations we first need at least one IKEv1 policy – also known as the
ISAKMP policy. We define all the security parameters for the IKE policy set and enable
the negotiations on the ‘untrust’ interface which terminates the VPN. Since PSK is used
as the authentication method, let us next configure a tunnel group for the VPN peer.
The tunnel is configured as Site-to-Site – or LAN-to-LAN (L2L) as ASA understands it.
In the tunnel group we define the PSK (palocisco) for this IPsec VPN peer.
With IKE settings configured, next we define our IPsec parameters. For this we create
a transform-set ‘ESP-AES1-SHA’. By default the tunnel mode is used. Finally we link
the VPN peer, transform-set and ACL together with the use of crypto map ‘VPN_MAP’.
The crypto map is then applied to the same interface as the IKE policies. PFS is not
configured.
26
As NAT is a common reason why traffic does not enter a VPN tunnel, in addition to
incorrect routing and security policies denying the traffic, let us additionally implement it
for the upstream traffic that will not pass the VPN tunnel. For this purpose we will need
an extra exempt NAT rule in addition to the basic NAT according to Listing 7 below.
The first NAT rule, an exempt NAT rule, specifies that traffic sourced from network
10.0.2.0/24 (zone ‘trust’) and destined to network 10.0.1.0/24 (zone ‘untrust’) is not to
be translated. The second NAT rule, a dynamic PAT rule, states that any traffic
sourced from the ‘trust’ zone and destined to the ‘untrust’ zone is to be dynamically
translated to use the outgoing interface’s IP address as a new source and get a new
custom source port assigned to this session. The keyword after-auto will guarantee
that the more specific NAT rules will have a higher priority during a NAT lookup.
In this chapter an issue at a time will be introduced to the VPN tunnel, reversing the
troubleshooting process of using commands and tools to determine the cause of failure
then deriving the proper corrective measure. In this case the cause of failure is known
beforehand, and unfolding the methods, to determine that this very incident is the
cause of failure, is the objective. For the test traffic simple ICMP Requests (Ping)
sourced by the hosts are used. At first the available troubleshooting tools will be cov-
ered.
System Log helpfully logs IKE/IPsec-related events among other events. Traffic Log
lists allowed and denied sessions matching the Security Policies that have logging en-
abled. While Traffic Log concentrates more on session history and traffic statistics,
Session Browser offers details regarding the ongoing sessions and their flows. Packet
Capture can capture packets at different stages of the firewall; received, inspected,
transmitted and dropped. The output can then be viewed in the Wireshark format.
On PA-200’s CLI for convenience’s sake, we will first modify the console/terminal out-
put such that it will use the whole screen worth of space to print the output.
For debugging IKE negotiations we determine our logging level: normal or debug. The
normal level, which is the default, will generally report about successes and failures,
but typically the debug level is required to find the cause. For real-time CLI logging we
specify tail command to follow the output of the management log called ‘ikemgr.log’
in real-time. The log can also be read later by replacing tail with less, which will
print the whole log file’s contents.
To verify the tunnel’s parameters, SPI, lifetime and statistics, we use the show vpn
flow command as such:
show vpn flow tunnel-id 1
detailed than the debugging level. Packet Capture Wizard can be used to confirm traffic
flow through ASA.
For VPN debugging ASA 5505 via CLI the following commands should prove useful:
The command debug crypto condition is used to narrow the debugging down to
only the interesting output, in this case the tunnel with the peer 172.16.1.1. The com-
mand debug crypto ikev1 127 will trigger a debug output via the console termi-
nal. This will cover both IKEv1 phases, while the debug crypto ipsec variant will
provide additional data regarding the IKEv1 Phase 2. The following integer can vary in
range 1–255 determining the debug level. Level 127 is generally informative and com-
pact enough. In comparison the debug level 254 will also include ISAKMP header in-
formation and is required in order to view the contents of the message payloads such
as the SA payload.
To verify the state of an IPsec tunnel and the related SAs via CLI we type:
The first displays the type, role and state for each active IKEv1 peer. While IKE SA
state is typically of interest to determine how far the negotiations have progressed, the
device in the role of a responder generally has easier time to determine the cause(s) of
failure. The second covers both IKEv1 and IKEv2 SAs as well as the chosen policy set
parameters. The third command displays extended information regarding all IPsec SAs
and their statistics. The fourth provides detailed IKEv1, IKEv2 and IPsec session infor-
mation with all policy parameters, lifetimes and Proxy-IDs present.
Due to the abundant amount of available troubleshooting tools, priority will be given to
the concise information provided by PA-200’s GUI System Log and ASA’s ASDM Log-
ging features. Should these fail to catch onto the issue, the secondary measure will be
CLI debugging followed by the rest of the methods. The troubleshooting subjects will
be divided into CASEs each covering multiple scenarios depending on the experiment
29
arrangements. Any changes done in a scenario are rolled back for the following sce-
nario.
Proxy-ID mismatch indicates that the tunnel endpoints have disagreement regarding
which source and destination networks are allowed to traverse the tunnel. Proxy-IDs
are negotiated during QM messages 1–2. While the IPsec policy proposal is contained
in the SA payload, Proxy-IDs are located in a separate Identification payload. During
the configuration phase ASA 5505 also refers to them as ACLs assigned to a crypto
map. Three different scenarios were used in experimentation.
In the first scenario Proxy-ID definitions were erased on PA-200, causing IKE Phase 2
to fail. Figures 19 and 20 below portray the issue on ASA 5505 ASDM’s end.
In both the role of a receiver and initiator ASDM Logging triggers errors during IKE
Phase 2 QM negotiations. However, only as a receiver is ASDM able to deduce that
this is due to a Proxy-ID mismatch. Additionally running debug crypto ikev1 127
will bring the details shown in the following Listing 8 when ASA is acting as the initiator.
30
This output essentially does not reveal the issue either, but tells us that the peer does
not send the continuation message or any kind of notification. Thus ASA’s end hangs
at the ‘QM_WAIT_MSG2’ state, being left waiting for the second message of QM which
never comes.
On PA-200’s end the logging output reflected similar results. Figures 21 and 22 below
depict the System Log output on PA-200:
Conveniently, without an ACL mapped to the crypto map, the ASA will not even attempt
to establish the tunnel. It will, however, reply to PA-200’s attempts to do so. Figures 23
and 24 show the log entries from both perspectives when PA initiates the negotiations:
Figures 23 and 24 reflect essentially the same output as Figures 21 and 19; the receiv-
er (ASA) rejects the initiator’s (PA) Proxy-ID proposal and just sends a notification that
the contents of the Identification payloads were no good.
In the third scenario the ACL on ASA’s side was switched to one allowing all traffic to
the tunnel. Configuration changes on ASA were as follows:
Having ASA initiate the negotiations will cause negotiations to break down with slight
delay. Figures 25 and 26 show how PA will not accept ASA’s proposal for Proxy-IDs,
and how ASA after determining that negotiations do not progress terminates the IPsec
negotiations and the IKE SA, as no IPsec SAs were successfully established.
32
Figures 27 and 28 below show PA-200 assuming the initiator’s role in turn.
As a receiver ASA agrees to establish the IPsec SA with the proposed Proxy-IDs par-
tially matching its own. Soon enough ASA realizes that its crypto map ACL does not
33
have matching IPsec SAs even though an IKE SA has been established. Thus ASA
continues offering its 0.0.0.0/0 network only to get rejected by PA time and time again.
ASA then continues to attempt creating a new IPsec SA pair.
PSK mismatch during IKE Phase 1 occurs when the receiver is unable to process the
MM message 5, which directly follows the DH key exchange. The Receiver’s inability to
process the packet will prompt it to resend the previous MM message 4. As a result
The Initiator will receive a duplicate message and end up resending MM message 5
and will expect MM message 6 in return.
To start off, Figures 29 and 30 below play out the scenario when PA initiates the nego-
tiations:
As seen in Figure 29, PA-200 ends up hanging and eventually time-outing the negotia-
tions. ASA 5505, being on the receiving end, realizes an error processing the encrypt-
ed packet and even suspects a mismatched PSK as proven by Figure 30. This deduc-
34
tion is based on the decryption resulting in invalid payloads. ASA sends this information
to PA as a separate message with Notification payload of type ‘PAYLOAD-
MALFORMED’. PA’s debugging does not indicate to trigger any actions on PA’s side
regarding this piece of information. Such details can be spotted by running Packet Cap-
ture on PA filtered by the peer addresses. Figure 31 shows the Packet Capture output.
Figures 32 and 33 show that as in the former case, the initiator (ASA) cannot determine
the issue by itself, while the receiver (PA) suspects a PSK mismatch, which is indeed
correct. The MM messages 5–6 are encrypted, so with decrypting them with incorrect
key(s), the payloads will end up invalid. However, PA withholds the information and the
MM messages 4–5 continue to be retransmitted. ASA as the initiator soon ends the
loop and informs PA via an encrypted message carrying a Delete payload specifying
the SPIs – or better known as initiator and responder cookies during Phase 1. Listing 9
presents how the error loop occurs.
Upon receiving and decrypting the message, PA chooses to ignore it, since the SA has
not been established yet, continuing to resend MM message 4. ASA, having already
deleted the SAD entry for this incomplete ISAKMP (IKE) SA, now sends PA a notifica-
tion of type ‘INVALID-COOKIE’ since the cookie (SPI) pair identifying the SA that was
being negotiated no longer exist in its database. Running Packet Capture on PA again
reveals how this exchange looks like as printed on Figure 34.
PA gets to receive two of such notifications before its negotiations timeout. It becomes
apparent that PA’s support for ISAKMP notifications is at a lower level than ASA.
IKE policy negotiation takes place during MM messages 1–2 and is used to agree on a
policy set comprising an encryption algorithm and key length, hash algorithm, DH
group, authentication method and key lifetime. Mismatches in the encryption algorithm,
hash algorithm, DH group and authentication method result in clear error messages,
whereas lifetime is more flexible and may thus result in difficult-to-spot misbehaviours
when a session reaches the rekeying stage.
In order to view IKE policy proposals as the receiver for the CLI debugging we will use
debug level on PA and 254 level on ASA as follows:
In the experiments the IKE policy parameters were changed on PA-200 one by one
and the output recorded; AES128 was changed to 3DES, SHA-1 to MD5 and DH5 to
DH14. Encryption and hash algorithm mismatches had the exact same error messag-
es, so these will be covered together. This will be followed by the DH group mismatch
with a more distinct reaction. Figures 35 and 36 picture the IKEv1 encryption and/or
hash mismatch scenario when PA-200 initiates the negotiations.
Figure 35. IKEv1 policy mismatch (encryption or hash algorithm): PA-200 as initiator
37
Figure 36. IKEv1 policy mismatch (encryption or hash algorithm): ASA 5505 as receiver
Here we may observe that PA’s proposal is rejected and the notification ‘NO-
PROPOSAL-CHOSEN’ is send. PA chooses to ignore this and simply resend the pro-
posal until negotiations timeout.
Figure 37. IKEv1 policy mismatch (encryption or hash algorithm): ASA 5505 as initiator
Figure 38. IKEv1 policy mismatch (encryption or hash algorithm): PA-200 as receiver
38
The scarce amount of output data can be explained by looking at PA’s CLI output as
shown in Listing 11 below.
While ASA did send a notification message, PA did no such thing. PA compares the
policy sets until the last one, determines the reason for rejection and ends the negotia-
tions there while ASA retries until timeout.
Next comes the DH group mismatch scenario. Having PA to initiate the negotiations,
the output shown in Figures 39 and 40 will be generated.
Figure 40. IKEv1 policy mismatch (DH group): ASA 5505 as receiver
39
The error messages and behavior are about the same for DH group mismatch as in the
previous scenario of encryption or hash algorithm mismatch. Only notable difference is
that ASA reports the DH group mismatch. Apparently for this software version DH
group 14 and above appears as ‘Unknown’ whereas group 2 would appear properly.
PA receives ASA’s mismatch notifications like seen in Figure 39.
In the case of ASA initiating the negotiations, PA would react in the same manner as
with encryption or hash mismatch; realizing DH group mismatch has occurred and end-
ing the negotiations without a word.
Finally we have the IKEv1 SA lifetime mismatch. This will be broken down into the fol-
lowing four scenarios:
1. PA initiates negotiations with a higher lifetime value (86400s > 28800s)
2. ASA initiates negotiations with a lower lifetime value (28800s < 86400s)
3. PA initiates negotiations with a lower lifetime value (7200s < 28800s)
4. ASA initiates negotiations with a higher lifetime value (28800s > 7200s).
In scenario 1 when PA proposes 86400 second lifetime value, ASA replies without
changing the lifetime value in the policy set. However, ASA has chosen to utilize its
lower lifetime of 28800 seconds. ASA will convey this information to PA at the end of
MM negotiations via a notification message ‘RESPONDER-LIFETIME’. The parties will
continue with mismatched IKE SA lifetimes until ASA prompts a rekeying process.
In scenario 2 when ASA proposes 28800 second lifetime value, PA agrees to the lower
key lifetime value over its higher 86400 second lifetime. ASA receives confirmation of
this in the MM message 2.
In scenario 3 when PA proposes 7200 second lifetime value, ASA will agree on the
proposed lower lifetime value over its higher 28800 second lifetime. No changes occur
to the following message exchanges.
In scenario 4 when ASA proposes 28800 second lifetime value, PA replies without
changing the lifetime value in the policy set. However, PA has chosen to utilize its low-
er lifetime of 7200 seconds. PA abstains from sending the notification message to ASA.
The parties will continue with mismatched IKE SA lifetimes until PA prompts a rekeying
process.
40
IKE SA lifetime mismatch thus does not generate blatant warning or error messages, at
most notification messages informing the peer. In the end vendors are given leeway in
deciding how the peers will react to these lifetime mismatch situations. [12, 22–23].
The IPsec policy negotiation takes place during QM messages 1–2 and is used to
agree on a policy set comprising an encapsulation protocol, encryption algorithm and
key length, hash algorithm, key lifetime in seconds and key lifesize in kilobytes. An op-
tional DH group is also negotiated if PFS is enabled. Tunnelling mode will be the solely
used encryption mode. Very much like during IKE policy set negotiation, IPsec policy
set mismatches in encapsulation protocol, encryption algorithm, hash algorithm or DH
group (PFS) result in clear error messages, whereas lifetime is more flexible.
In order to view IPsec policy proposals as the receiver for the CLI debugging we will
use debug level on PA and 254 level on ASA as follows:
In the experiments the IPsec policy parameters were changed on PA-200 one by one
and the output recorded; AES128 was changed to AES256, SHA-1 to MD5, PFS group
to DH2 when enabled and ESP encapsulation to AH. The more varied IPsec SA life-
time talks will be covered at the end of the section.
As with during IKE policy negotiations, IPsec encryption and hash algorithm mismatch-
es had the exact same error messages, so these will be covered together. This will be
followed by the DH group (PFS) mismatch and encapsulation protocol mismatch. Fig-
ures 41 and 42 depict the IPsec encryption and/or hash mismatch scenario when PA-
200 initiates the negotiations.
41
Figure 41. IPsec policy mismatch (encryption or hash algorithm): PA-200 as initiator
Figure 42. IPsec policy mismatch (encryption or hash algorithm): ASA 5505 as receiver
Having received PA’s IPsec policy set proposal, ASA rejects the mismatched set and
sends PA the notification message ‘NO-PROPOSAL-CHOSEN’. ASA then informs PA
of IKE SA deletion. PA still continues to offer its IPsec proposal until negotiation
timeout. Meanwhile ASA drops the following proposals which no longer have an asso-
ciated SA.
For the reverse interaction, Figures 43 and 44 reflect the occurring negotiations.
Figure 43. IPsec policy mismatch (encryption or hash algorithm): ASA 5505 as initiator
42
Figure 44. IPsec policy mismatch (encryption or hash algorithm): PA-200 as receiver
Just as above, once ASA’s IPsec proposal gets rejected by PA, PA sends the notifica-
tion message ‘NO-PROPOSAL-CHOSEN’. Listing 12 shows PA’s CLI output.
Having compared ASA’s proposal to its own, PA compiles the notification message to
be sent to ASA. Each consecutive time ASA’s proposal arrives, it performs the same
IPsec policy comparison and replies with the same notification message until ASA’s
negotiations timeout and notification for IPSEC and IKE SA deletion arrives.
When PFS is enabled on PA, but disabled on ASA, the System Log output will look
exactly the same when PA is in the initiator’s role as if the mismatch’s cause was en-
cryption or hash algorithm. This occurs since ASA does not have a PFS group config-
ured to compare the proposal’s group to. When PA, who has PFS enabled, is in the
receiving end, the PFS group mismatch is apparent on the GUI’s System Log alone.
43
PA’s PFS has been set to DH group 2 while ASA does not have it enabled, which
shows up as DH group 0. Figures 45 and 46 portray this exchange scenario.
In the event that PA proposes the usage of AH encapsulation to ASA who uses ESP –
or vice versa, the system log output will not change; same ‘NO-PROPOSAL-CHOSEN’
notification message is sent once again. The CLI debug output in the receiver’s point of
view changes slightly for both parties. ASA’s reaction is as shown in Listing 13.
When ASA in turn initiates the negotiations, PA’s debug output is as shown in Listing
14 below.
From PA’s CLI debugging output we can also note that AH indeed does not exercise
encryption as this field is filled with its hash algorithm (SHA1) instead. Interestingly
enough ASA has been confirmed to be the more aggressive peer with regards to SA
terminations be it as initiator or receiver.
Last but not least comes IPsec SA lifetime and lifesize mismatch. PA’s lifesize has
been left at the default zero value, which practically makes it unlimited, thus making
ASA’s specified lifesize value being preferred. Like with IKEv1 SA lifetime troubleshoot-
ing, this will be broken down into the following four scenarios:
1. PA initiates negotiations with a higher lifetime value (7200s > 3600s)
2. ASA initiates negotiations with a lower lifetime value (3600s < 7200s)
3. PA initiates negotiations with a lower lifetime value (1800s < 3600s)
4. ASA initiates negotiations with a higher lifetime value (3600s > 1800s).
In scenario 1 when PA proposes 7200 second lifetime value, ASA returns the proposal
and adds a notification payload specifying that it has chosen the lower lifetime value of
3600 seconds and lifesize of 4608000 KB. PA updates its IPsec SA lifetime and lifesize
45
Listing 15. ASA forcing its lower IPsec SA lifetime and lifesize onto PA.
In scenario 2 when ASA proposes 3600 second lifetime value, PA agrees to the lower
SA lifetime proposal and the preferred lifesize of 4608000 KB.
In scenario 3 when PA proposes 1800 second lifetime value, ASA agrees to the lower
SA lifetime proposal, but notifies PA of choosing its preferred lifesize as shown in List-
ing 16 below.
In scenario 4 when ASA proposes 3600 second lifetime value, PA decides to keep its
lower lifetime of 1800 seconds and includes a notification payload in its response. PA
also unconditionally agrees to the proposed IPsec SA lifesize of 4608000 KB. ASA in
turn chooses to update its lifetime value to PA’s lower 1800 seconds as shown in List-
ing 17 below.
In conclusion, IPsec SA lifetime negotiations between these two peers will always
agree on the lower lifetime value regardless of the negotiation role. While PA’s debug
46
output shows that ASA’s notification arrives, it does not explicitly state whether the noti-
fication is ignored or agreed upon until the lifetime is updated at the end of the negotia-
tions.
While the theoretical part of this study does not cover IKEv2 protocol in detail, introduc-
ing a scenario where one peer speaks IKEv1 and another IKEv2 is definitely worthwhile
to find out how it actually affects the negotiations. To implement the configurations for
IKEv2, we will use Cisco’s IKE migration feature, which requires an existing IPsec VPN
configuration for IKEv1 with PSK authentication and will create a matching configura-
tion for IKEv2. Issuing the CLI command migrate l2l on ASA will result in the fol-
lowing configuration changes shown in Listing 18.
IKEv2 now functions parallel to IKEv1 sharing the same transform set and crypto map.
IKEv2 will automatically use the same PSK as was configured for IKEv1. In order to
debug IKEv2 the following additional CLI debugging commands are introduced:
debug crypto ike-common 5
debug crypto ikev2 platform 4
debug crypto ikev2 protocol 4
The command debug crypto ike-common will print general tunnel management
output for both IKEv1 and IKEv2, thus making it useful to determine which protocol is
used to build a tunnel. The commands debug crypto ikev2 platform and de-
47
bug crypto ikev2 protocol generate messages for IKEv2 negotiation exchang-
es and protocol’s processing. The debugging level can be set within the range 1–255.
A lower debugging level is used here to just debug the version mismatch scenarios.
For the first scenario, we will temporarily disable IKEv1 forcing ASA to only negotiate
with IKEv2. To disable IKEv1 negotiations on the peer-facing interface, we issue:
no crypto ikev1 enable untrust
Initiating the negotiations from PA’s side will only prompt connection building and
teardown on the informational syslog level. PA will keep on sending its IKEv1 proposal
until the SA negotiation times out. ASA’s IKEv1 CLI debugging prints the output shown
in Listing 19.
After reversing the roles and initiating the negotiations from ASA, PA generates output
matching Listing 20.
48
[DEBUG]: isakmp.c:1043:isakmp_handler():
521 bytes message received from 172.16.2.1[500]
[PROTO_WARN]: ikev2.c:262:ikev2_input():
0:172.16.1.1[500] - 172.16.2.1[500]:0x103cc268:unknown ikev2 peer
So, in both cases the IKE proposals are silently discarded, and the initiating peer will
continue offering their proposal until negotiation timeout.
For the next scenario IKEv1 will be re-enabled on ASA to support both versions in par-
allel. When ASA begins the IKE SA negotiations it begins with the preferred IKEv2, but
after timing out it will proceed to attempt IKEv1 negotiations, which then succeed. PA
will as before discard the IKEv2 proposals and only reply when IKEv1 proposals are
received. Listing 22 shows the common IKE exchange occurring on ASA’s CLI.
49
Listing 22. ASA falls back to IKEv1 after failed IKEv2 negotiations.
For reference, it took 7 proposals and 2 minutes for IKEv2 to timeout before IKEv1 was
given the baton. Should PA initiate the negotiations, ASA would automatically revert to
IKEv1 as soon as the first proposal comes, since the ISAKMP (IKE) header comes with
a Version field.
IKE mode or exchange type is determined by the initiator in its first message. The re-
ceiver will then either agree or disagree on the proposed exchange type. By default
both ASA 5505 and PA-200 will use auto mode, which will initiate negotiations using
MM, but will agree to both MM and AM proposals. That being the case, the only sce-
nario to be covered is having PA configured for AM and ASA for MM. Figure 47 shows
PA-200 being configured for AM.
Now, on the ASA’s CLI we can disable AM and only allow MM with the command:
crypto ikev1 am-disable
When ASA receives PA’s IKE policy proposal, it only needs to look at the ISAKMP
header’s Exchange Type field to determine the following action:
[IKEv1]IP = 172.16.1.1, Aggressive Mode connections disabled on
interface ... dropping pkt
When PA receives ASA’s IKE policy proposal, the reaction is just as anticlimactic:
[PROTO_ERR]: ikev1.c:1341:isakmp_ph1begin_r(): IKE phase-1 request
for gateway VPN_Peer_ASA is rejected: main mode is not allowed by
configuration.
In both cases the receiver promptly discards the proposals. In these scenarios the initi-
ator will not be able to figure out why the negotiations fail.
NAT during transit introduces more design changes to the IPsec VPN configuration
than NAT-T alone can fix. It will also require changes to peer IP address and Local and
Remote Identifiers depending on which kind of address translation takes place. Once
NAT is confirmed to exist in transit, we are concerned about whether it is a static or
dynamic NAT. Any peer behind a dynamic NAT should only take on the role of an initia-
tor, since the other peer would not be able to reach the peer behind dynamic NAT
when the NAT entry is not in place. [13, 6–7].
While any NAT/PAT device in transit is known to modify the IP addresses and/or
TCP/UDP ports, yet another consideration is that IPsec traffic may end up dropped
altogether. The issue lies in ESP header not specifying port numbers. Some NAT/PAT
devices discard ESP encapsulated traffic on sight, since address translation cannot be
performed. [13, 7].
In this study we will enforce NAT on the third-party router such that ASA will be inside
NAT. The router configuration is as shown in Listing 23.
51
interface GigabitEthernet0/0
description LINK_to_PA200
ip address 172.16.1.2 255.255.255.252
ip nat outside
!
interface GigabitEthernet0/1
description LINK_to_ASA5505
ip address 172.16.2.2 255.255.255.252
ip nat inside
!
interface Loopback0
ip address 172.16.3.1 255.255.255.0
!
ip nat inside source static 172.16.2.1 172.16.3.2
With such a static NAT the NAT-designated inside local address 172.16.2.1 will be
known as the global address 172.16.3.2 behind the NAT-designated outside interface
and vice versa. The single NAT rule will create bi-directional address translations.
While addresses solely used for NAT, such as the 172.16.3.0/24 here, are not required
to be of a network connected to the NAT/PAT device in order to be available, Loop-
back0 was configured to clearly show where this network exists in the topology.
Configuration changes on PA are then necessary, since ASA will no longer be reacha-
ble via the previously known peer IP address 172.16.2.1.
ASA’s identity will remain as 172.16.2.1, but the packets destined to the peer will be
sent to 172.16.3.2 instead. Since ASA is inside NAT, it will still reach PA at the address
172.16.1.1.
NAT-T has already been enabled on PA-200 during the configuration stage. This soft-
ware version of ASA 5505 also seems to construct NAT-T payloads without explicitly
needing to configure the feature. Without further ado, let us look at how a successful
IKEv1 negotiation with NAT-T looks like, as shown in Figures 49 and 50.
Figure 50. NAT-T with ASA inside NAT; ASA 5505 as receiver
Once NAT Discovery has been confirmed during MM messages 3–4, IKEv1 starts us-
ing UDP port 4500 instead of 500. UDP encapsulation with source and destination port
4500 continues during IKE SA negotiations, IPsec SA negotiations and then the ESP
encapsulated data exchanges, which it was originally setup for. We may also verify the
address translations on the NAT Router as shown in Listing 24.
53
In another scenario ASA was placed behind dynamic PAT instead of static NAT. The
purpose of this was to see the PAT Router drop the ESP packets devoid of port num-
bers. However, this caused slightly unexpected results as after the successful negotia-
tions ESP traffic was not obstructed at all. The NAT translations appeared according to
the following Listing 25.
The hexadecimal value ‘0x7E34A2D0’ is the SPI for ASA’s inbound IPsec SA and PA’s
outbound IPsec SA. The value ‘2117378768’ on the other hand is the decimal value
matching that hexadecimal value. Likewise the hexadecimal value ‘0xDB335CEE’ is
the SPI for PA’s inbound IPsec SA and ASA’s outbound IPsec SA, and ‘3677576430’
its equivalent decimal number. In other words, due to a feature in Cisco IOS (Version
15.6 in this case), the ESP packets can be subjected to not only NAT but PAT as well.
In the final scenario PA’s IKE Gateway configurations were reverted to previously con-
figured without local identifiers and the address 172.16.2.1 as ASA’s peer address.
ASA was placed behind dynamic PAT instead of static NAT. In such a situation IKEv1
initiations from PA passed Router without translations, but the return traffic got trans-
lated as sourced by Router’s PA-facing address. PA views this as follows:
ikev1.c:456:ikev1_main(): malformed cookie received or the spi ex-
pired.
54
Since the peer IP address and cookie, equivalent to an SPI, are used to identify an IKE
SA, with the peer IP address changed, the message cannot be connected to any exist-
ing SA negotiations. PA will as usual continue offering its proposal to 172.16.2.1 until
timeout. In case ASA initiates the negotiations, PA will know straight away to discard
the packets from this unknown peer address (Router exercising PAT) as seen from the
CLI output:
[PROTO_ERR]: ikev1.c:1317:isakmp_ph1begin_r(): Couldn't find
configuration for IKE phase-1 request for peer IP 172.16.1.2[500].
This section will compile the relevant ISAKMP Notification messages that were discov-
ered during the troubleshooting section in this study. Since the errors and warnings
during debugging ASA and PA were varying and changed ever-so-slightly according to
each scenario, it seems more productive to gather the standardized notification mes-
sages that described the mismatch situations for future reference. Table 2 lists and
explains the notification message types appearing in the Notification Payload of
ISAKMP (IKE) headers.
Since these are messages specified in the protocol standards they are used by both
ASA and PA. However, during the testing it became obvious that PA does not support
the ‘INVALID-COOKIE’, ‘PAYLOAD-MALFORMED’ or ‘INVALID-ID-INFORMATION’
notification types in the scenarios covered in this study. Thus, we may conclude that
ASA implements a wider range of ISAKMP notification messages, thus making trou-
bleshooting easier for the peer.
IPsec, being by design a modular protocol framework, was easily implemented on and
supported by both PA-200 and ASA 5505. The terminology differed slightly between
the vendors such as ‘crypto map’ or transform-set’. The appendices will serve as refer-
ences for the differences between CLI commands, but going through them is outside of
this study.
Among the configuration steps, the most obvious variation between the two was ASA’s
requirement for a policy-based VPN arrangement. Another noteworthy difference was
support for DH groups; this software version of IOS for ASA would not support beyond
group 5, while PA could go for the more secure group 14. The focus on IKEv1 over
IKEv2 was also solidified by the fact that this PAN-OS version did not have IKEv2 sup-
port yet implemented. Thus, in this study it is no overstatement to consider software
versions playing at least as big a role as the vendor difference.
With regard to troubleshooting means, both devices have features for packet capture,
system logging, CLI debugging and status observations. ASA’s debugging capabilities
appear cleaner and less cryptic compared to PA which tends to leave the majority of
the message contents in hexadecimal form, though PA provides more insight on the
operations taking place in the background such as hashing. Then again, Cisco and
56
Cisco’s ASA have more documentation available than Palo Alto, so this study may not
have covered the most optimal troubleshooting tools – especially for PA-200.
8 Conclusion
The goal of this thesis was to configure and systematically troubleshoot an IPsec Site-
to-Site VPN. The results were to be used to aid future troubleshooting and to compare
the devices’ VPN troubleshooting capabilities.
The initial device setup for the practical part of the study went without notable issues.
The Configuration stage was based on various other studies which came with a les-
sons learned input. As a result the configuring was complete as soon as the policy-
based VPN constraint on ASA’s end was addressed.
As for lessons learned: with regard to information gathering, regardless of the vendor,
the receiver will have a clearer understanding of the underlying issue. The initiator will
hopefully have means to receive and decipher any notification messages sent by the
receiver to understand why the negotiations are not progressing as they should. In this
study, due to ASA 5505’s wider support for ISAKMP notifications, PA-200 had access
to considerably more information.
57
References
2 Morgan B., Lovering N. CCNP ISCW Official Exam Certification Guide. USA, Cis-
co Press; 2007.
3 Frankel S., Krishnan S. IP Security (IPsec) and Internet Key Exchange (IKE)
Document Roadmap [online]. Internet Engineering Task Force (IETF); 2011.
URL: https://tools.ietf.org/html/rfc6071.
Accessed: 17 March 2017.
5 Krawczyk H., Bellare M., Canetti R. HMAC: Keyed-Hashing for Message Authen-
tication [online]. Network Working Group; 1997.
URL: https://tools.ietf.org/html/rfc2104
Accessed: 20 March 2017.
6 Configuring Security for VPNs with IPsec [online]. Cisco Systems; 2012.
URL: http://www.cisco.com/en/US/docs/ios-
xml/ios/sec_conn_vpnips/configuration/15-2mt/sec-cfg-vpn-IPsec.html
Accessed: 20 March 2017.
9 Harkins D., Carrel D. The Internet Key Exchange (IKE) [online]. Network Working
Group; 1998.
URL: https://tools.ietf.org/html/rfc2409
Accessed: 22 March 2017.
10 Kaufman C., Hoffman P., Nir Y., Eronen P., Kivinen T. Internet Key Exchange
Protocol Version 2 (IKEv2) [online]. Internet Engineering Task Force; 2014.
URL: https://tools.ietf.org/html/rfc7296
Accessed: 25 March 2017.
11 Palo Alto Networks Enterprise Firewall PA-200 [online]. Virtual Graffiti Inc.; 2014.
URL: http://www.paloguard.com/Firewall-PA-200.asp
Accessed: 28 March 2017.
interface Vlan10
nameif trust
security-level 100
ip address 10.0.2.254 255.255.255.0
!
interface Vlan20
nameif untrust
security-level 0
ip address 172.16.2.1 255.255.255.252
!
ftp mode passive
clock timezone GMT 0
object network 172.16.2.2
host 172.16.2.2
description Default Gateway
object network 10.0.1.0_24
subnet 10.0.1.0 255.255.255.0
description PA_LAN
object network 10.0.2.0_24
subnet 10.0.2.0 255.255.255.0
description ASA_LAN
access-list PROXY-ID-ACL extended permit ip object 10.0.2.0_24 object 10.0.1.0_24
access-list FALSE-ACL extended permit ip any any
pager lines 100
logging enable
logging asdm informational
mtu trust 1500
mtu untrust 1500
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
nat (trust,untrust) source static 10.0.2.0_24 10.0.2.0_24 destination static 10.0.1.0_24
10.0.1.0_24 no-proxy-arp route-lookup
!
nat (trust,untrust) after-auto source dynamic any interface
route untrust 0.0.0.0 0.0.0.0 172.16.2.2 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
http server enable
http 10.0.2.1 255.255.255.255 trust
no snmp-server location
no snmp-server contact
crypto ipsec ikev1 transform-set ESP-AES1-SHA esp-aes esp-sha-hmac
crypto ipsec security-association pmtu-aging infinite
Appendix 1
3 (4)
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
tunnel-group 172.16.1.1 type ipsec-l2l
tunnel-group 172.16.1.1 ipsec-attributes
ikev1 pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect ip-options
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
Appendix 1
4 (4)
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
call-home
profile CiscoTAC-1
no active
destination address http
https://tools.cisco.com/its/service/oddce/services/DDCEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:b6c701a6071a2091643304f06c8e6fe6
: end.
Appendix 2
1 (5)