All Cisco-Network Study Notes: Troubleshooting Steps

Troubleshooting Steps

Before getting into the details of the different issues that might occur when the VPN client is connecting to the PIX firewall, this section examines a successful debug message pertaining to IPsec. Example 7-50 shows the debug output at level 7 for a successful Remote Access VPN tunnel. Most of the time, you can run these debugs at level 5 and will be able to isolate the problem. The debug at level 7 will be rarely needed; however, in the interest of showing details on the packet flow, debug level 7 is used in Example 7-50.

Example 7-50. Successful Tunnel Build-up for Remote Access VPN

PIX-A#debug crypto isakmp 7
PIX-A#debug crypto ipsec 7
PIX-A#
! Following are the payloads received from the VPN Clients
Jun 08 04:47:51 [IKEv1 DEBUG]: IP = 10.1.1.50, processing SA payload
Jun 08 04:47:51 [IKEv1 DEBUG]: IP = 10.1.1.50, processing ke payload
Jun 08 04:47:51 [IKEv1 DEBUG]: IP = 10.1.1.50, processing ISA_KE
! Removed other payloads received from the VPN Clients
Jun 08 04:47:51 [IKEv1 DEBUG]: IP = 10.1.1.50, Received Cisco Unity client VID
! Following line shows the Remote Access VPN client request found the VPN group
Jun 08 04:47:51 [IKEv1]: IP = 10.1.1.50, Connection landed on tunnel_group mygroup
! Following line indicates staring of IKE SA negotiation is getting processed.
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, processing IKE SA
! Following line indicates policy is acceptable
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, IKE SA Proposal # 1,
 Transform # 10 acceptable Matches global IKE entry # 1
! Following lines show ISA SA is being constructed
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, constructing ISA_SA
 for isakmp
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, constructing ke
 payload
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, constructing nonce
 payload
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, Generating keys for
 Responder...
! Removed other payloads from display
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, constructing xauth
 V6 VID payload
! Removed some other payloads from display
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, Send Altiga/Cisco
 VPN3000/Cisco ASA GW VID
Jun 08 04:47:51 [IKEv1]: IP = 10.1.1.50, IKE DECODE SENDING Message (msgid=0) with
 payloads : HDR + SA (1) + KE (4) + NONCE (10) + ID (5) + HASH (8) + VENDOR (13) +
 VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 36
 9
Jun 08 04:47:51 [IKEv1]: IP = 10.1.1.50, IKE DECODE RECEIVED Message (msgid=0) with
 payloads : HDR + HASH (8) + NOTIFY (11) + VENDOR (13) + VENDOR (13) + NONE (0)
 total length : 116
Jun 08 04:47:51 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, processing hash
! Removed output from displaying
Jun 08 04:47:54 [IKEv1 DEBUG]: process_attr(): Enter!
! Mode-config is processing the reply attributes to the VPN client
Jun 08 04:47:54 [IKEv1 DEBUG]: Processing MODE_CFG Reply attributes.
! Following 4 lines indicate the DNS and WINS are not configured, hence showing "cleared"
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: primary DNS = cleared
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: secondary DNS = cleared
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: primary WINS = cleared
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: secondary WINS = cleared
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: IP Compression = disabled
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IKEGetUserAttributes: Split Tunneling Policy = Disabled
! Following line indicates the X-Authentication is successful.
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, User
 (cisco) authenticated.
! Removed debug output from displaying
Jun 08 04:47:54 [IKEv1 DEBUG]: Processing cfg ACK attributes
Jun 08 04:47:54 [IKEv1]: IP = 10.1.1.50, IKE DECODE RECEIVED Message (msgid=64c2feef)
 with payloads : HDR + HASH (8) + ATTR (14) + NONE (0) total length : 197
Jun 08 04:47:54 [IKEv1 DEBUG]: process_attr(): Enter!
! Following attributes are request items from the VPN Clients. Removed some of the
! attributes request from the debug output.
Jun 08 04:47:54 [IKEv1 DEBUG]: Processing cfg Request attributes
Jun 08 04:47:54 [IKEv1 DEBUG]: MODE_CFG: Received request for IPV4 address!
Jun 08 04:47:54 [IKEv1 DEBUG]: MODE_CFG: Received request for IPV4 net mask!
Jun 08 04:47:54 [IKEv1 DEBUG]: MODE_CFG: Received request for DNS server address!
Jun 08 04:47:54 [IKEv1 DEBUG]: MODE_CFG: Received request for WINS server address!
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing blank hash
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing qm hash
Jun 08 04:47:54 [IKEv1]: IP = 10.1.1.50, IKE DECODE SENDING Message (msgid=64c2feef)
 with payloads : HDR + HASH (8) + ATTR (14) + NONE (0) total length : 159
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Delay Quick Mode processing, Cert/Trans Exch/RM DSID in progress
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Resume Quick Mode processing, Cert/Trans Exch/RM DSID completed
! Following line indicates that Phase I of IPSec Remote Access VPN is completed.
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, PHASE 1
 COMPLETED
! Following line indicates that this connection will use DPD for keep alive method.
Jun 08 04:47:54 [IKEv1]: IP = 10.1.1.50, Keep-alive type for this connection: DPD
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Starting phase 1 rekey timer: 41040000 (ms)
! Removed some debug output from this location
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Processing ID
! Remote Proxy is the VPN client IP address for Phase II which is shown in the line below
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, Received
 remote Proxy Host data in ID Payload: Address 192.168.0.1, Protocol 0, Port 0
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Processing ID
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, Received
 local IP Proxy Subnet data in ID Payload: Address 0.0.0.0, Mask 0.0.0.0, Protocol
 0, Port 0
Jun 08 04:47:54 [IKEv1]: QM IsRekeyed old sa not found by addr
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, IKE
 Remote Peer configured for SA: mydyn
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 processing IPSEC SA
! Following line shows there is a match for transform set between the client and the PIX
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 IPSec SA Proposal # 11, Transform # 1 acceptable Matches global IPsec SA entry # 1
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, IKE:
 requesting SPI!
Jun 08 04:47:54 [IKEv1 DEBUG]: IKE got SPI from key engine: SPI = 0x3dbb59a3
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 oakley constucting quick mode
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing blank hash
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing ISA_SA for ipsec
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Overriding Initiator's IPsec rekeying duration from 2147483 to 28800 seconds
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing ipsec nonce payload
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing proxy ID
! The local and Remote Proxy Identities for the VPN connection are shown in the
!following lines
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Transmitting Proxy Id:
 Remote host: 192.168.0.1  Protocol 0  Port 0
 Local subnet:  0.0.0.0  mask 0.0.0.0 Protocol 0  Port 0
! Timer negotiation for the tunnel
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Sending RESPONDER LIFETIME notification to Initiator
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 constructing qm hash
Jun 08 04:47:54 [IKEv1]: IP = 10.1.1.50, IKE DECODE SENDING Message (msgid=cef83b87)
 with payloads : HDR + HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NOTIFY
 (11) + NONE (0) total length : 176
Jun 08 04:47:54 [IKEv1]: IP = 10.1.1.50, IKE DECODE RECEIVED Message
 (msgid=cef83b87) with payloads : HDR + HASH (8) + NONE (0) total length : 48
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 processing hash
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 loading all IPSEC SAs
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Generating Quick Mode Key!
Jun 08 04:47:54 [IKEv1 DEBUG]: Group = mygroup, Username = cisco, IP = 10.1.1.50,
 Generating Quick Mode Key!
! Both inbound and outbound SAs are evident with the corresponding SPI numbers
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, Security
 negotiation complete for User (cisco) Responder, Inbound SPI = 0x3dbb59a3,
 Outbound SPI = 0xaaad36b4
Jun 08 04:47:54 [IKEv1 DEBUG]: IKE got a KEY_ADD msg for SA: SPI = 0xaaad36b4
Jun 08 04:47:54 [IKEv1 DEBUG]: pitcher: rcv KEY_UPDATE, spi 0x3dbb59a3
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, Starting
 P2 Rekey timer to expire in 27360 seconds
! The VPN Client route is added to the PIX as shown in the following line
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, Adding
 static route for client address: 192.168.0.1
! Following line indicates a successful phase II negotiation
Jun 08 04:47:54 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.1.1.50, PHASE 2
 COMPLETED (msgid=cef83b87)
PIX-A#

The Remote Access VPN Client problems can be categorized as follows:

Tunnel is not established
Tunnel is established completely but is unable to pass data

The next sections present a detailed discussion of these topics.

Tunnel Is Not Established

After you become familiar with the debug output on PIX Firewall (see the preceding section), and the log output from VPN Client (see Chapter 8, "Troubleshooting IPsec VPNs on VPN 3000 Series Concentrators") for a successful Remote Access VPN client connection, work through the following steps if your VPN tunnel is not coming up. You can also refer to the section in this chapter entitled "LAN-to-LAN Troubleshooting" for additional details on the following steps:

Step 1.

Enable ISAKMP on the interface.

Unlike the router, ISAKMP is not enabled by default on the PIX. Use the isakmp enable <interface> command to enable it on an interface of the PIX firewall.

Step 2.

Look for an attributes mismatch between VPN Client and PIX.

VPN clients propose only DH groups 2 and 5. If you configure DH groups other than types 2 and 5, the VPN client cannot build up a tunnel with the PIX Firewall, and you will receive a message similar to that shown in Example 7-51.

Example 7-51. debug Message when DH group 7 Is Configured

PIX-A(config)# debug crypto ipsec 5
PIX-A(config)# debug crypto isakmp 5
! Based on the groupname client used is used to find the corresponding tunnel.
PIX-A(config)# Jun 08 07:46:23 [IKEv1]: IP = 10.1.1.50, Connection landed on
 tunnel_group mygroup
! Following line indicates that the Phase I proposal is not acceptable. DH group
! is one of the attributes of Phase I. PIX is configured with DH 7, but the
! client supports only DH 2 or DH 5
Jun 08 07:46:23 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, All SA proposals
 found unacceptable
Jun 08 07:46:23 [IKEv1]: IP = 10.1.1.50, All IKE SA proposals found unacceptable!
Jun 08 07:46:23 [IKEv1 DEBUG]: Group = mygroup, IP = 10.1.1.50, IKE AM Responder FSM
 error history (struct &0x194c0f8) , : AM_DONE, EV_ERROR-->
 AM_BLD_MSG2, EV_PROCESS_SA-->AM_BLD_MSG2, EV_GROUP_LOOKUP-->AM_BLD_MSG2,
 EV_PROCESS_MSG
PIX-A(config)#

Step 3.

Look for group authentication failures.

IKE authentication can fail due to mismatches in group names or passwords. If you are using certificates and if rsa-sig is used as an IKE authentication method, configure isakmp identity hostname; otherwise, IKE authentication will fail. Example 7-52 shows the debug message indicating group password failure.

Example 7-52. debug Output of Wrong Group Name or Password for VPN Client Connection

PIX-A(config)# debug crypto isakmp 5
PIX-A(config)# debug crypto ipsec 5
PIX-A(config)# Jun 08 08:05:29 [IKEv1]: IP = 10.25.35.227, Connection landed on
 tunnel_group mygroup
Jun 08 08:05:29 [IKEv1 DEBUG]: Group = mygroup, IP = 10.25.35.227, IKE SA Proposal
 # 1, Transform # 10 acceptable Matches global IKE entry # 1
Jun 08 08:05:30 [IKEv1]: Group = mygroup, IP = 10.25.35.227, Received an un-encrypted
 INVALID_HASH_INFO notify message, dropping
! Following line clearly indicates the it's a pre-shared-key mismatch problem.
! On the VPN client, modify the profile and define the proper group name.
Jun 08 08:05:30 [IKEv1]: Group = mygroup, IP = 10.25.35.227, Error, peer has indicated
 that something is wrong with our message. This could indicate a pre-shared key
 mismatch.
Jun 08 08:05:30 [IKEv1]: Group = mygroup, IP = 10.25.35.227, Information Exchange
 processing failed
PIX-A(config)#

Step 4.

Look for user authentication (Extended Authentication or X-Auth) failures.

In addition to group authentication, user authentication is required for a successful user authentication Remote Access VPN connection. X-Auth occurs between Phase I And Phase II of the IPsec tunnel establishment process. Note that a LAN-to-LAN tunnel does not require the X-Authentication. X-Auth can be performed either locally on the PIX Firewall or by an AAA server. So, if the authentication fails with the AAA Server, you may want to try authenticating the tunnel with the local user database. Example 7-53 shows the debug message that appears on the PIX firewall when the user authentication fails.

Example 7-53. debug Output of Remote Access VPN with Either Bad Username or Password

PIX-A(config)# debug crypto isakmp 5
PIX-A(config)# debug crypto ipsec 5
PIX-A(config)# Jun 08 08:14:01 [IKEv1]: IP = 10.25.35.227, Connection landed on
 tunnel_group mygroup
Jun 08 08:14:01 [IKEv1 DEBUG]: Group = mygroup, IP = 10.25.35.227, IKE SA Proposal
 # 1, Transform # 10 acceptable Matches global IKE entry # 1
! Following line clearly indicates either a bad username or password.
Jun 08 08:14:15 [IKEv1]: Group = mygroup, Username = cisco, IP = 10.25.35.227, Remote
 peer has failed user authentication - check configured username and password
PIX-A(config)#

Step 5.

Enable mode config pushing.

Turning on authorization is a must for pushing (mode config pushing) the WINS or DNS information to the client. So, if PIX is unable to push the attributes, be sure that authorization is turned on.

Tunnel Is Established Completely But Unable to Pass Data

If the Remote Access VPN tunnel is established, but you cannot send any traffic across the tunnel, work through the following steps to correct the problem:


Step 1.	Check for valid IP addresses. If your IPsec tunnel is up, but unable to send any traffic across the tunnel, check to see if you are assigned a valid IP address. If you are not getting an IP address from the PIX, check to see how the PIX is configured to allocate an IP address. If the IP pool is defined locally on the PIX firewall, check to see if you are running short of IP addresses.
Step 2.	Be sure IPsec traffic is allowed. After IPsec tunnel traffic is decrypted, it has to go through the ACL check on the interface where the crypto map is (for example, outside interface) applied. To allow this traffic, you must either configure sysopt connection permit-ipsec to bypass the access-list, or you need to create an access-list on the interface as inbound to allow the IPsec traffic.
Step 3.	Bypass the NAT if necessary. For the Remote Access VPN client host to access the private network behind the PIX seamlessly, the NAT needs to be bypassed on the PIX firewall. If nat-control is turned on, you must bypass the NAT for the VPN traffic with the NAT 0 access list command. If nat-control is not turned on but the nat is configured on the inside interface that includes network address for the VPN traffic, you need to bypass the NAT in the same way as when the nat-control is turned on.
Step 4.	Enable NAT-T or IPsec over TCP. If your VPN client is behind a NAT/PAT device, be sure to turn on NAT-T or IPsec over TCP on both the PIX firewall and the VPN Client.
Step 5.	Be sure the required ports are open. If there is a firewall between the VPN client and the PIX firewall, be sure UDP/500, UDP/4500, TCP/10000, and ESP (IP/50) are open. If all other ports are open, but ESP is blocked, then even though the tunnel will be established, it will not be able to pass any traffic across the tunnel.