Diagnostic Commands and Tools
Troubleshooting IPsec VPN on the VPN 3000 Concentrator is very handy due to its useful debug and monitoring tools. This section discusses both of these tools in greater detail, and this discussion forms the foundation for the rest of the chapter.
Debug Tool
Event log is the debug tool in the VPN 3000 Concentrator, which is used to capture and view relevant debug messages for IPsec tunnel. Various event classes correspond to Internet Key Exchange (IKE), IPsec negotiation events, extended user authentication (x-auth) and Mode configuration (mode-config) information for VPN tunnel. If you are not yet sure what to look for, you can set all events classes to a specific severity level by going to Configuration > System > Events > General, which is good for a general view of the problem. Use syslog for bulk logging when you have a busy system with higher severities configured. Console logging is the most CPU-intensive, so be careful when turning on higher-level logging for all classes. Remember to turn logging back to defaults when you are finished collecting debug information.
Once you have some idea of the problem you are experiencing with the General events, enable more specific logging by using classes and going into Configuration > System > Events > Classes. Classes can be used to disable certain messages and for enabling a specific subset of event messages. Some of the most commonly used event classes are shown in Figure 8-1.
Figure 8-1. Viewing the Debug in the Event Filter Window
Configuring event classes and turning them on is a fairly simple process, and is the first step required to get the debug information. Once you log in to VPN concentrator using the browser, follow these steps to configure event classes:
Step 1. Browse to Configuration>System>Events Classes (see Figure 8-1).
Step 2. Select the Class Name you want to modify or click the Add button to add a new class.
Step 3. If you are adding a new class, in the new configuration window (see Figure 8-2), select Event Class from the Class Name drop-down.
Figure 8-2. Configuration of An Event Class Cert
[View full size image]
Step 4. Check the Enable checkbox.
Step 5. Raise the severity level for Events to Log from 1-5 (Default) to 1-9, for Debug (consult with Table 8-1 for more details on severity level).
Table 8-1. Event Severity Level Explanation Event Levels
Explanation
1 = FAULT
The highest priority severity. Indicates a CRASH or non-recoverable error.
23 WARNING
Indicates a problem that may require user action. Level 2 indicates a pending CRASH or severe problem. Level 3 indicates a potentially serious problem.
46 INFORMATIONAL
Level 4 provides the lowest level of information. More information is provided by 5 and 6.
79 DEBUG
Level 7 provides the lowest level of debugging information. More information is provided by 8 and 9.
10
High-Level Header Decode.
11
Low-Level Header Decode.
12
Hex Dump of Header.
13
Hex Dump of Packet.
Step 6. Follow Steps 2 to 5 for all necessary classes.
Step 7. Be sure to click the Save Needed button.
Figure 8-2 shows the configuration of the event class cert with Events to Log set to 9 so that it will show the debug information of a certificate-related message.
As you can see, you can control the number of messages you receive for a specific class based on the severity level, so it's important that you understand the meaning and level of detail you will receive based on the severity level. You will typically find yourself using severity level 1-9, as that is most appropriate for debugging. If you run into problems where packet level details are required by Cisco Developer, you may need to run it at severity level 1-13, because with a severity level of 10-13 you will see packet-level detail messages, which are usually analyzed by developers. Table 8-1 defines the various severity levels available for different classes.
Note
Raise the severity of classes to level 9 during troubleshooting and set it back to the default of (5) or uncheck the Enable button of classes when you are finished with troubleshooting. Level 10-13 is most useful with Authdecode and RADIUS.
Once you configure the event classes, the next and final step to view the log is to go to the Monitoring tab. See the following section, Monitoring Tool, for more details on how to view the event log.
Monitoring Tool
The Monitoring Tool produces results that are equivalent to the output of different show commands for an IPsec session on the router or PIX firewall. Additionally, this tool presents the debug level output for the classes configured in the previous section, and statistics pertaining to the VPN Concentrator itself. The Monitoring Tool produces statistics, which display, on various activities that are going through or to the Concentrator. These statistics are invaluable first-hand information to isolate the problem fairly quickly.
The most important information under the Monitoring section is the Event Log, which, as mentioned before, is equivalent to debug output on the router or PIX firewall. You can view the events for the event classes configured in the "Debug Tool" section in two ways: Filterable Event Log or Live Event Log. For the Live Event Log, you merely click on Live Event Log and messages will be displayed in real time. The Live Event Log window scrolls very quickly with huge amounts of events being generated by the Concentratorevents that pertain to the VPN tunnel build-up process. You may, therefore, find this Live Event Log useful only in a few circumstances, and only if you are very familiar with the packet flow of VPN tunnel build-up process. Quite often you may want to use the Filterable Event Log option to capture the debug for the following reasons:
With Filterable Event Log, you can save the log into a text file so that you can analyze the log offline, search for a specific message of interest, or send it to Cisco support for in-depth analysis.
You can apply a filter when getting the log, based on event class, client IP, and or based on Group Name, in the case of Remote Access Client. This eliminates many messages that you may not be interested in seeing and reduces the time required to analyze the log to a greater extent.
Figure 8-3 shows how to use Filterable Event Log to display the debug output for IKE-related messages of a VPN tunnel build-up process.
Figure 8-3. Viewing Debug Output for IKE in Event Log Window
[View full size image]
Capturing the debug for a failure of tunnel build-up process will not help unless you know how to properly read and interpret the log message that is being captured. Figure 8-4 is an attempt to show you how to read one of the lines taken from the event log shown in Figure 8-3 (the previous figure).
Figure 8-4. Event Log: Message Header Explained
[View full size image]
Whether you are troubleshooting LAN-to-LAN or Remote Access VPN, always analyze the log on both sides of the tunnel. In the case of LAN-to-LAN, you need to analyze the log messages of the other side (it can be another VPN Concentrator, IOS router, PIX firewall, and so on). In the case of Remote Access VPN, you need to get and analyze the VPN client logs (see the "VPN Client Log" section for details on this). Under some rare circumstances, you may be able to isolate the problem merely by looking at the log from one side of the IPsec tunnel. However, to isolate the issue efficiently and in a timely manner, analyzing the log on both sides of the tunnel end points is a must. There are numerous problems, which you might not be able to analyze just by analyzing one side of the log.
As mentioned before in this section, the Monitoring Tool produces some very valuable statistics in addition to the debug level log for different classes. Following is a list of some very important statistics available under the Monitoring tab that help in isolating the connectivity issues of the IPsec tunnel and in finding out statistics on the health of the VPN concentrator itself:
The Monitoring > Routing Table provides you all the routing entries built up in the routing table of the concentrator. These are the first statistics you need to examine if you have problems building up the tunnel or sending traffic once the tunnel is up.
Just as with debug ip packet detail on the router, you can turn on the event log with class IPDBG to see if the packet is hitting the concentrator. This is especially important when you have a routing problem in your LAN and need to see if the tunnel is not getting built up because the packet is not hitting the concentrator. Alternatively, for this you can rely on various stats under Monitoring > Statistics > MIB-2 Stats. For example, if interface stats in increases over the time of test, it ensures that packets are reaching the concentrator. The ARP table has the other very important stats for troubleshooting Layer 2 issues in your LAN before the packet goes through the IPsec tunnel. For example, if you create an IP pool of overlapping addresses with your private LAN, there is a very good chance that remote access client will not be able to send data due to ARP conflict; you can find that information by looking at the ARP table.
The Monitoring > System Status gives information on the overall health of the VPN concentrator. If you ever run into a problem that requires analysis with a VPN performance issue, these are the statistics you need to look at.