Troubleshooting Steps
Before delving into the details of how to troubleshoot connectivity problems, it is important to understand the meaning of different connection flags to quickly identify the problem with connections. You can use combinations of show, debug, and the syslog message that are discussed in this section, to troubleshoot connectivity problems.
Figure 3-5 shows the connection flags for both inbound and outbound connections across the PIX firewall for TCP connections.
Figure 3-5. PIX Firewall Connection Flags Information for Both Inbound and Outbound Connections
The following numbered steps show the sequence of the packets and the connections flags for the outbound connection (see Figure 3-5) through the PIX firewall:
| 1. | When PIX receives an initial SYN packet from the inside network, the SYN packet is permitted by the access-list on the ingress interface, a translation (xlate) is built up (if NAT is configured), and the connection is created with the flag saA, which can be verified with the show connection command. |
| 2. | The outside device responds to the SYN packet with a SYN+ACK. The connection flags are updated to reflect this, and now show A. Again, this information can be verified with show connection command output. |
| 3. | The inside device responds to the SYN+ACK with an ACK, and this completes the TCP 3-way handshake. The connection is now considered up (the U flag will be shown with the show connection command). |
| 4. | When the outside device sends the first data packet, the connection is updated, and an I is added to the connection flags to indicate that the PIX received inbound data on that connection. |
| 5. | Finally, after the inside device sends a data packet, the connection is updated to include the O flag. |
Understanding the connection flags for the teardown of the connection is very important. Following is the sequence of packets and the corresponding connection flags for the outbound connection:
| 1. | When PIX receives a FIN packet from the inside, PIX updates the connection flags by adding an f. |
| 2. | The outside device immediately responds to the FIN packet with a FIN+ACK. The connection flags are updated to reflect this with UfFR flags. |
| 3. | The inside device responds to the FIN+ACK with a final ACK, and the PIX tears down the connection. Thus, there are no more connection flags, because the connection no longer exists. |
The sequence of events that takes place for the inbound connection is similar to that of the outbound connection on the PIX firewall but it has different flags. Details are not discussed here, but refer to Figure 3-5 for a summary of connection flags for the inbound connection.
If you have connectivity problems through the PIX firewall, you need to analyze the connection syslog ID, which is in the following form:
PIX-6-302014: Teardown TCP connection number for interface_name:real_address/
real_port to interface_name:real_address/real_port duration time bytes number
[reason] [(user)]
The reason field is important to understand. Table 3-9 outlines the reasons for connection terminations.
| Connection Flags | Meaning |
|---|---|
| Reset-I | Reset was from Inside |
| Reset-O | Reset was from Outside |
| TCP FINs | Normal Close Down Sequence |
| FIN Timeout | Force Termination After 15 Seconds |
| SYN Timeout | Force Termination After 2 Minutes |
| Xlate[*] Clear | Command Line Removal |
| Deny | Terminate by Application Inspection |
| SYN Control | Back Channel Initiation from Wrong Side |
| Uauth Deny | Deny by URL Filter |
| Unknown | Catch All Error |
[*] An xlate is created dynamically when a packet from a high-security-level interface initiates an outbound connection to a lower-security-level interface.
If you cannot make connection across the PIX firewall, the problem might be with the access-list, NAT, routing, and so on. Work through the following steps to correct the connectivity problem:
| 1. | Is the initial packet reaching the PIX firewall? This is the first and most important step to troubleshoot the connectivity problem across the firewall. You must be sure the PIX is receiving the initial packet on its interface. If ACL is configured on the interface, you can find out if the packet is reaching to the PIX interface by executing the show access-list command. You should see the hit counts incrementing on the show access-list command output. Another way you can verify if the packets are reaching is by executing the show interface command as shown in Example 3-16. You should see that the Input counters are incrementing. The software input queue is an indicator of traffic load, and "No buffers" indicates packet drops, typically due to bursty traffic. Example 3-16. Interface Statistics of Inside Interface
| |
| 2. | Is the initial packet allowed by the ACL? For outbound connection across the PIX firewall, you do not have to configure an ACL to allow the traffic. By default, all traffic is allowed. However, for the inbound connection, you must configure an ACL to allow the traffic. Execute the show access-list command and be sure that you see that the hit count on ingress interface ACL increases over time. Remember that creating an ACL is not sufficient; you also must apply the ACL on the interface in the proper direction. Execute show running-config access-group command to verify if indeed the access-list is applied to the correct interface. | |
| 3. | Is translation being built up? If Steps 12 are verified, the next step is to verify if the translation is being built up for the connection (this is mandatory when the nat-control is turned on or when nat-control is turned off but NAT is configured to perform the translation). If the translation is not being built up, first be sure the nat/global is configured correctly. For example, to verify that the nat/global is configured for the outbound connection, execute the following commands: PIX# show run nat-control You need to ensure that the NAT identification corresponds to the global identification. If some hosts are able to connect through the PIX, but some are not, the problem might be that you are running short of addresses from the global pool. To be on the safe side, it is recommended to configure a PAT address along with the NAT pool (when the nat-control is turned on). To verify that a host is building up translation, use the following command: For example, to find out if inside host 10.1.1.70 is building up the translation, execute the following command: PIX# show xlate local 10.1.1.70 debug For translation issues, the following syslog message will be generated: PIX-3-305005: No translation group found for tcp src inside:10.1.1.70/11039 dst outside To find out the translation and connection entry created for a specific host (10.1.1.70), execute the following command: PIX# show local 10.1.1.70 If the NAT is not working as expected, run the following debug command to get the details of the NAT translation process: PIX# debug pix process | |
| 4. | Is the connection being built up? If the translation is built up, next make sure that the connection is built up. This can be verified with the show conn or show conn detail commands to find the connection flag. The meaning of different connection flags is in Figure 3-5. You also need to analyze the syslog to see why the connection is being reset. Refer to Table 3-9 to find out the meaning of different connection resets. For example, if you see connection reset reason as Reset-I, the inside host is resetting the connection. | |
| 5. | Is routing set up correctly on PIX? If you see the translation being built up on the PIX but it does not know where to forward the traffic, you must ensure that you have the route set up correctly. For outbound traffic towards the Internet, you must ensure that the default gateway is set up on the outside interface. If the default route is missing to reach to 198.133.219.25 from the inside host 10.1.1.1, you will get the following syslog message: PIX-6-110001: No route to 198.133.219.25 from 10.1.1.70 Use the following commands to find out if the route indeed exists: PIX(config)# show route | include 198.133.219.25 In Figure 3-4, if you want the users from network 10.1.2.0/24 to be able to access 198.133.219.25, in addition to a default gateway configured on the outside interface, you must configure a static route pointing to the inside router as follows: PIX(config)# route inside 10.1.2.0 255.255.255.0 10.1.1.2 | |
| 6. | Is the return traffic coming back to the PIX? If the global pool or the translated IP address of the initial packet is not part of the same network as the outside interface network of the PIX, the next-hop router will not get the proxy Address Resolution Protocol (ARP) from the PIX. Hence, it is mandatory to create a static route on the next-hop router for the translated network so that the next hop router directs the packets back to the outside interface of the PIX Firewall. For example, in Figure 3-4, if you are using any address on the global pool which is part of network 20.1.1.0/24, you do not need any route defined on the router where the PIX outside interface is connected. This is because, with nat/global, PIX will proxy ARP the MAC address from the global pool to the router that is connected to the outside router. If the global pool is defined with the IP address not part of the outside interface network (20.1.1.0/24), for example, the 30.1.1.0/24 network, you must define a static route on the outside router for network 30.1.1.0/24 pointing to the outside interface (20.1.1.1) of the PIX firewall. | |
| 7. | Is the router or the PIX ARPing correctly? If everything else is set up correctly, and you are still unable to make connection across the PIX firewall, execute the show arp command on the PIX to see if the next-hop address has an ARP entry. For example, if you have a default gateway setup pointing to 20.1.1.10, on the PIX you should see an ARP entry for the 20.1.1.10 address. If the ARP entry is built as shown by the show arp command, execute the debug arp command and then execute the clear arp command to see if the ARP is having issues. ARP can be a problem for the next-hop router (especially on the outside router). When you make changes to the global pool, static and so on, it is recommended to clear the ARP on the outside router. | |
| 8. | Packets are dropped by Accelerated Security Path (ASP). PIX Firewall contains a system for packet inspection, called the Accelerated Security Path (ASP). All packets have to go through the ASP logic, and if any packet violates the logic, that packet will be dropped by the PIX firewall. Sometimes what the ASP considers to be illegitimate packets may be legitimate for you. So it's important to find out what packets the ASP is dropping. This section presents some techniques that you can use to help troubleshoot the problem. Execute the show asp drop command to find out counter values for the reasons for packet drops by the ASP. Issue the command clear asp drop then show asp drop to get a baseline of the drops so far, then send the traffic that is not making it through the firewall, and then issue show asp drop again, and check which counters are incrementing. Following is a sample output of the show asp drop command. PIX# show asp drop To find the details on what packets are dropped by the ASP, turn on capture with the following command: PIX# capture capture_name type asp-drop drop-code all packet-length 1518 Drop-code can be all or a specific one in the preceding command. However, because you typically don't know why the PIX is dropping the packet, you cannot specify a specific drop-code; therefore you will usually use all as drop-code. You might need to run interface capture in addition to ASP drop capture to troubleshoot ASP drop issues. Additionally, instead of sending the debug output to the console, redirect it to the logging buffer with the following commands: PIX(config)# logging list mylist message 711001 |
[After going through the preceding steps, you should be able to resolve most of the connectivity problems. However, if you run into a problem with the PIX Firewall version, you might need to engage the Technical Assistance Center (TAC) for further analysis.