Cisco Network Security Troubleshooting Handbook

part 2 chapter 4e gectik
gördügünüz gibi calismalara devam ediyoruz hersey sizin icin..

Protecting Network Resources

Protecting Network Resources

To protect the network resources behind the PIX firewall, you must undertake the following actions:

• Anti-spoofing configuration You can accomplish anti-spoofing configuration with the command ip verify reverse-path on all interfaces of the PIX firewall. This means that the firewall rejects any packet that has a source address that is not expected to be on that interface. If the PIX is an Internet firewall, it should reject all packets coming from the Internet that claim to be from a private network. Similarly, it should reject all packets coming from the private network with source addresses that are not part of the private network, as anti-spoofing is not optional in either direction.

• Prevention from DOS attack Set embryonic and maximum connection counts on static and nat statements to prevent network resources from DoS attack.

Best Practices

Best Practices

This section looks into some of the best practices for PIX Firewall when deployed to the production network, including:

• Protecting the PIX firewall itself Physical and logical security

• Protecting the network resources Anti-spoofing configuration, and denial of service (DOS) attacks

Protecting the PIX Firewall Itself

The PIX firewall protecting your network should be secured by itself. Following are some of the recommendations for securing the firewall that is protecting your network from the hostile environment:

• Physical security As the firewall is one of the most critical network devices in your network, proper action should be taken to protect it from a hacker's physical access. Having console access can help the hackers to change the policy, which can cause severe harm to the network. It is recommended that you turn off the password recovery ability on the PIX with Version 7.0.

• Link bandwidth policing Under no circumstances should you allow more traffic to flow to a PIX firewall interface than it can handle. This can happen when your network or ISP is affected with a virus such as code red, and so on. So, proper QoS needs to be configured to police the bandwidth on both upstream and downstream routers. Though PIX Firewall has a good mechanism for protecting against DoS attacks, the mechanism is for protecting internal network resources, not for the PIX from DoS. Therefore, an effective method to implement QoS on upstream and downstream routers is to police bandwidth.

• Secure access for management purposes There are multiple ways to allow access for managing the PIX Firewall. Among them, Telnet is the only one that is not secured. Arguably, the console is not secured either if it is connected via reverse Telnet. So, for remote management, it is best to configure SSH instead of Telnet on the PIX firewall.

• Access control and password Be sure to configure access control for the Console port before providing access for managing the PIX. In a non-AAA environment, use passwd, and enable password commands. If AAA is configured on the PIX, be sure to apply AAA for connections. SSH with AAA is highly recommended for device management.

• Regular monitoring by an IPS device Though PIX has IPS capacity, it is very limited. IPS devices such as the sensor or IDSM blade could be useful for analyzing the exploitation. In the absence of an IPS monitoring device, you might rely on PIX syslog to analyze the traffic, which is not a very effective solution in a large network due to the amount of syslog.

• Syslog monitoring if IPS is absent In the absence of IPS devices, syslog may be used to monitor the traffic coming in or going out of your network. However, while configuring the syslog, you might need to pay extra attention to the level of detail you want to configure to collect syslog. If you leave the debug-level logging on, that might cause slow performance, because of the amount of CPU cycles it uses. Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you do not want to see.

Common Problems and Resolutions

Common Problems and Resolutions

This section looks into some confusions and commonly asked questions regarding PIX firewall in a question-and-answer format.

1	Can a PIX firewall function in both transparent and routed modes?
Answer:	No, it can function either in routed or transparent Mode.
2	Is load balancing possible with a PIX firewall?
Answer:	Yes, with Version 7.0, you can configure asymmetric routing with active/active failover setup. However, the load balancing must be configured on upstream or downstream routers.
3	Is NAT-control turned on by default on PIX firewall?
Answer:	In PIX Firewall Version 7.0, the NAT-control command is turned off by default, which means that unless there is a matching source/destination NAT statement for the packet, NAT will not occur. However, unlike the older version of the code, if there is no match on the source or destination NAT, the packet will be allowed.
4	Can non-IP protocols be routed through the PIX firewall?
Answer:	Yes, other protocols such as Internetwork Packet Exchange [IPX] and AppleTalk can function, but only if you configure transparent firewall, which is new in Version 7.0.
5	Is it possible to configure a transparent firewall in multiple contexts? If so, are there any restrictions?
Answer:	Yes, a transparent firewall can be configured in multiple contexts. In both single and multiple contexts, there can be only one inside and one outside interface, unlike the routed mode, in which more than two interfaces are possible. However, in multiple contexts mode with a transparent firewall, you cannot share the same interface into multiple contexts.
6	Can policing traffic be configured inbound on the PIX firewall?
Answer:	No. Unlike with routers, only outbound policing is possible on PIX Version 7.0.
7	Is ESMTP supported through the PIX firewall?
Answer:	Yes, with Version 7.0, it is possible to send ESMTP traffic across PIX firewall inspected.
8	Is it possible to pass traffic between equal security level interfaces?
Answer:	You can permit communication between interfaces with equal security levels by using the following command: PIX(config)# same-security-traffic permit inter-interface
9	Is it possible route the packets back to the same interface as PIX learns the packet from?
Answer:	Yes, permitting traffic in and out of the same interface is possible with the following command: PIX(config)# same-security-traffic permit intra-interface
10	Can I configure time-based access-list on the PIX firewall?
Answer:	Yes, beginning with PIX Firewall Version 7.0, time-based access-lists are available.

Troubleshooting Steps

Troubleshooting Steps

Before working through some of the common scenarios, it would be helpful for you to examine the syslog messages that are shown on the secondary PIX firewall after you turn on logging on as shown in Example 3-29.

Example 3-29. Syslog on the Secondary PIX

PIX(config)# logging on PIX(config)# logging monitor 7 PIX(config)# 111008: User 'enable_15' executed the 'logging con 7' command. Detected an Active mate. Switching to Standby Switching to Standby.

After the units are synchronized with each other, you can find out the status of a unit on both the primary and secondary with the show failover command. Example 3-30 shows the output of the show failover command on the primary unit.

Example 3-30. Monitoring Failover Status on Primary PIX Firewall

PIX(config)# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 5 seconds Last Failover at: 14:52:29 EST Wed Feb 9 2005 This host: Primary - Active Active time: 14805 (sec) Interface outside (192.168.1.1): Normal Interface inside (10.1.1.1): Normal Interface stateful (1.1.1.1): Normal Other host: Secondary - Standby Active time: 250 (sec) Interface outside (192.168.1.2): Normal Interface inside (10.1.1.2): Normal Interface stateful (1.1.1.2): Normal Stateful Failover Logical Update Statistics Link : stateful Stateful Obj xmit xerr rcv rerr General 34036 0 1054 0 PIX#

Work through the following steps to troubleshoot the failover problem:

Step 1.	Level 1 syslog will give the reasons for a failover. So always check the syslog to determine the root cause. For example, if the switch port failed on the inside interface of Active Firewall, you would see the following message on the Primary (Active) firewall. 411002: Line protocol on Interface inside, changed state to down 105007: (Primary) Link status 'Down' on interface 1 104002: (Primary) Switching to STNDBYinterface check, mate is healthier Syslog from Secondary (Standby) Firewall will report the following message: 104001: (Secondary) Switching to ACTIVEmate want me Active
Step 2.	Execute show interface on both PIX firewalls to make sure they are up.
Step 3.	Test the connectivity by pinging to the Failover interface IP. Be sure to allow the ICMP for the interfaces.
Step 4.	If the primary and secondary are connected in two different switches, be sure that all VLANS are trunked between the switches.
Step 5.	Be sure to turn on dot1Q across the board on the switch

Asymmetrical Routing Support

Asymmetrical Routing Support

For better performance and reliability, you may have the network set up with redundant connections to the same Internet Service Provider (ISP) or two different ISPs as shown in Figure 3-11. This poses a problem for PIX, as the ISP may receive traffic from one PIX but might return traffic back to the other PIX. This is because these two PIX firewall units (or pairs) do not share session information. Therefore, while the return packet is allowed by the PIX that allowed the initial connection, it is denied by the other PIX. This is also how it works in Failover mode when it is deployed with Active/Standby Mode.

Figure 3-11. Asymmetric Routing Support with Active/Active FO Setup

The same problem can occur when running active/active failover. A unit may receive a packet that belongs to its peer. If this happens, the receiving unit will forward the packet back to its peer for processing. Stateful failover must be enabled to support asymmetric routing.

Figure 3-11 shows two units running Active/Active failover, where Unit 1 has context ctx1 active and Unit 2 has context ctx2 active. An inside host initiates a connection through context ctx1 of Unit 1 (solid line). Context ctx1 creates the connection, replicates the connection to Unit 2, and then forwards the packet. If the return packet is routed through context ctx2 of Unit 2, connection information already exists in Unit 2's connection table. But, because context ctx1 is not active in Unit 2, it forwards the packet back to Unit 1 (arrows).

Following are some of the restrictions and cautions you should remember when configuring asymmetric routing in Active/Active Failover mode.

• Remember that multiple context PIX does not support VPN, IPS, and other features.

• Shared interface setup requires NAT.

• Under the asymmetric routing A/A FO, packets are forwarded by Layer 2.

• The supported scenario is for outbound traffic being routed through one unit and inbound traffic being routed to a different unit for a given connection. The scenario where traffic traveling in the same direction for a connection gets routed to different units should not happen if routing is configured properly. The upstream or downstream router should set up the load-balancing policy such that routers are not performing per packet load balancing to the High Availability (HA) cluster.

To configure asymmetric routing support, you need to use asr-group command under the interfaces of contexts. Asymmetric routing support is needed for the outside interface only; you need to have the following commands configured on both ctx1 and ctx2:

PIX/ctx1(config)# interface Ethernet 0.30
PIX/ctx1(config-if)# asr-group 1
PIX/ctx2(config)# interface Ethernet 0.40
PIX/ctx2(config-if)# asr-group 1

Initialization, Configuration Synchronization/Command Replicati

Initialization, Configuration Synchronization/Command Replication

When a unit boots up, the system failover group will contact the active peer to obtain the running configuration. If both units boot up at the same time, the System failover group of the Primary unit will become active and synchronize its configuration to the Secondary unit. After configuration syncing has finished, the state machine of the user failover group will start running to elect the active unit for each group and start the Active/Active failover.

Even though both units could be actively processing user traffic, the command replication is uni-directional. The command will be replicated to the peer only if failover group 1 is in active state. Users will be warned if they try to enter a configuration command from the standby unit. This means that a user context can be in the active state, but the user will need to enter commands from the user context of the standby unit if it is bound to failover group 2.

Configuration Examples

Work through the steps that follow to configure Active/Active Failover:

Step 1.	Verify that both units have exactly the same hardware configuration and proper license.
Step 2.	If a unit is in single context mode, use the command: mode multiple to bring it to multiple security contexts mode.
Step 3.	Configure the basic failover parameters in the system context of the primary unit. Example 3-26 shows the configuration needed for active/active failover. Example 3-26. Active/Active Failover Setup PIX(config)# failover lan unit primary PIX(config)# failover lan interface folink ethernet2 PIX(config)# failover link stfo Ethernet3 PIX(config)# failover interface ip folink 1.1.1.1 255.255.255.0 standby 1.1.1.2 PIX(config)# failover interface ip stfo 2.2.2.1 255.255.255.0 standby 2.2.2.2 ! This command is optional but recommended PIX(config)# failover key cisco123 PIX(config)# failover lan enable PIX(config)#
Step 4.	Use the failover group command to configure a failover group. Example 3-27 shows how to configure two failover groups with the preemption option. Example 3-27. Failover Group Configuration ! Execute failover group command that will take you to the subcommand mode PIX(config)# failover group 1 ! Primary unit has higher priority PIX(config-failover)# primary ! Preempt peer if bootup as Standby PIX(config-failover)# preempt PIX(config-failover)# exit PIX(config)# failover group 2 ! Secondary unit has higher priority PIX(config-failover)# secondary PIX(config-failover)# preempt PIX(config-failover)# exit PIX(config)#
Step 5.	Bind the user contexts to the failover group. Assume there are two contexts, ctx1 and ctx2, in addition to admin. PIX(config)# context ctx1 PIX(config-context)# join-failover-group 1 PIX(config-context)# exit PIX(config)# context ctx2 PIX(config-context)# join-failover-group 2 PIX(config-context)# exit PIX(config)#
Step 6.	Type the failover command to enable active/active failover.
Step 7.	Configure the bootstrap failover configuration on the secondary unit as shown in Example 3-28. Example 3-28. The Initial Configuration on the Standby PIX PIX(config)# failover lan unit secondary PIX(config)# failover lan interface folink ethernet2 PIX(config)# failover link stfo Ethernet3 PIX(config)# failover interface ip folink 1.1.1.1 255.255.255.0 standby 1.1.1.2 PIX(config)# failover interface ip stfo 2.2.2.1 255.255.255.0 standby 2.2.2.2 ! This command is optional but recommended PIX(config)# failover key cisco123 PIX(config)# failover lan enable PIX(config)# failover
Step 8.	After failover is up, on the Primary unit (i.e., Active for failover group 1), issue the write memory command to save the configuration to flash.

Hardware and License Requirements

Hardware and License Requirements

You must have the following minimum requirement for hardware and licensing to configure active/active failover:

• Both units need to have the same hardware configuration.

• Both units must have an unrestricted (UR) License, or the primary unit must have a UR license and the secondary unit must have an active/active failover-only license.

System and User Failover Group

To support active/active failover, PIX 7.0 failover added support for failover groups. Each failover group has its own state machine and can switch over independently. There are two types of failover group: system failover group and user failover group. The concept is similar to the system and user context.

The system group is used internally by the failover process. Its main purpose is to allow the failover process to manage the unit-wide activities under the failover group scheme. These activities include: unit health monitoring, failover command interface health monitoring, running config synchronizing, and so on. There is only one system failover group per unit, and it is created automatically when the user enables the failover. The system context is bound to the system failover group.

User failover groups are used to manage the user contexts under the active/active failover scheme. Figure 3-10 shows a simple example of two user contexts that are bound to two user failover groups.

Note

PIX Firewall Version 7.0 supports a maximum of two (user) failover groups. However, there can be more than two user contexts configured in the system.

The failover-group subcommand under the context command is used to bind a user context to a failover group. The failover group 1 is the default failover group. If a user context is not bound to a failover group through the command interface, it is bound to failover group 1 by default. Failover group 1 must be created first and must be the last group to be removed. If a context is bound to a failover group, the failover group cannot be deleted unless the binding is removed.

Active/Active Model

Active/Active Model

The active/standby model is not the best way to use the resources. Ideally, both of the units in the failover should be used, and act as backups for each other in the event of failure. This is exactly what active/active failover can provide, as introduced in Version 7.0. In this model, both units can actively process firewall traffic while at the same time serving as backups for each other. In PIX Version 7.0, only a two-node setup is possible, in which each device handles 50 percent of the traffic. When one unit fails, the remaining unit takes over and processes 100 percent of the traffic.

Remember that in this model, PIX itself does not perform the load balancing of the traffic. So, it is your responsibility to engineer the network to route 50 percent of the traffic to each unit. This can be accomplished either by static routes or dynamic routing protocols on both upstream and downstream routers.

A new command, failover groups, is being introduced to support active/active failover. Conceptually it is very similar to the Cisco's Multi-Group Hot Standby Router Protocol (HSRP) or Virtual Router Redundancy Protocol (VRRP). As mentioned before, PIX 7.0 supports a two-node active/active failover configuration with two failover groups.

The active/active failover feature leverages the Security Context feature. Figure 3-10 shows a typical two-node cluster. On unit A, ctx1 is active and ctx2 is standby. Unit B has ctx2 as active and ctx1 is standby. If unit A fails, unit B will have both contexts active and will process 100 percent of the firewall traffic.

Figure 3-10. Active/Active FO setup

Each failover group contains separate state machines to keep track of the failover state. The command failover active group can be used to change the active state of a failover group.

Case Studies

Case Studies

This section explores the failover features available on PIX Firewall in greater detail (the discussion is based on information from the New Product Introduction training by PIX marketing team with modification to be able to easily comprehend).

The resiliency of the connections through the PIX firewall can be achieved with failover, which means that if one PIX failed, the other PIX still would be available to process the packets. Starting from PIX Version 7.0, PIX can be configured in one of the following two failover modes:

• Active/standby model

• Active/active model

Active/Standby Model

In this model, one PIX acts as the active PIX that processes the traffic at the time, and the other unit acts as standby. In the event of failure of the active unit, the standby unit becomes active and starts processing packets. If stateful failover is configured, the transition of connections from active to standby is very smooth. The new features added to Version 7.0 are as follows:

• Stateful failover for VPN traffic This release introduces stateful failover for VPN connections. All security association (SA) state information and key material is synchronized automatically between the failover pair members, and this provides a highly resilient VPN solution.

• Non-Stop Online Software Upgrades This version allows you to perform software upgrades of failover pairs without affecting network uptime or connections flowing through the units. This is because of the ability being introduced in this version to perform inter-version state sharing between PIX failover pairs. This allows you to perform software upgrades to maintenance releases (for example, 7.0(1) upgrading to 7.0(2)) without affecting traffic flowing through the pair. There is no impact in both active/standby failover environments and active/active environments.

Reverse DNS & IDENT Protocol

Reverse DNS & IDENT Protocol

If you can connect to the servers but need to wait a long time before the data comes back, you might run into a problem with reverse DNS or IDENTD. To work around the problem with DNS issues, be sure to allow UDP/53 to the DNS server from the destination server, provided that the DNS server is behind the PIX firewall. To get around the problem for IDENT protocol, be sure to allow TCP/113 through the PIX firewall so that the server can communicate with the client on TCP/113.

High Memory Utilization

High Memory Utilization

If you cannot make a new connection across the PIX firewall, but your existing connections are working well, most likely you are running into low memory issues. Work through the steps that follow to identify and correct the problem:

Step 1.	Check the amount of free memory available with the following command: PIX# show memory Free memory: 40955872 bytes (61%) Used memory: 26152992 bytes (39%) ------------- ---------------- Total memory: 67108864 bytes (100%) PIX# This output will show the memory available. If free memory is low, investigate the reason for the low memory.
Step 2.	Analyze the syslog message to find out if you received the following message for running short of memory: %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 3.	Identify what is utilizing the memory (RAM) most heavily on the PIX. Before you can make that identification, you must know how memory is utilized on the PIX. Several things use up memory, such as the PIX image (run from RAM), configuration, the IPsec database, Xlates (translations), and connections. Most of the time, if the problem appears suddenly, the problem is with either translations or connections. So, you need to check the translations count with the show xlate command as shown below: PIX# show xlate count 350 in use, 400 most used PIX# The translation count in this example looks normal. Now, quickly find the connection count with the following command: PIX# show conn count 161723 in use, 161723 most used PIX# So, it appears that with only 350 translations, 16,1723 connections are made, which is an indication that your network may be infected by a virus or worm.
Step 4.	Now analyze the traffic load and find out from which interface the vast majority of traffic is coming. You can find this information with the command shown in Example 3-23. Example 3-23. Examining Traffic Load with the show traffic Command PIX# show traffic ! Following statistics is for outside interface outside: received (in 50.000 secs): 2170 packets 501050 bytes 65 pkts/sec 21752 bytes/sec transmitted (in 50.000 secs): 187629 packets 9455480 bytes 7601 pkts/sec 394156 bytes/sec ! Following statistics is for outside interface inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 50.000 secs): 7500 packets 124631 bytes 65 pkts/sec 6725 bytes/sec PIX# As you can see in the previous example, the inside interface is receiving the vast majority of traffic very fast (7208 pkts/sec). So, at this point, you can be fairly certain that a virus or a worm has infected the inside hosts.
Step 5.	Identify the hosts that are generating all the connections. Example 3-24 shows how to find out the hosts that are generating all these connections: Example 3-24. Identifying Hosts that Are Generating All These Connections PIX# show local-host | include host|count/limit local host: <10.1.1.10>, TCP connection count/limit = 2/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.20>, TCP connection count/limit = 5/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.50>, TCP connection count/limit = 12/unlimited UDP connection count/limit = 0/unlimited . . . local host: <10.1.1.100>, TCP connection count/limit = 151501/unlimited UDP connection count/limit = 0/unlimited PIX# The previous example shows that the majority of the traffic is indeed coming from 10.1.1.100.
Step 6.	Now that you have found the host that is generating all this traffic, look at the connections, and find the details of the type of connection as shown in Example 3-25. Example 3-25. Connections Generated by the Host PIX# show local-host 10.1.1.100 Interface inside: 300 active, 300 maximum active, 0 denied local host: <10.1.1.100>, TCP connection count/limit = 166108/unlimited TCP embryonic count = 166108 UDP connection count/limit = 0/unlimited Xlate(s): Global 209.165.201.21 Local 10.1.1.100 Conn(s): TCP out 65.101.32.157:135 in 10.1.1.100:34580 idle 0:01:43 Bytes 0 flags saA TCP out 65.103.108.191:135 in 10.1.1.100:8688 idle 0:01:43 Bytes 0 flags saA TCP out 65.100.205.160:135 in 10.1.1.100:7774 idle 0:01:43 Bytes 0 flags saA TCP out 65.101.182.19:135 in 10.1.1.100:39193 idle 0:01:43 Bytes 0 flags saA . . . . . . PIX# The preceding example shows that all connections are embryonic connections, which reaffirms that the host is infected.
Step 7.	To cure the problem, patch the machine, and take any other actions recommended by the anti-virus software vendor. If the patch is not possible immediately, shut down the machine and unplug it from the network.
Step 8.	As a temporary work-around, you can limit the maximum number of connections for the specific host as follows: PIX(config)# nat (inside) 1 10.1.1.100 255.255.255.255 50 0 For the remainder of the network, configure for unlimited connections as follows: PIX(config)# nat (inside) 1 0.0.0.0 0.0.0.0 0 0 Finally, be sure to clear the local host with the following command: PIX(config)# clear local-host 10.1.1.100
Step 9.	If the memory is leaked by a specific process, you may run into problem with memory on the PIX firewall even under normal load of traffic. Under this circumstance, use the output of show process memory over time to see which process is continually getting more memory allocated, but not freeing it back. Then look for specific bug on the code.

Large ACL

ACL comprises multiple Access Controls Elements (ACEs), which are processed sequentially in top-down fashion. So, it is very important that you configure the entries that are most likely to be matched towards the top of the access-list. The performance impact of an access-list increases linearly with the increase in the number of ACEs. So, try to summarize the address ranges of the ACEs as much as possible. The smaller the list is, the less the CPU is used and the shorter the time required to search sequentially for an entry. If you have a huge number of ACEs even after summarization, configure turbo ACL. Note that the turbo ACL feature is available only on PIX Firewall Version 6.2 and above.

Performance Issues

Performance Issues

When you run into high CPU or low memory on the PIX, you might observe one or more of the following symptoms:

• A packet drops across the firewall.

• You are unable to log in to PIX either via Telnet, SSH or even the console.

• You are unable to execute any command on the CLI even if you can connect to it.

This high CPU and low memory condition on the PIX firewall can be caused by normal or abnormal traffic. Examples of abnormal traffic are attacks, worms, viruses, and so on, in the network. However, if you have just deployed a new PIX in the network, and you are having performance issues, you might be reaching the CPU or memory limit on the PIX. This can happen if you have an additional feed or bandwidth increment for your existing PIX. It can also happen because of misconfiguration on the switch that causes the redirecting of all traffic into the port of the PIX firewall. However, if performance deteriorates suddenly, chances are that either you are running into a configuration issue on the PIX or your network is under attack. This section describes troubleshooting steps for isolating issues with CPU and memory problems.

High CPU Utilization

It is important to keep the CPU utilization under 60 percent. If utilization exceeds 60 percent, you must examine the traffic and see if you need to consider a firewall with higher performance, or if you are under attack.

Work through the following steps to identify and correct high CPU utilization problem:

Step 1.	Find out the summary of CPU utilization. In both single and multiple mode, execute the following command to obtain the summary of the CPU usage at any time: PIX# show cpu [usage] [context {all | context_name}] A sample output of the show cpu usage command is as follows: PIX# show cpu usage CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0% PIX# A sample output of CPU usage for a specific context in multiple context mode follows: PIX/context_name(config)# show cpu usage context admin CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0% PIX/context name(config)# You can find out the same information by executing the command from the context itself as follows: PIX/context_name(config)# show cpu usage context admin CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0% PIX/context name(config)# The following command shows how to display the CPU utilization for all contexts: PIX(config)# show cpu usage context all CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0% 5 sec 1 min 5 min Context Name 0% 0% 0% admin 59% 59% 59% system 41% 41% 41% PIX# If the CPU utilization is showing high, move to the next step to find out which process is causing the high CPU utilization.
Step 2.	Identify the process that is utilizing maximum CPU cycles. Execute the show process command on the PIX firewall and find out the run time of the process. You should compare the other processes with different poll processes, for example, the 557poll process. Example 3-18 shows that the Logger process is utilizing the maximum CPU cycles other than 557poll. Example 3-18. Output of show processes on PIX Firewall PIX(config)# show processes PC SP STATE Runtime SBASE Stack Process Hsi 001eab19 008a5a74 00557910 0 008a4aec 3628/4096 arp_timer Lsi 001f00bd 00a28dbc 00557910 0 00a27e44 3832/4096 FragDBGC Lwe 00119abf 02d280dc 0055b070 0 02d27274 3688/4096 dbgtrace Lwe 003e4425 02d2a26c 00557dd8 74440 02d28324 6936/8192 Logger Crd 001e26fb 0533940c 00557d88 6070290 05338484 3684/4096 557poll Lsi 00300a29 04c0f504 00557910 0 04c0e57c 3944/4096 xlate clean .... PIX(config)# As it is not clear how severely CPU cycles are utilized by the Logger process, take a show process command output in one-minute intervals. Now take the difference, and list the difference of CPU utilization as shown in Example 3-19. Example 3-19. The Differences in CPU Utilization by Different Process Process_Name Runtime (msec) Logger 35340 pix/intf3 28410 557poll 8250 i82543_timer 5180 i82542_timer 2330 As you can see, the Logger process is utilizing the maximum CPU cycles. Starting with PIX firewall version 7.0, you can find out the same information just discussed on which process is utilizing the highest CPU cycles by executing show processes cpu-hog. This is preferred to the method that was just explained.
Step 3.	Examine the process and take corrective action. As the Logger process is using the maximum CPU, review the configuration for syslog as shown in Example 3-20. Example 3-20. The Output of the show log Command PIX(config)# show log Syslog logging: enabled Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged ! Following line shows huge amount of trap level logging, which mean this much ! of syslog information is written to the syslog server. But this in total since ! the PIX is rebooted last. Trap logging: level warnings, 6929312 messages logged Logging to inside 172.16.171.10 History logging: disabled . . . PIX(config)# In Example 3-20, it is not conclusive how quickly the PIX is generating so many messages to the syslog. Therefore, execute the show log command again as shown in Example 3-21 and compare the new output with the previously taken output that was shown in Example 3-20. Example 3-21. The show log Output Taken After a Few Minutes PIX(config)# show log Syslog logging: enabled Buffer logging: level alerts, 0 messages logged !Notice the amount of messages logged after few minutes Trap logging: level warnings, 9152372 messages logged Logging to inside 172.16.171.10 PIX(config)#
Step 4.	Investigate the Syslog messages. At this stage, you should be fairly certain that the syslog is causing the high CPU utilization problem. So you need to determine if it is because of attack or misconfiguration. If the syslog server is available, perform the analysis from the syslog server, or turn on the buffer logging. For this specific example, assume that the syslog server is down, which will cause the PIX to generate the syslog messages as shown in Example 3-22. Example 3-22. show log Output When the Syslog Server Is Unreachable PIX(config)# show log Buffer logging: level warnings, 41527 messages logged Trap logging: level warnings, 9553127 messages logged Logging to inside 172.16.171.10 . . . 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside 400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside PIX(config)# In the previous example, ICMP Unreachable is generated by the syslog server for each syslog message PIX is sending to the syslog server. The problem is aggravated when the Intrusion Prevention System (IPS) is configured, because for every ICMP unreachable message, PIX is generating an IPS message and sending it to the syslog server, which in turn generates another similar message. And this cycle continues until the syslog server is rechargeable. To correct the problem, you need to bring up the syslog server or turn off logging. You might consider turning of IPS until the syslog server is up again.
Step 5.	Follow the same procedure to correct other problems. For example, if you are under attack, examining the syslog along with sniffer capture will assist in finding the hosts that are infected, so that you can correct the problem by patching the host, or, as a temporary work-around, configuring the PIX with ACL to drop the bad packets on the interface. Actions that need to be taken differ depending on the problem.
Step 6.	Re-examine the CPU output, and repeat as necessary.

For additional details on troubleshooting performance issues on the PIX firewall, refer to the following link:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a008009491c.shtml

To find out the function of different processes on the PIX, refer to the following link:

http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml

Troubleshooting Steps

Troubleshooting Steps

Use the show service-policy command to get statistics, and observe their behavior to determine operational status. There is no low latency queue or priority queue in multiple context modes.

Limitations of Virtual Firewall

Limitations of Virtual Firewall

Before delving into the details of configuration and troubleshooting of Virtual Firewall, it is important to understand some of the limitations of the Firewall implementation:

• Transparent Firewall in Multiple Contexts In routed mode, the classifier classifies the packet based on source VLAN, or destination IP address. So it is possible to share the same VLAN across multiple contexts with the help of NAT. But, in transparent mode, NAT cannot be configured. Therefore, classification is based only on source VLAN. This means it is not possible to share the same VLAN across multiple contexts.

• NAT Zero Access List and Shared Interfaces in routed mode On the PIX Firewall, you can configure "NAT zero access-list" to bypass the NAT for the traffic defined on the ACL. If you do that and if the VLAN is shared, the classifier will not know the destination address per context, so communication will fail.

• Hosts on Shared Interfaces Cannot Initiate Outbound Connections If you share the inside VLAN among multiple contexts, this will cause problems, unless you have the destination address translation configured. Usually outbound traffic is Internet-bound, and configuration address translation may not be possible. Therefore, it is strongly recommended not to share the inside VLAN for outbound traffic among multiple contexts.

Configuration Steps

Figure 3-9 shows two context deployments of the PIX firewall.

Figure 3-9. Two Context Deployment of the PIX Firewall

Work through the following steps to configure multiple contexts on the PIX firewall based on Figure 3-9:

1.	Change the mode of operation from the single to multiple with the following command: PIX(config)# mode multiple WARNING: This command will change the behavior of the device WARNING: This command will initiate a Reboot Proceed with change mode? [confirm] Convert the system configuration? [confirm] . . . . . . Once the PIX is booted up, execute the following show command and be sure that the security context mode is shown as multiple: PIX# show mode Security context mode: multiple PIX# After the PIX is converted to the multiple context, once you log in to the PIX, you will be taken to the System Resource Space (context). From here, you can configure the rest of the other contexts. The context admin will be created by default, which can be verified with the following command: PIX# show running-config context admin context admin config-url flash:/admin.cfg ! PIX#
2.	Configure interfaces. You need to configure all interfaces on the system context before they can be mapped to the user security context. For this setup, the inside and outside (both interfaces) are configured as trunk carrying VLAN 10 and 20 on the inside and VLAN 30 and 40 on the outside. For the outside interface, configure the PIX in the system context as follows: PIX(config)# interface Ethernet 0 PIX(config-if)# speed auto PIX(config-if)# duplex auto PIX(config)# interface Ethernet 0.30 PIX(config-subif)# vlan 30 PIX(config-subif)# interface Ethernet 0.40 PIX(config-subif)# vlan 40 Similarly, configure the inside interface for VLAN 10 and 20 as follows: PIX(config)# interface Ethernet 1 PIX(config-if)# speed auto PIX(config-if)# duplex auto PIX(config)# interface Ethernet 1.10 PIX(config-subif)# vlan 10 PIX(config-subif)# interface Ethernet 0.20 PIX(config-subif)# vlan 20 PIX(config-subif)#
3.	Create contexts. From system context, you can create a new context or change the role of admin context. To do this, you need to create a new context that you want to designate as admin context. For instance, if you want to configure contexts ctx1 and ctx2, and want to make ctx1 your admin context, you need to create the two new contexts as follows: PIX(config)# context ctx1 PIX(config-ctx)# config-url flash:/ctx1.cfg PIX(config)# context ctx2 PIX(config-ctx)# config-url flash:/ctx2.cfg PIX(config-ctx)# Now assign the admin context role to ctx1 context as follows: PIX(config)# admin-context ctx1 Finally remove the admin context that was created by default as follows: PIX(config)# no context admin Verify the contexts you have just created: PIX# show context Context Name Interfaces URL *ctx1 flash:/ctx1.cfg ctx2 flash:/ctx2.cfg Total active Security Contexts: 2 PIX#
4.	Associate interfaces to the contexts. You need to associate the interfaces from the system context with the allocate-interface commands. For example, to allocate interfaces Ethernet0.30, and Ethernet1.10, you can use the following commands: PIX(config)# context ctx1 PIX(config-ctx)# allocate-interface Ethernet0.30 PIX(config-ctx)# allocate-interface Ethernet1.10 Similarly, the following interface mappings are for context ctx2: PIX(config)# context ctx2 PIX(config-ctx)# allocate-interface Ethernet0.40 PIX(config-ctx)# allocate-interface Ethernet1.20 After creation of context, verify it with the following command: PIX# show context Context Name Interfaces URL *ctx1 Ethernet0.30, Ethernet1.10 flash:/ctx1.cfg ctx2 Ethernet0.40, Ethernet0.20 flash:/ctx2.cfg Total active Security Contexts: 2 PIX#
5.	Configure the interfaces on the context. From system context, go to the respective contexts to configure the interfaces with changeto command. The following commands show how to configure the interfaces on the context ctx1: PIX(config)# changeto context ctx1 PIX/ctx1(config)# interface Ethernet0.30 PIX/ctx1(config-if)# ip address 192.168.1.1 255.255.255.0 PIX/ctx1(config-if)# nameif outside PIX/ctx1(config-if)# exit The following commands are used to configure the inside interface on context ctx1: PIX/ctx1(config)# interface Ethernet1.10 PIX/ctx1(config-if)# ip address 10.1.1.1 255.255.255.0 PIX/ctx1(config-if)# nameif inside PIX/ctx1(config-if)# exit Configure interfaces for context ctx2 in the same way as for context ctx1: PIX(config)# changeto context ctx2 PIX/ctx2(config)# interface Ethernet0.40 PIX/ctx2(config-if)# ip address 192.168.2.1 255.255.255.0 PIX/ctx2(config-if)# nameif outside PIX/ctx2(config-if)# exit PIX/ctx2(config)# interface Ethernet1.20 PIX/ctx2(config-if)# ip address 10.1.2.1 255.255.255.0 PIX/ctx2(config-if)# nameif inside PIX/ctx2(config-if)# exit
6.	At this stage, you can treat both ctx1 and ctx2 contexts as individual firewalls and define the policies, NAT, and so on just as you would do for a regular PIX firewall. To go back to the system context, execute the following command: PIX(config-ctx)# changeto context system PIX(config)#

Virtual Firewall

Virtual Firewall

Beginning with PIX Firewall Version 7.0, you can logically partition a single PIX firewall into multiple logical PIX firewalls. Each logical PIX firewall can have its own security policy and administration control. This logical PIX firewall is called a Security Context, which is discussed next.

Security Context

Depending on your type of platform and the license, you can have up to 50 security contexts on the PIX, which means you can create up to 50 logical PIXen out of a single PIX firewall. As mentioned before, each security context is an independent firewall with its own security policy, interfaces, and administrators. Almost all the required features to provide the firewalling are possible with multiple contexts: firewall features, IPS, and management, to name a few. Note, however, that with multiple mode some of the features are not supported, including VPN (you can still establish VPN for management purposes only), Web VPN, Dynamic Routing protocols, and Multicast.

As soon as you convert the PIX firewall from single to multiple mode, it creates the system resource space and the admin context:

• System Resource After converting into multiple mode, when you log in to the PIX, you are taken to the System Resource space. From System Resource, the system administrator adds and manages contexts. The System Resource configuration identifies basic settings for the PIX firewall. The system Resource space does not include any network interfaces or network settings for itself; rather, when the system needs to access network resources (such as downloading the contexts from the server), it uses the admin context, which can have interface and network connectivity.

• Admin Context The admin context is created as soon as you convert the PIX firewall into multiple mode. It is just like any other context, except that users for admin context can access system administrator rights and can access the system and all other contexts. Typically, the admin context provides network access to network-wide resources, such as a syslog server or a context configuration server.

• User Security Context Apart from the admin context, you can create an individual security context. The limit is up to 50 based on your PIX model. Each security context is just like a single PIX firewall.

How the Virtual Firewall Works

When PIX operates in multiple context mode, each packet that enters the PIX firewall must be classified, so that the PIX can determine to which context it should send a packet. This is the job of a component of the PIX software called classifier. To classify the packets, the classifier goes through the following order to check with the destination IP address of the packet:

1.	Context Interface IP Address If the destination address of the packet is the interface IP address of the context (for instance, outside the interface IP address of a context) that means the packet is for a specific context, hence the classifier marks the packet for that context. An example of these types of packets is an SSH connection to a specific context to mange the context.
2.	Source Interface (VLAN) If the destination address of the packet is not one of the interface IP addresses of the context, the next check is performed based on the source VLAN of the packet. For example, if Ethernet 1 and Ethernet 2 interfaces are connected to VLAN100 and VLAN200, respectively, and if these interfaces are mapped to the context 1 and context 2, when the packet enters into PIX through Ethernet 1, the classifier will forward the packet to context 1, which is obvious. To make it little more complex, assume that Ethernet 1 is configured as trunks for VLAN100 and VLAN200, and VLAN100 is mapped to context1 and VLAN200 is mapped to context 2. With this setup, if the packet reaches to the PIX using VLAN 200, the classifier will forward the packet to context 2. Figure 3-7 shows an example to illustrate this point. Figure 3-7. Source VLAN Varies with Multiple Contexts [View full size image] This example shows VLAN 10, 20, and 30 mapped to separate contexts, so for outbound traffic; the classification will be made based on these sources' VLAN. Note that the PIX inside interface can be configured as a trunk for these VLANs, and then mapped to the individual VLAN to the respective contexts. For example, VLAN 10 should be mapped to Context A. For return traffic, the classifier would already know which context the return traffic belongs to. Even though the outside VLAN 500 is shared for the outside interface across all three contexts, the classifier builds up the knowledge base on which context the return packet should belong to based on the source address translation of the initial outbound traffic.
3.	Destination Address If the packet is not destined for the interface IP address of any context, and if the source VLAN of the packet is shared as shown in Figure 3-8 for the inbound connection, the classifier makes the decision on which context the packet needs to be forwarded based on the destination IP address. Figure 3-8. VLAN Sharing with Multiple Contexts Therefore, PIX classifier needs to have the knowledge of the destination network for each context. The classifier learns this destination network based on the translation configured on the contexts. Figure 3-8 illustrates this point.

For inbound traffic initiated from outside, the classifier looks only at static statements where the global interface matches the source interface of the packet. So in Figure 3-6, to allow the inbound connection initiated from outside, configure static NAT on each classifier for the inside network.

Troubleshooting Steps

Troubleshooting Steps

There are a combination of show and debug commands available to troubleshoot problems with transparent firewall. You can determine the mode of firewall with the following command:

PIX# show firewall

You can find out the mac-address-table on the PIX in transparent mode with the following command:

PIX# show mac-address-table [interface_name]

The following is sample output from the show mac-address-table command that shows the entire table:

PIX# show mac-address-table
interface mac address type Time Left
-----------------------------------------------------------------------
outside 0009.7cbe.2100 static -
inside 0010.7cbe.6101 static -
inside 0009.7cbe.5101 dynamic 10
PIX#

The following command displays the mac-address-table only for inside interface:

PIX# show mac-address-table inside
interface mac address type Time Left
-----------------------------------------------------------------------
inside 0010.7cbe.6101 static -
inside 0009.7cbe.5101 dynamic 10
PIX#

Use the following debug command in addition to the show commands to troubleshoot the transparent firewall issues:

• debug arp-inspection To track the code path of arp forwarding and arp inspection module in transparent firewall.

• debug mac-address-table To track insert/delete/update to the bridge table maintained for transparent firewall.

• debug l2-indication To track code path for processing of layer 2 (l2) indications.

You need to turn on syslog level 4 to see all messages pertaining to transparent problem. This can help in identifying problems pertaining to transparent firewall.

• MAC Spoofing If you receive a MAC address entry that conflicts with the existing static entry (MAC address to a specific interface), you will get the following syslog message:

  %PIX-3-321001: Deny MAC address , possible spoof attempt on interface 
  

  For example, if you have a static MAC address defined pointing to the DMZ interface, and if you receive the same MAC address dynamically from the inside interface, then it will be considered as MAC spoofing.

• ARP Inspection If the ARP inspection module drops a packet, the following syslog message will be generated:

  [View full width]
  %PIX-3-321002: ARP inspection check failed for arp  received from host
    on interface . This host is advertising MAC Address
    for IP Address , which is currently statically assigned to MAC
   Address .
  

• Host Movement If a MAC address of a host is moved from one interface to the other, the following syslog message will be generated:

  %PIX-4-411001: MAC  moved from  to 
  

  Here interface-1 is the name of the interface from where the host has moved. Interface-2 is the name of the interface to where the host has moved.

• L2 Table Flooding When the bridge table is full and an attempt is made to add one more entry by the Mac-address of the host, the following syslog message will be generated:

  [View full width]
  %PIX-4-411002: Detected bridge table full while inserting MAC  on interface
  . Number of entries = 
  

Configuration Steps

Configuration Steps

Work through the following steps to configure transparent firewall based on Figure 3-6:

1.	Turn on transparent firewall. Convert the PIX firewall with the following command: PIX(config)# show firewall Firewall mode: Router PIX(config)# firewall transparent Switched to transparent mode PIX(config)#
2.	Assign an IP address for management. Configure the IP address for the PIX firewall for management purposes. This address cannot be used as a default gateway for the host. The IP address should be of the same subnet of the other hosts. PIX(config)# ip address 10.1.1.1 255.255.255.0
3.	Configure interfaces (inside and outside). Bring up both inside and outside interfaces and be sure not to use the IP addresses on the interface. The following command will turn on the outside interface: PIX(config)# interface Ethernet 0 PIX(config-if)# nameif outside PIX(config-if)# security-level 0 PIX(config-if)# no shutdown Configure the inside interface with the following commands: PIX(config)# interface Ethernet 1 PIX(config-if)# nameif inside PIX(config-if)# security-level 100 PIX(config-if)# no shutdown
4.	Configure an access-list (optional). You can configure an access-list optionally, and filter the traffic. The command syntax is as follows: [View full width] PIX(config)# [no] access-list ethertype [unicast | multicast | broadcast] For example, if you want to allow only the IPX traffic, and deny the rest, your ACL configuration will be the like the following: PIX(config)# access-list 100 ethertype permit ipx Then, you need to apply the access-list on the interface. If the access-list 100 needs to be applied on the inside interface, the configuration will look like this: PIX(config)# access-group 100 in interface inside
5.	Configure ARP inspection. ARP inspection is turned off by default, which means that all ARP packets are allowed through the PIX firewall. You can control the flow of ARP packets by enabling ARP inspection. When you enable ARP inspection, PIX Firewall compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the actions as follows: a. If the IP address, MAC address, and source interface match an ARP entry, the packet is allowed through. b. If there is a mismatch between the MAC address, the IP address, or the interface, the PIX firewall drops the packet. c. If the ARP packet does not match any entries in the static ARP table, you can set the PIX firewall to either forward the packet out all interfaces (flood), or to drop the packet. ARP inspection prevents network devices from ARP spoofing. ARP spoofing can lead an attacker to a man-in-the-middle attack. For example, a host sends an ARP request to the gateway router; the gateway router responds with the gateway router MAC address. The attacker, however, sends another ARP response to the host with the attacker's MAC address instead of the router's MAC address. The host, thinking that the hacker's MAC address is the valid destination, starts forwarding the packet. The attacker can now intercept all the host traffic before forwarding it on to the router. ARP inspection ensures that an attacker cannot send an ARP response with the attacker MAC address, if the correct MAC address and the associated IP address are in the static ARP table. Note In multiple context mode, the commands in this chapter can be entered in a security context, but not in the system context. The dedicated management interface, if present, never floods packets even if this parameter is set to flood. You can add a static ARP Entry with the following command: arp interface_name ip_address mac_address ARP inspection compares ARP packets with static ARP entries in the ARP table. The following command allows ARP responses from the router at 10.1.1.1 with the MAC address 0009.7cbe.2100 on the outside interface. Enter the following command. PIX(config)# arp outside 10.1.1.3 0009.7cbe.2100 Note that the transparent firewall uses dynamic ARP entries in the ARP table for traffic to and from PIX Firewall, such as management traffic. To enable ARP inspection, use the following command: PIX(config)# arp-inspection interface name enable flood | no-flood] Here flood forwards non-matching ARP packets out all interfaces, and no-flood drops non-matching packets. Note that the default setting is to flood non-matching packets. To restrict ARP through the PIX firewall to only static entries, set this command to no-flood. The following command enables ARP inspection on the outside interface, and to drop all non-matching ARP Packets. PIX(config)# arp-inspection outside enable no-flood To view the current settings for ARP inspection on all interfaces, enter the following command: PIX(config)# show arp-inspection
6.	Configure the MAC address table. Just as with a normal bridge or switch, PIX Firewall learns and builds a MAC address table by inserting the MAC address with the source interface. Unlike a normal bridge or switch, if the destination is not present on the MAC table, PIX does not flood on all interfaces. Instead it does the following: a. If the packets are for devices directly connected, PIX firewall generates an ARP request for the destination IP address so that it can learn which interface receives the ARP response. b. If the packets for devices are not directly connected, PIX Firewall generates a ping to the destination IP address so that PIX Firewall can learn which interface receives the ping reply. The original packet is dropped. You can build up the MAC table dynamically or statically. By default, each interface automatically learns the MAC addresses of entering traffic, and PIX Firewall adds corresponding entries to the MAC address table. You can disable MAC address learning if desired; however, unless you add MAC addresses to the table statically, no traffic can pass through the PIX Firewall. To disable MAC address learning, enter the following command. PIX(config)# mac-learn interface_name disable The no form of this command re-enables MAC address learning. The clear configure mac-learn command re-enables MAC address learning on all interfaces. You can add static MAC addresses to the MAC address table if desired. One benefit to adding static entries is to guard against MAC spoofing. If a client with the same MAC address as a static entry attempts to send traffic to an interface that does not match the static entry, the security appliance drops the traffic and generates a system message. To add a static MAC address to the MAC address table, enter the following command: PIX(config)# mac-address-table static interface_name mac_address The interface name is the source interface. The default timeout value for dynamic MAC address table entries is five minutes, but you can change the timeout. To change the timeout, enter the following command. PIX(config)# mac-address-table aging-time timeout_value The timeout_value (in minutes) is between 5 and 720 (12 hours). The default is 5 minutes.

Troubleshooting Steps

Troubleshooting Steps

Before delving into the details of how to troubleshoot connectivity problems, it is important to understand the meaning of different connection flags to quickly identify the problem with connections. You can use combinations of show, debug, and the syslog message that are discussed in this section, to troubleshoot connectivity problems.

Figure 3-5 shows the connection flags for both inbound and outbound connections across the PIX firewall for TCP connections.

Figure 3-5. PIX Firewall Connection Flags Information for Both Inbound and Outbound Connections

The following numbered steps show the sequence of the packets and the connections flags for the outbound connection (see Figure 3-5) through the PIX firewall:

1.	When PIX receives an initial SYN packet from the inside network, the SYN packet is permitted by the access-list on the ingress interface, a translation (xlate) is built up (if NAT is configured), and the connection is created with the flag saA, which can be verified with the show connection command.
2.	The outside device responds to the SYN packet with a SYN+ACK. The connection flags are updated to reflect this, and now show A. Again, this information can be verified with show connection command output.
3.	The inside device responds to the SYN+ACK with an ACK, and this completes the TCP 3-way handshake. The connection is now considered up (the U flag will be shown with the show connection command).
4.	When the outside device sends the first data packet, the connection is updated, and an I is added to the connection flags to indicate that the PIX received inbound data on that connection.
5.	Finally, after the inside device sends a data packet, the connection is updated to include the O flag.

Understanding the connection flags for the teardown of the connection is very important. Following is the sequence of packets and the corresponding connection flags for the outbound connection:

1.	When PIX receives a FIN packet from the inside, PIX updates the connection flags by adding an f.
2.	The outside device immediately responds to the FIN packet with a FIN+ACK. The connection flags are updated to reflect this with UfFR flags.
3.	The inside device responds to the FIN+ACK with a final ACK, and the PIX tears down the connection. Thus, there are no more connection flags, because the connection no longer exists.

The sequence of events that takes place for the inbound connection is similar to that of the outbound connection on the PIX firewall but it has different flags. Details are not discussed here, but refer to Figure 3-5 for a summary of connection flags for the inbound connection.

If you have connectivity problems through the PIX firewall, you need to analyze the connection syslog ID, which is in the following form:

PIX-6-302014: Teardown TCP connection number for interface_name:real_address/
 real_port to interface_name:real_address/real_port duration time bytes number
 [reason] [(user)]

The reason field is important to understand. Table 3-9 outlines the reasons for connection terminations.

Table 3-9. Reasons for Connection Termination

Connection Flags	Meaning
Reset-I	Reset was from Inside
Reset-O	Reset was from Outside
TCP FINs	Normal Close Down Sequence
FIN Timeout	Force Termination After 15 Seconds
SYN Timeout	Force Termination After 2 Minutes
Xlate[*] Clear	Command Line Removal
Deny	Terminate by Application Inspection
SYN Control	Back Channel Initiation from Wrong Side
Uauth Deny	Deny by URL Filter
Unknown	Catch All Error

^[*] An xlate is created dynamically when a packet from a high-security-level interface initiates an outbound connection to a lower-security-level interface.

If you cannot make connection across the PIX firewall, the problem might be with the access-list, NAT, routing, and so on. Work through the following steps to correct the connectivity problem:

1.	Is the initial packet reaching the PIX firewall? This is the first and most important step to troubleshoot the connectivity problem across the firewall. You must be sure the PIX is receiving the initial packet on its interface. If ACL is configured on the interface, you can find out if the packet is reaching to the PIX interface by executing the show access-list command. You should see the hit counts incrementing on the show access-list command output. Another way you can verify if the packets are reaching is by executing the show interface command as shown in Example 3-16. You should see that the Input counters are incrementing. The software input queue is an indicator of traffic load, and "No buffers" indicates packet drops, typically due to bursty traffic. Example 3-16. Interface Statistics of Inside Interface PIX-A# show interface inside Interface Ethernet1 "inside", is up, line protocol is up Hardware is i82559, BW 100 Mbps Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps) MAC address 000d.2807.097a, MTU 1500 IP address 192.168.1.1, subnet mask 255.255.255.0 32 packets input, 2100 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 36 packets output, 2304 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collisions, 0 deferred 0 lost carrier, 0 no carrier input queue (curr/max blocks): hardware (128/128) software (0/1) output queue (curr/max blocks): hardware (0/1) software (0/1) Received 35 VLAN untagged packets, 1610 bytes Transmitted 36 VLAN untagged packets, 1008 bytes Dropped 0 VLAN untagged packets PIX-A# To see the details of the packet going across or reaching the PIX firewall, run the capture command as shown in the "Diagnostic Commands and Tools" section.
2.	Is the initial packet allowed by the ACL? For outbound connection across the PIX firewall, you do not have to configure an ACL to allow the traffic. By default, all traffic is allowed. However, for the inbound connection, you must configure an ACL to allow the traffic. Execute the show access-list command and be sure that you see that the hit count on ingress interface ACL increases over time. Remember that creating an ACL is not sufficient; you also must apply the ACL on the interface in the proper direction. Execute show running-config access-group command to verify if indeed the access-list is applied to the correct interface.
3.	Is translation being built up? If Steps 12 are verified, the next step is to verify if the translation is being built up for the connection (this is mandatory when the nat-control is turned on or when nat-control is turned off but NAT is configured to perform the translation). If the translation is not being built up, first be sure the nat/global is configured correctly. For example, to verify that the nat/global is configured for the outbound connection, execute the following commands: PIX# show run nat-control PIX# show running-config nat PIX# show running-config global PIX# show running-config static PIX# show run alias You need to ensure that the NAT identification corresponds to the global identification. If some hosts are able to connect through the PIX, but some are not, the problem might be that you are running short of addresses from the global pool. To be on the safe side, it is recommended to configure a PAT address along with the NAT pool (when the nat-control is turned on). To verify that a host is building up translation, use the following command: [View full width] show xlate [global | local [netmask ]] [gport | lport ] [debug] For example, to find out if inside host 10.1.1.70 is building up the translation, execute the following command: PIX# show xlate local 10.1.1.70 debug NAT from inside:10.1.1.70 to outside:20.1.1.70 flags - idle 0:00:10 timeout 3:00:00 PIX# For translation issues, the following syslog message will be generated: [View full width] PIX-3-305005: No translation group found for tcp src inside:10.1.1.70/11039 dst outside :198.133.219.25/80 PIX-3-305006: regular translation creation failed for tcp src inside:10.1.1.70/11040 dst outside:198.133.219.25/80 To find out the translation and connection entry created for a specific host (10.1.1.70), execute the following command: PIX# show local 10.1.1.70 If the NAT is not working as expected, run the following debug command to get the details of the NAT translation process: PIX# debug pix process
4.	Is the connection being built up? If the translation is built up, next make sure that the connection is built up. This can be verified with the show conn or show conn detail commands to find the connection flag. The meaning of different connection flags is in Figure 3-5. You also need to analyze the syslog to see why the connection is being reset. Refer to Table 3-9 to find out the meaning of different connection resets. For example, if you see connection reset reason as Reset-I, the inside host is resetting the connection.
5.	Is routing set up correctly on PIX? If you see the translation being built up on the PIX but it does not know where to forward the traffic, you must ensure that you have the route set up correctly. For outbound traffic towards the Internet, you must ensure that the default gateway is set up on the outside interface. If the default route is missing to reach to 198.133.219.25 from the inside host 10.1.1.1, you will get the following syslog message: PIX-6-110001: No route to 198.133.219.25 from 10.1.1.70 Use the following commands to find out if the route indeed exists: PIX(config)# show route | include 198.133.219.25 In Figure 3-4, if you want the users from network 10.1.2.0/24 to be able to access 198.133.219.25, in addition to a default gateway configured on the outside interface, you must configure a static route pointing to the inside router as follows: PIX(config)# route inside 10.1.2.0 255.255.255.0 10.1.1.2
6.	Is the return traffic coming back to the PIX? If the global pool or the translated IP address of the initial packet is not part of the same network as the outside interface network of the PIX, the next-hop router will not get the proxy Address Resolution Protocol (ARP) from the PIX. Hence, it is mandatory to create a static route on the next-hop router for the translated network so that the next hop router directs the packets back to the outside interface of the PIX Firewall. For example, in Figure 3-4, if you are using any address on the global pool which is part of network 20.1.1.0/24, you do not need any route defined on the router where the PIX outside interface is connected. This is because, with nat/global, PIX will proxy ARP the MAC address from the global pool to the router that is connected to the outside router. If the global pool is defined with the IP address not part of the outside interface network (20.1.1.0/24), for example, the 30.1.1.0/24 network, you must define a static route on the outside router for network 30.1.1.0/24 pointing to the outside interface (20.1.1.1) of the PIX firewall.
7.	Is the router or the PIX ARPing correctly? If everything else is set up correctly, and you are still unable to make connection across the PIX firewall, execute the show arp command on the PIX to see if the next-hop address has an ARP entry. For example, if you have a default gateway setup pointing to 20.1.1.10, on the PIX you should see an ARP entry for the 20.1.1.10 address. If the ARP entry is built as shown by the show arp command, execute the debug arp command and then execute the clear arp command to see if the ARP is having issues. ARP can be a problem for the next-hop router (especially on the outside router). When you make changes to the global pool, static and so on, it is recommended to clear the ARP on the outside router.
8.	Packets are dropped by Accelerated Security Path (ASP). PIX Firewall contains a system for packet inspection, called the Accelerated Security Path (ASP). All packets have to go through the ASP logic, and if any packet violates the logic, that packet will be dropped by the PIX firewall. Sometimes what the ASP considers to be illegitimate packets may be legitimate for you. So it's important to find out what packets the ASP is dropping. This section presents some techniques that you can use to help troubleshoot the problem. Execute the show asp drop command to find out counter values for the reasons for packet drops by the ASP. Issue the command clear asp drop then show asp drop to get a baseline of the drops so far, then send the traffic that is not making it through the firewall, and then issue show asp drop again, and check which counters are incrementing. Following is a sample output of the show asp drop command. PIX# show asp drop Frame drop: Flow is denied by access rule 3001 First TCP packet not SYN 3000 Bad TCP flags 30 TCP failed 3 way handshake 51 TCP RST/FIN out of order 71 TCP SEQ in SYN/SYNACK invalid 10 TCP ACK in SYNACK invalid 10 TCP packet SEQ past window 40 TCP RST/SYN in window 60 TCP DUP and has been ACKed 16 DNS Guard id not matched 80 PIX# To find the details on what packets are dropped by the ASP, turn on capture with the following command: PIX# capture capture_name type asp-drop drop-code all packet-length 1518 Drop-code can be all or a specific one in the preceding command. However, because you typically don't know why the PIX is dropping the packet, you cannot specify a specific drop-code; therefore you will usually use all as drop-code. You might need to run interface capture in addition to ASP drop capture to troubleshoot ASP drop issues. Additionally, instead of sending the debug output to the console, redirect it to the logging buffer with the following commands: PIX(config)# logging list mylist message 711001 PIX(config)# logging buffered mylist PIX(config)# logging debug-trace PIX(config)# debug fixup tcp PIX(config)# debug pix process

After going through the preceding steps, you should be able to resolve most of the connectivity problems. However, if you run into a problem with the PIX Firewall version, you might need to engage the Technical Assistance Center (TAC) for further analysis.