All Cisco-Network Study Notes: Performance Issues

Performance Issues

When you run into high CPU or low memory on the PIX, you might observe one or more of the following symptoms:

A packet drops across the firewall.
You are unable to log in to PIX either via Telnet, SSH or even the console.
You are unable to execute any command on the CLI even if you can connect to it.

This high CPU and low memory condition on the PIX firewall can be caused by normal or abnormal traffic. Examples of abnormal traffic are attacks, worms, viruses, and so on, in the network. However, if you have just deployed a new PIX in the network, and you are having performance issues, you might be reaching the CPU or memory limit on the PIX. This can happen if you have an additional feed or bandwidth increment for your existing PIX. It can also happen because of misconfiguration on the switch that causes the redirecting of all traffic into the port of the PIX firewall. However, if performance deteriorates suddenly, chances are that either you are running into a configuration issue on the PIX or your network is under attack. This section describes troubleshooting steps for isolating issues with CPU and memory problems.

High CPU Utilization

It is important to keep the CPU utilization under 60 percent. If utilization exceeds 60 percent, you must examine the traffic and see if you need to consider a firewall with higher performance, or if you are under attack.

Work through the following steps to identify and correct high CPU utilization problem:

Step 1.

Find out the summary of CPU utilization.

In both single and multiple mode, execute the following command to obtain the summary of the CPU usage at any time:

PIX# show cpu [usage] [context {all | context_name}]

A sample output of the show cpu usage command is as follows:

PIX# show cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0%
PIX#

A sample output of CPU usage for a specific context in multiple context mode follows:

PIX/context_name(config)# show cpu usage context admin
CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0%
PIX/context name(config)#

You can find out the same information by executing the command from the context itself as follows:

PIX/context_name(config)# show cpu usage context admin
CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0%
PIX/context name(config)#

The following command shows how to display the CPU utilization for all contexts:

PIX(config)# show cpu usage context all
CPU utilization for 5 seconds = 1%; 1 minute: 0%; 5 minutes: 0%
5 sec 1 min 5 min Context Name
0% 0% 0% admin
59% 59% 59% system
41% 41% 41% 
PIX#

If the CPU utilization is showing high, move to the next step to find out which process is causing the high CPU utilization.

Step 2.

Identify the process that is utilizing maximum CPU cycles.

Execute the show process command on the PIX firewall and find out the run time of the process. You should compare the other processes with different poll processes, for example, the 557poll process. Example 3-18 shows that the Logger process is utilizing the maximum CPU cycles other than 557poll.

Example 3-18. Output of show processes on PIX Firewall

PIX(config)# show processes
   PC       SP       STATE      Runtime      SBASE      Stack Process
Hsi 001eab19 008a5a74 00557910         0   008a4aec  3628/4096 arp_timer
Lsi 001f00bd 00a28dbc 00557910         0   00a27e44  3832/4096 FragDBGC
Lwe 00119abf 02d280dc 0055b070         0   02d27274  3688/4096 dbgtrace
Lwe 003e4425 02d2a26c 00557dd8     74440   02d28324  6936/8192 Logger
Crd 001e26fb 0533940c 00557d88   6070290   05338484  3684/4096 557poll
Lsi 00300a29 04c0f504 00557910         0   04c0e57c  3944/4096 xlate clean
....
PIX(config)#

As it is not clear how severely CPU cycles are utilized by the Logger process, take a show process command output in one-minute intervals. Now take the difference, and list the difference of CPU utilization as shown in Example 3-19.

Example 3-19. The Differences in CPU Utilization by Different Process

Process_Name           Runtime (msec)
Logger                 35340
pix/intf3              28410
557poll                 8250
i82543_timer            5180
i82542_timer            2330

As you can see, the Logger process is utilizing the maximum CPU cycles. Starting with PIX firewall version 7.0, you can find out the same information just discussed on which process is utilizing the highest CPU cycles by executing show processes cpu-hog. This is preferred to the method that was just explained.

Step 3.

Examine the process and take corrective action.

As the Logger process is using the maximum CPU, review the configuration for syslog as shown in Example 3-20.

Example 3-20. The Output of the show log Command

PIX(config)# show log
Syslog logging: enabled
   Standby logging: disabled
   Console logging: disabled
   Monitor logging: disabled
   Buffer logging: level alerts, 0 messages logged
! Following line shows huge amount of trap level logging, which mean this much
! of syslog information is written to the syslog server. But this in total since
! the PIX is rebooted last.
   Trap logging: level warnings, 6929312 messages logged
       Logging to inside 172.16.171.10
   History logging: disabled
. . .
PIX(config)#

In Example 3-20, it is not conclusive how quickly the PIX is generating so many messages to the syslog. Therefore, execute the show log command again as shown in Example 3-21 and compare the new output with the previously taken output that was shown in Example 3-20.

Example 3-21. The show log Output Taken After a Few Minutes

PIX(config)# show log
Syslog logging: enabled
   Buffer logging: level alerts, 0 messages logged
!Notice the amount of messages logged after few minutes
   Trap logging: level warnings, 9152372 messages logged
       Logging to inside 172.16.171.10
PIX(config)#

Step 4.

Investigate the Syslog messages.

At this stage, you should be fairly certain that the syslog is causing the high CPU utilization problem. So you need to determine if it is because of attack or misconfiguration. If the syslog server is available, perform the analysis from the syslog server, or turn on the buffer logging.

For this specific example, assume that the syslog server is down, which will cause the PIX to generate the syslog messages as shown in Example 3-22.

Example 3-22. show log Output When the Syslog Server Is Unreachable

PIX(config)# show log
    Buffer logging: level warnings, 41527 messages logged
    Trap logging: level warnings, 9553127 messages logged
        Logging to inside 172.16.171.10
. . .
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
400011: IDS:2001 ICMP unreachable from 172.16.171.10 to 172.16.171.1 on interface inside
PIX(config)#

In the previous example, ICMP Unreachable is generated by the syslog server for each syslog message PIX is sending to the syslog server. The problem is aggravated when the Intrusion Prevention System (IPS) is configured, because for every ICMP unreachable message, PIX is generating an IPS message and sending it to the syslog server, which in turn generates another similar message. And this cycle continues until the syslog server is rechargeable.

To correct the problem, you need to bring up the syslog server or turn off logging. You might consider turning of IPS until the syslog server is up again.

Step 5.

Follow the same procedure to correct other problems. For example, if you are under attack, examining the syslog along with sniffer capture will assist in finding the hosts that are infected, so that you can correct the problem by patching the host, or, as a temporary work-around, configuring the PIX with ACL to drop the bad packets on the interface. Actions that need to be taken differ depending on the problem.

Step 6.

Re-examine the CPU output, and repeat as necessary.

For additional details on troubleshooting performance issues on the PIX firewall, refer to the following link:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a008009491c.shtml

To find out the function of different processes on the PIX, refer to the following link:

http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml