High Memory Utilization

High Memory Utilization

If you cannot make a new connection across the PIX firewall, but your existing connections are working well, most likely you are running into low memory issues. Work through the steps that follow to identify and correct the problem:

Step 1.	Check the amount of free memory available with the following command: PIX# show memory Free memory: 40955872 bytes (61%) Used memory: 26152992 bytes (39%) ------------- ---------------- Total memory: 67108864 bytes (100%) PIX# This output will show the memory available. If free memory is low, investigate the reason for the low memory.
Step 2.	Analyze the syslog message to find out if you received the following message for running short of memory: %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 3.	Identify what is utilizing the memory (RAM) most heavily on the PIX. Before you can make that identification, you must know how memory is utilized on the PIX. Several things use up memory, such as the PIX image (run from RAM), configuration, the IPsec database, Xlates (translations), and connections. Most of the time, if the problem appears suddenly, the problem is with either translations or connections. So, you need to check the translations count with the show xlate command as shown below: PIX# show xlate count 350 in use, 400 most used PIX# The translation count in this example looks normal. Now, quickly find the connection count with the following command: PIX# show conn count 161723 in use, 161723 most used PIX# So, it appears that with only 350 translations, 16,1723 connections are made, which is an indication that your network may be infected by a virus or worm.
Step 4.	Now analyze the traffic load and find out from which interface the vast majority of traffic is coming. You can find this information with the command shown in Example 3-23. Example 3-23. Examining Traffic Load with the show traffic Command PIX# show traffic ! Following statistics is for outside interface outside: received (in 50.000 secs): 2170 packets 501050 bytes 65 pkts/sec 21752 bytes/sec transmitted (in 50.000 secs): 187629 packets 9455480 bytes 7601 pkts/sec 394156 bytes/sec ! Following statistics is for outside interface inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 50.000 secs): 7500 packets 124631 bytes 65 pkts/sec 6725 bytes/sec PIX# As you can see in the previous example, the inside interface is receiving the vast majority of traffic very fast (7208 pkts/sec). So, at this point, you can be fairly certain that a virus or a worm has infected the inside hosts.
Step 5.	Identify the hosts that are generating all the connections. Example 3-24 shows how to find out the hosts that are generating all these connections: Example 3-24. Identifying Hosts that Are Generating All These Connections PIX# show local-host | include host|count/limit local host: <10.1.1.10>, TCP connection count/limit = 2/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.20>, TCP connection count/limit = 5/unlimited UDP connection count/limit = 0/unlimited local host: <10.1.1.50>, TCP connection count/limit = 12/unlimited UDP connection count/limit = 0/unlimited . . . local host: <10.1.1.100>, TCP connection count/limit = 151501/unlimited UDP connection count/limit = 0/unlimited PIX# The previous example shows that the majority of the traffic is indeed coming from 10.1.1.100.
Step 6.	Now that you have found the host that is generating all this traffic, look at the connections, and find the details of the type of connection as shown in Example 3-25. Example 3-25. Connections Generated by the Host PIX# show local-host 10.1.1.100 Interface inside: 300 active, 300 maximum active, 0 denied local host: <10.1.1.100>, TCP connection count/limit = 166108/unlimited TCP embryonic count = 166108 UDP connection count/limit = 0/unlimited Xlate(s): Global 209.165.201.21 Local 10.1.1.100 Conn(s): TCP out 65.101.32.157:135 in 10.1.1.100:34580 idle 0:01:43 Bytes 0 flags saA TCP out 65.103.108.191:135 in 10.1.1.100:8688 idle 0:01:43 Bytes 0 flags saA TCP out 65.100.205.160:135 in 10.1.1.100:7774 idle 0:01:43 Bytes 0 flags saA TCP out 65.101.182.19:135 in 10.1.1.100:39193 idle 0:01:43 Bytes 0 flags saA . . . . . . PIX# The preceding example shows that all connections are embryonic connections, which reaffirms that the host is infected.
Step 7.	To cure the problem, patch the machine, and take any other actions recommended by the anti-virus software vendor. If the patch is not possible immediately, shut down the machine and unplug it from the network.
Step 8.	As a temporary work-around, you can limit the maximum number of connections for the specific host as follows: PIX(config)# nat (inside) 1 10.1.1.100 255.255.255.255 50 0 For the remainder of the network, configure for unlimited connections as follows: PIX(config)# nat (inside) 1 0.0.0.0 0.0.0.0 0 0 Finally, be sure to clear the local host with the following command: PIX(config)# clear local-host 10.1.1.100
Step 9.	If the memory is leaked by a specific process, you may run into problem with memory on the PIX firewall even under normal load of traffic. Under this circumstance, use the output of show process memory over time to see which process is continually getting more memory allocated, but not freeing it back. Then look for specific bug on the code.

Large ACL

ACL comprises multiple Access Controls Elements (ACEs), which are processed sequentially in top-down fashion. So, it is very important that you configure the entries that are most likely to be matched towards the top of the access-list. The performance impact of an access-list increases linearly with the increase in the number of ACEs. So, try to summarize the address ranges of the ACEs as much as possible. The smaller the list is, the less the CPU is used and the shorter the time required to search sequentially for an entry. If you have a huge number of ACEs even after summarization, configure turbo ACL. Note that the turbo ACL feature is available only on PIX Firewall Version 6.2 and above.