The next two chapters are focused primarily on essential TCP/IP
troubleshooting skills and tools. Here in this chapter, we will
explain
show
and
debug
commands. In addition, generic commands
such as
ping
and
traceroute
will be applied to network problems. Problem isolation
techniques that are used in troubleshooting LANs will be outlined and implemented. Finally,
the use and kinds of access lists will be examined.
The next chapter, Chapter 38, “TCP/IP Routing Protocol Troubleshooting,” focuses on the
IP routing protocols, including RIP, IGRP, EIGRP, OSPF, and BGP. In addition to an explanation
of these routing protocols, the
show
and
debug
commands used specifically for them will
be examined. The final part of Chapter 38 explores redistribution issues and solutions.
Many of the
show
and
debug
commands are not protocol-specific. Though these commands do
not deal exclusively with the TCP/IP protocol, they are used in troubleshooting many TCP/IP problems
and therefore are included in this chapter for completeness. As is the case with the
show
and
debug
commands, logging and core dumps are not limited to the TCP/IP but can be used to contribute
to troubleshooting TCP/IP problems and are also included in this chapter for completeness.
In addition to all the detailed problem-solving techniques presented in these two chapters,
quick reference summary charts are located at the end of Chapter 38. These tables help to
quickly associate a cause to many TCP/IP symptoms.
Troubleshooting Commands
We will cover several troubleshooting tools in this chapter, each of which is part of the Cisco IOS.
There are many
show
commands that are supported by the router. In addition to
show
commands,
a tool called debug is used to see specific information regarding packet transfer and exchange.
Part of effectively using these tools is using them without adversely affecting the router and its
many processes. Here you will learn the specifics of several troubleshooting commands, along with
the information needed in order to use them without causing additional problems on your network.
We start with non-intrusive, Cisco-specific
show
commands. After discussing the
show
commands, we move on to the debug tool. To finalize this section, we discuss some non-
Cisco-specific troubleshooting tools:
ping
and
traceroute
.
show
Commands
A large number of
show
commands are supported by Cisco IOS. Explaining them all is beyond the
scope of this book. The most effective and useful
show
commands are described in the following
sections, and Table 37.1 lists the ones most frequently used. To get an idea of all of the
show
commands,
execute the
show ?
command from the router prompt.
show version
This command is used to display the system hardware and software versions. It also provides
information about how long the router was running and the reason it was last restarted. Review
the following output of the show version command:
Router_B>show version
Cisco Internetwork Operating System Software
IOS (tm) RSP Software (RSP-JSV-M), Version 12.1(16), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 09-Jul-02 07:36 by kellythw
Image text-base: 0x60010958, data-base: 0x614C4000
ROM: System Bootstrap, Version 11.1(8)CA1, EARLY DEPLOYMENT RELEASE
SOFTWARE (fc1)
BOOTLDR: RSP Software (RSP-BOOT-M), Version 12.1(16), RELEASE SOFTWARE (fc1)
Router_B uptime is 35 weeks, 1 day, 6 hours, 18 minutes
System returned to ROM by reload at 00:12:02 EST Tue Oct 8 2002
System restarted at 23:52:37 EST Mon Oct 7 2002
System image file is "slot0:rsp-jsv-mz.121-16.bin"
cisco RSP4 (R5000) processor with 131072K/2072K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on
G.703/E1 software, Version 1.0.
G.703/JT2 software, Version 1.0.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
Bridging software.
TN3270 Emulation software.
Chassis Interface.
5 VIP2 controllers (2 FastEthernet)(6 HSSI)(1 ATM).
1 VIP2 R5K controller (8 Serial).
2 FastEthernet/IEEE 802.3 interface(s)
8 Serial network interface(s)
6 HSSI network interface(s)
1 ATM network interface(s)
123K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
No slave installed in slot 7.
Configuration register is 0x102
Router_B>
As you can see, the output contains a great deal of information. We’ll move through it
field by field.
The first field indicates the revision of software that is actively running on the router. In this
case, it is Cisco IOS 12.1(16).
The next field is the bootstrap version, which indicates the Cisco IOS that is used in case
the IOS isn’t found. This IOS is stored on the PROMs or flash memory of the router. The
router boots by using 11.1(8)CA. This allows the router to actually boot so that you can fix
software problems.
Current router status information is located in the field following the bootstrap information.
This output tells you the length of time the router has been up and the last date it was reloaded.
If an error caused the router to reload, the error message is included in this field. Finally, the file
that was used while booting is listed.
Directly after this section is a line that tells the type of processor used and the amount of
DRAM present. The DRAM is displayed in the format value1/value2 bytes of memory.
value1 is the amount of local memory present; value2 is the amount of I/O memory present.
The total DRAM in the router is the sum of these two values.
The final section describes the route processor and amount of RAM. At the end of the
section, all interface processors are listed, followed by the number of interfaces. The last
three lines indicate the different amounts and types of memory.
show startup-config and running-config
These two commands are used to view the syntax of the router’s configuration. The show
startup-config command displays the contents of the configuration that was written to
NVRAM. The show running-config, show config, and write term commands are all
equivalent. The results of these commands display the configuration that was loaded into
memory and is running on the router.
Although you should already be familiar with these two commands, here is a very good troubleshooting
tip: Compare the two configurations when working on network problems. It is always
possible that configuration changes were made to the running configuration but not copied to the
startup configuration. There may be extra or missing commands in the configuration versions. You
may be able to solve the problem of missing commands in the running configuration quickly by
copying the startup-config to the running-config.
These commands provide you with global, protocol, and interface information. You can
analyze them for proper configuration and then make changes, if needed. Many problems
can be isolated by viewing the configuration. What usually happens is that you will see something
that wasn’t there before, see something that shouldn’t be there, or notice that something
is missing that needs to be there. For this technique to work, you must be familiar with the
router and its configuration. If backups are made of the configurations, you can compare
them to the running-config to look for differences.
show buffers
The buffers come configured with default settings. They can be modified, if necessary, but if you
do this it’s usually a good idea to have a Cisco TAC engineer look at the memory allocation and
suggest the new buffer settings. Following is an example of the buffer settings:
Router_B>show buffers
Buffer elements:
999 in free list (500 max allowed)
2594679003 hits, 0 misses, 500 created
Public buffer pools:
Small buffers, 104 bytes (total 480, permanent 480):
455 in free list (20 min, 1000 max allowed)
243410950 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 360, permanent 360):
357 in free list (20 min, 800 max allowed)
374760214 hits, 8298 misses, 5776 trims, 5776 created
2275 failures (0 no memory)
Big buffers, 1524 bytes (total 360, permanent 360):
358 in free list (10 min, 1200 max allowed)
274949626 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 40, permanent 40):
40 in free list (5 min, 1200 max allowed)
12900991 hits, 173 misses, 519 trims, 519 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 40, permanent 40):
40 in free list (3 min, 120 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 0):
3 in free list (3 min, 52 max allowed)
2459 hits, 2 misses, 8716 trims, 8720 created
0 failures (0 no memory)
Interface buffer pools:
IPC buffers, 4096 bytes (total 312, permanent 312):
312 in free list (104 min, 1040 max allowed)
696006349 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Header pools:
You can view six buffer distinctions in this output: small, middle, big, very big, large, and
huge. Each division is allocated a particular amount of buffer space. These allocations are
determined at router bootup and vary by interface type. The show buffers output details
the buffer name and size, with the buffer size following immediately after its name. The
(total 120, permanent 120) for the small pool specifies that there are a total of 120
spaces allocated to the small pool. The permanent means that the 120 buffer spaces are permanently
assigned to the small buffer pool. When a buffer’s space is permanent, it cannot
be deallocated and given back to the system memory for other uses.
In the next field, you can see the number of free buffer spaces that are open to accepting a packet.
Each pool maintains a minimum and maximum threshold, which the pool uses to decide whether
more buffer space needs to be allocated to it. This is seen in the min and max allowed indicators.
The last two lines of information given for each pool describe the activity happening there.
This information, which includes all hits, misses, trims, created spaces, and failures, is described
in the following list:
Hits The number of times the pool was used successfully.
Misses The number of times a packet tried to find a space within a pool but found no available
spaces. In this case, the packet is not discarded; rather, a space is created for it.
Trims The number of spaces removed from the pool because the amount exceeded the number
of allowed buffer spaces. This value is only meaningful on dynamically allocated buffer pools;
static pools cannot be trimmed.
Created The number of spaces created to accommodate requests for space when there wasn’t
enough at the time the request was made or if there were fewer than the min of a certain type
of buffer available. Once the space is no longer needed, it will be trimmed.
Failures The number of times a buffer pool tried unsuccessfully to create space. When a failure
occurs, the requesting packet is dropped.
The last field is the no memory field, which records the number of failures that occurred due
to the lack of sufficient system memory required to create additional buffer space.
If you observe a significant increase in the number of misses while monitoring buffers with the
show buffers command, the pool can be tuned by assigning different values to the max-free, minfree,
and permanent parameters. Increasing the values for these parameters overrides the system
defaults—instead of having to create additional spaces on demand within a pool, the spaces can be
statically allocated and assigned. This helps you avoid racking up missed and failed packet statuses.
You can adjust these parameters with the following command:
buffers {small | middle | big | verybig | large | huge | type number}
{permanent | max-free | min-free | initial} number
The type represents interface type, and number is the number to be assigned to the specified
parameter.
Table 37.3 depicts the sizes of the buffer space within a pool. When a packet needs to be stored
in a buffer, it requests space from the pool in proportion to its size requirement. For example, a
full-size Ethernet packet at a 1500MTU requires one buffer space from the big buffer pool.
show stack
The show stack command is not very useful to you, but it is invaluable information for the
Cisco TAC. An example of output from the command appears in this section. As you can see,
it won’t make a lot of sense to the user. The information is sent to Cisco, and Cisco runs it
through a stack decode that provides the information relevant to system problems.
Stacks are used to provide information on the router’s processes and processor utilization.
The output displayed is from a healthy router. If the router were to crash, the latest
stack information is saved so it can be captured once the router comes back up. The data contains information regarding the reason for the reload and any errors that are attributed
to the crash.
Router_A#show stack
Minimum process stacks:
Free/Size Name
10288/12000 Init
5196/6000 Router Init
9672/12000 Virtual Exec
Interrupt level stacks:
Level Called Unused/Size Name
1 49917 8200/9000 Network Interrupt
2 2 8372/9000 Network Status Interrupt
3 0 9000/9000 OIR interrupt
4 0 9000/9000 PCMCIA Interrupt
5 2561 8652/9000 Console Uart
6 0 9000/9000 Error Interrupt
7 27140712 8608/9000 NMI Interrupt Handler
Router_A#
show tech-support
The show tech-support command is a compilation of several show commands (version,
running-config, controllers, stacks, interfaces, diagbus, buffers, process
memory, process cpu, context, boot, flash bootflash, ip traffic, and controllers
cbus). It should be noted that, although these are the typical commands issued by show
tech-support, the commands can vary depending on hardware and software levels. You
can get most of the information you need by issuing the show tech-support command,
instead of issuing all of the commands separately.
The show tech-support command does not allow you to scroll through its output on the router
because of the enormous amount of information that is displayed. To capture the output, you need
a terminal with a large line-buffer setting, or you can log the output directly to a terminal.
show access-lists
The show access-lists command is useful to view the access list configuration without sorting
through the running or start-up configuration. In addition to displaying the line entries of the
access list, the command uses the access list number to define what type of access list is being displayed.
The output from the show access-lists command follows:
Router_B#show access-lists
Extended IP access-list 105
permit ip 172.16.0.0 0.0.255.255 any (97160 matches)
permit ip 10.0.0.0 0.255.255.255 any
deny ip any any (102463 matches)
Novell access-list 801
permit 606E3000 (3245 matches)
permit 506E3074
permit B06F2E00 (655 matches)
permit D06F2EFE
permit 717B012C
permit E06F2E67
permit F9BE0714 (5038 matches)
permit A054AB00
permit 617B07C4
permit 017B1900
This information gives you a summary of each access list on the router. The access list type
is defined, and the number assigned to it is shown. Each line of the list is displayed individually.
The list also specifies matchups between networks and wildcard masks.
show memory
The show memory command is helpful for diagnosing memory problems such as allocation failures,
low amounts of free memory, and so on. In the following output, you can see that the first
field has the memory divided between processor memory and fast memory. The fields are selfexplanatory;
they describe the total, used, and free amounts of memory. As you will see in the
“Process Commands” section later, the output here is very similar to the show processes
memory command.
Router_C>show memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Proc 60DC38E0 52676384 34896328 17780056 15823612 14764584
Fast 60DA38E0 131072 128344 2728 27282684
Processor memory
Address Bytes Prev. Next Ref PrevF NextF Alloc PC What
60DC38E0 1056 0 60DC3D2C 1 601342A4 List Elements
60DC3D2C 2656 60DC38E0 60DC47B8 1 601342A4 List Headers
60DC47B8 9000 60DC3D2C 60DC6B0C 1 60135498 Interrupt Stack
60DC6B0C 9000 60DC47B8 60DC8E60 1 60135498 Interrupt Stack
Interface Commands
Interface commands deal with detailed interface settings and configurations. Because each
type of interface uses particular protocols and technologies, the show interface command
is capable of displaying all data related to a specified interface. Table 37.4 lists the useful
interface-related show commands. Here in this section, we will focus on the show interface
and show ip interface commands.
show queueing and show queue
To verify the configuration and operation of the queuing system, you can issue the following
two commands:
show queueing [fair | priority | custom]
show queue [interface-type interface-number]
Following are the results from these commands on Router C. Because weighted fair queuing
is the only type of queuing that has been enabled on this router, it wasn’t necessary to issue the
optional command options fair, custom, or priority.
Router_C#show queueing
Current fair queue configuration:
Interface Discard Dynamic Reserved
threshold queue count queue count
Serial0 96 256 0
Serial1 64 256 0
Current priority queue configuration:
Current custom queue configuration:
Current RED queue configuration:
Router_C#
This command output shows that weighted fair queuing is enabled on both serial interfaces,
and that the discard threshold for Serial 0 was changed from 64 to 96. There’s a maximum
of 256 dynamic queues for both interfaces—the default value. The lines following the
interface information are empty because their corresponding queuing algorithms aren’t configured
yet.
The next command, show queue, displays more detailed information pertaining to the
specified interface:
Router_C#show queue serial0
Input queue: 0/75/0 (size/max/drops); Total output drops: 0
Queueing strategy: weighted fair
TABLE 3 7 . 4 show interface Commands
show interface Command Information Produced
queuing/queue Queuing configuration and contents
interface interface-type interface-number Interface status and configuration
ip interface Information specifically related to IP
interfaces
Troubleshooting Commands 1093
Output queue: 0/1000/96/0 (size/max total/threshold/ drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Router_C#
show interface
As mentioned, the show interface command has many derivatives. Table 37.5 lists many of
the options that are available with this command.
It is important to recognize that the interface processors listed are there
because they are present on the router. For example, you won’t see a Token
Ring interface listed unless there is a Token Ring interface on the router.
Now look at sample outputs from an Ethernet and a serial interface. After each sample, we
will go through a detailed explanation:
Router_A#show interface Ethernet 5/4
Ethernet5/4 is up, line protocol is up
Hardware is cxBus Ethernet, address is 009a.822e.51b6 (bia 90.323f.acdb)
Description: Connection to Router_B
Internet address is 172.16.1.1/24
TABLE 3 7 . 5 show interface Command Options
show interface Command Option Information Produced
atm interface-type ATM interface
ethernet interface-type IEEE 802.3
serial interface-type Serial
hssi interface-type High-Speed Serial Interface (HSSI)
accounting Interface accounting
fair-queue Interface Weighted Fair Queueing (WFQ) info
rate-limit Interface rate-limit info
mac-accounting Interface MAC accounting info
1094 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/
255, load 33/255
Encapsulation ARPA, loopback not set, keepalive set (10
sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 101553 drops; input queue 0/75, 1327
drops
5 minute input rate 247000 bits/sec, 196 packets/sec
5 minute output rate 1329000 bits/sec, 333 packets/sec
421895792 packets input, 2524672293 bytes, 1 no
buffer
Received 453382 broadcasts, 0 runts, 0 giants
6 input errors, 1 CRC, 5 frame, 0 overrun, 494
ignored, 0 abort
0 input packets with dribble condition detected
618578101 packets output, 977287695 bytes, 0
underruns
0 output errors, 30979588 collisions, 1 interface
resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffers copied, 0 interrupts, 0 failures
Router_A#
This output starts with the most pertinent information—the physical interface and line protocol
status. In this case, both are up. There is much argument as to what constitutes an “up”
interface. It is very simple—the controller sends a signal that there are electrons flowing through
the physical interface. So, just doing a no shut on an interface brings it into an “up” status,
even if nothing is plugged into the interface. Line protocol is up means that the interface
is able to send itself a frame and receive it back.
The next fields contain the layer 2 MAC address, the interface description, and the layer 3
IP address. Below the interface address information, you’ll find the line settings for the interface;
MTU, bandwidth, delay, reliability, and load are listed. These values are used to calculate
a distance-vector protocol route metric.
Default Ethernet encapsulation for Cisco is ARPA. You can see that this is true and that the
keep-alive is the default at 10 seconds. This line is very important when troubleshooting Ethernet
problems. If the encapsulation type is not compatible with other machines on the network,
you will have communication problems. In order to better demonstrate this, let’s examine the
example given in the following paragraph.
Troubleshooting Commands 1095
When the router broadcasts from an interface, it uses the encapsulation that is configured.
Look at Figure 37.1. In this case, an ARPA frame (#1) is sent. If the hosts on the network do not
understand ARPA, they do not respond to the broadcast. On the other hand, if a host broadcast
uses a SNAP frame (#2), the router is designed to understand any incoming frame encapsulation
and can respond to the broadcast. Another bit of useful information that the router adds to the
ARP table is the encapsulation type of that host. Then, the next time that the router wants to speak
with the given host, it uses the documented frame type instead of the type configured on the interface.
Here’s a look at the ARP table (notice that the Type field is SNAP):
Router_C>show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.1.1 - 0010.296a.a820 ARPA Ethernet5/0
Internet 172.16.1.22 62 0010.29d1.68a0 SNAP Ethernet5/0
Router_C>
Continuing on with the output from the show interface command, you can see a great deal
of statistical information. The counters for the interface have not been cleared since the router
booted. Queuing type for the interface is first in, first out (FIFO). You should be familiar with
the next few fields, because the input and output queue were previously discussed in the sections
describing the show queueing and show queue commands. Here, you have statistical information
that displays the number of drops. The interface traffic statistics follow.
FIGURE 3 7 . 1 Ethernet frame encapsulation compatibility
Workstation 172.16.1.2
Workstation
172.16.1.1
Ethernet
ARPA
frame
(#1)
SNAP
encapsulation
(#2)
1096 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
Statistical information includes the number of packets that travel across the interface and
the bandwidth utilization. The following fields are dedicated to Ethernet troubleshooting.
The cyclic redundancy check field counts the number of frames that were received that
do not pass the CRC test. Next are frame errors and overruns. Overruns occur when the
receiver on the interface receives frames faster than it can move them to the hardware buffer
on the interface. The ignore signal is sent if there are buffer problems.
Output errors consist of underruns and collisions. The other fields are counters for the
physical interface: resets, lost carrier, and no carrier. These are followed by more
buffer error counters.
Now let’s review the output from a serial interface:
Router_D#sho int s1/0
Serial1/0 is up, line protocol is up
Hardware is cxBus Serial
Description: Connection to frame-relay cloud
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, rely 255/
255, load 1/255
Encapsulation FRAME-RELAY, loopback not set, keepalive
set (10 sec)
LMI enq sent 195167, LMI stat recvd 195165, LMI upd
recvd 10, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 1023 LMI type is CISCO frame relay DTE
Broadcast queue 0/64, broadcasts sent/dropped 0/0,
interface broadcasts 908350
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/4 (size/max/drops); Total output
drops: 22795
Queueing strategy: weighted fair
Output queue: 0/64/22795 (size/threshold/drops)
Conversations 0/59 (active/max active)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 7000 bits/sec, 9 packets/sec
5 minute output rate 9000 bits/sec, 8 packets/sec
55695166 packets input, 3680326698 bytes, 1 no buffer
Received 0 broadcasts, 0 runts, 0 giants
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored,
1 abort
56424159 packets output, 569801054 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
8656902 output buffers copied, 0 interrupts, 0
Troubleshooting Commands 1097
failures
3 carrier transitions
RTS up, CTS up, DTR up, DCD up, DSR up
Router_D#
This output has a lot of Frame Relay information that we will discuss in Chapter 39, “Troubleshooting
Serial Line and Frame Relay Connectivity.” For now, we’ll just review the fields of information
that are available by using this command. You can see that the first line is the interface status
line. The metric values are also listed. Following the Frame Relay information, you see the interface
traffic statistics. At the bottom of the output are the buffer error fields, as well as the physical interface
counters. A carrier transition is counted any time the carrier status change occurs. (We will
explore this output in Chapter 39.)
show ip interface
The show ip interface command provides information specific to the TCP/IP configuration
of the specified interface. Information regarding the interface status, IP address, subnet mask,
broadcast address, and applied access lists is all contained in the show ip interface command
output. In addition, the command also provides information on proxy ARP, which will be
explained in further detail later in this chapter; helper addresses, which are used for DHCP configurations;
the status of Network Address Translation (NAT); and many other items. The
amount of output from the show ip interface command for a particular interface is second
only to that of the show interface command. Here is an example:
Router_B#show ip interface serial 0
Serial0 is up, line protocol is up
Internet address is 172.16.30.6/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is enabled
Multicast reserved groups joined: 224.0.0.10
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP multicast fast switching is enabled
1098 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
Probe proxy name replies are disabled
Gateway Discovery is disabled
Policy routing is disabled
Network address translation is disabled
Router_B#
Process Commands
Process commands deal directly with the processes running on the router. If the standard show
processes command is issued, you get a result similar to a ps –ef executed on a Unix box. The
output details each process, including process ID number (PID), time running, and stack information.
This output is too general to be used effectively while troubleshooting, but there are
two very important process command options that can be executed to refine this output.
The two options available with the show processes command are cpu and memory, as
described in Table 37.6. Each of these options refines the processes’ output and makes it more
useful and user-friendly.
show processes cpu
The output from this command, shown later in this section, relates the router’s processes and CPU
utilization. The first line of the output displays the router’s CPU utilization over three periods. In
addition, you will notice that the CPU utilization for the five-second interval has two percentages:
15 percent and 6 percent. The first number is the average CPU utilization for all processes on
the router over the last five seconds. The second number is the percentage of the CPU spent on
interrupt-driven processes. In general, interrupt-driven tasks are core to the router’s ability to
route packets. Examples of these tasks include fast- or process-switched packets, input from the
console or auxiliary ports, and corrections of memory-alignment issues. Items such as maintaining
VTY sessions and responding to SNMP queries are non-interrupt-driven processes that would
only show up in the first percentage.
Underneath the CPU utilization line, you can see the processes running on the router. Starting
from the left, you can see the PID, followed by the runtime and other data. The three columns that
TABLE 3 7 . 6 show processes Commands
show processes Command Information Produced
cpu Amount of CPU time being spent on each process
memory Memory statistics
Troubleshooting Commands 1099
deal with CPU utilization detail the percentage of CPU cycles used by the specified process. The
process description is found in the far-right column:
Router_C>show processes cpu
CPU utilization for five seconds: 15%/6%; one minute: 7%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 76 1564143 0 0.00% 0.00% 0.00% 0 Load Meter
2 0 1 0 0.00% 0.00% 0.00% 0 LAPF Input
3 3638844 872510 4170 0.00% 0.04% 0.00% 0 Check heaps
4 4 28 142 0.00% 0.00% 0.00% 0 Pool Manager
5 0 2 0 0.00% 0.00% 0.00% 0 Timers
. . . [output removed] . . .
When the overall CPU utilization gets high, you can identify which process is using the most
CPU cycles, and then focus your attention on that process. For example, if the IP-EIGRP CPU
utilization runs high, you can determine that there is a problem within EIGRP, perhaps a routing
loop or some other instability.
show processes memory
The second helpful option for the show processes command is memory, which is used to associate
memory utilization with the router’s processes. Here is a sample output:
Router_D>show processes memory
Total: 52503792, Used: 45141524, Free: 7362268
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 54400 304 8898364 0 0 *Init*
0 0 632 3906083084 632 0 0 *Sched*
0 0 700723436 729437084 472484 1091352 0 *Dead*
1 0 96 0 6876 0 0 SSCOP Input
2 0 0 0 6780 0 0 Check heaps
3 0 17262036 152680 6916 12351248 260336 Pool Manager
. . . [output removed] . . .
The first line details the total, used, and free amounts of system memory. Following that, you
see the PID, allocated, freed, and holding memory. This means that the processor has allocated
a given amount of memory to the process; if the process does not need all of that memory, it
frees some of it and retains the rest.
TCP/IP Protocol Commands
We will discuss the major TCP/IP protocol commands in this section. In addition to the
TCP/IP-related commands listed here, other protocol-related commands are covered later in
this book. These cover protocols including HDLC, Frame Relay, X.25, and ISDN.
1100 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
Table 37.7 lists the frequently used IP options for the show command.
show ip access-list
This command provides information regarding a specified access list, or all access lists that
fall within the 1–199 range. When various access lists are configured on the router, the show
ip access-list command shows named IP access lists only. (Named access lists are explained
later in this chapter.) From the following sample output, you can see that it lists both standard
and extended lists:
Standard IP access list 5
permit 172.16.14.2
permit 172.16.91.140
permit 172.16.10.51
permit 172.16.1.7
permit 172.16.155.0, wildcard bits 0.0.0.255
Extended IP access list 152
deny ip any 172.16.91.0 0.0.0.63 log (268436 matches)
deny ip any host 172.16.91.66 log (81058 matches)
permit tcp any any established (8809 matches)
permit ip host 172.16.2.55 any
permit ip host 172.60.22.10 any (2194226 matches)
permit ip host 172.140.64.8 any (7930443 matches)
permit ip 172.16.10.0 0.0.255.255 any (9076 matches)
TABLE 3 7 . 7 Frequently Used show IP Command Options
show ip Command Option Information Produced
access-lists IP access lists
accounting The active IP accounting database
arp Information regarding the IP ARP entries in the ARP cache
interface IP interface status and configuration
protocols Information regarding the IP routing protocols running on
a router
route IP routing table
traffic IP protocol statistics
Troubleshooting Commands 1101
show ip arp
This command provides information contained in the router’s ARP cache, including the IP
address, MAC address, encapsulation type, and interface from which the MAC was learned.
Here is a sample:
Router_C#show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.60.1 - 0010.7bd9.2881 ARPA Ethernet0/1
Internet 172.16.50.2 - 0010.7bd9.2880 ARPA Ethernet0/0
Internet 172.16.50.1 6 0000.0c09.99cc ARPA Ethernet0/0
Router_C#
show ip protocols
This command provides information about the IP routing protocols that run on the router. The
sample output shown here includes only EIGRP information, because that is all that is being run
on the router. As you can see, global filters are not applied. Metric values are displayed for each
individual routing protocol. Route redistribution information is also provided:
Router_B#show ip protocols
Routing Protocol is "eigrp 100"
Outgoing update filter list for all interfaces is not
set
Incoming update filter list for all interfaces is not
set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
EIGRP maximum hopcount 100
EIGRP maximum metric variance 1
Redistributing: eigrp 100
Automatic network summarization is not in effect
Routing for Networks:
172.16.0.0
Routing Information Sources:
Gateway Distance Last Update
Distance: internal 90 external 170
Router_B#
show ip route
This command returns information stored in the IP route table. The command can be issued as
a general command, and all IP routes and corresponding information will be displayed. Additionally,
you can specify a given network, and the command will return information regarding
that network only.
1102 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
This section contains two examples of the show ip route command. Notice that the two
outputs are different. The general command provides summary information for every IP route
in the route table. However, when a network is specified, the results are much more detailed.
Items such as the exact routing protocol responsible for learning the route, the source interface,
and the next-hop router’s IP address are all included:
Router_A>show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF
inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA
external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2,
E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, *
- candidate default
U - per-user static route, o - ODR
Gateway of last resort is not set
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
D 172.16.50.0/24 [90/2195456] via 172.16.30.6, 00:00:19, Serial1
C 172.16.30.4/30 is directly connected, Serial1
Router_A>
Router_A>show ip route 172.16.50.0
Routing entry for 172.16.50.0/24
Known via "eigrp 100", distance 90, metric 2195456, type internal
Redistributing via eigrp 100
Last update from 172.16.30.6 on Serial1, 00:02:03 ago
Routing Descriptor Blocks:
* 172.16.30.6, from 172.16.30.6, 00:02:03 ago, via
Serial1
Route metric is 2195456, traffic share count is 1
Total delay is 21000 microseconds, minimum bandwidth
is 1544 Kbit
Reliability 128/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1
Router_A>
Troubleshooting Commands 1103
show ip traffic
This command returns information pertaining to IP traffic statistics. When the command is
issued, the output is organized according to the IP protocol. Here is a sample:
Router_B#show ip traffic
IP statistics:
Rcvd: 400 total, 400 local destination
0 format errors, 0 checksum errors,
0 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options,
0 with options
Opts: 0 end, 0 nop, 0 basic security,
0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert,
0 cipso
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 0 received, 0 sent
Mcast: 398 received, 401 sent
Sent: 404 generated, 0 forwarded
0 encapsulation failed, 0 no route
ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0
unreachable
0 echo, 0 echo reply, 0 mask requests, 0 mask
replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply
0 mask requests, 0 mask replies, 0 quench, 0
timestamp
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
UDP statistics:
Rcvd: 0 total, 0 checksum errors, 0 no port
Sent: 0 total, 0 forwarded broadcasts
1104 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
TCP statistics:
Rcvd: 0 total, 0 checksum errors, 0 no port
Sent: 0 total
Probe statistics:
Rcvd: 0 address requests, 0 address replies
0 proxy name requests, 0 where-is requests, 0
other
Sent: 0 address requests, 0 address replies (0 proxy)
0 proxy name replies, 0 where-is replies
EGP statistics:
Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no
listener
Sent: 0 total
IGRP statistics:
Rcvd: 0 total, 0 checksum errors
Sent: 0 total
OSPF statistics:
Rcvd: 0 total, 0 checksum errors
0 Hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
IP-IGRP2 statistics:
Rcvd: 402 total
Sent: 406 total
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0, Register Stops: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 00
DVMRP: 0/0, PIM: 0/0
Troubleshooting Commands 1105
ARP statistics:
Rcvd: 0 requests, 0 replies, 0 reverse, 0 other
Sent: 1 requests, 5 replies (0 proxy), 0 reverse
Router_B#
debug Commands
The debug commands and options are very powerful tools. The messages produced by the
debugging process give detailed information and provide insight into what is happening on a
very low level.
This power does not come free of charge. In most cases, debugging requires every packet to
be process-switched, meaning that the route processor has to look at every packet entering the
router in order for valid information to be obtained. In addition, the router must run and manage
many other processes. Debugging can cause a great deal of additional overhead on a router.
Therefore, it is important to use the tool with discretion. Use it to provide additional information
on an existing problem, not to monitor a router. As a rule of thumb, debug commands
should not be run on a router that already has a CPU utilization greater than 50 percent.
Because most problems are reported while a network is in production, the last thing you
want to do is crash a router or cause unnecessary overhead by using the debug tool. By focusing
the application of the debug command by using various command options and access lists, you
can effectively troubleshoot problems without causing additional ones.
Always remember to turn the debugging function off after you obtain the
necessary data. If left on, it can cause another network problem.
There are two tricks to successfully using the debug tool. First, make sure that your router
is configured to apply timestamps to all messages. This is done with the following commands:
Router_A(config)#service timestamps debug datetime msec localtime
Router_A(config)#service timestamps log datetime msec localtime
Next, make sure that you see these messages. By default, error and debug messages are sent
only to the console. If you are telnetted to the router, you will not see the debug or log messages
unless you issue the following command:
Router_A#terminal monitor
You can turn the messages off again by issuing the no form of the command:
Router_A#terminal no monitor
If the output messages from the debug become excessive, it becomes difficult, if not impossible,
to enter commands. Should this happen, there are two commands that you can issue to stop the
messages. The first one was already mentioned (terminal no monitor, or term no mon for
1106 Chapter 37 Cisco Diagnostic Commands and TCP/IP Troubleshooting
short). In this case, you type, but you don’t see anything echo back. It can get confusing. Remember
that the text messages that echo to the screen are not entered on the command line of the
router. You can safely type term no mon and press Enter, even with hundreds of messages scrolling
past you on the screen. The router eventually recognizes and processes the command. That
stops the messages from scrolling down the screen, but it does not stop the processor from looking
at every packet.
To stop the debug process altogether, the easiest way is to type the shorthand form of
undebug all, like this:
Router_A#un all
It is short and sweet, yet effective. It works especially well when the router seems to be having
a runaway. This command stops all debug processes and all associated messages. It can be
entered safely while messages are scrolling wildly down the screen. It may take the router a few
CPU cycles to accept the command and actually stop the debug process, so don’t panic.
As an alternative, you can also have the un all command ready to go if you allow multiple
telnet sessions to the same router. In this instance, you would telnet to the router twice. In one
of the telnet sessions, set up the terminal monitor command so that you would receive the
debug output. In the other window, type in the undebug all command but do not press Enter.
Then return to your first telnet session and execute the debug command you need. If the output
is overwhelming, go back to your other telnet session and hit Enter. As was the case before, it
may take several seconds for the router to process the command and the messages to stop
appearing on the screen.
Limiting Debug Output
Because of the potential impact to the router, you should take precautions whenever you use
debug commands. Be as specific as possible when entering the debug commands so that you look
only at information relevant to your issue. In addition to the commands themselves, you can apply
access lists to the debug commands to further limit the information you are examining.
For example, if you wanted to see ping (ICMP) packets going between stations with IP
addresses of 10.20.20.20 and 10.30.30.30, you could create an access list like this:
access-list 100 permit icmp host 10.20.20.20 host 10.30.30.30
Then apply this access list to the debug command as shown here:
Router_C#debug ip packet detail 100
IP packet debugging is on (detailed) for access list 100
Router_C#
IP: s=10.20.20.20 (Serial0), d=10.30.30.30 (Serial1), g=10.5.30.30, len
100, forward ICMP type=8, code=0
In this manner, only ICMP packets going from 10.20.20.20 to 10.30.30.30 are shown in the
debug output, rather than all of the packets going through the router.
Troubleshooting Commands 1107
As with the show commands, there are global-, interface-, and protocol-related debugging
options. Because these tools and commands are used and discussed often in upcoming chapters,
they are only summarized here according to usage.