Troubleshooting by Layer

The earlier sections of this chapter explained a general troubleshooting methodology. When going
through this methodology, it is often helpful to approach the problem in a logical manner that leverages
the OSI model. Therefore, Cisco has started backing a model of troubleshooting that does just
that. This model has three distinct approaches: bottom-up, top-down, and divide-and-conquer.
Bottom-Up Troubleshooting Approach
As the name implies, when you use the
bottom-up troubleshooting
approach, you start with the
bottom—the Physical layer of the OSI model—and work your way up to the top—the Application
layer. This approach is used when you suspect the problem is at the Physical layer, or when you
are troubleshooting a complex network problem. In these situations, ensuring that the core components
required for networking are in place can go a long way toward isolating the problem.

The downside to bottom-up troubleshooting is that it can require the checking of each interface
along the path to see if errors are occurring there. Depending on the length of the path from
the end points of the problem, this process can be very time-consuming. In these cases, determining
the most likely culprit based on the symptoms of the trouble can save a lot of time.
Top-Down Troubleshooting Approach
If you suspect that the problem lies in a piece of software, then
top-down troubleshooting
should be used. You start by testing the application and work down the OSI layers to find the
source of the problem. The challenge to this type of troubleshooting is that you need to check
all the user’s network applications in order to find the one that is causing the errors. This is a
potentially time-consuming troubleshooting method if there are a large number of applications
that could be the source of the trouble.
Divide-and-Conquer Troubleshooting Approach
The
divide-and-conquer troubleshooting
approach allows you to select the specific layer (Data
Link, Network, or Transport) of the OSI model in which to begin troubleshooting. You make
your selection based on experience with similar problems in the past, along with the specific
symptoms of the current trouble. After selecting the layer you wish to start with, the next task
is to determine the direction of the problem by determining whether the problem exists at,
above, or below this layer. Most commonly this is done by studying output from the IOS commands
on the router or through analysis of the output of network management tools. Once the
direction of the problem is determined, you continue troubleshooting through the OSI model in
that direction until you isolate the difficulty.
Often you can check the first four layers (Physical through Transport) by using
the
traceroute
command.
Summary
With the complexity of today’s networks, it is important to adhere to a troubleshooting model
to aid in efficiently and effectively isolating and resolving network problems.
Various methods of problem isolation and the troubleshooting method itself help administrators
pinpoint problem areas and foresee future trouble. Troubleshooting skills are gained
through experience. It is unreasonable to expect that you can jump in on your first network failure
and be able to solve it quickly. Experience is the best teacher. Following a problem-solving
model helps you to reach a timely solution to network failures. It helps to know your network,
but the “shooting-from-the-hip” style of troubleshooting is nowhere near as effective as a
methodical and logical process.

Using the three steps of the Cisco troubleshooting model in order is a clear, calculated, and
logical way to make a network run more smoothly. The three methods of problem isolation
(bottom-up, top-down, and divide-and-conquer) are more subjective, and it is up to each individual
to use the appropriate method for the problem that they are facing. It is important to document
changes so you have a trail of what was done on the network. Finally, it’s important to
reverse any network alterations that did not correct the problem.
Exam Essentials
Know the three steps to the Cisco troubleshooting model and the function that each step performs.
The three steps to the Cisco troubleshooting model are gather symptoms, isolate the
problem, correct the problem, and repeat if necessary. Once a problem is resolved, documentation
should be updated.
Know the troubleshooting methodologies and how to use them.
These troubleshooting
methodologies are bottom-up, top-down, and divide-and-conquer. In addition to understanding
them, know when it is most appropriate to use each method.
Be able to apply the Cisco troubleshooting methodology to example situations.
Know how
to apply each step of the troubleshooting model in real-life scenarios. You should be able to
determine what step in a troubleshooting scenario is next in the series, and understand how to
correlate a task with the correct step in the process. 1042

Looks Can Be Deceiving

One common mistake when observing the results of a change is seeing symptoms go away
and assuming that the problem has been solved. For example, assume that users are complaining
about slow response time while accessing the Internet. In the course of troubleshooting,
you find and correct some non-optimally-configured interface settings on the
router on the users’ segment. You then go back to the user who originally reported the problem.
She reports that everything is running fine now. However, she neglects to mention the
fact that there was a shift change, and now only two people are connecting to the Internet
where there used to be 50. The next day, when all of the users are back online, the problem
repeats itself. If an analysis of the observations had been done, it would have demonstrated
that the traffic flow to the Internet had dropped off and that this could be a contributing factor
to the improvement in response times.
As is demonstrated in this example, failure to analyze your observations creates the risk that
important information can be overlooked and the problem will recur. To avoid this possibility,
make sure to look at the entire scope of the problem. Use your network management tools to
help you determine whether the problem is really resolved. You can also look at your network
baseline information to find out what the “normal” traffic pattern looks like. In this example, it
should show a sharp drop-off in utilization when the shift changes. This would tell you that the
improvement in connection speed may not be due to the interface changes you’ve made, but
rather due to a lower volume of traffic. More verification may be needed.

Step 3: Correct the Problem

The investigation gave you three leads about the source of the problem. Now it’s a matter of
checking out each possibility and determining which one is most likely the source of the issue.

The majority of the possibilities point directly at the host machine, so start there. The first
two causes are host configuration issues. Now, assume that you’ve checked the TCP/IP configuration
on the host and everything is configured properly. You can eliminate the host machine
as the culprit.
You then move on to the remaining possible cause, which is an access list on the router. While
looking at the configuration on the router, you see that an access list is applied to the Ethernet
interface directly connected to the host segment. After reviewing the syntax of the access list, you
determine that it is the cause of the failure.
Great—you’ve found the problem. Now what? Once you find the problem, you must decide
what is needed to fix it. In this case, it is an access-list problem, so there are some special considerations
about how to restore functionality. You must be careful in your actions here, because that
access list may contain other entries that provide security or other network administrative functionality.
You can’t just remove the list—you could cause new problems as you fix the original one.
The best thing to do in this situation is to make a copy of the access list in a text editor,
and then make changes that are specific to your problem. When editing the access list, change
its number. After all of the changes are made in your text editor, ensure that you have a current
backup of the configuration on the router in case you need to restore the original configuration.
Then paste the modified access list back into the router. Finally, go to the interface
and apply the new access list. By following this procedure, the access list is never removed
from the interface.
Obviously, you have now changed the access-list number that is applied to the interface, so
any documentation that refers to the original number will need to be updated. If the access list
that was causing the problem was applied only to Ethernet 0, you can now safely remove
the old list, update this list with the corrections to address your problem, and put it back on the
router. Then reapply this list to Ethernet 0. As was the case before, the access list is never
removed from the interface.
When you are going through the troubleshooting methodology, it is important that you
don’t fix one problem and cause another. Before implementing any changes, think it through or
discuss it with coworkers to pick it apart, and make sure that your solution will fix the problem
without doing anything to create adverse side effects.
Another good practice when implementing changes is to change only one thing at a time, if
possible. If multiple changes must be made, it is best to make the changes in small sets. This way
it is easier to keep track of what was done, what worked, and what didn’t. Observing the effects
of a change becomes much more effective if only a single change is made at a time. There is nothing
worse than troubleshooting your self-induced errors in addition to the original difficulties!
To summarize, follow these practices and guidelines to making changes:

Make one change or a set of related changes at a time, and then observe the results.

Make non-impacting changes—this means trying not to cause other problems while implementing
the changes. The more transparent the change, the better.

Do not create security holes when changing access lists, TACACS+, RADIUS, or other
security-oriented configurations.

Most importantly, make sure you can revert to the original configuration if unforeseen problems
occur as a result of the change. Always have a backup or copy of the configuration.

In the preceding paragraphs, there were references to observing the results of the changes.
Observing results consists of using the exact same methods and commands that were used to
obtain information to gather symptoms—to see whether the changes you implemented had the
results you want. By making a change and then testing its effectiveness, you move toward the
correct solution.
It may take one or more changes to fix the problem, but you should observe each change separately
to monitor progress and to make sure that the alteration doesn’t create any adverse
effects. After the first change is made, you should be able to gather enough information to learn
whether or not the modification was effective, even if it doesn’t entirely solve the problem.

If the changes made have corrected the problem, move on and document the modifications
that were made to the network. If the changes did not work, you need to go back and either
gather more information or try another one of the potential issues that you identified while
isolating the problem.
Iterations—repetitions of certain steps within the troubleshooting model—are simply ways
of whittling away at a larger problem. By implementing changes and monitoring the results, you
can move toward solving the overall problem.

Iterations of the troubleshooting process allow you to focus with more and more detail on
the possible causes of the failure. The result of focusing on the problem is your ability to identify
more specific possibilities for the failure.
The iteration process has its own set of steps: While working through the process, you might
get more ideas of possible sources of the trouble. Write them down; if the current changes do
not work, you have notes about some other options. If you feel that you have exhausted all of
the possible causes, you should probably go back and gather more information. You will probably
find additional clues.
This is also the time to undo any changes that had adverse effects or that did not fix the problem.
Make sure to document what was done, so it will be easier to undo the any configuration
modifications.

Document the Changes
The network problem has been officially resolved after you’ve implemented a change, observed
that the symptoms have disappeared, and can successfully execute the tests that were used to aid
in gathering information about the problem. In this example, the way to verify that the problem
is solved is for Host A to try to ftp to Host Z. If this test is successful, then the problem is resolved.
In the previous sections, we have emphasized that documentation is an integral part of troubleshooting.
When you keep track of the alterations that were made, the routers, switches, or hosts that
were changed, and when the changes occurred, you have valuable information for future reference.
There is always the possibility that something you changed might have affected something else and
you didn’t notice it. If this happens, you will have documentation to refer to, so you can undo the
changes. Or if a similar problem occurs in the future, you can refer to these documents to resolve the
new problem, based on what was done the last time. Later chapters in this book will give you more
information about documentation and establishing baseline information.

Step 2: Isolate the Problem

This step within the troubleshooting model is used to contemplate the possible causes of the failure.
Obviously, it is quite easy to create a very long list of possible causes. That’s why it’s so
important to gather as much relevant information as you can in the gathering symptoms phase.
By defining the problem and assigning the corresponding boundaries, the resulting list of possible
causes diminishes because the entries in the list will be focused on the actual problem and
not on “possible” problems.
First, review what you know about your sample problem:

Host A can’t ftp to Host Z.

Host A can’t ftp to any host on Campus B.

Host A can’t ping to anywhere outside its own network.

Host A can ftp to any host on its own network.

All other hosts on Host A’s network can ftp to Host Z, as well as to other hosts.
Based on what you know, you now need to list possible causes. These possible causes are
as follows:

No default gateway is configured on Host A.

The wrong subnet mask is configured.

There is a misconfigured access list on the router connected to the switch on Campus A.
If you had not gathered such specific information in step 1, this list could have included all
possible problems with any piece of equipment between Host A and Host Z. That would have
been a long list, and it would take a lot of time to eliminate all of the possible causes.
Remember that because these are only
possible
causes, you still have to choose the most
likely option, implement it, and observe to see whether the changes made were effective. When
the list of possible problems is long, it may require more iterations of the problem-solving steps
to actually solve the problem. In this example, you have only three possible causes, so this is a
much more manageable list. Although there may be other possible causes that you can think of
(and it’s great that you can do that), for this example and in the interest of simplicity, only these
three are listed.
Here’s where it gets interesting. You now have to check each of these possibilities and fix
them if they are the cause of the problem.

Step 1: Gather Symptoms

As you can see, the user’s problem is vague; you need more information if you are to solve the
problem any time soon. This is where the first step comes in. Gathering symptoms is the step in
the troubleshooting model when details about the problem are gathered from as many sources
as is practical. These symptoms can come from a number of sources, including but not limited
to the network devices, users, monitoring tools, and console messages.
Now, while you still have the user on the line, the first step is to ask him what he means when
he says he can’t “get to” Host Z. The user then defines the situation by telling you that he can’t
ftp to Host Z. Ask the user if he experiences any other difficulties or if this is the only one. Verify
where the user is currently located. After these preliminary questions, you’ll have a basic idea of
what is and isn’t working. Unfortunately, you can’t simply assume that FTP is broken, because
there are many other pieces of the network that can contribute to this problem.
At this point, the problem is still pretty vague and needs more definition. Additional information
should include data that excludes other possibilities and helps pinpoint the actual problem.
An example in the case we’re discussing is to verify whether you can ping, traceroute, or
telnet to Host Z, thus reducing the number of possible causes.
Depending on the user and situation, you may or may not be able to get more detailed information.
It is up to you as a network engineer or administrator to solve the problem, which
means that you may have to get the information yourself.
It is important that you gain as much information as possible to actually define the problem
correctly. Without a proper and specific definition of the problem, it will be much harder to isolate
and resolve. Information that is useful for gathering symptoms is listed in Table 33.1.

TABLE 3 3 . 1
Useful Information for Gathering Symptoms
Information Example
Symptoms Can’t telnet, ftp, or get to the WWW.
Reproducibility Is this a one-time occurrence, or does it always happen?
Timeline When did it start? How long did it last? How often does it occur? Has the
current configuration ever worked properly?
Scope What are you able to access successfully via Telnet or FTP? Which WWW
sites can you reach, if any? Who else does this affect?
Baseline Info Were any recent changes made to the network configurations?

All of this information can be used to guide you to the actual problem and to create the problem
statement. Use your network topology diagram and check each item in Table 33.1. Once
you are done talking to the user, you need to define what is working and what isn’t.
Figure 33.4 is a picture of your network. Although the large X on the Frame Relay cloud represents
that there is an FTP connectivity issue, it does not indicate the location of the failure.
Right now, all you know is that a single user cannot ftp to Host Z.

Reproduce the Problem
Before spending time and effort trying to solve this problem, verify that it is still a problem.
Troubleshooting is a waste of time and resources if the problem can’t be reproduced. It’s just
like a dog chasing its tail. If the issue is intermittent, further steps should be taken to capture as
much information as possible about the event the next time it does occur. This will help narrow
down the scope of items you will look at.

Understand the Timeline
In addition to verifying whether the problem is reproducible, it is important to investigate the
frequency of the problem. For instance, maybe it happens only once or twice a day. By establishing
a timeframe you can more readily identify any possible causes. In addition, you need to
know whether this is the first time the user has attempted this function. There is a different set
of variables involved with an item that worked yesterday but not today than there is with something
that fails during first-time use. Obviously, if it worked yesterday, you can look at what
changed overnight and look for something that is broken. If the user has never used this feature
before, there may be an existing access list or other security device that has only now been activated
by the user’s initial use of this application.
Determine the Scope of a Problem
Next, you need to find out whether anyone else is unable to ftp to Host Z. If others can ftp to Host Z
(for the sake of this example, assume that they can), you can be pretty sure that the problem is
specific to the user, either on their station or on the destination host. This step determines the scope
of the problem and helps to differentiate between a user-specific problem and a more widely spread
problem. Figure 33.5 shows that other hosts can ftp to Host Z without any problems.

Cisco Troubleshooting Model

Imagine trying to solve a network failure by using a different approach every time. With today’s
complex networks, the possible scenarios would be innumerable. Because so many different
things can go wrong within a network, it’s possible to start from many different points. Not
only is this an ineffective method of troubleshooting, but it’s also time-consuming, and time is
very valuable in a “network down” situation.
Cisco has designed an effective
troubleshooting model
that contains three steps. A troubleshooting
model is a list of troubleshooting steps or processes that can be followed to provide an
efficient manner of resolving network problems. The headings in this section contain information
specific to each step of the troubleshooting model. After the three steps are completed and the
problem is resolved, a few more actions follow, such as documenting the problem-solving events.
To be effective when troubleshooting and to achieve faster resolution times, follow the model
outlined in Figure 33.2. This flow chart shows the three steps.
The troubleshooting process begins when a network failure is reported to you. The following
are brief descriptions of the steps to take:
1.
Gather symptoms.
At this point in the process, it is important gather and document the
symptoms of the problem that is being experienced.
2.
Isolate the problem.
After identifying the symptoms, the administrator looks for commonalities
in the symptoms and tries to determine at what layer of the OSI model the problem
is occurring. During this phase, it may be necessary to go back and gather more symptoms.

3.
Correct the problem.
Based on the information that was gathered and the determinations that
were made in the previous two steps, the network administrator now makes the changes necessary
to correct the problem. Once the corrective steps have been taken, the administrator
observes the results of the changes to ensure that the problem was corrected. If the problem
was not corrected, then the changes made should be backed out and the administrator should
start the troubleshooting process over with the gather symptoms stage. If the changes do correct
the problem, then the administrator should update the necessary documentation. The
final item of importance when correcting the problem is to make sure that you make only one
change at a time. This will ensure that you do not make unnecessary changes, which could
introduce new problems.
The best way to understand how Cisco’s model works and how you should use it is by looking
at an example. For this example, assume you are in charge of operational support of the network
pictured in Figure 33.3. There are two campus networks, connected via a Frame Relay
cloud. Within each network, VLANs are connected to a Catalyst 6500 switch and then to a core
router that has a connection to the Frame Relay cloud in one way or another.

The fun begins when you get a call from a user who “can’t get to Host Z.” Based on this
information, let’s apply Cisco’s troubleshooting model to solve the user’s difficulty and fix the
problem in the network.

The Complexity of Internetworks

When a network failure occurs, time is of the essence. When a production network goes down,
several things are affected. The most important of these is the bottom line—network failures
cost money.
A good example is a call-center network. The company relies on the network to be available
for its employees so that they can take phone orders, answer inquiries, or perform other
business transactions that generate income. A failure in this environment needs to be diagnosed
and repaired in a timely manner. The longer the network is down, the more money the
company loses.
To minimize monetary and productivity losses, network failures must be resolved quickly.
Troubleshooting is an integral part of getting this done. Intimate knowledge of a network also
facilitates rapid resolution. Armed with a few troubleshooting skills and intimate knowledge of
your network, you can solve most problems rather quickly, thus saving money.
Hold on a minute. What if you’re new on the job and you don’t yet have an intimate
knowledge of the network? You can probably get up to speed quickly enough, right?
Although that may have been the case in the past, getting up to speed becomes an overwhelming
challenge in today’s complex networks. These networks consist of many facets of
routing, dial-up, switching, video, WAN (ISDN, Frame Relay, ATM, and others), LAN, and
VLAN technologies.
Figure 33.1 gives you an idea of how these technologies intertwine. Notice that ATM, Frame
Relay, Token Ring, Ethernet, and FDDI all are present. Each technology has its own properties
and commands to allow for troubleshooting. Various protocols are used for each of these technologies.
In addition, different applications require specific network resources. (At least the
seven-layer OSI model, which you will review in Chapter 36, “Protocol Attributes,” is used to
maintain a common template when designing new technologies and protocols.) It would take
you a long time to master all of the technologies implemented in the network and to be able to
solve network problems based on your knowledge of the network alone. All of these factors
contribute to today’s complex network environments.
There must be an easier, more logical way to efficiently and successfully troubleshoot without
having to become intimately familiar with every network environment. Well, you’ll be
happy to know that there is an easier option—following a troubleshooting model, which is discussed
in detail in this chapter. By following a troubleshooting model, the need for intimate
knowledge of the network is reduced. A troubleshooting model should be adopted to help
resolve network malfunctions and reduce downtime.
Let’s move on to discuss Cisco’s model in detail.

Troubleshooting Methodology

Troubleshooting is a skill that takes time and experience to fully
develop. To be successful when diagnosing and repairing network
failures, a good set of troubleshooting tools and skills is essential.
The information presented here is the foundation for the rest of the information covered on the
exam. This chapter emphasizes the importance of following a specific set of troubleshooting steps
when you try to diagnose and solve network problems. An effective troubleshooting methodology
is needed because of the complexity of today’s network environments. As a Cisco Certified Network
Professional (CCNP), you need to understand and know how to apply an efficient and systematic
troubleshooting methodology. Otherwise, you would be required to have a very intimate understanding
of the network you are troubleshooting. It is imperative that you learn troubleshooting
skills and understand the information available to you while solving network problems.

Centralized Security in Remote Access Networks Exam Essentials

Understand the components of AAA. You should know that AAA is the acronym for
authentication, authorization, and accounting. Authentication is used to verify a user’s
authenticity, usually with a username and password. Authorization is used to determine
which services are available to a verified user. Accounting is used to audit the user’s activity
on the system to provide tracking.

Know the services provided by CiscoSecure. The CiscoSecure software runs on Windows NT and
Unix and provides a Java-based web client for configuration. The software provides RADIUS
and TACACS+ services for authentication, authorization, and accounting. The software can
store and retrieve user information with outside databases, including Oracle and Sybase.
Understand the functions provided by each AAA component, including the six accounting types.
In addition to the AAA functions of authorizing and authenticating a user for access to various functions
in the router, the accounting function can audit commands, connections, EXEC, network, system,
and resources.
Know how to configure AAA services for Cisco IOS. AAA has been updated since its initial
inception; the command aaa new-model is used so the user can utilize the new AAA commands.
There are many AAA commands used to configure authentication, authorization, and accounting
on a Cisco device. Each service command begins with the aaa prefix. You don’t need to
know the AAA commands for Cisco Catalyst series switches, but they are included in this chapter
for completeness.
Understand the differences between packet-mode and character-mode services. Packet-mode
services are typically dial-up connections, including asynchronous and ISDN access. Characterbased
services are connections such as login, exec, NASI, and commands. Most of these services
terminate at the access device, which is typical of character-mode services.
Know that aaa new-model requires additional commands to configure correctly. Invoking the
aaa new-model command with no other parameters will lock the administrator out of the router.
1026

Centralized Security in Remote Access Networks Summary

To have a complete security policy in place, authorization, authentication, and accounting
(AAA) must be implemented on a network. AAA not only allows full control over dial-up connections,
but login and exec access to devices. Tracking and auditing is accomplished through
the accounting services in AAA.
CiscoSecure is software that allows for centralized control over access to every device in your
network. It will run on Windows NT and Unix and provides RADIUS as well as TACACS+
authentication, authorization, and accounting services.
The two access modes, which are controlled by AAA, are character-mode and packet-mode
connections. Character-mode connections usually terminate at the access server or router, and
packet-mode connections are those that pass traffic through an access server or router.
Configuration of AAA services for Cisco devices has many facets. The administrator must
first configure how to authenticate users and then define which services those users will be
allowed to access. The optional accounting feature can be used to audit the user’s activity on
the system.
The use of a virtual template is a technology that enables the security server to supply the
access server with user-specific dialer profile information. Instead of each access server containing
user-specific dialer profile information, this information is kept on the security server and
downloaded to the access server when the user is authenticated.

Virtual Profiles

Virtual Profiles
Virtual profiles and virtual templates provide ways to apply centralized, user-specific
parameters to multiple access servers and their physical interfaces. This can greatly reduce
the impact of changes to widely distributed access points.
As suggested by the name, there is a difference between a virtual profile and the element it
replaces—the dialer profile. Dialer profiles maintain information on a single access server for
specific users. The virtual profile adds the following:
 User-specific configurations served from the AAA server
 An open methodology for defining both standards-based and vendor-specific parameters
After the user authenticates the system, a virtual template is applied to the virtual access
interface. User parameters are then obtained from the AAA server (security server) and applied
to the virtual access interface. This solution allows for better scalability and easier administration
than would be allowed with standard dialer profiles. As a result, the virtual profile is actually
a combination of the physical interface, generic information stored in a virtual template on
the access server, and user-specific parameters stored on the security server.
If you want to expand your understanding of virtual profiles and their usage,
refer to the Cisco website (www.cisco.com). 1024

Accounting Configuration

The accounting function records who did what and for how long. Because of this, it relies
upon the authentication process to provide part of the audit trail. For this reason, it is recommended
that accounts be established with easily identified usernames—typically a lastname,
first-initial configuration. This information is coupled with six accounting types, as
described in Table 32.5.

The configuration of accounting is fairly simple, but there are a few choices that should be
considered. Table 32.6 provides a subset of the more common commands. Administrators
will need to balance the desire to obtain complete accounting records against the overhead
incurred. In Table 32.6, there is a function that is being accounted for that includes commands,
connections, system events, and so on. There is a method used to account for those
functions that includes start-stop, stop-only, and wait-start, and the server type to send this
information to.

AAA Accounting Commands
Command Description
aaa accounting command
level method server
Audits all commands at a specified level by using the specified
method. (The options are start-stop, stop-only, and wait-start.) Sends
this information to the server type (TACACS+ or RADIUS) specified.
aaa accounting
connection method
server
Audits all outbound connections (including Telnet and rlogin) to the
specified server type by using the specified method.
aaa accounting exec
method server
Audits the EXEC process with the specified method to the specified
server type.
aaa accounting network
method server
Audits network service requests (including SLIP, PPP, and ARAP
requests) to the specified server type by using the specified method.

aaa accounting system
method server
Audits system-level events by using the specified method to the
specified server type. This includes reload, for example. Because
a router reload is one of the ultimate denial-of-service attacks, it
would be useful to know what user identification was used to issue
the command.
aaa accounting
function start-stop
server
Documents the start and stop of a particular type of session specified
by the function parameter to the specified server type. Audit information
is sent in the background, negating any delay for the user.
aaa accounting
function stop-only
server
Sends a stop accounting notice at the end of a user process specified
by the function parameter to the specified server type.
aaa accounting
function wait-start
server
Similar to aaa accounting start-stop, this command documents
the start of a particular type of session specified by the function
parameter to the specified server type. However, the user is not permitted
to continue until the accounting server acknowledges the log
entry. This can delay user access.
aaa accounting
function method
{tacacs+ | radius}
Enables accounting information to be sent to the TACACS+ or
RADIUS accounting server for the specified function by using the
specified method.

One area in which accounting transcends security is charge-back. If accurate start
and stop times are recorded, a company could charge users for their time on the
system to offset the cost. Internet service providers (ISPs) have long considered
this as an alternative to the flat-rate model currently found in the United States.

Authorization Commandss

Recall that authorization is the AAA process responsible for granting permission to access particular
components in the network. The administrator will need to define these permissions
based on corporate policy and user privileges. It is important to note that although a TACACS+
file was included in the previous section to illustrate authentication, the actual authorization
controls were not included.
980 Chapter 32  Centralized Security in Remote Access Networks
The commands associated with authorization include parameters for the protocols that are
to be used and the method used for authorization. These commands are used after the authentication
phase of AAA, and they are described in Table 32.4.

TABLE 3 2 . 4 AAA Authorization Commands
Command Description
aaa authorization
network method
Performs authorization security on all network services—including
SLIP, PPP, and ARAP—using the method specified by the method
parameter. The method could be TACACS+, RADIUS, local, and so on.
aaa authorization
exec method
Authorizes the EXEC process with the specified AAA method.
aaa authorization
commands level 15
method
Authorizes all EXEC commands used at the specified level (0–15) by using
the specified method. In this example, this is level 15, which is regarded
as full authorization and normally associated with enable mode.
aaa authorization
config-commands
Uses AAA authorization for configuration mode commands.
aaa authorization
reverse-access
method
Uses AAA authorization specified by the method parameter for reverse
Telnet connections.
aaa authorization
function ifauthenticated
Permits the user to use the requested function only if the user is
authenticated.
aaa authorization
function local
Uses the local database for authorization for the specified function.
This database is stored on the router’s configuration in NVRAM.
aaa authorization
function radius
Uses RADIUS for authorization of the specified function.
aaa authorization
function tacacs+
Uses TACACS+ for authorization of the specified function.

A Sample TACACS+ Configuration File

A Sample TACACS+ Configuration File
The easiest way to understand the authorization function is to examine a configuration file that
controls authorized services. Look at the following sample configuration file that controls
authorized services:
#TACACS+ V2.1 configuration file
#created 5/14/03
#edited 8/26/03
#
#If user doesn't appear in the config file user/etc/password
default authentication = file /etc/passwd
accounting file = /home1/logs/tacacs+.accounting
#Must be same as router IOS "tacacs-server key"
key = tjelkprp
#
user=netops {
member=operator
login=cleartext dilbert
}
user=rpadjen {
# Robert Padjen
default service=permit
login=cleartext yummy
}
group=operator {
name="Network Operator"
cmd=debug {
permit .*
}
cmd=write {
permit terminal
}
cmd=clear {
permit .*
}
cmd=show {
#permit show commands
permit .*
}
}
user=shayna {
# Shayna Padjen
member=operator_plus
login=cleartext flatshoe
}
group=operator_plus {
name="Network Operator Plus"
cmd=debug {
permit .*
}
cmd=write {
permit terminal

}
cmd=clear {
permit .*
}
#permit show commands
cmd=show {
permit .*
}
cmd=configure {
permit terminal
}
cmd=interface {
permit .*
}
cmd=shutdown {
permit .*
}
cmd=no {
permit shutdown
}
}
This file establishes a number of user accounts and authorization rights. The first group,
operator, is provided with basic diagnostic and administrative functions, while the operator_
plus group is enhanced with shutdown, interface, and configure commands. All commands
are available to one administrator. Note that Shayna is a member of operator_plus,
and Rob is allowed full access.
Pay particular attention to a few additional items about this specific configuration file. First,
the passwords are in cleartext, meaning that anyone with access to the server can obtain them.
Most configuration files are encrypted. Second, observe that restrictions can be quite granular
and could include functions such as Ping while blocking extended ping.
Please refer to the documentation that accompanies your server for syntax and
configuration instructions specific to your installation.

Authorization Configurationn

Authorization defines the network services that are available to an individual or group. It provides
an easy means of allowing privileged-mode (enable-mode) access while restricting the
commands that can be executed. For example, you might want to isolate most enable commands
to a single administrator or manager, while allowing operators to perform limited diagnostic
functions. More experienced operators would be granted higher levels of authorization—
for example, they might be permitted to shut down an interface. The unrestricted enable-mode
administrator would be required for additional functions.
Use care in restricting administrative rights to the router. Although this is a
helpful option when allocating rights to vendors and other parties, too restrictive
a policy will lead to the distribution of the unrestricted account information,
which can create a larger security risk.

Catalyst Switch Configuration

On the Cisco Catalyst series switch platform running Catalyst Operating System (CatOS), the
authentication commands present themselves differently, but the resulting behavior is the same.
The following configuration, like the router configuration, uses TACACS+ for login and enable
(privileged) mode:
#tacacs+
set tacacs server 10.1.98.36 primary
set tacacs server 10.1.5.36
set tacacs attempts 3
set tacacs directedrequest disable
set tacacs key tjelkprp
set tacacs timeout 5
set authentication login tacacs enable
set authentication login local enable
set authentication enable tacacs enable
set authentication enable local enable

Again, this configuration file is an excerpt from the Catalyst switch configuration file—
displayed with the show config command. There are two TACACS+ servers defined; however,
notice that one is defined as primary. On the router, the first server listed is defaulted
to primary, but the switch allows for the primary’s configuration by using the primary keyword.
Don’t be too concerned with understanding the switch configuration—the test
focuses only on the router-based commands. The configuration is provided here so readers
who have not previously experienced Catalyst commands can become familiar with them.
The remainder of this chapter focuses only on the router commands.

The switch commands in this chapter are based on version 4.5.5 of the Catalyst
code. There might be minor differences with other versions. show config or
write terminal are often used to show the configuration information.

Authentication Configuration

Authentication is configured differently on Cisco routers and switches; however, the general parameters
are similar. In broad terms, the administrator must first instruct the device to use an authentication
protocol and then provide the IP address for communications to the security server.
Router Configuration
The following is extracted from the full configuration file of the router to highlight the commands
used for AAA configuration:
aaa new-model
aaa authentication login default tacacs+ enable
aaa authentication enable default tacacs+ enable
aaa accounting exec start-stop tacacs+

tacacs-server host 10.1.98.36
tacacs-server host 10.1.5.36
tacacs-server key tjelkprp
The preceding output is an example of a typical router configuration. This output starts the
AAA service, establishes authentication services for both the login and enable processes, and
audits the start and end times of each access. The two TACACS+ servers noted here are defined,
and the preshared key is assigned.
In this example (which uses TACACS+), the aaa authentication command is used to
define the type of authentication protocol. The enable keyword at the end of the two authentication
commands allows the local enable secret password (use of the enable password would
be used if the secret is not defined, but this is not recommended from a security perspective)
to be used if network connectivity is lost between the security server and router; however, this
also can be considered a security risk. This risk is minor, considering that the attacker would
have to physically access the router or compromise the internal network sufficiently to change
routes or block packets. Here, the tacacs-server command is being used to define the IP
address of each TACACS+ server. In this example, the server key is being used to provide basic
security over the communications link to the security server. Note that this configuration
includes an aaa accounting command, which instructs the router to log the start and stop
times of an exec session to the TACACS+ server.

AAA Configuration

Although AAA was designed to centralize access control, it still requires configuration on each
and every network device. Fortunately, after AAA is configured, there are few instances when
the administrator will need to alter its configuration—for example, when the encryption key is
changed. Aside from such minor alterations, all changes—including those for user accounts—
are invoked at the security server. This configuration process lets the router or access device
know about the type of security to be used, the location of the security server, and the passwords
or other information needed to facilitate communications.
In addition to these configuration commands, the administrator must establish
network-level connectivity between the access device and the security server.
This might require access list modification or route entries.

Table 32.3 outlines some of the AAA commands, including those for authentication and
accounting. The configurations that relate to these commands are shown later in this section.

Overview of AAA Commands and Configuration
Command Description
aaa new-model Enables AAA services on the router. new-model reflects changes from
the initial implementation, which is no longer supported. In the
absence of other AAA commands, the local database will be used for
username and password. If no database is present and no other AAA
method is specified, this command will lock out the router.
aaa authentication
login default tacacs+
enable
Configures TACACS+ to be the default method used for login-level
access. If TACACS+ is unavailable, use the local enable password.
aaa authentication
enable default
tacacs+ enable
Configures TACACS+ to be the default method used for enable-level
access. If TACACS+ is unavailable, use the local enable password.
aaa accounting exec
start-stop tacacs+
Configures the accounting process, logging the start and stop times
of each exec session access.
tacacs-server host
10.1.98.36
Specifies the IP address of the TACACS+ server. The singleconnection
parameter can be used to improve performance by
maintaining a single TCP session as opposed to starting a separate
session for each authentication.
tacacs-server key
tjelkprp
Specifies the encryption key to be used for communications between
the router and TACACS+ server.

Packet-Mode Connections

Packet-mode connections
include most dial-up connections, including the following:

async

group-async

serial
 ISDN BRI
 ISDN PRI
Packet-mode connections typically secure connections that pass traffic through the network
device. You use the ppp, network, and arap AAA commands to control packet-mode connections.
Table 32.2 offers a list with explanations of these commands.
These sections do not provide a complete breakdown of all possible commands,
but instead they introduce the more common commands. Please refer
to the documentation specific to your version of the IOS for a current listing of
all commands and options or use the incorporated Help function.

Packet-Mode Authentication Commands
Command Description
aaa authentication ppp
user if-needed tacacs+
AAA is used for PPP packet-mode challenges. The list user is
used first, and if unsuccessful, TACACS+ will be used.
aaa authorization network
tacacs+ if-authenticated
TACACS+ is used to determine whether the user is permitted to
make packet-mode connections if the user is authenticated.
interface async16 ppp
authentication chap user
This is a new command for this chapter in that it associates
an AAA function with an interface. Specifically, line async16 is
instructed to use the list user for CHAP authentication. Note that
an AAA server (RADIUS, and so on) is not used.

Character-Mode Connections

Character-mode connections
describe character-based access, including access via the VTY,
TTY, AUX (auxiliary), and CON (console) ports. Although such access might be through a

packet-based network—Telnet, for example—the connection is still viewed as being character
based. The AAA commands that configure character-mode access are as follows:

login

exec

nasi

connection

arap

enable

command
Character-mode access usually includes connections only to the router or network device.
Table 32.1 includes explanations of these commands.

TABLE 3 2 . 1
Character-Mode Authentication and Authorization Commands
Command Description
aaa authentication enable
default tacacs+ enable
Uses TACACS+ to determine whether the user can access
enabled mode. If TACACS+ is unavailable, the local enable
password will be used.
aaa authorization exec
tacacs+ local
Determines whether the user is allowed access to the EXEC
shell. This example provides for TACACS+ authentication, and
should TACACS+ fail, it permits authorization via the local
database. The local database is populated with the
username
command.
aaa authorization command
n
tacacs+ local
Runs authorization for all commands at privilege level
n
(a
number between 0 and 15). Every line entered by a user can be
controlled and authorized by TACACS+, although performance
can suffer.
username
user
password
password
Creates or adds to the local database with a username of
user
and the password of password. This database is stored in the
router’s configuration file in NVRAM (nonvolatile random
access memory), and it can be accessed upon authentication
failure depending on configuration.

Router Access Modes

A Cisco router can be accessed by using one of two access modes. These are broadly categorized
as character mode and packet mode. In essence, the difference between these modes can be best
understood by looking at the commands that configure character and packet modes. You
should understand the difference in the modes and use this section as an introduction to the configuration
command syntax.

How AAA Works

It is important to remember that AAA is simply a grouping of three security functions—authentication,
authorization, and accounting. Most texts examine each component as an isolated process,
and although this is perhaps more accurate, here they have been placed into a three-step
process to better communicate the interactions between each service. For example, it is perfectly
valid to use only authentication and authorization while omitting accounting, but if you do so,
administrators will lose the auditing benefits that are provided by the auditing service.


Step 1: Authentication
Authentication
is the first facet of the three security elements, and it provides a basis for the remaining
two components. Authentication provides the “who” in the AAA model. Like journalists who
ask themselves the questions they must answer to make their story good (Who?, What?, Where?,
When?, and How?), administrators need to ask who is involved in their system; it is one of the fundamental
pieces of information they need to set up their system. Unfortunately, in computing, as in
non-computing situations, it can be fairly simple to lie about one’s identity.
To facilitate the authentication process, most systems require both a username and a
password—it is hoped the password will be maintained in confidence in order to preclude
the potential of compromise. By requiring two elements of identity, the computer-based
system doubles the likelihood that the user is accurately identified.
However, it is possible to obtain, lie about, or guess both pieces of information. The likelihood
of accurate authentication is stronger if a physical element is added. In non-computing situations,
this might include a passport or driver’s license; in the computer world, it might include a tokenbased
device. As presented in the CiscoSecure section of this chapter, there are many products that
can provide this service as a software receiver of the physical code card data.

Step 2: Authorization
After the identity of the user has been established, a decision must be made regarding what
rights that user can exercise. This is called
authorization
, and is assigned by the administrator
based on the requirements and business policies of the organization. An example of authorization
would include permissions to access a remote access device or the ability to print a file.
Because authentication and authorization are so involved and dependent on each other, they are
regarded as a single security component in most environments.
Step 3: Accounting
Whereas authentication and authorization work to prevent unauthorized access,
accounting
provides a means of verifying that only authorized users obtain access. In addition, accounting
is used to audit the actions of an authorized user.
An accounting record relies on the authenticity of the authentication process—a fraudulent
user might provide a valid login, but the accounting feature provides the audit trail required to
assess the damage. This log provides a record of when an activity occurred and what action was
performed—connecting to a router, for example.

CiscoSecure’s Response

CiscoSecure’s Response to Brute Force and Denial-of-Service Attacks
The CiscoSecure product, like other such products, has the capability to disable accounts automatically
in response to brute force attacks. This is accomplished by
intruder detection
, in which
the software assumes that the party is an intruder after a certain number of failed logins. A
brute
force attack
is one in which the attacker bombards the system with login attempts. Ultimately,
such an attack can lead to access—especially when passwords and account information are relatively
simple. By detecting such an attack, products can disable the account before it is compromised.
Frequently, such logic is limited to the number of attempts per unit of time, however. For
example, a brute force rule might allow five bad login attempts per hour before locking the
account for a day, or it might detect three bad passwords and then lock the account until the
administrator releases it.
Unfortunately, most solutions to a brute force attack lead to another type of attack: denialof-
service. A
denial-of-service attack
usually does not lead to the access of private information;
rather, as the name suggests, it prevents legitimate users from obtaining that data or
using the resource. Administrators must balance the impact of brute force compromises
against the potential of blocking access to legitimate users as a result of this protection. As
with most products, including CiscoSecure and others, the responsibility to balance access
control with access is placed on the administrator.

CiscoSecure’s Response

CiscoSecure’s Response to Brute Force and Denial-of-Service Attacks
The CiscoSecure product, like other such products, has the capability to disable accounts automatically
in response to brute force attacks. This is accomplished by
intruder detection
, in which
the software assumes that the party is an intruder after a certain number of failed logins. A
brute
force attack
is one in which the attacker bombards the system with login attempts. Ultimately,
such an attack can lead to access—especially when passwords and account information are relatively
simple. By detecting such an attack, products can disable the account before it is compromised.
Frequently, such logic is limited to the number of attempts per unit of time, however. For
example, a brute force rule might allow five bad login attempts per hour before locking the
account for a day, or it might detect three bad passwords and then lock the account until the
administrator releases it.
Unfortunately, most solutions to a brute force attack lead to another type of attack: denialof-
service. A
denial-of-service attack
usually does not lead to the access of private information;
rather, as the name suggests, it prevents legitimate users from obtaining that data or
using the resource. Administrators must balance the impact of brute force compromises
against the potential of blocking access to legitimate users as a result of this protection. As
with most products, including CiscoSecure and others, the responsibility to balance access
control with access is placed on the administrator.

Authentication, Authorization, and Accounting

Regarded as distinct elements, authentication, authorization, and accounting (AAA) all work cooperatively
to establish and enforce a security model. This model is the result of a
security policy
, which
should define an overall set of standards that will be used by the organization to secure and protect
its assets. This policy can include definitions of access rights that will be assigned to different groups
and the protocols that will be used for various functions. For example, one policy statement might
include that TACACS
+
is the sole protocol used and that SSH, a secure tool used for administration,
is preferred over Telnet.

It is important to understand how authentication, authorization, and accounting work
together to promote and support a security model. In this chapter, you will learn about how
AAA works, as well as how AAA functions in Cisco’s router access modes. AAA services are
the basic tenet of Cisco remote access solutions, and, although their presentation has been left
to the end of the book, you should find that the Physical and Network layers supplement these
concepts well. This includes physical security, the use of access lists, static or authenticated IP
routing, and other security techniques.

CiscoSecure 2

The CiscoSecure product is Cisco’s security server solution. This product incorporates many
services, including TACACS
+
and RADIUS servers, as well as logging functionality.
CiscoSecure uses web-based interfaces and Java to provide multiple administrators with
access to the server. Though the product supports both Internet Explorer and Netscape, it ships
with a Netscape FastTrack Server, and some administrators find it to be more reliable with the
Netscape client. CiscoSecure also relies on a relational database to manage accounts and store
information—currently it supports the Oracle and Sybase database platforms.
For enhanced security, administrators can choose to use
one-time challenge tokens
. These tokens
provide for the use of a different password for each login—a tactic that prevents session replay and
other techniques that would otherwise compromise security. Token cards from CRYPTOCard,
Enigma Logic, and Security Dynamics Technologies are supported with CiscoSecure.

Sequenced Data Transfer

Systems send protocol data units (PDUs) to one another, and each level of the OSI model has
its own type of PDU. Figure 36.4 shows the PDU names for all seven OSI layers. For example,
the Application layer’s PDU name is layer 7 PDU. Although this convention can be used for all
layers, some layers use other names as well. For instance, a layer 3 PDU is called a
packet
and
a layer 2 PDU is called a
frame
. When a system sends data to another system, the data has to
be fragmented so that it fits the MTU (maximum transmission unit). Therefore, several frames
may be needed to transfer the original data. Connection-oriented protocols assign a sequence
number to each outgoing and incoming PDU. This is
sequenced data transfer
.
Figure 36.5 shows you how sequencing works. There is a possibility that the destination system
will receive the PDUs out of order. If this happens, the protocol on the destination system
uses the sequence numbers to put the PDUs back into the correct order so that the original data
is obtained. 1101

Connection-Oriented Protocols

Connection-oriented protocols
contain inherent functions that control the connection as well as
data transfer. These functions are very detailed in the procedures that are followed to enable
reliable and error-free data transfer. When a source open system needs to transfer data to a destination
open system, the connection-oriented protocols actually establish a communication
pipe. The
pipe,
as it is called here, is nothing more than a logical connection between two open
systems. A great deal of information is used to establish this communication pipe, however.
In order to establish a connection, the two open systems must share certain information that
allows them to negotiate terms and finally establish a link. The information includes the common
protocol that will be used, required resources, and available resources. Look at Figure 36.3. This
figure shows the steps taken as communication is established between two open systems when
using TCP, a connection-oriented protocol.

The originating system first sends a connection request to the destination system. This
request contains information that the two systems need to agree upon before the connection can
be established. Some of the information includes the common protocol, protocol parameters,
and required resources.
Protocol parameters
are the window sizes and other possible parameters.
The
window size
is the amount of data that a station can transmit before needing an
acknowledgment from the destination system that all the data was received without error, or
that errors existed and part of the data will need to be retransmitted.
Required resources
can
include necessary bandwidth, specific port numbers, and other network resources.
The destination system receives this connection request; if it can accommodate the common
protocol, protocol attributes, and required resources, it replies with a connection accept. If, for
some reason, the destination system cannot accommodate any of the requirements sent by the
originating system, the destination system responds with a connection deny. A denied connection
can result from a blocked port on the destination system, insufficient bandwidth between
the systems, or other unavailable requested resources.
Assuming that a connection is established between the two systems, data and control information
is exchanged during the life of the connection. This data exchange can be considered a
dialog
. First, the originating system sends data until the window size is reached. That system
then waits for a response from the destination system. The destination system sends control
information that informs the originating system what needs to happen next. The transmission
can be an acknowledgment that all data in the transmission was received without error and that
the originating system can send the next batch of data. In addition, the destination system can
send a message informing the originating system that some of the data was missing, corrupted,
or had other errors that require the data to be retransmitted.
The foregoing procedure can be summarized with the description of three processes. You will
learn more about each of these processes in the following sections:
Sequenced data transfer
Each packet of a session is assigned a sequence number.
Flow control
Acknowledgments are required after a specified amount of data has been sent.
Error control
Verification of contiguous and nonerroneous packets.

Global Protocol Classifications

As mentioned, each layer of the OSI model utilizes specific protocols that enable the layer to perform
the necessary functions and communicate with adjacent layers. Each protocol has specific
properties based on the functions that it needs to accomplish. Throughout all seven layers, there
are two major protocol classifications: connection-oriented and connectionless. 1098.2.

The OSI Reference Model

This section is a review of the OSI model, which was originally discussed in
CCNA: Cisco Certified
Network Associate Study Guide, 4th ed
., by Todd Lammle (Sybex, 2004). The
OSI model
(the Open Systems Interconnection reference model) is the template used to design applications
or protocols that allow nonhomogenous computers or networks to communicate with one
another. The ISO (International Organization for Standardization) developed the OSI model.
The OSI model consists of seven layers. Each layer communicates directly with its adjacent layers,
as well as with the corresponding layer of the destination system (depicted in Figure 36.1).
Communication between layers facilitates the transfer of data up and down the OSI model. Communication
between the corresponding layers of the source system and the destination system
enables two heterogeneous networks or computers to understand each other.
The OSI template defines the services and roles that each layer is to provide. Because each
layer provides different services and functions, the layers need to communicate so that the data
can be transmitted up and down the seven layers and onto the destination system. The following
list summarizes the responsibility of each of the seven layers, starting from the Physical layer
and working up to the Application layer:
Physical
This layer sends and receives bits with values of 1s and 0s. The Physical layer is
in charge of determining how it sends these values. If the physical connection between two
machines is fiber-optic, then the Physical layer has to use light to transmit the 1s and 0s. If
the connection is electrical, then electrical signals are sent to represent the 1s and 0s.

Data Link
This layer takes all the data that is accumulated as packets are handed from one layer
to the next and then packages it into frames. The Data Link layer equates the Network layer address
(IP address) to a data link address, or MAC address, of the next hop. Once the physical address is
known, the frame is sent to that address. The receiving interface uses the Data Link layer to extract
the packet from the frame, discards the frame, and then sends the packet up to the Network layer.
Network
This layer defines the topology of the network through the use of logical addressing.
Routing protocols use this information to route packets.
Transport
This layer takes care of end-to-end communications. It is responsible for the
connection to the destination system, as well as for packet segmentation and assembly. The
Transport layer includes both connection-oriented and connectionless protocols (for example,
TCP and UDP).
Session
This layer is responsible for coordinating communication among applications, which
it does through dialog-control methods.
Presentation
This layer negotiates syntax, so it is responsible for the proper method of presenting
the data to the Application layer. Some of the Presentation layer functions are compression/
decompression and encryption/decryption of data.

Application
This is the user and application interface. The Application layer is responsible for
data exchange and job management. It also handles file, print, message, database, and application
services.
You saw how the logical data flow of the OSI model works, but look at Figure 36.2, in
which you can see the actual data flow. This figure depicts data that is handed from the Application
layer all the way down to the Physical layer. At that point, the data is transmitted
across any variety of physical media to the next hop, or destination system. Once the 1s and
0s arrive at the Physical layer of the destination system, the information is sent to layer 2 (the
Data Link layer). This layer discards the frame, and then the extracted packet is handed up
to the Network layer. The network packet header is stripped off, and the resulting packet is
handed up to the Transport layer. This process is repeated for each layer until it arrives at the
Application layer.
Now that each layer of the OSI reference model has been explained briefly, you need to
focus on the functions of each layer in detail. This detail provides the necessary background
and information to effectively troubleshoot network problems that occur within specific layers
of the OSI model.

Protocol Attributes

THE CCNP EXAM TOPICS COVERED IN THIS
BLOG INCLUDE THE FOLLOWING:

Verify network connectivity.

Use the optimal troubleshooting approach in resolving
network problems.

Minimize downtime during troubleshooting.

Use Cisco IOS commands to identify problems.

Determine the layer or layers on which a problem is occurring.

As you know, to successfully troubleshoot network problems, it is
important to have a good understanding of how network components,
including PCs and servers, communicate with each other.
Without this basic knowledge, troubleshooting a network problem is like trying to read a book in
a foreign language. The information is there, but it just isn’t comprehensible. Although the troubleshooting
model discussed in Chapter 33, “Troubleshooting Methodology,” provides the method of
retrieving all the necessary information, the data is useless without an understanding of the information
presented.
This chapter is a review of the protocols used by layers 2, 3, and 4 of the OSI model. We briefly
review the seven layers of the OSI model, and then discuss how they communicate with one another.
We then discuss layer 2 and layer 3 protocols. More specific information on some of the material
covered here can be found in later chapters and is cross-referenced here where appropriate.

End-System Documentation and Troubleshooting Exam Essentials

Know what end-system network configuration tables are and the information they contain.
End-system network configuration tables are used to record key settings of end systems in the
network. Items commonly included in an end-system network configuration table are system
name, system manufacturer/model, CPU speed, RAM, storage, system purpose, media type,
interface speed, VLAN, IP address, default gateway, subnet mask, WINS, DNS, operating system
(including version), network-based applications, high-bandwidth applications, and lowlatency
applications.

Know what end-system network topology diagrams are and the information they contain.
End-system network topology diagrams are graphical representations of the network and are
usually built with many of the same components as the end-system network configuration
tables. Some common components of the end-system network topology diagram are system
name, connection to the network, system purpose, VLAN, IP address, subnet mask, and network
applications.
Know the commands to discover information and troubleshoot end systems. There are Unix
and Windows versions of the discovery and troubleshooting commands, and many of them correlate
directly to Cisco IOS commands. Some of these commands are arp, ifconfig, ipconfig,
netstat, ping, route, telnet, and traceroute.

End-System Documentation and Troubleshooting Summary

End-system documentation is just as important as the network documentation in terms of the overall
documentation strategy. The two main components that make up end-system documentation are
the end-system network configuration table and the end-system network topology table.
End-system network configuration tables are documents that show the key configuration
parameters in place on the end systems in the network. Some of the common items in an endsystem
network configuration table are the system name, system manufacturer/model, CPU
speed, RAM, storage, system purpose, media type, interface speed, VLAN, IP address, default
gateway, subnet mask, WINS, DNS, operating system (including version), network-based applications,
high-bandwidth applications, and low-latency applications. The specific items included
on the end-system network configuration table depend on the purpose of the documentation. In
most cases, the end-system table is kept in a spreadsheet or database format. As is the case with
all the documentation covered in this book, be sure to keep hardcopies of the documents to use
in the event of a network outage.
End-system network topology diagrams are graphical representations of the end systems in
the network. In many cases, they are just additions to the network topology diagram; however,
they can be their own entity. The data included in an end-system network topology diagram is
usually a small subset of that maintained in the end-system network configuration tables. The
topology diagrams are meant to make the network administrator better able to visualize the
path across the network. Some of the standard items that go into an end-system network topology
are system name, connection to the network, system purpose, VLAN, IP address, subnet
mask, and network applications.
Finally, in this chapter we covered a number of commands that can be used to effectively
troubleshoot problems on end systems. These commands include ping and its record route
option, traceroute, arp, route, nbtstat, netstat, and ipconfig. All of these commands
have Windows NT/2000/XP and Unix equivalents, and most have a direct relationship to a
Cisco IOS command.

The nbtstat Command

As was touched on earlier, Windows systems can also use WINS (NetBIOS) to resolve names
into IP addresses. In these cases the ipconfig /displaydns command will not show these
associations. In order to view this information you need to use the nbtstat command. The
options that are available for this command are listed in the following example:
C:\>nbtstat /?
Displays protocol statistics and current TCP/IP connections using NBT
(NetBIOS over TCP/IP).
NBTSTAT [ [-a RemoteName] [-A IP address] [-c] [-n]
[-r] [-R] [-RR] [-s] [-S] [interval] ]
-a (adapter status) Lists the remote machine's name table given its name
-A (Adapter status) Lists the remote machine's name table given its IP
address.

-c (cache) Lists NBT's cache of remote [machine] names and their
IP addresses
-n (names) Lists local NetBIOS names.
-r (resolved) Lists names resolved by broadcast and via WINS
-R (Reload) Purges and reloads the remote cache name table
-S (Sessions) Lists sessions table with the destination IP addresses
-s (sessions) Lists sessions table converting destination IP
addresses to computer NETBIOS names.
-RR (ReleaseRefresh) Sends Name Release packets to WINS and then, starts
Refresh
RemoteName Remote host machine name.
IP address Dotted decimal representation of the IP address.
interval Redisplays selected statistics, pausing interval seconds between
each display. Press Ctrl+C to stop redisplaying statistics.
The two options that you will most likely use in a troubleshooting situation are the -c and
-R options. The -c option is used to display the current name resolution cache, and the -R option
is used to clear this cache. Sample output from both of these commands is displayed here:
C:\>nbtstat -c
Local Area Connection:
Node IpAddress: [10.1.1.1] Scope Id: []
NetBIOS Remote Cache Name Table
Name Type Host Address Life [sec]
------------------------------------------------------------
MICHELE <20> UNIQUE 10.2.2.2 570
NERMAL <20> UNIQUE 10.100.100.100 580
NERMAL <00> UNIQUE 10.100.100.100 575
PICASO <42> UNIQUE 10.8.8.8 415
ALEX <20> UNIQUE 10.9.9.9 582
LEAH <20> UNIQUE 10.10.10.10 492
\Device\NetBT_Tcpip_{D84EDBA9-F40F-4AFF-8409-24613C6A325B}:
C:\> nbtstat -R
Successful purge and preload of the NBT Remote Cache Name Table.

The ipconfig Command

The “Creating an End-System Network Configuration Table” section earlier in this chapter
introduced the ipconfig command used with the /all option. While this option is useful for
gathering information on a system, other options in the ipconfig command are helpful
for troubleshooting purposes.
The first of these options are the /release and /renew options, which are used to release
and renew DHCP addresses. Here are two examples:
C:\>ipconfig /release
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 0.0.0.0
Subnet Mask . . . . . . . . . . . : 0.0.0.0
Default Gateway . . . . . . . . . :

C:\WINDOWS\system32>ipconfig /renew
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 10.22.5.3
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.22.5.1
The next option useful in troubleshooting is the /displaydns option. It allows you to see the
DNS-name-to-IP-address cache that is on the workstation. The following output is from an XP
machine right after pinging www.cisco.com:
C:\>ipconfig /displaydns
Windows IP Configuration
ns1.cisco.com
----------------------------------------
Record Name . . . . . : ns1.cisco.com
Record Type . . . . . : 1
Time To Live . . . . : 86227
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 128.107.241.185
1.0.0.127.in-addr.arpa
----------------------------------------
Record Name . . . . . : 1.0.0.127.in-addr.arpa.
Record Type . . . . . : 12
Time To Live . . . . : 0
Data Length . . . . . : 4
Section . . . . . . . : Answer
PTR Record . . . . . : localhost

ns2.cisco.com
----------------------------------------
Record Name . . . . . : ns2.cisco.com
Record Type . . . . . : 1
Time To Live . . . . : 86227
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 192.135.250.69
www.cisco.com
----------------------------------------
Record Name . . . . . : www.cisco.com
Record Type . . . . . : 1
Time To Live . . . . : 86227
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 198.133.219.25
Record Name . . . . . : ns1.cisco.com
Record Type . . . . . : 1
Time To Live . . . . : 86227
Data Length . . . . . : 4
Section . . . . . . . : Additional
A (Host) Record . . . : 128.107.241.185
Record Name . . . . . : ns2.cisco.com
Record Type . . . . . : 1
Time To Live . . . . : 86227
Data Length . . . . . : 4
Section . . . . . . . : Additional
A (Host) Record . . . : 192.135.250.69
localhost
----------------------------------------
Record Name . . . . . : localhost
Record Type . . . . . : 1

Time To Live . . . . : 0
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 127.0.0.1
In addition to the local DNS cache on the machine, Internet Explorer (IE) keeps its
own name resolution cache. By default, names are cached in Internet Explorer
versions 4.0 or higher for 30 minutes, and for 24 hours in IE versions below 4.0.
Shutting down Internet Explorer and restarting it will refresh this cache.
The final option for the ipconfig command that we will discuss is /flushdns, which is complementary
to the /displaydns option. The /flushdns option clears out all entries in the DNS
cache on the workstation. This works out well for troubleshooting stale DNS entries. The output
of the command is shown here:
C:\>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.
C:\>

The netstat Command

The netstat command is used to display current connections to the end system. This can be useful
in a troubleshooting scenario to assist in the verification of connectivity. In addition to the IP
addresses of the connections, the netstat command also shows the port the connections are
using. The Windows NT/2000/XP options and sample output of the command are shown here:
C:\>netstat /?
Displays protocol statistics and current TCP/IP network connections.
NETSTAT [-a] [-e] [-n] [-o] [-s] [-p proto] [-r] [interval]
-a Displays all connections and listening ports.
-e Displays Ethernet statistics. This may be combined with the -s
option.
-n Displays addresses and port numbers in numerical form.
-o Displays the owning process ID associated with each connection.
-p proto Shows connections for the protocol specified by proto; proto
may be any of: TCP, UDP, TCPv6, or UDPv6. If used with the -s
option to display per-protocol statistics, proto may be any of:
IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, or UDPv6.
-r Displays the routing table.
-s Displays per-protocol statistics. By default, statistics
are shown for IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, and
UDPv6; the -p option may be used to specify a subset of the
default.
interval Redisplays selected statistics, pausing interval seconds
between each display. Press CTRL+C to stop redisplaying
statistics. If omitted, netstat will print the current
configuration information once.
C:\>netstat -n
Active Connections
Proto Local Address Foreign Address State
TCP 10.12.1.11:3718 10.215.198.192:80 ESTABLISHED
TCP 10.12.1.11:3719 10.215.198.153:80 ESTABLISHED
TCP 10.12.1.11:3722 10.215.198.6:80 ESTABLISHED
TCP 10.12.1.11:3724 10.12.1.100:139 ESTABLISHED
TCP 10.12.1.11:3726 10.255.37.1:23 ESTABLISHED
The Unix version of the command is very similar to the Windows version. Its options and
sample output are as follows:
unix1% netstat -help
usage: netstat [-adgimnprsDMv] [-I interface] [interval]
unix1% netstat -n
TCP
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
----------------- ----------------- ----- ------ ----- ------ ------
10.4.132.58.32891 10.4.132.58.162 57344 0 57344 0 ESTAB
10.4.132.58.162 10.4.132.58.32891 57344 0 57344 0 ESTAB
10.4.132.58.53074 10.4.128.10.1960 24820 0 8760 0 ESTAB
10.4.132.58.38090 10.104.108.13.1960 62780 0 8760 0 ESTAB
...
...

The route Command

If an end station has multiple interfaces, it can be useful to know which of these interfaces is
being used for particular destinations. In theses cases, for both Windows NT/2000/XP stations
and Unix stations, you can use the route command. The following are the options and the syntax
for displaying the routing table for Windows NT/2000/XP:
C:\>route /?
Manipulates network routing tables.
ROUTE [-f] [-p] [command [destination] [MASK netmask] [gateway]
[METRIC metric] [IF interface]
-f Clears the routing tables of all gateway entries. If this is
used in conjunction with one of the commands, the tables are
cleared prior to running the command.
-p When used with the ADD command, makes a route persistent across
boots of the system. By default, routes are not preserved when
the system is restarted. Ignored for all other commands, which
always affect the appropriate persistent routes. This option
is not supported in Windows 95.
command One of these:
PRINT Prints a route
ADD Adds a route
DELETE Deletes a route
CHANGE Modifies an existing route
destination Specifies the host.
MASK Specifies that the next parameter is the 'netmask' value.
netmask Specifies a subnet mask value for this route entry. If not
specified, it defaults to 255.255.255.255.
gateway Specifies gateway.
interface The interface number for the specified route.
METRIC Specifies the metric, ie. cost for the destination.
All symbolic names used for destination are looked up in the network Database
file NETWORKS. The symbolic names for gateway are looked up in the host name
database file HOSTS.
If the command is PRINT or DELETE. Destination or gateway can be a wildcard,
(wildcard is specified as a star '*'), or the gateway argument may be omitted.
If Dest contains a * or ?, it is treated as a shell pattern, and only matching
destination routes are printed. The '*' matches any string, and '?' matches
any one char. Examples: 157.*.1, 157.*, 127.*, *224*.
Diagnostic Notes:
Invalid MASK generates an error, that is when (DEST & MASK) !=
DEST.
Example> route ADD 157.0.0.0 MASK 155.0.0.0 157.55.80.1 IF 1
The route addition failed: The specified mask parameter is
invalid.
(Destination & Mask) != Destination.
Examples:
> route PRINT
> route ADD 157.0.0.0 MASK 255.0.0.0 157.55.80.1 METRIC 3 IF 2
destination^ ^mask ^gateway metric^ ^
Interface^
If IF is not given, it tries to find the best interface for a
given gateway.
> route PRINT
> route PRINT 157* .... Only prints those matching 157*
> route CHANGE 157.0.0.0 MASK 255.0.0.0 157.55.80.5 METRIC 2 IF 2
CHANGE is used to modify gateway and/or metric only.
> route PRINT
> route DELETE 157.0.0.0
> route PRINT
C:\>route print
=======================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ... 00 04 f2 cd 65 1f...... NVIDIA nForce MCP Networking Adapter -
➥Packet Scheduler Miniport
=======================================================================
=======================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.12.1.1 10.12.1.11 20
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
10.12.1.0 255.255.255.0 10.12.1.11 10.12.1.11 20
10.12.1.11 255.255.255.255 127.0.0.1 127.0.0.1 20
10.12.1.255 255.255.255.255 10.12.1.11 10.12.1.11 20
224.0.0.0 240.0.0.0 10.12.1.11 10.12.1.11 20
255.255.255.255 255.255.255.255 10.12.1.11 10.12.1.11 1
Default Gateway: 10.12.1.1
=======================================================================
Persistent Routes:
None
For the Unix side of things, the options and sample printout are as follows:
unix1% route
usage: route [ -fnqv ] cmd [[ - ] args ]
unix1% route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.12.1.0 0.0.0.0 255.255.255.0 U 0 0 0 hme0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 10.12.1.1 0.0.0.0 UG 0 0 0 hme0

In addition to printing out the routing table, the route command can also be used to add or
delete static routes if they are needed.

The arp Command

Although the arp command was covered in the earlier discovery section, it is being repeated
here because it can be a very meaningful part of the troubleshooting process. As is the case on
the routers, sometimes it is necessary to verify that the layer-2-to-layer-3 translation is working
as expected on the end system. In both Unix and Windows NT/2000/XP systems, the command
to display this information is arp -a. The command options and sample output from an XP box
are as follows:
C:\>arp /?
Displays and modifies the IP-to-Physical address translation tables
used by address resolution protocol (ARP).
ARP -s inet_addr eth_addr [if_addr]
ARP -d inet_addr [if_addr]
ARP -a [inet_addr] [-N if_addr]
-a Displays current ARP entries by interrogating the current
protocol data. If inet_addr is specified, the IP and Physical
addresses for only the specified computer are displayed. If
more than one network interface uses ARP, entries for each ARP
table are displayed.
-g Same as -a.
inet_addr Specifies an internet address.
-N if_addr Displays the ARP entries for the network interface specified
by if_addr.
-d Deletes the host specified by inet_addr. inet_addr may be
wildcarded with * to delete all hosts.
-s Adds the host and associates the Internet address inet_addr
with the Physical address eth_addr. The Physical address
address is given as 6 hexadecimal bytes separated by hyphens.
The entry is permanent.
eth_addr Specifies a physical address.
if_addr If present, this specifies the Internet address of the
interface whose address translation table should be modified.
If not present, the first applicable interface will be used.
Example:
> arp -s 157.55.85.212 00-aa-00-62-c6-09 .... Adds a static entry.
> arp -a .... Displays the arp table.
C:\>arp -a
Interface: 10.12.1.11 --- 0x2
Internet Address Physical Address Type
10.12.1.1 00-06-5a-23-06-f9 dynamic
Similar output from a Unix machine is shown next.
unix1% arp
Usage: arp hostname
arp -a
arp -d hostname
arp -s hostname ether_addr [temp] [pub] [trail]
arp -f filename
unix1% arp -a
Net to Media Table
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- ----- ---------------
hme0 10.12.1.1 255.255.255.255 00:06:5a:23:06:f9
hme0 10.12.1.68 255.255.255.255 00:04:f2:cd:65:1f
hme0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00
In addition to displaying information about the translation from layer 2 to layer 3, the arp
command can also be used to add and delete entries to the ARP table.