All Cisco-Network Study Notes: 11/17/08

güncel

arastirmaya devam ediyoruz
en kapsamlı icerikleri bu siteden bulabileceksiniz..

Overview of IDSM-2 Blade on the Switch

Overview of IDSM-2 Blade on the Switch
The IDSM-2 blade provides up to 600 Mbps of Intrusion Detection and Intrusion Prevention services. Just like IDS/IPS Appliance, IDSM-2 module can operate in either Promiscuous or Inline Mode (see Chapter 14 for more details). The only difference between the IDS/IPS Appliance and the IDSM-2 blade is that in version 5.0, IDS/IPS Appliance can operate in both the Inline and the Promiscuous modes, but the IDSM-2 blade can operate either one or the other. This is because the IDSM-2 module has only two sniffing interfaces, so it cannot be configured for both Inline and Promiscuous mode; it requires at least three interfaces to configure both Promiscuous and Inline mode. For the same reason, you cannot configure multiple Inline pairs on IDSM-2, which is possible on IDS/IPS Appliance. This section covers the following items that are used for the seamless operation of IDSM-2 blade:

Software and hardware requirements

Slot assignment on the switch

Front Panel indicator lights and how to use them

Installing an IDSM-2 blade on the switch

Removing an IDSM-2 blade from the switch

Ports supported on an IDSM-2 blade

Software and Hardware Requirements
There are specific hardware and software requirements on the Catalyst switch for IDSM-2 support, as follows:

Catalyst software release 7.5(1) or later with Supervisor Engine 1A with Multilayer Switch Feature Card 2 (MSFC2)

Catalyst software release 7.5(1) or later with Supervisor Engine 2 with MSFC2 or PFC2

Cisco IOS software release 12.2(14)SY with Supervisor Engine 2 with MSFC2

Cisco IOS software release 12.1(19)E or later with Supervisor Engine 2 with MSFC2

Cisco IOS software release 12.1(19)E1 or later with Supervisor Engine 1A with MSFC2

Cisco IOS software release 12.2(14)SX1 with Supervisor Engine 720

Cisco IDS/IPS software release 4.0 or later

Any Catalyst 6500 series switch chassis or 7600 router

You can use "Software Advisor" from the following link to find out the supported versions of the Catalyst 6500 switches:

http://www.cisco.com/kobayashi/support/tac/tools.shtml#software

Slot Assignment on the Switch
Before you select a slot on the switch, you must understand some of the restrictions for assigning the appropriate slot for the IDSM-2 module on the switch. The following list will help in determining how to select the appropriate slot for the IDSM-2 blade:

Catalyst 6503 A 3-slot chassis that reserves one slot for the Supervisor and offers two line card slots for other modules. The first slot is reserved for either the Supervisor 2 or Supervisor 720.

Catalyst 6006 and 6506 A 6-slot chassis that reserves one slot for the Supervisor and offers five line card slots for other modules. If a Supervisor 2 is used, then Slot 1 is reserved for its use. If a Supervisor 720 is used, then either slot 5 or 6 must be used for this Supervisor.

Catalyst 6509 A 9-slot chassis that reserves one slot for the Supervisor and offers eight line card slots for other modules. If a Supervisor 2 is used, then Slot 1 is reserved for its use. If a Supervisor 720 is used, then either slot 5 or 6 must be used for this Supervisor.

Catalyst 6509-NEBS-A A 9-slot chassis that reserves one slot for the Supervisor and offers eight line card slots for other modules. If a Supervisor 2 is used, then Slot 1 is reserved for its use. If a Supervisor 720 is used, then either slot 5 or 6 must be used for this Supervisor.

Catalyst 6513 A 13-slot chassis that reserves one slot for the Supervisor and offers 12 line card slots for other modules. If a Supervisor 2 is used, then Slot 1 is reserved for its use. If a Supervisor 720 is used, then either slot 7 or 8 must be used for this Supervisor.

The IDSM-2 can reside in any slot that is not reserved for the Supervisor. The IDSM-2 has been tested to work with the Supervisor 1A (with MSFC2), the Supervisor 2, and the Supervisor 720.

Note

Cisco supports only eight IDSM-2s per chassis.

Front Panel Indicator Lights and How to Use Them
The IDSM-2 has a status indicator and a Shutdown button. Locating and understanding the meaning of different light colors is very important. Table 15-1 explains them.

Table 15-1. IDSM-2 States as Indicated by the Status Indicator Color
Description

Green
All diagnostics tests pass, so IDSM-2 is operational.

Red
A diagnostic other than an individual port test failed.

Amber
The IDSM-2 is running through its boot and self-test diagnostics sequence, or the IDSM-2 is disabled, or the IDSM-2 is in the shutdown state.

Off
The IDSM-2 power is off.

Installing the IDSM-2 Blade on the Switch
All Catalyst 6500 series switches support hot swapping, which lets you install, remove, replace, and rearrange modules without turning off the system power. When the system detects that a module has been installed or removed, it runs diagnostic and discovery routines, acknowledges the presence or absence of the module, and resumes system operation with no operator intervention.

To install the IDSM-2 in the Catalyst 6500 series switch, follow these steps:

Step 1. Choose a slot on the Catalyst 6500 switch for the IDSM-2 blade. Note that the supervisor engine must be installed in slot 1; a redundant supervisor engine can be installed in slot 2. If a redundant supervisor engine is not required, slots 2 through 9 (slots 2 through 6 on the 6-slot chassis and slots 2 through 11 on the 13-slot chassis) are available for modules.

Step 2. Hold the IDSM-2 with one hand, and place your other hand under the IDSM-2 carrier to support it.

Step 3. Verify that you have correctly installed the IDSM-2 and can bring it online by executing the command show module on the switch.

Removing the IDSM-2 Blade from the Switch
You must first shut down the IDSM-2 before removing it from a Catalyst 6500 series switch. Removing the module without going through a shutdown may damage your module, or corrupt your IDS/IPS Operating System on the module. Work through the following procedure to remove the IDSM-2 blade from the Catalyst 6500 series switch:

Step 1. Shut down the IDSM-2 blade by one of the following methods:

Log in to the IDSM-2 and execute the reset powerdown command to shut down IDSM-2 blade.

To shut down the module from CatOS, execute the set module shutdown module_number command.

If you are running Native IOS on the switch and want to shut down the IDSM-2 module, execute hw-module module module_number shutdown command.

As a last resort, you can press the Shutdown button on the IDSM-2 blade itself. Shutdown may take several minutes.

Step 2. Verify that the IDSM-2 blade shuts down by executing the show module command on the switch. Do not remove the IDSM-2 until the status indicator is amber or off.

Step 3. Carefully pull the IDSM-2 straight out of the slot, keeping your other hand under the carrier to guide it.

Ports Supported on IDSM-2 Blade
Eight ports appear to the switch for each IDSM-2 blade. This can be verified by executing command show port IDSM_Module_Slot_Number in CatOS. Example 15-1 shows eight ports on the module on CatOS:

Example 15-1. Ports Appearing on the Switch Running CatOS
Console> (enable) show port 2
* = Configured MAC Address
Port Name Status Vlan Duplex Speed Type
----- -------------------- ---------- ---------- ------ ----- ------------
! Port 2/1 is the TCP reset interface. This interface should be in the same VLAN as the
! sniffing interface
2/1 connected trunk full 1000 Intrusion Detection
! Port 2/2 is the Command and Control Interface
2/2 connected 251 full 1000 Intrusion Detection
2/3 disable 1 full 1000 Intrusion Detection
2/4 disable 1 full 1000 Intrusion Detection
2/5 disable 1 full 1000 Intrusion Detection
2/6 disable 1 full 1000 Intrusion Detection

! Ports 2/7 and 2/8 are the Sniffing Interfaces
2/7 monitor trunk full 1000 Intrusion Detection
2/8 connected trunk full 1000 Intrusion Detection
Console> (enable)

The IDSM-2 module uses the following four IP ports:

Command and control port IDSM-2 uses GigabitEthernet0/2 interface as the command and control port that contains the IP address. If you are running CatOS, this is identified as port slot/2. In Example 15-1, 2/2 is the command and control port for the blade seated in slot 2. If you are running Native IOS, this port is identified as a management port, which is internally mapped to slot/2 port. Hence, on the 6K, set this port to a VLAN appropriate to the IP address you give the sensor. Try to avoid using VLAN 1, which is the default.

Capture/Sniffing Ports The module has two sniffing ports that are seen by the switch as ports 7 and 8. In Example 15-1, these ports are 2/7 and 2/8. If you are running Native IOS, these ports are identified as data-port 1 and 2. Data-port 1 is internally mapped to slot/7, and data-port 2 is mapped to slot/8 on the Native IOS switch. On the IDSM-2 blade, these ports are identified as GigabitEthernet0/7 and GigabitEthernet0/8 respectively. If you are using Inline mode, these two ports should be in two different VLANs. In Promiscuous mode, you can use either Switched Port Analyzer (SPAN) or Virtual Access Control List (VACL) to direct traffic to these ports. As the ports are Gigabit ports, and IDSM-2 blade cannot handle more than 600 mbps traffic, it is not necessary to use both ports in Promiscuous mode.

Reset Port In CatOS, port slot/1 is used for reset. Remember that the reset port must be assigned to the same VLAN as the sniffing port(s) to perform the TCP resets.

Troubleshooting Cisco IDS Network Module (NM-CIDS)

Chapter 16. Troubleshooting Cisco IDS Network Module (NM-CIDS)
Intrusion Prevention Systems (IPS) on routers come in two flavors: integrated IPS features, and external network modules called NM-CIDS. As the NM-CIDS uses the same code base as IPS Sensor, all the troubleshooting techniques pertaining to Sensor discussed in Chapter 14, "Troubleshooting Cisco Intrusion Prevention System," are applied here with some minor exceptions (for example, the inline feature of IPS that is supported on IPS Sensor is not supported on NM-CIDS). Hence, this chapter does not repeat the troubleshooting information on IPS operations that are performed on NM-CIDS. Instead the chapter focuses on configuration and troubleshooting of the Cisco IOS Router and NM-CIDS configuration issues. The chapter concludes with Best Practices specifically for NM-CIDS.

Troubleshooting Cisco Intrusion Prevention System

Chapter 14. Troubleshooting Cisco Intrusion Prevention System
Cisco Secure Intrusion Prevention System (IPS) comprises three main components: first, the IPS Sensor, which is the sniffing or inline component of Cisco Intrusion Prevention System; second, a management section of IPS (IPS MC, for example), which will be discussed in Chapter 18, "Troubleshooting IDM and IDS/IPS Management Console (IDS/IPS MC);" and third, a reporting tool (for example, a Security Monitor, which is discussed in Chapter 22, "Troubleshooting IEV and Security Monitors"). This chapter focuses on how to troubleshoot issues with IPS software on the sensor. As IPS software implementation is the same on all platforms (Sensor Appliance, IDSM-2 blade (IDSM stands for intrusion detection system module), NM (Network Module)-CIDS blade, and ASA-SSM (Security Services Module)), troubleshooting software issues on IPS software are the same across these platforms.

First, the chapter discusses the building blocks of IPS and some high-level details of IPS operation, and then it discusses troubleshooting tools and techniques. Once you understand the level of detail required to use the tools and apply techniques, you will see the main problem area categories and how to troubleshoot them efficiently and quickly. The chapter finishes with a detailed case study on different capturing techniques on various switches that are needed for IPS to function correctly, and concludes with a section on Best Practices.

Overview of IPS Sensor Software

Overview of IPS Sensor Software
Troubleshooting IPS issues demands that you understand the underlying architecture of IPS Software. This not only helps you feel comfortable with the product, but helps you to be a very efficient and confident troubleshooter, qualities that can distinguish you from others. To begin becoming an effective troubleshooter, you need to start with the software building blocks, which are the foundation of IPS software. Sections that follow discuss the following Sensor software topics:

IPS deployment architecture

IPS sensor software building blocks

Communication protocols

Modes of sensor operation

Hardware and interfaces supported

IPS Deployment Architecture
As mentioned before, there are primarily three components of an IPS deployment for a single sensor as depicted in Figure 14-1.

Figure 14-1. A Typical IPS Deployment in the Network

[View full size image]

Typically more than one sensor is managed by a single IPS MC and by the Security Monitor. The three main components of an IPS deployment are as follows:

Sensor A sensor can be an Appliance (IDS 42xx series), IDSM-2 blade, NM-CIDS module, or ASA-SSM module. You can deploy the sensor in Inline mode or Promiscuous mode. In Inline mode, the traffic is redirected through the sensor, and in Promiscuous mode, the traffic is sniffed to the sensor. The sensor will be discussed thoroughly in this chapter.

Management Console A management console for the sensor is used to configure and deploy the sensor software to the sensor. IPS Device Manager (IDM), an integral part of the sensor software, can be used to configure and upgrade the software on the sensor. If you want to manage more than one sensor, you can use the IPS MC, which is discussed thoroughly in Chapter 18. IPS MC communicates with sensor using SSH and SSL. More details on the communication protocol used between IPS MC (or IDM) and the sensor are discussed in the "Communication Protocol" section.

Monitoring Device A monitoring device is used to pull the events from the sensor, which then displays the events in real time. You can also correlate events, and generate static reports as you need. Several monitoring devices support pulling events from the sensor. Security Monitor is one such monitoring device, which is discussed in detail in Chapter 22, "Troubleshooting IEV and Security Monitors."

Although it's extremely important to understand all three components of the IPS Architecture, the sensor is the core of everything. To understand the sensor in depth, you need to comprehend the IPS Software building blocks.

IPS Software Building Blocks
There are several interoperating applicationsprocesses that go into making IPS software on the Linux platform. These applications and processes are described as follows:

MainApp

AnalysisEngine

Command Line Interface (CLI)

The applications that make up the sensor can be verified by using the show version output.

MainApp
This application initializes the sensor, starts and stops other applications, and performs upgrades. Following are some of the components of the MainApp application:

ctlTransSource (Control Transaction server) This allows sensors to communicate control transactions with each other. Sensor currently provides Network Access Controller master blocking sensor capability with this application.

Event Store This is an indexed store used to store IPS events (error, status, and alert system messages) that is accessible through the CLI, IDM, Adaptive Security Device Manager (ASDM), or Remote Data Exchange Protocol (RDEP).

InterfaceApp This handles bypass and physical settings and defines paired interfaces. Physical settings are speed, duplex, and administrative state.

LogApp This component writes all the application's log messages to the log file and the application's error messages to the Event Store.

Network Access Controller (NAC) This manages remote network devices (firewalls, routers, and switches) to provide blocking capabilities when an alert event has occurred. Network Access Controller creates and applies access control lists (ACLs) on the controlled network device or uses the shun command on the firewalls (PIX and FWSM).

NotificationApp Sends Simple Network Management Protocol (SNMP) traps when triggered by alert, status, and error events. NotificationApp uses the public domain SNMP agent. SNMP GETs provide information about the general health of the sensor.

Web Server This application is the sensor's web server, which is capable of both HTTP and HTTPS communications. The web server uses several servlets (idm, event-server, transaction-server and iplog-server) to perform their respected tasks as follows:

- idm (Intrusion Detection Device Manager) servlet provides the IDM web-based management interface.

- event-server servlet is used to serve events to external management applications such as Security Monitor.

- transaction-server allows external management applications such as the IDS MC to initiate control transactions with the sensor. Control transactions are used to configure and control sensors.

- iplog-server is used to serve IP logs to external systems such as Security Monitor.

AuthenticationApp This application configures and manages authentication on the sensor. It verifies that users are authorized to perform Command Line Interface (CLI), IDM, ASDM, or RDEP actions.

AnalysisEngine
This application is the actual sensing engine. The AnalysisEngine performs packet capture, processes the signature and alarm channel configurations, and generates alert events based on its configuration and the IP traffic. The AnalysisEngine stores its events in the Event Store.

CLI
This application is the calling line ID (CLI) shell application that is started when you log in to the sensor through Telnet or SSH. All accounts created through the CLI will use the CLI as their shell (except the service accountonly one service account is allowed). Which CLI commands are allowed depends on the privilege of the user.

Communication Protocols
There are several communication protocols used on the sensor. These protocols enable Sensor's processes to communicate with each other and external management devices to communicate with sensors. Following are three types of communication methods:

Inter-process Communication Within Sensor, all applications use an interprocess communication application programming interface (API) called IDAPI to handle internal communications and to interact with each otherfor example, after AnalysisEngine captures and analyzes the network traffic on its interfaces. When a signature is matched, AnalysisEngine generates an alert, which is stored in the Event Store. The communication between the AnalysisEngine and the Event Store uses IDAPI.

External Communication with Management Console Management Console, such as IPS MC, communicates with Sensor using SSH protocol when it initially pulls the sensor's configuration. Configuration and Sensor software upgrade is pushed by IPS MC using the RDEP2 or Cisco Intrusion Detection Event Exchange (CIDEE), which uses HTTP over SSL. RDEP2 is discussed in detail in this section. IDM, which is an integral part of the IPS software, uses HTTPS (HTTP over SSL).

External communication with Reporting devices An IPS Reporting tool such as Security Monitor communicates with sensors using a protocol called RDEP2 or CIDEE. RDEP2 uses the industry standards HTTP, Transaction Layer Security (TLS) and Secure Sockets Layer (SSL) and Extensible Markup Language (XML) to provide a standardized interface between RDEP2 agents (between the Security Monitor as client and the sensor as Server). CIDEE specifies the extensions to Security Device Event Exchange (SDEE) that are used by the Cisco IPS. SDEE is a product-independent standard for communicating security device events. SDEE is an enhancement to the current version of RDEP2 that adds extensibility features that are needed for communicating events generated by various types of security devices. RDEP2, SDEE, and CIDEE are discussed next.

RDEP2 RDEP2 is a communication protocol used to exchange IPS event, IP log, configuration, and control messages between IPS clients (for example, Security Monitor) and IPS servers (for example, Sensor). RDEP2 communications consist of request and response messages. RDEP2 clients initiate request messages to RDEP2 servers. RDEP2 servers respond to request messages with response messages.

RDEP2 defines three classes of request/response messages: event, IP log, and transaction messages. Event messages include IPS alert, status, and error messages. Clients use IP log requests to retrieve IP log data from servers. Transaction messages are used to configure and control IPS servers.

RDEP2 uses HTTP, TLS and SSL and XML to provide a standardized interface between RDEP2 monitoring tool (Security Monitor) and the IPS sensor. RDEP2 uses HTTP message formats and message exchange protocol to exchange messages between the monitoring device and the sensor.

IPS sensors can accept connections from one to ten RDEP2 clients (Security Monitor) simultaneously. Clients selectively retrieve data by time range, type of event (alert, error, or status message), and level (alert = high, medium, low, or informational; error = high, medium, or low). Events are retrieved by a query (a single bulk get), a subscription (a real-time persistent connection), or both. Communications are secured by TLS or SSL. The monitoring application sends an RDEP2 event request to the sensor's web server, which passes it to the event server. The event server queries the event store through Intrusion Detection Application Programming Interface (IDAPI), and then returns the result. For retrieving events, the sensor is backward-compatible to RDEP, even though the new standard for retrieval is RDEP2. It is strongly recommended to use RDEP2 to retrieve events and send configuration changes for IPS 5.0.

IPS MC also sends configuration changes to the sensor through RDEP2. The IPS MC sends an RDEP2 control transaction to the sensor's web server, which passes it to the control transaction server. The control transaction server passes the control transaction through IDAPI to the appropriate application, waits for the application's response, and then returns the result.

SDEE SDEE, developed by Cisco Systems, Inc., is a product-independent standard for communicating security device events. SDEE is an enhancement to the current version of RDEP2 that adds extensibility features that are needed for communicating events generated by various types of security devices.

Systems that use SDEE to communicate events to clients are referred to as SDEE providers. SDEE specifies that events can be transported using the HTTP or HTTP over SSL and TLS protocols. When HTTP or HTTPS is used, SDEE providers act as HTTP servers, whereas SDEE clients are the initiators of HTTP requests.

CIDEE CIDEE specifies the extensions to SDEE that are used by the Cisco IPS. The CIDEE standard specifies all possible extensions that are supported by IPS. Specific systems may implement a subset of CIDEE extensions. However, any extension that is designated as being required must be supported by all systems. CIDEE supports the following events:

- This is used for error events. Error events are generated by the CIDEE provider when the provider detects an error or warning condition. The event contains an error code and textual description of the error.

- This is used for status message events. Status message events are generated by CIDEE providers to indicate that something of potential interest occurred on the host. Different types of status messages can be reported in the status eventone message per event. Each type of status message contains a set of data elements that are specific to the type of occurrence that the status message is describing. The information in many of the status messages might be useful for audit purposes. Errors and warnings are not considered status information and are reported using rather than .

- This is used for a block request event. It is generated to indicate that a block action is to be initiated by the service that handles network blocking.

IDIOM IDIOM is a data format standard that defines the event messages that are reported by the IPS, in addition to the operational messages that are used to configure and control intrusion detection systems. These messages consist of XML documents that conform to the IDIOM XML schema.

IDCONF IPS 5.0 manages its configuration using XML documents. IDCONF specifies the XML schema including IPS 5.0 control transactions. The IDCONF schema does not specify the contents of the configuration documents, but rather the framework and building blocks from which the configuration documents are developed. It provides mechanisms that let the IPS managers and CLI ignore features that are not configurable by certain platforms or functions through the use of the feature-supported attribute. IDCONF messages are exchanged over RDEP2 and are wrapped inside IDIOM request and response messages. IDIOM, for the most part, has been superseded by IDCONF, SDEE, and CIDEE.

Modes of Sensor Operation
Starting from IPS version 5.0, Sensor can function in one of the following modes:

Inline

Inline bypass

Promiscuous

Combined

Modes are described in the sections that follow.

Inline Mode
Running Inline mode puts the IPS sensor directly into the traffic flow. Because an inline IPS sensor sits in the path of traffic flow, it stops attacks by dropping malicious traffic before it reaches the intended target, thus providing a protective service. Not only is the inline device processing information on Layers 3 and 4, but it is also analyzing the contents and payload of the packets for more sophisticated embedded attacks (Layers 3 to 7). This deeper analysis lets the system identify and stop or block attacks that normally would pass through a traditional firewall device.

In Inline mode, a packet comes in through the first interface of the pair of the sensor and out the second interface of the pair. The packet is sent to the second interface of the pair unless that packet is being denied or modified by a signature. You can configure ASA-SSM to operate inline even though it has only one sensing interface.

In Inline mode, in addition to Blocking, and TCP reset, which are available in Promiscuous mode, additional action types include the following:

request-block-connection Request SHUN of connection

request-block-host Request SHUN of attacker host

deny-attacker-inline Do not transmit packets with source address of attacker

deny-packet-inline Do not transmit the single packet causing alert

deny-connection-inline Do not transmit packets on this TCP connection

log-attacker-packets Activate packet logging for attacker address

log-victim-packets Activate packet logging for victim address

log-pair-packets Activate packet logging for attacker/victim address pair

reset-tcp-connection Send TCP RST packets to terminate connection

produce-alert Write evIdsAlert to EventStore

produce-verbose-alert Write evIdsAlert to EventStore with triggerPacket

request-snmp-trap Write evIdsAlert to EventStore with SNMP request in AlarmTraits

Note

Even though you can run IPS 5.0 software on IDS-4210, and Cisco IDS Network Module (NM-CIDS), these two sensors do not operate in Inline mode.

Inline Bypass Mode
Because the traffic is passing through the sensor in Inline mode, it is a point of failure for the network behind it. To mitigate this risk, a software driver-level Bypass mode option is available. The Bypass mode will unconditionally copy packets from one interface to the other. The Bypass has an Automatic mode that will activate it during sensorApp configuration operations, or if sensorApp is unresponsive. Automatic bypass mode is turned on by default, which is the recommended configuration. However, if you want to turn Bypass on permanently, there is a serious security consequence, as all the packets will bypass IPS processing subsystems.

Note

Bypass mode functions only when the operating system is running. If the sensor is powered off or shut down, Bypass mode does not worktraffic is not passed to the sensor.

Promiscuous Mode
Packets do not flow through the IPS in Promiscuous mode. The sensor analyzes a copy of the monitored traffic rather than the actual redirected packet. The advantage of operating in Promiscuous mode is that the IPS does not affect the packet flow. The disadvantage of operating in Promiscuous mode, however, is that the IPS cannot stop malicious traffic from reaching its intended target for certain types of attacks, such as atomic attacks (single-packet attacks). The response actions implemented by promiscuous IPS devices are post-event responses and often require assistance from other networking devices, for example, the Blocking feature. TCP reset is another action type in Promiscuous mode, but not a guaranteed mechanism to stop attacks. Although such response actions can prevent some classes of attacks, for atomic attacks, the single packet has the chance of reaching the target system before the promiscuous-based sensor can apply an ACL modification on a managed device (such as a firewall, switch, or router) or perform TCP reset on the connection.

Combined Modes
Sensor platforms with three or more interfaces can combine both Inline and Promiscuous traffic input, or with four or more interfaces, you can have two separate inline feeds. The combinations are flexible; the only rule is that Inline mode takes two interfaces. Note that if you have the same traffic coming in on multiple interfaces, irregular results will ensue. You might see duplicate non-TCP alarms, or with TCP you might get many 13xx alarms or TCP stream collisions resulting in no alarm.

Hardware and Interfaces Supported
It is important to know which sensors support IPS version 5.0. Table 14-1 summarizes all the sensors that support IPS 5.0. This information is taken from the following link, so refer to this link for additional details:

Table 14-1. Supported Sensors for Version IPS 5.0 Model Name
Part Number
Optional Interfaces

IDS-4210
IDS-4210

IDS-4210-K9

IDS-4210-NFR

IDS-4215
IDS-4215-K9

IDS-4215-4FE-K9
IDS-4FE-INT=

IDS-4235
IDS-4235-K9
IDS-4FE-INT=

IDS-4250
IDS-4250-TX-K9

IDS-4250-SX-K9

IDS-4250-XL-K9
IDS-4FE-INT=, IDS-4250-SX-INT=, IDS-XL-INT=

IDS-XL-INT=

IPS-4240
IPS-4240-K9

IPS-4255
IPS-4255-K9

ASA-SSM-10
ASA-SSM-AIP-10-K9

ASA-SSM-20
ASA-SSM-AIP-20-K9

IDSM-2
WS-SVC-IDSM2-K9

NM-CIDS
NM-CIDS-K9

Many times, knowing and using the proper interfaces for different purposes is a challenge, and the source of many problems with IPS operation. Table 14-2 summarizes all the available interfaces that are supported on sensors. This information is taken from the following location:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/csids/csids11/hwguide/hwintro.htm#wp490986

Table 14-2. Interface Support for Appliances and Modules Running IPS 5.0 Base Chassis
Added PCI Cards
Interfaces Supporting Inline
Possible Port Combinations
Interfaces Not Supporting Inline

IDS-4210

None
N/A
All

IDS-4215

None
N/A
All

IDS-4215
4FE
FastEthernet0/1

4FE

FastEthernet1/0

FastEthernet1/1

FastEthernet1/2

FastEthernet1/3
1/0<->1/1

1/0<->1/2

1/0<->1/3

1/1<->1/2

1/1<->1/3

1/2<->1/3

0/1<->1/0

0/1<->1/1

0/1<->1/2

0/1<->1/3
FastEthernet0/0

IDS-4235

None
N/A
All

IDS-4235
4FE
4FE

FastEthernet1/0

FastEthernet1/1

FastEthernet1/2

FastEthernet1/3
1/0<->1/1

1/0<->1/2

1/0<->1/3

1/1<->1/2

1/1<->1/3

1/2<->1/3
GigabitEthernet0/0

GigabitEthernet0/1

IDS-4250

None
N/A
All

IDS-4250
4FE
4FE

FastEthernet1/0

FastEthernet1/1

FastEthernet1/2

FastEthernet1/3
1/0<->1/1

1/0<->1/2

1/0<->1/3

1/1<->1/2

1/1<->1/3

1/2<->1/3
GigabitEthernet0/0

GigabitEthernet0/1

IDS-4250
SX
None
N/A
All

IDS-4250
XL
2 SX of the XL

GigabitEthernet2/0

GigabitEthernet2/1
2/0<->2/1
GigabitEthernet0/0

GigabitEthernet0/1

IDSM-2

port 7 and 8

GigabitEthernet0/7

GigabitEthernet0/8
0/7<->0/8
GigabitEthernet0/2

IPS-4240

4 onboard GE

GigabitEthernet0/0

GigabitEthernet0/1

GigabitEthernet0/2

GigabitEthernet0/3
0/0<->0/1

0/0<->0/2

0/0<->0/3

0/1<->0/2

0/1<->0/3

0/2<->0/3
Management0/0

IPS-4255

4 onboard GE

GigabitEthernet0/0

GigabitEthernet0/1

GigabitEthernet0/2

GigabitEthernet0/3
0/0<->0/1

0/0<->0/2

0/0<->0/3

0/1<->0/2

0/1<->0/3

0/2<->0/3
Management0/0

NM-CIDS

None
N/A
All

ASA-SSM-10

GigabitEthernet0/1
By security context
GigabitEthernet0/0

ASA-SSM-20

GigabitEthernet0/1
By security context
GigabitEthernet0/0

User/NAS Import Options

User/NAS Import Options
This feature allows changes either online or offline, and allows updating of the CS ACS database with a colon-delimited file. The following are the actions available for user and NAS:

Users: add, change, and delete

NAS: add and delete

You must restart CSRadius and CSTacacs for changes to take effect.

The following are some of the important points about importing:

The first line must contain ONLINE or OFFLINE.

This determines if the CSAuth service needs to be stopped during this process.

CSUtils cannot distinguish between multiple instances of an external database.

CSUtil will use the first instance of an external database.

Import User Information
You can add users to the existing database with the entry shown in Example 13-17. This entry adds the user Joe to group 2 in the CS ACS database. It also points authentication for this user to the internal CS ACS database with a password of my1Password.

Example 13-17. Adding a User to CS ACS
ADD:Joe:PROFILE:2:CSDB:my1Password

To change the CS ACS profile for Joe, use the command shown in Example 13-18. This entry updates Joe to group 3 and points the password to the NT domain database.

Example 13-18. Updating a User to CS ACS
UPDATE:Joe:PROFILE:3:EXT_NT

The DELETE entry can be used to delete users as shown in Example 13-19.

Example 13-19. Deleting a User from CS ACS
DELETE:Joe

Import NAS Information
Use the entry shown in Example 13-20 to add an NAS to the CS ACS database. This entry adds the router named router1, using the shared secret of my1NAS. This NAS will use RADIUS.

Example 13-20. Adding NAS
ADD_NAS:router1:IP:10.10.10.10:KEY:my1NAS:VENDER:"RADIUS (Cisco IOS/PIX)"

If you need to delete a specific NAS, use the command shown in Example 13-21, which deletes NAS router1.

Example 13-21. How to Delete a Specific NAS
DEL_NAS:router1

You can also choose to run all the previously shown procedures using a single text file. Example 13-22 shows a sample text file that contains multiple actions for different users.

Example 13-22. import.txt File Whose Content Can Be Imported Once
OFFLINE
ADD:user01:CSDB:userpassword:PROFILE:1
ADD:user02:EXT_NT:PROFILE:2
ADD:chapuser:CSDB:hello:CHAP:chappw:PROFILE:3
ADD:mary:EXT_NT:CHAP:achappassword
ADD:joe:EXT_SDI
ADD:user4:CSDB:user4password
ADD:user5:CSDB_UNIX:unixpassword
UPDATE:user9:PROFILE:10
DELETE:user10
ADD_NAS:router1:IP:10.10.10.10:KEY:my1NAS:VENDOR:"TACACS+ (Cisco
IOS)":NDG:"California"
DEL_NAS:router2

Compact User Database
When you delete a user from the CS ACS database, the record is marked as deleted. You might need to compact the database to actually remove the "deleted records". Compacting the database addresses this issue. When you compact a database, it first dumps the data, then creates a new database, and finally imports all the data that was dumped earlier. The following is the syntax for compacting a database:

csutil.exe -q -d n -l

Example 13-23 shows the sample of database compact run.

Example 13-23. Sample Database Compact Command
C:\Program Files\CiscoSecure ACS v3.3\Utils>net stop CSAuth
The CSAuth service is stopping.
The CSAuth service was stopped successfully.

C:\Program Files\CiscoSecure ACS v3.3\Utils>csutil -q -d -n -l
CSUtil v3.3(2.2), Copyright 1997-2004, Cisco Systems Inc
Done

Initializing database....
Done

Initializing database...
Loading database from dump.txt...
Done

C:\Program Files\CiscoSecure ACS v3.3\Utils>

Export User and Group Information
Export User and Group Information may be useful for troubleshooting the configuration issue by Cisco support. You will need to stop CSAuth before exporting this information.

To export user information to users.txt, enter the following command:

csutil.exe u

To export group information to groups.txt, enter the following command:

csutil.exe g

Other features of CSUtil.exe include the following:

Export Registry information to setup.txt.

Decode CS ACS internal error codes.

Recalculate Cyclic Redundancy Check (CRC) values for manually copied files.

Import user-defined RADIUS vendors and VSA sets.

Reports and Activity (Real-time Troubleshooting)

Diagnostic Commands and Tools
Cisco Secure ACS has extensive logging capability that allows an administrator to troubleshoot any issue pertaining to CS ACS Server itself (for example, replication) or an AAA requests problem (for example, an authentication problem) from NAS. This section explores these tools and how to use them efficiently.

Reports and Activity (Real-time Troubleshooting)
The Failed Attempts log under the Reports and Activity from the GUI is the quickest and best way to find out the reasons for authentication failure. Failed Attempts logs are turned on by default. However, if you want to add additional fields to the Default, you may by browsing to System Configuration > Logging > CSV Failed Attempts. In the CSV Failed Attempts File Configuration page, move desired attributes from Attributes to Logged Attributes. Then click on Submit. These additional attributes are shown under Reports and Activity. Occasionally, you might need to look at the Passed Authentications to troubleshoot authorization or NAS Access Restriction (NAR) issues. By default, the Passed Authentication log is not turned on. To turn it on, go to System Configuration > Logging > CSV Passed Authentications, and check Log to CSV Passed Authentications report under Enable Logging. There are other logs available for different services. For instance, for replication issues, there is a corresponding CSV file called Database Replication under Reports and Activity.

Radtest and Tactest
These tools are available to simulate AAA requests from the CS ACS server itself, which eliminates any possibilities of NAS configuration issues. This is especially important for troubleshooting the authentication issues with external user database authentication, for example, Microsoft Active Directory (AD) or Secure ID server. These tools are installed as part of CS ACS installation and located at C:\Program Files\CiscoSecure ACS v3.3\Utils>. More details on how to run these tools can be found at the following location: http://www.cisco.com/en/US/products/sw/secursw/ps2086/products_tech_note09186a00800afec1.shtml#auth_of

Package.cab File
Package.cab is the result of execution of the CSSupport utility, which includes all the log files for every service that we have discussed in the section entitled "CS ACS Architecture." Before running the CSSupport utility as shown in the paragraphs that follow, to capture the debug level logging, be sure to collect the "FULL" logging (on CS ACS, System Configuration > Service Control > Level of detail > Choose FULL > Restart). This is shown in Figure 13-3. Also be sure to check Manage Directory and set the appropriate option.

Figure 13-3. Turning on Full Logging on CS ACS

[View full size image]

Once you set up the logging level to "FULL", run a few tests that are sure to fail and then run cssupport.exe as shown below:

C:\Program Files\CiscoSecure ACS v3.3\utils\CSSupport.exe

The Package.cab file contains a good deal of meaningful information, but the amount of information may be overwhelming. So, being able to read the file efficiently is a key to success in isolating issues from the Package.cab file logs. Before getting into any more detail, you need to understand what goes into the makings of the package.cab file. Figure 13-4 shows the unzipped version of package.cab with a listing of files (icons are arranged by type).

Figure 13-4. Listing of Files in package.cab

[View full size image]

The following are short descriptions of the files of package.cab:

CSV Files CSV files contain the information about Audit log, Accounting, and Failed and Passed Authentication. Most of the files contain statistics, but to troubleshoot issues, Failed and Passed Authentication files are often used in conjunction with the log files that are discussed in the paragraphs that follow. The CSV files are created every day. Each file name without the date is the Active file. So, Failed Attempts active.csv is the active file, which stores the Failed Attempts information from the NAS.

Log Files Every service discussed in the "CS ACS Architecture" section of this chapter has a corresponding log file. These files contain extensive logs about each and every service. For instance, auth.log contains all the current log information of CSAuth service. Just like CSV files, log files are created every day and the active log file is the one without the date in its name.

User Database Files Three files go into making the CS ACS database. These files are user.dat, user.idx, and varsdb.mdb. You should not manipulate these files. Unless otherwise requested by Cisco, capturing these files is not necessary when running the CSSupport.exe utility.

Registry File ACS.reg contains the Registry information of the CS ACS Server. Substantial CS ACS configuration (for example, NAS) goes into the Windows Registry. So, reading this file may be required for some troubleshooting. Do not import this file into another server; instead, open it with a text editor of your choice.

Other Files Another useful file is MSInfo.txt, which contains the server and the OS information. The resource.txt file contains the resource information on the server, and SecEventDump.txt, AppEventDump.txt, and SysEventDump.txt contain an additional event dump on the server that may be used occasionally to troubleshoot any issues with the server itself.

As mentioned before, reading these files efficiently to isolate the problem is a key to success in troubleshooting CS ACS. To illustrate how to analyze the files, examine an example. The example assumes that a regular login authentication by the CS ACS Server is failing. The NAS debug does not give any conclusive output that indicates the reason for the failure.

To analyze this, first look at the Failed Attempts active.csv file to see why the user is failing. Quite often the information obtained from this file gives you the reason, so that no further analysis is needed; however, that's not always the case. For this example, assume that you have no conclusive reason for failure from the CSV file. However, you do have the username. The next step is to analyze the auth.log, because that contains more detailed information.

So, you search the username in the auth.log file. In this case, unfortunately, you receive no results from the search based on that username. So there must be a problem. It could be that CSTacacs service cannot process and forward the authentication request to the CSAuth service. Because you see the authentication failure in the Failed Attempts log, the authentication request must be reaching the CS ACS, and the first service that receives that packet is the CSTacacs, as the communication protocol configured between NAS and CS ACS is TACACS+. So, you need to analyze the TCS.log file, which contains all the activities that CSTacacs performs. As expected, you see the user request coming from the NAS. However, the user request is not being forwarded to the CSAuth service. After a little investigation, we find that NAR is configured for this user and, hence, packets are being dropped by the CSTacacs service; therefore, they are not being forwarded to the CSAuth service. Hence, you do not see the user in the auth.log. For every AAA request failure, you must look at the Failed Attempt first, and then search for the username in the auth.log. If an additional detail is needed, you need to analyze either the TCS.log or the RDS.log. Note that both CSTacacs and CSRadius form the communication bridge between the NAS and CS ACS, and CSAuth is the communication bridge between the CSTacacs/CSRadius and any external user databases such as Active Directory, NDS, and so on.

Reports and Activity (Real-time Troubleshooting)

CS ACS with Novell NDS integration

Categorization of Problem Areas
The problem areas of CS ACS can be categorized as follows:

Installation and upgrade issues

CS ACS with Active Directory integration

CS ACS with Novell NDS integration

CS ACS with ACE Server (Secure ID [SDI]) integration

Replication issues

Network access restrictions (NAR) issues

Downloadable ACL issues

Installation and Upgrade Issues
If you follow the procedure properly, installation or upgrade is a fairly easy process for both CS ACS on Windows and CS ACS Appliance. This section examines the installation and upgrade procedure, important issues to be aware of, things that may go wrong, and how to resolve the problems.

CS ACS on Windows Platform
Depending on the version of CS ACS that needs to be installed, check the following documentation to make sure all the minimum requirements for the Operating System version, Service Packs (SPs), and so on, are met. Otherwise, abnormal failure might occur that might not be diagnosed or supported by Cisco TAC unless the documented minimum requirement is fulfilled.

http://www.cisco.com/warp/public/480/csnt.html

Installation steps are intuitive, and therefore they are not covered here.

Upgrading from an older to a new version is a little more complex than installing a new version. However, if you work through the following steps carefully, you can minimize the chance of upgrade failure substantially:

Step 1. Review the prerequisites for installation of the version that you are trying to upgrade. If you must perform an incremental upgrade, for instance, from CS ACS 2.3 on NT platform to CS ACS 3.3 on Win 2K platform, define the strategy.

Step 2. Back up the database using C:\Program Files\CiscoSecure ACS v3.3\Utils>CSUtil -b (full backup including NAS information) and C:\Program Files\CiscoSecure ACS v3.3\Utils>CSUtil -d (partial backup, only users/groups information) options, and save the files offline in a different location.

Step 3. Run the setup.exe file of the new version.

Step 4. If the standard upgrade procedure in Step 3 fails, run the uninstall shield or uninstaller from the control panel, and choose the option during uninstall to keep the old database. Then install the new version. These procedures should find the information saved by the uninstall procedure and import it.

Step 5. If Step 4 fails, chances are very high that your Registry has been corrupted. If so, uninstall the CS ACS completely, and run the clean.exe files, which come in the CS ACS CD. These files will clean up the Registry. Then proceed with the installation. In the newer version (for instance, CS ACS 3.3), the Clean utility comes as setup.exe within the Clean directory, which is in the ACS Utilities\Support\ directory of the installation CD.

Step 6. If all the services started on the newer version, import the dump.txt that you have created in Step 2 with the csutil -d option, which contains only the user and group information. You still need to define the NASs. If there is a small number of NASs, this may work.

Step 7. If you have a large number of NASs, build another server with a version that runs the old version of code and import the database that is created in Step 2 with the csutil -b option, which includes the whole database that has the NAS information in it. Then follow Steps 26.

You should be aware of the following important facts if you are trying to upgrade one of the older CS ACS versions or from the trial version:

The minimum CS ACS version requirement to run on the Windows platform is CS ACS 2.5.

If you are upgrading CS ACS from 2.3 on a Windows NT platform to CS ACS 3.3 on the Windows 2000 platform, be sure to upgrade to CS ACS 2.6 on the NT platform first, and be sure the database upgraded and data migrated properly. As CS ACS 2.6 can run on Windows 2000, upgrade the OS of your CS ACS server to Windows 2000 after ensuring that the service packs and other prerequisites are fulfilled, and, finally, upgrade to the target version of CS ACS, which is CS ACS 3.3.

If you are running a trial version, to migrate that version to production, just upgrade or install the production CS ACS version on top of the trial version. For example, you can install the CS ACS 3.1 production version over the CS ACS 2.6 trial version, or install the CS ACS 3.3 production version over the CS ACS 3.3 trial version.

CS ACS installation or upgrade may fail for the following reasons:

Running an unsupported version of OS, service pack (SP), or browser.

CS ACS services are crashing.

If you are running a supported browser and service pack but CS ACS is still crashing, upgrade to the latest build of the CS ACS release that you are running. There may be a bug that has been fixed in the latest build of that release. If you are running the latest release, provide Cisco TAC with the package.cab file or, at least, run drwtsn32 in a DOS prompt, with the following box checked: Dump Symbol Table.

CS ACS with Active Directory Integration
To integrate with the Active Directory, Cisco Secure ACS can be installed in one of the following modes:

Standalone Server If CS ACS is installed on a standalone server, CS ACS can authenticate Windows users only against the local SAM database.

Domain Controller If CS ACS is installed on a Primary Domain Controller (PDC) or Backup Domain Controller (BDC), it will be able to authenticate Windows users who are defined in any trusted domain.

Member Server CS ACS on a member server will also authenticate users defined in any trusted domains. However, lack of permissions could cause issues with domain lists, authentication, and Remote Access Service (RAS) flag fetching.

Cisco Secure ACS services run under the local system account on the server. The local system account has almost the same privileges as the administrator.

When a new external WindowsNT/2000 database is defined on CS ACS, CS ACS fetches the list of domains trusted by the domain of the computer where the server is installed. CS ACS fetches the list of trusted domains only to populate it to Java control. The user can add domains manually as well. CS ACS uses the list of enumerated domains to determine the order in which they will be checked when an external authentication is presented.

When a new mapping between Windows NT/2000 user groups and Cisco Secure ACS user group is defined, CS ACS obtains and displays the list of the user groups defined in the selected Windows domain.

When a windows user is being authenticated, CS ACS uses Microsoft's Network Logon on behalf of the user to verify the user's credentials. This is a noninteractive login, as opposed to a desktop login.

CS ACS fetches the following information about that user:

List of user groups to which the user belongs.

Callback flag.

Values are set on the MS user definition page, which includes Admin set phone #, and user set (send by the client during authentication).

Dialin permission (RAS flag).

Password status.

Microsoft Point-to-Point Encryption (MPPE) keys (there are two, a 56-bit and 128-bit key).

Until CS ACS version 3.0, there were no "hooks" into the Security Accounts Manager (SAM) database to change the password through CS ACS. CS ACS 3.0 uses an API to change MS-CHAP passwords, but the MS-CHAPv2 protocol must be supported end-to-end.

Table 13-3 shows the trust relationship for CS ACS with the domain controller when the CS ACS is on the member server of Domain A.

Table 13-3. Trust Relationship of CS ACS and Windows Domain Controller When CS ACS Is on a Member Server of Domain A Task
Trust Direction
Description

Fetch list of domains trusted by Domain A.
A trusts other domains.
The list includes domains trusted by A.

Fetch list of user groups from a trusted Domain B.
B trusts A.
CS ACS reads information (accesses resources) on Domain B.

Authenticate a user with account on Domain B.
A trusts B.
CS ACS performs the network logon with user name. The user with an account on Domain B is going to access a computer in Domain A.

Fetch information (callback, and so on) on user with account on Domain B.
B trusts A.
CS ACS reads information (accesses resources) on Domain B.

Change password of a user with account on Domain B (CS ACS v3.0).
B trusts A.
CS ACS changes information (Access ressources) on Domain B.

Configuration Steps
The following steps are required to integrate CS ACS with the domain controller:

On the domain controller serving the CS ACS server follow these steps:

Step 1. Create a user.

Step 2. Make the user hard to hack by giving it a very long, complicated password.

Step 3. Make the user a member of the Domain Administrator group.

Step 4. Make the user a member of the Administrators group.

On the Windows 2000 server running CS ACS, follow these steps:

Step 1. Add a new user to the proper local group. Go to Start > Settings > Control Panel > Administrative Tools > Computer Management. Open Local Users and Groups and then Groups. Double-click the Administrators group. Click Add. Choose the domain from the Look in box. Double-click the user created earlier to add it. Click OK.

Step 2. Give the new user special rights on CS ACS server. Go to Start > Settings > Control Panel > Administrative Tools > Local Security Policy > Local Policies. Open User Rights Assignment. Double-click on Act as part of the operating system. Click Add. Choose the domain from the Look in box. Double-click the user created earlier to add it. Click OK. Double-click Log on as a service. Click Add. Choose the domain from the Look in box. Double-click the user created earlier to add it. Click OK.

Step 3. Set the CS ACS services to run as the created user. Open Start > Settings > Control Panel > Administrative Tools > Services. Double-click the CSADMIN entry. Click the Log On tab. Click This Account and then the Browse button. Choose the domain, and double-click the user created earlier. Click OK. Repeat for the remainder of the CS services.

Step 4. Wait for Windows to apply the security policy changes, or reboot the server. If you rebooted the server, skip the rest of these instructions. Otherwise, stop and then start the CSADMIN service. Open the CS ACS GUI. Click on System Config. Click on Service Control. Click Restart.

Note

If the Domain Security Policy is set to override settings for "Act as part of the operating system" and "Log on as a service" rights, the user rights changes listed in the previous steps also need to be made there.

Troubleshooting Steps
This section discusses some of the common issues that you may run into when integrating with Active Directory.

Windows Group to CS ACS Group Mapping Problems
During Configuring of Group mapping, the user sees a pop-up window. If you are having problems with Group mapping, you may see the following message:

Failed to enumerate Windows groups. If you are using AD consult the installation
guide for information

Possible causes of the problems are as follows:

CS ACS services do not have privileges to execute the NetGroupEnum function Refer to Configuration steps discussed for "CS ACS with Active Directory Integration" in the preceding section to correct the permission issue.

NetBIOS over Transmission Control Protocol (TCP) is not enabled NetBIOS over TCP must be enabled; otherwise, group mapping will fail.

Domain Name System (DNS) is not working correctly You may try to reregister DNS with commands: "ipconfig/flushdns" then "ipconfig/registerdns" at the DOS prompt.

Remote Procedure Call (RPC) is not working correctly (for example, after applying the blaster patch) In that case, consult with Microsoft.

Domain Controller (DCs) are not time-synchronized Run the command net time /Domain: to synchronize time.

Different service packs If you run different SPs on different DCs, you may run into this problem. Apply the same patch to fix the problem.

NetLogon Services are not running NetLogon Services must up and running on all DCs.

Check that no firewall (FW) packet filters are installed If there is a packet-filtering firewall installed, be sure to select Yes on DNS properties to "allow dynamic updates".

CS ACS Maps User to Wrong Group of CS ACS (Default Group)
After successful user authentication based on the group mapping configuration, the user is mapped to a specific CS ACS group. The following list summarizes some of the reasons why the user may be mapped to the wrong CS ACS group:

Misconfiguration of group mapping If the user belongs to both group X and group Y, CS ACS assigns the user according to the order in which the user was configured.

Service accounts under which CS ACS services are running do not have permission to validate groups for another user Log in as user, under the CS ACS services that are running. Check if you have access to get the groups of another user.

CS ACS with Novell NDS Integration
This section works through the configuration steps that lead in turn to sections that cover troubleshooting steps.

Configuration Steps
Use the following steps to configure an NDS database with CS ACS on Windows.

Step 1. Consult with your Novell NetWare administrator to get administrator context information for CS ACS and the names of the Tree, Container, and Context details.

Step 2. On CS ACS, click on External User Databases > Database Configuration > Novell NDS > Configure.

Step 3. In the Novell NDS configuration window, enter a name for the configuration. This is for information purposes only.

Step 4. Enter the Tree name.

Step 5. Enter the full Context List, with items separated by dots(.). You can enter more than one context list. If you do, separate the lists with a comma and space. For example, if your Organization is Corporation, your Organization Name is Chicago, and you want to enter two Context names, Marketing and Engineering, you would enter: Engineering.Chicago.Corporation, Marketing.Chicago.Corporation. You do not need to add users in the Context List.

Step 6. Click Submit. Changes take effect immediately; you do not need to restart the Cisco Secure ACS.

Caution

If you click Delete, your NDS database is deleted.

Step 7. Then perform the Group Mappings (between the Novell NDS Database Groups and CS ACS Groups) by browsing to External User Databases > Database Group Mappings > Novell NDS.

Step 8. Finally, configure the unknown user policy by selecting Check the following external user databases and moving the Novell NDS database from the External Databases to the Selected Databases text box on the External User Databases > Unknown User Policy page.

Troubleshooting Steps
You can isolate any problem that you may have with the troubleshooting steps in the sections that follow.

Novell Client Is Not Installed
You must install the Novell client on the CS ACS server, so that CS ACS can talk to the Novell NDS database. If you do not have the Novell client installed on the CS ACS, and you try to configure Novell NDS database settings from the External User Database > Database Configuration > Novel NDS, you will receive an error message similar to the following:

An error has occurred while processing the External Database Configuration Page
because of an internal error..

Revise the Configuration on CS ACS
Most of the time, the Novell NDS authentication failure is caused by misconfiguration. Therefore, check to see if the tree name, context, and container name are all specified correctly. Start with one container in which users are present; later more containers can be added if needed.

Check Admin Username
Check the admin username to be sure it is correct, and that you have defined a fully qualified path. For example, instead of admin, define admin.cisco, as the latter is a fully qualified name.

Example 13-1 shows the incorrect provision of admin credentials.

Example 13-1. Incorrect Admin Credentials
AUTH 03/22/2005 10:40:21 I 0360 0676 External DB [NDSAuth.dll]: Tree 224462640 could
not log in with admin credentials supplied

Perform Group Mapping
Performing Group Mapping is an excellent test to ensure the admin context can connect and pull the group information from the Novell NDS database. Therefore, if you are unable to map groups, the admin user does not have permission to list the groups. Under that circumstance, check that the admin can list users in the other domain. One way to verify that is as follows: on the CS ACS Server, using Nwadmin, examine the groups from the other domain. If you cannot do so, consult with the Novell administrator.

Authentication Failure with a Bad Password
Before looking at authentication that has failed either due to the wrong username or a bad password, it's extremely important to understand and be familiar with the sequence of events that occur within CS ACS with Novell NDS authentication. Therefore, closely observe the successful user authentication log shown in Example 13-2.

Example 13-2. Successful User Authentication Against NDS Database
AUTH 03/22/2005 12:20:56 I 5081 1764 Start RQ1026, client 2 (127.0.0.1)
! As the user doesn't exist on the local database, CS ACS is tagging this as unknown user
AUTH 03/22/2005 12:20:56 I 4683 1764 Attempting authentication for Unknown User
'cisco'
! The following two lines indicate that Novell NDS is configured to this user
! authentication. This is being done by selecting Novell NDS database for Unknown User
! Policy
AUTH 03/22/2005 12:20:56 I 1280 1764 ReadSupplierRegistry: Novell NDS loaded
AUTH 03/22/2005 12:20:56 I 0863 1764 pvAuthenticateUser: authenticate 'cisco'
against Novell NDS
! Following lines indicate that CS ACS is trying to lock a thread for this user
! Authentication.
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: User cisco waiting
for lock
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: User cisco waiting
in lock
! A new thread is getting initialized here for the user authentication under ndstest
tree
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: Initializing thread
0 for tree ndstest
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Starting Thread 0
! The following two lines indicate that the user authentication is under works
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Waiting for work
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Got work
! This is where the user is authenticated.
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Authenticated
cisco.OU=SJ.TESTING.LAB, Response 0
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: Back from Wait for
user cisco with code 0
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: Response 0 from
successful Tree ndstest
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Response 0 from Tree
ndstest
AUTH 03/22/2005 12:20:56 I 0360 0472 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Waiting for work
! Following three lines indicates that the group mappings between Novell NDS and CS ACS
! are successful. Third line in particular indicates that user is mapped to CS ACS Group
! number 150.
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: Added
'sj_acs.SJ.testing.LAB' to Group List for user: cisco.OU=SJ.TESTING.LAB
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: There were 1 Groups
for this user: cisco.OU=SJ.TESTING.LAB
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: User cisco
authenticated into group 150
AUTH 03/22/2005 12:20:56 I 0360 1764 External DB [NDSAuth.dll]: User cisco out from lock
AUTH 03/22/2005 12:20:56 I 3421 1764 User cisco password type changed
AUTH 03/22/2005 12:20:56 I 1586 1764 User cisco feature flags changed
AUTH 03/22/2005 12:20:56 I 1586 1764 User cisco feature flags changed
AUTH 03/22/2005 12:20:56 I 5081 1764 Done RQ1026, client 2, status 0

As mentioned before, it is extremely important to understand the sequence of events that occur with a successful user authentication as shown in Example 13-2, before you can analyze and find the cause of failure for a bad user password. With the knowledge gained from Example 13-2, examine example 13-3, which shows failed authentication due to a bad password.

Example 13-3. Shows a Failed Authentication Attempt Due to Bad Password to NDS Database
AUTH 08/13/2003 14:11:47 I 0276 2212 External DB [NDSAuth.dll]: User cisco waiting
for lock
AUTH 08/13/2003 14:11:47 I 0276 2212 External DB [NDSAuth.dll]: User cisco waiting
in lock
AUTH 08/13/2003 14:11:47 I 0276 2212 External DB [NDSAuth.dll]: Initializing thread
0 for tree ndstest
AUTH 08/13/2003 14:11:47 I 0276 1968 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Got work
AUTH 08/13/2003 14:11:50 I 0276 1968 External DB [NDSAuth.dll]: Response 1 from Tree
ndstest
AUTH 08/13/2003 14:11:50 I 0276 1968 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Waiting for work
! In the following line, code 102 indicates that authentication fails due to bad
username
! or wrong password.
AUTH 08/13/2003 14:11:53 I 0276 2212 External DB [NDSAuth.dll]: Back from Wait for
user cisco with code 102
! Then eventually it times out trying.
AUTH 08/13/2003 14:11:53 I 0276 2212 External DB [NDSAuth.dll]: Timeout trying User
cisco
AUTH 08/13/2003 14:11:53 I 0276 2212 External DB [NDSAuth.dll]: User cisco out from
lock

Authentication Failure When the User Does Not Exist
If the user does not exist on the Novell NDS database or the user enters the wrong username, the authentication will fail, giving the same error code as a bad password (error code 102). Example 13-4 shows the output when the user does not exist on the database.

Example 13-4. Failed Authentication Due to Unknown User
AUTH 08/13/2003 14:13:24 I 0276 2212 External DB [NDSAuth.dll]: User cisco123
waiting for lock
AUTH 08/13/2003 14:13:24 I 0276 2212 External DB [NDSAuth.dll]: User cisco123
waiting in lock
AUTH 08/13/2003 14:13:24 I 0276 2212 External DB [NDSAuth.dll]: Initializing thread
0 for tree ndstest
AUTH 08/13/2003 14:13:24 I 0276 1968 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Got work
AUTH 08/13/2003 14:13:24 I 0276 1968 External DB [NDSAuth.dll]: Response 1 from Tree
ndstest
AUTH 08/13/2003 14:13:24 I 0276 1968 External DB [NDSAuth.dll]: Thread 0 for tree
ndstest Waiting for work
AUTH 08/13/2003 14:13:26 I 5094 2220 Worker 3 processing message 275.
AUTH 08/13/2003 14:13:26 I 5081 2220 Start RQ1012, client 27 (127.0.0.1)
AUTH 08/13/2003 14:13:26 I 5081 2220 Done RQ1012, client 27, status 0
AUTH 08/13/2003 14:13:26 I 5094 2220 Worker 3 processing message 276.
AUTH 08/13/2003 14:13:26 I 5081 2220 Start RQ1031, client 27 (127.0.0.1)
AUTH 08/13/2003 14:13:26 I 5081 2220 Done RQ1031, client 27, status 0
! In the following line, the code 102 is an indication that user authentication failed
! either due to bad username or wrong password.
AUTH 08/13/2003 14:13:30 I 0276 2212 External DB [NDSAuth.dll]: Back from Wait for
user cisco123 with code 102
! Eventually will timeout
AUTH 08/13/2003 14:13:30 I 0276 2212 External DB [NDSAuth.dll]: Timeout trying User
cisco123
AUTH 08/13/2003 14:13:30 I 0276 2212 External DB [NDSAuth.dll]: User cisco123 out
from lock
AUTH 08/13/2003 14:13:30 I 0276 2212 External DB [NTAuthenDLL.dll]: Starting
authentication for user [cisco123]
! Following lines indicate that NT/2K domain is also configured next in order, so
! attempting authentication to NT/2K domain as well and eventually fails.
AUTH 08/13/2003 14:13:30 I 0276 2212 External DB [NTAuthenDLL.dll]: Attempting NT/
2000 authentication
AUTH 08/13/2003 14:13:30 E 0276 2212 External DB [NTAuthenDLL.dll]: NT/2000
authentication FAILED (error 1326L)
AUTH 08/13/2003 14:13:30 I 1547 2212 Unknown User 'cisco123' was not authenticated

Wrong Group Mapping
After successful user authentication, the user is mapped to a specific CS ACS group. Two things determine which CS ACS group the user is mapped to: the Novell NDS group or groups the user belongs to, and the CS ACS group mapping configuration under the External Database Configuration page. If there are problems with proper group assignment by CS ACS after successful Novell NDS user authentication, analyze the auth.log file to find out which NDS database groups a specific user belongs to, and if the same group or groups are mapped to the desired CS ACS group. Examine the following example. Assume that the user belongs to all the following groups and maps to the CS ACS Group 10:

superuser.xyz

http_only.xyz

http_ftp.xyz

http_netmeeting.xyz

Analyze the log as shown in Example 13-5.

Example 13-5. Sample Output: User Saad Belongs to Multiple Groups That Do Not Match with the Group Mapped to CS ACS
AUTH 10/13/2004 10:20:49 I 0259 1340 External DB [NDSAuth.dll]: Initializing
thread 0 for tree XYZ
AUTH 10/13/2004 10:20:49 I 0259 0676 External DB [NDSAuth.dll]: Thread 0 for
tree XYZ Got work
AUTH 10/13/2004 10:20:52 A 0259 0676 External DB [NDSAuth.dll]: Login
Attempt: Context 'MKT.DH.XYZ' User 'saad.MKT.DH.XYZ'
Password 'saad' result 0
AUTH 10/13/2004 10:20:52 I 0259 0676 External DB [NDSAuth.dll]:
Authenticated saad.MKT.DH.XYZ, Response 0
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: Back from
Wait for user saad with code 0
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: Response 0
from successful Tree XYZ
AUTH 10/13/2004 10:20:52 I 0259 0676 External DB [NDSAuth.dll]: Response 0
from Tree XYZ
AUTH 10/13/2004 10:20:52 I 0259 0676 External DB [NDSAuth.dll]: Thread 0 for
tree XYZ Waiting for work
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: Added
'Everyone.MKT.DH.XYZ' to Group List for user:
saad.MKT.DH.XYZ
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: Added
'http netmeeting.XYZ' to Group List for user:
saad.MKT.DH.XYZ
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: There were 2
Groups for this user: saad.MKT.DH.XYZ
AUTH 10/13/2004 10:20:52 I 0259 1340 External DB [NDSAuth.dll]: User saad
authenticated into group 0

So, from Example 13-5, you see that user saad belongs to NDS groups "Everyone.MKT.DH.XYZ" and "http_netmeeting.XYZ". Thus, the user does not meet the requirements to be mapped to group 10 on CS ACS, as both of the groups are not mapped on the CS ACS to group 10. As any unmatched group defaults to the CS ACS Default Group, saad is mapped to Group 0. So, the user must belong to all the NDS groups in the mapping, to be mapped into the configured CS ACS group, not just one.

On CS ACS to map this user into group 10, you need a map, which has one of the following combinations of NDS groups:

Everyone.MKT.DH.XYZ

http_netmeeting.XYZ

Everyone.MKT.DH.XYZ' and 'AAA_http netmeeting.XYZ'

It does not matter if a user also belongs in other NDS groups, in addition to those listed in the mapping, but the user must belong in all the NDS groups listed in a mapping to be mapped to a proper CS ACS group.

CS ACS with ACE Server (Secure ID [SDI]) Integration
Cisco Secure ACS can integrate with a few token servers, but this chapter discusses only the ACE server. The ACE server is also known as the SDI server, so both names will be used interchangeably throughout this chapter. Because the implementation of other token servers is very similar to the implementation of the ACE server, the discussion of ACS integration with ACE is applicable for the other token servers as well. The SDI server can be installed on the same machine on which Cisco Secure ACS is running, or on a separate machine. ACE client software is required on the system running Cisco Secure ACS software.

Installation and Configuration Steps
Use the following steps to install and configure CS ACS with SDI Software

Step 1. Install the ACE server as per ACE direction.

Step 2. Bring the ACE server into host configuration mode (run sdadmin).

Step 3. Be sure you have configured the hostname/ip-address of Cisco Secure ACS system as a client in the ACE server setup. This can be verified under Client > Edit Client from ACE Server Host configuration window. For CS ACS Windows client, encryption should be Data Encryption Standard (DES), because the client is Windows, and you have to choose Net OS Client. When you click the User Activations tab, you must see the SDI user under Directly Activated Lists.

Step 4. Be sure the user is activated on the clientthe client is the system on which Cisco Secure ACS is installed. This can be verified under Users > Edit Users > Client Activations. In this window you will see a list of available clients. Choose the right one and move them under Clients Directly Activated On.

Step 5. Be sure the CS ACS client and the SDI server can perform forward and reverse lookups of each other (that is, ping by name or IP).

Step 6. Copy the SDI server's sdconf.rec to the CS ACS client; this can reside anywhere on the CS ACS client.

Step 7. The installation of the ACE client on Windows may differ slightly by version. Run agent.exe to initiate the installation process of the ACE client. During installation, when asked to install Network Access Protection Software, answer No, and leave the root certificate box blank. Then go to Next. When prompted, specify the path to the sdconf.rec file, including the file name.

Step 8. After the client installation and reboot, go to Windows Control Panel > SDI Agent > Test Authentication with Ace Server > Ace/Server Test Directly and enter the username, code, and card configured on the Ace server to perform an authentication test and check the communication between the SDI client and the server. If this test does not work, it means the SDI client is not communicating with the SDI server. It also means the CS ACS Windows will not be able to communicate with the SDI Server. This is because CS ACS uses an SDI client interface to communicate with the SDI server.

Step 9. Then install CS ACS on Windows as usual.

Step 10. From Navigation, go to External User Databases > Database Configurations > Configure. ACS should be able to find the SDI Dynamic Linked Library (DLL).

Step 11. Go back to External User Databases. Click on Unknown User Policy and check the second radio button. Then move the SDI database from External Databases to Selected databases.

Step 12. Go back to External User Databases and click on Database Group Mapping > SDI Database > Cisco Secure ACS group to pick the group that will be mapped to SDI group.

Step 13. Go to Group setup and edit the settings for the group that was mapped to SDI. In this case, it is Default Group. Add appropriate attributes for TACACS+ & RADIUS depending on what kind of service the user will use (either Shell or PPP).

Troubleshooting Steps
Use the following step-by-step procedures to troubleshoot the SDI issues with CS ACS:

Step 1. First, authenticate the user with the ACE test agent.

Step 2. If this works, confirm that the card is synchronized with the database. Be sure to use DES encryption on the SDI server when the card is initialized. Choosing SDI will not work.

Step 3. If this does not work, resynchronize from the Token menu in host configuration mode. Click on Token > Edit Token, and then choose the token that you want to resynchronize. You will have a menu to resynchronize.

Step 4. Next, bring up the activity monitor (Report > Log Monitor > Activity Monitor) on the ACE server while attempting Telnet authentication to a device.

Step 5. Then check to see if there are any errors on the activity monitor on the ACE server.

Step 6. If the ACE server works, but there is a problem with the dial users, check the settings on the network access servers (NASs) to be sure that Password Authentication Protocol (PAP) is configured. Then try connecting as a non-SDI user.

Step 7. If that works, connecting as an SDI user should work. Put the username in the username tab and the passcode in the password tab on Dial-up Networking.

Step 8. If the client from where you are dialing is configured to bring up the post terminal screen after dialing, then be sure the following AAA statement is on the NAS:

aaa authentication ppp default if-needed tacacs+/Radius

The key is to use >if-needed>. This means that if the user is already authenticated by the following AAA statement:

aaa authentication login default tacacs+/radius

then you do not have to authenticate the user again when doing PPP. This also applies when using the normal PAP password.

Here are some common problems that you might face with SDI and CS ACS integration:

The ACE log displays the message "Passcode accepted", but the user still fails Check the CS ACS Failed Attempts log to determine the cause of the problem. The failure could be due to authorization issues.

The ACE log displays the message "Access Denied, passcode incorrect" This is an ACE problem with the passcode. During this time, the CS ACS Failed Attempts log shows either the message External DB auth failed or External DB user invalid or bad password.

The ACE log displays the message "User not in database" Check the ACE database. During this time, the CS ACS Failed Attempts log shows either the message External DB auth failed or External DB user invalid or bad password.

The ACE log displays the message "User not on agent host" This is an ACE configuration problem. To solve this problem, configure the user on the agent host.

The CS ACS log displays the message "External database not operational" If the ACE log does not show any attempts, confirm the operation with the ACE client test authentication and check to be sure that the ACE/server authentication engine is running.

The CS ACS log displays the message "CS user unknown" or "Cached token rejected/expired" with nothing in the ACE log If the network device is sending a Challenge Handshake Authentication Protocol (CHAP) request and the CS ACS does not have an enumerated ACE user with a separate CHAP password, CS ACS does not send the user to ACE because token-only authentication requires PAP.

Replication Issues
Replication allows the CS ACS Server to maintain distributed databases. This helps the NAS to improve fault tolerance (by providing a backup server) or to improve performance (by sharing throughput across several servers). Replication can be configured as a straightforward master-to-slave relationship, or as a pipeline, or even as a tree in which each slave automatically replicates to its children upon receipt of replicated data from its parent.

Configuration
Replication is configured by the GUI. The GUI is used on both the master and slave to configure both ends of the replication.

The following are the steps required for replication on the master (IP Address 10.1.1.1) and the slave (IP address 30.1.1.1). Use the following steps to configure the master CS ACS:

Step 1. Log in to the primary CS ACS server GUI.

Step 2. Turn on Distributed System Settings and enable Cisco Secure Database Replication optionsfound under Interface Configuration->Advanced Options.

Step 3. In the Network Configuration section, add each secondary server to the AAA Servers table as shown in Figure 13-5. The Traffic Type should be left defaulting to inbound/outbound unless there is a good reason to do otherwise.

Figure 13-5. Slave CS ACS Server Entry Configuration on the Primary CS ACS Server

[View full size image]

Step 4. In the navigation bar, click System Configuration. Then click Cisco Secure Database Replication, which brings up the Database Replication Setup page.

Step 5. Select the Send check box for each database component to send to the secondary server as shown in Figure 13-6.

Figure 13-6. Replication Component Configuration on the Master CS ACS

[View full size image]

Step 6. Select a scheduling option from one of the four options: Manually, Automatically Triggered Cascade, Every X Minutes, or At Specific times. To set up Auto Replication, you must not select manually, and the Scheduling Option must be set up on Master, not on the slave.

Step 7. Under the Replication Partners, add the secondary CS ACS server to the Replication Partner column as shown in Figure 13-7.

Figure 13-7. Replication Partner Configuration on Master

[View full size image]

Step 8. Click Submit. Note that Accept Replication from does not have any meaning on the master.

Use the following steps to configure steps required the slave CS ACS server:

Step 1. Follow the preceding Steps 1-4, which were outlined for the master server.

Step 2. Click the Receive check box for each database component to be received from a primary server as shown in Figure 13-8.

Figure 13-8. Replication Components Configuration on the Slave

[View full size image]

Step 3. Leave the Scheduling Option set to Manually.

Step 4. Do not add the primary server to the Replication Partner column, under the Replication Partners; the replication partner column should be blank as shown in Figure 13-9.

Figure 13-9. Replication Partners Configuration on the Slave

[View full size image]

Step 5. From the Accept Replication From drop-down list, select Master Server. If you have more than one Master server, select Any Known Cisco Secure ACS for Windows 2000/NT Server.

Step 6. Click Submit.

Troubleshooting Steps
Before getting into the details of some of the common problems that you might face with replication, it is useful to examine the internal workings of replication.

On the master, a dedicated thread within the CSAuth service continually monitors the time and controls the outbound replication when due. The following actions are performed:

1. Lockout and wait for any open transactions to complete (authentication's and admin's updates).

2. Dump the required registry keys and copy/compress required files to a temporary location.

3. Release lockouts, allowing normal service to resume.

For each child replication partner, CSAuth then performs the following tasks:

1. Connects to remote CSAuth server.

2. Exchanges version and component specifications.

3. Copies permitted files onto remote AAA server.

4. Initiates replication take-up on the remote AAA server. This request also specifies where the files are located on the remote AAA server.

On the slave, the CSAuth service performs the following task:

1. Opens a connection as per any client request.

2. Responds to the master's request for version and replication information checks by verifying that the two servers are running the same software version. The master will have sent a list of components it is allowed to send. The slave then removes any components it is not allowed to receive. The resultant list is returned to the master.

3. Performs remote file copy operations as the master transmits the replication set.

When the slave receives the replication take-up command from the master, it performs the following:

1. Lockout and wait for any open transactions to complete (authentications and administrator updates).

2. Load the required Registry keys. Uncompress and save required files from the temporary location.

3. Release lockouts, allowing normal service to resume.

4. Restart any services as configured in the Registry (default CSRadius and CSTacacs).

5. Kick the replication thread so that if so configured, "cascade" replication occurs.

To view the actions performed by the CSAuth service for replication as described in the previous steps, analyze the auth.log file. The replication CSV file gives a summary version of the status of replication. Therefore, to troubleshoot the replication issues, first look at the CSV file, which is under Reports & Activity, or in the logs directory of the CS ACS installation. If the problem cannot be resolved with the CSV file, you can analyze the auth.log (...\csauth\logs\auth.log), which shows details of failure (be sure to turn logging to FULL).

To see only the replication log on auth.log file, search for string "dbreplicate(out)" on the master and "dbreplicate(in)" on the slave server. Example 13-6 shows the output of auth.log file on the Master Server.

Example 13-6. Replication Only Log on the Master Server
DBReplicate(OUT) attempting to exchange sync info with host acs1
DBReplicate(OUT) attempting to tx files to host acs1
DBReplicate(OUT) - one or more files could not be sent to acs1
DBReplicate(OUT) to host acs1 was denied
Etc.

The following are some of the issues that you might encounter, and their resolutions.

Compatible version numbers Both ends of a replication must be running the same version. However, they may be running different builds of the same version, that is, 2.3(1.24) and 2.3(1.29). This has caused a few problems in clients replicating from 2.3 betas to 2.3 FCS systems due a change in Registry encryption. This might be rectified in the future but is worth bearing in mind.

Master being rejected by the slave The slave AAA server may reject connection attempts from the master due to the master PC having several network adapter cards. This is because the slave can see the IP address of the wrong card, that is, it does not see the one configured in its network configuration. Either the second card can be removed, or another dummy AAA server (with the second IP address) can be added.

Secret value/shared key problems Both AAA servers must share the same secret key.

Proxy entries on slave overwritten If you use replication to update several distributed masters, each with its own proxy distribution table, you might see that the slave will keep on losing its proxy table. This is a misconfiguration, and can be resolved by unchecking the component Distribution Table on the tx/rx list.

Replication timeout After issuing the replication take-up request, the master waits up to 5 minutes before assuming that the slave died and reporting it as such. Should the slave be configured so that it uses RDBMS synchronization, it is possible that a large number of transactions could block the slave from accepting the replication before the master gives up. Try to avoid this type of setting.

Master CS ACS entry is missing on the slave server In the auth.log file of the secondary server, you may see the following message:

Worker 2 message from unknown host x.x.x.x - closing conn 41

So, on the slave, configure the master CS ACS entry.

CS ACS's own server entry is missing from the server list When CS ACS is installed, its own entry is created as an AAA server. If you remove that, you may run into problems with the CS ACS replication, and the auth.log could display the following message:

"ERROR Inbound database replication aborted - check IP address for this AAA
Server"

This appears when the AAA server's list does not contain the CS ACS machine itself. You need to investigate further the Registry of both master and slave. After checking the Registry of both master and slave, you might find that the slave machine on the AAA Servers list has only the master configured and does not have its own definition. On the slave CS ACS, only MASTER (Master CS ACS) is configured as an AAA server (type=1) as shown in Example 13-7.

Example 13-7. Registry on the Slave Server
[HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\CiscoAAAv3.1\Hosts\MASTER]
"key"=hex:36,2e,70,77,e6,24,01,37,73,59,da,d9,b3,61,1d,d9,de,47,79,e2,28,b4,cd,\
27,42,11,7d,a4,c9,6e,bd,85
"vendor"=dword:ffffffff
"protocol"=dword:00000063
"type"=dword:00000001
"direction"=dword:00000003
"acct Packet Filter"=dword:00000005
"network device group"=dword:00000006
"ip"=hex(7):31,00,37,00,32,00,2e,00,32,00,35,00,2e,00,35,00,37,00,2e,00,39,00,\
31,00,00,00,00,00
"lastModified"=hex:50,7c,26,f8,fb,0d,c3,01

On the master server, there are two AAA servers configured : MASTER(the Master itself) and SLAVE(the Slave CS ACS) as shown in Example 13-8.

Example 13-8. Registries of the CS ACS Servers
[HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\CiscoAAAv3.1\Hosts\CPFACS01]
"key"=hex:eb,e5,07,d8,98,ad,fe,5b,f2,88,81,e8,83,1f,e0,00,36,d6,57,03,f7,fd,dc,\
5b,61,89,6a,a6,a8,78,b9,5b
"vendor"=dword:ffffffff
"protocol"=dword:00000063
"type"=dword:00000001
"direction"=dword:00000003
"acct Packet Filter"=dword:00000005
"network device group"=dword:00000000
"ip"=hex(7):31,00,37,00,32,00,2e,00,32,00,35,00,2e,00,35,00,37,00,2e,00,39,00,\
31,00,00,00,00,00
"lastModified"=hex:80,06,62,dd,e5,07,c3,01

[HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\CiscoAAAv3.1\Hosts\HQACS02]
"key"=hex:58,a5,28,70,6e,50,f6,80,64,7c,fe,1f,70,c1,b1,bb,8d,d7,0f,1f,7a,11,ac,\
64,86,b3,4a,c9,a5,37,db,a4
"vendor"=dword:ffffffff
"protocol"=dword:00000063
"type"=dword:00000001
"direction"=dword:00000003
"acct Packet Filter"=dword:00000005
"network device group"=dword:00000006
"ip"=hex(7):31,00,37,00,32,00,2e,00,32,00,35,00,2e,00,35,00,37,00,2e,00,39,00,\
32,00,00,00,00,00
"lastModified"=hex:c0,be,de,ab,fb,0d,c3,01

Add a profile of the slave CS ACS to the slave CS ACS AAA Server's list to fix the problem.

Bi-directional replication between two CS ACS Servers Bi-directional replication is not supported. If you configure the bi-directional replication, you will see messages like the one in Example 13-9 in the auth.log file and the replication will fail. So avoid configuring bidirectional replication.

Example 13-9. Auth.log Output for Bidirectional Replication
03/22/2005 21:26:19 ERROR Inbound database replication from ACS
'dumb' denied
03/22/2005 21:36:36 INFO Outbound replication cycle starting...
03/22/2005 21:36:42 ERROR ACS 'dumb' has denied replication request

NAT or Firewall device between the replication partners If there is a Network Address Translation (NAT) or firewall device between the replication partners, you may see the following message on the auth.log

denied - shared secret mismatch

From CS ACS version 3.1 and above, the IP address is embedded in the data and used as part of the server verification process. Hence, if the NAT device changes the source IP of the server, replication no longer works. To get around the problem, you may want to configure the firewall or NAT device so that it keeps the same source IP address. In addition to taking care of the NAT issue, be sure to allow TCP/2000 for the communication to take place.

Replication is not working for some components A master CS ACS has a dirty flag for each replication component. If no data changes on the master between replication cycles, nothing is replicated. To see which components are being replicated, be sure that user.dat, user.idx, and varsdb.mdb are being transmitted. The users are listed in user.dat. The advanced settings for users and groups are in varsdb.mdb.

Slave server does not have enough space If you have a 10-MB database that needs to be replicated to a slave CS ACS, there must be enough space to hold the compressed file set, you need space in temp during de-compress, and you need space for uncompressed files. For a 10-MB database, 50-MB should be comfortable.

Network Access Restrictions (NARs) Issues
Network Access Restrictions (NARs) allow you to define additional authorization conditions that must be met before a user can gain access to the network. Even though it is an authorization condition, NAR is indeed an authentication process.

Cisco Secure ACS supports two basic types of network access restrictions:

IP-based restrictions in which the originating request relates to an existing IP address

Non-IP-based filters for all other cases in which the automatic number identification (ANI) may be used

A non-IP-based NAR is a list of permitted or denied "calling"/"point of access" locations that you can employ in restricting an AAA client when you do not have an IP-based connection established. The non-IP-based NAR generally uses the calling line ID (CLI) number and the Dialed Number Identification Service (DNIS) number.

Configuration Steps
To illustrate the functionality of NAR, consider the following example. Table 13-4 shows how users belonging to a default group can connect via Router1. Users in Group1 can connect only via AP11 and Group2 users get access via Router2 and AP11.

Table 13-4. Configuration Metric Group
User
Router1
Router2
AP11

Default
Laci
Pass
Fail
Fail

Group1
Magi
Fail
Fail
Pass

Group2
Torri
Fail
Pass
Pass

This can be accomplished in several ways: using per "User defined Network Access Restriction" and using "Shared Profile Component for NAR", whereas all NAR is applied on Group-Settings.

The following are the configuration steps for NAR:

Step 1. Define Router1, Router2, and AP11 as AAA client under AAA client, if not defined already.

Step 2. Define users listed in Table 13-4 and map them to corresponding groups.

Step 3. Go to each group's settings and under Access Restrictions, define the parameters as shown in Figure 13-10 for all three devices such as Default Group, which allows only Router1.

Figure 13-10. NAR Configuration for Group

[View full size image]

Step 4. If Shared Profile components is the component that is used instead, then from Navigation, click on Shared Profile Component, and define three NARs: NAR-AP, NAR-router1, and NAR-router2 in a manner similar to that shown in Figure 13-11 for Router1. The name of the NAR is NAR-router1.

Figure 13-11. Shared Profile Component for NAR

[View full size image]

Step 5. Now apply the NAR defined in Step 4 on all three groups under network access restrictions in a manner similar to that shown in Figure 13-12, which is configured for Default Group.

Figure 13-12. NAR Applied on Default Group

[View full size image]

Note

This setup can also be extended to use Network Device Groups (NDG). Configuration steps are exactly the same, but instead of selecting NAS, select NDG.

Troubleshooting Steps
The best way to troubleshoot the NAR issues is to use Failed Attempts or Passed Authentications reports to understand why access was or was not granted to a certain user. Usually, the caller ID, network access server (NAS) port, and NAS IP address fields are available and can be used to debug the session. When the reason for acceptance or denial is unclear, you can add the Filter Information field to these reports (both to Failed Attempts and Passed Authentications). This field will provide additional data. However, remember that you must use the Shared Profile Component (SPC) NARs configuration to get this additional information. With traditional NARs, it is hard to find the cause of acceptance or denial, as the first message (No Filter Activated) always appears regardless of the results. For additional details, look at the RDS.log or TCS.log and see how the packet is coming to the CS ACS Server from NAS and if it is getting forwarded to the CSAuth service, which shows the information on auth.log.

Example 13-10 shows the auth.log file for successful authentication.

Example 13-10. Snippet of auth.log File Shows NAR Passing
03/22/2005,18:41:21,Authen OK,laci,Default Group,10.0.0.2,66,10.0.0.171,,,,,,All
Access Filters Passed.,,router1,,,No,laci,cisco,,,
03/22/2005,18:42:08,Authen OK,torri,Group 2,10.0.0.2,66,10.0.0.172,,,,,,Access
Filter NAR-router2 from Group 2 permitted on Filter Line: 'router2 (Port=*) (IP=*)'.
This is sufficient to satisfy an 'Any Selected' SPC NAR
config.,,router2,,,No,torri,cisco,,,

03/22/2005,18:42:59,Authen OK,magi,Group 1,0040.9638.8e9a,99,10.0.0.1,,,,,,All
Access Filters Passed.,,ap11,,,No,magi,cisco,,,

03/22/2005,18:43:10,Authen OK,torri,Group 2,0040.9638.8e9a,99,10.0.0.1,,,,,,Access
Filter NAR-AP from Group 2 permitted on Filter Line: 'ap11 (Port=*) (CLI=*)
(DNIS=*)'. This is sufficient to satisfy an 'Any Selected' SPC NAR
config.,,ap11,,,No,torri,cisco,,,

Example 13-11 shows the auth.log file for unsuccessful authentication.

Example 13-11. Snippet of auth.log File Shows NAR Failing
03/22/2005,18:41:39,Authen failed,magi,Group 1,10.0.0.2,User Access
Filtered,,,66,10.0.0.171,Access Filter NAR-AP from Group 1 denied on Filter Line:
'* (Port=*) (IP=*)'. This is sufficient to reject an 'All Selected' SPC NAR
config.,,,,,,,router1,,,,No,magi,cisco,,,

03/22/2005,18:41:44,Authen failed,torri,Group 2,10.0.0.2,User Access
Filtered,,,66,10.0.0.171,No Access Filters
Passed.,,,,,,,router1,,,,No,torri,cisco,,,

03/22/2005,18:41:57,Authen failed,laci,Default Group,10.0.0.2,User Access
Filtered,,,66,10.0.0.172,Access Filter NAR-router1 from Default Group Did not permit
any criteria. This is sufficient to reject an
'All Selected' SPC NAR config.,,,,,,,router2,,,,No,laci,cisco,,,
03/22/2005,18:42:03,Authen failed,magi,Group 1,10.0.0.2,User Access
Filtered,,,66,10.0.0.172,Access Filter NAR-AP from Group 1 denied on Filter Line:
'* (Port=*) (IP=*)'. This is sufficient to reject an 'All Selected' SPC NAR
config.,,,,,,,router2,,,,No,magi,cisco,,,

03/22/2005,18:42:45,Authen failed,laci,Default Group,0040.9638.8e9a,User Access
Filtered,,,97,10.0.0.1,Access Filter NAR-router1 from Default Group denied on Filter
Line: '* (Port=*) (CLI=*) (DNIS=*)'. This is sufficient to reject an 'All Selected'
SPC NAR config.,,,,,,,ap11,,,,No,laci,cisco,,,

The following link contains some interesting and useful discussion about NAR and how to troubleshoot it efficiently:

http://www.cisco.com/en/US/products/sw/secursw/ps2086/products_white_paper09186a00801a8fd0.shtml

Downloadable ACL Issues
You can download ACL from the CS ACS Server to NAS to control which resources the user can access after getting access to the network. There are three ways to configure this, as described in the sections that follow.

Downloading ACL per User Basis Using Filter-id
With this option, you need to define the ACL in the NAS, and on the CS ACS server, and you need to define the name of the ACL using the Internet Engineering Task Force (IETF) RADIUS attribute 11 as shown in Figure 13-13.

Figure 13-13. ACL Named Defined with IETF Radius Attribute

[View full size image]

Using Cisco AV-Pair
On the NAS, you must have authorization turned on; otherwise, the AV pair will not be applied on the NAS. On CS ACS, you need to configure this under Group Profile. Click on Group Setup > Select a Group from drop-down, and then click on Edit Settings. In the Group Setup page, select Cisco IOS/PIX RADIUS from the Jump To drop-down. Then check "[009\001] cisco-av-pair" box, and define your ACL in the rectangle text box underneath as shown in Figure 13-14.

Figure 13-14. ACL Defined Using Cisco AV Pair

[View full size image]

Note

The variable x (ip:inacl# . . .) must be defined with different numbers (for example 1, 2, 3 and so on) if multiple ACL entries are defined in the ACL. This ensures the order of the ACE in the ACL when downloaded to the NAS. If the same number is used for variable x for multiple ACL entries, then it may work, but the order of ACL entries will not be maintained, which may cause unexpected problems.

Using Shared Profile Components
If you are running CS ACS 3.0 and 3.1, under the Shared Profile component, the only option available to download ACL is for the PIX Firewall, which is called "Downloadable PIX ACL." From CS ACS version 3.2 and above, "Downloadable IP ACL" is available under "Shared Profile Components", which is supported by PIX firewall, VPN Concentrator 4.0 code and above, and IOS version 12.3T and above.

The following sequence of events occurs if ACLs via Shared Profile Components are used:

1. The first stage is to download the name and version of the ACL where the device validates it against the local ACL cache.

2. In the second stage, the device (if needed) requests the ACL content, where CS ACS uses inacl for downloading the ACL.

The following step-by-step example shows the user fred when the Shared Profile component ACL is configured.

Step 1. Access a request from PIX (initial user authentication).

User-Name[1] = "fred"
User-Password[2] =
...

...

Step 2. Access an acceptance from CS ACS (authentication response and ACL set assignment).

AV-Pair[26/9/1] ="ACS:CiscoSecure-Defined-ACL=#ACSACL#-myAcl-
1e45bc4890fa12b2
...

...

Step 3. Access a request from PIX (initiation of ACL download).

User-Name = "#ACSACL#-myAcl-1e45bc4890fa12b2"
...

...

Step 4. Access a challenge from CS ACS (first ACL fragment returnedmore to follow).

State[24] = "TBD"
AV-Pair[26/9/1] = "ip:inacl#1 = permit tcp any anyestablished"
AV-Pair[26/9/1] = "ip:inacl#2 = permit ip any any"
...

...
AV-Pair[26/9/1] = "ip:inacl#60 = deny icmp 2.2.2.0 0.0.0.255
9.9.9.00.0.0.255"

Troubleshooting Steps
Most of the time, downloadable ACL issues arise from mis-configuration either on the NAS or on the CS ACS server. So, first you must always perform a sanity check on the configuration, if there is any issue with downloadable ACL. Analyze auth.log and RDS.log files to find out what information CS ACS is sending to the NAS as reply attributes for authorization. Then look at the debug information on the NAS to see if the NAS understands the ACL that CS ACS is sending. Keep the following points in mind when you configure downloadable ACL:

1. Perform a sanity check on the configuration both on the NAS and CS ACS server.

2. Be sure to have the authorization turned on for the NAS; otherwise, even though CS ACS sends the ACL name or the ACL itself, NAS will not install it. In the authorization debug, you will see that ACL is downloading from CS ACS, but when you execute show access-list command, you will not see the ACL being installed.

3. If using filter ID to download only the ACL name to NAS, be sure that ACL is defined locally on the NAS first. Then be sure that the name of the ACL matches on both CS ACS and the NAS. Note that name is case sensitive.

4. When an AV pair is used to download ACL, be sure to define the ACL entries with different numbers to maintain the order of ACL entries.