C H A P T E R 7

Solving Serial Connection Problems

This chapter suggests ways to handle problems associated with serial connections
and is organized as follows:
• Using the show interfaces serial Command, page 7-2
• Using Extended ping Tests, page 7-18
• Using debug Commands, page 7-21
• Troubleshooting Clocking Problems, page 7-22
• Adjusting Buffers, page 7-26
• Using Loopback Tests, page 7-30

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-1
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Using the show interfaces serial Command

The output of the show interfaces serial EXEC command displays statistics
related to the status of serial interfaces. You can use the show interfaces serial
command to diagnose serial line connectivity problems in a WAN environment.

Syntax
The following is the syntax of the show interfaces serial command:

show interfaces serial [slot | interface] [accounting]

where:
• slot (optional) identifies the slot of a particular card in the Cisco ICS 7750
(legal slot values are 1 through 8).
• interface (optional) identifies a particular interface, such as serial 0.
• accounting (optional) displays the number of packets of each protocol type
that has been sent through the interface.

Understanding Command Output

The following sections describe the most important fields of the command output
for troubleshooting serial connections.
• Interface and Line Protocol Status, page 7-5
• Output Drops, page 7-10
• Input Drops, page 7-11
• Input Errors, page 7-12
• Interface Resets, page 7-16
• Carrier Transitions, page 7-17
Example 7-1 shows an example of output from the show interfaces serial
command for a synchronous serial interface. For a description of the most
significant fields as indicated by the callouts—the numbers in square
brackets—see Table 7-1.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-2 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Example 7-1 Output of show interfaces serial Command

Cisco ICS 7750# show interfaces serial

[1] Serial 0 is up, line protocol is up
Hardware is MCI Serial
Internet address is 150.136.190.203, subnet mask is 255.255.255.0
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,rely 255/255, load 1/255
Encapsulation HDLC, loopback not set, keepalive set (10 sec)
Last input 0:00:07, output 0:00:00, output hang never
[2] Output queue 0/40, 0 drops; [3] input queue 0/75, 0 drops
Five minute input rate 0 bits/sec, 0 packets/sec
Five minute output rate 0 bits/sec, 0 packets/sec
16263 packets input, 1347238 bytes, 0 no buffer
Received 13983 broadcasts, 0 runts, 0 giants
[4] 2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 2 abort
[5] 1 carrier transitions
22146 packets output, 2383680 bytes, 0 underruns
0 output errors, 0 collisions, [6] 2 interface resets, 0 restarts

Table 7-1 Descriptions for the show interfaces serial Field

Callout Field Description

1 Interface and line protocol status Indicates whether the interface hardware is
currently active (whether CD1 is present) or that it
has been disabled by an administrator.
2 Output drops Number of packets in the output queue. Each
number is followed by a slash, the maximum size
of the queue, and the number of packets dropped
because of a full queue.
3 Input drops Number of packets in the input queue. Each
number is followed by a slash, the maximum size
of the queue, and the number of packets dropped
because of a full queue.
4 Input errors, including CRC Total number of errors related to no buffer2, runt3,
errors and framing errors. giant4, CRC5, frame6, overrun7, ignored8, and
abort. Other input-related errors can also
increment the count, so this sum might not balance
with the other counts.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-3
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-1 Descriptions for the show interfaces serial Field (continued)

Callout Field Description

5 Carrier transitions Number of times the CD signal of a serial interface
has changed state. For example, if DCD9 goes
down and comes up, the carrier transition counter
increments two times. This field also indicates
modem or line problems if the CD line is changing
state often.
6 Interface resets Number of times an interface has been completely
reset, which can happen if packets queued for
transmission are not sent within several seconds.
On a serial line, a reset can be caused by a
malfunctioning modem that is not supplying the
transmit clock signal or by a cable problem. If the
system notices that the CD line of a serial interface
is up, but the line protocol is down, it periodically
resets the interface in an effort to restart it.
Interface resets can also occur when an interface is
looped back or shut down.
1. CD = carrier detect
2. No buffer errors occur when a packet is discarded because there was no buffer space. Bursts of noise on serial lines are often
responsible for no buffer events. Compare with ignored packets.
3. Runts are packets that are discarded because they are smaller than the medium’s minimum packet size.
4. Giants are packets that are discarded because they are larger than the medium’s maximum packet size.
5. CRC = cyclic redundancy checksum. A CRC error occurs when the CRC generated by the originating station or far-end device
does not match the checksum calculated from the data received. On a serial link, CRCs usually indicate noise, gain hits, or
other transmission problems on the data link.
6. Framing errors occur when packets are received with a CRC error and a noninteger number of octets. On a serial line, this is
usually the result of noise or other transmission problems.
7. Overruns represent the number of times that the receiver hardware is unable to hand received data to a hardware buffer
because the input rate exceeds the receiver’s ability to handle the data.
8. Ignored packets are those that are discarded because the interface hardware does not have enough internal buffers. Compare
with no buffer.
9. DCD = data carrier detect

Cisco ICS 7750 Administration and Troubleshooting Guide

7-4 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Interface and Line Protocol Status

Table 7-2 identifies six possible problem conditions in the interface status line of
the show interfaces serial command display (see callout 1 in Example 7-1):
• Serial x is down, line protocol is down (DTE mode)
• Serial x is up, line protocol is down (DTE mode)
• Serial x is up, line protocol is down (DCE mode)
• Serial x is up, line protocol is up (looped)
• Serial x is up, line protocol is down (disabled)
• Serial x is administratively down, line protocol is down

Table 7-2 show interfaces serial Status Line Conditions

Status Line Possible

Condition Problem Solution
Serial x is down, Typically indicates that the Step 1 Check the LEDs on the CSU/DSU2 to
line protocol is router is not sensing a CD see if CD is active, or insert a breakout
down (DTE1 signal. box (see Chapter 4, “System
mode) Troubleshooting Guidelines”) on the
• Telephone company
line to check for CD signal.
problem—Line is down
or not connected to Step 2 Verify that you are using the proper
CSU/DSU cable and interface (refer to the
Cisco ICS 7750 Hardware Installation
• Faulty or incorrect
Guide).
cabling.
Step 3 Insert a breakout box and check all
• Hardware failure
control leads.
(CSU/DSU)
Step 4 Contact your leased-line or other carrier
service to see if there is a problem.
Step 5 Swap faulty parts.
Step 6 If you suspect that an 3ASI or MRP4
card is faulty, change the serial line to
another interface. If the connection
comes up, there is a problem with the
previously connected interface.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-5
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-2 show interfaces serial Status Line Conditions (continued)

Status Line Possible

Condition Problem Solution
Serial x is up, line • Local or remote router is Step 1 Put the modem, CSU, or DSU in local
protocol is down misconfigured loopback mode and use the show
(DTE mode) interfaces serial command to
• Keepalives are not being
determine whether the line protocol
sent by remote router
comes up.
• Leased-line or other
If the line protocol comes up, a
carrier service
telephone company problem or a failed
problem—noisy line or
remote router is probably the cause.
misconfigured or failed
switch
• Timing problem on Step 2 If the problem appears to be on the
cable—SCTE5 not set on remote end, repeat Step 1 on the remote
CSU/DSU modem, CSU, or DSU.
• Failed local or remote Step 3 Verify all cabling. Make certain that the
CSU/DSU cable is attached to the correct interface,
the correct CSU/DSU, and the correct
• Failed local or remote
telephone company network
hardware
termination point.
Step 4 Enable the debug serial interface
EXEC command.
Step 5 If the line protocol does not come up in
local loopback mode and if the output of
the debug serial interface command
shows that the keepalive counter is not
incrementing, an MRP hardware
problem is likely. Swap the MRP.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-6 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-2 show interfaces serial Status Line Conditions (continued)

Status Line Possible

Condition Problem Solution
Step 6 If the line protocol comes up and the
keepalive counter increments, the
problem is not in the local ASI or MRP.
Troubleshoot the serial line as
described in “Troubleshooting
Clocking Problems” on page 7-22 and
“Using Loopback Tests” on page 7-30.
Step 7 If you suspect that the ASI or MRP
hardware is faulty, change the serial line
to an unused interface. If the connection
comes up, there is a problem with the
previously connected interface.
Serial x is up, line • Missing clockrate Step 1 Add the clockrate interface
protocol is down interface configuration configuration command on the serial
(DCE6 mode) command interface.
• DTE device does not Step 2 Set the DTE device to SCTE mode if
support or is not set up possible. If your CSU/DSU does not
for SCTE mode support SCTE, you might have to
(terminal timing) disable SCTE on the Cisco router
interface. (See “Inverting the Transmit
• Failed remote CSU or
Clock” on page 7-26.)
DSU
• Failed or incorrect cable Step 3 Verify that the correct cable is being
used.
• Router hardware failure Step 4 If the line protocol is still down, there is
a possible hardware failure or cabling
problem. Insert a breakout box and
observe leads. (See Chapter 4, “System
Troubleshooting Guidelines.”)
Step 5 Replace faulty parts as necessary.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-7
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-2 show interfaces serial Status Line Conditions (continued)

Status Line Possible

Condition Problem Solution
Serial x is up, line Loop exists in circuit. The Step 1 Use the show running-config
protocol is up sequence number in the privileged EXEC command. This will
(looped) keepalive packet changes to a enable you to look for any loopback
random number when a loop interface configuration command
is detected initially. entries.
If the same random number is Step 2 If you find a loopback interface
returned over the link, a loop configuration command entry, use the
exists. no loopback interface configuration
command to remove the loop.
Step 3 If you do not find the loopback
interface configuration command,
examine the CSU/DSU to determine
whether they are configured in manual
loopback mode. If they are, disable
manual loopback.
Step 4 Reset the CSU or DSU and inspect the
line status. If the line protocol comes
up, no other action is needed.
Step 5 If the CSU or DSU is not configured in
manual loopback mode, contact the
leased-line or other carrier service for
line troubleshooting assistance.
Serial x is up, line • High error rate because Step 1 Troubleshoot the line with a serial
protocol is down of telephone company analyzer and breakout box. (See
(disabled) service problem Chapter 4, “System Troubleshooting
• CSU or DSU hardware Guidelines.”) Look for toggling CTS 37
and DSR 48 signals.
problem
• Bad router hardware Step 2 Loop CSU/DSU (DTE loop). If the
problem continues, it is likely that there
is a hardware problem. If the problem
does not continue, it is likely that there
is a telephone company problem.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-8 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-2 show interfaces serial Status Line Conditions (continued)

Status Line Possible

Condition Problem Solution
Step 3 Swap out bad hardware as required
(CSU/DSU, switch, local ASI or MRP,
or remote router).
Serial x is • Router configuration Step 1 Check the ASI or MRP configuration
administratively includes the shutdown for the shutdown command.
down, line interface configuration
protocol is down command
• Duplicate IP address Step 2 Use the no shutdown interface
configuration command to remove the
shutdown command.
Step 3 Verify that there are no identical IP
addresses using the show
running-config privileged EXEC
command or the show interfaces EXEC
command.
Step 4 If there are duplicate addresses, resolve
the conflict by changing one of the IP
addresses.
1. DTE = data terminal equipment. Computers and terminals are the most common DTE types.
2. CSU/DSU = channel service unit/data service unit
3. ASI = analog station interface
4. MRP = multiservice route processor
5. SCTE = serial clock transmit external
6. DCE = data communications equipment. Modems (analog) and CSU/DSUs (digital) are the most common DCE types.
7. CTS = Clear To Send
8. DSR = Data Set Ready

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-9
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Output Drops
Output drops appear in the output of the show interfaces serial command when
the system is attempting to hand off a packet to a transmit buffer but no buffers
are available (see callout 2 in Example 7-1).

Symptom Increasing number of output drops on serial link

Possible Cause Input rate to serial interface exceeds bandwidth available on

serial link

Action The following steps are suggested when you encounter this symptom:

Step 1 Minimize periodic broadcast traffic such as routing and Service Advertising
Protocol (SAP) updates by using access lists or by other means. For example, to
increase the delay between SAP updates, use the ipx sap-interval interface
configuration command.
Step 2 Increase the output hold queue size in small increments by using the hold-queue
out interface configuration command.
Step 3 On affected interfaces, turn off fast switching for heavily used protocols. For
example, to turn off IP fast switching, enter the no ip route-cache interface
configuration command. For the command syntax for other protocols, refer to the
Cisco IOS IP Configuration Guide, Release 12.2 and the Cisco IOS IP Command
Reference, Volume 2 of 3: Routing Protocols, Release 12.2.
Step 4 Implement priority queuing on slower serial links by configuring priority lists.
For information on configuring priority lists, refer to the “Congestion
Management” section in the Cisco IOS Quality of Service Solutions Configuration
Guide, Release 12.2 and the Cisco IOS Quality of Service Solutions Command
Reference, Release 12.2 publications.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-10 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Input Drops
Input drops appear in the output of the show interfaces serial command when too
many packets from that interface are still being processed in the system (see
callout 3 in Example 7-1).

Symptom Increasing number of input drops on serial link

Possible Cause Input rate exceeds the capacity of the MRP or input queues
exceed the size of output queues

Action The following steps are suggested when you encounter this symptom:

Step 1 Increase the output queue size on common destination interfaces for the interface
that is dropping packets. Use the hold-queue out interface configuration
command.
Step 2 Reduce the input queue size by using the hold-queue in interface configuration
command to force input drops to become output drops. Output drops have less
impact on the performance of the router than do input drops.

Note Input drop problems typically occur when heavy traffic is being routed
between Ethernet and serial interfaces. ASIs, MRPs, and routers may drop
packets during these congested periods.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-11
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Input Errors
Possible sources for input errors that appear in the show interfaces serial
command output (see callout 4 in Example 7-1) are as follows:
• CRC errors
• Framing errors
• Aborted transmission
The most likely sources are summarized in the list of possible problems that
follows and in Table 7-3.

Note Any input error value for CRC errors, framing errors, or aborts above
1 percent of the total interface traffic suggests a link problem that you should
isolate and repair.

Symptom Increasing number of input errors in excess of 1 percent of total

interface traffic

Possible Cause The following problems can result in this symptom:

• Faulty telephone company equipment
• Noisy serial line
• Incorrect clocking configuration (SCTE not set)
• Incorrect cable or cable too long
• Bad cable or connection
• Bad CSU or DSU
• Bad ASI or MRP hardware
• Data converter or other device being used between MRP and DSU

Note Cisco strongly discourages the use of data converters when you
are connecting an ASI or MRP to a WAN or serial network.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-12 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Action The following steps are suggested when you encounter this symptom:

Step 1 Use a serial analyzer to isolate the source of the input errors. If you detect errors,
it is likely that there is a hardware problem or a clock mismatch in a device outside
the ASI or MRP.
Step 2 Use the loopback and ping tests to isolate the specific problem source. For more
information, see “Using Extended ping Tests” on page 7-18 and “Using Loopback
Tests” on page 7-30.
Step 3 Look for patterns. For example, if errors occur at a consistent interval, they could
be related to a periodic function such as the sending of routing updates.

Table 7-3 Troubleshooting Serial Line Input Errors

Input Error Type Possible

(Field Name) Problem Solution
CRC errors CRC errors occur when Step 1 Ensure that the line is clean enough for
the CRC calculation does transmission requirements. Shield the cable
not pass, indicating that if necessary.
data is corrupted.
• Noisy serial line Step 2 Ensure that the cable is within the
recommended length—no more than 50 ft
• Serial cable too long
(15.24 m), or 25 ft (7.62 m) for a T1 link.
• SCTE mode is not Step 3 Ensure that all devices are properly
enabled on DSU configured for a common line clock. Set
SCTE on the local and remote DSU. If your
• CSU line clock is
CSU/DSU does not support SCTE, see
incorrectly
“Inverting the Transmit Clock” on
configured
page 7-26.
• Ones density problem Step 4 Ensure that the local and remote CSU/DSU
on T1 link (incorrect are configured for the same framing and
framing or coding coding scheme as that used by the
specification) leased-line or other carrier service—for
example, ESF1/B8ZS2.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-13
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-3 Troubleshooting Serial Line Input Errors (continued)

Input Error Type Possible

(Field Name) Problem Solution
Step 5 Contact your leased-line or other carrier
service and have the service perform
integrity tests on the line.
Framing errors A framing error occurs Step 1 Make sure that the physical cable meets the
when a packet does not specified transmission requirements. Shield
end on an 8-bit byte the cable if necessary. Ensure that you are
boundary. Possible causes using the correct cable and that it is firmly
include: seated in its connectors.
• The serial line has Step 2 Ensure that the cable is within the
signal noise on it or recommended length—no more than 50 ft
other interference. (15.24 m), or 25 ft (7.62 m) for a T1 link.
• Improperly designed Step 3 Ensure that all devices are properly
cable; serial cable is configured to use a common line clock. Set
too long; the cable SCTE on the local and remote DSU. If your
from the CSU or DSU CSU/DSU does not support SCTE, see
to the router is not “Inverting the Transmit Clock” later in this
shielded chapter.
• SCTE mode is not Step 4 Ensure that the local and remote CSU/DSU
enabled on the DSU; are configured for the same framing and
the CSU line clock is coding scheme as that used by the
configured leased-line or other carrier service—for
incorrectly; one of the example, ESF/B8ZS.
clocks is configured
for local clocking
• Ones density problem Step 5 Contact your leased-line or other carrier
on T1 link (incorrect service and have the service perform
framing or coding integrity tests on the line.
specification)

Cisco ICS 7750 Administration and Troubleshooting Guide

7-14 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Table 7-3 Troubleshooting Serial Line Input Errors (continued)

Input Error Type Possible

(Field Name) Problem Solution
Aborted Aborts indicate an illegal Step 1 Ensure that all devices are properly
transmission sequence of one bits configured to use a common line clock. Set
(more than 7 in a row). SCTE on the local and remote DSU. If your
CSU/DSU does not support SCTE, see
• SCTE mode is not
“Inverting the Transmit Clock” later in this
enabled on DSU
chapter.
• CSU line clock is Step 2 Shield the cable if necessary. Make sure the
incorrectly cable is within the recommended
configured length—no more than 50 ft (15.24 m), or
25 ft (7.62 m) for a T1 link. (Refer to the
• Serial cable is too
Cisco ICS 7750 Hardware Installation
long or cable from the
Guide.) Ensure that all connections are
CSU or DSU to the
good.
router is not shielded
• Ones density problem Step 3 Check the hardware at both ends of the link.
on T1 link (incorrect Swap faulty equipment as necessary.
framing or coding
specification)
• Packet terminated in Step 4 Lower data rates and observe whether aborts
middle of decrease.
transmission (typical
cause is an interface
reset or a framing
error)
• Hardware Step 5 Use local and remote loopback tests to
problem—bad circuit, determine where aborts are occurring. (See
bad CSU/DSU, or bad “Using Loopback Tests” on page 7-30.)
sending interface on
remote router
Step 6 Contact your leased-line or other carrier
service and have the service perform
integrity tests on the line.
1. ESF = Extended Super Frame
2. B8ZS = binary eight-zero substitution

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-15
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Interface Resets
Interface resets that appear in the output of the show interfaces serial command
are the result of missed keepalive packets (see callout 5 in Example 7-1).

Symptom Increasing number of interface resets on serial link

Possible Cause The following problems can result in this symptom:

• Congestion on link (typically associated with output drops)
• Bad line causing CD transitions
• Possible hardware problem at the CSU, DSU, or switch

Action When interface resets are occurring, examine other fields of the show
interfaces serial command output to determine the source of the problem.
Assuming an increase in interface resets is being recorded, examine the
following fields:

Step 1 If there are a high number of output drops in the show interfaces serial output,
see the section “Output Drops” on page 7-10.
Step 2 Check the carrier transitions field in the show interfaces serial display. If carrier
transitions are numerous while interface resets are being registered, the problem
is probably a bad link or CSU/DSU. Contact your leased-line or carrier service
and swap faulty equipment as necessary.
Step 3 Examine the input errors field in the show interfaces serial display. If input errors
are numerous while interface resets are increasing, the problem is probably a bad
link or CSU/DSU. Contact your leased-line or other carrier service and swap
faulty equipment as necessary.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-16 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using the show interfaces serial Command

Carrier Transitions
Carrier transitions appear in the output of the show interfaces serial command
whenever there is an interruption in the carrier signal, such as an interface reset at
the remote end of a link (see callout 6 in Example 7-1).

Symptom Increasing carrier transitions count on serial link

Possible Cause The following problems can result in this symptom:

• Line interruptions by an external source, such as physical separation of
cabling, T1 alarms, or lightning striking somewhere along the network
• Faulty switch, DSU, ASI, or MRP hardware

Action The following steps are suggested when you encounter this symptom:

Step 1 Check hardware at both ends of the link by attaching a breakout box or a serial
analyzer and testing to determine source of problems. (See Chapter 4, “System
Troubleshooting Guidelines.”)
Step 2 If an analyzer or breakout box does not identify any external problems, check the
ASI or MRP hardware.
Step 3 Swap faulty equipment as necessary.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-17
 Chapter 7 Solving Serial Connection Problems
Using Extended ping Tests

Using Extended ping Tests

The ping command is particularly useful when numerous input errors are being
registered in the output from the show interfaces serial command. (See “Input
Errors” on page 7-12.)
Cisco internetworking devices provide a mechanism to automate the sending of
many ping packets in sequence. Example 7-2 illustrates the menu you can use to
specify extended ping options. This example specifies 20 successive pings (in the
Repeat count field). However, when testing the components on your serial line,
you should specify a much larger number, such as 1000 pings.

Example 7-2 Extended ping Specification Menu

Cisco ICS 7750# ping

Protocol [ip]:
Target IP address: 129.44.12.7
Repeat count [5]: 20
Datagram size [100]: 64
Timeout in seconds [2]:
Extended commands [n]: yes
Source address:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]: 0xffff
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 20, 64-byte ICMP Echos to 129.44.12.7, timeout is 2 seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent, round-trip min/avg/max = 1/3/4 ms

In general, perform serial line ping tests as follows:

Step 1 Put the CSU/DSU into local loopback mode.

Step 2 Configure the extended ping command to send different packet sizes and data
patterns.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-18 78-10169-02 Rev. B0
Chapter 7 Solving Serial Connection Problems
Using Extended ping Tests

Example 7-3 and Example 7-4 illustrate two useful ping tests, an all-zeros
1500-byte ping and an all-ones 1500-byte ping, respectively.

Note The packet size in both examples (1500 bytes) is specified by the
Datagram size field. The data pattern (all zeroes in Example 7-3, all
ones in Example 7-4) is specified by a hexadecimal value in the Data
pattern field.

Example 7-3 All-Zeros 1500-Byte ping Test

Cisco ICS 7750#ping

Protocol [ip]:
Target IP address: 192.169.51.22
Repeat count [5]: 100
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 192.169.51.14
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]: 0x0000
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 192.169.51.22, timeout is 2
seconds:
Packet has data pattern 0x0000
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100),
round-trip min/avg/max = 4/6/8 ms

Example 7-4 All-Ones 1500-Byte ping Test

Cisco ICS 7750#ping

Protocol [ip]:
Target IP address: 192.169.51.22
Repeat count [5]: 100
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 192.169.51.14
Type of service [0]:

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-19
 Chapter 7 Solving Serial Connection Problems
Using Extended ping Tests

Set DF bit in IP header? [no]:

Validate reply data? [no]:
Data pattern [0xABCD]: 0xffff
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 192.169.51.22, timeout is 2
seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100),
round-trip min/avg/max = 4/6/8 ms

Step 3 Examine the show interfaces serial command output and determine whether
input errors have increased. (See “Input Errors” on page 7-12.) If input errors
have not increased, the local hardware (DSU, cable, ASI, and MRP card) is
probably in good condition. Assuming that this test sequence was prompted by the
appearance of a large number of CRC and framing errors, a clocking problem is
likely. Check the CSU/DSU for a timing problem. (See “Troubleshooting
Clocking Problems” on page 7-22.)
Step 4 If you determine that the clocking configuration is correct and is operating
properly, put the CSU/DSU into remote loopback mode.
Step 5 Repeat the ping test and look for changes in the input error statistics.
Step 6 If input errors increase, there is either a problem in the serial line or on the
CSU/DSU. Contact the WAN service provider and swap the CSU/DSU. If
problems persist, contact your technical support representative.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-20 78-10169-02 Rev. B0
Chapter 7 Solving Serial Connection Problems
Using debug Commands

Using debug Commands

The output of debug privileged EXEC commands provides information relating
to protocol status and network activity for many internetworking events.

Caution Use debug commands with care. Enabling debugging can significantly disrupt
the operation of a heavily loaded router. When you finish using a debug
command, remember to disable it with its specific no debug command or with
the no debug all command.

Following are some debug commands that are useful when troubleshooting serial
and WAN problems. For more information about the function and output of each
of these commands, refer to the Cisco IOS Debug Command Reference
publication.
• debug serial interface verifies whether HDLC keepalive packets are
incrementing. If they are not, an ASI card, MRP card, or the network might
have a timing problem.
• debug arp indicates whether an ASI or MRP is sending information about or
learning about other cards s or routers (with ARP packets) on the other side
of the WAN. Use this command when some nodes on a TCP/IP network are
responding but others are not.
• debug ppp negotiation shows Point-to-Point Protocol (PPP) packets
transmitted during PPP startup, where PPP options are negotiated.
• debug ppp packet shows PPP packets being sent and received. This
command displays low-level packet dumps.
• debug ppp errors shows PPP errors (such as illegal or malformed frames)
associated with PPP connection negotiation and operation.
• debug ppp chap shows PPP Challenge Handshake Authentication Protocol
(CHAP) and Password Authentication Protocol (PAP) packet exchanges.
• debug serial packet shows SMDS packets being sent and received. This
display also prints out error messages to indicate why a packet was not sent
or was received erroneously. For SMDS, the command dumps the entire
SMDS header and some payload data when an SMDS packet is transmitted
or received.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-21
 Chapter 7 Solving Serial Connection Problems
Troubleshooting Clocking Problems

Troubleshooting Clocking Problems

Clocking conflicts in serial connections can lead to chronic loss of connection
service or to degraded performance. The following sections discuss issues related
to clocking problems:
• Overview of Clocking, page 7-22
• Causes of Clocking Problems, page 7-23
• Detecting Clocking Problems, page 7-23
• Isolating Clocking Problems, page 7-24
• Solutions to Clocking Problems, page 7-25
• Inverting the Transmit Clock, page 7-26

Overview of Clocking
The CSU/DSU derives the data clock from the data that passes through it. To
recover the clock, the CSU/DSU hardware must receive at least one 1-bit value
for every 8 bits of data that pass through it (this is known as ones density.)
Maintaining ones density allows the hardware to recover the data clock reliably.
Newer T1 implementations commonly use Extended Superframe Format (ESF)
framing with Binary 8-Zero Substitution (B8ZS) coding. B8ZS provides a scheme
by which a special code is substituted whenever eight consecutive zeros are sent
through the serial link. This code is then interpreted at the remote end of the
connection. This technique guarantees ones density independent of the data
stream.
Older T1 implementations use D4 (also known as Superframe Format [SF])
framing and Alternate Mark Inversion (AMI) coding. AMI does not utilize a
coding scheme like B8ZS. This restricts the type of data that can be transmitted
because ones density is not maintained independent of the data stream.
Another important element in serial communications is serial clock transmit
external (SCTE) terminal timing. SCTE is the clock echoed back from the data
terminal equipment (DTE) device (such as an ASI or MRP) to the data
communications equipment (DCE) device (such as a modem or CSU/DSU).

Cisco ICS 7750 Administration and Troubleshooting Guide

7-22 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Troubleshooting Clocking Problems

When the DCE device uses SCTE instead of its internal clock to sample data from
the DTE, it is better able to sample the data without error even if there is a
phase-shift in the cable between the CSU/DSU and the router. Using SCTE is
highly recommended for serial transmissions faster than 64 kbps. If your
CSU/DSU does not support SCTE, see “Inverting the Transmit Clock” on
page 7-26.

Causes of Clocking Problems

In general, clocking problems in serial WAN interconnections can be attributed to
one of the following causes:
• Incorrect DSU configuration
• Incorrect CSU configuration
• Cables out of specification (longer than 50 ft [15.24 m] or unshielded)
• Noisy or poor patch panel connections
• Several cables connected together

Detecting Clocking Problems

To detect clocking conflicts on a serial interface, look for input errors as follows:

Step 1 Use the show interfaces serial command on the ASI, MRPs, or routers at both
ends of the link.
Step 2 Examine the command output for CRC, framing errors, and aborts (see “Input
Errors” on page 7-12).
Step 3 If either of the preceding steps indicates errors exceeding an approximate range
of 0.5 to 2.0 percent of traffic on the interface, clocking problems are likely to
exist somewhere in the WAN.
Step 4 Isolate the source of the clocking conflicts as outlined in “Isolating Clocking
Problems” on page 7-24.
Step 5 Bypass or repair any faulty patch panels.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-23
 Chapter 7 Solving Serial Connection Problems
Troubleshooting Clocking Problems

Isolating Clocking Problems

After you determine that clocking conflicts are the most likely cause of input
errors, the following procedure will help you isolate the source of those errors:

Step 1 Perform a series of ping tests and loopback tests (both local and remote), as
described in the sections “Using Extended ping Tests” on page 7-18 and “Using
Loopback Tests” on page 7-30.
Step 2 Determine which end of the connection is the source of the problem or if the
problem is in the line. In local loopback mode, use different patterns and sizes in
the ping tests (for example, use 1500-byte datagrams). Using a single pattern and
packet size may not force errors to materialize, particularly when a serial cable to
the ASI, MRP, or CSU/DSU is the problem.
Step 3 Use the show interfaces serial EXEC command and determine whether input
error counts are increasing and where they are accumulating.

If input errors are accumulating on both ends of the connection, clocking of the
CSU is the most likely problem.
If only one end is experiencing input errors, there is probably a DSU clocking or
cabling problem.
Aborts on one end suggest that the other end is sending bad information or that
there is a line problem.

Note Always refer to the show interfaces serial command output, and log any
changes in error counts or note that the error count does not change.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-24 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Troubleshooting Clocking Problems

Solutions to Clocking Problems

Possible causes of clocking problems are as follows:
• Incorrect CSU configuration
• Incorrect DSU configuration
• Cable to ASI or MRP out of specification
Table 7-4 describes suggested ways to solve these clocking problems.

Table 7-4 Clocking Problems and Solutions

Possible
Problem Solution
Incorrect CSU configuration Step 1 Determine whether the CSUs at both ends agree on the
clock source (local or line).
Step 2 If the CSUs do not agree, configure them so that they
do. Usually the line is the source.
Step 3 Check the LBO1 setting on the CSU to ensure that the
impedance matches that of the physical line. For
information on configuring your CSU, consult your
CSU hardware documentation.
Incorrect DSU configuration Step 1 Determine whether the CSUs at both ends have SCTE
mode enabled.
Step 2 If SCTE is not enabled on both ends of the connection,
enable it.
For any interface that is connected to a line of 128 kbps
or faster, SCTE must be enabled. If your DSU does not
support SCTE, see “Inverting the Transmit Clock” on
page 7-26.
Step 3 Make sure that ones density is maintained—the DSU
must use the same framing and coding schemes (such as
ESF and B8ZS) used by the leased-line or other carrier
service.
Check with your leased-line provider for information
on their framing and coding schemes.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-25
 Chapter 7 Solving Serial Connection Problems
Adjusting Buffers

Table 7-4 Clocking Problems and Solutions (continued)

Possible
Problem Solution
Step 4 If your carrier service uses AMI coding, either invert
the transmit clock on both sides of the link or run the
DSU in bit-stuff mode. For information on configuring
your DSU, consult your DSU hardware documentation.
Cable to MRP out of Step 1 If the cable is longer than 50 ft (15.24 m), use a shorter
specification cable.
Step 2 If the cable is unshielded, replace it with shielded cable.
1. LBO = Line Build Out

Inverting the Transmit Clock

If you are attempting serial connections at speeds greater than 64 kbps with a
CSU/DSU that does not support SCTE, you might have to invert the transmit
clock on the router. Inverting the transmit clock compensates for phase-shifts
between the data and clock signals.
The specific command used to invert the transmit clock varies between platforms.
To ensure that you are using the correct command syntax for your ASI, MRP, or
router, refer to the “Interface Commands (interface fastethernet - loopback line)”
chapter in the Cisco IOS Interface Command Reference, Release 12.2 publication.

Adjusting Buffers
Excessively high bandwidth utilization results in reduced overall performance
and can cause intermittent failures. However, increasing the bandwidth might not
be necessary or immediately practical. One way to resolve marginal serial-line
overutilization problems is to control how MRPs use data buffers.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-26 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Adjusting Buffers

Caution In general, do not adjust system buffers unless you are working closely with a
Cisco technical support representative. You can severely affect the
performance of your hardware and your network if you incorrectly adjust the
system buffers on MRPs or routers.

Use one of the following three options to control how buffers are used:
• Adjust parameters associated with system buffers
• Specify the number of packets held in input or output queues (hold queues)
• Prioritize how traffic is queued for transmission (priority output queuing)
The configuration commands associated with these options are described in Cisco
IOS configuration guide and command reference publications.
The following section focuses on identifying situations in which you are likely to
use these options to help resolve connectivity and performance problems in serial
and WAN interconnections.

Tuning System Buffers

There are two general buffer types on Cisco ASIs, MRPs, and routers: hardware
buffers and system buffers. Only the system buffers are directly configurable by
system administrators.
The hardware buffers are used as the receive and transmit buffers associated with
each interface and, in the absence of any special configuration, are dynamically
managed by the system software.
The system buffers are associated with the main system memory and are allocated
to different sized memory blocks. A useful command for determining the status
of your system buffers is the show buffers EXEC command. Example 7-5 shows
the output from the show buffers command.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-27
 Chapter 7 Solving Serial Connection Problems
Adjusting Buffers

Example 7-5 show buffers Command Output

Cisco ICS 7750>show buffers

Buffer elements:
401 in free list (500 max allowed)
87777499 hits, 0 misses, 0 created
Small buffers, 104 bytes (total 120, permanent 120):
114 in free list (20 min, 250 max allowed)
70005538 hits, 6 misses, 2 trims, 2 created
Middle buffers, 600 bytes (total 90, permanent 90):
88 in free list (10 min, 200 max allowed)
25696696 hits, 27 misses, 27 trims, 27 created
Big buffers, 1524 bytes (total 90, permanent 90):
90 in free list (5 min, 300 max allowed)
8214530 hits, 15 misses, 366 trims, 366 created
Large buffers, 5024 bytes [2](total 5, permanent 5):
5 in free list (0 min, 30 max allowed)
15017 hits, 12 misses, [1] 16354 trims, 16354 created
Huge buffers, 18024 bytes (total 3, permanent 0):
2 in free list (0 min, 4 max allowed)
297582 hits, 17 misses, 30 trims, 33 created
0 failures [3](0 no memory)

The show buffers command output in Example 7-5 indicates large numbers in the
trims and created fields for large buffers. (See callout 1.) If this is the case, you
can increase your serial link performance by increasing the max-free value
configured for your system buffers.
Use the buffers max-free number global configuration command to increase the
number of free system buffers. The value you configure should be approximately
150 percent of the figure indicated in the total field of the show buffers command
output. (See callout 2.) Repeat this process until the show buffers output no
longer indicates trims and created buffers.
If the show buffers command output shows a large number of failures in the no
memory field (see callout 3), you must reduce the usage of the system buffers or
increase the amount of shared or main memory (physical RAM) on the ASI, MRP
or router (refer to Installing Memory, PVDM, and VPN Modules in ASI Cards,
MRP Cards, and SPE Cards in the Cisco ICS 7750).

Cisco ICS 7750 Administration and Troubleshooting Guide

7-28 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Adjusting Buffers

Implementing Hold Queue Limits

Hold queues are buffers used by each ASI, MRP, or router interface to store
outgoing or incoming packets. Use the hold-queue interface configuration
command to increase the number of data packets queued before the routing device
begins to drop packets.
Use this command to prevent packets from being dropped and to improve serial
link performance under the following conditions:
• You have an application that cannot tolerate drops, and the protocol is able to
tolerate longer delays.
• The interface is very slow (bandwidth is low or anticipated utilization is
likely to sporadically exceed available bandwidth).

Note When you increase the number specified for an output hold queue, you might
need to increase the number of system buffers. The value you should use
depends on the size of the packets associated with the traffic anticipated for
the network.

Using Priority Queuing to Reduce Bottlenecks

Priority queuing is a list-based control mechanism that allows traffic to be
prioritized for each interface. Priority queuing involves two steps:

Step 1 Create a priority list by protocol type and level of priority.

Step 2 Assign the priority list to a specific interface.
Both of these steps use versions of the priority-list global configuration
command. In addition, you can apply further traffic control by referencing
access-list global configuration commands from priority-list specifications. For
examples of defining priority lists and for details about command syntax
associated with priority queuing, refer to the “Congestion Management” section
in the Cisco IOS Quality of Service Solutions Configuration Guide, Release 12.2 and
the Cisco IOS Quality of Service Solutions Command Reference, Release 12.2
publications.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-29
 Chapter 7 Solving Serial Connection Problems
Using Loopback Tests

Note Priority queuing automatically creates four hold queues of varying sizes,
which override any hold queue specification included in your configuration.

Use priority queuing to prevent packets from being dropped and to improve serial
link performance under the following conditions:
• When the interface is slow, many types of traffic are being transmitted, and
you want to improve terminal traffic performance.
• If you have a serial link that is intermittently experiencing very heavy loads
(such as file transfers occurring at specific times), you can use priority lists
to select which types of traffic should be discarded during heavy traffic
periods.
In general, start with the default number of queues when implementing priority
queues. After enabling priority queuing, monitor output drops with the show
interfaces serial EXEC command. If you notice that output drops are occurring
in the traffic queue you have specified to be high priority, increase the number of
packets that can be queued (using the queue-limit keyword option of the
priority-list global configuration command).

Using Loopback Tests

If the output of the show interfaces serial EXEC command indicates that the
serial line is up but the line protocol is down (see “Interface and Line Protocol
Status” on page 7-5), use CSU/DSU loopback tests to determine the source of the
problem. Perform the local loop test first, then the remote test.

Note To use these tests, the internetworking system must be attached to a CSU or
DSU, or to a multiplexer with built-in CSU/DSU functionality. Because there
is no concept of a loopback in X.25 or Frame Relay packet-switched network
(PSN) environments, loopback tests do not apply to X.25 and Frame Relay
networks.

Cisco ICS 7750 Administration and Troubleshooting Guide

7-30 78-10169-02 Rev. B0
 Chapter 7 Solving Serial Connection Problems
Using Loopback Tests

Local Loopback Tests for HDLC or PPP Links

The following is a general procedure for performing loopback tests in conjunction
with built-in system diagnostic capabilities.

Step 1 Place the CSU/DSU in local loop mode (refer to your vendor documentation). In
local loop mode, the use of the line clock (from the T1 service) is terminated, and
the DSU is forced to use the local clock.
Step 2 Use the show interfaces serial EXEC command (see “Interface and Line Protocol
Status” on page 7-5) to determine whether the line status changes from “line
protocol is down” to “line protocol is up (looped),” or if it remains down.
Step 3 If the line protocol comes up when the CSU or DSU is in local loopback mode, it
suggests that the problem is occurring on the remote end of the serial connection.
If the status line does not change state, there is a possible problem in the ASI,
MRP, router, connecting cable, or CSU/DSU.
Step 4 If the problem appears to be local, use the debug serial interface privileged
EXEC command. (See “Using debug Commands” on page 7-21.)
Step 5 Take the CSU/DSU out of local loop mode. When the line protocol is down, the
debug serial interface command output will indicate that keepalive counters are
not incrementing.
Step 6 Place the CSU/DSU in local loop mode again. This should cause the keepalive
packets to begin to increment. Specifically, the values for mineseen and yourseen
keepalives will increment every 10 seconds. This information will appear in the
debug serial interface output.
If the keepalives do not increment, there may be a timing problem on the MRP
card or on the network. For information on correcting timing problems, see
“Troubleshooting Clocking Problems” on page 7-22.
Step 7 Check the local router and CSU/DSU hardware and any attached cables. Ensure
that the cables are within the recommended lengths (no more than 50 ft [15.24 m],
or 25 ft [7.62 m] for a T1 link). Verify that the cables are attached to the proper
ports. Swap faulty equipment as necessary.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-31
 Chapter 7 Solving Serial Connection Problems
Using Loopback Tests

Example 7-6 shows the output from the debug serial interface command for an
HDLC serial connection, with missed keepalives (see callouts 1 through 4)
causing the line to go down (see callout 5) and the interface to reset.

Example 7-6 debug serial interface Command Output

Cisco ICS 7750# debug serial interface

Serial1: HDLC myseq 636119, mineseen 636119, yourseen 515032, line up
Serial1: HDLC myseq 636120, mineseen 636120, yourseen 515033, line up
Serial1: HDLC myseq 636121, mineseen 636121, yourseen 515034, line up
Serial1: HDLC myseq 636122, mineseen 636122, yourseen 515035, line up
Serial1: HDLC myseq 636123, mineseen 636123, yourseen 515036, line up
Serial1: HDLC myseq 636124, mineseen 636124, yourseen 515037, line up
Serial1: HDLC myseq 636125, mineseen 636125, yourseen 515038, line up
Serial1: HDLC myseq 636126, mineseen 636126, yourseen 515039, line up
Serial1: HDLC myseq 636127, mineseen 636127, yourseen 515040, line up
Serial1: HDLC myseq 636128, [1] mineseen 636127, yourseen 515041, line up
Serial1: HDLC myseq 636129, mineseen 636129, yourseen 515042, line up
Serial1: HDLC myseq 636130, mineseen 636130, yourseen 515043, line up
Serial1: HDLC myseq 636131, [2] mineseen 636130, yourseen 515044, line up
Serial1: HDLC myseq 636132, [3] mineseen 636130, yourseen 515045, line up
Serial1: HDLC myseq 636133, [4] mineseen 636130, yourseen 515046, [5] line down

Remote Loopback Tests for HDLC or PPP Links

If you determine that the local hardware is functioning properly but you still
encounter problems when attempting to establish connections over the serial link,
try using the remote loopback test to isolate the problem.

Note This remote loopback test assumes that HDLC encapsulation is being used and
that the preceding local loop test was performed immediately before this test.

Step 1 Put the remote CSU or DSU into loopback mode (refer to the vendor
documentation).
Step 2 Using the show interfaces serial EXEC command, determine whether the line
protocol remains up with the status line indicating “Serial x is up, line protocol is
up (looped),” or if it goes down with the status line indicating “line protocol is
down.”

Cisco ICS 7750 Administration and Troubleshooting Guide

7-32 78-10169-02 Rev. B0
Chapter 7 Solving Serial Connection Problems
Using Loopback Tests

Step 3 If the line protocol remains up (looped), the problem is probably at the remote end
of the serial connection (between the remote CSU/DSU and the remote router).
Perform both local and remote tests at the remote end to isolate the problem
source.
Step 4 If the line status changes to “line protocol is down” when remote loopback mode
is activated, make certain that ones density is being properly maintained. The
CSU/DSU must be configured to use the same framing and coding schemes used
by the leased-line or other carrier service (such as ESF and B8ZS).
Step 5 If problems persist, contact your WAN network manager or the WAN service
organization.

Cisco ICS 7750 Administration and Troubleshooting Guide

78-10169-02 Rev. B0 7-33
 Chapter 7 Solving Serial Connection Problems
Using Loopback Tests

Cisco ICS 7750 Administration and Troubleshooting Guide

7-34 78-10169-02 Rev. B0