Cisco Internetwork Troubleshooting Isolating the Problem at the Physical and Data Link Layers © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT Common Symptoms of Physical Layer Problems There is no connectivity on the interface as seen from the data link layer. No component on the failing interface appears to be functional above the physical layer. The network is functional, but is operating either consistently or intermittently at …. There are framing errors.There are line coding errors.There are synchronization errors.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Other Possible Symptoms of Physical Layer Problems There is an increased number of interface errors. LEDs are off, flashing, or in a state other than the expected state during normal operation. There are console messages.There are system log file messages.There are management system alarms.There is excessive utilization.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Common Symptoms of Data Link Layer Problems There is no connectivity on the link as seen from the network layer. No component on the failing link appears to be functional above the data link layer. The link is functional, but is operating either consistently or intermittently at …. There are framing errors.There are encapsulation errors.There are address resolution errors.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Other Possible Symptoms of Data Link Layer Problems There are excessive CRC errors and frame check sequence errors.There are large quantities of broadcast traffic.A MAC address is cycling between ports.There are console messages.There are system log file messages.There are management system alarms.

© 2005 Cisco Systems, Inc. All rights reserved. CIT traceroute [destination] router> Identifies the path a packet takes through the network. [no] debug ? router# Displays a list of options for enabling or disabling debugging events on a device. ping {host | ip-address} Sends an echo request packet to an address, then waits for a reply. router> General Cisco Commands to Isolate Physical and Data Link Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show version router> Displays the Cisco IOS software version and all installed hardware configurations. show ip interface brief router> Displays a summary of the status of all interfaces on a device. show interfaces [type number] router> Displays the operational status of an interface including the amount and type of traffic being sent and received. Cisco Commands to Isolate Physical Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show cdp neighbors detail router> Displays the device type, IP address, and Cisco IOS version of neighboring devices. show controllers router> Displays current internal status information for the interface controller cards. debug [async | ethernet-interface | frame-relay | isdn | ppp | serial] router# Captures events on physical interfaces. Cisco Commands to Isolate Physical Layer Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip arp router> Displays entries in the Address Resolution Protocol (ARP) table. debug [arp | serial | ppp] router# Captures events related to data link layer protocols. Cisco Commands to Isolate Data Link Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT ifconfig -a Displays IP information. terminal% traceroute Identifies the path that a packet takes through the network. terminal% arp -a Displays the current mappings of the IP address to the MAC address in the ARP table. C:\> General End-System Commands to Isolate Physical and Data Link Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Click for Animation Next Animation Example: Isolating a Serial Interface Problem at the Physical Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran(config)#interface serial 1/0 SanFran(config-if)#ip address SanFran(config-if)#no shutdown Dec 12 11:15:52: %LINK-3-UPDOWN: Interface Serial1/0, changed state to up Dec 12 11:15:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up SanFran(config-if)# Dec 12 11:16:15: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to down SanFran(config-if)# Isolating a Serial Interface Problem at the Physical Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show ip interface brief Interface IP-Address OK? Method Status Protocol FastEthernet0/ YES NVRAM up up Serial1/ YES manual up down SanFran# Gathering Information About the Interfaces on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanJose#show ip interface brief Interface IP-Address OK? Method Status Protocol FastEthernet0/ YES NVRAM up up Serial1/ manual administratively down down SanJose# Gathering Information About the Interfaces on SanJose

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Isolating a Frame Relay Problem at the Data Link Layer Click for Animation Next Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Orlando# Dec 16 21:14:24: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to down Dec 16 21:14:24: %DUAL-5-NBRCHANGE: IP-EIGRP 101: Neighbor (Serial1/0) is down: interface down Orlando# Isolating a Frame Relay Problem at the Data Link Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Orlando#show ip interface brief Interface IP-Address OK? Method Status Protocol FastEthernet0/ YES NVRAM up up Serial1/ YES NVRAM up down Orlando# Gathering Information About the Interfaces on Orlando

© 2005 Cisco Systems, Inc. All rights reserved. CIT Daytona# Dec 16 21:16:36: %DUAL-5-NBRCHANGE: IP-EIGRP 101: Neighbor (Serial1/0) is down: holding time expired Daytona# Daytona#show ip interface brief Interface IP-Address OK? Method Status Protocol FastEthernet0/ YES NVRAM up up Serial1/ YES NVRAM up up Daytona# Gathering Information About the Interfaces on Daytona

© 2005 Cisco Systems, Inc. All rights reserved. CIT Orlando#show running-config interface serial 1/0 Building configuration... Current configuration : 162 bytes ! interface Serial1/0 description Frame Relay link to Daytona ip address encapsulation frame-relay frame-relay lmi-type ansi end Orlando# Analyzing the Running Configuration of Orlando

© 2005 Cisco Systems, Inc. All rights reserved. CIT Orlando#show interface serial 1/0 Serial1/0 is up, line protocol is down Hardware is PowerQUICC Serial Description: Frame Relay link to Daytona Internet address is /30 MTU 1500 bytes, BW 128 Kbit, DLY usec, reliability 254/255, txload 1/255, rxload 1/255 Encapsulation FRAME-RELAY, loopback not set Keepalive set (10 sec) LMI enq sent 232, LMI stat recvd 73, LMI upd recvd 0, DTE LMI down LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 0 LMI type is ANSI Annex D frame relay DT. Analyzing Output to Isolate a Data Link Layer Problem on Orlando (Cont)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Last clearing of "show interface" counters 00:38: packets input, 3984 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 37 input errors, 0 CRC, 36 frame, 0 overrun, 0 ignored, 1 abort 268 packets output, bytes, 0 underruns 0 output errors, 0 collisions, 50 interface resets 0 output buffer failures, 0 output buffers swapped out 112 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up Orlando# Analyzing Output to Isolate a Data Link Layer Problem on Orlando

© 2005 Cisco Systems, Inc. All rights reserved. CIT Orlando#debug frame-relay lmi Frame Relay LMI debugging is on Displaying all Frame Relay LMI data Orlando# Dec 16 21:43:13.655: Serial1/0(out): StEnq, myseq 174, yourseen 19, DTE down Dec 16 21:43:13.655: datagramstart = 0x3B5BC14, datagramsize = 14 Dec 16 21:43:13.655: FR encap = 0x Dec 16 21:43:13.655: AE 13 Dec 16 21:43:13.655: Dec 16 21:43:23.656: Serial1/0(out): StEnq, myseq 175, yourseen 19, DTE down Dec 16 21:43:23.656: datagramstart = 0x3999F54, datagramsize = 14 Dec 16 21:43:23.656: FR encap = 0x Dec 16 21:43:23.656: AF 13 Dec 16 21:43:23.656: Dec 16 21:43:33.656: Serial1/0(out): StEnq, myseq 176, yourseen 19, DTE down Dec 16 21:43:33.656: datagramstart = 0x3B5BE94, datagramsize = 14 Dec 16 21:43:33.656: FR encap = 0x Dec 16 21:43:33.656: B0 13. Orlando#undebug all All possible debugging has been turned off Orlando# Analyzing Output to Isolate a Data Link Layer Problem on Orlando (Cont)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Click for Animation Next Animation Example: Isolating Ethernet Problems at the Physical and Data Link Layers

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#ping Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !..!. Success rate is 40 percent (2/5), round-trip min/avg/max = 1/3/4 ms Isolating Ethernet Problems at the Physical and Data Link Layers

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show interfaces ethernet 0 Ethernet0 is up, line protocol is up Hardware is Lance, address is fcb0 (bia fcb0) Internet address is /24 MTU 1500 bytes, BW Kbit, DLY 1000 usec, rely 255/255, load 1/255 Encapsulation ARPA, loopback not set, keepalive set (10 sec) ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:01, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 2218 packets input, bytes, 0 no buffer Received 6 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 input packets with dribble condition detected 2739 packets output, bytes, 0 underruns 15 output errors, 0 collisions, 9 interface resets 0 babbles, 0 late collision, 0 deferred 15 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Analyzing Command Output to Isolate an Ethernet Problem on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT Click for Animation Next Animation Example: Isolating Fast Ethernet Problems at the Physical and Data Link Layers

© 2005 Cisco Systems, Inc. All rights reserved. CIT Tampa#show interfaces fastethernet 0/1 FastEthernet0/1 is up, line protocol is up Hardware is Fast Ethernet, address is 000a.8a (bia 000a.8a ) MTU 1500 bytes, BW Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex, 100Mb/s input flow-control is off, output flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:12, output 00:00:01, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 26 packets input, 3233 bytes, 0 no buffer Received 25 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 25 multicast, 0 pause input 0 input packets with dribble condition detected packets output, bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 250 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out Tampa# Isolating Fast Ethernet Problems at the Physical and Data Link Layers

© 2005 Cisco Systems, Inc. All rights reserved. CIT Tampa_SW#show interfaces fastethernet 0/1 FastEthernet0/1 is up, line protocol is up Hardware is Fast Ethernet, address is c44.5c81 (bia c44.5c81) MTU 1500 bytes, BW Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s input flow-control is off, output flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec packets input, bytes, 0 no buffer Received 1358 broadcasts, 0 runts, 0 giants, 0 throttles 10 input errors, 10 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected packets output, bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out Tampa_SW# Analyzing Command Output to Isolate a Fast Ethernet Problem

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Isolating a Problem at the Physical and Data Link Layers Click for Animation Animations Done

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip route … Gateway of last resort is not set /16 is variably subnetted, 7 subnets, 2 masks D /25 [90/ ] via , 23:19:43, FastEthernet0/5 C /25 is directly connected, Loopback0 D /25 [90/156160] via , 23:19:43, FastEthernet0/5 D /25 [90/ ] via , 23:19:43, FastEthernet0/5 D /25 [90/ ] via , 23:19:43, FastEthernet0/5 C /25 is directly connected, FastEthernet0/5 B /16 [200/0] via , 23:18:53, Null0 Seattle# Gathering Information About Network Connectivity on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip interface brief Interface IP-Address OK? Method Status Protocol Vlan1 unassigned YES NVRAM up up Vlan YES NVRAM up down Vlan YES NVRAM up down FastEthernet0/1 unassigned YES unset up up FastEthernet0/2 unassigned YES unset up up FastEthernet0/3 unassigned YES unset up up FastEthernet0/4 unassigned YES unset up up FastEthernet0/ YES NVRAM up up FastEthernet0/6 unassigned YES NVRAM up up... Gathering Information About the Interfaces on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show vlan VLAN Name Status Ports default active Fa0/6, Fa0/7, Fa0/8, Fa0/9 Fa0/10, Fa0/11, Fa0/12, Fa0/13 Fa0/14, Fa0/15, Fa0/16, Fa0/17 Fa0/18, Fa0/19, Fa0/20, Fa0/21 Fa0/22, Fa0/23, Fa0/24, Gi0/1 Gi0/2 2 test_carole active 3 test_jerry active 4 test_libby active 1002 fddi-default active 1003 token-ring-default active... Gathering Information About VLAN Status on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show vtp status VTP Version : 2 Configuration Revision : 1 Maximum VLANs supported locally : 1005 Number of existing VLANs : 8 VTP Operating Mode : Server VTP Domain Name : CIT VTP Pruning Mode : Disabled VTP V2 Mode : Disabled VTP Traps Generation : Enabled MD5 digest : 0x04 0x03 0xB0 0xDB 0xA0 0x15 0xF7 0x7C Configuration last modified by at :25:34 Local updater ID is on interface Vl27 (lowest numbered VLAN interface found) Seattle# Gathering Information about VTP Database Status on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater Device ID Local Intrfce Holdtme Capability Platform Port ID Elmhurst Fas 0/3 149 R S I WS-C3550-2Fas 0/9 Elmhurst Fas 0/4 149 R S I WS-C3550-2Fas 0/10 Lenexa Fas 0/1 148 R S I WS-C3550-2Fas 0/9 Lenexa Fas 0/2 148 R S I WS-C3550-2Fas 0/10 Olympia Fas 0/5 152 R 1760 Fas 0/0 DEV_test Fas 0/6 148 S I WS-C3550-2Fas 0/2 Seattle# Gathering Information about Cisco Discovery Protocol Neighbors on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show cdp neighbors fas 0/6 detail Device ID: DEV_test Entry address(es): IP address: Platform: cisco WS-C , Capabilities: Switch IGMP Interface: FastEthernet0/6, Port ID (outgoing port): FastEthernet0/1 Holdtime : 166 sec Version : Cisco Internetwork Operating System Software IOS (tm) C3550 Software (C3550-I5Q3L2-M), Version 12.1(11)EA1a, RELEASE SOFTWARE (fc1) Copyright (c) by cisco Systems, Inc. Compiled Thu 17-Oct-02 23:29 by antonino advertisement version: 2 Protocol Hello: OUI=0x00000C, Protocol ID=0x0112; payload len=27, value= FFFFFFFF010221FF AB7D1C180FF0000 VTP Management Domain: 'CIT' Duplex: full Seattle# Analyzing Information About a Cisco Discovery Protocol Neighbor

© 2005 Cisco Systems, Inc. All rights reserved. CIT Guidelines for Isolating Problems at the Physical and Data Link Layers Check operational status and data error rates.Verify proper interface configurations.Check for bad cables or connections.Check for correct cable pinout.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Isolating Problems at the Physical and Data Link Layers

© 2005 Cisco Systems, Inc. All rights reserved. CIT Summary Problems at the physical layer cause intermittent or consistent loss of connectivity across that link. Devices experiencing a problem at the data link layer have physical connectivity, but will not transfer data through the failed interface. Analyzing the output of an appropriate command or application helps you to isolate a problem at the physical or data link layer. Using an effective and systematic technique allows you to successfully isolate a problem at the physical or data link layer.