Cisco Internetwork Troubleshooting Isolating the Problem at the Network Layer © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT Common Symptoms of Network Layer Problems There is connectivity on the link as seen from the transport layer. No component on the failing link appears to be functional above the network layer. The network is functional, but is operating either consistently or intermittently at ….

© 2005 Cisco Systems, Inc. All rights reserved. CIT Possible Symptoms of Network Layer Problems Packets are delivered to incorrect destinations.There are console messages.There are system log file messages.There are management system alarms.There is unexpected routing behavior.There are empty, inconsistent, or incomplete routing tables.Pings succeed only a percentage of the time.

© 2005 Cisco Systems, Inc. All rights reserved. CIT show running-config router# Displays the contents of the currently running configuration file. General Cisco Command to Isolate Network Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip route router> Displays the current state of the routing table. debug ip routing router# Displays information on routing table updates and route-cache updates. show ip protocols router> Displays the current state of active routing protocols. Cisco Commands to Isolate Routing Table Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip ospf router> Displays general information about the OSPF routing process. show ip ospf interface router> Displays OSPF-related interface information. show ip ospf neighbor router> Displays information about OSPF neighbors on a per- interface basis. Cisco Commands to Isolate OSPF Routing Protocol Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip eigrp neighbors router> Displays the neighbors discovered by the EIGRP routing process. show ip eigrp interfaces router> Displays information about interfaces configured for EIGRP. show ip eigrp topology router> Displays entries in EIGRP topology table. Cisco Commands to Isolate EIGRP Routing Protocol Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip bgp router> Displays entries in BGP routing table. show ip bgp summary router> Displays the status of all BGP connections. show ip bgp neighbors router> Displays information about the BGP and TCP connections to neighbors. Cisco Commands to Isolate BGP Routing Protocol Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT debug ip ospf [adj | events | packet] router# Displays information related to OSPF processing. debug ip eigrp [neighbor | notifications] router# Displays information related to EIGRP processing. debug ip bgp [dampening | events | keepalives | updates] router# Displays information related to BGP processing. Cisco Commands to Debug Routing Protocol Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip traffic router> Displays statistics about IP traffic, such as format errors, bad hops, encapsulation failures, unknown routes, and probe proxy requests. debug ip icmp router# Displays information about ICMP transactions. debug ip packet {access-list number} router# Displays IP debugging information and IP security option (IPSO) security transactions for a specified access list. Cisco Commands to Isolate IP Traffic Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip access-lists [access-list number | access-list- name] router> Displays the contents of all access lists or for a specified numbered or named access list. Cisco Command to Isolate Access List Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT route print Displays the current IP routing table on Windows machines. C:\> route -n Displays the current IP routing table in UNIX or Mac OS X operating systems. terminal% End-System Commands to Isolate Network Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Isolating an Access List Problem at the Network Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is to network D EX /16 [170/ ] via , 3d21h, Serial0/0:0 D EX /16 [170/ ] via , 3d21h, Serial0/0: /16 is variably subnetted, 13 subnets, 2 masks D /26 [90/ ] via , 3d21h, Serial0/0:0 D /26 [90/ ] via , 3d21h, Serial0/0:0 C /26 is directly connected, Serial0/0:1 Isolating an Access List Problem at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT C /26 is directly connected, Serial1/1 D EX /16 [170/ ] via , 3d21h, Serial0/0:0 D /26 [90/ ] via , 3d21h, Serial0/0:0 C /26 is directly connected, FastEthernet0/0.2 C /26 is directly connected, FastEthernet0/0.3 C /26 is directly connected, FastEthernet0/0.1 C /26 is directly connected, Serial0/0:0 C /26 is directly connected, Serial1/0 C /26 is directly connected, FastEthernet0/0.4 C /26 is directly connected, Loopback0 D EX /16 [170/ ] via , 3d21h, Serial0/0:0 D EX /16 [170/ ] via , 3d21h, Serial0/0:0 D EX /16 [170/ ] via , 3d21h, Serial0/0:0 D*EX /0 [170/ ] via , 3d21h, Serial0/0:0 Columbia> Isolating an Access List Problem at the Network Layer (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>ping cisco Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 16/18/20 ms Columbia> Verifying Connectivity to Cisco Systems Across the Default Route

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore>show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is to network D EX /16 [170/284160] via , 3d21h, FastEthernet0/0 D EX /16 [170/284160] via , 3d21h, FastEthernet0/ /16 is variably subnetted, 13 subnets, 2 masks C /26 is directly connected, Loopback0 D /26 [90/156160] via , 3d21h, FastEthernet0/0 C /26 is directly connected, Serial0/0:1 C /26 is directly connected, Serial1/1 D EX /16 [170/284160] via , 3d21h, FastEthernet0/0 Gathering Information from the Routing Table on Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT C /26 is directly connected, FastEthernet0/0 D /26 [90/ ] via , 3d21h, Serial0/0:0 D /26 [90/ ] via , 3d21h, Serial0/0:0 D /26 [90/ ] via , 3d21h, Serial0/0:0 C /26 is directly connected, Serial0/0:0 C /26 is directly connected, Serial1/0 D /26 [90/ ] via , 3d21h, Serial0/0:0 D /26 [90/ ] via , 3d21h, Serial0/0:0 D EX /16 [170/284160] via , 3d21h, FastEthernet0/0 D EX /16 [170/284160] via , 3d21h, FastEthernet0/0 D EX /16 [170/284160] via , 3d21h, FastEthernet0/0 D*EX /0 [170/28416] via , 3d21h, FastEthernet0/0 Baltimore> Gathering Information from the Routing Table on Baltimore (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore>ping cisco Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms Baltimore> Verifying Connectivity to Cisco Systems Across the Default Route

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is to network B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h /16 is variably subnetted, 11 subnets, 2 masks D /26 [90/156160] via , 3d22h, FastEthernet0/5 C /26 is directly connected, Loopback0 D /26 [90/ ] via , 3d21h, FastEthernet0/5 C /26 is directly connected, FastEthernet0/5 Gathering Information from the Routing Table on Washington

© 2005 Cisco Systems, Inc. All rights reserved. CIT B /16 [200/0] via , 4d15h, Null0 D /26 [90/ ] via , 3d21h, FastEthernet0/5 D /26 [90/ ] via , 3d21h, FastEthernet0/5 D /26 [90/ ] via , 3d21h, FastEthernet0/5 D /26 [90/ ] via , 3d21h, FastEthernet0/5 D /26 [90/ ] via , 3d21h, FastEthernet0/5 D /26 [90/ ] via , 3d21h, FastEthernet0/5 B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h /16 is variably subnetted, 2 subnets, 2 masks C /27 is directly connected, Vlan27 B /16 [20/0] via , 4d15h /16 is variably subnetted, 2 subnets, 2 masks C /28 is directly connected, Vlan28 S* /0 [1/0] via Washington> Gathering Information from the Routing Table on Washington (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>show ip protocol Routing Protocol is "eigrp 202. Redistributing: static, eigrp 202, bgp 21. Routing Protocol is "bgp 21". Routing Information Sources: Gateway Distance Last Update (this router) 200 4d15h d15h d15h Distance: external 20 internal 200 local 200 Washington> Gathering Information to Isolate a Network Layer Problem

© 2005 Cisco Systems, Inc. All rights reserved. CIT Lenexa>show ip route. B /24 [20/0] via , 4d16h /30 is subnetted, 1 subnets C is directly connected, FastEthernet0/13 B /16 [20/0] via , 4d22h B /16 [20/0] via , 00:06:04 B /16 [20/0] via , 4d22h B /16 [20/0] via , 4d22h /27 is subnetted, 1 subnets C is directly connected, Vlan27 B /16 [20/0] via , 4d22h /28 is subnetted, 1 subnets C is directly connected, Vlan28 B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h Viewing the Routing Table on Lenexa

© 2005 Cisco Systems, Inc. All rights reserved. CIT /8 is variably subnetted, 7 subnets, 4 masks B /30 [20/0] via , 4d16h B /30 [20/0] via , 4d16h C /25 is directly connected, Loopback0 B /16 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h B /24 [20/0] via , 4d16h S* /0 [1/0] via Lenexa> Viewing the Routing Table on Lenexa (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>show ip bgp BGP table version is 27, local router ID is Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> i *> i s> / i s> / i s> / i s> / i s> / i s> / i s> / i s> / i s> / i Viewing the BGP Routing Table on Washington

© 2005 Cisco Systems, Inc. All rights reserved. CIT s> / i * i *> i * i *> i Network Next Hop Metric LocPrf Weight Path * i *> i * i *> i Washington> Viewing the BGP Routing Table on Washington (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>show ip protocol Routing Protocol is "eigrp 202. Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap CIT CIT Maximum path: 1 Routing for Networks: Routing Information Sources: Gateway Distance Last Update (this router) 200 4d15h d15h d15h Distance: external 20 internal 200 local 200 Washington> Viewing the Status of Routing Protocols on Washington

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>show access-lists Standard IP access list 21 permit permit permit permit permit , wildcard bits Standard IP access list CIT permit , wildcard bits (5 matches) check=200 permit , wildcard bits (8 matches) check=192 permit , wildcard bits (12 matches) check=180 permit , wildcard bits (10 matches) check=170 permit , wildcard bits (10 matches) check=160 Extended IP access list dhcp_glean_acl (per-user) permit udp any eq bootpc host eq bootps Washington> Viewing Access List Information on Washington

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Isolating an IP Addressing Problem at the Network Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington>ping Columbia_sw Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..... Success rate is 0 percent (0/5) Washington> Washington>telnet columbia. Columbia> Columbia>ping Columbia_sw Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..... Success rate is 0 percent (0/5) Columbia> Isolating an IP Addressing Problem at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>ping Columbia Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..... Success rate is 0 percent (0/5) Columbia_SW> Testing Connectivity to the Columbia Router

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>telnet columbia_sw Trying Columbia_SW ( )... % Connection timed out; remote host not responding Columbia> Testing Connectivity to the Columbia_SW Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show ip interface brief Interface IP-Address OK? Method Status Protocol FastEthernet0/0 unassigned YES manual up up FastEthernet0/ YES manual up up FastEthernet0/ YES manual up up FastEthernet0/ YES manual up up FastEthernet0/ YES manual up up Serial1/ YES manual up up Serial1/ YES manual up up Virtual-Access1 unassigned YES unset up up Serial0/0: YES manual up up Serial0/0: YES manual up up Loopback YES manual up up Columbia> Viewing the Current Interface Status on Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater Device ID Local Intrfce Holdtme Capability Platform Port ID Baltimore Ser 0/0:1 179 R 1760 Ser 0/0:1 Baltimore Ser 0/0:0 179 R 1760 Ser 0/0:0 Columbia_SW Fas 0/0 134 S I WS-C2950T-Fas 0/1 Columbia> Viewing Information About Devices Neighboring Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show cdp neighbors detail. Device ID: Columbia_SW Entry address(es): IP address: Platform: cisco WS-C2950T-24, Capabilities: Switch IGMP Interface: FastEthernet0/0, Port ID (outgoing port): FastEthernet0/1 Holdtime : 156 sec. Columbia> Viewing Detailed Information About Devices Neighboring Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show logging Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns) Console logging: disabled. Log Buffer (65536 bytes):. Dec 17 15:33:29: %IP-4-DUPADDR: Duplicate address on FastEthernet0/0.1, sourced by 000a.8a44.de40. Columbia> Analyzing Console Messages on Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Animation Done Click for Animation Example: Isolating Problems at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Olympia>show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is not set /25 is subnetted, 9 subnets D [90/ ] via , 00:01:52, Serial1/0.1 C is directly connected, Serial1/0.1 C is directly connected, Loopback0 D [90/ ] via , 00:01:52, Serial1/0.1 D [90/ ] via , 00:01:52, Serial1/0.1 D [90/ ] via , 00:38:14, Serial1/0.1 D [90/ ] via , 00:01:52, Serial1/0.1 C is directly connected, Serial1/1.1 C is directly connected, FastEthernet0/0 Olympia> Gathering Information from the Routing Table on Olympia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route Gateway of last resort is to network B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h B /16 [20/0] via , 4d15h... Gathering Information from the Routing Table on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT C /25 is directly connected, Loopback0 C /25 is directly connected, FastEthernet0/5 B /16 [200/0] via , 00:29:01, Null /27 is subnetted, 1 subnets C is directly connected, Vlan /28 is subnetted, 1 subnets C is directly connected, Vlan /8 is variably subnetted, 2 subnets, 2 masks O IA /26 [110/2] via , 00:25:37, Vlan28 [110/2] via , 00:25:37, Vlan27 O IA /25 [110/2] via , 00:25:37, Vlan28 [110/2] via , 00:25:37, Vlan27 S* /0 [1/0] via Seattle# Gathering Information from the Routing Table on Seattle (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#ping Olympia Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms Seattle# Verifying Connectivity to Olympia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface FULL/BDR 00:00: Vlan FULL/DR 00:00: Vlan FULL/DR 00:00: Vlan FULL/BDR 00:00: Vlan28 Seattle# Gathering Information on OSPF Neighbors

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip ospf interface fas 0/5 FastEthernet0/5 is up, line protocol is up Internet Address /25, Area 5 Process ID 505, Router ID , Network Type NON_BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) , Interface address No backup designated router on this network Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5 Hello due in 00:00:06 Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 2, maximum is 2 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 0, Adjacent neighbor count is 0 Suppress hello for 0 neighbor(s) Seattle# Gathering Information on an OSPF Interface on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Olympia>show ip ospf interface fast 0/0 FastEthernet0/0 is up, line protocol is up Internet Address /25, Area 5 Process ID 505, Router ID , Network Type NON_BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) , Interface address No backup designated router on this network Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5 Hello due in 00:00:06 Index 3/3, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 4, maximum is 4 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 0, Adjacent neighbor count is 0 Suppress hello for 0 neighbor(s) Olympia> Gathering Information on an OSPF Interface on Olympia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#debug ip ospf adj OSPF adjacency events debugging is on Seattle#debug ip ospf eve Seattle#.Dec 28 21:15:56.167: OSPF: Rcv hello from area 0 from Vlan Dec 28 21:15:56.167: OSPF: End of hello processing.Dec 28 21:15:56.243: OSPF: Rcv hello from area 0 from Vlan Dec 28 21:15:56.243: OSPF: End of hello processing.Dec 28 21:15:57.939: OSPF: Rcv hello from area 0 from Vlan Dec 28 21:15:57.943: OSPF: End of hello processing.Dec 28 21:15:58.011: OSPF: Rcv hello from area 0 from Vlan Dec 28 21:15:58.011: OSPF: End of hello processing.Dec 28 21:16:06.168: OSPF: Rcv hello from area 0 from Vlan Dec 28 21:16:06.168: OSPF: End of hello processing... Seattle#debug all All possible debugging has been turned off Seattle# Gathering Information on OSPF Processes

© 2005 Cisco Systems, Inc. All rights reserved. CIT Guidelines for Isolating Problems at the Network Layer Ping a device across the connection in question.Test connectivity at each hop of a path.Troubleshoot in both directions along an IP path.Use a network diagram. Identify a single pair of problematic source and destination devices.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Summary Problems at the network layer have symptoms that distinguish them from problems at other layers. Analyzing the output of an appropriate Cisco command or application helps you to isolate a problem at the network layer. Applying commands at an end system can be a useful source of information for isolating network layer problems. Using an effective and systematic technique allows you to successfully isolate problems at the network layer.