Cisco Internetwork Troubleshooting Correcting the Problem at the Network Layer © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT [no] ip domain lookup router(config)# This command enables the IP DNS-based host name- to-address translation. To disable, enter the no form of this command. General Cisco Command to Correct Network Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT interface {interface-type number} router(config)# Accesses a specified interface while in global configuration mode. ip address ip-address mask [secondary] router(config-if)# Specifies a primary or secondary IP address for an interface. [no] ip redirects router(config-if)# Enables or disables the sending of redirect messages through the same interface on which they were received. Cisco Commands to Correct IP Interface Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT bandwidth {kilobits} router(config-if)# Communicates the bandwidth value of an interface to the higher-level protocols. [no] ip proxy-arp router(config-if)# Enables or disables proxy ARP on an interface. router(config-if)# ip mroute-cache Enables IP multicast fast switching or multicast distributed switching. Cisco Commands to Correct IP Interface Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT access-list {access-list-number} {deny | permit} protocol source source-wildcard destination destination-wildcard [log] router(config)# Defines an extended access list. ip access-list {standard | extended} {access-list-name} router(config)# Defines a standard or extended named access list. ip access-group {access-list-number | access-list-name} router(config)# Applies an extended access list. Cisco Commands to Correct Access List Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT ip route prefix mask address [distance] router(config)# Configures a static route. ip route {ip-address | interface-type number} [distance] router(config)# Configures a default route. router(config-if)# ip route-cache Enables the use of high-speed switching caches. Cisco Commands to Correct IP Routing Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT ip split-horizon router(config-if)# Enables split horizon. [no] passive interface router(config-router)# Enables and disables the sending of routing updates on a specified interface. router(config-router)# network network-number [mask network-mask] Specifies a list of networks for a routing process. Cisco Commands to Correct IP Routing Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT arp -d End-system command for deleting entries from an ARP table. route add Adds static IP routes to a routing table. General End-System Commands to Correct Problems at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT ifconfig Configures IP information on hosts running Mac OS X and UNIX. UNIX/Mac OS X Command to Correct Problems at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting an Access List Problem at the Network Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington(config)#ip access-list standard CIT Washington(config-std-nacl)# remark Include Cisco network as well Washington(config-std-nacl)# permit Washington(config-std-nacl)#exit Washington# Dec 17 14:19:48: %SYS-5-CONFIG_I: Configured from console by console Washington# Correcting an Access List Problem at the Network Layer 1. Enter interface configuration mode. 2. Enter the remark statement. 4. Exit interface configuration mode. 3. Enter the permit statement.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington#show access-lists Standard IP access list 21 permit permit permit permit permit , wildcard bits Standard IP access list CIT permit , wildcard bits (4 matches) check=158 permit , wildcard bits (6 matches) check=152 permit , wildcard bits (10 matches) check=142 permit , wildcard bits (8 matches) check=134 permit , wildcard bits (8 matches) check=126 permit , wildcard bits Extended IP access list dhcp_glean_acl (per-user) permit udp any eq bootpc host eq bootps Washington# Verifying the Updated Access List

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington#show ip bgp BGP table version is 27, local router ID is Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> i *> i s> / i s> / i s> / i s> / i s> / i s> / i s> / i s> / i s> / i Viewing the BGP Routing Table on Washington

© 2005 Cisco Systems, Inc. All rights reserved. CIT s> / i * i *> i * i *> i Network Next Hop Metric LocPrf Weight Path * i *> i * i *> i Washington# Viewing the BGP Routing Table on Washington (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington#debug ip routing Washington#clear ip bgp 77 Dec 17 14:23:46: %BGP-5-ADJCHANGE: neighbor Down User reset Dec 17 14:23:46: %BGP-5-ADJCHANGE: neighbor Down User reset. Dec 17 14:24:05: %BGP-5-ADJCHANGE: neighbor Up Dec 17 14:24:05.475: RT: Nexthop for /24 updated Washington#undebug all All possible debugging has been turned off Washington# Debugging and Clearing the BGP Table on the Washington Router

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington#show ip bgp BGP table version is 38, local router ID is Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> i. * i *> i Washington# Checking for Routing Updates in the BGP Table

© 2005 Cisco Systems, Inc. All rights reserved. CIT Washington#show ip route Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default, U - per-user static route, o - ODR P - periodic downloaded static route B /16 [20/0] via , 00:03: /28 is subnetted, 1 subnets C is directly connected, Vlan28 B /24 [20/0] via , 00:03:15 S* /0 [1/0] via Washington# Checking for Routing Updates in the IP Routing Table

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore>show ip route. D EX /16 [170/284160] via , 00:05:06, FastEthernet0/0 D EX /16 [170/284160] via , 00:05:06, FastEthernet0/0 D EX /16 [170/284160] via , 00:05:06, FastEthernet0/0 D EX /24 [170/284160] via , 00:05:06, FastEthernet0/0 D*EX /0 [170/28416] via , 3d22h, FastEthernet0/0 Baltimore> Checking for Routing Table Updates on Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show ip route. D EX /16 [170/ ] via , 00:06:48, Serial0/0:0 D EX /16 [170/ ] via , 00:06:48, Serial0/0:0 D EX /16 [170/ ] via , 00:06:48, Serial0/0:0 D EX /24 [170/ ] via , 00:06:48, Serial0/0:0 D*EX /0 [170/ ] via , 3d22h, Serial0/0:0 Columbia> Checking for Routing Table Updates on Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting an IP Addressing Problem at the Network Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>enable Columbia_SW#configure terminal Enter configuration commands, one per line. End with CNTL/Z. Columbia_SW(config)#interface fastethernet 0/1 Columbia_SW(config-if)#ip address % IP addresses may not be configured on L2 links. Columbia_SW(config-if)# Correcting an IP Addressing Problem at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW#show running-config interface fastEthernet 0/1 Building configuration... Current configuration : 310 bytes ! interface FastEthernet0/1 switchport trunk native vlan 901 switchport mode trunk no ip address duplex full speed 100 storm-control broadcast level storm-control multicast level storm-control unicast level storm-control action shutdown storm-control action trap end Columbia_SW# Examining the Configuration of the Columbia_SW Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW#show ip interface brief Interface IP-Address OK? Method Status Protocol Vlan1 unassigned YES manual administratively down down Vlan YES manual up up FastEthernet0/1 unassigned YES unset up up. Columbia_SW# Verifying the Interface Status on the Columbia_SW Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW(config)#logging console.Dec 17 17:33:43: %IP-4-DUPADDR: Duplicate address on Vlan901, sourced by a157 Columbia_SW(config-if)#interface vlan901 Columbia_SW(config-if)#ip address Columbia_SW(config-if)#exit Columbia_SW#.Dec 17 17:33:58: %SYS-5-CONFIG_I: Configured from console by console Changing the IP Address on VLAN901

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>ping columbia Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:.!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms Columbia_SW> Verifying Connectivity Between Columbia_SW and Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>ping columbia_SW Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms Columbia> Columbia>exit [Connection to columbia closed by foreign host] Washington>ping columbia_sw Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/32/36 ms Washington> Verifying Connectivity Between the Three Devices

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting Problems at the Network Layer Animations Done Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show run int fas 0/5 Building configuration... Current configuration : 175 bytes ! interface FastEthernet0/5 description Link to Olympia no switchport ip address ip ospf network non-broadcast duplex full speed 100 end Seattle# Verifying the Network Issue on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Olympia#show run int fas 0/0 Building configuration... Current configuration : 156 bytes ! interface FastEthernet0/5 description Link to Seattle no switchport ip address ip ospf network non-broadcast duplex full speed 100 end Olympia# Verifying OSPF Network Type on Olympia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Olympia#conf t Enter configuration commands, one per line. End with CNTL/Z. Olympia(config)#int fas 0/0 Olympia(config-if)#no ip ospf net non Olympia(config-if)#^Z Olympia# Correcting an OSPF Network Type Problem at the Network Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#conf t Enter configuration commands, one per line. End with CNTL/Z. Seattle(config)#int fas 0/5 Seattle(config-if)#no ip ospf net non Seattle(config-if)#^Z Seattle#.Dec 28 21:17:10: %SYS-5-CONFIG_I: Configured from console by console Seattle#.Dec 28 21:17:49: %OSPF-5-ADJCHG: Process 505, Nbr on FastEthernet0/5 from LOADING to FULL, Loading Do Seattle# Correcting an OSPF Network Type Problem at the Network Layer (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip ospf neigh FULL/BDR 00:00: Vlan FULL/DR 00:00: Vlan FULL/DR 00:00: Vlan FULL/BDR 00:00: Vlan FULL/BDR 00:00: FastEthernet0/5 Seattle# Verifying OSPF Neighbors on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Olympia#show ip ospf neigh Neighbor ID Pri State Dead Time Address Interface FULL/DR 00:00: FastEthernet0/0 Olympia# Verifying OSPF Neighbors on Olympia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Seattle#show ip route ospf /16 is variably subnetted, 11 subnets, 2 masks O N /25 [110/1682] via , 00:03:24, FastEthernet0/5 O /25 [110/782] via , 00:03:24, FastEthernet0/5 O /25 [110/2] via , 00:03:24, FastEthernet0/5 O N /25 [110/1682] via , 00:03:24, FastEthernet0/5 O N /25 [110/1682] via , 00:03:24, FastEthernet0/5 O N /25 [110/1682] via , 00:03:24, FastEthernet0/5 O N /25 [110/1682] via , 00:03:24, FastEthernet0/5 O /25 [110/1563] via , 00:03:24, FastEthernet0/5... Seattle# Verifying the OSPF Routes on Seattle

© 2005 Cisco Systems, Inc. All rights reserved. CIT Cisco Systems Cisco TAC Internetwork Troubleshooting Handbook Cisco Systems technologies reference Support Resources for Correcting Network Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Procedure for Correcting Network Layer Problems 1 Verify that you have a valid saved configuration for any device on which you intend to modify the configuration. 3 Evaluate and document the results of each change that you make. 4 Verify that the changes you made actually fixed the problem without introducing any new problems. 5 Continue making changes until the problem appears to be solved. 6 If necessary, get input from outside resources. 2 Make initial configuration changes. 7 Once the problem is resolved, document the solution.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Summary Use the appropriate end system or Cisco commands and applications to correct an isolated network layer problem. Use the appropriate end-system command and application to correct problems at the network level. These are some network layer support resources: –Cisco Systems TAC –Internetwork Troubleshooting Handbook –Cisco Systems technologies reference Following a systematic procedure increases the chances that you will successfully and effectively correct an isolated problem at the network layer.

Completed Troubleshooting Logs © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch a) router rip network b) router eigrp 101 redistribute static Troubleshooting LogTrouble Ticket D Core Router/Switch a) Users cannot get server (workgroup network not defined in RIP) b) Users cannot get to the Internet (EIGRP not redistributing static route)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router a) no router eigrp 11 router eigrp 101 ! had wrong AS number passive-interface default no passive-interface Serial0/0:0 no passive-interface Serial0/0:1 no passive-interface Serial1/0.1 no passive-interface Serial1/1.1 no passive-interface FastEthernet0/0 network network network network network network auto-summary eigrp log-neighbor-changes ! b) interface Serial0/0:0 ip address 172.2x.1x Troubleshooting LogTrouble Ticket D Distribution Router a) Does not form EIGRP adjacencies due to wrong AS number b) No neighbor with Access Rtr on Serial 0/0:0 (has ip address of ser 0/0:1 )

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router router eigrp x0x no passive-interface Serial1/0.1 no passive-interface Serial1/1.1 passive-interface Serial1/0 passive-interface Serial1/1 ! interface Serial1/0.1 point-to-point ip address 172.2x.1x [128|192] ! interface Serial1/1.1 point-to-point ip address 172.2x.1x [128|192] ! Troubleshooting LogTrouble Ticket D Access Router a) Does not form EIGRP adjacencies (passive interfaces, wrong prefixes – even pods are /36, odd pods are /25)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Switch a) interface FastEthernet0/1 no shutdown b) pc's have static IP overlapping w/ DHCP server – enable DHCP on PCs Troubleshooting LogTrouble Ticket D Access Switch a) PCs cannot connect to anywhere b) PCs getting network error messages when try to ping

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! a) interface Vlan27 no ip address ! interface Vlan28 ip address interface Vlan27 ip address ! Troubleshooting LogTrouble Ticket E Core Router/Switch a) No one can connect to Internet or CIT core (or Elmhurst or Lenexa – VLANs have swapped IP addresses) b) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! b) interface FastEthernet0/15 no description Link to Orlando no ip address no speed 100 no duplex full switch shut ! interface FastEthernet0/5 description Link to Orlando no switchport ip address speed 100 duplex full no shut ! Troubleshooting LogTrouble Ticket E Core Router/Switch (Cont.) b) Trunk on wrong interface c) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! c) router bgp 11 neighbor remote-as 77 neighbor remote-as 77 neighbor distribute-list CIT in neighbor distribute-list CIT in ! logging console ! Troubleshooting LogTrouble Ticket E Core Router/Switch (Cont.) c) No BGP neighbors (wrong AS for core)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router ! a) router eigrp 101 network network network network network network no network no network no network no network no network no network ! mistyped network statements ! Troubleshooting LogTrouble Ticket E Distribution Router a) Missing EIGRP routes (network statement issues) b) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router ! b) interface Serial1/0.1 frame-relay map ip broadcast ! interface Serial1/1.1 frame-relay map ip broadcast no frame-relay map ip ! Troubleshooting LogTrouble Ticket E Distribution Router (Cont.) b) Frame Relay links not forming EIGRP neighbors (missing broadcasts, wrong DLCI)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! a) interface FastEthernet0/0 no ip address ! interface FastEthernet0/0.1 ip address ! b) controller t1 0/0 no loopback local line ! c) interface Serial1/0 bandwidth 128 ! interface Serial1/1 bandwidth 64 Troubleshooting LogTrouble Ticket E Access Router a) No connectivity to access switch (IP address on wrong port) b) No connectivity to dist rtr (controller running loopback mode) c) Traffic flows over wrong links (bandwidth mistyped)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Switch ! interface FastEthernet0/1 switchport trunk native vlan 901 switchport mode trunk duplex full speed 100 no shutdown ! interface FastEthernet0/10 no switchport mode trunk switchport trunk native vlan 901 no ip address shut ! Troubleshooting LogTrouble Ticket E Access Switch Users cannot connect to anything (trunk on wrong port)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! a) examples shown for pod 4 ! interface range FastEthernet0/1 - 2 switchport trunk encapsulation isl ! interface range FastEthernet0/3 - 4 switchport trunk encapsulation dot shutdown no shutdown ! ! b) vtp mode transparent no vlan vlan 27 vlan 28 ! ! c) router ospf 404 router-id ! do clear ip ospf process y Troubleshooting LogTrouble Ticket F Core Router/Switch a) EtherChannels are down, due to trunk encapsulation mismatch b) VTP mode is server, VLANs are getting overwritten (instructor can do this a few times, so all pods have issue at least once) c) Duplicate OSPF RID with core (even pods with Elmhurst, odd pods with Lenexa)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router ! a) examples shown for pod 4 router ospf 404 area 4 nssa ! ! b) router eigrp 404 redistribute ospf 404 metric no redistribute ospf 44 metric ! ! c) router eigrp 404 no passive-interface Serial1/0.1 no passive-interface Serial1/1.1 passive-interface Serial1/0 passive-interface Serial1/1 ! Troubleshooting LogTrouble Ticket F Distribution Router a) Area 4 missing nssa statement b) OSPF routes not redistributed due to wrong AS number c) no EIGRP neighbors, since frame subinterfaces are passive for EIGRP

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! a) router eigrp 404 network network ! b) interface Serial1/0 keepalive ! interface Serial1/1 keepalive ! ! c) interface Serial1/0.1 no ip add interface Serial1/1.1 ip address interface Serial1/0.1 ip address ! Troubleshooting LogTrouble Ticket F Access Router a) Missing routes for two end user VLANs b) Frame links down, no keepalives c) IP addresses on frame links are swapped

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Switch Nothing neededNo issues here Troubleshooting LogTrouble Ticket F Access Switch