Friday, February 13, 2015

Configuring Cisco Wireless LAN Controller (WLC)

I've been configuring Cisco WLC 2504 and 5508 for quite some time now and only got the chance to blog it. I always use PuTTY or HyperTerminal to console to the WLC since SecureCRT v7.2.1 gets stuck after this line:

Would you like to terminate autoinstall? [yes] 

My test setup includes a Cisco AIR-CAP1602E-E-K9, a Cisco Catalyst 3650 switch and WLC 2504. The C stands for controller-based AP while the Cisco AIR-SAP1602E is a standalone AP. I've manually set the country, local date and time on the WLC.



The AP doesn't seem to join the WLC when the date and time aren't manually configured or set to non-working NTP. The AP also doesn't join when the country is set other than the specified country Stock Keeping Unit (SKU). Once the WLC is in production, you could configure it for NTP afterwards.

Here's a Cisco WLAN compliance matrix to specified approved countries and this is what the initial boot up and setup wizard looks like:


100%

36110929 bytes read
Launching...
Launching images...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 688, tty '': '/etc/init.d/rcS'
Detecting Hardware ...
Installing ether-pow driver - 0x6008
starting pid 879, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....passed!
XML config selected
Validating XML configuration

 XML Config version file is corrupted :Move all files to tmp XML dir

XML config version file is corrupted: Removed xml dir
touch: /mnt/application/xml/clis/XMLInvalidTagConfig.txt: No such file or direct
ory
octeon_device_init: found 1 DPs
/dev/fpga: No such device or address
readCPUConfigData: cardid 0x6060001
XML validation can not be done file open failed
sh: cannot create /mnt/application/xml/clis/XMLInvalidTagConfig.txt: nonexistent
 directory
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 7.6.110.0
Firmware Version PIC 16.0


Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Error (256) found in fsck check - attempt to repair.
Initializing Licensing Services:
License daemon start initialization.....
ok

License daemon running.....
Starting Statistics Service: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DU
TY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(2)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2040
Fastpath CPU0.00: Core 0 Initialization: ok
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.01: Core 1 Initialization: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Management Frame Protection: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting DNS Services: ok
Starting Licensing Services: ok
Starting Redundancy: ok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services:  ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server:  enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
   Web Server:    CLI:    Secure Web: Web Authentication Certificate not found (
error). If you cannot access management interface via HTTPS please reconfigure V
irtual Interface.

(Cisco Controller)


Welcome to the Cisco Wizard Configuration Tool
Use the '-' character to backup


Would you like to terminate autoinstall? [yes]:     // HANGS UP TO THIS POINT WHEN USING SecureCRT

System Name [Cisco_3e:de:84] (31 characters max):
AUTO-INSTALL: process terminated -- no configuration loaded

Enter Administrative User Name (24 characters max): admin
Enter Administrative Password (3 to 24 characters):Cisco123     // REQUIRES UPPER AND LOWER CASE LETTERS AND REPEATED NUMBERS THRICE ISN'T ALLOWED
Re-enter Administrative Password: Cisco123

Enable Link Aggregation (LAG) [yes][NO]: no

Management Interface IP Address: 172.27.197.15
Management Interface Netmask: 255.255.255.192
Management Interface Default Router: 172.27.197.1
Management Interface VLAN Identifier (0 = untagged):
Management Interface Port Num [1 to 4]: 1
Management Interface DHCP Server IP Address: 172.27.197.1

Virtual Gateway IP Address: 1.1.1.1      // FOR WIRELESS ROAMING

Multicast IP Address: 239.0.0.1

Mobility/RF Group Name: MY_WIRELESS

Network Name (SSID): MY_WIRELESS

Configure DHCP Bridging Mode [yes][NO]: no

Allow Static IP Addresses [YES][no]: no

Configure a RADIUS Server now? [YES][no]: no
Warning! The default WLAN security policy requires a RADIUS server.
Please see documentation for more details.

Enter Country Code list (enter 'help' for a list of countries) [US]: SG   // REQUIRED FOR AP TO JOIN

Enable 802.11b Network [YES][no]: yes
Enable 802.11a Network [YES][no]: yes
Enable 802.11g Network [YES][no]: yes
Enable Auto-RF [YES][no]: yes

Configure a NTP server now? [YES][no]: no    // REQUIRED FOR AP TO JOIN OR SET LOCAL CLOCK
Configure the system time now? [YES][no]: yes
Enter the date in MM/DD/YY format: 02/02/15 
Enter the time in HH:MM:SS format: 14:23:00

Configuration correct? If yes, system will save it and reset. [yes][NO]: yes


After the wizard, the WLC will reboot and it's now accessible via HTTPS. IE tends to work smoothly with the WLC GUI.




Here are some useful show commands on the WLC:

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.100.0     // AP MUST USE SAME VERSION
Bootloader Version............................... 1.0.16
Field Recovery Image Version..................... 1.0.0
Firmware Version................................. PIC 16.0


Build Type....................................... DATA + WPS

System Name...................................... MY_WLC
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1279
IP Address....................................... 172.27.197.15
Last Reset....................................... Power on reset
System Up Time................................... 41 days 11 hrs 42 mins 59 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... SG  - Singapore
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +45 C
Fan Status....................................... 4400 rpm

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 2
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ B0:FA:EB:8B:XX:YY
Maximum number of APs supported.................. 25     // CAN INCREASE VIA LICENSING


If the WLC and AP are running on different versions, the AP will log the error below. The APs are currently shipped with version 7.6 as of this writing. You can directly upgrade to 7.6  if you're on WLC version 7.0 or 7.4.



*Mar  1 00:02:40.694: %CAPWAP-3-ERRORLOG: Discovery response from MWAR 'Cisco_a5:5f:44'running version 7.0.220.0 is rejected.      // WLC IS ONVERSION 7.0; CHECK THE UPGRADE PATH IN CISCO.COM



 (Cisco Controller) >show run-config

Press Enter to continue...

System Inventory
NAME: "Chassis"    , DESCR: "Cisco 2500 Series Wireless LAN Controller"
PID: AIR-CT2504-K9,  VID: V01,  SN: PSZ17220XYZ

Burned-in MAC Address............................ 10:F3:11:A4:7A:BC
Maximum number of APs supported.................. 15
Press Enter to continue or <ctrl-z> to abort

<OUTPUT TRUNCATED>

(Cisco Controller) >show ap summary

Number of APs.................................... 1

Global AP User Name.............................. Not Configured
Global AP Dot1x User Name........................ Not Configured

AP Name             Slots  AP Model              Ethernet MAC       Location
      Country  IP Address       Clients
------------------  -----  --------------------  -----------------  ------------
----  -------  ---------------  -------
APfc5b.3937.3abc     2     AIR-CAP1602E-E-K9     fc:5b:39:37:3a:bc  default loca
tion  SG       172.27.197.8     0

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 1

Base Mac             AP EthernetMac       AP Name                 IP Address         Status
04:da:d2:ce:12:34    6c:20:56:13:ab:cd   APfc5b.3937.3abc            172.27.197.22      Joined   


(Cisco Controller) >debug capwap events enable

(Cisco Controller) >debug disable-all

 (Cisco Controller) save config

Are you sure you want to save? (y/n) y

Configuration Saved!

Sunday, February 8, 2015

Configure Policy Based Routing (PBR)

It's been close to a year since the CCNP Security exams were renewed. There's already some few rants on when the new 300-207 SITCS book will be officially released. Cisco guru Keith Barker also tweeted around late last year that he's just waiting for some updates from folks at Cisco before starting recording his video nuggets for SITCS.


While waiting for the official Cisco press book (by September 2015) and CBT Nugget videos to be released, I've been busy doing ROUTE and SWITCH labs in conjunction with my CCDP studies. This is one of the lab I did for ROUTE regarding policy based routing or PBR.


R1(config)#interface loopback1
*Dec 27 23:49:35.519: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1,
R1(config-if)#description R1 LAN
R1(config-if)#ip address 192.168.1.1 255.255.255.0
R1(config-if)#interface s0/0/0
R1(config-if)#description R1 --> R2
R1(config-if)#ip address 172.16.12.1 255.255.255.248
R1(config-if)#clock rate 128000
R1(config-if)#bandwidth 128
R1(config-if)#no shutdown
*Dec 27 23:51:05.339: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
R1(config-if)#interface s0/0/1
R1(config-if)#description R1 --> R3
R1(config-if)#ip address 172.16.13.1 255.255.255.248
R1(config-if)#bandwidth 64
R1(config-if)#no shutdown
*Dec 27 23:51:38.891: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to down


R2(config)#interface loopback2
R2(config-if)#
*Dec 27 22:34:44.835: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback2, changed state to up
R2(config-if)#description R2 LAN
R2(config-if)#ip address 192.168.2.1 255.255.255.0
R2(config-if)#interface s0/0/0
R2(config-if)#description R2 --> R1
R2(config-if)#ip address 172.16.12.2 255.255.255.248
R2(config-if)#bandwidth 128
R2(config-if)#no shutdown
R2(config-if)#
*Dec 27 22:35:32.047: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Dec 27 22:35:33.047: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
R2(config-if)#do ping 172.16.12.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
R2(config-if)#interface s0/0/1
R2(config-if)#description R2 --> R3
R2(config-if)#ip address 172.16.23.2 255.255.255.248
R2(config-if)#clock rate 128000
R2(config-if)#bandwidth 128
R2(config-if)#no shutdown
R2(config-if)#
*Dec 27 22:36:11.723: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to down


R3(config)#interface loopback3
R3(config-if)#
*Dec 28 00:00:46.055: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback3, changed state to up
R3(config-if)#description R3 LAN
R3(config-if)#ip address 192.168.3.1 255.255.255.0
R3(config-if)#interface s0/0/0
R3(config-if)#description R3 --> R1
R3(config-if)#ip address 172.16.13.3 255.255.255.248
R3(config-if)#clock rate 64000
R3(config-if)#bandwidth 64
R3(config-if)#no shutdown
R3(config-if)#
*Dec 28 00:01:47.459: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Dec 28 00:01:48.459: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
R3(config-if)#do ping 172.16.13.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
R3(config-if)#interface s0/0/1
R3(config-if)#description R3 --> R2
R3(config-if)#ip address 172.16.23.3 255.255.255.248
R3(config-if)#bandwidth 128
R3(config-if)#no shutdown
R3(config-if)#
*Dec 28 00:02:25.503: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to up
*Dec 28 00:02:26.503: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/1, changed state to up
R3(config-if)#do ping 172.16.23.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
R3(config-if)#interface s0/1/0
R3(config-if)#description R3 --> R4
R3(config-if)#ip address 172.16.34.3 255.255.255.248
R3(config-if)#clock rate 64000
R3(config-if)#bandwidth 64
R3(config-if)#no shutdown
R3(config-if)#
*Dec 28 00:03:16.587: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to down


R4(config)#interface loopback4
R4(config-if)#
*Dec 27 23:19:15.119: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback4, changed state to up
R4(config-if)#description R4 LAN A
R4(config-if)#ip address 192.168.4.1 255.255.255.128
R4(config-if)#interface loopback5
R4(config-if)#
*Dec 27 23:19:43.931: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback5, changed state to up
R4(config-if)#description R4 LAN B
R4(config-if)#ip address 192.168.4.129 255.255.255.128
R4(config-if)#interface s0/0/0
R4(config-if)#description R4 --> R3
R4(config-if)#ip address 172.16.34.4 255.255.255.248
R4(config-if)#bandwidth 64
R4(config-if)#no shutdown
R4(config-if)#
*Dec 27 23:22:38.507: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Dec 27 23:22:39.507: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
R4(config-if)#do ping 172.16.34.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms


R3#show ip interface brief | exclude unassigned
Interface                  IP-Address      OK? Method Status                Protocol
Serial0/0/0                172.16.13.3     YES manual up                    up 
Serial0/0/1                172.16.23.3     YES manual up                    up 
Serial0/1/0                172.16.34.3     YES manual up                    up 
Loopback3                  192.168.3.1     YES manual up                    up 

R3#show protocols | exclude down
Global values:
  Internet Protocol routing is enabled
Serial0/0/0 is up, line protocol is up
  Internet address is 172.16.13.3/29
Serial0/0/1 is up, line protocol is up
  Internet address is 172.16.23.3/29
Serial0/1/0 is up, line protocol is up
  Internet address is 172.16.34.3/29
Loopback3 is up, line protocol is up
  Internet address is 192.168.3.1/24

R3#show interface description | exclude down
Interface                      Status         Protocol Description
Se0/0/0                        up             up       R3 --> R1
Se0/0/1                        up             up       R3 --> R2
Se0/1/0                        up             up       R3 --> R4
Lo3                            up             up       R3 LAN


R1(config)#router eigrp 1
R1(config-router)#network 192.168.1.0
R1(config-router)#network 172.16.12.0 0.0.0.7
R1(config-router)#network 172.16.13.0 0.0.0.7
R1(config-router)#no auto-summary


R2(config)#router eigrp 1
R2(config-router)#network 192.168.2.0
R2(config-router)#network 172.16.12.0 0.0.0.7
R2(config-router)#
*Dec 27 22:52:07.707: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.12.1 (Serial0/0/0) is up: new adjacency
R2(config-router)#network 172.16.23.0 0.0.0.7
R2(config-router)#no auto-summary
R2(config-router)#
*Dec 27 22:52:24.883: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.12.1 (Serial0/0/0) is resync: summary configured   // PREFER TO ISSUE FIRST THE COMMAND no auto-summary


R3(config)#router eigrp 1
R3(config-router)#no auto-summary
R3(config-router)#network 192.168.3.0
R3(config-router)#network 172.16.13.0 0.0.0.7
R3(config-router)#
*Dec 28 00:15:12.431: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.13.1 (Serial0/0/0) is up: new adjacency
R3(config-router)#network 172.16.23.0 0.0.0.7
R3(config-router)#
*Dec 28 00:15:19.771: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.23.2 (Serial0/0/1) is up: new adjacency
R3(config-router)#network 172.16.34.0 0.0.0.7


R4(config)#router eigrp 1
R4(config-router)#network 192.168.4.0
R4(config-router)#network 172.16.34.0 0.0.0.7
R4(config-router)#
*Dec 27 23:31:22.139: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.34.3 (Serial0/0/0) is up: new adjacency
R4(config-router)#no auto-summary
R4(config-router)#
*Dec 27 23:31:36.979: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.34.3 (Serial0/0/0) is resync: summary configured


R1#show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
1   172.16.13.3             Se0/0/1           14 00:01:56   20  2280  0  24
0   172.16.12.2             Se0/0/0           11 00:03:03    9  1140  0  22


R2#show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
1   172.16.23.3             Se0/0/1           10 00:04:22   12  1140  0  23
0   172.16.12.1             Se0/0/0           12 00:05:37   10  1140  0  24


R3#show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
2   172.16.34.4             Se0/1/0           12 00:03:58   24  2280  0  7
1   172.16.23.2             Se0/0/1           10 00:04:43   16  1140  0  23
0   172.16.13.1             Se0/0/0           10 00:04:50   23  2280  0  25


R4#show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
0   172.16.34.3             Se0/0/0           14 00:00:51   30  2280  0  28


R1#tclsh
R1(tcl)#foreach address {
+>(tcl)#172.16.12.1
+>(tcl)#172.16.12.2
+>(tcl)#172.16.13.1
+>(tcl)#172.16.13.3
+>(tcl)#172.16.23.2
+>(tcl)#172.16.23.3
+>(tcl)#172.16.34.3
+>(tcl)#172.16.34.4
+>(tcl)#192.168.1.1
+>(tcl)#192.168.2.1
+>(tcl)#192.168.3.1
+>(tcl)#192.168.4.1
+>(tcl)#192.168.4.129
+>(tcl)#} { ping $address }

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.129, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/60 ms


R2#tclsh
R2(tcl)#foreach address {
+>(tcl)#172.16.12.1
+>(tcl)#172.16.12.2
+>(tcl)#172.16.13.1
+>(tcl)#172.16.13.3
+>(tcl)#172.16.23.2
+>(tcl)#172.16.23.3
+>(tcl)#172.16.34.3
+>(tcl)#172.16.34.4
+>(tcl)#192.168.1.1
+>(tcl)#192.168.2.1
+>(tcl)#192.168.3.1
+>(tcl)#192.168.4.1
+>(tcl)#192.168.4.129
+>(tcl)#} { ping $address }

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/42/44 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/42/44 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.129, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/42/44 ms


R3#tclsh
R3(tcl)#foreach address {
+>(tcl)#172.16.12.1
+>(tcl)#172.16.12.2
+>(tcl)#172.16.13.1
+>(tcl)#172.16.13.3
+>(tcl)#172.16.23.2
+>(tcl)#172.16.23.3
+>(tcl)#172.16.34.3
+>(tcl)#172.16.34.4
+>(tcl)#192.168.1.1
+>(tcl)#192.168.2.1
+>(tcl)#192.168.3.1
+>(tcl)#192.168.4.1
+>(tcl)#192.168.4.129
+>(tcl)#} { ping $address }

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/60 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.129, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms


R4#tclsh
R4(tcl)#foreach address {
+>(tcl)#172.16.12.1
+>(tcl)#172.16.12.2
+>(tcl)#172.16.13.1
+>(tcl)#172.16.13.3
+>(tcl)#172.16.23.2
+>(tcl)#172.16.23.3
+>(tcl)#172.16.34.3
+>(tcl)#172.16.34.4
+>(tcl)#192.168.1.1
+>(tcl)#192.168.2.1
+>(tcl)#192.168.3.1
+>(tcl)#192.168.4.1
+>(tcl)#192.168.4.129
+>(tcl)#} { ping $address }

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/42/44 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/41/44 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.23.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/29/36 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.34.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/60 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/43/44 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/32 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.4.129, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms


R1#show ip route    // OBSERVE THE NEXT-HOP IP BEFORE PBR
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     172.16.0.0/29 is subnetted, 4 subnets
D       172.16.34.0 [90/41024000] via 172.16.13.3, 00:08:10, Serial0/0/1
D       172.16.23.0 [90/21024000] via 172.16.12.2, 00:08:25, Serial0/0/0
C       172.16.12.0 is directly connected, Serial0/0/0
C       172.16.13.0 is directly connected, Serial0/0/1
     192.168.4.0/25 is subnetted, 2 subnets
D       192.168.4.0 [90/41152000] via 172.16.13.3, 00:07:25, Serial0/0/1
D       192.168.4.128 [90/41152000] via 172.16.13.3, 00:07:25, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback1
D    192.168.2.0/24 [90/20640000] via 172.16.12.2, 00:08:25, Serial0/0/0
D    192.168.3.0/24 [90/21152000] via 172.16.12.2, 00:08:25, Serial0/0/0


R4#traceroute 192.168.1.1 source 192.168.4.1

Type escape sequence to abort.
Tracing the route to 192.168.1.1

  1 172.16.34.3 12 msec 12 msec 16 msec
  2 172.16.23.2 20 msec 20 msec 20 msec
  3 172.16.12.1 28 msec 24 msec *

R4#traceroute 192.168.1.1 source 192.168.4.129

Type escape sequence to abort.
Tracing the route to 192.168.1.1

  1 172.16.34.3 12 msec 12 msec 16 msec
  2 172.16.23.2 20 msec 20 msec 20 msec
  3 172.16.12.1 28 msec 24 msec *


R3#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     172.16.0.0/29 is subnetted, 4 subnets
C       172.16.34.0 is directly connected, Serial0/1/0
C       172.16.23.0 is directly connected, Serial0/0/1
D       172.16.12.0 [90/21024000] via 172.16.23.2, 00:14:01, Serial0/0/1
C       172.16.13.0 is directly connected, Serial0/0/0
     192.168.4.0/25 is subnetted, 2 subnets
D       192.168.4.0 [90/40640000] via 172.16.34.4, 00:13:01, Serial0/1/0
D       192.168.4.128 [90/40640000] via 172.16.34.4, 00:13:01, Serial0/1/0
D    192.168.1.0/24 [90/21152000] via 172.16.23.2, 00:14:02, Serial0/0/1     // R3 PREFERS VIA R2 HOP TO REACH R1 LAN IP SUBNET
D    192.168.2.0/24 [90/20640000] via 172.16.23.2, 00:14:02, Serial0/0/1
C    192.168.3.0/24 is directly connected, Loopback3


R3#show interfaces s0/0/0
Serial0/0/0 is up, line protocol is up
  Hardware is GT96K Serial
  Description: R3 --> R1
  Internet address is 172.16.13.3/29
  MTU 1500 bytes, BW 64 Kbit/sec, DLY 20000 usec,    
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  Last input 00:00:01, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     525 packets input, 37448 bytes, 0 no buffer
     Received 205 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     497 packets output, 36003 bytes, 0 underruns
     0 output errors, 0 collisions, 7 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

R3#show interfaces s0/0/1
Serial0/0/1 is up, line protocol is up
  Hardware is GT96K Serial
  Description: R3 --> R2
  Internet address is 172.16.23.3/29
  MTU 1500 bytes, BW 128 Kbit/sec, DLY 20000 usec,   
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  Last input 00:00:02, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 96 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     568 packets input, 41722 bytes, 0 no buffer
     Received 208 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     565 packets output, 42025 bytes, 0 underruns
     0 output errors, 0 collisions, 7 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up
R3#show ip eigrp topology 192.168.1.0
IP-EIGRP (AS 1): Topology entry for 192.168.1.0/24
  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 21152000
  Routing Descriptor Blocks:
  172.16.23.2 (Serial0/0/1), from 172.16.23.2, Send flag is 0x0
      Composite metric is (21152000/20640000), Route is Internal    // SUCCESSOR ROUTE, INSALLTED IN ROUTING TABLE
      Vector metric:
        Minimum bandwidth is 128 Kbit  
        Total delay is 45000 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1500
        Hop count is 2
  172.16.13.1 (Serial0/0/0), from 172.16.13.1, Send flag is 0x0
      Composite metric is (40640000/128256), Route is Internal   // METRIC FOR R1 LAN IP SUBNET IS HIGHER
      Vector metric:
        Minimum bandwidth is 64 Kbit  
        Total delay is 25000 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1500
        Hop count is 1


R3(config)#ip access-list standard PBR_ACL
R3(config-std-nacl)#remark ACL matches R4 LAN B traffic
R3(config-std-nacl)#permit 192.168.4.128 0.0.0.127
R3(config-std-nacl)#exit
R3(config)#route-map ?
  WORD  Route map tag

R3(config)#route-map R3_TO_R1 ?
  <0-65535>  Sequence to insert to/delete from existing route-map entry
  deny       Route map denies set operations
  permit     Route map permits set operations
  <cr>

R3(config)#route-map R3_TO_R1 permit ?
  <0-65535>  Sequence to insert to/delete from existing route-map entry
  <cr>

R3(config)#route-map R3_TO_R1 permit
R3(config-route-map)#?
Route Map configuration commands:
  continue     Continue on a different entry within the route-map
  default      Set a command to its defaults
  description  Route-map comment
  exit         Exit from route-map configuration mode
  help         Description of the interactive help system
  match        Match values from routing table
  no           Negate a command or set its defaults
  set          Set values in destination routing protocol

R3(config-route-map)#match ?
  as-path           Match BGP AS path list
  clns              CLNS information
  community         Match BGP community list
  extcommunity      Match BGP/VPN extended community list
  interface         Match first hop interface of route
  ip                IP specific information
  ipv6              IPv6 specific information
  length            Packet length
  local-preference  Local preference for route
  mdt-group         Match routes corresponding to MDT group
  metric            Match metric of route
  mpls-label        Match routes which have MPLS labels
  nlri              BGP NLRI type
  policy-list       Match IP policy list
  route-type        Match route-type of route
  source-protocol   Match source-protocol of route
  tag               Match tag of route

R3(config-route-map)#match ip ?
  address       Match address of route or match packet
  next-hop      Match next-hop address of route
  route-source  Match advertising source address of route

R3(config-route-map)#match ip address ?
  <1-199>      IP access-list number
  <1300-2699>  IP access-list number (expanded range)
  WORD         IP access-list name
  prefix-list  Match entries of prefix-lists

R3(config-route-map)#match ip address PBR_ACL
R3(config-route-map)#set ?
  as-path           Prepend string for a BGP AS-path attribute
  automatic-tag     Automatically compute TAG value
  clns              OSI summary address
  comm-list         set BGP community list (for deletion)
  community         BGP community attribute
  dampening         Set BGP route flap dampening parameters
  default           Set default information
  extcomm-list      Set BGP/VPN extended community list (for deletion)
  extcommunity      BGP extended community attribute
  global            Set to global routing table
  interface         Output interface
  ip                IP specific information
  ipv6              IPv6 specific information
  level             Where to import route
  local-preference  BGP local preference path attribute
  metric            Metric value for destination routing protocol
  metric-type       Type of metric for destination routing protocol
  mpls-label        Set MPLS label for prefix
  nlri              BGP NLRI type
  origin            BGP origin code
  tag               Tag value for destination routing protocol
  traffic-index     BGP traffic classification number for accounting
  vrf               Define VRF name
  weight            BGP weight for routing table

R3(config-route-map)#set ip ?
  address     Specify IP address
  default     Set default information
  df          Set DF bit
  global      global routing table
  next-hop    Next hop address
  precedence  Set precedence field
  qos-group   Set QOS Group ID
  tos         Set type of service field
  vrf         VRF name

R3(config-route-map)#set ip next-hop ?
  A.B.C.D              IP address of next hop
  dynamic              application dynamically sets next hop
  peer-address         Use peer address (for BGP only)
  recursive            Recursive next-hop
  verify-availability  Verify if nexthop is reachable

R3(config-route-map)#set ip next-hop 172.16.13.1
R3(config-route-map)#exit
R3(config)#interface s0/1/0
R3(config-if)#ip ?
Interface IP configuration subcommands:
  access-group        Specify access control for packets
  accounting          Enable IP accounting on this interface
  address             Set the IP address of an interface
  admission           Apply Network Admission Control
  auth-proxy          Apply authentication proxy
  authentication      authentication subcommands
  bandwidth-percent   Set EIGRP bandwidth limit
  bgp                 BGP interface commands
  broadcast-address   Set the broadcast address of an interface
  cef                 Cisco Express Forwarding interface commands
  cgmp                Enable/disable CGMP
  ddns                Configure dynamic DNS
  dhcp                Configure DHCP parameters for this interface
  directed-broadcast  Enable forwarding of directed broadcasts
  dns                 Configure DNS server
  dvmrp               DVMRP interface commands
  flow                NetFlow related commands
  header-compression  IPHC options
  hello-interval      Configures IP-EIGRP hello interval
  helper-address      Specify a destination address for UDP broadcasts
  hold-time           Configures IP-EIGRP hold time
  idle-group          Specify interesting packets for idle-timer
  igmp                IGMP interface commands
  information-reply   Enable sending ICMP Information Reply messages
  inspect             Apply inspect name
  ips                 Create IPS rule
  irdp                ICMP Router Discovery Protocol
  load-sharing        Style of load sharing
  local-proxy-arp     Enable local-proxy ARP
  mask-reply          Enable sending ICMP Mask Reply messages
  mobile              Mobile IP support
  mrm                 Configure IP Multicast Routing Monitor tester
  mroute-cache        Enable switching cache for incoming multicast packets
  mtu                 Set IP Maximum Transmission Unit
  multicast           IP multicast interface commands
  nat                 NAT interface commands
  nbar                Network-Based Application Recognition
  next-hop-self       Configures IP-EIGRP next-hop-self
  ospf                OSPF interface commands
  pgm                 PGM Reliable Transport Protocol
  pim                 PIM interface commands
  policy              Enable policy routing
  proxy-arp           Enable proxy ARP
  proxy-mobile        Enable Proxy Mobile IP services
  rarp-server         Enable RARP server for static arp entries
  rbscp               RBSCP subfeatures for this interface
  redirects           Enable sending ICMP Redirect messages
  rgmp                Enable/disable RGMP
  rip                 Router Information Protocol
  route-cache         Enable fast-switching cache for outgoing packets
  router              IP router interface commands
  rsvp                RSVP Interface Commands
  rtp                 RTP parameters
  sap                 Session Announcement Protocol interface commands
  security            DDN IP Security Option
  service             IP service
  split-horizon       Perform split horizon
  summary-address     Perform address summarization
  tcp                 TCP header compression and other parameters
  traffic-export      Configure this interface for exporting ip traffic
  unnumbered          Enable IP processing without an explicit address
  unreachables        Enable sending ICMP Unreachable messages
  urd                 Configure URL Rendezvousing
  verify              Enable per packet validation
  virtual-reassembly  Enable Virtual Fragment Reassembly
  vrf                 VPN Routing/Forwarding parameters on the interface
  wccp                WCCP interface commands

R3(config-if)#ip policy ?
  route-map  Policy route map

R3(config-if)#ip policy route-map ?
  WORD  Route map name

R3(config-if)#ip policy route-map R3_TO_R1
R3(config-if)#end
R3#
*Dec 28 00:42:40.059: %SYS-5-CONFIG_I: Configured from console by console
R3#show route-map
route-map R3_TO_R1, permit, sequence 10
  Match clauses:
    ip address (access-lists): PBR_ACL
  Set clauses:
  Policy routing matches: 0 packets, 0 bytes    // NO PACKETS MATCHING THE ACL YET


R3(config)#access-list 1 permit 192.168.4.0 0.0.0.255     // CREATE STANDARD ACL TO CAPTURE R4 LAN

TRAFFIC
R3(config)#exit
R3#
*Dec 28 00:45:19.615: %SYS-5-CONFIG_I: Configured from console by console
R3#debug ip policy ?
  <1-199>  Access list
  dynamic  dynamic PBR
  <cr>

R3#debug ip policy 1     // DEBUG PBR
Policy routing debugging is on for access list 1

R3#show debug

Policy Routing:
  Policy routing debugging is on for access list 1


R3#
*Dec 28 00:55:39.295: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, po
licy rejected -- normal forwarding
*Dec 28 00:55:48.307: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, po
licy rejected -- normal forwarding
*Dec 28 00:55:48.323: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, po
licy rejected -- normal forwarding
*Dec 28 00:55:48.335: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding
*Dec 28 00:55:57.355: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding
*Dec 28 00:55:57.375: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding
*Dec 28 00:55:57.395: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding
*Dec 28 00:56:06.423: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding
*Dec 28 00:56:06.451: IP: s=192.168.4.1 (Serial0/1/0), d=192.168.1.1, len 28, FI
B policy rejected(no match) - normal forwarding


R4#traceroute 192.168.1.1 source 192.168.4.1

Type escape sequence to abort.
Tracing the route to 192.168.1.1

  1 172.16.34.3 12 msec 12 msec 16 msec
  2 172.16.23.2 20 msec 20 msec 20 msec
  3 172.16.12.1 28 msec 24 msec *      // R3 > R2 > R1


R3#
*Dec 28 01:00:19.183: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
policy match
*Dec 28 01:00:19.183: IP: route map R3_TO_R1, item 10, permit
*Dec 28 01:00:19.183: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1 (Serial0/
0/0), len 28, policy routed
*Dec 28 01:00:19.183: IP: Serial0/1/0 to Serial0/0/0 172.16.13.1
*Dec 28 01:00:28.199: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
policy match
*Dec 28 01:00:28.199: IP: route map R3_TO_R1, item 10, permit
*Dec 28 01:00:28.199: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1 (Serial0/
0/0), len 28, policy routed
*Dec 28 01:00:28.199: IP: Serial0/1/0 to Serial0/0/0 172.16.13.1
*Dec 28 01:00:28.215: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
policy match
*Dec 28 01:00:28.215: IP: route map R3_TO_R1, item 10, permit
*Dec 28 01:00:28.215: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1 (Serial0/
0/0), len 28, policy routed
*Dec 28 01:00:28.215: IP: Serial0/1/0 to Serial0/0/0 172.16.13.1
*Dec 28 01:00:28.227: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
FIB policy match
*Dec 28 01:00:28.231: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
PBR Counted
*Dec 28 01:00:28.231: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, g=172.16
.13.1, len 28, FIB policy routed
*Dec 28 01:00:37.255: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
FIB policy match
*Dec 28 01:00:37.255: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
PBR Counted
*Dec 28 01:00:37.255: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, g=172.16
.13.1, len 28, FIB policy routed
*Dec 28 01:00:37.283: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
FIB policy match
*Dec 28 01:00:37.283: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, len 28,
PBR Counted
*Dec 28 01:00:37.283: IP: s=192.168.4.129 (Serial0/1/0), d=192.168.1.1, g=172.16
.13.1, len 28, FIB policy routed


R4#traceroute 192.168.1.1 source 192.168.4.129

Type escape sequence to abort.
Tracing the route to 192.168.1.1

  1 172.16.34.3 12 msec 12 msec 16 msec
  2 172.16.13.1 28 msec 28 msec *      // R3 > R1


R3#show route-map
route-map R3_TO_R1, permit, sequence 10
  Match clauses:
    ip address (access-lists): PBR_ACL
  Set clauses:
    ip next-hop 172.16.13.1
  Policy routing matches: 15 packets, 480 bytes     // PACKETS MATCHED PBR ACL

Sunday, February 1, 2015

Configure IP SLA Tracking and Path Control

I recently visited the "City of Smiles, Bangkok" for an IT project. I love their mangoes with sweet sticky rice and young coconut. Their mangoes is similar to our Philippine (Cebu) mangoes. I also noticed that you would see the image of their king scattered everywhere.



Back in my ROUTE lab, I did IP SLA and path control using tracking and floating static route. We've implemented this type of scenario at work for large sites that's dual-homed or have a secondary WAN link.


R1(config)#interface loopback0
R1(config-if)#
*Jan  1 01:32:43.871: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
R1(config-if)#description R1 LAN
R1(config-if)#ip address 192.168.1.1 255.255.255.0
R1(config-if)#interface s0/0/0
R1(config-if)#description R1 --> ISP1
R1(config-if)#ip address 209.165.201.2 255.255.255.252
R1(config-if)#clock rate 128000
R1(config-if)#bandwidth 128
R1(config-if)#no shutdown
R1(config-if)#
*Jan  1 01:33:46.195: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
R1(config-if)#interface s0/0/1
R1(config-if)#description R1 --> ISP2
R1(config-if)#ip address 209.165.202.130 255.255.255.252
R1(config-if)#bandwidth 128
R1(config-if)#no shutdown
R1(config-if)#
*Jan  1 01:34:24.539: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to down


ISP1(config)#interface loopback0
ISP1(config-if)#description
*Jan  1 00:17:19.591: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
ISP1(config-if)#description Simulated Internet Web Server
ISP1(config-if)#ip address 209.165.200.254 255.255.255.255
ISP1(config-if)#interface loopback1
ISP1(config-if)#
*Jan  1 00:17:55.571: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up
ISP1(config-if)#description ISP1 DNS Server
ISP1(config-if)#ip address 209.165.201.30 255.255.255.255
ISP1(config-if)#interface s0/0/0
ISP1(config-if)#description ISP1 --> R1
ISP1(config-if)#ip address 209.165.201.1 255.255.255.252
ISP1(config-if)#bandwidth 128
ISP1(config-if)#no shutdown
ISP1(config-if)#
*Jan  1 00:18:58.511: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Jan  1 00:18:59.511: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
ISP1(config-if)#do ping 209.165.201.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.201.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
ISP1(config-if)#interface s0/0/1
ISP1(config-if)#description ISP1 --> ISP2
ISP1(config-if)#ip address 209.165.200.225 255.255.255.252
ISP1(config-if)#clock rate 128000
ISP1(config-if)#bandwidth 128
ISP1(config-if)#no shutdown
ISP1(config-if)#
*Jan  1 00:20:21.971: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to up
*Jan  1 00:20:22.971: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/1, changed state to up


ISP2(config)#interface loopback0
ISP2(config-if)#
*Jan  1 00:58:26.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
ISP2(config-if)#description Simulated Internet Web Server
ISP2(config-if)#ip address 209.165.200.254 255.255.255.255
ISP2(config-if)#interface loopback1
*Jan  1 00:58:59.423: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up
ISP2(config-if)#description ISP2 DNS Server
ISP2(config-if)#ip address 209.165.202.158 255.255.255.255
ISP2(config-if)#interface s0/0/0
ISP2(config-if)#description ISP2 --> R1
ISP2(config-if)#ip address 209.165.202.129 255.255.255.252
ISP2(config-if)#clock rate 128000
ISP2(config-if)#bandwidth 128
ISP2(config-if)#no shutdown
ISP2(config-if)#
*Jan  1 00:59:59.771: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Jan  1 01:00:00.771: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
ISP2(config-if)#do ping 209.165.202.130

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.202.130, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms
ISP2(config-if)#interface s0/0/1
ISP2(config-if)#description ISP2 --> ISP1
ISP2(config-if)#ip address 209.165.200.226 255.255.255.252
ISP2(config-if)#bandwidth 128
ISP2(config-if)#no shutdown
ISP2(config-if)#do ping 209.165.200.225

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.200.225, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms


R1#show interfaces description
Interface                      Status         Protocol Description
Fa0/0                          admin down     down
Fa0/1                          admin down     down
Se0/0/0                        up             up       R1 --> ISP1
Se0/0/1                        up             up       R1 --> ISP2
Lo0                            up             up       R1 LAN

R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.201.1


ISP1(config)#ip route 192.168.1.0 255.255.255.0 209.165.201.2
ISP1(config)#router eigrp 1
ISP1(config-router)#no auto-summary
ISP1(config-router)#network 209.165.200.224 0.0.0.3
ISP1(config-router)#network 209.165.201.0 0.0.0.31


ISP2(config)#ip route 192.168.1.0 255.255.255.0 209.165.202.130
ISP2(config)#router eigrp 1
ISP2(config-router)#no auto-summary
ISP2(config-router)#network 209.165.200.224 0.0.0.3
ISP2(config-router)#
*Jan  1 01:27:41.067: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 209.165.200.225 (Serial0/0/1) is up: new adjacency
ISP2(config-router)#network 209.165.202.128 0.0.0.31


R1#tclsh
R1(tcl)#foreach address {
+>(tcl)#209.165.200.254
+>(tcl)#209.165.201.30
+>(tcl)#209.165.202.158
+>(tcl)#} {
+>(tcl)#ping $address source 192.168.1.1
+>(tcl)#}

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.200.254, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.201.30, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.165.202.158, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms


R1#tclsh
R1(tcl)#foreach address {
+>(tcl)#209.165.200.254
+>(tcl)#209.165.201.30
+>(tcl)#209.165.202.158
+>(tcl)#} {
+>(tcl)#trace $address source 192.168.1.1
+>(tcl)#}

Type escape sequence to abort.
Tracing the route to 209.165.200.254

  1 209.165.201.1 8 msec 8 msec *    // ALL TRACEROUTE GOES TO ISP1 HOP
Type escape sequence to abort.
Tracing the route to 209.165.201.30

  1 209.165.201.1 8 msec 8 msec *
Type escape sequence to abort.
Tracing the route to 209.165.202.158

  1 209.165.201.1 8 msec 8 msec 4 msec
  2 209.165.200.226 12 msec 8 msec *


R1(config)#ip sla ?
  <1-2147483647>          Entry Number
  enable                  Enable Event Notifications
  ethernet-monitor        IP SLAs Auto Ethernet configuration
  group                   Group Configuration or Group Scheduling
  key-chain               Use MD5 authentication for IP SLAs Control Messages
  logging                 Enable Syslog
  low-memory              Configure Low Water Memory Mark
  reaction-configuration  IP SLAs Reaction-Configuration
  reaction-trigger        IP SLAs Trigger Assignment
  reset                   IP SLAs Reset
  responder               Enable IP SLAs Responder
  restart                 Restart An Active Entry
  schedule                IP SLAs Entry Scheduling

R1(config)#ip sla 11
R1(config-ip-sla)#?
IP SLAs entry configuration commands:
  dhcp         DHCP Operation
  dns          DNS Query Operation
  ethernet     Ethernet Operations
  exit         Exit Operation Configuration
  frame-relay  Frame-relay Operation
  ftp          FTP Operation
  http         HTTP Operation
  icmp-echo    ICMP Echo Operation
  icmp-jitter  ICMP Jitter Operation
  path-echo    Path Discovered ICMP Echo Operation
  path-jitter  Path Discovered ICMP Jitter Operation
  tcp-connect  TCP Connect Operation
  udp-echo     UDP Echo Operation
  udp-jitter   UDP Jitter Operation
  voip         Voice Over IP Operation

R1(config-ip-sla)#icmp-echo ?
  Hostname or A.B.C.D  Destination IP address or hostname, broadcast disallowed

R1(config-ip-sla)#icmp-echo 209.165.201.30
R1(config-ip-sla-echo)#?
IP SLAs echo Configuration Commands:
  default            Set a command to its defaults
  exit               Exit operation configuration
  frequency          Frequency of an operation
  history            History and Distribution Data
  no                 Negate a command or set its defaults
  owner              Owner of Entry
  request-data-size  Request data size
  tag                User defined tag
  threshold          Operation threshold in milliseconds
  timeout            Timeout of an operation
  tos                Type Of Service
  verify-data        Verify data
  vrf                Configure IP SLAs for a VPN Routing/Forwarding instance

R1(config-ip-sla-echo)#frequency ?
  <1-604800>  Frequency in seconds (default 60)

R1(config-ip-sla-echo)#frequency 10
R1(config-ip-sla-echo)#exit
R1(config)#ip sla ?
  <1-2147483647>          Entry Number
  enable                  Enable Event Notifications
  ethernet-monitor        IP SLAs Auto Ethernet configuration
  group                   Group Configuration or Group Scheduling
  key-chain               Use MD5 authentication for IP SLAs Control Messages
  logging                 Enable Syslog
  low-memory              Configure Low Water Memory Mark
  reaction-configuration  IP SLAs Reaction-Configuration
  reaction-trigger        IP SLAs Trigger Assignment
  reset                   IP SLAs Reset
  responder               Enable IP SLAs Responder
  restart                 Restart An Active Entry
  schedule                IP SLAs Entry Scheduling

R1(config)#ip sla schedule ?
  <1-2147483647>  Entry number

R1(config)#ip sla schedule 11 ?
  ageout      How long to keep this Entry when inactive
  life        Length of time to execute in seconds
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R1(config)#ip sla schedule 11 life ?
  <0-2147483647>  Life seconds (default 3600)
  forever         continue running forever

R1(config)#ip sla schedule 11 life forever ?
  ageout      How long to keep this Entry when inactive
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R1(config)#ip sla schedule 11 life forever start-time ?
  after     Start after a certain amount of time from now
  hh:mm     Start time (hh:mm)
  hh:mm:ss  Start time (hh:mm:ss)
  now       Start now
  pending   Start pending

R1(config)#ip sla schedule 11 life forever start-time now

R1#show ip sla ?
  application             IP SLAs Application
  authentication          IP SLAs Authentication Information
  auto                    Auto IP SLAs show commands
  configuration           IP SLAs Configuration
  enhanced-history        IP SLAs Enhanced History
  ethernet-monitor        IP SLAs Auto Ethernet Monitor
  event-publisher         IP SLAs Event Publisher
  group                   IP SLAs Group Scheduling/Configuration
  history                 IP SLAs History
  reaction-configuration  IP SLAs Reaction Configuration
  reaction-trigger        IP SLAs Reaction Trigger
  responder               IP SLAs Responder Information
  statistics              IP SLAs Statistics

R1#show ip sla configuration ?
  <1-2147483647>  Entry Number
  |               Output modifiers
  <cr>

R1#show ip sla configuration 11
IP SLAs, Infrastructure Engine-II.
Entry number: 11   
Owner:
Tag:
Type of operation to perform: icmp-echo  
Target address/Source address: 209.165.201.30/0.0.0.0  
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Operation timeout (milliseconds): 5000
Verify data: No
Vrf Name:
Schedule:
   Operation frequency (seconds): 10  (not considered if randomly scheduled)
   Next Scheduled Start Time: Start Time already passed  
   Group Scheduled : FALSE
   Randomly Scheduled : FALSE
   Life (seconds): Forever   
   Entry Ageout (seconds): never
   Recurring (Starting Everyday): FALSE
   Status of entry (SNMP RowStatus): Active
Threshold (milliseconds): 5000 (not considered if react RTT is configured)
Distribution Statistics:
   Number of statistic hours kept: 2
   Number of statistic distribution buckets kept: 1
   Statistic distribution interval (milliseconds): 20
History Statistics:
   Number of history Lives kept: 0
   Number of history Buckets kept: 15
   History Filter Type: None
Enhanced History:


R1#show ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 11  
Type of operation: icmp-echo
        Latest RTT: 7 milliseconds
Latest operation start time: *02:29:26.851 UTC Thu Jan 1 2015
Latest operation return code: OK   
Number of successes: 19   
Number of failures: 0
Operation time to live: Forever   


R1(config)#ip sla 22
R1(config-ip-sla)#icmp-echo 209.165.202.158    // ISP2 DNS
R1(config-ip-sla-echo)#frequency 10
R1(config-ip-sla-echo)#exit
R1(config)#ip sla schedule 22 life forever start-time now

R1#show ip sla configuration 22
IP SLAs, Infrastructure Engine-II.
Entry number: 22
Owner:
Tag:
Type of operation to perform: icmp-echo
Target address/Source address: 209.165.202.158/0.0.0.0
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Operation timeout (milliseconds): 5000
Verify data: No
Vrf Name:
Schedule:
   Operation frequency (seconds): 10  (not considered if randomly scheduled)
   Next Scheduled Start Time: Start Time already passed
   Group Scheduled : FALSE
   Randomly Scheduled : FALSE
   Life (seconds): Forever
   Entry Ageout (seconds): never
   Recurring (Starting Everyday): FALSE
   Status of entry (SNMP RowStatus): Active
Threshold (milliseconds): 5000 (not considered if react RTT is configured)
Distribution Statistics:
   Number of statistic hours kept: 2
   Number of statistic distribution buckets kept: 1
   Statistic distribution interval (milliseconds): 20
History Statistics:
   Number of history Lives kept: 0
   Number of history Buckets kept: 15
   History Filter Type: None
Enhanced History:


R1#show ip sla statistics 22
IPSLAs Latest Operation Statistics

IPSLA operation id: 22
Type of operation: icmp-echo
        Latest RTT: 16 milliseconds
Latest operation start time: *02:31:35.067 UTC Thu Jan 1 2015
Latest operation return code: OK
Number of successes: 6
Number of failures: 0
Operation time to live: Forever


R1(config)#no ip route 0.0.0.0 0.0.0.0 209.165.201.1
R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.201.1 ?
  <1-255>    Distance metric for this route
  name       Specify name of the next hop
  permanent  permanent route
  tag        Set tag for this route
  track      Install route depending on tracked item
  <cr>

R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.201.1 5   // FLOATING STATIC ROUTE WITH AD OF 5

R1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 209.165.201.1 to network 0.0.0.0  

     209.165.201.0/30 is subnetted, 1 subnets
C       209.165.201.0 is directly connected, Serial0/0/0
     209.165.202.0/30 is subnetted, 1 subnets
C       209.165.202.128 is directly connected, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback0
S*   0.0.0.0/0 [5/0] via 209.165.201.1  


R1(config)#track ?
  <1-500>     Tracked object
  resolution  Tracking resolution parameters
  timer       Polling interval timers

R1(config)#track 1 ?
  application  Application
  interface    Select an interface to track
  ip           IP protocol
  list         Group objects in a list
  stub-object  Stub tracking object

R1(config)#track 1 ip ?
  route  IP route
  sla    IP Service Level Agreement

R1(config)#track 1 ip sla ?
  <1-2147483647>  Entry number

R1(config)#track 1 ip sla 11 ?
  reachability  Reachability
  state         Return code state
  <cr>

R1(config)#track 1 ip sla 11 reachability
R1(config-track)#?
Tracking instance configuration commands:
  default        Set a command to its defaults
  default-state  Default object state
  delay          Tracking delay
  exit           Exit from tracking configuration mode
  no             Negate a command or set its defaults

R1(config-track)#delay ?
  down  Delay down change notification
  up    Delay up change notification

R1(config-track)#delay down ?
  <0-180>  Seconds to delay

R1(config-track)#delay down 10 up ?
  <0-180>  Seconds to delay

R1(config-track)#delay down 10 up 1


R1#debug ip routing   // DEBUG ROUTING CHANGES
IP routing debugging is on
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.201.1 2 ?
  name       Specify name of the next hop
  permanent  permanent route
  tag        Set tag for this route
  track      Install route depending on tracked item
  <cr>

R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.201.1 2 track 1
R1(config)#
*Jan  1 02:36:57.055: RT: closer admin distance for 0.0.0.0, flushing 1 routes  
*Jan  1 02:36:57.055: RT: NET-RED 0.0.0.0/0
*Jan  1 02:36:57.055: RT: add 0.0.0.0/0 via 209.165.201.1, static metric [2/0]  
*Jan  1 02:36:57.055: RT: NET-RED 0.0.0.0/0
*Jan  1 02:36:57.055: RT: default path is now 0.0.0.0 via 209.165.201.1
*Jan  1 02:36:57.055: RT: new default network 0.0.0.0
*Jan  1 02:36:57.055: RT: NET-RED 0.0.0.0/0
*Jan  1 02:37:02.055: RT: NET-RED 0.0.0.0/0
R1(config)#do show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 209.165.201.1 to network 0.0.0.0

     209.165.201.0/30 is subnetted, 1 subnets
C       209.165.201.0 is directly connected, Serial0/0/0
     209.165.202.0/30 is subnetted, 1 subnets
C       209.165.202.128 is directly connected, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback0
S*   0.0.0.0/0 [2/0] via 209.165.201.1   // ROUTE WITH BETTER (LOWER) AD OF 2


R1(config)#track 2 ip sla 22 reachability
R1(config-track)#delay down 10 up 1
R1(config-track)#exit
R1(config)#ip route 0.0.0.0 0.0.0.0 209.165.202.129 3 track 2
R1(config)#do show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 209.165.201.1 to network 0.0.0.0

     209.165.201.0/30 is subnetted, 1 subnets
C       209.165.201.0 is directly connected, Serial0/0/0
     209.165.202.0/30 is subnetted, 1 subnets
C       209.165.202.128 is directly connected, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback0
S*   0.0.0.0/0 [2/0] via 209.165.201.1    // AD OF 2 IS BETTER THAN AD OF 3


ISP1(config)#interface loopback1
ISP1(config-if)#shutdown
ISP1(config-if)#
*Jan  1 01:27:38.019: %LINK-5-CHANGED: Interface Loopback1, changed state to administratively down
*Jan  1 01:27:39.019: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to down


R1#show debug
IP routing:
  IP routing debugging is on


R1#
*Jan  1 02:45:28.055: RT: NET-RED 0.0.0.0/0
*Jan  1 02:45:49.559: %TRACKING-5-STATE: 1 ip sla 11 reachability Up->Down  
*Jan  1 02:45:49.559: RT: del 0.0.0.0 via 209.165.201.1, static metric [2/0]   // DELETES DEFAULT ROUTE WITH AD OF 2
*Jan  1 02:45:49.559: RT: delete network route to 0.0.0.0
*Jan  1 02:45:49.559: RT: NET-RED 0.0.0.0/0
*Jan  1 02:45:49.559: RT: NET-RED 0.0.0.0/0
*Jan  1 02:45:49.559: RT: add 0.0.0.0/0 via 209.165.202.129, static metric [3/0]   // INSTALLS NEXT HIGHEST DEFAULT ROUTE TO ISP2 WITH AD OF 3
*Jan  1 02:45:49.559: RT: NET-RED 0.0.0.0/0
*Jan  1 02:45:49.559: RT: default path is now 0.0.0.0 via 209.165.202.129
*Jan  1 02:45:49.563: RT: new default network 0.0.0.0
*Jan  1 02:45:49.563: RT: NET-RED 0.0.0.0/0
R1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 209.165.202.129 to network 0.0.0.0  

C    1.0.0.0/8 is directly connected, FastEthernet0/0
     209.165.201.0/30 is subnetted, 1 subnets
C       209.165.201.0 is directly connected, Serial0/0/0
     209.165.202.0/30 is subnetted, 1 subnets
C       209.165.202.128 is directly connected, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback0
S*   0.0.0.0/0 [3/0] via 209.165.202.129   


R1#show ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 11
Type of operation: icmp-echo
        Latest RTT: NoConnection/Busy/Timeout  
Latest operation start time: *02:47:46.851 UTC Thu Jan 1 2015
Latest operation return code: No connection  
Number of successes: 113
Number of failures: 16  
Operation time to live: Forever



IPSLA operation id: 22
Type of operation: icmp-echo
        Latest RTT: 7 milliseconds
Latest operation start time: *02:47:55.063 UTC Thu Jan 1 2015
Latest operation return code: OK
Number of successes: 102
Number of failures: 2
Operation time to live: Forever


R1#traceroute 209.165.200.254 source 192.168.1.1

Type escape sequence to abort.
Tracing the route to 209.165.200.254

  1 209.165.202.129 8 msec 8 msec *    // R1 HOPS TO ISP2 TO REACH ISP1 WEB SERVER


ISP1(config)#interface loopback1
ISP1(config-if)#no shutdown
*Jan  1 01:32:08.915: %LINK-3-UPDOWN: Interface Loopback1, changed state to up
*Jan  1 01:32:09.915: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up


R1#
*Jan  1 02:49:28.055: RT: NET-RED 0.0.0.0/0
*Jan  1 02:50:10.579: %TRACKING-5-STATE: 1 ip sla 11 reachability Down->Up   
*Jan  1 02:50:10.579: RT: closer admin distance for 0.0.0.0, flushing 1 routes  
*Jan  1 02:50:10.579: RT: NET-RED 0.0.0.0/0
*Jan  1 02:50:10.579: RT: add 0.0.0.0/0 via 209.165.201.1, static metric [2/0]  
*Jan  1 02:50:10.579: RT: NET-RED 0.0.0.0/0
*Jan  1 02:50:10.579: RT: default path is now 0.0.0.0 via 209.165.201.1   // DEFAULT STATIC ROUTE IS BACK TO ISP1 WITH AD OF 2
*Jan  1 02:50:10.579: RT: new default network 0.0.0.0
*Jan  1 02:50:10.579: RT: NET-RED 0.0.0.0/0
*Jan  1 02:50:15.579: RT: NET-RED 0.0.0.0/0
*Jan  1 02:50:28.055: RT: NET-RED 0.0.0.0/0


R1#show ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 11
Type of operation: icmp-echo
        Latest RTT: 8 milliseconds
Latest operation start time: *02:51:16.851 UTC Thu Jan 1 2015
Latest operation return code: OK   
Number of successes: 121
Number of failures: 29
Operation time to live: Forever



IPSLA operation id: 22
Type of operation: icmp-echo
        Latest RTT: 16 milliseconds
Latest operation start time: *02:51:25.063 UTC Thu Jan 1 2015
Latest operation return code: OK
Number of successes: 123
Number of failures: 2
Operation time to live: Forever


R1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 209.165.201.1 to network 0.0.0.0   

C    1.0.0.0/8 is directly connected, FastEthernet0/0
     209.165.201.0/30 is subnetted, 1 subnets
C       209.165.201.0 is directly connected, Serial0/0/0
     209.165.202.0/30 is subnetted, 1 subnets
C       209.165.202.128 is directly connected, Serial0/0/1
C    192.168.1.0/24 is directly connected, Loopback0
S*   0.0.0.0/0 [2/0] via 209.165.201.1