Friday, May 17, 2019

Troubleshooting an MPLS VPN Network

MPLS LSP Ping

Because MPLS cannot work without IP on the network level, you can still use the IP ping when the network is running MPLS. The ping packets are labeled and label-switched throughout the network. Why invent MPLS LSP ping? Well, IP ping is insufficient for verifying the correctness of the MPLS LSP. Although it can verify whether the connectivity is present on the IP level, it does not verify whether the LSP is broken. If you have a plain IP-over-MPLS network and LDP is broken between two LSRs, ping indicates that there is no problem as the echo request makes it to the destination and the echo reply makes it back to the source. Between the two LSRs where the LDP session is broken, the packets are no longer labeled. The ping indicates falsely that everything is okay, when in reality the LSP is broken.

LSPs can break for any number of reasons, while the IP connectivity remains fine. Following are
some reasons an LSP could be broken:
* The LDP session is down.
* MPLS is not enabled on an LSR (or one interface).
* The LFIB has a wrong entry for that LSP (wrong in/out label or wrong outgoing next-hop
information).
* The software and hardware LFIB have a discrepancy.


MPLS LSP Traceroute


The goal of traceroute is to test the path, whereas the goal of ping is to test the connectivity. The goal of MPLS LSP traceroute is to test the path of the LSP and verify the control and data plane on every LSR along the path of the LSP.

An MPLS LSP traceroute is nothing more than an MPLS echo request. The difference with MPLS LSP ping is that MPLS LSP traceroute sends several MPLS echo request packets with increasing MPLS TTL. The first MPLS LSP traceroute probe has MPLS TTL 1, and for every subsequent probe, the TTL is increased by 1. One additional TLV that is included with the echo request for MPLS LSP traceroute is the Downstream Mapping TLV. If an LSR receives the MPLS echo request with TTL 1, it replies to it. If the LSR is not the egress LSR for the FEC and everything checks out fine, the LSR replies as one of the downstream routers. Therefore, it replies with a return code of 8 and the appropriate downstream TLV information (MTU, Address Type, Downstream IP Address, Downstream Interface Address, Multipath information, Downstream Label, and Protocol) filled in. If the LSR is the egress LSR for the FEC, it does not need to fill in the Downstream Mapping information but should reply with a return code of 3.

The MPLS echo request for an MPLS traceroute holds the following information:
* MPLS Echo Header
* Target FEC Stack TLV
* Downstream Mapping TLV
* A PAD TLV (optional)

The echo reply will hold the following information:
* MPLS Echo Header
* An Error Code TLV (optional)
* A PAD TLV (optional)
* Downstream Mapping TLV


R1-PE1#traceroute 10.5.5.5 
Type escape sequence to abort.
Tracing the route to 10.5.5.5
VRF info: (vrf in name/id, vrf out name/id)
  1 10.12.0.2 [MPLS: Label 202 Exp 0] 52 msec 48 msec 28 msec
  2 10.23.0.3 [MPLS: Label 304 Exp 0] 32 msec 60 msec 36 msec
  3 10.34.0.4 [MPLS: Label 402 Exp 0] 48 msec 8 msec 32 msec
  4 10.45.0.5 64 msec 92 msec 48 msec


R3-P2#show mpls ldp ?
  backoff           LDP session setup backoff table
  bindings          Show the LDP Label Information Base (LIB)
  capabilities      Display LDP Capabilities information
  discovery         Display sources for locally generated LDP Discovery Hello
                    PDUs
  graceful-restart  Show Graceful Restart summary
  igp               IGP-related info
  neighbor          Display LDP neighbor information
  parameters        Display LDP configuration parameters

R3-P2#show mpls ldp bindings
  lib entry: 10.1.1.1/32, rev 4
        local binding:  label: 301
        remote binding: lsr: 10.4.4.4:0, label: 400
        remote binding: lsr: 10.2.2.2:0, label: 200
  lib entry: 10.2.2.2/32, rev 6
        local binding:  label: 302
        remote binding: lsr: 10.4.4.4:0, label: 401
        remote binding: lsr: 10.2.2.2:0, label: imp-null
  lib entry: 10.3.3.3/32, rev 10
        local binding:  label: imp-null
        remote binding: lsr: 10.4.4.4:0, label: 405
        remote binding: lsr: 10.2.2.2:0, label: 205
  lib entry: 10.4.4.4/32, rev 12
        local binding:  label: 303
        remote binding: lsr: 10.4.4.4:0, label: imp-null
        remote binding: lsr: 10.2.2.2:0, label: 201
  lib entry: 10.5.5.5/32, rev 14
        local binding:  label: 304
        remote binding: lsr: 10.4.4.4:0, label: 402
        remote binding: lsr: 10.2.2.2:0, label: 202
  lib entry: 10.12.0.0/24, rev 2
        local binding:  label: 300
        remote binding: lsr: 10.4.4.4:0, label: 403
        remote binding: lsr: 10.2.2.2:0, label: imp-null
  lib entry: 10.23.0.0/24, rev 8
        local binding:  label: imp-null
        remote binding: lsr: 10.4.4.4:0, label: 404
        remote binding: lsr: 10.2.2.2:0, label: imp-null
  lib entry: 10.34.0.0/24, rev 16
        local binding:  label: imp-null
        remote binding: lsr: 10.4.4.4:0, label: imp-null
        remote binding: lsr: 10.2.2.2:0, label: 203
  lib entry: 10.45.0.0/24, rev 18
        local binding:  label: 305
        remote binding: lsr: 10.4.4.4:0, label: imp-null
        remote binding: lsr: 10.2.2.2:0, label: 204

R3-P2#show mpls ldp bindings ?
  A.B.C.D             Destination prefix
  advertisement-acls  Show advertisement access lists
  all                 Display LIB information in all VRFs
  detail              Show detailed information
  local               Display only locally assigned labels
  local-label         Match locally assigned label values
  neighbor            Display labels from LDP neighbor
  remote-label        Match remotely assigned label values
  vrf                 VRF Routing/Forwarding instance information
  |                   Output modifiers
  <cr>

R3-P2#show mpls ldp bindings local    // PER PLATFORM LABEL; USE SAME LABELS IN OTHER INTERFACES
  lib entry: 10.1.1.1/32, rev 4
        local binding:  label: 301
  lib entry: 10.2.2.2/32, rev 6
        local binding:  label: 302
  lib entry: 10.3.3.3/32, rev 10
        local binding:  label: imp-null      // DIRECTLY CONNECTED
  lib entry: 10.4.4.4/32, rev 12
        local binding:  label: 303
  lib entry: 10.5.5.5/32, rev 14
        local binding:  label: 304
  lib entry: 10.12.0.0/24, rev 2
        local binding:  label: 300
  lib entry: 10.23.0.0/24, rev 8
        local binding:  label: imp-null
  lib entry: 10.34.0.0/24, rev 16
        local binding:  label: imp-null
  lib entry: 10.45.0.0/24, rev 18
        local binding:  label: 305

R3-P2#show mpls ldp bindings 10.5.5.5 32 
  lib entry: 10.5.5.5/32, rev 14
        local binding:  label: 304
        remote binding: lsr: 10.4.4.4:0, label: 402      // TWO LABELS RECEIVED DUE TO LIBERAL LABEL RETENTION MODE
        remote binding: lsr: 10.2.2.2:0, label: 202

R3-P2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#no mpls ip
R3-P2(config)#
*Nov 12 15:41:25.884: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (1) is DOWN (LDP unconfigured globally)
*Nov 12 15:41:25.888: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (2) is DOWN (LDP unconfigured globally)
R3-P2(config)#
R3-P2(config)#mpls ip
R3-P2(config)#
*Nov 12 15:41:33.612: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (3) is UP
*Nov 12 15:41:33.644: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (4) is UP

R3-P2#show mpls ldp bindings local     
  lib entry: 10.1.1.1/32, rev 2
        local binding:  label: 306
  lib entry: 10.2.2.2/32, rev 4
        local binding:  label: 307
  lib entry: 10.3.3.3/32, rev 6
        local binding:  label: imp-null
  lib entry: 10.4.4.4/32, rev 8
        local binding:  label: 308
  lib entry: 10.5.5.5/32, rev 10
        local binding:  label: 309
  lib entry: 10.12.0.0/24, rev 12
        local binding:  label: 310
  lib entry: 10.23.0.0/24, rev 14
        local binding:  label: imp-null
  lib entry: 10.34.0.0/24, rev 16
        local binding:  label: imp-null
  lib entry: 10.45.0.0/24, rev 18
        local binding:  label: 311

R3-P2#show mpls ldp bindings 10.5.5.5 32
  lib entry: 10.5.5.5/32, rev 10
        local binding:  label: 309      // LABEL 304 PREVIOUSLY
        remote binding: lsr: 10.2.2.2:0, label: 202       // NO CHANGE
        remote binding: lsr: 10.4.4.4:0, label: 402      // NO CHANGE

R3-P2#show mpls forwarding-table 10.5.5.5   
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
309        402        10.5.5.5/32      0             Fa1/0      10.34.0.4


R3-P2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#access-list 1 permit 10.5.5.5
R3-P2(config)#end
R3-P2#
*Nov 12 16:03:51.768: %SYS-5-CONFIG_I: Configured from console by console
R3-P2#debug mpls ?
  access-list  MPLS ACCESS
  events       MPLS events
  ip           Debug MPLS IP information
  l2transport  Any transport over MPLS debug
  ldp          Label Distribution Protocol
  lspv         LSP Verification debug
  netflow      MPLS Egress NetFlow Accounting
  packet       mpls packet debug
  static       MPLS Static labels
  traffic-eng  MPLS Traffic Engineering debug

R3-P2#debug mpls ldp ?
  address-to-session  Address to LDP session mapping services
  advertisements      LDP label and address advertisements
  backoff             LDP session establishment backoff
  bindings            Label bindings and other Label Information Base (LIB)
                      changes
  capabilities        Capabilities debug
  dev                 MPLS development interested events
  graceful-restart    LDP Graceful Restart events
  iccp                InterChassis Control Protocol debug
  igp                 IGP-related interactions
  messages            LDP messages
  peer                LDP peer
  prev-label          Previous label protocol changes
  refcount            LDP instance reference count tracking
  session             LDP session
  targeted-neighbors  LDP targeted neighbor events
  transport           LDP transport and discovery

R3-P2#debug mpls ldp bindings ?
  peer-acl    Peer access list filter
  prefix-acl  Prefix access list filter
  <cr>

R3-P2#debug mpls ldp bindings prefix-acl ?
  WORD  ACL name

R3-P2#debug mpls ldp bindings prefix-acl 1       // LIMIT DEBUG BASED ON ACL 1 USING PREFIX 10.5.5.5
LDP Label Information Base (LIB) changes debugging is on for prefix ACL 1
R3-P2#
R3-P2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#no mpls ip
R3-P2(config)#
*Nov 12 16:04:32.496: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (3) is DOWN (LDP unconfigured globally)
*Nov 12 16:04:32.504: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (4) is DOWN (LDP unconfigured globally)
*Nov 12 16:04:32.708: tagcon: tibent(10.5.5.5/32): deleted
R3-P2(config)#
*Nov 12 16:04:34.100: tc_handle_bg_timer_event: TC not enabled, ctx # 0
*Nov 12 16:04:34.100: tagcon: (default) Deassign peer id; 10.2.2.2:0: id 2
*Nov 12 16:04:34.104: tagcon: (default) Deassign peer id; 10.4.4.4:0: id 3
R3-P2(config)#
R3-P2(config)#mpls ip  
R3-P2(config)#
R3-P2(config)#
*Nov 12 16:04:43.892: tib: prefix recurs walk start: 10.5.5.5/32, tableid: 0; tib entry not found
*Nov 12 16:04:43.892: tib: get path labels: 10.5.5.5/32(0), nh tableid: 0, Fa1/0, nh 10.34.0.4
*Nov 12 16:04:43.892: tagcon: tibent(10.5.5.5/32): created; find route tags request
*Nov 12 16:04:43.892: tib: Assign 10.5.5.5/32 nh 10.34.0.4 real label
*Nov 12 16:04:43.896: tagcon: tibent(10.5.5.5/32): label 304 (#10) assigned
*Nov 12 16:04:43.896: tib: add a route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), remote label Unknown
*Nov 12 16:04:43.896: tib: update route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), with remote label Unknown
*Nov 12 16:04:43.896: tagcon: announce labels for: 10.5.5.5/32; nh 10.34.0.4, Fa1/0, inlabel 304, outlabel unknown (from 0.0.0.0:0), get path labels
*Nov 12 16:04:44.008: tib: prefix recurs walk start: 10.5.5.5/32, tableid: 0
*Nov 12 16:04:44.008: tib: get path labels: 10.5.5.5/32(0), nh tableid: 0, Fa1/0, nh 10.34.0.4
*Nov 12 16:04:44.012: tib: Assign 10.5.5.5/32 nh 10.34.0.4 real label
*Nov 12 16:04:44.012: tib: found route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), remote label Unknown
*Nov 12 16:04:44.016: tib: update route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), with remote label Unknown
*Nov 12 16:04:44.020: tagcon: announce labels for: 10.5.5.5/32; nh 10.34.0.4, Fa1/0, inlabel 304, outlabel unknown (from 0.0.0.0:0), get path labels
*Nov 12 16:04:44.100: tagcon: (default) Assign peer id; 10.2.2.2:0: id 0
*Nov 12 16:04:44.100: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (1) is UP
*Nov 12 16:04:44.156: tagcon: (default) Assign peer id; 10.4.4.4:0: id 1
*Nov 12 16:04:44.156: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (2) is UP
*Nov 12 16:04:44.156: tagcon: 10.4.4.4:0: 10.45.0.4 added to addr<->ldp ident map
*Nov 12 16:04:44.160: tagcon: 10.4.4.4:0: 10.34.0.4 added to addr<->ldp ident map
*Nov 12 16:04:44.160: tagcon: 10.4.4.4:0: 10.4.4.4 added to addr<->ldp ident map
*Nov 12 16:04:44.160: tib: 10.5.5.5/32:: learn binding 402 from 10.4.4.4:0
*Nov 12 16:04:44.164: tib: a new binding to be added
*Nov 12 16:04:44.164: tagcon: tibent(10.5.5.5/32): label 402 from 10.4.4.4:0 added
*Nov 12 16:04:44.164: tib: next hop for route 10.5.5.5/32(0, 10.34.0.4, Fa1/0) is mapped to peer 10.4.4.4:0
*Nov 12 16:04:44.164: tib: invoke iprm label announcement for 10.5.5.5/32
*Nov 12 16:04:44.164: tib: prefix recurs walk start: 10.5.5.5/32, tableid: 0
*Nov 12 16:04:44.164: tib: get path labels: 10.5.5.5/32(0), nh tableid: 0, Fa1/0, nh 10.34.0.4
*Nov 12 16:04:44.164: tib: Assign 10.5.5.5/32 nh 10.34.0.4 real label
*Nov 12 16:04:44.164: tib: found route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), remote label Unknown
*Nov 12 16:04:44.164: tib: update route info for 10.5.5.5/32(0, 10.34.0.4, Fa1/0), with remote label 402 from 10.4.4.4:0
*Nov 12 16:04:44.164: tagcon: announce labels for: 10.5.5.5/32; nh 10.34.0.4, Fa1/0, inlabel 304, outlabel 402 (from 10.4.4.4:0), get path labels
*Nov 12 16:04:44.176: tagcon: 10.2.2.2:0: 10.12.0.2 added to addr<->ldp ident map
*Nov 12 16:04:44.176: tagcon: 10.2.2.2:0: 10.2.2.2 added to addr<->ldp ident map
*Nov 12 16:04:44.176: tagcon: 10.2.2.2:0: 10.23.0.2 added to addr<->ldp ident map
*Nov 12 16:04:44.180: tib: 10.5.5.5/32:: learn binding 202 from 10.2.2.2:0
*Nov 12 16:04:44.180: tib: a new binding to be added
*Nov 12 16:04:44.180: tagcon: tibent(10.5.5.5/32): label 202 from 10.2.2.2:0 added
*Nov 12 16:04:44.180: tib: next hop for route 10.5.5.5/32(0, 10.34.0.4, Fa1/0) is not mapped to peer 10.2.2.2:0
*Nov 12 16:04:44.180: tib: skip iprm label announcement for 10.5.5.5/32
R3-P2(config)#
R3-P2(config)#end
R3-P2#u all
All possible debugging has been turned off

R3-P2#show mpls ldp bindings 10.5.5.5 32
  lib entry: 10.5.5.5/32, rev 10
        local binding:  label: 304
        remote binding: lsr: 10.4.4.4:0, label: 402
        remote binding: lsr: 10.2.2.2:0, label: 202


MPLS LDP Hello Message uses UDP Port 646 and it's sourced from the neighbor's interface IP address.



Expand User Datagram Protocol > click Source Port:646 > right-click > Prepare a Filter > Selected


In order to view the whole LDP session, type || tcp.port == 646 on the search filter (Expression) > press Enter.


MPLS LFIB exchange is established via TCP three-way handshake on port 646.





R3-P2#show mpls ldp neighbor
    Peer LDP Ident: 10.2.2.2:0; Local LDP Ident 10.3.3.3:0
        TCP connection: 10.2.2.2.646 - 10.3.3.3.47607      // R3 HAS THE HIGHER ROUTER ID (LOOPBACK IP) AND INITIATED THE TCP SESSION (3-WAY HANDSHAKE) USING HIGHER RANDOM (EPHEMERAL) SOURCE PORT GOING TO MPLS WELL KNOWN PORT 646
        State: Oper; Msgs sent/rcvd: 37/36; Downstream
        Up time: 00:21:56
        LDP discovery sources:
          FastEthernet1/1, Src IP addr: 10.23.0.2
        Addresses bound to peer LDP Ident:
          10.12.0.2       10.2.2.2        10.23.0.2         // LDP NEIGHBOR ALSO ADVERTISE ITS KNOWN NETWORKS
    Peer LDP Ident: 10.4.4.4:0; Local LDP Ident 10.3.3.3:0
        TCP connection: 10.4.4.4.57765 - 10.3.3.3.646
        State: Oper; Msgs sent/rcvd: 37/37; Downstream
        Up time: 00:21:56
        LDP discovery sources:
          FastEthernet1/0, Src IP addr: 10.34.0.4
        Addresses bound to peer LDP Ident:
          10.45.0.4       10.34.0.4       10.4.4.4


R5-PE2#show control-plane host open-ports    // VERIFY OPENED PORTS FOBGP AND MPLS PING/TRACEROUTE
Active internet connections (servers and established)
Prot               Local Address             Foreign Address                  Service    State
 tcp                     *:51815             192.168.1.2:179         IOS host service ESTABLIS
 tcp                        *:23                         *:0                   Telnet   LISTEN
 tcp                     *:31529                10.4.4.4:646         IOS host service ESTABLIS
 tcp                     *:30089                10.1.1.1:179         IOS host service ESTABLIS
 tcp                       *:179                         *:0                      BGP   LISTEN
 tcp                       *:179                         *:0                      BGP   LISTEN
 udp                      *:3503                         *:0                LSPV Echo   LISTEN


R3-P2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 11 subnets, 2 masks
O        10.1.1.1/32 [110/3] via 10.23.0.2, 1d04h, FastEthernet1/1
O        10.2.2.2/32 [110/2] via 10.23.0.2, 1d04h, FastEthernet1/1
C        10.3.3.3/32 is directly connected, Loopback0
O        10.4.4.4/32 [110/2] via 10.34.0.4, 1d04h, FastEthernet1/0
O        10.5.5.5/32 [110/3] via 10.34.0.4, 1d04h, FastEthernet1/0
O        10.12.0.0/24 [110/2] via 10.23.0.2, 1d04h, FastEthernet1/1
C        10.23.0.0/24 is directly connected, FastEthernet1/1
L        10.23.0.3/32 is directly connected, FastEthernet1/1
C        10.34.0.0/24 is directly connected, FastEthernet1/0
L        10.34.0.3/32 is directly connected, FastEthernet1/0
O        10.45.0.0/24 [110/2] via 10.34.0.4, 1d04h, FastEthernet1/0

R3-P2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#ip route 10.2.2.2 255.255.255.255 10.34.0.4     // WRONG STATIC ROUTE
R3-P2(config)#
R3-P2(config)#no mpls ip
R3-P2(config)#
*Nov 12 16:49:13.188: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (3) is DOWN (LDP unconfigured globally)
*Nov 12 16:49:13.196: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (4) is DOWN (LDP unconfigured globally)
R3-P2(config)#
*Nov 12 16:49:13.356: %MPLS_PACKET-4-NOLFDSB: MPLS packet received on non MPLS enabled interface FastEthernet1/0 L3 type 0x8847 label {303 6 1 254}
R3-P2(config)#
R3-P2(config)#mpls ip
R3-P2(config)#
*Nov 12 16:49:23.056: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (1) is UP      // LDP NEIGHBOR R2 DIDN'T GO UP; IT CAN'T REACH R2 LOOPBACK IP 10.2.2.2
R3-P2(config)#
R3-P2(config)#do show mpls ldp neighbor
    Peer LDP Ident: 10.4.4.4:0; Local LDP Ident 10.3.3.3:0
        TCP connection: 10.4.4.4.14003 - 10.3.3.3.646
        State: Oper; Msgs sent/rcvd: 12/12; Downstream
        Up time: 00:00:13
        LDP discovery sources:
          FastEthernet1/0, Src IP addr: 10.34.0.4
        Addresses bound to peer LDP Ident:
          10.45.0.4       10.34.0.4       10.4.4.4       
R3-P2(config)#
R3-P2(config)#no ip route 10.2.2.2 255.255.255.255 10.34.0.4     // RESTORE ROUTING FOR 10.2.2.2 DYNAMICALLY VIA OSPF
R3-P2(config)#
*Nov 12 16:50:11.792: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (2) is UP
R3-P2(config)#
R3-P2(config)#do show mpls ldp neighbor   // MPLS ESTABLISH LDP NEIGHBORSHIP USING HIGHEST LOOPBACK IP
    Peer LDP Ident: 10.4.4.4:0; Local LDP Ident 10.3.3.3:0
        TCP connection: 10.4.4.4.14003 - 10.3.3.3.646
        State: Oper; Msgs sent/rcvd: 13/13; Downstream
        Up time: 00:01:00
        LDP discovery sources:
          FastEthernet1/0, Src IP addr: 10.34.0.4
        Addresses bound to peer LDP Ident:
          10.45.0.4       10.34.0.4       10.4.4.4       
    Peer LDP Ident: 10.2.2.2:0; Local LDP Ident 10.3.3.3:0
        TCP connection: 10.2.2.2.646 - 10.3.3.3.24330
        State: Oper; Msgs sent/rcvd: 12/12; Downstream
        Up time: 00:00:11
        LDP discovery sources:
          FastEthernet1/1, Src IP addr: 10.23.0.2
        Addresses bound to peer LDP Ident:
          10.12.0.2       10.2.2.2        10.23.0.2    


Verify CUSTA SITE 1 has network connectivity to 192.168.1.99 in CUSTA SITE 2. Troubleshoot backwards on the LSP: CUSTA-SITE2 > R5-PE2 > R4-P3 > R3-P2 > R2-P1 > R1-PE1 > CUSTA-SITE1.

CUSTA-SITE2#show ip interface brief
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            unassigned      YES unset  administratively down down   
FastEthernet1/0            192.168.1.2     YES manual up                    up     
FastEthernet1/1            unassigned      YES unset  administratively down down   
Loopback1                  192.168.1.99    YES manual up                    up     

R5-PE2#show bgp vpnv4 unicast vrf CUSTA label
   Network          Next Hop      In label/Out label
Route Distinguisher: 65000:1 (CUSTA)
   172.16.1.0/30    10.1.1.1        nolabel/107
   172.16.1.99/32   10.1.1.1        nolabel/108
   192.168.1.0/30   0.0.0.0         509/nolabel(CUSTA)
   192.168.1.99/32  192.168.1.2     510/nolabel

R5-PE2#show bgp vpnv4 unicast vrf CUSTA 192.168.1.99
BGP routing table entry for 65000:1:192.168.1.99/32, version 13
Paths: (1 available, best #1, table CUSTA)
  Advertised to update-groups:
     2        
  Refresh Epoch 1
  Local
    192.168.1.2 from 0.0.0.0 (10.5.5.5)
      Origin incomplete, metric 1, localpref 100, weight 32768, valid, sourced, best
      Extended Community: RT:1:1
      mpls labels in/out 510/nolabel

R5-PE2#show mpls forwarding-table vrf CUSTA 192.168.1.99
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
510        No Label   192.168.1.99/32[V]   \
                                       2370          Fa1/0      192.168.1.2

R5-PE2#show ip cef vrf CUSTA 192.168.1.99
192.168.1.99/32
  nexthop 192.168.1.2 FastEthernet1/0


R1-PE1#show bgp vpnv4 unicast vrf CUSTA label
   Network          Next Hop      In label/Out label
Route Distinguisher: 65000:1 (CUSTA)
   172.16.1.0/30    0.0.0.0         107/nolabel(CUSTA)
   172.16.1.99/32   172.16.1.2      108/nolabel
   192.168.1.0/30   10.5.5.5        nolabel/509
   192.168.1.99/32  10.5.5.5        nolabel/510

R1-PE1#show mpls forwarding-table 10.5.5.5
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
106        205        10.5.5.5/32      0             Fa0/0      10.12.0.2

R1-PE1#show ip cef vrf CUSTA 192.168.1.99
192.168.1.99/32
  nexthop 10.12.0.2 FastEthernet0/0 label 205 510      // TOP LABEL: 209; BOTTOM LABEL: 510



CUSTA-SITE2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
CUSTA-SITE2(config)#line vty 0 4          // ENABLE TELNET FOR TESTING
CUSTA-SITE2(config-line)#no login
CUSTA-SITE2(config-line)#privilege 15
CUSTA-SITE2(config-line)#privilege level 15
CUSTA-SITE2(config-line)#end
CUSTA-SITE2#


CUSTA-SITE1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/30 is directly connected, FastEthernet1/0
L        172.16.1.2/32 is directly connected, FastEthernet1/0
C        172.16.1.99/32 is directly connected, Loopback1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
O E2     192.168.1.0/30 [110/1] via 172.16.1.1, 1d16h, FastEthernet1/0
O E2     192.168.1.99/32 [110/1] via 172.16.1.1, 1d16h, FastEthernet1/0

CUSTA-SITE1#traceroute 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 172.16.1.1 16 msec 20 msec 16 msec      // R1-PE1
  2 10.12.0.2 [MPLS: Labels 205/510 Exp 0] 108 msec 72 msec 92 msec     // R2-P1
  3 10.23.0.3 [MPLS: Labels 309/510 Exp 0] 128 msec 72 msec 72 msec    // R3-P2
  4 10.34.0.4 [MPLS: Labels 405/510 Exp 0] 84 msec 72 msec 88 msec     // R4-P3 (PHP)
  5 192.168.1.1 [MPLS: Label 510 Exp 0] 72 msec 40 msec 64 msec      // R5-PE2; BOTTOM LABEL:510
  6 192.168.1.2 96 msec 80 msec 68 msec       // CUSTA-SITE2; NATIVE IP

CUSTA-SITE1#telnet 192.168.1.99
Trying 192.168.1.99 ... Open

CUSTA-SITE2#


Perform Wireshark packet capture between R2-P1 and R3-P2. Notice the Ethernet II (Layer 2/Data Link) has a Type: MPLS label switched packet (MPLS is Layer 2.5). Also the MPLS Label: 310 has a Bottom of Label Stack: 1

CUSTA-SITE1#ping 192.168.1.99      
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.99, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/90/104 ms



The ICMP reply packet is the reverse Label Switch Path (LSP) from CUSTA-SITE2 (192.168.1.99) to CUSTA-SITE1 (172.16.1.2).


There's a slight difference with the commands used in configuring and troubleshooting a VRF.

Configure the CE routers to allow Telnet for testing purposes.

CUSTA-SITE1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
CUSTA-SITE1(config)#line vty 0 4
CUSTA-SITE1(config-line)#no login
CUSTA-SITE1(config-line)#privilege level 15
CUSTA-SITE1(config-line)#end
CUSTA-SITE1#


CUSTB-SITE1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
CUSTB-SITE1(config)#line vty 0 4
CUSTB-SITE1(config-line)#no login
CUSTB-SITE1(config-line)#privilege level 15
CUSTB-SITE1(config-line)#end
CUSTB-SITE1#


CUSTB-SITE2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
CUSTB-SITE2(config)#line vty 0 4
CUSTB-SITE2(config-line)#no login
CUSTB-SITE2(config-line)#privilege level 15
CUSTB-SITE2(config-line)#end
CUSTB-SITE2#


R1-PE1#ping 172.16.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)


R1-PE1#ping 172.16.1.2      // PING FAILED SINCE IP IS NOT IN GLOBAL ROUTING TABLE
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R1-PE1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.1.1.1/32 is directly connected, Loopback0
O        10.2.2.2/32 [110/2] via 10.12.0.2, 1w3d, FastEthernet0/0
O        10.3.3.3/32 [110/3] via 10.12.0.2, 1w3d, FastEthernet0/0
O        10.4.4.4/32 [110/4] via 10.12.0.2, 1w3d, FastEthernet0/0
O        10.5.5.5/32 [110/5] via 10.12.0.2, 1w3d, FastEthernet0/0
C        10.12.0.0/24 is directly connected, FastEthernet0/0
L        10.12.0.1/32 is directly connected, FastEthernet0/0
O        10.23.0.0/24 [110/2] via 10.12.0.2, 1w3d, FastEthernet0/0
O        10.34.0.0/24 [110/3] via 10.12.0.2, 1w3d, FastEthernet0/0
O        10.45.0.0/24 [110/4] via 10.12.0.2, 1w3d, FastEthernet0/0

R1-PE1#show ip route ? 
  Hostname or A.B.C.D  Network to display information about or hostname
  bgp                  Border Gateway Protocol (BGP)
  connected            Connected
  dhcp                 Show routes added by DHCP Server or Relay
  eigrp                Enhanced Interior Gateway Routing Protocol (EIGRP)
  isis                 ISO IS-IS
  lisp                 Locator ID Separation Protocol (LISP)
  list                 IP Access list
  loops                RIB routes forming loops
  mobile               Mobile routes
  multicast            Multicast global information
  next-hop-override    Show next-hop-overrides too
  nhrp                 Next Hop Resolution Protocol (NHRP)
  odr                  On Demand stub Routes
  ospf                 Open Shortest Path First (OSPF)
  ospfv3               OSPFv3
  profile              IP routing table profile
  repair-paths         Show repair paths too
  rip                  Routing Information Protocol (RIP)
  static               Static routes
  summary              Summary of all routes
  supernets-only       Show supernet entries only
  tag                  Route Tag
  track-table          Tracked static table
  vrf                  Display routes from a VPN Routing/Forwarding instance
  |                    Output modifiers
  <cr>

R1-PE1#show ip route vrf CUSTA

Routing Table: CUSTA
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/30 is directly connected, FastEthernet1/0
L        172.16.1.1/32 is directly connected, FastEthernet1/0
O        172.16.1.99/32 [110/2] via 172.16.1.2, 2d19h, FastEthernet1/0
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
B        192.168.1.0/30 [200/0] via 10.5.5.5, 2d13h
B        192.168.1.99/32 [200/1] via 10.5.5.5, 2d13h


R1-PE1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R1-PE1(config)#ip route ?
  A.B.C.D  Destination prefix
  profile  Enable IP routing table profile
  static   Allow static routes
  vrf      Configure static route for a VPN Routing/Forwarding instance

R1-PE1(config)#ip route vrf CUSTA 10.1.1.0 255.255.255.0 172.16.1.2
R1-PE1(config)#end

R1-PE1#show ip route vrf CUSTA

Routing Table: CUSTA
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/24 is subnetted, 1 subnets
S        10.1.1.0 [1/0] via 172.16.1.2
      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/30 is directly connected, FastEthernet1/0
L        172.16.1.1/32 is directly connected, FastEthernet1/0
O        172.16.1.99/32 [110/2] via 172.16.1.2, 2d19h, FastEthernet1/0
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
B        192.168.1.0/30 [200/0] via 10.5.5.5, 2d13h
B        192.168.1.99/32 [200/1] via 10.5.5.5, 2d13h


R1-PE1#show ip route vrf CUSTB

Routing Table: CUSTB
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/30 is directly connected, FastEthernet1/1
L        172.16.1.1/32 is directly connected, FastEthernet1/1
D        172.16.1.100/32 [90/156160] via 172.16.1.2, 2d19h, FastEthernet1/1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
B        192.168.1.0/30 [200/0] via 10.5.5.5, 2d19h
B        192.168.1.100/32 [200/0] via 10.5.5.5, 2d19h


R1-PE1#ping vrf CUSTA 172.16.1.2       // CUSTA-SITE1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/16/24 ms

R1-PE1#ping vrf CUSTB 172.16.1.2     // CUSTB-SITE1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/12/24 ms

R1-PE1#ping vrf CUSTA 192.168.1.2      // CUSTA-SITE2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/91/132 ms

R1-PE1#ping vrf CUSTB 192.168.1.2      // CUSTB-SITE2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/64/84 ms


R1-PE1#traceroute ?             
  WORD       Trace route to destination address or hostname
  appletalk  AppleTalk Trace
  clns       ISO CLNS Trace
  ethernet   Ethernet Traceroute
  ip         IP Trace
  ipv6       IPv6 Trace
  ipx        IPX Trace
  mpls       MPLS LSP Trace
  vrf        Select VPN routing instance
  <cr>

R1-PE1#traceroute vrf CUSTA 172.16.1.99       // MPLS TRACEROUTE USES HIGH NUMBERED UDP PORTS
Type escape sequence to abort.
Tracing the route to 172.16.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 172.16.1.2 20 msec 12 msec 12 msec       // DIRECTLY CONNECTED (NO MPLS)

R1-PE1#traceroute vrf CUSTA 192.168.1.2
Type escape sequence to abort.
Tracing the route to 192.168.1.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.12.0.2 [MPLS: Labels 205/509 Exp 0] 44 msec 56 msec 44 msec      // SAME DESTINATION IP BUT DIFFERENT EGRESS MPLS LABEL; Exp IS THE EXPERIMENTAL BITS USED FOR QoS
  2 10.23.0.3 [MPLS: Labels 309/509 Exp 0] 64 msec 44 msec 40 msec
  3 10.34.0.4 [MPLS: Labels 405/509 Exp 0] 48 msec 56 msec 44 msec
  4 192.168.1.1 52 msec 36 msec 52 msec
  5 192.168.1.2 84 msec 48 msec 56 msec

R1-PE1#traceroute vrf CUSTB 192.168.1.2
Type escape sequence to abort.
Tracing the route to 192.168.1.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.12.0.2 [MPLS: Labels 205/507 Exp 0] 40 msec 60 msec 60 msec
  2 10.23.0.3 [MPLS: Labels 309/507 Exp 0] 48 msec 40 msec 44 msec
  3 10.34.0.4 [MPLS: Labels 405/507 Exp 0] 40 msec 52 msec 48 msec
  4 192.168.1.1 40 msec 48 msec 60 msec
  5 192.168.1.2 64 msec 72 msec 60 msec


R1-PE1#telnet 192.168.1.2 ?
  /debug             Enable telnet debugging mode
  /encrypt           Negotiate telnet encryption
  /ipv4              Force use of IP version 4
  /ipv6              Force use of IP version 6
  /line              Enable telnet line mode
  /noecho            Disable local echo
  /quiet             Suppress login/logout messages
  /route:            Enable telnet source route mode
  /source-interface  Specify source interface
  /stream            Enable stream processing
  /terminal-type     Set terminal type
  /vrf               Select VPN routing/forwarding instance
  <0-65535>          Port number
  bgp                Border Gateway Protocol (179)
  chargen            Character generator (19)
  cmd                Remote commands (rcmd, 514)
  daytime            Daytime (13)
  discard            Discard (9)
  domain             Domain Name Service (53)
  drip               Dynamic Routing Information Protocol (3949)
  echo               Echo (7)
  exec               Exec (rsh, 512)
  finger             Finger (79)
  ftp                File Transfer Protocol (21)
  ftp-data           FTP data connections (20)
  gopher             Gopher (70)
  hostname           NIC hostname server (101)
  ident              Ident Protocol (113)
  irc                Internet Relay Chat (194)
  klogin             Kerberos login (543)
  kshell             Kerberos shell (544)
  login              Login (rlogin, 513)
  lpd                Printer service (515)
  nntp               Network News Transport Protocol (119)
  pim-auto-rp        PIM Auto-RP (496)
  pop2               Post Office Protocol v2 (109)
  pop3               Post Office Protocol v3 (110)
  smtp               Simple Mail Transport Protocol (25)
  sunrpc             Sun Remote Procedure Call (111)
  tacacs             TAC Access Control System (49)
  talk               Talk (517)
  telnet             Telnet (23)
  time               Time (37)
  uucp               Unix-to-Unix Copy Program (540)
  whois              Nicname (43)
  www                World Wide Web (HTTP, 80)
  <cr>

R1-PE1#telnet 192.168.1.2 /vrf ?
  WORD  VPN Routing/forwarding instance name

R1-PE1#telnet 192.168.1.2 /vrf CUSTA
Trying 192.168.1.2 ... Open
CUSTA-SITE2#


R1-PE1#telnet 192.168.1.2 /vrf CUSTB
Trying 192.168.1.2 ... Open

CUSTB-SITE2#


CUSTA-SITE1#traceroute 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 172.16.1.1 48 msec 44 msec 16 msec
  2 10.12.0.2 [MPLS: Labels 205/510 Exp 0] 88 msec 72 msec 76 msec      // YOU CAN OPTIONALLY HIDE THE MPLS LABEL AND TTL IN A TRACEROUTE OUTPUT
  3 10.23.0.3 [MPLS: Labels 309/510 Exp 0] 72 msec 76 msec 48 msec
  4 10.34.0.4 [MPLS: Labels 405/510 Exp 0] 72 msec 72 msec 60 msec
  5 192.168.1.1 [MPLS: Label 510 Exp 0] 92 msec 60 msec 44 msec
  6 192.168.1.2 52 msec 64 msec 64 msec


R1-PE1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R1-PE1(config)#no mpls ?
  discovery    Configure LSP discovery
  export       Configure periodic export of MPLS PAL table
  ip           Dynamic MPLS forwarding for IP
  label        Label properties
  ldp          Label Distribution Protocol
  oam          OAM configuration
  static       MPLS static application
  traffic-eng  Configure Traffic Engineering parameters

R1-PE1(config)#no mpls ip ?
  default-route   Allow MPLS forwarding for ip default route
  propagate-ttl   Propagate IP TTL into the label stack
  ttl-expiration  Control MPLS TTL expiration behavior
  <cr>

R1-PE1(config)#no mpls ip propagate-ttl        // HIDES MPLS LABELS FOR CE TO CE ROUTER TRACEROUTE


R2-P1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R2-P1(config)#no mpls ip propagate-ttl


R3-P2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#no mpls ip propagate-ttl


R4-P3#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R4-P3(config)#no mpls ip propagate-ttl


R5-PE2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R5-PE2(config)#no mpls ip propagate-ttl


CUSTA-SITE1#traceroute 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 172.16.1.1 16 msec 16 msec 16 msec
  2 192.168.1.1 [MPLS: Label 510 Exp 0] 88 msec 80 msec 80 msec     // CISCO DEVICE USES UDP IN TRACEROUTE; HIDES P ROUTER/LABELS
  3 192.168.1.2 88 msec 100 msec 132 msec


R1-PE1#traceroute vrf CUSTA 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.1.1 [MPLS: Label 510 Exp 0] 140 msec 88 msec 52 msec        // SEES DESTINATION IP AS ONE HOP
  2 192.168.1.2 100 msec 96 msec 40 msec

R1-PE1#ping ?
  WORD      Ping destination address or hostname
  atm       ATM echo
  clns      CLNS echo
  ethernet  Ethernet echo
  ip        IP echo
  ipv6      IPv6 echo
  mpls      MPLS echo
  srb       srb echo
  tag       Tag encapsulated IP echo
  vrf       Select VPN routing instance
  <cr>

R1-PE1#ping mpls ?
  ipv4         Target specified as an IPv4 address
  pseudowire   Target VC specified as an IPv4 address and VC ID
  traffic-eng  Target specified as TE tunnel interface
  <cr>

R1-PE1#ping mpls ipv4 ?
  A.B.C.D  {/nn || A.B.C.D}  Target FEC address with mask

R1-PE1#ping mpls ipv4 10.5.5.5/32
Sending 5, 100-byte MPLS Echos to 10.5.5.5/32,
     timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/49/60 ms

R1-PE1#ping mpls ipv4 10.5.5.5/32 ?     
  destination          Destination address or address range
  dsmap                Request dsmap from replying router
  exp                  EXP bits in mpls header
  flags                Flag options
  force-explicit-null  force an explicit null label to be added
  interval             Send interval between requests in msec
  output               Output options
  pad                  Pad TLV pattern
  repeat               Repeat count
  reply                Reply mode
  revision             Echo Packet TLV versioning
  size                 Packet size
  source               Source specified as an IP address
  sweep                Sweep range of sizes
  timeout              Timeout in seconds
  ttl                  Time to live
  verbose              verbose output mode
  <cr>

R1-PE1#ping mpls ipv4 10.5.5.5/32 verbose
Sending 5, 100-byte MPLS Echos to 10.5.5.5/32,
     timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
!    size 100, reply addr 10.45.0.5, return code 3      // ! INDICATES MPLS PING OK;  MPLS PING USES UDP PORT 3503; RETURN CODE 3 INDICATES THE EGRESS LSR (R5-PE2) IN LSP RESPONDED AN ECHO REPLY
!    size 100, reply addr 10.45.0.5, return code 3
!    size 100, reply addr 10.45.0.5, return code 3
!    size 100, reply addr 10.45.0.5, return code 3
!    size 100, reply addr 10.45.0.5, return code 3

Success rate is 100 percent (5/5), round-trip min/avg/max = 40/48/56 ms

R1-PE1#traceroute ?
  WORD       Trace route to destination address or hostname
  appletalk  AppleTalk Trace
  clns       ISO CLNS Trace
  ethernet   Ethernet Traceroute
  ip         IP Trace
  ipv6       IPv6 Trace
  ipx        IPX Trace
  mpls       MPLS LSP Trace
  vrf        Select VPN routing instance
  <cr>

R1-PE1#traceroute mpls ?
  ipv4         Target specified as an IPv4 address
  multipath    LSP Multipath Traceroute
  traffic-eng  Target specified as TE tunnel interface
  <cr>

R1-PE1#traceroute mpls ipv4 ?
  A.B.C.D  {/nn || A.B.C.D}  Target FEC address with mask

R1-PE1#traceroute mpls ipv4 10.5.5.5/32      // TO VERIFY MPLS LSP OR IF USING NATIVE IP
Tracing MPLS Label Switched Path to 10.5.5.5/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 10.12.0.1 MRU 1500 [Labels: 205 Exp: 0]
L 1 10.12.0.2 MRU 1500 [Labels: 309 Exp: 0] 40 ms
L 2 10.23.0.3 MRU 1500 [Labels: 405 Exp: 0] 48 ms
L 3 10.34.0.4 MRU 1504 [Labels: implicit-null Exp: 0] 52 ms      // PHP
! 4 10.45.0.5 44 ms        // NATIVE IPv4

R1-PE1#traceroute mpls ipv4 10.5.5.5/32 ?      
  destination          Destination address or address range
  exp                  EXP bits in mpls header
  flags                Flag options
  force-explicit-null  force an explicit null label to be added
  output               Output options
  reply                Reply mode
  revision             Echo Packet TLV versioning
  source               Source specified as an IP address
  timeout              Timeout in seconds
  ttl                  Maximum time to live
  verbose              verbose output mode
  <cr>

R1-PE1#traceroute mpls ipv4 10.5.5.5/32 verbose
Tracing MPLS Label Switched Path to 10.5.5.5/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 10.12.0.1 10.12.0.2 MRU 1500 [Labels: 205 Exp: 0]       // MAXIMUM RECEIVE UNIT (MRU)
L 1 10.12.0.2 10.23.0.3 MRU 1500 [Labels: 303 Exp: 0] 60 ms, ret code 8       // L INDICATES DOWNSTREAM ROUTER SENDS AN ECHO REPLY; RETURN CODE 8 INDICATES THE PACKET WAS CORRECTLY LABELED BUT IT'S NOT THE TARGET EGRESS LSR
L 2 10.23.0.3 10.34.0.4 MRU 1500 [Labels: 415 Exp: 0] 56 ms, ret code 8
L 3 10.34.0.4 10.45.0.5 MRU 1504 [Labels: implicit-null Exp: 0] 52 ms, ret code 8
! 4 10.45.0.5 60 ms, ret code 3      //   RETURN CODE 3 INDICATES EGRESS LSR IN LSP


R3-P2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#no mpls ip
R3-P2(config)#
*Dec 11 09:39:12.541: %LDP-5-NBRCHG: LDP Neighbor 10.2.2.2:0 (3) is DOWN (LDP unconfigured globally)
*Dec 11 09:39:12.545: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (4) is DOWN (LDP unconfigured globally)
R3-P2(config)#
*Dec 11 09:39:16.757: %MPLS_PACKET-4-NOLFDSB: MPLS packet received on non MPLS enabled interface FastEthernet1/1 L3 type 0x8847 label {303 0 1 1}


R2-P1#
*Dec 11 09:39:11.097: %LDP-5-NBRCHG: LDP Neighbor 10.3.3.3:0 (1) is DOWN (TCP connection closed by peer)


R4-P3(config)#
*Dec 11 09:39:20.181: %LDP-5-NBRCHG: LDP Neighbor 10.3.3.3:0 (2) is DOWN (TCP connection closed by peer)

  
R1-PE1#traceroute mpls ipv4 10.5.5.5/32
Tracing MPLS Label Switched Path to 10.5.5.5/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 10.12.0.1 MRU 1500 [Labels: 205 Exp: 0]
B 1 10.12.0.2 MRU 1504 [No Label] 12 ms      // ISSUE OR A BREAK IN LSP
. 2 *
B 3 10.12.0.2 MRU 1504 [No Label] 28 ms
B 4 10.12.0.2 MRU 1504 [No Label] 36 ms
B 5 10.12.0.2 MRU 1504 [No Label] 40 ms
B 6 10.12.0.2 MRU 1504 [No Label] 24 ms
B 7 10.12.0.2 MRU 1504 [No Label] 44 ms
B 8 10.12.0.2 MRU 1504 [No Label] 20 ms
B 9 10.12.0.2 MRU 1504 [No Label] 40 ms
B 10 10.12.0.2 MRU 1504 [No Label] 44 ms
B 11 10.12.0.2 MRU 1504 [No Label] 40 ms
B 12 10.12.0.2 MRU 1504 [No Label] 40 ms
B 13 10.12.0.2 MRU 1504 [No Label] 36 ms
B 14 10.12.0.2 MRU 1504 [No Label] 40 ms
B 15 10.12.0.2 MRU 1504 [No Label] 40 ms
B 16 10.12.0.2 MRU 1504 [No Label] 40 ms
B 17 10.12.0.2 MRU 1504 [No Label] 36 ms
B 18 10.12.0.2 MRU 1504 [No Label] 44 ms
B 19 10.12.0.2 MRU 1504 [No Label] 40 ms
B 20 10.12.0.2 MRU 1504 [No Label] 36 ms
B 21 10.12.0.2 MRU 1504 [No Label] 40 ms
B 22 10.12.0.2 MRU 1504 [No Label] 36 ms
B 23 10.12.0.2 MRU 1504 [No Label] 40 ms
B 24 10.12.0.2 MRU 1504 [No Label] 40 ms
B 25 10.12.0.2 MRU 1504 [No Label] 36 ms
B 26 10.12.0.2 MRU 1504 [No Label] 72 ms
B 27 10.12.0.2 MRU 1504 [No Label] 56 ms
B 28 10.12.0.2 MRU 1504 [No Label] 40 ms
B 29 10.12.0.2 MRU 1504 [No Label] 40 ms
B 30 10.12.0.2 MRU 1504 [No Label] 40 ms


R1-PE1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R1-PE1(config)#no mpls ip propagate-ttl ?
  forwarded  Propagate IP TTL for forwarded traffic
  local      Propagate IP TTL for locally originated traffic
  <cr>

R1-PE1(config)#no mpls ip propagate-ttl forwarded        // HIDES MPLS LABELS FOR CE TO CE ROUTER BUT NOT FOR MPLS VPN NETWORK: PE <> P <> PE


R2-P1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R2-P1(config)#no mpls ip propagate-ttl forwarded


R3-P2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R3-P2(config)#no mpls ip propagate-ttl forwarded


R4-P3#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R4-P3(config)#no mpls ip propagate-ttl forwarded


R5-PE2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R5-PE2(config)#no mpls ip propagate-ttl forwarded


CUSTA-SITE1#traceroute 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 172.16.1.1 52 msec 100 msec 24 msec
  2 192.168.1.1 [MPLS: Label 510 Exp 0] 92 msec 96 msec 44 msec
  3 192.168.1.2 92 msec 36 msec 116 msec


R1-PE1#traceroute vrf CUSTA 192.168.1.99
Type escape sequence to abort.
Tracing the route to 192.168.1.99
VRF info: (vrf in name/id, vrf out name/id)
  1 10.12.0.2 [MPLS: Labels 205/510 Exp 0] 116 msec 68 msec 68 msec
  2 10.23.0.3 [MPLS: Labels 309/510 Exp 0] 64 msec 60 msec 60 msec
  3 10.34.0.4 [MPLS: Labels 405/510 Exp 0] 76 msec 64 msec 40 msec
  4 192.168.1.2 84 msec 64 msec 44 msec


R1-PE1#show mpls forwarding-table      // LFIB ACCOUNTING PER LSP
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
100        Pop Label  10.2.2.2/32      0             Fa0/0      10.12.0.2  
101        Pop Label  10.23.0.0/24     0             Fa0/0      10.12.0.2  
102        201        10.3.3.3/32      0             Fa0/0      10.12.0.2  
103        202        10.34.0.0/24     0             Fa0/0      10.12.0.2  
104        203        10.4.4.4/32      0             Fa0/0      10.12.0.2  
105        204        10.45.0.0/24     0             Fa0/0      10.12.0.2  
106        205        10.5.5.5/32      0             Fa0/0      10.12.0.2  
107        No Label   172.16.1.0/30[V] 38614         aggregate/CUSTA
108        No Label   172.16.1.99/32[V]   \
                                       0             Fa1/0      172.16.1.2 
109        No Label   172.16.1.0/30[V] 15940         aggregate/CUSTB
110        No Label   172.16.1.100/32[V]   \
                                       0             Fa1/1      172.16.1.2 


R5-PE2#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
R5-PE2(config)#access-list ?
  <1-99>            IP standard access list
  <100-199>         IP extended access list
  <1100-1199>       Extended 48-bit MAC address access list
  <1300-1999>       IP standard access list (expanded range)
  <200-299>         Protocol type-code access list
  <2000-2699>       IP extended access list (expanded range)
  <2700-2799>       MPLS access list
  <700-799>         48-bit MAC address access list
  compiled          Enable IP access-list compilation
  dynamic-extended  Extend the dynamic ACL absolute timer
  rate-limit        Simple rate-limit specific access list

R5-PE2(config)#access-list 2701 ?
  deny    Specify packets to reject
  permit  Specify packets to forward
  remark  access list entry comment

R5-PE2(config)#access-list 2701 permit ?
  WORD  mpls label table name or 'any'

R5-PE2(config)#access-list 2701 permit any ?
  <0-1048575>  mpls label number
  any          any label number
  <cr>

R5-PE2(config)#access-list 2701 permit any any ?
  <0-7>  mpls exp value
  any    any exp value
  <cr>

R5-PE2(config)#access-list 2701 permit any any any ?
  <0-1>  mpls eos value
  any    any eos value
  <cr>

R5-PE2(config)#access-list 2701 permit any any any any ?
  <cr>

R5-PE2(config)#access-list 2701 permit any any any any
R5-PE2(config)#end
R5-PE2#
*Nov 30 23:03:16.043: %SYS-5-CONFIG_I: Configured from console by console
R5-PE2#debug mpls ?
  access-list  MPLS ACCESS
  events       MPLS events
  ip           Debug MPLS IP information
  l2transport  Any transport over MPLS debug
  ldp          Label Distribution Protocol
  lspv         LSP Verification debug
  netflow      MPLS Egress NetFlow Accounting
  packet       mpls packet debug
  static       MPLS Static labels
  traffic-eng  MPLS Traffic Engineering debug

R5-PE2#debug mpls packet ?
  <2700-2799>        MPLS label access list
  Async              Async interface
  Auto-Template      Auto-Template interface
  BVI                Bridge-Group Virtual Interface
  CDMA-Ix            CDMA Ix interface
  CTunnel            CTunnel interface
  Dialer             Dialer interface
  FastEthernet       FastEthernet IEEE 802.3
  GMPLS              MPLS interface
  LISP               Locator/ID Separation Protocol Virtual Interface
  LongReachEthernet  Long-Reach Ethernet interface
  Loopback           Loopback interface
  MFR                Multilink Frame Relay bundle interface
  Multilink          Multilink-group interface
  Null               Null interface
  Port-channel       Ethernet Channel of interfaces
  Tunnel             Tunnel interface
  Vif                PGM Multicast Host interface
  Virtual-PPP        Virtual PPP interface
  Virtual-Template   Virtual Template interface
  Virtual-TokenRing  Virtual TokenRing
  error              error conditions
  exception          exception conditions
  process            process packet information
  rate-limit         rate limit mpls packet debugs
  vmi                Virtual Multipoint Interface
  <cr>

R5-PE2#debug mpls packet 2701     // CAUTION IS NEEDED WHEN DEBUGGING MPLS; LIMIT INFO WITH ACL


R5-PE2#debug mpls events
MPLS events debugging is on
R5-PE2#
R5-PE2#debug mpls packet
Packet debugging is on


R4-P3#configure terminal              
Enter configuration commands, one per line.  End with CNTL/Z.
R4-P3(config)#no mpls ip
R4-P3(config)#
*Nov 30 23:08:13.911: %LDP-5-NBRCHG: LDP Neighbor 10.3.3.3:0 (1) is DOWN (LDP unconfigured globally)
*Nov 30 23:08:13.915: %LDP-5-NBRCHG: LDP Neighbor 10.5.5.5:0 (2) is DOWN (LDP unconfigured globally)
R4-P3(config)#
*Nov 30 23:08:14.715: %MPLS_PACKET-4-NOLFDSB: MPLS packet received on non MPLS enabled interface FastEthernet1/0 L3 type 0x8847 label {409 6 1 253}
R4-P3(config)#
R4-P3(config)#mpls ip  
R4-P3(config)#
*Nov 30 23:08:31.311: %LDP-5-NBRCHG: LDP Neighbor 10.5.5.5:0 (1) is UP
*Nov 30 23:08:31.379: %LDP-5-NBRCHG: LDP Neighbor 10.3.3.3:0 (2) is UP


R5-PE2#
*Nov 30 23:08:14.275: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (1) is DOWN (TCP connection closed by peer)
R5-PE2#
R5-PE2#
*Nov 30 23:08:31.515: %LDP-5-NBRCHG: LDP Neighbor 10.4.4.4:0 (1) is UP
R5-PE2#
*Nov 30 23:08:41.611: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 253} - ipv4 data s:10.12.0.2 d:172.16.1.1 ttl:255 tos:C0 prot:1       // {LABEL EXP TTL}
*Nov 30 23:08:41.707: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 253} - ipv4 data s:10.12.0.2 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:41.763: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 253} - ipv4 data s:10.12.0.2 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:41.815: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 254} - ipv4 data s:10.23.0.3 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:41.883: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 254} - ipv4 data s:10.23.0.3 d:172.16.1.1 ttl:255 tos:C0 prot:1
R5-PE2#
*Nov 30 23:08:41.931: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 254} - ipv4 data s:10.23.0.3 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:41.995: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 255} - ipv4 data s:10.34.0.4 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:42.067: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 255} - ipv4 data s:10.34.0.4 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:42.115: MPLS turbo: Fa0/0: rx: Len 190 Stack {510 6 255} - ipv4 data s:10.34.0.4 d:172.16.1.1 ttl:255 tos:C0 prot:1
*Nov 30 23:08:42.175: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 1} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:4 tos:0 prot:11
*Nov 30 23:08:42.175: FIBfwd-proc: packet routed to FastEthernet1/0 192.168.1.2(1)
*Nov 30 23:08:42.179: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet1/0 nh 192.168.1.2 uhp 1 deag 0 ttlexp 1
*Nov 30 23:08:42.179: FIBfwd-proc: CUSTA:172.16.1.0/30 process level forwarding
*Nov 30 23:08:42.179: FIBfwd-proc: depth 0 first_idx 0
R5-PE2# paths 1 long 0(0)
*Nov 30 23:08:42.179: FIBfwd-proc: try path 0 (of 1) v4-rcrsv-10.1.1.1 first short ext 6AA1435C(0)
*Nov 30 23:08:42.179: FIBfwd-proc: v4-rcrsv-10.1.1.1 valid short mbl
*Nov 30 23:08:42.179: FIBfwd-proc: label[0] 107 connid 0 link ILLEGAL
*Nov 30 23:08:42.179: FIBfwd-proc: ip_pak_table 1 ip_nh_table 0 if none nh 10.1.1.1 deag 0 chg_if 0 via fib 6A95F444 path type recursive
*Nov 30 23:08:42.179: FIBfwd-proc: depth 1 first_idx 0 paths 1 long 0(0)
*Nov 30 23:08:42.179: FIBfwd-proc: try path 0 (of 1) v4-anh-10.45.0.4-Fa0/0 first short ext 67A72DBC(0)
*Nov 30 23:08:42.179: FIBfwd-proc: v4-anh-10.45.0.4-Fa0/0 valid short
*Nov 30 23:08:42.179: FIBfwd-proc: label[1] 412 connid 0 link TAG
*Nov 30 23:08:42.183: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 deag 0 chg_if 0 via fib 0 path type attached nexthop
*Nov 30 23:08:42.187: FIBfwd-proc: packet routed to FastEthernet0/0 10.45.0.4(1) with labels 107 412
*Nov 30 23:08:42.191: FIBip
R5-PE2#v4-packet-proc: packet routing succeeded
*Nov 30 23:08:42.191: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 ttlexp 0
*Nov 30 23:08:42.195: FIBfwd-proc: sending link TAG ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 chgif 0 ttlexp 0 rec 0
*Nov 30 23:08:42.239: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 1} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:4 tos:0 prot:11
*Nov 30 23:08:42.243: FIBfwd-proc: packet routed to FastEthernet1/0 192.168.1.2(1)
*Nov 30 23:08:42.243: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet1/0 nh 192.168.1.2 uhp 1 deag 0 ttlexp 1
*Nov 30 23:08:42.243: FIBfwd-proc: CUSTA:172.16.1.0/30 process level forwarding
*Nov 30 23:08:42.243: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)
*Nov 30 23:08:42.243: FIBfwd-proc: try path 0 (of 1) v4-rcrsv-10.1.1.1 first short ext 6AA1435C(0)
*Nov 30 23:08:42.243: FIBfwd-proc: v4-rcrsv-10.1.1.1 valid short mbl
*Nov 30 23:08:42.
R5-PE2#243: FIBfwd-proc: label[0] 107 connid 0 link ILLEGAL
*Nov 30 23:08:42.243: FIBfwd-proc: ip_pak_table 1 ip_nh_table 0 if none nh 10.1.1.1 deag 0 chg_if 0 via fib 6A95F444 path type recursive
*Nov 30 23:08:42.243: FIBfwd-proc: depth 1 first_idx 0 paths 1 long 0(0)
*Nov 30 23:08:42.243: FIBfwd-proc: try path 0 (of 1) v4-anh-10.45.0.4-Fa0/0 first short ext 67A72DBC(0)
*Nov 30 23:08:42.243: FIBfwd-proc: v4-anh-10.45.0.4-Fa0/0 valid short
*Nov 30 23:08:42.243: FIBfwd-proc: label[1] 412 connid 0 link TAG
*Nov 30 23:08:42.243: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 deag 0 chg_if 0 via fib 0 path type attached nexthop
*Nov 30 23:08:42.243: FIBfwd-proc: packet routed to FastEthernet0/0 10.45.0.4(1) with labels 107 412
*Nov 30 23:08:42.243: FIBipv4-packet-proc: packet routing succeeded
*Nov 30 23:08:42.243: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 ttlexp 0
*Nov 30 23:08:42.243: FIBfwd-proc: sending
R5-PE2# link TAG ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 chgif 0 ttlexp 0 rec 0
*Nov 30 23:08:42.283: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 1} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:4 tos:0 prot:11
*Nov 30 23:08:42.283: FIBfwd-proc: packet routed to FastEthernet1/0 192.168.1.2(1)
*Nov 30 23:08:42.283: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet1/0 nh 192.168.1.2 uhp 1 deag 0 ttlexp 1
*Nov 30 23:08:42.283: FIBfwd-proc: CUSTA:172.16.1.0/30 process level forwarding
*Nov 30 23:08:42.283: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)
*Nov 30 23:08:42.283: FIBfwd-proc: try path 0 (of 1) v4-rcrsv-10.1.1.1 first short ext 6AA1435C(0)
*Nov 30 23:08:42.283: FIBfwd-proc: v4-rcrsv-10.1.1.1 valid short mbl
*Nov 30 23:08:42.283: FIBfwd-proc: label[0] 107 connid 0 link ILLEGAL
*Nov 30 23:08:42.283: FIBfwd-proc: ip_pak_table 1 ip_nh_table 0 if none nh 10.1.1.1 deag 0 chg_if 0 via fib 6A95F444 path type recursive
*Nov 30 23:08:42.283: FIBfwd-proc: depth 1 first_idx 0 paths 1 long 0(0)
*Nov 30 23:08:42.283: FIBfwd-proc: try path 0 (of 1) v4-anh-10.45.0.4-Fa0/0 first short ext 67A72DBC(0)
*No
R5-PE2#v 30 23:08:42.283: FIBfwd-proc: v4-anh-10.45.0.4-Fa0/0 valid short
*Nov 30 23:08:42.283: FIBfwd-proc: label[1] 412 connid 0 link TAG
*Nov 30 23:08:42.283: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 deag 0 chg_if 0 via fib 0 path type attached nexthop
*Nov 30 23:08:42.283: FIBfwd-proc: packet routed to FastEthernet0/0 10.45.0.4(1) with labels 107 412
*Nov 30 23:08:42.283: FIBipv4-packet-proc: packet routing succeeded
*Nov 30 23:08:42.283: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 ttlexp 0
*Nov 30 23:08:42.283: FIBfwd-proc: sending link TAG ip_pak_table 1 ip_nh_table 65535 if FastEthernet0/0 nh 10.45.0.4 uhp 0 deag 0 chgif 0 ttlexp 0 rec 0
*Nov 30 23:08:42.323: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 2} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:5 tos:0 prot:11
*Nov 30 23:08:42.367: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 2} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:5 tos:0 prot:11
*Nov
R5-PE2# 30 23:08:42.419: MPLS turbo: Fa0/0: rx: Len 56 Stack {510 0 2} - ipv4 data s:172.16.1.1 d:192.168.1.99 ttl:5 tos:0 prot:11
*Nov 30 23:08:44.587: tagcon: background scan: Start
*Nov 30 23:08:44.587: tagcon: background scan: End
R5-PE2#
R5-PE2#undebug all
All possible debugging has been turned off