Saturday, August 27, 2022

Cisco DevNet Sandbox Free Virtual Lab

The Cisco DevNet Sandbox is a free online tool where you can test and develop various Cisco technologies, i.e. Cloud, Security, SD WAN, etc. without owning or setup any Cisco hardware or virtual server. You just register or use your existing CCO login.

There are a lot of Sandbox virtual labs to choose from. This is useful if you need to test a new IOS-XE or NX-OS feature without blowing up your production network. You can also do a Proof of Concept (POC) or Proof of Value (POV) without having to ask from your Cisco Account Manager for a loan hardware. It's also a great learning tool for those studying their Cisco certifications, i.e. CCNA, CCNP, etc.


Click a specific Sandbox Lab. In this case I chose the Cisco Modeling Labs (CML) Enterprise > click Reserve.

The Sandbox lab setup takes around 10 minutes to complete. You can check the time left on the Setup green icon on the top right corner. You'll receive an email about the Sandbox lab chosen and a download link for the Cisco AnyConnect VPN client software installer. I used Cisco AnyConnect version 4.x in my laptop.

Once the Setup is finished, the Sandbox Setup changed to Active (top right) and nodes will have a green dot. You'll also receive another email informing the Sandbox lab is ready and gives you a temporary username/password login.


Copy/paste the AnyConnect VPN address/URL > click Connect > type the username and password provided in the email.

 

Open an SSH client > select the Service: Telnet > type the Sandbox component IP address > click OK. I used TeraTerm in this case.


There's also a PDF network diagram you can view or download in the Instructions section.

dist-rtr01#show version

Cisco IOS XE Software, Version 17.03.02

Cisco IOS Software [Amsterdam], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.3.2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Sat 31-Oct-20 13:16 by mcpre

 

 

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.

All rights reserved.  Certain components of Cisco IOS-XE software are

licensed under the GNU General Public License ("GPL") Version 2.0.  The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0.  For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

 

 

ROM: IOS-XE ROMMON

 

dist-rtr01 uptime is 5 minutes

Uptime for this control processor is 7 minutes

System returned to ROM by reload

System image file is "bootflash:packages.conf"

Last reload reason: factory-reset

 

 

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

 

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

 

If you require further assistance please contact us by sending email to

export@cisco.com.

 

License Level: ax

License Type: N/A(Smart License Enabled)

Next reload license Level: ax

 

The current throughput level is 1000 kbps

 

 

Smart Licensing Status: UNREGISTERED/No Licenses in Use

 

cisco CSR1000V (VXE) processor (revision VXE) with 1105351K/3075K bytes of memory.

Processor board ID 9LFWOLT44MQ

Router operating mode: Autonomous

6 Gigabit Ethernet interfaces

32768K bytes of non-volatile configuration memory.

3012036K bytes of physical memory.

6188032K bytes of virtual hard disk at bootflash:.

 

Configuration register is 0x2102

 

 

dist-rtr01#show ip interface brief

Interface              IP-Address      OK? Method Status                Protocol

GigabitEthernet1       10.10.20.175    YES TFTP   up                    up

GigabitEthernet2       172.16.252.21   YES TFTP   up                    up

GigabitEthernet3       172.16.252.25   YES TFTP   up                    up

GigabitEthernet4       172.16.252.2    YES TFTP   up                    up

GigabitEthernet5       172.16.252.10   YES TFTP   up                    up

GigabitEthernet6       172.16.252.17   YES TFTP   up                    up

Loopback0              unassigned      YES unset  administratively down down

 

 

dist-rtr01#show ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP

       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       H - NHRP, G - NHRP registered, g - NHRP registration summary

       o - ODR, P - periodic downloaded static route, l - LISP

       a - application route

       + - replicated route, % - next hop override, p - overrides from PfR

       & - replicated local route overrides by connected

 

Gateway of last resort is not set

 

      172.16.0.0/16 is variably subnetted, 19 subnets, 3 masks

O        172.16.101.0/24 [110/41] via 172.16.252.9, 00:07:16, GigabitEthernet5

                         [110/41] via 172.16.252.1, 00:09:37, GigabitEthernet4

O        172.16.102.0/24 [110/41] via 172.16.252.9, 00:09:26, GigabitEthernet5

                         [110/41] via 172.16.252.1, 00:07:16, GigabitEthernet4

O        172.16.103.0/24 [110/41] via 172.16.252.9, 00:07:11, GigabitEthernet5

                         [110/41] via 172.16.252.1, 00:07:16, GigabitEthernet4

O        172.16.104.0/24 [110/41] via 172.16.252.9, 00:07:11, GigabitEthernet5

                         [110/41] via 172.16.252.1, 00:07:16, GigabitEthernet4

O        172.16.105.0/24 [110/41] via 172.16.252.9, 00:07:11, GigabitEthernet5

                         [110/41] via 172.16.252.1, 00:07:16, GigabitEthernet4

C        172.16.252.0/30 is directly connected, GigabitEthernet4

L        172.16.252.2/32 is directly connected, GigabitEthernet4

O        172.16.252.4/30 [110/2] via 172.16.252.18, 00:11:04, GigabitEthernet6

C        172.16.252.8/30 is directly connected, GigabitEthernet5

L        172.16.252.10/32 is directly connected, GigabitEthernet5

O        172.16.252.12/30

           [110/2] via 172.16.252.18, 00:11:04, GigabitEthernet6

C        172.16.252.16/30 is directly connected, GigabitEthernet6

L        172.16.252.17/32 is directly connected, GigabitEthernet6

C        172.16.252.20/30 is directly connected, GigabitEthernet2

L        172.16.252.21/32 is directly connected, GigabitEthernet2

C        172.16.252.24/30 is directly connected, GigabitEthernet3

L        172.16.252.25/32 is directly connected, GigabitEthernet3

O        172.16.252.28/30

           [110/2] via 172.16.252.18, 00:11:04, GigabitEthernet6

O        172.16.252.32/30

           [110/2] via 172.16.252.18, 00:11:04, GigabitEthernet6

 

Once you're finished, click End (stop hand icon on top right).

To clear the cached VPN address, go to Cisco AnyConnect Security Mobility Client folder:

C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client


Select preferences_global (an XML file) > delete.

You can verify the content of the XML file by opening it in Notepad. Notice it has the Cisco Sandbox URL and public IP address.


Saturday, August 6, 2022

Cisco Nexus 5000 Fabric Extender (FEX) Configuration

After configuring a Virtual PortChannel (vPC) between two parent Cisco Nexus 5000 switches in my previous post, the next step is to configure a Nexus 2000 Fabric Extender (FEX) modules. These are remote line cards which uplinks to a parent Cisco Nexus 5K, 6K, 7K or 9K switches.

N5K-1# configure terminal

N5K-1(config)# feature fex   // ENABLE FEX

N5K-1(config)#

N5K-1(config)# interface Eth1/7

N5K-1(config-if)#  description FEX101 -  RACK 1

N5K-1(config-if)#  switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)# channel-group 101

 

N5K-1(config-if)# interface port-channel101

N5K-1(config-if)#  description FEX101 -  RACK 1

N5K-1(config-if)#  switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)#  vpc 101

  


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------               Offline   N2K-C2248TP-E-1GE   FOX2520PABC   // STATE WILL CHANGE FROM DISCOVERED TO OFFLINE

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

  


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  SERVER_RACK_1        Image Download   N2K-C2248TP-E-1GE   FOX2520PABC   // FEX 2K WILL DOWNLOAD IMAGE FROM PARENT N5K SWITCH

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

 

N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------               Offline   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)#

N5K-1(config-if)# 2005 Apr 14 19:42:10 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101 Module 1: Cold boot


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------           Online Sequence   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

2005 Apr 14 19:42:26 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

 


N5K-1config-if)# show fex   // IT TOOK AROUND 10 MINS FOR FEX 2K TO COMPLETE ITS UPGRADE

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  FEX0101                Online   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:44 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

 

N5K-1(config-if)# show fex detail

FEX: 101 Description: FEX0101   state: Online

  FEX version: 7.3(8)N1(1) [Switch version: 7.3(8)N1(1)]    // DOWNLOADED NX-OS FROM PARENT N5K SWITCH

  FEX Interim version: 7.3(8)N1(1)

  Switch Interim version: 7.3(8)N1(1)

  Extender Serial: FOX2520ABC

  Extender Model: N2K-C2248TP-E-1GE,  Part No: 73-12345-67

  Card Id: 149, Mac Addr: 28:af:fd:19:12:34, Num Macs: 64

  Module Sw Gen: 12594  [Switch Sw Gen: 21]

  Post level: complete

  Pinning-mode: static    Max-links: 1

  Fabric port for control traffic: Eth1/7

  FCoE Admin: false

  FCoE Oper: true

  FCoE FEX AA Configured: false

  Fabric interface state:

    Po101 - Interface Up. State: Active

    Eth1/7 - Interface Up. State: Active

  Fex Port        State  Fabric Port

       Eth101/1/1  Down       Po101

       Eth101/1/2  Down       Po101

       Eth101/1/3  Down       Po101

       Eth101/1/4  Down       Po101

       Eth101/1/5  Down       Po101

       Eth101/1/6  Down       Po101

       Eth101/1/7  Down       Po101

       Eth101/1/8  Down       Po101

       Eth101/1/9  Down       Po101

      Eth101/1/10  Down       Po101

      Eth101/1/11  Down       Po101

      Eth101/1/12  Down       Po101

      Eth101/1/13  Down       Po101

      Eth101/1/14  Down       Po101

      Eth101/1/15  Down       Po101

      Eth101/1/16  Down       Po101

      Eth101/1/17  Down       Po101

      Eth101/1/18  Down       Po101

      Eth101/1/19  Down       Po101

      Eth101/1/20  Down       Po101

      Eth101/1/21  Down       Po101

      Eth101/1/22  Down       Po101

      Eth101/1/23  Down       Po101

      Eth101/1/24  Down       Po101

      Eth101/1/25  Down       Po101

      Eth101/1/26  Down       Po101

      Eth101/1/27  Down       Po101

      Eth101/1/28  Down       Po101

      Eth101/1/29  Down       Po101

      Eth101/1/30  Down       Po101

      Eth101/1/31  Down       Po101

      Eth101/1/32  Down       Po101

      Eth101/1/33  Down       Po101

      Eth101/1/34  Down       Po101

      Eth101/1/35  Down       Po101

      Eth101/1/36  Down       Po101

      Eth101/1/37  Down       Po101

      Eth101/1/38  Down       Po101

      Eth101/1/39  Down       Po101

      Eth101/1/40  Down       Po101

      Eth101/1/41  Down       Po101

      Eth101/1/42  Down       Po101

      Eth101/1/43  Down       Po101

      Eth101/1/44  Down       Po101

      Eth101/1/45  Down       Po101

      Eth101/1/46  Down       Po101

      Eth101/1/47  Down       Po101

      Eth101/1/48  Down       Po101

Logs:

04/14/2005 19:32:34.118822: Module register received

04/14/2005 19:32:34.120820: Image Version Mismatch

04/14/2005 19:32:34.122113: Registration response sent

04/14/2005 19:32:34.122404: Requesting satellite to download image

04/14/2005 19:32:34.968158: Deleting route to FEX

04/14/2005 19:32:34.975847: Module disconnected

04/14/2005 19:32:34.977323: Module Offline

04/14/2005 19:32:48.807084: Deleting route to FEX

04/14/2005 19:32:48.814678: Module disconnected

04/14/2005 19:32:48.816369: Deleting route to FEX

04/14/2005 19:32:48.823789: Module disconnected

04/14/2005 19:32:48.826038: Offlining Module

04/14/2005 19:32:48.826685: Offlining Module

04/14/2005 19:33:34.065205: Deleting route to FEX

04/14/2005 19:33:34.072753: Module disconnected

04/14/2005 19:33:34.073898: Offlining Module

04/14/2005 19:33:34.075512: Deleting route to FEX

04/14/2005 19:33:34.082780: Module disconnected

04/14/2005 19:33:34.085162: Offlining Module

04/14/2005 19:33:34.108909: Deleting route to FEX

04/14/2005 19:33:34.116541: Module disconnected

04/14/2005 19:33:34.118758: Offlining Module

04/14/2005 19:34:06.942831: Deleting route to FEX

04/14/2005 19:34:06.950497: Module disconnected

04/14/2005 19:34:06.952049: Offlining Module

04/14/2005 19:34:06.953282: Deleting route to FEX

04/14/2005 19:34:06.960850: Module disconnected

04/14/2005 19:34:06.963198: Offlining Module

04/14/2005 19:34:06.988662: Deleting route to FEX

04/14/2005 19:34:06.996309: Module disconnected

04/14/2005 19:34:06.998510: Offlining Module

04/14/2005 19:35:30.639182: Module register received

04/14/2005 19:35:30.641271: Image Version Mismatch

04/14/2005 19:35:30.642573: Registration response sent

04/14/2005 19:35:30.642863: Requesting satellite to download image

04/14/2005 19:40:46.980527: Image preload successful.

04/14/2005 19:40:48.410573: Deleting route to FEX

04/14/2005 19:40:48.418185: Module disconnected

04/14/2005 19:40:48.419680: Module Offline

04/14/2005 19:40:48.421660: Deleting route to FEX

04/14/2005 19:40:48.428859: Module disconnected

04/14/2005 19:40:48.431149: Offlining Module

04/14/2005 19:40:48.452809: Deleting route to FEX

04/14/2005 19:40:48.460347: Module disconnected

04/14/2005 19:40:48.462656: Offlining Module

04/14/2005 19:42:18.759564: Module register received

04/14/2005 19:42:18.763150: Registration response sent

04/14/2005 19:42:18.809369: create module inserted event.

04/14/2005 19:42:18.810380: Module Online Sequence

04/14/2005 19:42:25.044678: Module Online

 

 

N5K-1(config)# fex 101

N5K-1(config-fex)# ?

  description  FEX description

  diagnostic   Diagnostic commands

  hardware     FEX Card type

  no           Negate the command

  pinning      Pinning configurations

  port         Configure a port

  serial       Chassis serial number

  show         Show running config

  type         FEX Card type

  end          Go to exec mode

  exit         Exit from command interpreter

  pop          Pop mode from stack or restore from name

  push         Push current mode to stack or save it under name

  where        Shows the cli context you are in

 

N5K-1(config-fex)# description RACK_1

N5K-1(config-fex)# pinning max-links 1

Change in Max-links will cause traffic disruption.

N5K-1(config-fex)# serial ?

  WORD  Serial number (Max Size 20)

 

N5K-1(config-fex)# serial FOX2520PABC    // BEST PRACTICE TO CONFIGURE serial AND type FOR EASY TROUBLESHOOTING; FROM show fex detail

Changing serial will offline fex.

N5K-1config-fex)# type ?

  N2148T     Fabric Extender 48x1G 4x10G Module

  N2224TP    Fabric Extender 24x1G 2x10G SFP+ Module

  N2232P     Fabric Extender 32x10G 8x10G Module

  N2232TM    Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TM-E  Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TP    Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TT    Fabric Extender 32x10GBase-T 8x10GBase-T Module

  N2248PQ    Fabric Extender 48x10G SFP+ 4x40G QSFP Module

  N2248T     Fabric Extender 48x1G 4x10G Module

  N2248TP-E  Fabric Extender 48x1G 4x10G Module

  N2332TQ    Fabric Extender 32x10GBase-T 4x40G QSFP Module

  N2348TQ    Fabric Extender 48x10GBase-T 6x40G QSFP Module

  N2348TQ-E  Fabric Extender 48x10GBase-T 6x40G QSFP Module

  N2348UPQ   Fabric Extender 48x10G SFP+ 6x40G QSFP Module

  NB22DELL   Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22FJ     Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22HP     Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22IBM    Fabric Extender 14x10G SFP+ 8x10G SFP+ Module

 

N5K-1(config-fex)# type N2248TP-E


N5K-1(config-fex)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  SERVER_RACK_1                Online   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

You'll see the new ports reflected in the show run output.

N5K-1# show run

interface Ethernet101/1/1

interface Ethernet101/1/2

interface Ethernet101/1/3

<OUTPUT TRUNCATED>
 

interface Ethernet101/1/46

interface Ethernet101/1/47

interface Ethernet101/1/48

 

N5K-1# show run interface Ethernet101/1/1

!Command: show running-config interface Ethernet101/1/1
!Time: Thu Apr 14 20:27:52 2005

version 7.3(8)N1(1)

interface Ethernet101/1/1


Perform the same configuration steps on the other parent N5K-2 switch since their configuration is independent from each other. The FEX 2K module will immediately show up as Connected then change to Online status.

N5K-2# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2519PDEF

---       --------             Connected   N2K-C2248TP-E-1GE   FOX2520PABC


N5K-2# 2005 Apr 14 19:42:10 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101Module 1: Cold boot

2005 Apr 14 19:42:44 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 21:16:24 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_ENV_ERROR: FEX-101 Module 1: Check environment alarms.

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

N5K-2# 2005 Apr 14 21:16:31 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

 

N5K-2# show  fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101        FEX0101                Online   N2K-C2248TP-E-1GE   FOX2520ABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

Saturday, July 2, 2022

Cisco Nexus Switch Virtual PortChannel (vPC) Failure Scenarios

Here's a Cisco link regarding the different Nexus vPC terminologies, best practice and failure scenarios for the Peer-Link and Peer-Keepalive. I tried to simulate different failure scenarios in my Nexus switch lab.


Peer-Keepalive Failure (mgmt0 via Layer 3):

- Only the heartbeat between Primary and Secondary Nexus peer will be lost

- vPC adjacency will NOT break/fail

- There's no change in vPC role (Primary/Secondary)

- vPC will still run as normal/forward traffic

 - Ensure NMS monitoring for the Nexus mgmt0 interface



N5K-1# show run interface mgmt0

 

!Command: show running-config interface mgmt0

!Time: Mon Jul 19 02:56:36 2021

 

version 7.3(8)N1(1)

 

interface mgmt0

  vrf member management

  ip address 10.10.2.8/23

 

 

N5K-1# show run vpc

!Command: show running-config vpc
!Time: Thu Jul 22 08:30:50 2021

version 7.3(8)N1(1)
feature vpc

vpc domain 1

  role priority 10

  peer-keepalive destination 10.10.2.9 source 10.10.2.8


interface port-channel1
  vpc peer-link

interface port-channel100
  vpc 100


I shutdown the switchport connected to NK5-1 mgmt0.

 

SW01#configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

SW01(config)#interface Gi1/0/6

SW01(config-if)#shutdown

 

 

N5K-1# show interface mgmt0

mgmt0 is down (Link not connected)

 

  Hardware: GigabitEthernet, address: 00de.fb78.0123 (bia 00de.fb78.0112)

  Internet Address is 10.10.2.8/23

 

 

The Peer-Keepalive status changed to peer is not reachable but peer adjacency is still formed ok.

 

N5K-1# 2021 Jul 19 02:28:59 N5K-1 %$ VDC-1 %$ %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 1, VPC peer keep-alive receive has failed

 

N5K-1# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer adjacency formed ok     

vPC keep-alive status             : peer is not reachable through peer-keepalive

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : primary                      

Number of vPCs configured         : 294

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    up     1,99  

 

<OUTPUT TRUNCATED>

 

 

N5K-1# show vpc peer-keepalive

 

vPC keep-alive status             : peer is not reachable through peer-keepalive

--Send status                   : Success

--Last send at                  : 2021.07.19 02:29:59 804 ms

--Sent on interface             :

--Receive status                : Failed

--Last update from peer         : (65) seconds, (174) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.9

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

 

The FEX module state is still Online.

 

N5K-1# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

100    FEX100                Online   N2K-C2348UPQ-10GE   FOC22401234


 

The same output is seen on the Nexus peer switch.

 

N5K-2# 2021 Jul 19 02:28:59 N5K-2 %$ VDC-1 %$ %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 1, VPC peer keep-alive receive has failed

 

N5K-2# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer adjacency formed ok     

vPC keep-alive status             : peer is not reachable through peer-keepalive

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : secondary                    

Number of vPCs configured         : 294

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    up     1,99      

 

<OUTPUT TRUNCATED>

 

 

N5K-2# show vpc peer-keepalive

 

vPC keep-alive status             : peer is not reachable through peer-keepalive

--Send status                   : Success

--Last send at                  : 2021.07.19 02:30:46 803 ms

--Sent on interface             : mgmt0

--Receive status                : Failed

--Last update from peer         : (112) seconds, (807) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.8

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

 

N5K-2# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

100    FEX100                Online   N2K-C2348UPQ-10GE   FOC22401234

 

 

The vPC Peer-Keepalive status immediately changed to alive after I unshut the switch port on N5K-1 mgmt0,

 

SW01(config)#interface Gi1/0/6

SW01(config-if)#no shutdown

 

 

N5K-1# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer adjacency formed ok     

vPC keep-alive status             : peer is alive                

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : primary                      

Number of vPCs configured         : 294

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    up     1,99        

 

<OUTPUT TRUNCATED>


 

N5K-1# show vpc peer-keepalive

 

vPC keep-alive status             : peer is alive                

--Peer is alive for             : (84) seconds, (386) msec

--Send status                   : Success

--Last send at                  : 2021.07.19 02:36:40 813 ms

--Sent on interface             : mgmt0

--Receive status                : Success

--Last receive at               : 2021.07.19 02:36:40 854 ms

--Received on interface         : mgmt0

--Last update from peer         : (0) seconds, (336) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.9

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

 

N5K-2# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer adjacency formed ok     

vPC keep-alive status             : peer is alive                

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : secondary                    

Number of vPCs configured         : 294

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    up     1,99                                                     

 

<OUTPUT TRUNCATED>

 

 

N5K-2# show vpc peer-keepalive

 

vPC keep-alive status             : peer is alive                

--Peer is alive for             : (114) seconds, (258) msec

--Send status                   : Success

--Last send at                  : 2021.07.19 02:37:11 851 ms

--Sent on interface             : mgmt0

--Receive status                : Success

--Last receive at               : 2021.07.19 02:37:11 834 ms

--Received on interface         : mgmt0

--Last update from peer         : (0) seconds, (227) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.8

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

Peer-Link failure (Port-channel 1):

- All the vPC member ports/FEX on the Secondary Nexus switch will be suspended

- All traffic will flow via the Primary Nexus switch

- This will prevent a "split-brain" scenario

- Traffic on Orphan port/device (i.e. trunk to a standalone switch or router) connected to Secondary Nexus switch will fail or "blackhole"

- Create a Port-Channel with multiple interfaces for Peer-link

 

N5K-1# show run interface po1

 

!Command: show running-config interface port-channel1

!Time: Mon Jul 19 02:55:48 2021

 

version 7.3(8)N1(1)

 

interface port-channel1

  switchport mode trunk

  spanning-tree port type network

  vpc peer-link

 

N5K-1# show port-channel summary

Flags:  D - Down        P - Up in port-channel (members)

        I - Individual  H - Hot-standby (LACP only)

        s - Suspended   r - Module-removed

        S - Switched    R - Routed

        U - Up (port-channel)

        M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

1     Po1(SU)     Eth      LACP      Eth1/23(P)   Eth1/24(P)

<OUTPUT TRUNCATED>


I disabled Port-Channnel 1 Peer-Link on N5K-1 switch. N5K-2 vPC Port-Channel interface immediately became suspended and FEX went offline.

N5K-1# configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

N5K-1(config)# interface port-channel1

N5K-1(config-if)# shutdown

 

 

N5K-2# 2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary

2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number FOX25191234)

2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is offline


 

N5K-2# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer link is down            

vPC keep-alive status             : peer is alive                

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : secondary                    

Number of vPCs configured         : 6  

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    down   -                                                        

 

vPC status

----------------------------------------------------------------------------

id     Port        Status Consistency Reason                     Active vlans

------ ----------- ------ ----------- -------------------------- -----------

100    Po100       down   failed      Peer-link is down          -          

 

 

N5K-2# show vpc peer-keepalive

 

vPC keep-alive status             : peer is alive                

--Peer is alive for             : (1343) seconds, (564) msec

--Send status                   : Success

--Last send at                  : 2021.07.19 03:03:14 84 ms

--Sent on interface             : mgmt0

--Receive status                : Success

--Last receive at               : 2021.07.19 03:03:14 84 ms

--Received on interface         : mgmt0

--Last update from peer         : (0) seconds, (334) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.8

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

 

N5K-2# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

100    FEX100               Offline   N2K-C2348UPQ-10GE   FOC22401234

 

 

Only N5K-1 FEX is online. This is to prevent "split-brain" traffic on the peer switch N5K-2.

 

N5K-1# show vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

 

vPC domain id                     : 1  

Peer status                       : peer link is down            

vPC keep-alive status             : peer is alive                

Configuration consistency status  : success

Per-vlan consistency status       : success                      

Type-2 consistency status         : success

vPC role                          : primary                      

Number of vPCs configured         : 294

Peer Gateway                      : Disabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Operational Layer3 Peer-router    : Disabled

Auto-recovery status              : Enabled (timeout = 240 seconds)

 

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans   

--   ----   ------ --------------------------------------------------

1    Po1    down   -    

 

 

N5K-1# show vpc peer-keepalive

 

vPC keep-alive status             : peer is alive                

--Peer is alive for             : (1278) seconds, (446) msec

--Send status                   : Success

--Last send at                  : 2021.07.19 03:02:07 934 ms

--Sent on interface             : mgmt0

--Receive status                : Success

--Last receive at               : 2021.07.19 03:02:07 882 ms

--Received on interface         : mgmt0

--Last update from peer         : (0) seconds, (428) msec

 

vPC Keep-alive parameters

--Destination                   : 10.10.2.9

--Keepalive interval            : 1000 msec

--Keepalive timeout             : 5 seconds

--Keepalive hold timeout        : 3 seconds

--Keepalive vrf                 : management

--Keepalive udp port            : 3200

--Keepalive tos                 : 192

 

 

N5K-1# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

100    FEX100                Online   N2K-C2348UPQ-10GE   FOC22401234


 

I re-enabled the Port-Channel 1 and it took around a couple of minutes for the FEX in N5K-2 to back back online.

 

N5K-1# configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

N5K-1(config)# interface port-channel1

N5K-1(config-if)# no shutdown

 

 

N5K-2# 2021 Jul 19 03:05:44 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX105-2-SOHMS_ENV_ERROR: FEX-100 Module 1: Check environment alarms.

2021 Jul 19 03:05:48 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online

2021 Jul 19 03:05:48 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line

2021 Jul 19 03:05:50 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online


N5K-2# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

100    FEX100                Online   N2K-C2348UPQ-10GE   FOC22401234


In summary, you can tolerate a separate Peer-Keepalive failure and a separate Peer-Link failure. This will give enough time to troubleshoot and fix the problem (usually at Layer 1). Avoid a Peer-Keepalive followed by a Peer-Link failure at all cost, otherwise traffic instability/split-brain will occur.