My Network Lab

Saturday, August 6, 2022

Cisco Nexus 5000 Fabric Extender (FEX) Configuration

After configuring a Virtual PortChannel (vPC) between two parent Cisco Nexus 5000 switches in my previous post, the next step is to configure a Nexus 2000 Fabric Extender (FEX) modules. These are remote line cards which uplinks to a parent Cisco Nexus 5K, 6K, 7K or 9K switches.

N5K-1# configure terminal

N5K-1(config)# feature fex // ENABLE FEX

N5K-1(config)#

N5K-1(config)# interface Eth1/7

N5K-1(config-if)# description FEX101 - RACK 1

N5K-1(config-if)# switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)# channel-group 101

N5K-1(config-if)# interface port-channel101

N5K-1(config-if)# description FEX101 - RACK 1

N5K-1(config-if)# switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)# vpc 101

N5K-1(config-if)# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 -------- Offline N2K-C2248TP-E-1GE FOX2520PABC // STATE WILL CHANGE FROM DISCOVERED TO OFFLINE

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

N5K-1(config-if)# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 SERVER_RACK_1 Image Download N2K-C2248TP-E-1GE FOX2520PABC // FEX 2K WILL DOWNLOAD IMAGE FROM PARENT N5K SWITCH

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

N5K-1(config-if)# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 -------- Offline N2K-C2248TP-E-1GE FOX2520PABC

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

N5K-1(config-if)#

N5K-1(config-if)# 2005 Apr 14 19:42:10 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101 Module 1: Cold boot

N5K-1(config-if)# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 -------- Online Sequence N2K-C2248TP-E-1GE FOX2520PABC

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

2005 Apr 14 19:42:26 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

N5K-1config-if)# show fex // IT TOOK AROUND 10 MINS FOR FEX 2K TO COMPLETE ITS UPGRADE

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 FEX0101 Online N2K-C2248TP-E-1GE FOX2520PABC

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:44 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

N5K-1(config-if)# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 7.3(8)N1(1) [Switch version: 7.3(8)N1(1)] // DOWNLOADED NX-OS FROM PARENT N5K SWITCH

FEX Interim version: 7.3(8)N1(1)

Switch Interim version: 7.3(8)N1(1)

Extender Serial: FOX2520ABC

Extender Model: N2K-C2248TP-E-1GE, Part No: 73-12345-67

Card Id: 149, Mac Addr: 28:af:fd:19:12:34, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

Post level: complete

Pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/7

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Po101 - Interface Up. State: Active

Eth1/7 - Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Down Po101

Eth101/1/2 Down Po101

Eth101/1/3 Down Po101

Eth101/1/4 Down Po101

Eth101/1/5 Down Po101

Eth101/1/6 Down Po101

Eth101/1/7 Down Po101

Eth101/1/8 Down Po101

Eth101/1/9 Down Po101

Eth101/1/10 Down Po101

Eth101/1/11 Down Po101

Eth101/1/12 Down Po101

Eth101/1/13 Down Po101

Eth101/1/14 Down Po101

Eth101/1/15 Down Po101

Eth101/1/16 Down Po101

Eth101/1/17 Down Po101

Eth101/1/18 Down Po101

Eth101/1/19 Down Po101

Eth101/1/20 Down Po101

Eth101/1/21 Down Po101

Eth101/1/22 Down Po101

Eth101/1/23 Down Po101

Eth101/1/24 Down Po101

Eth101/1/25 Down Po101

Eth101/1/26 Down Po101

Eth101/1/27 Down Po101

Eth101/1/28 Down Po101

Eth101/1/29 Down Po101

Eth101/1/30 Down Po101

Eth101/1/31 Down Po101

Eth101/1/32 Down Po101

Eth101/1/33 Down Po101

Eth101/1/34 Down Po101

Eth101/1/35 Down Po101

Eth101/1/36 Down Po101

Eth101/1/37 Down Po101

Eth101/1/38 Down Po101

Eth101/1/39 Down Po101

Eth101/1/40 Down Po101

Eth101/1/41 Down Po101

Eth101/1/42 Down Po101

Eth101/1/43 Down Po101

Eth101/1/44 Down Po101

Eth101/1/45 Down Po101

Eth101/1/46 Down Po101

Eth101/1/47 Down Po101

Eth101/1/48 Down Po101

Logs:

04/14/2005 19:32:34.118822: Module register received

04/14/2005 19:32:34.120820: Image Version Mismatch

04/14/2005 19:32:34.122113: Registration response sent

04/14/2005 19:32:34.122404: Requesting satellite to download image

04/14/2005 19:32:34.968158: Deleting route to FEX

04/14/2005 19:32:34.975847: Module disconnected

04/14/2005 19:32:34.977323: Module Offline

04/14/2005 19:32:48.807084: Deleting route to FEX

04/14/2005 19:32:48.814678: Module disconnected

04/14/2005 19:32:48.816369: Deleting route to FEX

04/14/2005 19:32:48.823789: Module disconnected

04/14/2005 19:32:48.826038: Offlining Module

04/14/2005 19:32:48.826685: Offlining Module

04/14/2005 19:33:34.065205: Deleting route to FEX

04/14/2005 19:33:34.072753: Module disconnected

04/14/2005 19:33:34.073898: Offlining Module

04/14/2005 19:33:34.075512: Deleting route to FEX

04/14/2005 19:33:34.082780: Module disconnected

04/14/2005 19:33:34.085162: Offlining Module

04/14/2005 19:33:34.108909: Deleting route to FEX

04/14/2005 19:33:34.116541: Module disconnected

04/14/2005 19:33:34.118758: Offlining Module

04/14/2005 19:34:06.942831: Deleting route to FEX

04/14/2005 19:34:06.950497: Module disconnected

04/14/2005 19:34:06.952049: Offlining Module

04/14/2005 19:34:06.953282: Deleting route to FEX

04/14/2005 19:34:06.960850: Module disconnected

04/14/2005 19:34:06.963198: Offlining Module

04/14/2005 19:34:06.988662: Deleting route to FEX

04/14/2005 19:34:06.996309: Module disconnected

04/14/2005 19:34:06.998510: Offlining Module

04/14/2005 19:35:30.639182: Module register received

04/14/2005 19:35:30.641271: Image Version Mismatch

04/14/2005 19:35:30.642573: Registration response sent

04/14/2005 19:35:30.642863: Requesting satellite to download image

04/14/2005 19:40:46.980527: Image preload successful.

04/14/2005 19:40:48.410573: Deleting route to FEX

04/14/2005 19:40:48.418185: Module disconnected

04/14/2005 19:40:48.419680: Module Offline

04/14/2005 19:40:48.421660: Deleting route to FEX

04/14/2005 19:40:48.428859: Module disconnected

04/14/2005 19:40:48.431149: Offlining Module

04/14/2005 19:40:48.452809: Deleting route to FEX

04/14/2005 19:40:48.460347: Module disconnected

04/14/2005 19:40:48.462656: Offlining Module

04/14/2005 19:42:18.759564: Module register received

04/14/2005 19:42:18.763150: Registration response sent

04/14/2005 19:42:18.809369: create module inserted event.

04/14/2005 19:42:18.810380: Module Online Sequence

04/14/2005 19:42:25.044678: Module Online

N5K-1(config)# fex 101

N5K-1(config-fex)# ?

description FEX description

diagnostic Diagnostic commands

hardware FEX Card type

no Negate the command

pinning Pinning configurations

port Configure a port

serial Chassis serial number

show Show running config

type FEX Card type

end Go to exec mode

exit Exit from command interpreter

pop Pop mode from stack or restore from name

push Push current mode to stack or save it under name

where Shows the cli context you are in

N5K-1(config-fex)# description RACK_1

N5K-1(config-fex)# pinning max-links 1

Change in Max-links will cause traffic disruption.

N5K-1(config-fex)# serial ?

WORD Serial number (Max Size 20)

N5K-1(config-fex)# serial FOX2520PABC // BEST PRACTICE TO CONFIGURE serial AND type FOR EASY TROUBLESHOOTING; FROM show fex detail

Changing serial will offline fex.

N5K-1config-fex)# type ?

N2148T Fabric Extender 48x1G 4x10G Module

N2224TP Fabric Extender 24x1G 2x10G SFP+ Module

N2232P Fabric Extender 32x10G 8x10G Module

N2232TM Fabric Extender 32x10GBase-T 8x10G SFP+ Module

N2232TM-E Fabric Extender 32x10GBase-T 8x10G SFP+ Module

N2232TP Fabric Extender 32x10GBase-T 8x10G SFP+ Module

N2232TT Fabric Extender 32x10GBase-T 8x10GBase-T Module

N2248PQ Fabric Extender 48x10G SFP+ 4x40G QSFP Module

N2248T Fabric Extender 48x1G 4x10G Module

N2248TP-E Fabric Extender 48x1G 4x10G Module

N2332TQ Fabric Extender 32x10GBase-T 4x40G QSFP Module

N2348TQ Fabric Extender 48x10GBase-T 6x40G QSFP Module

N2348TQ-E Fabric Extender 48x10GBase-T 6x40G QSFP Module

N2348UPQ Fabric Extender 48x10G SFP+ 6x40G QSFP Module

NB22DELL Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

NB22FJ Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

NB22HP Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

NB22IBM Fabric Extender 14x10G SFP+ 8x10G SFP+ Module

N5K-1(config-fex)# type N2248TP-E

N5K-1(config-fex)# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 SERVER_RACK_1 Online N2K-C2248TP-E-1GE FOX2520PABC

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

You'll see the new ports reflected in the show run output.

N5K-1# show run

interface Ethernet101/1/1

interface Ethernet101/1/2

interface Ethernet101/1/3

interface Ethernet101/1/46

interface Ethernet101/1/47

interface Ethernet101/1/48

N5K-1# show run interface Ethernet101/1/1

!Command: show running-config interface Ethernet101/1/1
!Time: Thu Apr 14 20:27:52 2005

version 7.3(8)N1(1)

interface Ethernet101/1/1

Perform the same configuration steps on the other parent N5K-2 switch since their configuration is independent from each other. The FEX 2K module will immediately show up as Connected then change to Online status.

N5K-2# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

--- -------- Discovered N2K-C2248TP-E-1GE FOX2519PDEF

--- -------- Connected N2K-C2248TP-E-1GE FOX2520PABC

N5K-2# 2005 Apr 14 19:42:10 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101Module 1: Cold boot

2005 Apr 14 19:42:44 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 21:16:24 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_ENV_ERROR: FEX-101 Module 1: Check environment alarms.

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

N5K-2# 2005 Apr 14 21:16:31 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

N5K-2# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 FEX0101 Online N2K-C2248TP-E-1GE FOX2520ABC

--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF

Saturday, July 2, 2022

Cisco Nexus Switch Virtual PortChannel (vPC) Failure Scenarios

Here's a Cisco link regarding the different Nexus vPC terminologies, best practice and failure scenarios for the Peer-Link and Peer-Keepalive. I tried to simulate different failure scenarios in my Nexus switch lab.

Peer-Keepalive Failure (mgmt0 via Layer 3):

- Only the heartbeat between Primary and Secondary Nexus peer will be lost

- vPC adjacency will NOT break/fail

- There's no change in vPC role (Primary/Secondary)

- vPC will still run as normal/forward traffic

- Ensure NMS monitoring for the Nexus mgmt0 interface

N5K-1# show run interface mgmt0

!Command: show running-config interface mgmt0

!Time: Mon Jul 19 02:56:36 2021

version 7.3(8)N1(1)

interface mgmt0

vrf member management

ip address 10.10.2.8/23

N5K-1# show run vpc

!Command: show running-config vpc
!Time: Thu Jul 22 08:30:50 2021

version 7.3(8)N1(1)
feature vpc

vpc domain 1

role priority 10

peer-keepalive destination 10.10.2.9 source 10.10.2.8

interface port-channel1
vpc peer-link

interface port-channel100
vpc 100

I shutdown the switchport connected to NK5-1 mgmt0.

SW01#configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

SW01(config)#interface Gi1/0/6

SW01(config-if)#shutdown

N5K-1# show interface mgmt0

mgmt0 is down (Link not connected)

Hardware: GigabitEthernet, address: 00de.fb78.0123 (bia 00de.fb78.0112)

Internet Address is 10.10.2.8/23

The Peer-Keepalive status changed to peer is not reachable but peer adjacency is still formed ok.

N5K-1# 2021 Jul 19 02:28:59 N5K-1 %$ VDC-1 %$ %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 1, VPC peer keep-alive receive has failed

N5K-1# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is not reachable through peer-keepalive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 294

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 up 1,99

N5K-1# show vpc peer-keepalive

vPC keep-alive status : peer is not reachable through peer-keepalive

--Send status : Success

--Last send at : 2021.07.19 02:29:59 804 ms

--Sent on interface :

--Receive status : Failed

--Last update from peer : (65) seconds, (174) msec

vPC Keep-alive parameters

--Destination : 10.10.2.9

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

The FEX module state is still Online.

N5K-1# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

100 FEX100 Online N2K-C2348UPQ-10GE FOC22401234

The same output is seen on the Nexus peer switch.

N5K-2# 2021 Jul 19 02:28:59 N5K-2 %$ VDC-1 %$ %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 1, VPC peer keep-alive receive has failed

N5K-2# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is not reachable through peer-keepalive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 294

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 up 1,99

N5K-2# show vpc peer-keepalive

vPC keep-alive status : peer is not reachable through peer-keepalive

--Send status : Success

--Last send at : 2021.07.19 02:30:46 803 ms

--Sent on interface : mgmt0

--Receive status : Failed

--Last update from peer : (112) seconds, (807) msec

vPC Keep-alive parameters

--Destination : 10.10.2.8

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

N5K-2# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

100 FEX100 Online N2K-C2348UPQ-10GE FOC22401234

The vPC Peer-Keepalive status immediately changed to alive after I unshut the switch port on N5K-1 mgmt0,

SW01(config)#interface Gi1/0/6

SW01(config-if)#no shutdown

N5K-1# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 294

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 up 1,99

N5K-1# show vpc peer-keepalive

vPC keep-alive status : peer is alive

--Peer is alive for : (84) seconds, (386) msec

--Send status : Success

--Last send at : 2021.07.19 02:36:40 813 ms

--Sent on interface : mgmt0

--Receive status : Success

--Last receive at : 2021.07.19 02:36:40 854 ms

--Received on interface : mgmt0

--Last update from peer : (0) seconds, (336) msec

vPC Keep-alive parameters

--Destination : 10.10.2.9

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

N5K-2# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 294

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 up 1,99

N5K-2# show vpc peer-keepalive

vPC keep-alive status : peer is alive

--Peer is alive for : (114) seconds, (258) msec

--Send status : Success

--Last send at : 2021.07.19 02:37:11 851 ms

--Sent on interface : mgmt0

--Receive status : Success

--Last receive at : 2021.07.19 02:37:11 834 ms

--Received on interface : mgmt0

--Last update from peer : (0) seconds, (227) msec

vPC Keep-alive parameters

--Destination : 10.10.2.8

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

Peer-Link failure (Port-channel 1):

- All the vPC member ports/FEX on the Secondary Nexus switch will be suspended

- All traffic will flow via the Primary Nexus switch

- This will prevent a "split-brain" scenario

- Traffic on Orphan port/device (i.e. trunk to a standalone switch or router) connected to Secondary Nexus switch will fail or "blackhole"

- Create a Port-Channel with multiple interfaces for Peer-link

N5K-1# show run interface po1

!Command: show running-config interface port-channel1

!Time: Mon Jul 19 02:55:48 2021

version 7.3(8)N1(1)

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

N5K-1# show port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

1 Po1(SU) Eth LACP Eth1/23(P) Eth1/24(P)

I disabled Port-Channnel 1 Peer-Link on N5K-1 switch. N5K-2 vPC Port-Channel interface immediately became suspended and FEX went offline.

N5K-1# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

N5K-1(config)# interface port-channel1

N5K-1(config-if)# shutdown

N5K-2# 2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary

2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number FOX25191234)

2021 Jul 19 02:59:03 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is offline

N5K-2# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer link is down

vPC keep-alive status : peer is alive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 6

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 down -

vPC status

----------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

------ ----------- ------ ----------- -------------------------- -----------

100 Po100 down failed Peer-link is down -

N5K-2# show vpc peer-keepalive

vPC keep-alive status : peer is alive

--Peer is alive for : (1343) seconds, (564) msec

--Send status : Success

--Last send at : 2021.07.19 03:03:14 84 ms

--Sent on interface : mgmt0

--Receive status : Success

--Last receive at : 2021.07.19 03:03:14 84 ms

--Received on interface : mgmt0

--Last update from peer : (0) seconds, (334) msec

vPC Keep-alive parameters

--Destination : 10.10.2.8

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

N5K-2# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

100 FEX100 Offline N2K-C2348UPQ-10GE FOC22401234

Only N5K-1 FEX is online. This is to prevent "split-brain" traffic on the peer switch N5K-2.

N5K-1# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer link is down

vPC keep-alive status : peer is alive

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 294

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Operational Layer3 Peer-router : Disabled

Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po1 down -

N5K-1# show vpc peer-keepalive

vPC keep-alive status : peer is alive

--Peer is alive for : (1278) seconds, (446) msec

--Send status : Success

--Last send at : 2021.07.19 03:02:07 934 ms

--Sent on interface : mgmt0

--Receive status : Success

--Last receive at : 2021.07.19 03:02:07 882 ms

--Received on interface : mgmt0

--Last update from peer : (0) seconds, (428) msec

vPC Keep-alive parameters

--Destination : 10.10.2.9

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : management

--Keepalive udp port : 3200

--Keepalive tos : 192

N5K-1# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

100 FEX100 Online N2K-C2348UPQ-10GE FOC22401234

I re-enabled the Port-Channel 1 and it took around a couple of minutes for the FEX in N5K-2 to back back online.

N5K-1# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

N5K-1(config)# interface port-channel1

N5K-1(config-if)# no shutdown

N5K-2# 2021 Jul 19 03:05:44 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX105-2-SOHMS_ENV_ERROR: FEX-100 Module 1: Check environment alarms.

2021 Jul 19 03:05:48 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online

2021 Jul 19 03:05:48 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line

2021 Jul 19 03:05:50 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online

N5K-2# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

100 FEX100 Online N2K-C2348UPQ-10GE FOC22401234

In summary, you can tolerate a separate Peer-Keepalive failure and a separate Peer-Link failure. This will give enough time to troubleshoot and fix the problem (usually at Layer 1). Avoid a Peer-Keepalive followed by a Peer-Link failure at all cost, otherwise traffic instability/split-brain will occur.