After configuring a Virtual PortChannel (vPC) between two parent Cisco Nexus 5000 switches in my previous post, the next step is to configure a Nexus 2000 Fabric Extender (FEX) modules. These are remote line cards which uplinks to a parent Cisco Nexus 5K, 6K, 7K or 9K switches.
N5K-1# configure terminal
N5K-1(config)#
feature fex // ENABLE FEX
N5K-1(config)#
N5K-1(config)#
interface Eth1/7
N5K-1(config-if)# description FEX101 - RACK 1
N5K-1(config-if)# switchport mode fex-fabric
N5K-1(config-if)#
fex associate 101
N5K-1(config-if)#
channel-group 101
N5K-1(config-if)#
interface port-channel101
N5K-1(config-if)# description FEX101 - RACK 1
N5K-1(config-if)# switchport mode fex-fabric
N5K-1(config-if)#
fex associate 101
N5K-1(config-if)# vpc 101
N5K-1(config-if)#
show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 -------- Offline N2K-C2248TP-E-1GE FOX2520PABC // STATE WILL CHANGE FROM DISCOVERED TO OFFLINE
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
N5K-1(config-if)#
show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 SERVER_RACK_1 Image Download N2K-C2248TP-E-1GE FOX2520PABC // FEX 2K WILL DOWNLOAD IMAGE FROM PARENT N5K SWITCH
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
N5K-1(config-if)#
show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 -------- Offline N2K-C2248TP-E-1GE FOX2520PABC
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
N5K-1(config-if)#
N5K-1(config-if)# 2005 Apr 14 19:42:10 N5K-1 %$ VDC-1 %$
%SATCTRL-FEX101-2-SATCTRL: FEX-101 Module 1: Cold boot
N5K-1(config-if)#
show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 -------- Online Sequence N2K-C2248TP-E-1GE FOX2520PABC
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
N5K-1(config-if)#
2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
2005 Apr 14 19:42:25
N5K-1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line
2005 Apr 14 19:42:26 N5K-1 %$ VDC-1 %$
%PFMA-2-FEX_STATUS: Fex 101 is online
N5K-1config-if)#
show fex // IT TOOK AROUND 10 MINS FOR
FEX 2K TO COMPLETE ITS UPGRADE
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101
Online N2K-C2248TP-E-1GE FOX2520PABC
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
N5K-1(config-if)#
2005 Apr 14 19:42:44 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR:
FEX-101 System minor alarm on power supply 2: failed
2005 Apr 14 19:42:47
N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime
diag detected major event: Voltage failure on power supply: 2
2005 Apr 14 19:42:47
N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor
alarm on power supply 2: failed
N5K-1(config-if)#
show fex detail
FEX: 101 Description: FEX0101 state: Online
FEX version:
7.3(8)N1(1) [Switch version: 7.3(8)N1(1)] // DOWNLOADED NX-OS FROM PARENT N5K SWITCH
FEX Interim version: 7.3(8)N1(1)
Switch Interim version: 7.3(8)N1(1)
Extender Serial: FOX2520ABC
Extender Model: N2K-C2248TP-E-1GE, Part No: 73-12345-67
Card Id: 149, Mac Addr: 28:af:fd:19:12:34,
Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
Post level: complete
Pinning-mode: static Max-links: 1
Fabric port
for control traffic: Eth1/7
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po101 -
Interface Up. State: Active
Eth1/7 -
Interface Up. State: Active
Fex Port State
Fabric Port
Eth101/1/1 Down
Po101
Eth101/1/2 Down
Po101
Eth101/1/3 Down
Po101
Eth101/1/4 Down
Po101
Eth101/1/5 Down
Po101
Eth101/1/6 Down
Po101
Eth101/1/7 Down
Po101
Eth101/1/8 Down
Po101
Eth101/1/9 Down
Po101
Eth101/1/10 Down
Po101
Eth101/1/11 Down
Po101
Eth101/1/12 Down
Po101
Eth101/1/13 Down
Po101
Eth101/1/14 Down
Po101
Eth101/1/15 Down
Po101
Eth101/1/16 Down
Po101
Eth101/1/17 Down
Po101
Eth101/1/18 Down
Po101
Eth101/1/19 Down
Po101
Eth101/1/20 Down
Po101
Eth101/1/21 Down
Po101
Eth101/1/22 Down
Po101
Eth101/1/23 Down
Po101
Eth101/1/24 Down
Po101
Eth101/1/25 Down
Po101
Eth101/1/26 Down
Po101
Eth101/1/27 Down
Po101
Eth101/1/28 Down
Po101
Eth101/1/29 Down
Po101
Eth101/1/30 Down
Po101
Eth101/1/31 Down
Po101
Eth101/1/32 Down
Po101
Eth101/1/33 Down
Po101
Eth101/1/34 Down
Po101
Eth101/1/35 Down
Po101
Eth101/1/36 Down
Po101
Eth101/1/37 Down
Po101
Eth101/1/38 Down
Po101
Eth101/1/39 Down
Po101
Eth101/1/40 Down
Po101
Eth101/1/41 Down
Po101
Eth101/1/42 Down
Po101
Eth101/1/43 Down
Po101
Eth101/1/44 Down
Po101
Eth101/1/45 Down
Po101
Eth101/1/46 Down
Po101
Eth101/1/47 Down
Po101
Eth101/1/48 Down
Po101
Logs:
04/14/2005
19:32:34.118822: Module register received
04/14/2005
19:32:34.120820: Image Version Mismatch
04/14/2005
19:32:34.122113: Registration response sent
04/14/2005
19:32:34.122404: Requesting satellite to download image
04/14/2005
19:32:34.968158: Deleting route to FEX
04/14/2005
19:32:34.975847: Module disconnected
04/14/2005
19:32:34.977323: Module Offline
04/14/2005
19:32:48.807084: Deleting route to FEX
04/14/2005
19:32:48.814678: Module disconnected
04/14/2005
19:32:48.816369: Deleting route to FEX
04/14/2005
19:32:48.823789: Module disconnected
04/14/2005
19:32:48.826038: Offlining Module
04/14/2005
19:32:48.826685: Offlining Module
04/14/2005
19:33:34.065205: Deleting route to FEX
04/14/2005
19:33:34.072753: Module disconnected
04/14/2005
19:33:34.073898: Offlining Module
04/14/2005
19:33:34.075512: Deleting route to FEX
04/14/2005
19:33:34.082780: Module disconnected
04/14/2005
19:33:34.085162: Offlining Module
04/14/2005
19:33:34.108909: Deleting route to FEX
04/14/2005
19:33:34.116541: Module disconnected
04/14/2005
19:33:34.118758: Offlining Module
04/14/2005
19:34:06.942831: Deleting route to FEX
04/14/2005
19:34:06.950497: Module disconnected
04/14/2005
19:34:06.952049: Offlining Module
04/14/2005
19:34:06.953282: Deleting route to FEX
04/14/2005
19:34:06.960850: Module disconnected
04/14/2005
19:34:06.963198: Offlining Module
04/14/2005
19:34:06.988662: Deleting route to FEX
04/14/2005
19:34:06.996309: Module disconnected
04/14/2005
19:34:06.998510: Offlining Module
04/14/2005
19:35:30.639182: Module register received
04/14/2005
19:35:30.641271: Image Version Mismatch
04/14/2005
19:35:30.642573: Registration response sent
04/14/2005
19:35:30.642863: Requesting satellite to download image
04/14/2005
19:40:46.980527: Image preload successful.
04/14/2005
19:40:48.410573: Deleting route to FEX
04/14/2005
19:40:48.418185: Module disconnected
04/14/2005
19:40:48.419680: Module Offline
04/14/2005
19:40:48.421660: Deleting route to FEX
04/14/2005
19:40:48.428859: Module disconnected
04/14/2005
19:40:48.431149: Offlining Module
04/14/2005
19:40:48.452809: Deleting route to FEX
04/14/2005
19:40:48.460347: Module disconnected
04/14/2005
19:40:48.462656: Offlining Module
04/14/2005
19:42:18.759564: Module register received
04/14/2005
19:42:18.763150: Registration response sent
04/14/2005
19:42:18.809369: create module inserted event.
04/14/2005
19:42:18.810380: Module Online Sequence
04/14/2005 19:42:25.044678: Module Online
N5K-1(config)# fex
101
N5K-1(config-fex)# ?
description
FEX description
diagnostic
Diagnostic commands
hardware
FEX Card type
no
Negate the command
pinning
Pinning configurations
port
Configure a port
serial
Chassis serial number
show
Show running config
type
FEX Card type
end
Go to exec mode
exit
Exit from command interpreter
pop
Pop mode from stack or restore from name
push
Push current mode to stack or save it under name
where
Shows the cli context you are in
N5K-1(config-fex)#
description RACK_1
N5K-1(config-fex)#
pinning max-links 1
Change in Max-links will cause traffic disruption.
N5K-1(config-fex)#
serial ?
WORD
Serial number (Max Size 20)
N5K-1(config-fex)#
serial FOX2520PABC // BEST PRACTICE TO CONFIGURE serial AND type FOR EASY TROUBLESHOOTING; FROM show
fex detail
Changing serial will offline fex.
N5K-1config-fex)#
type ?
N2148T
Fabric Extender 48x1G 4x10G Module
N2224TP
Fabric Extender 24x1G 2x10G SFP+ Module
N2232P
Fabric Extender 32x10G 8x10G Module
N2232TM
Fabric Extender 32x10GBase-T 8x10G SFP+ Module
N2232TM-E
Fabric Extender 32x10GBase-T 8x10G SFP+ Module
N2232TP
Fabric Extender 32x10GBase-T 8x10G SFP+ Module
N2232TT
Fabric Extender 32x10GBase-T 8x10GBase-T Module
N2248PQ
Fabric Extender 48x10G SFP+ 4x40G QSFP Module
N2248T
Fabric Extender 48x1G 4x10G Module
N2248TP-E Fabric Extender 48x1G 4x10G Module
N2332TQ
Fabric Extender 32x10GBase-T 4x40G QSFP Module
N2348TQ
Fabric Extender 48x10GBase-T 6x40G QSFP Module
N2348TQ-E
Fabric Extender 48x10GBase-T 6x40G QSFP Module
N2348UPQ
Fabric Extender 48x10G SFP+ 6x40G QSFP Module
NB22DELL
Fabric Extender 16x10G SFP+ 8x10G SFP+ Module
NB22FJ
Fabric Extender 16x10G SFP+ 8x10G SFP+ Module
NB22HP
Fabric Extender 16x10G SFP+ 8x10G SFP+ Module
NB22IBM
Fabric Extender 14x10G SFP+ 8x10G SFP+ Module
N5K-1(config-fex)#
type N2248TP-E
N5K-1(config-fex)#
show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101
SERVER_RACK_1
Online N2K-C2248TP-E-1GE FOX2520PABC
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
You'll see the new ports reflected in the show run output.
N5K-1# show run
interface Ethernet101/1/1
interface Ethernet101/1/2
interface Ethernet101/1/3
<OUTPUT TRUNCATED>
interface Ethernet101/1/46
interface Ethernet101/1/47
interface Ethernet101/1/48
N5K-1# show run interface Ethernet101/1/1
!Command: show running-config interface Ethernet101/1/1
!Time: Thu Apr 14 20:27:52 2005
version 7.3(8)N1(1)
interface Ethernet101/1/1
Perform the same configuration steps on the other parent N5K-2 switch since their configuration is independent from each other. The FEX 2K module will immediately show up as Connected then change to Online status.
N5K-2# show
fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
--- -------- Discovered N2K-C2248TP-E-1GE FOX2519PDEF
---
--------
Connected N2K-C2248TP-E-1GE FOX2520PABC
N5K-2# 2005 Apr 14
19:42:10 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101Module 1: Cold
boot
2005 Apr 14 19:42:44
N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor
alarm on power supply 2: failed
2005 Apr 14 19:42:47
N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime
diag detected major event: Voltage failure on power supply: 2
2005 Apr 14 19:42:47
N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor
alarm on power supply 2: failed
2005 Apr 14 21:16:24
N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_ENV_ERROR: FEX-101 Module 1: Check
environment alarms.
2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$
%PFMA-2-FEX_STATUS: Fex 101 is online
2005 Apr 14 21:16:28
N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line
N5K-2# 2005 Apr 14
21:16:31 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
N5K-2# show fex
FEX
FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101
FEX0101 Online N2K-C2248TP-E-1GE FOX2520ABC
--- -------- Discovered N2K-C2248TP-E-1GE FOX2520PDEF
