Saturday, August 6, 2022

Cisco Nexus 5000 Fabric Extender (FEX) Configuration

After configuring a Virtual PortChannel (vPC) between two parent Cisco Nexus 5000 switches in my previous post, the next step is to configure a Nexus 2000 Fabric Extender (FEX) modules. These are remote line cards which uplinks to a parent Cisco Nexus 5K, 6K, 7K or 9K switches.

N5K-1# configure terminal

N5K-1(config)# feature fex   // ENABLE FEX

N5K-1(config)#

N5K-1(config)# interface Eth1/7

N5K-1(config-if)#  description FEX101 -  RACK 1

N5K-1(config-if)#  switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)# channel-group 101

 

N5K-1(config-if)# interface port-channel101

N5K-1(config-if)#  description FEX101 -  RACK 1

N5K-1(config-if)#  switchport mode fex-fabric

N5K-1(config-if)# fex associate 101

N5K-1(config-if)#  vpc 101

  


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------               Offline   N2K-C2248TP-E-1GE   FOX2520PABC   // STATE WILL CHANGE FROM DISCOVERED TO OFFLINE

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

  


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  SERVER_RACK_1        Image Download   N2K-C2248TP-E-1GE   FOX2520PABC   // FEX 2K WILL DOWNLOAD IMAGE FROM PARENT N5K SWITCH

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

 

N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------               Offline   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)#

N5K-1(config-if)# 2005 Apr 14 19:42:10 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101 Module 1: Cold boot


N5K-1(config-if)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101     --------           Online Sequence   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 19:42:25 N5K-1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

2005 Apr 14 19:42:26 N5K-1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

 


N5K-1config-if)# show fex   // IT TOOK AROUND 10 MINS FOR FEX 2K TO COMPLETE ITS UPGRADE

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  FEX0101                Online   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

N5K-1(config-if)# 2005 Apr 14 19:42:44 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-1 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

 

N5K-1(config-if)# show fex detail

FEX: 101 Description: FEX0101   state: Online

  FEX version: 7.3(8)N1(1) [Switch version: 7.3(8)N1(1)]    // DOWNLOADED NX-OS FROM PARENT N5K SWITCH

  FEX Interim version: 7.3(8)N1(1)

  Switch Interim version: 7.3(8)N1(1)

  Extender Serial: FOX2520ABC

  Extender Model: N2K-C2248TP-E-1GE,  Part No: 73-12345-67

  Card Id: 149, Mac Addr: 28:af:fd:19:12:34, Num Macs: 64

  Module Sw Gen: 12594  [Switch Sw Gen: 21]

  Post level: complete

  Pinning-mode: static    Max-links: 1

  Fabric port for control traffic: Eth1/7

  FCoE Admin: false

  FCoE Oper: true

  FCoE FEX AA Configured: false

  Fabric interface state:

    Po101 - Interface Up. State: Active

    Eth1/7 - Interface Up. State: Active

  Fex Port        State  Fabric Port

       Eth101/1/1  Down       Po101

       Eth101/1/2  Down       Po101

       Eth101/1/3  Down       Po101

       Eth101/1/4  Down       Po101

       Eth101/1/5  Down       Po101

       Eth101/1/6  Down       Po101

       Eth101/1/7  Down       Po101

       Eth101/1/8  Down       Po101

       Eth101/1/9  Down       Po101

      Eth101/1/10  Down       Po101

      Eth101/1/11  Down       Po101

      Eth101/1/12  Down       Po101

      Eth101/1/13  Down       Po101

      Eth101/1/14  Down       Po101

      Eth101/1/15  Down       Po101

      Eth101/1/16  Down       Po101

      Eth101/1/17  Down       Po101

      Eth101/1/18  Down       Po101

      Eth101/1/19  Down       Po101

      Eth101/1/20  Down       Po101

      Eth101/1/21  Down       Po101

      Eth101/1/22  Down       Po101

      Eth101/1/23  Down       Po101

      Eth101/1/24  Down       Po101

      Eth101/1/25  Down       Po101

      Eth101/1/26  Down       Po101

      Eth101/1/27  Down       Po101

      Eth101/1/28  Down       Po101

      Eth101/1/29  Down       Po101

      Eth101/1/30  Down       Po101

      Eth101/1/31  Down       Po101

      Eth101/1/32  Down       Po101

      Eth101/1/33  Down       Po101

      Eth101/1/34  Down       Po101

      Eth101/1/35  Down       Po101

      Eth101/1/36  Down       Po101

      Eth101/1/37  Down       Po101

      Eth101/1/38  Down       Po101

      Eth101/1/39  Down       Po101

      Eth101/1/40  Down       Po101

      Eth101/1/41  Down       Po101

      Eth101/1/42  Down       Po101

      Eth101/1/43  Down       Po101

      Eth101/1/44  Down       Po101

      Eth101/1/45  Down       Po101

      Eth101/1/46  Down       Po101

      Eth101/1/47  Down       Po101

      Eth101/1/48  Down       Po101

Logs:

04/14/2005 19:32:34.118822: Module register received

04/14/2005 19:32:34.120820: Image Version Mismatch

04/14/2005 19:32:34.122113: Registration response sent

04/14/2005 19:32:34.122404: Requesting satellite to download image

04/14/2005 19:32:34.968158: Deleting route to FEX

04/14/2005 19:32:34.975847: Module disconnected

04/14/2005 19:32:34.977323: Module Offline

04/14/2005 19:32:48.807084: Deleting route to FEX

04/14/2005 19:32:48.814678: Module disconnected

04/14/2005 19:32:48.816369: Deleting route to FEX

04/14/2005 19:32:48.823789: Module disconnected

04/14/2005 19:32:48.826038: Offlining Module

04/14/2005 19:32:48.826685: Offlining Module

04/14/2005 19:33:34.065205: Deleting route to FEX

04/14/2005 19:33:34.072753: Module disconnected

04/14/2005 19:33:34.073898: Offlining Module

04/14/2005 19:33:34.075512: Deleting route to FEX

04/14/2005 19:33:34.082780: Module disconnected

04/14/2005 19:33:34.085162: Offlining Module

04/14/2005 19:33:34.108909: Deleting route to FEX

04/14/2005 19:33:34.116541: Module disconnected

04/14/2005 19:33:34.118758: Offlining Module

04/14/2005 19:34:06.942831: Deleting route to FEX

04/14/2005 19:34:06.950497: Module disconnected

04/14/2005 19:34:06.952049: Offlining Module

04/14/2005 19:34:06.953282: Deleting route to FEX

04/14/2005 19:34:06.960850: Module disconnected

04/14/2005 19:34:06.963198: Offlining Module

04/14/2005 19:34:06.988662: Deleting route to FEX

04/14/2005 19:34:06.996309: Module disconnected

04/14/2005 19:34:06.998510: Offlining Module

04/14/2005 19:35:30.639182: Module register received

04/14/2005 19:35:30.641271: Image Version Mismatch

04/14/2005 19:35:30.642573: Registration response sent

04/14/2005 19:35:30.642863: Requesting satellite to download image

04/14/2005 19:40:46.980527: Image preload successful.

04/14/2005 19:40:48.410573: Deleting route to FEX

04/14/2005 19:40:48.418185: Module disconnected

04/14/2005 19:40:48.419680: Module Offline

04/14/2005 19:40:48.421660: Deleting route to FEX

04/14/2005 19:40:48.428859: Module disconnected

04/14/2005 19:40:48.431149: Offlining Module

04/14/2005 19:40:48.452809: Deleting route to FEX

04/14/2005 19:40:48.460347: Module disconnected

04/14/2005 19:40:48.462656: Offlining Module

04/14/2005 19:42:18.759564: Module register received

04/14/2005 19:42:18.763150: Registration response sent

04/14/2005 19:42:18.809369: create module inserted event.

04/14/2005 19:42:18.810380: Module Online Sequence

04/14/2005 19:42:25.044678: Module Online

 

 

N5K-1(config)# fex 101

N5K-1(config-fex)# ?

  description  FEX description

  diagnostic   Diagnostic commands

  hardware     FEX Card type

  no           Negate the command

  pinning      Pinning configurations

  port         Configure a port

  serial       Chassis serial number

  show         Show running config

  type         FEX Card type

  end          Go to exec mode

  exit         Exit from command interpreter

  pop          Pop mode from stack or restore from name

  push         Push current mode to stack or save it under name

  where        Shows the cli context you are in

 

N5K-1(config-fex)# description RACK_1

N5K-1(config-fex)# pinning max-links 1

Change in Max-links will cause traffic disruption.

N5K-1(config-fex)# serial ?

  WORD  Serial number (Max Size 20)

 

N5K-1(config-fex)# serial FOX2520PABC    // BEST PRACTICE TO CONFIGURE serial AND type FOR EASY TROUBLESHOOTING; FROM show fex detail

Changing serial will offline fex.

N5K-1config-fex)# type ?

  N2148T     Fabric Extender 48x1G 4x10G Module

  N2224TP    Fabric Extender 24x1G 2x10G SFP+ Module

  N2232P     Fabric Extender 32x10G 8x10G Module

  N2232TM    Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TM-E  Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TP    Fabric Extender 32x10GBase-T 8x10G SFP+ Module

  N2232TT    Fabric Extender 32x10GBase-T 8x10GBase-T Module

  N2248PQ    Fabric Extender 48x10G SFP+ 4x40G QSFP Module

  N2248T     Fabric Extender 48x1G 4x10G Module

  N2248TP-E  Fabric Extender 48x1G 4x10G Module

  N2332TQ    Fabric Extender 32x10GBase-T 4x40G QSFP Module

  N2348TQ    Fabric Extender 48x10GBase-T 6x40G QSFP Module

  N2348TQ-E  Fabric Extender 48x10GBase-T 6x40G QSFP Module

  N2348UPQ   Fabric Extender 48x10G SFP+ 6x40G QSFP Module

  NB22DELL   Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22FJ     Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22HP     Fabric Extender 16x10G SFP+ 8x10G SFP+ Module

  NB22IBM    Fabric Extender 14x10G SFP+ 8x10G SFP+ Module

 

N5K-1(config-fex)# type N2248TP-E


N5K-1(config-fex)# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101  SERVER_RACK_1                Online   N2K-C2248TP-E-1GE   FOX2520PABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

You'll see the new ports reflected in the show run output.

N5K-1# show run

interface Ethernet101/1/1

interface Ethernet101/1/2

interface Ethernet101/1/3

<OUTPUT TRUNCATED>
 

interface Ethernet101/1/46

interface Ethernet101/1/47

interface Ethernet101/1/48

 

N5K-1# show run interface Ethernet101/1/1

!Command: show running-config interface Ethernet101/1/1
!Time: Thu Apr 14 20:27:52 2005

version 7.3(8)N1(1)

interface Ethernet101/1/1


Perform the same configuration steps on the other parent N5K-2 switch since their configuration is independent from each other. The FEX 2K module will immediately show up as Connected then change to Online status.

N5K-2# show fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2519PDEF

---       --------             Connected   N2K-C2248TP-E-1GE   FOX2520PABC


N5K-2# 2005 Apr 14 19:42:10 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL: FEX-101Module 1: Cold boot

2005 Apr 14 19:42:44 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 Module 1: Runtime diag detected major event: Voltage failure on power supply: 2

2005 Apr 14 19:42:47 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_DIAG_ERROR: FEX-101 System minor alarm on power supply 2: failed

2005 Apr 14 21:16:24 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX101-2-SOHMS_ENV_ERROR: FEX-101 Module 1: Check environment alarms.

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

2005 Apr 14 21:16:28 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line

N5K-2# 2005 Apr 14 21:16:31 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

 

N5K-2# show  fex

  FEX         FEX           FEX              FEX              Fex      

Number    Description      State            Model            Serial    

------------------------------------------------------------------------

101        FEX0101                Online   N2K-C2248TP-E-1GE   FOX2520ABC

---       --------            Discovered   N2K-C2248TP-E-1GE   FOX2520PDEF

 

No comments:

Post a Comment