Cisco Nexus 9000 NX-OS Upgrade

The Cisco Nexus 9000 (N9K) switch upgrade procedure was simplified compared to upgrading an older Nexus platform. You can use the Cisco Nexus upgrade online tool to check and follow the upgrade path. 

In this scenario, I needed to upgrade a Cisco N9K switch NX-OS: 10.1.x > 10.2(x)M > 10.3(x)M and also upgrade the 10.3.x.M EPLD image in the last step. You just need to upgrade the EPLD image once and it should be the same version of the final target NX-OS code.

 

Check first if the Nexus switch has enough space in bootflash.

N9K# dir bootflash:
       4096    Jun 06 03:30:14 2024  .rpmstore/
       4096    Jun 06 03:30:35 2024  .swtam/
       1782    Jun 06 03:32:12 2024  20240606_033204_poap_25583_init.log
       1782    Jun 06 03:50:51 2024  20240606_035045_poap_25617_init.log
     455215    Jul 24 06:29:15 2024  20240724_041734_poap_25508_1.log
    2097294    Jul 24 06:17:20 2024  20240724_041734_poap_25508_init.log
 1777998029    Jun 06 03:34:10 2024  aci-n9000-dk9.14.1.2g.bin
       4096    Jun 06 03:30:37 2024  eem_snapshots/
       4096    Jun 06 03:30:34 2024  evt_log_snapshot/
       4096    Jun 06 03:42:13 2024  home/
       4096    Jun 06 03:36:26 2024  lost+found/
        952    Jun 06 03:53:54 2024  lpssutil_lpss_log
 1964521472    Jun 06 03:26:35 2024  nxos.10.1.x.bin
          0    Jun 06 03:33:22 2024  platform-sdk.cmd
      32665    Jul 24 06:28:44 2024  poap_retry_debugs.log
       4096    Jun 06 03:31:08 2024  scripts/
       4096    Jun 06 03:32:07 2024  virt_strg_pool_bf_vdc_1/
       4096    Jun 06 03:31:07 2024  virtual-instance/
         59    Jul 24 04:16:26 2024  virtual-instance.conf
       4132    Aug 01 06:17:33 2024  vlan.dat

Usage for bootflash://sup-local
 4148449280 bytes used
112038109184 bytes free
116186558464 bytes total

Transfer the NX-OS from an FTP/TFTP server to bootflash.

N9K# copy ftp://ftpuser:password@172.27.2.3/nxos64-cs.10.2.x.M.bin bootflash:nxos64-cs.10.2.x.M.bin
Enter vrf (If no input, current vrf 'default' is considered): <
[#####################    ]        42.57MB

***** Transfer of file Completed Successfully *****
Copy complete, now saving to disk (please wait)...
Copy complete.

N9K# dir bootflash:
       4096    Jun 06 03:30:14 2024  .rpmstore/
       4096    Jun 06 03:30:35 2024  .swtam/
       1782    Jun 06 03:32:12 2024  20240606_033204_poap_25583_init.log
       1782    Jun 06 03:50:51 2024  20240606_035045_poap_25617_init.log
     455215    Jul 24 06:29:15 2024  20240724_041734_poap_25508_1.log
    2097294    Jul 24 06:17:20 2024  20240724_041734_poap_25508_init.log
 1777998029    Jun 06 03:34:10 2024  aci-n9000-dk9.14.1.2g.bin
       4096    Jun 06 03:30:37 2024  eem_snapshots/
       4096    Jun 06 03:30:34 2024  evt_log_snapshot/
       4096    Jun 06 03:42:13 2024  home/
       4096    Jun 06 03:36:26 2024  lost+found/
        952    Jun 06 03:53:54 2024  lpssutil_lpss_log
 1964521472    Jun 06 03:26:35 2024  nxos.10.1.1.bin
  2019112448    Aug 11 14:31:28 2024  nxos64-cs.10.2.x.M.bin
          0    Jun 06 03:33:22 2024  platform-sdk.cmd
      32665    Jul 24 06:28:44 2024  poap_retry_debugs.log
       4096    Jun 06 03:31:08 2024  scripts/
       4096    Jun 06 03:32:07 2024  virt_strg_pool_bf_vdc_1/
       4096    Jun 06 03:31:07 2024  virtual-instance/
         59    Jul 24 04:16:26 2024  virtual-instance.conf
       4132    Aug 01 06:17:33 2024  vlan.dat

Usage for bootflash://sup-local
 6144655360 bytes used
110041903104 bytes free
116186558464 bytes total

Verify the MD5 checksum and compare it to the checksum published in the Cisco download website. It only took a few seconds to complete this process.

N9K# show file bootflash:nxos64-cs.10.2.x.M.bin md5sum
8ec58aa31b351833cb85e78b69868123


Verify the compatibility of the new NX-OS.

N9K# show install all impact nxos bootflash:nxos64-cs.10.2.x.M.bin
Installer will perform impact only check. Please wait.

Verifying image bootflash:/nxos64-cs.10.2.x.M.bin for boot variable "nxos".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.2.x.M.bin.
[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos64-cs.10.2.x.M.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS


Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  default upgrade is not hitless


Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1        nxos                                   10.1(x)               10.2(x)           yes
     1        bios     v01.09(10/08/2023):v01.09(10/08/2023)    v01.09(10/08/2023)            no

Use the install command to initiate the NX-OS upgrade. Type 'y' to proceed and the Nexus switch will auto reboot.

N9K# install all nxos bootflash:nxos64-cs.10.2.x.M.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive

Verifying image bootflash:/nxos64-cs.10.2x.M.bin for boot variable "nxos".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.2.x.M.bin.
[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos64-cs.10.2.x.M.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS


Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  default upgrade is not hitless


Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1        nxos                                   10.1(1)               10.2(7)           yes
     1        bios     v01.09(10/08/2023):v01.09(10/08/2023)    v01.09(10/08/2023)            no


Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)?  [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
2024 Aug 12 08:01:51 N9K %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Successfully deactivated virtual service 'guestshell+'  


Finishing the upgrade, switch will reboot in 10 seconds.
N9K#


CISCO MODULE
BIOS Ver: 1.09
Switch G10
RC Revision:  02.05.00

Memory Information:
 MRC Revision:00.50.00
 Total  DRAM: 32768 MB
 Memory TOLM: 80000000
 PCIE   BASE: 80000000          Size : 10000000
 PCI32  BASE: 90000000          Limit: FBFFFFFF
 PCI64  BASE: 80000000000       Limit: 83FFFFFFFFF
 UC    START: 80000000000       End  : 84000000000
ME Operational Firmware Version: 06:3.0.3.214

DIMM Information:
 Clock Speed: 1067MHz
 Socket: 0x0 Channel: 0x0 Number: 0x0 Presence: Yes Size: 16GB
 Socket: 0x0 Channel: 0x0 Number: 0x1 Presence: No
 Socket: 0x0 Channel: 0x1 Number: 0x0 Presence: Yes Size: 16GB
 Socket: 0x0 Channel: 0x1 Number: 0x1 Presence: No

 Detected CISCO IOFPGA
 Booting from Primary BIOS
 Code Signing Results: 0x00000000
 Booting from Upgrade FPGA
 IOFPGA Subsystem Vendor ID 0x10ee
 FPGA Revision          : 0x00000013
 FPGA ID                : 0x16905123
 FPGA Date              : 0x20230456
 Power Debug Register1  : 0x00000000
 Power Debug Register2  : 0x110a000f
 Reset Cause Register   : 0x00000001
 Boot Ctrl Register     : 0x0000e0ff
 FPGA Update Status     : 0x80000020

Detected CISCO MIFPGA
 MIFPGA Version        : 0x00000123
 MIFPGA Date           : 0x20240456
 MIFPGA Update Status  : 0x00000020
 MIFPGA ID             : 0x21310123


<OUTPUT TRUNCATED>

Security Lock
Booting bootflash:/nxos64-cs.10.2.x.M.bin
Trying diskboot
 Filesystem type is ext2fs, partition type 0x83

It took around 5 mins for the NX-OS upgrade to complete.


N9K login: 2024 Aug 12 08:05:37 N9K %$ VDC-1 %$ %USER-1-SYSTEM_MSG: Registering VNTAGC with dchal 1  - vntagc
2024 Aug 12 08:05:37 N9K %$ VDC-1 %$ %USER-1-SYSTEM_MSG: Registering MTM with dchal 1  - mtm
2024 Aug 12 08:05:39 N9K%$ VDC-1 %$  %USER-0-SYSTEM_MSG: NX mode supported by this card - 0x1fc  - nf
2024 Aug 12 08:05:47 N9K %$ VDC-1 %$ %ASCII-CFG-2-CONFIG_REPLAY_STATUS: Ascii Replay Started.
2024 Aug 12 08:05:52 N9K %$ VDC-1 %$ %ASCII-CFG-2-CONFIG_REPLAY_STATUS: Ascii Replay Done.
2024 Aug 12 08:05:53 N9K %$ VDC-1 %$ %ASCII-CFG-2-CONF_CONTROL: System ready
2024 Aug 12 08:06:11 N9K %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Thirdparty RPMs installation succeeded - /thirdparty_install.py
2024 Aug 12 08:06:19 N9K %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Successfully activated virtual service 'guestshell+'  
2024 Aug 12 08:06:19 N9K %$ VDC-1 %$ %VMAN-2-GUESTSHELL_ENABLED: The guest shell has been enabled. The command 'guestshell' may be used to access it, 'guestshell destroy' to remove it.

Verify the new NX-OS using the show version command.

N9K# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2024, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.

Software
  BIOS: version 01.09
  NXOS: version 10.2(x) [Maintenance Release]
  BIOS compile time:  10/08/2023
  NXOS image file is: bootflash:///nxos64-cs.10.2.x.M.bin
  NXOS compile time:  2/28/2024 12:00:00 [02/17/2024 21:17:45]

Hardware
  cisco Nexus9000 C93xx-xx Chassis
  Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 32813048 kB of memory.
  Processor Board ID FDO282217Q1
  Device name: N9K
  bootflash:  115805708 kB

Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 34 second(s)

Last reset at 526211 usecs after Mon Aug 12 08:02:04 2024
  Reason: Reset due to upgrade
  System version: 10.1(1)
  Service:

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):

The memory space in bootflash isn't large enough, so you'll need to delete the previous NX-OS image file. Repeat the same steps until the final target NX-OS code. Ensure there's at least one previous NX-OS code for backup.

/bootflash/nxos64-cs.10.3.x.M.bin: Write could not complete, check free space on device

Error during copy

***** Transfer of file aborted *****

Copy failed. Removing file nxos64-cs.10.3.x.M.bin

N9K# delete bootflash:nxos.10.1.x.bin
Do you want to delete "/nxos.10.1.x.bin" ? (yes/no/abort)   [y] y

Once the final NX-OS is running, you'll upgrade the EPLD image. Use an identical image version with the main NX-OS.

N9K# install all nxos nxos64-cs.10.3.x.M.bin epld n9000-epld.10.3.x.M.img

Installer will perform compatibility check first. Please wait.

Installer is forced disruptive

 

Verifying image bootflash:/nxos64-cs.10.3.x.M.bin for boot variable "nxos".

[####################] 100% -- SUCCESS

 

Verifying EPLD/FPGA image bootflash:/n9000-epld.10.3.x.M.img.

[####################] 100% -- SUCCESS

 

Verifying image type.

[####################] 100% -- SUCCESS

 

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.3.x.M.bin.

[####################] 100% -- SUCCESS

 

Preparing "bios" version info using image bootflash:/nxos64-cs.10.3.x.M.bin.

[####################] 100% -- SUCCESS

 

Performing module support checks.

[####################] 100% -- SUCCESS

 

Notifying services about system upgrade.

[####################] 100% -- SUCCESS

 

 

Compatibility check is done:

Module  bootable          Impact  Install-type  Reason

------  --------  --------------  ------------  ------

     1       yes      disruptive         reset  default upgrade is not hitless

    27       yes      disruptive          none  default upgrade is not hitless

 

 

Images will be upgraded according to following table:

Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required

------  ----------  ----------------------------------------  --------------------  ------------

     1       lcn9k                                   10.3(x)               10.3(x)            no

    27        nxos                                   10.3(x)               10.3(x)            no

    27        bios     v01.09(10/08/2023):v01.09(10/08/2023)    v01.09(10/08/2023)            no

 

 

FPGA microcode will be upgraded according to following table:

Module  Type   EPLD              Running-Version   New-Version  Upg-Required

------  ----  -------------      ---------------   -----------  ------------

    27   SUP  MI FPGA                   0x20        0x18             No

    27   SUP  IO FPGA                   0x13        0x13             No

 

EPLD Upgrade may result in multiple modules going offline.

 

Additional info for this installation:

--------------------------------------

 

Service "vpc" in vdc 1: Vpc is enabled, Please make sure both Vpc peer switches have same boot mode using 'show boot mode' and proceed

 

 

Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)?  [n] y

 

 

Install is in progress, please wait.

 

Setting boot variables.

[[####################] 100% -- SUCCESS

 

Performing configuration copy.

[####################] 100% -- SUCCESS

 

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

 

Module 27: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

 

Performing EPLD/FPGA upgrade bootflash:/n9000-epld.10.3.x.M.img.

[####################] 100% -- SUCCESS

 

All EPLD/FPGAs are up to date.

 

 

Install has been successful.

 

Juniper Root Password Recovery

Here's a useful Juniper link in recovering the root password.

I tried to login to my virtual MX device but couldn't access it, so I've performed a root password recovery.

vMX1 (ttyd0)

 

login: root

Password:

Login incorrect

login: root

Password:

Login incorrect

login: root

Password:

Login incorrect

 

vMX1 (ttyd0)

 

login:

 

 

// RELOAD JUNIPER DEVICE, CONNECT TO CONSOLE

 

 

Booting from Hard Disk...

Loading /boot/loader

Consoles: serial port 

BIOS drive A: is disk0

BIOS drive C: is disk1

BIOS 639kB/1047424kB available memory

 

FreeBSD/i386 bootstrap loader, Revision 1.2

(builder@larth.juniper.net, Sat Jun  7 07:19:45 UTC 2014)

Loading /boot/defaults/loader.conf

/kernel text=0x927168 data=0x55514+0x11417c syms=[0x4+0xa2e10+0x4+0xedc20]

/boot/modules/if_bge.ko text=0xfeec data=0x370+0x80 syms=[0x4+0xe40+0x4+0xe12]

/boot/modules/if_em.ko text=0x1465c data=0x7e4+0x14 syms=[0x4+0x18d0+0x4+0x1c89]

<OUTPUT TRUNCATED>

 

// PRESS SPACE BAR WHEN YOU SEE THIS PROMPT

 

Hit [Enter] to boot immediately, or space bar for command prompt.

 

 

Type '?' for a list of commands, 'help' for more detailed help.

OK ?

Available commands:

  heap             show heap usage

  reboot           reboot the system

  bcachestat       get disk block cache stats

  autoboot         boot automatically after a delay

  boot             boot a file or loaded kernel

  nextboot         set next boot device

  more             show contents of a file

  read             read input from the terminal

  echo             echo arguments

  unset            unset a variable

  set              set a variable

  show             show variable(s)

  ?                list commands

  help             detailed help

  install          install JUNOS

  include          read commands from a file

  ls               list files

  lsmod            list loaded modules

  unload           unload all modules

  load             load a kernel or module

  pnpscan          scan for PnP devices

  recover          initiate recovery process from compact flash

  boot-conf        load kernel and modules, then autoboot

  read-conf        read a configuration file

  enable-module    enable loading of a module

  disable-module   disable loading of a module

  toggle-module    toggle loading of a module

  show-module      show module load data

 

 

// TYPE boot -s

 

OK boot -s

platform_early_bootinit: M/T/EX Series Early Boot Initialization

mtx_platform_set_re_type: Found Re type 160

GDB: debug ports: sio

GDB: current port: sio

KDB: debugger backends: ddb gdb

KDB: current backend: ddb

Copyright (c) 1996-2014, Juniper Networks, Inc.

All rights reserved.

Copyright (c) 1992-2007 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

        The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

JUNOS 14.1R1.10 #0: 2014-06-07 09:37:07 UTC

    builder@larth.juniper.net:/volume/build/junos/14.1/release/14.1R1.10/obj-i386/junos/bsd/kernel

Timecounter "i8254" frequency 1193182 Hz quality 0

CPU: QEMU Virtual CPU version 2.5+ (1895.61-MHz 686-class CPU)

  Origin = "GenuineIntel"  Id = 0x663  Stepping = 3

 

 <OUTPUT TRUNCATED>

// TO INITIATE PASSWORD RECOVER TYPE recovery

Attaching /packages/jbase via /dev/mdctl...

Mounted jbase package on /dev/md0...

System watchdog timer disabled

Enter full pathname of shell or 'recovery' for root password recovery or RETURN for /bin/sh: recovery

 

Performing filesystem consistency checks ...

/dev/ad0s1a: 2685 files, 247617 used, 1608782 free (2 frags, 402195 blocks, 0.0% fragmentation)

/dev/ad0s1e: 10 files, 8 used, 515311 free (3 frags, 128827 blocks, 0.0% fragmentation)

/dev/ad0s1f: 197 files, 15972 used, 4522583 free (67 frags, 1130629 blocks, 0.0% fragmentation)

 

Performing mount of main filesystems ...

Verified manifest signed by PackageProductionEc_2014

Verified boot-modules signed by PackageProductionEc_2014

Verified jboot signed by PackageProductionEc_2014

Verified jbase-14.1R1.10 signed by PackageProductionEc_2014

 

<OUTPUT TRUNCATED>

 

Performing initialization of management services ...

 

Performing checkout of management services ...

 

NOTE: Once in the CLI, you will need to enter configuration mode using

NOTE: the 'configure' command to make any required changes. For example,

NOTE: to reset the root password, type:

NOTE:    configure

NOTE:    set system root-authentication plain-text-password

NOTE:    (enter the new password when asked)

NOTE:    commit

NOTE:    exit

NOTE:    exit

NOTE: When you exit the CLI, you will be asked if you want to reboot

NOTE: the system

 

 

// RECONFIGURE THE ROOT PASSWORD, SAVE AND REBOOT

 

Starting CLI ...

root> configure

Entering configuration mode

 

[edit]

root# set system root-authentication plain-text-password

New password:

Retype new password:

 

[edit]

root# commit

 

[edit]

root# exit

 

Exiting configuration mode

 

root> exit

 

Reboot the system? [y/n] y

Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done

Waiting (max 60 seconds) for system process `vnlru' to stop...done

Waiting (max 60 seconds) for system process `bufdaemon' to stop...done

Waiting (max 60 seconds) for system process `syncer' to stop...

Syncing disks, vnodes remaining...0 0 0 0 0 0 done

 

syncing disks... All buffers synced.

Uptime: 5m36s

Normal shutdown (no dump device defined)

Rebooting...

 

 

vMX1 (ttyd0)

 

login: root

Password:

 

--- JUNOS 14.1R1.10 built 2014-06-07 09:37:07 UTC

root@vMX1% cli

root@vMX1>

Cisco Feature Navigator

You can use the Cisco Feature Navigator online tool to check if a certain feature is supported on a specific Cisco platform or software version.

Select either Login (need CCO account) or Guest. Just click on Guest.

Click Browse > Routing.

You can either search for Product (platform/model) or Features.

Select Features > search/type: cipher > TLS 1.2 > Browse.

Cisco Flexible NetFlow (FNF) Restrictions

I've configured Flexible NetFlow (FNF) in a Cisco ASR1K router and sourced it from GigabitEthernet0 (Mgmt-intf), which is the default out-of-band management interface. I wasn't getting any NetFlow statistics in the collector server and found out in this Cisco link that FNF export isn't supported in the management port (below is a snippet). You'll notice in the show flow exporter statistics indicated a "failed to send" flow.

 

Restrictions for Flexible NetFlow

The following are restrictions for Flexible NetFlow:

 

Flexible NetFlow export is not supported on the Ethernet management port, Gi0/0. 

 

ASR1K#show flow exporter statistics

Flow Exporter MyNetflow:

  Packet send statistics (last cleared 00:00:05 ago):

    Successfully sent:         0                     (0 bytes)

    Reason not given:          49                    (64052 bytes)

 

  Client send statistics:

    Client: Option options application-name

      Records added:           0

      Bytes added:             0

 

    Client: Option options application-attributes

      Records added:           0

      Bytes added:             0

 

    Client: Flow Monitor MyNetflow

      Records added:           1138

        - failed to send:      1138

      Bytes added:             62590

        - failed to send:      62590

 

So I used a different source interface (sub-interface) and VRF which can still reach the NetFlow collector server. Notice the "failed to send" flows had disappeared.

ASR1K#show flow exporter statistics
Flow Exporter MyNetflow:
  Packet send statistics (last cleared 00:01:30 ago):
    Successfully sent:         2422                  (3292436 bytes)

  Client send statistics:
    Client: Option options application-name
      Records added:           2252
        - sent:                2252
      Bytes added:             186916
        - sent:                186916

    Client: Option options application-attributes
      Records added:           0
      Bytes added:             0

    Client: Flow Monitor MysNetflow

      Records added:           55823

        - sent:                55823

      Bytes added:             3070265

        - sent:                3070265

Configure QinQ (802.1ad) in a Cisco IOS-XE Router

Here's a Cisco link in configuring QinQ (802.1ad) or double VLAN tagging in a Cisco IOS-XE router.

SUMMARY STEPS

1.  enable
2. configure terminal
3. interface type number
4. dot1q tunneling ethertype ethertype
5. interface type number . subinterface-number
6. encapsulation dot1q vlan-id second-dot1q {any | vlan-id | vlan-id - vlan-id [ vlan-id - vlan-id ]}
7. pppoe enable [group group-name ] [max-sessions max-sessions-number ]
8. exit
9. Repeat Step 5 to configure another subinterface.
10. Repeat Step 6 and Step 7 to specify the VLAN tags to be terminated on the subinterface.
11. end 

 

Router(config)#interface GigabitEthernet0/0/1

Router(config-if)#dot1q ?

  tunneling  dot1q tunneling configuration

 

Router(config-if)#dot1q tunneling ?

  ethertype  ethertype used for dot1q tunneling packets

 

Router(config-if)#dot1q tunneling ethertype ?

  0x88A8  dot1q tunneling etype 0x88A8

  0x9100  dot1q tunneling etype 0x9100

  0x9200  dot1q tunneling etype 0x9200

 

Router(config-if)#dot1q tunneling ethertype 0x88A8

Router(config-if)#end

 

Router#show run interface Gi0/0/1   

Building configuration...

 

Current configuration : 173 bytes

!

interface GigabitEthernet0/0/1

 no ip address

 negotiation auto

 dot1q tunneling ethertype 0x88A8

end

 

 

Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#interface GigabitEthernet0/0/1.210
Router(config-subif)#encapsulation ?
  dot1Q            IEEE 802.1Q Virtual LAN
  priority-tagged  Priority-tagged (VLAN 0)

Router(config-subif)#encapsulation dot1q ?
  <1-4094>  IEEE 802.1Q VLAN ID required

Router(config-subif)#encapsulation dot1q 20 ?
  native        Make this as native vlan
  second-dot1q  Configure this subinterface as a 1Q-in-1Q subinterface
  <cr>          <cr>

Router(config-subif)#encapsulation dot1q 20 second-dot1q ?
  WORD  Second (inner) VLAN IDs e.g. 100,200-300,400,500-600
  any   All Inner VLAN IDs not configured on another subinterface

Router(config-subif)#encapsulation dot1q 20 second-dot1q 200
Router(config-subif)#end

 

Router#show run interface Gi0/0/1.200   

Building configuration...

 

Current configuration : 216 bytes

!

interface GigabitEthernet0/0/1.200

 encapsulation dot1Q 20 second-dot1q 200

 ip vrf forwarding CUSTOMER-A

 ip address 192.168.200.1 255.255.255.0

end

 

Managing Configuration Files in a Cisco ASR920

This Cisco link covers the management of configuration files in a Cisco ASR920. You wouldn't find the startup-config in the Cisco ASR920 bootflash memory.

ASR920# dir

Directory of bootflash:/

 

   11  drwx            16384  Oct 16 2015 21:37:46 +00:00  lost+found

15105  drwx             4096  Oct 31 2016 22:31:05 +00:00  .prst_sync

30209  drwx             4096   Apr 8 2021 21:34:51 +00:00  .installer

   13  -rw-             1182  Oct 16 2015 21:51:23 +00:00  CAT1941V21234_1445051234.lic

   14  -rw-             1176  Oct 16 2015 21:51:56 +00:00  CAT1941V25678_1445055678.lic

135937  drwx             4096   Apr 8 2021 21:25:23 +00:00  core

45313  drwx             4096  Oct 16 2015 21:56:47 +00:00  .rollback_timer

   15  -rw-                0  Oct 16 2015 21:57:06 +00:00  tracelogs.809

120833  drwx           270336  May 29 2024 04:49:54 +00:00  tracelogs

60417  drwx             4096  Jul 15 2014 01:09:40 +00:00  usb_modem

   32  -rw-        311284732   Mar 9 2018 10:36:53 +00:00  asr920-universalk9_npe.03.x.x.S.x-3.S7-ext.bin

   30  -rw-              822  Mar 22 2016 16:11:14 +00:00  usb_modem_stats.txt

   31  -rw-        467318537  Dec 16 2020 02:27:50 +00:00  asr920-universalk9_npe.16.x.x.SPA.bin

151044  drwx             4096   Apr 8 2021 21:39:04 +00:00  onep

   33  -rw-           317446  Oct 31 2016 22:24:58 +00:00  crashinfo_RP_00_00_20161031-222450-UTC

   34  -rw-          1586124  Dec 16 2020 01:53:56 +00:00  asr920_x_x_xr_s_rommon.pkg

151042  drwx             4096   Aug 4 2021 09:33:14 +00:00  .dbpersist

151043  drwx             4096   Apr 8 2021 21:39:09 +00:00  license_evlog

 

1339412480 bytes total (366039040 bytes free)

 

The startup-config is found in the nvram directory.

 

ASR920#dir nvram:

Directory of nvram:/

 

32769  -rw-            25314                    <no date>  startup-config

32770  ----             3689                    <no date>  private-config

32771  -rw-            25314                    <no date>  underlying-config

    1  ----              462                    <no date>  persistent-data

    2  -rw-               17                    <no date>  ecfm_ieee_mib

    3  -rw-             3257                    <no date>  ifIndex-table

 

33554432 bytes total (33519233 bytes free)

 

To view the startup-config, use the more nvram:startup-config command:

ASR920#more nvram:startup-config
!
! Last configuration change at 10:25:49 UTC Thu May 16 2024 by nx
! NVRAM config last updated at 10:25:51 UTC Thu May 16 2024 by nx
!
version 16.12
no service pad
service timestamps debug datetime msec show-timezone
service timestamps log datetime msec show-timezone
service password-encryption
no platform punt-keepalive disable-kernel-core
platform bfd-debug-trace 1
platform xconnect load-balance-hash-algo mac-ip-instanceid
platform tcam-parity-error enable
platform tcam-threshold alarm-frequency 1
!
hostname ASR920
!
boot-start-marker
boot system bootflash:asr920-universalk9_npe.16.x.x.SPA.bin
boot system bootflash:asr920-universalk9_npe.03.xx.0x.S.1xx-3.S7-ext.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family

<OUTPUT TRUNCATED>

 

To perform a file transfer in a Cisco ASR920, use the copy nvram:startup-config <ftp/tftp://<REMOTE IP> command. Make sure the source interface can reach the remote file server.

ASR920#ping 172.27.5.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.5.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 196/196/196 ms

ASR920(config)#ip tftp source-interface Loopback0
ASR920(config)#end

ASR920#copy nvram:startup-config tftp://172.27.5.3
Address or name of remote host [172.27.5.3]?
Destination filename [startup-confg]?
!!
25314 bytes copied in 10.516 secs (2407 bytes/sec)

 

I renamed the file to startup-config2 (to avoid any file conflict), transferred it to the new Cisco ASR920, boot the new config using copy nvram:startup-config2 running-config and issued write memory to save config. Booting the new startup-config to running-config only took a few seconds.

This greatly speeds up staging a new router (in this case RMA replacement). It also lessens the human error if you have a big configuration file.

RMA_ASR920#copy nvram:startup-config2 running-config

Destination filename [running-config]?

 

%Log packet overrun, PC 0x111F6508, format:

User:%s  logged command:%s

% Previously established LDP sessions will not be affected by this change.

% OSPF: Reference bandwidth is changed.

        Please ensure reference bandwidth is consistent across all routers.

25080 bytes copied in 4.564 secs (5495 bytes/sec)

ASR920#     // NOTICE THE HOSTNAME IMMEDIATELY CHANGED

ASR920#show ip interface brief

% Authorization failed.     // I'M UNABLE TO ISSUE COMMANDS SINCE IT HAS AAA CONFIGURED

Cisco ASR920 License Install (non Smart License)

I had to RMA a Cisco ASR920 router due to a faulty port and unable to detect the inserted SFP. I also requested the Cisco TAC License team to generate a new license file (Traditional/PAK license) based on the feature licenses on the previous ASR920 (bundled in a single license). You'll need to use the new chassis serial number to generate the new PAK license. 

The RMA ASR920 came with the default IOS-XE version 16.9.3, so it's easy for to upgrade to any 16.x code. I had a previous post regarding the IOS-XE upgrade in a Cisco ASR920. The RMA also doesn't come with new hot-swappable power supply unit (PSU) so you had to swap out the PSU from the old to new ASR920.

RMA_ASR920#copy tftp://172.27.5.3/CAT24071234_20240621080644.lic bootflash:

Destination filename [CAT24071234_20240621080644.lic]?

Accessing tftp://172.27.5.3/CAT24071234_20240621080644.lic...

Loading CAT24071234_20240621080644.lic from 172.27.5.3 (via GigabitEthernet0): !

[OK - 3326 bytes]

 

3326 bytes copied in 22.088 secs (151 bytes/sec)

 

 

RMA_ASR920#dir

Directory of bootflash:/

 

   11  drwx            16384   Jul 1 2020 01:41:57 +00:00  lost+found

46273  drwx             4096   Jul 1 2020 01:42:01 +00:00  .prst_sync

   12  -rw-        445862301   Jul 1 2020 01:57:10 +00:00  asr920-universalk9_npe.16.09.03.SPA.bin

23137  drwx            20480  Jun 21 2024 19:00:37 +00:00  tracelogs

 7713  drwx             4096  Jun 21 2024 16:43:44 +00:00  .installer

 

<OUTPUT TRUNCATED>

   16  -rw-             3326  Jun 21 2024 19:04:11 +00:00  CAT24071234_20240621080644.lic

 

1241329664 bytes total (253034496 bytes free)

 

 

RMA_ASR920#license install bootflash:CAT24071234_20240621080644.lic

Installing licenses from "bootflash:CAT24071234_20240621080644.lic

Installing...Feature:24portGE-4port10GE...Successful:Supported

Installing...Feature:advancedmetroipaccess...Successful:Supported

Installing...Feature:metroaccess...Failed:

% Error: Duplicate license

2/3 licenses were successfully installed

1/3 licenses were existing licenses

0/3 licenses were failed to install

 

 

I noticed the installed and my target advancedmetroipaccess License State was Active, Not in Use.

 

RMA_ASR920#show license

Index 1 Feature: advancedmetroipaccess         

        Period left: Life time

        License Type: Permanent

        License State: Active, Not in Use

        License Count: Non-Counted

        License Priority: Medium

Index 2 Feature: metroipaccess                 

        Period left: Not Activated

        License Type: Evaluation

        License State: Active, Not in Use, EULA not accepted

        License Count: Non-Counted

        License Priority: None

Index 3 Feature: metroaccess                   

        Period left: Life time

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

Index 4 Feature: atm                           

Index 5 Feature: oc3                           

Index 6 Feature: oc12                          

Index 7 Feature: 1588                          

Index 8 Feature: 10GEupgradelicense            

Index 9 Feature: 24portGE-4port10GE            

        Period left: Life time

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

Index 10 Feature: 12x1GEupgradelicense          

 

 

RMA_ASR920#show license all

License Store: Primary License Storage

StoreIndex: 1   Feature: metroaccess                       Version: 1.0

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

StoreIndex: 2   Feature: 24portGE-4port10GE                Version: 1.0

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

StoreIndex: 3   Feature: advancedmetroipaccess             Version: 1.0

        License Type: Permanent

        License State: Active, Not in Use

        License Count: Non-Counted

        License Priority: Medium

License Store: Built-In License Storage

StoreIndex: 0   Feature: advancedmetroipaccess             Version: 1.0

        License Type: Evaluation

        License State: Inactive

            Evaluation total period: 8  weeks 4  days

            Evaluation period left: 8  weeks 4  days

            Period used: 0  minute  0  second 

        License Count: Non-Counted

        License Priority: None

StoreIndex: 1   Feature: metroipaccess                     Version: 1.0

        License Type: Evaluation

        License State: Active, Not in Use, EULA not accepted

            Evaluation total period: 8  weeks 4  days

            Evaluation period left: 8  weeks 4  days

            Period used: 0  minute  0  second 

        License Count: Non-Counted

        License Priority: None

StoreIndex: 2   Feature: metroaccess                       Version: 1.0

        License Type: Evaluation

        License State: Inactive

            Evaluation total period: 8  weeks 4  days

            Evaluation period left: 8  weeks 4  days

            Period used: 0  minute  0  second 

        License Count: Non-Counted

        License Priority: None

 

Per the Cisco link, I had to configure the license boot level and perform a reload command to take effect.

RMA_ASR920#show run | i license boot

license boot level metroaccess

RMA_ASR920#configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

RMA_ASR920(config)#license ?

  accept   Accept all further License Agreements

  agent    Configure LIC_AGENT

  boot     license boot config commands

  feature  License features

  smart    Smart licensing

  udi      license udi

 

RMA_ASR920(config)#license boot ?

  level  which level to boot

 

RMA_ASR920(config)#license boot level ?

  advancedmetroipaccess  Advanced Metro IP Access License Level

  metroaccess            Metro Access License Level

  metroipaccess          Metro IP Access License Level

 

RMA_ASR920(config)#license boot level advancedmetroipaccess ?

  <cr>  <cr>

 

RMA_ASR920(config)#license boot level advancedmetroipaccess

% use 'write' command to make license boot config take effect on next boot

 

RMA_ASR920(config)#end

RMA_ASR920#write memory

Building configuration...

[OK]

 

RMA_ASR920#show run | i license boot

license boot level advancedmetroipaccess

 

 

RMA_ASR920#show license all

License Store: Primary License Storage

StoreIndex: 1   Feature: metroaccess                       Version: 1.0

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

StoreIndex: 2   Feature: 24portGE-4port10GE                Version: 1.0

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

StoreIndex: 3   Feature: advancedmetroipaccess             Version: 1.0

        License Type: Permanent

        License State: Active, Not in Use

        License Count: Non-Counted

        License Priority: Medium

 

 

RMA_ASR920#show version

Cisco IOS XE Software, Version 16.x.x

Cisco IOS Software [Gibraltar], ASR920 Software (PPC_LINUX_IOSD-UNIVERSALK9_NPE-M), Version 16.x.x, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Thu 09-Jul-20 17:13 by mcpre

 

 

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.

All rights reserved.  Certain components of Cisco IOS-XE software are

licensed under the GNU General Public License ("GPL") Version 2.0.  The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0.  For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

 

 

ROM: IOS-XE ROMMON

 

RMA_ASR920 uptime is 23 hours, 12 minutes

Uptime for this control processor is 23 hours, 17 minutes

System returned to ROM by reload

System image file is "bootflash:asr920-universalk9_npe.16.x.x.SPA.bin"

Last reload reason: Reload Command

 

 

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

 

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

 

If you require further assistance please contact us by sending email to

export@cisco.com.

 

License Level: metroaccess

License Type: Permanent

Next reload license Level: advancedmetroipaccess

 

 

Smart Licensing Status: Smart Licensing is DISABLED

 

cisco ASR-920-x-x (Freescale P2020) processor (revision 1.2 GHz) with 890398K/6147K bytes of memory.

Processor board ID CAT24071234

24 Gigabit Ethernet interfaces

4 Ten Gigabit Ethernet interfaces

32768K bytes of non-volatile configuration memory.

3670016K bytes of physical memory.

1231647K bytes of eMMC flash at bootflash:.

 

Configuration register is 0x2102

 

 

RMA_ASR920#reload

Proceed with reload? [confirm]

 

 

Notice the advancemetroipaccess License State was changed to Active, In Use after a reload.

 

RMA_ASR920#show license

Index 1 Feature: advancedmetroipaccess         

        Period left: Life time

        License Type: Permanent

        License State: Active, In Use

        License Count: Non-Counted

        License Priority: Medium

Index 2 Feature: metroipaccess                 

        Period left: Not Activated

        License Type: Evaluation

        License State: Active, Not in Use, EULA not accepted

        License Count: Non-Counted

        License Priority: None

Index 3 Feature: metroaccess                   

        Period left: Life time

        License Type: Permanent

        License State: Active, Not in Use

        License Count: Non-Counted

        License Priority: Medium