My Network Lab

Tuesday, May 3, 2022

Cisco Nexus 5000 Switch NX-OS and Kickstart Code Upgrade

I upgraded a Cisco Nexus 3000 switch in a previous post. This time, I needed to upgrade a Cisco Nexus 5000 switch to NX-OS 7.3(8)N1(1).

I verified the current NX-OS software of the Nexus 5K switch to be 7.3(1)N1(1) using the show version command.

N5K# show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie

s_home.html

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

BIOS: version 2.1.7

Power Sequencer Firmware:

Module 1: v4.0

Fabric Power Sequencer Firmware: Module 1: version v4.0

Microcontroller Firmware: version v0.0.0.15

QSFP Microcontroller Firmware:

Module 1: v2.0.0.0

CXP Microcontroller Firmware:

Module not detected

kickstart: version 7.3(1)N1(1)

system: version 7.3(1)N1(1)

BIOS compile time: 06/16/2016

kickstart image file is: bootflash:///n6000-uk9-kickstart.7.3.1.N1.1.bin

kickstart compile time: 10/5/2016 19:00:00 [10/06/2016 05:19:57]

system image file is: bootflash:///n6000-uk9.7.3.1.N1.1.bin

system compile time: 10/5/2016 19:00:00 [10/06/2016 05:21:39]

Hardware

cisco Nexus 5672UP Chassis ("Nexus 5672UP Supervisor")

Intel(R) Xeon(R) CPU @ 1.80 with 8243284 kB of memory.

Processor Board ID FOC21091234

Device name: N5K

bootflash: 7692288 kB

Kernel uptime is 1 day(s), 15 hour(s), 25 minute(s), 23 second(s)

Last reset at 736681 usecs after Thu Apr 7 05:04:50 2005

Reason: Reset Requested by CLI command reload

System version: 7.3(1)N1(1)

Service:

plugin

Core Plugin, Ethernet Plugin

There's an upgrade path to follow per the Cisco Nexus 7.3(8)N1(1) Release Notes: 7.3(1) > 7.3(2) > 7.3(8). The said code release also list down the supported hardware such as the Nexus switch platforms, line card modules and Fabric Extenders (FEX).

So I upgraded first to 7.3(2)N1(1) and downloaded the main NX-OS and Kickstart image (BIOS).

I deleted some old image files to give space in bootflash using the delete command. It's best practice to keep the previous image in case you need to roll back or downgrade.

N5K# dir bootflash:

4096 Jan 01 00:02:07 2001 .patch/

24 Feb 16 13:02:35 2001 20010216_130235_poap_4722_dhcp.pcap

984128 Feb 16 14:14:59 2001 20010216_130235_poap_4722_init.log

24 Apr 07 05:12:28 2005 20050407_051228_poap_5299_dhcp.pcap

734565 Apr 07 05:32:44 2005 20050407_051228_poap_5299_init.log

35792384 Feb 17 11:25:28 2001 bootflah:

2518 Feb 16 14:37:40 2001 config

630 Jan 01 00:08:06 2001 license_FOC21081234_26_1.lic

4096 Dec 01 06:55:44 2001 lost+found/

37969408 Mar 25 06:53:39 2017 n6000-uk9-kickstart.7.1.0.N1.1b.bin

35792384 Feb 17 11:29:49 2001 n6000-uk9-kickstart.7.3.1.N1.1.bin

310229868 Mar 25 06:54:05 2017 n6000-uk9.7.1.0.N1.1b.bin

324046051 Feb 17 11:29:26 2001 n6000-uk9.7.3.1.N1.1.bin

4096 Feb 17 11:39:26 2001 scripts/

4096 Jan 01 00:02:06 2001 vdc_2/

4096 Jan 01 00:02:06 2001 vdc_3/

4096 Jan 01 00:02:06 2001 vdc_4/

4096 Feb 17 11:35:19 2001 virtual-instance/

4096 Jan 01 00:02:10 2001 virtual-instance-stby-sync/

11352 Apr 07 04:57:37 2005 vlan.dat

1654746 Apr 07 05:03:36 2005 vtp_debug.log

N5K# delete n6000-uk9-kickstart.7.1.0.N1.1b.bin

Do you want to delete "/n6000-uk9-kickstart.7.1.0.N1.1b.bin" ? (yes/no/abort) [y] y

N5K# delete n6000-uk9.7.1.0.N1.1b.bin

Do you want to delete "/n6000-uk9.7.1.0.N1.1b.bin" ? (yes/no/abort) [y] y

N5K# dir

4096 Jan 01 00:02:07 2001 .patch/

24 Feb 16 13:02:35 2001 20010216_130235_poap_4722_dhcp.pcap

984128 Feb 16 14:14:59 2001 20010216_130235_poap_4722_init.log

24 Apr 07 05:12:28 2005 20050407_051228_poap_5299_dhcp.pcap

734565 Apr 07 05:32:44 2005 20050407_051228_poap_5299_init.log

35792384 Feb 17 11:25:28 2001 bootflah:

2518 Feb 16 14:37:40 2001 config

630 Jan 01 00:08:06 2001 license_FOC21081234_26_1.lic

4096 Dec 01 06:55:44 2001 lost+found/

35792384 Feb 17 11:29:49 2001 n6000-uk9-kickstart.7.3.1.N1.1.bin

324046051 Feb 17 11:29:26 2001 n6000-uk9.7.3.1.N1.1.bin

4096 Feb 17 11:39:26 2001 scripts/

4096 Jan 01 00:02:06 2001 vdc_2/

4096 Jan 01 00:02:06 2001 vdc_3/

4096 Jan 01 00:02:06 2001 vdc_4/

4096 Feb 17 11:35:19 2001 virtual-instance/

4096 Jan 01 00:02:10 2001 virtual-instance-stby-sync/

11352 Apr 07 04:57:37 2005 vlan.dat

1654746 Apr 07 05:03:36 2005 vtp_debug.log

Use the copy command to transfer image files with the vrf keyword if using a source VRF.

N5K# copy ftp://ftpuser:ftp123@10.1.1.14/n6000-uk9-kickstart.7.3.2.N1.1.bin bootflash:

Enter vrf (If no input, current vrf 'default' is considered):

N5K# copy ftp://ftpuser:ftp123@10.1.1.14/n6000-uk9-kickstart.7.3.2.N1.1.bin bootflash: vrf ?

default Known VRF name

management Known VRF name

N5K# copy ftp://ftpuser:ftp123@10.1.1.14/n6000-uk9-kickstart.7.3.2.N1.1.bin bootflash: vrf management

Password: // ftp123

[#### ] 4.37MB

***** Transfer of file Completed Successfully *****

Copy complete, now saving to disk (please wait)...

N5K# copy ftp://ftpuser:ftp123@10.1.1.14/n6000-uk9.7.3.2.N1.1.bin bootflash: vrf management

Password:

[######### ] 10.49MB

***** Transfer of file Completed Successfully *****

Copy complete, now saving to disk (please wait)...

N5K# dir

4096 Jan 01 00:02:07 2001 .patch/

24 Feb 16 13:02:35 2001 20010216_130235_poap_4722_dhcp.pcap

984128 Feb 16 14:14:59 2001 20010216_130235_poap_4722_init.log

24 Apr 07 05:12:28 2005 20050407_051228_poap_5299_dhcp.pcap

734565 Apr 07 05:32:44 2005 20050407_051228_poap_5299_init.log

35792384 Feb 17 11:25:28 2001 bootflah:

2518 Feb 16 14:37:40 2001 config

630 Jan 01 00:08:06 2001 license_FOC2108R181_26_1.lic

35792384 Feb 17 11:29:49 2001 n6000-uk9-kickstart.7.3.1.N1.1.bin

35289088 Apr 08 23:42:56 2005 n6000-uk9-kickstart.7.3.2.N1.1.bin

324046051 Feb 17 11:29:26 2001 n6000-uk9.7.3.1.N1.1.bin

343742244 Apr 08 23:47:43 2005 n6000-uk9.7.3.2.N1.1.bin

4096 Feb 17 11:39:26 2001 scripts/

4096 Jan 01 00:02:06 2001 vdc_2/

4096 Jan 01 00:02:06 2001 vdc_3/

4096 Jan 01 00:02:06 2001 vdc_4/

4096 Feb 17 11:35:19 2001 virtual-instance/

4096 Jan 01 00:02:10 2001 virtual-instance-stby-sync/

11352 Apr 07 04:57:37 2005 vlan.dat

1654746 Apr 07 05:03:36 2005 vtp_debug.log

Usage for bootflash://

1220804608 bytes used

6136344576 bytes free

7357149184 bytes total

Verify the MD5 or SHA512 checksum using the show file command for any corruption during the file transfer. Just compare to the MD5 output on the Cisco download site.

N5K# show file bootflash:///n6000-uk9-kickstart.7.3.2.N1.1.bin md5sum

af50c5a3435dd971da57aa44b967598a

N5K# show file bootflash:///n6000-uk9.7.3.2.N1.1.bin md5sum

ef5943379bbef4be2800cb60b13f922b

It's also good practice to verify the new images using the show install command before the actual install or code upgrade.

N5K# show install all impact kickstart bootflash:///n6000-uk9-kickstart.7.3.2.N1.1.bin system bootflash:///n6000-uk9.7.3.2.N1.1.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin for boot variable "kickstart".

[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.3.2.N1.1.bin for boot variable "system". // THIS WILL TAKE AROUND 5 MINUTES

[####################] 100% -- SUCCESS

Verifying image type.

[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n6000-uk9.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n6000-uk9.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n6000-uk9.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Performing module support checks.

[####################] 100% -- SUCCESS

Notifying services about system upgrade.

[####################] 100% -- SUCCESS

Compatibility check is done:

Module bootable Impact Install-type Reason

------ -------- -------------- ------------ ------

1 yes disruptive reset Non-disruptive install not supported if L3 was enabled

2 yes disruptive reset Non-disruptive install not supported if L3 was enabled

Images will be upgraded according to following table:

Module Image Running-Version New-Version Upg-Required

------ ---------------- ---------------------- ---------------------- ------------

1 system 7.3(1)N1(1) 7.3(2)N1(1) yes

1 kickstart 7.3(1)N1(1) 7.3(2)N1(1) yes

1 bios v2.1.7(06/16/2016) v2.1.7(06/16/2016) no

1 power-seq v4.0 v4.0 no

1 fabric-power-seq v4.0 v4.0 no

2 power-seq v4.0 v4.0 no

1 microcontroller v0.0.0.15 v0.0.0.15 no

To perform the actual Nexus switch code upgrade, use the install command.

N5K# install all kickstart bootflash:///n6000-uk9-kickstart.7.3.2.N1.1.bin system bootflash:///n6000-uk9.7.3.2.N1.1.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin for boot variable "kickstart".

[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.3.2.N1.1.bin for boot variable "system".

[####################] 100% -- SUCCESS

Verifying image type.

[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n6000-uk9.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n6000-uk9.7.3.2.N1.1.bin.

[####################] 100% -- SUCCESS

Performing module support checks.

[####################] 100% -- SUCCESS

Notifying services about system upgrade.

[####################] 100% -- SUCCESS

Compatibility check is done:

Module bootable Impact Install-type Reason

------ -------- -------------- ------------ ------

1 yes disruptive reset Non-disruptive install not supported if L3 was enabled

2 yes disruptive reset Non-disruptive install not supported if L3 was enabled

Images will be upgraded according to following table:

Module Image Running-Version New-Version Upg-Required

------ ---------------- ---------------------- ---------------------- ------------

1 system 7.3(1)N1(1) 7.3(2)N1(1) yes

1 kickstart 7.3(1)N1(1) 7.3(2)N1(1) yes

1 bios v2.1.7(06/16/2016) v2.1.7(06/16/2016) no

1 power-seq v4.0 v4.0 no

1 fabric-power-seq v4.0 v4.0 no

2 power-seq v4.0 v4.0 no

1 microcontroller v0.0.0.15 v0.0.0.15 no

Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)? y

Install is in progress, please wait.

Performing runtime checks.

[####################] 100% -- SUCCESS

Setting boot variables.

[####################] 100% -- SUCCESS

Performing configuration copy.

[####################] 100% -- SUCCESS

Finishing the upgrade, switch will reboot in 10 seconds.

N5K# [156233.485298] Shutdown Ports..

[156233.520690] writing reset reason 49,

2005 Apr 9 00:29:48 N5K %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [156233.485298] Shutdown Ports.. - kernel

2005 Apr 9 00:29:48 N5K %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [156233.520690] writing reset reason 49, - kernel

Broadcast message from root (Sat Apr 9 00:29:53 2005):

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "icmpv6" (PID 4321) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "statsclient" (PID 4273) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "fs-daemon" (PID 4149) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "vman" (PID 4208) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "Radius Daemon" (PID 4460) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "Security Daemon" (PID 4249) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "res_mgr" (PID 4217) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "urib" (PID 4258) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "Cert_enroll Daemon" (PID 4251) is forced exit.

Apr 9 00:29:53 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "bios_daemon" (PID 4242) is forced exit.

INIT: Sending processes the KILL signal

Sending all processes the TERM signal...

Apr 9 00:30:00 %LIBSYSMGR-3-SIGTERM_FORCE_EXIT Service "icmpv6" (PID 4321) is forced exit.

Apr 9 00:30:00 %ICMPV6-3-MTS_RECV icmpv6 [4321] Error returned from mts_recv(), errno: Interrupted system call

Apr 9 00:30:00 %ADJMGR-2-MTS_RECV adjmgr [4283] MTS queue receive failed on AM LOW PRI queue: Interrupted system call

Sending all processes the KILL signal...

Unmounting filesystems...

[156252.392985] Resetting board

N5K-C5672UP BIOS v.2.1.7, Fri 06/17/2016, 02:54 PM

N5K-C5672UP BIOS v.2.1.7, Thu 06/16/2016

Booting kickstart image: bootflash:/n6000-uk9-kickstart.7.3.2.N1.1.bin

...............................................................................

.............................Image verification OK

Booting kernel

The Nexus code upgrade took around 7 mintues to complete. Perform similar steps to upgrade to 7.3(8)N1(1).

System is coming up ... Please wait ...

2005 Apr 9 00:36:51 N5K %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online

System is coming up ... Please wait ...

User Access Verification

N5K login: admin

Password:

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

N5K# show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie

s_home.html

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

BIOS: version 2.1.7

Power Sequencer Firmware:

Module 1: v4.0

Fabric Power Sequencer Firmware: Module 1: version v4.0

Microcontroller Firmware: version v0.0.0.15

QSFP Microcontroller Firmware:

Module not detected

SFP Microcontroller Firmware:

Module 1: v1.3.0.0

CXP Microcontroller Firmware:

Module not detected

kickstart: version 7.3(2)N1(1)

system: version 7.3(2)N1(1)

BIOS compile time: 06/16/2016

kickstart image file is: bootflash:///n6000-uk9-kickstart.7.3.2.N1.1.bin

kickstart compile time: 5/12/2017 23:00:00 [05/13/2017 11:15:12]

system image file is: bootflash:///n6000-uk9.7.3.2.N1.1.bin

system compile time: 5/12/2017 23:00:00 [05/13/2017 11:16:49]

Hardware

cisco Nexus 5672UP Chassis ("Nexus 5672UP Supervisor")

Intel(R) Xeon(R) CPU @ 1.80 with 8243276 kB of memory.

Processor Board ID FOC21091234

Device name: N5K

bootflash: 7692288 kB

Kernel uptime is 0 day(s), 0 hour(s), 6 minute(s), 49 second(s)

Last reset at 567632 usecs after Sat Apr 9 00:29:53 2005

Reason: Disruptive upgrade

System version: 7.3(1)N1(1)

Service:

plugin

Core Plugin, Ethernet Plugin

Active Package(s)

Friday, April 1, 2022

Configure IP SLA udp-jitter in a Cisco ASR1K

Here's a nice link regarding IP SLA in a Cisco router. I needed to configure a udp-jitter probe between two ASR1K routers to monitor its path latency, jitter and packet loss. I just need to configure ip sla responder on ASR2 and the rest of the IP SLA config would be on ASR1

ASR2#configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

ASR2(config)#ip sla responder

ASR2(config)#end

ASR2#write memory

Building configuration...

[OK]

ASR2#show run | section ip sla

ip sla responder

-----

ASR1#configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

ASR1(config)#ip sla 100

ASR1(config-ip-sla)# udp-jitter 10.1.1.8 16384 source-ip 10.1.1.7 codec g729a

ASR1(config-ip-sla-jitter)# tos 184

ASR1(config-ip-sla-jitter)# frequency 30

ASR1(config-ip-sla-jitter)# owner ASR1 > ASR2 TUNNEL

ASR1(config-ip-sla-jitter)#

ASR1(config-ip-sla-jitter)#ip sla schedule 100 life forever start-time now

ASR1(config)#end

ASR1#show run | sec ip sla 100

ip sla 100

udp-jitter 10.1.1.8 16384 source-ip 10.1.1.7 codec g729a

tos 184

owner ASR1 > ASR2 TUNNEL

frequency 30

ASR1#write memory

Building configuration...

[OK]

ASR1#show run | sec ip sla 100

ip sla 100

udp-jitter 10.1.1.8 16384 source-ip 10.1.1.7 codec g729a

tos 184

owner ASR1 > ASR2 TUNNEL

frequency 30

Just allow a few minutes for the IP SLA probe to run and collect statistics. You can also add monitoring/graph in an NMS for viewing historical data.

ASR1#show ip sla statistics 100 details

IPSLAs Latest Operation Statistics

IPSLA operation id: 100

Type of operation: udp-jitter

Latest RTT: 19 milliseconds

Latest operation start time: 12:56:08 UTC Thu Mar 17 2022

Latest operation return code: OK

Over thresholds occurred: FALSE

RTT Values:

Number Of RTT: 999 RTT Min/Avg/Max: 17/19/29 milliseconds

Latency one-way time:

Number of Latency one-way Samples: 999

Source to Destination Latency one way Min/Avg/Max: 9/10/19 milliseconds

Destination to Source Latency one way Min/Avg/Max: 8/9/18 milliseconds

Source to Destination Latency one way Sum/Sum2: 10565/113077

Destination to Source Latency one way Sum/Sum2: 9150/84308

Jitter Time:

Number of SD Jitter Samples: 997

Number of DS Jitter Samples: 998

Source to Destination Jitter Min/Avg/Max: 0/1/9 milliseconds

Destination to Source Jitter Min/Avg/Max: 0/1/9 milliseconds

Source to destination positive jitter Min/Avg/Max: 1/1/9 milliseconds

Source to destination positive jitter Number/Sum/Sum2: 286/475/1329

Source to destination negative jitter Min/Avg/Max: 1/1/9 milliseconds

Source to destination negative jitter Number/Sum/Sum2: 293/478/1288

Destination to Source positive jitter Min/Avg/Max: 1/1/7 milliseconds

Destination to Source positive jitter Number/Sum/Sum2: 180/229/425

Destination to Source negative jitter Min/Avg/Max: 1/1/9 milliseconds

Destination to Source negative jitter Number/Sum/Sum2: 181/230/468

Interarrival jitterout: 0 Interarrival jitterin: 0

Jitter AVG: 1

Over Threshold:

Number Of RTT Over Threshold: 0 (0%)

Packet Loss Values:

Loss Source to Destination: 1

Source to Destination Loss Periods Number: 1

Source to Destination Loss Period Length Min/Max: 1/1

Source to Destination Inter Loss Period Length Min/Max: 205/794

Loss Destination to Source: 0

Destination to Source Loss Periods Number: 0

Destination to Source Loss Period Length Min/Max: 0/0

Destination to Source Inter Loss Period Length Min/Max: 0/0

Out Of Sequence: 0 Tail Drop: 0 Packet Late Arrival: 0

Packet Skipped: 0

Voice Score Values:

Calculated Planning Impairment Factor (ICPIF): 11

MOS score: 4.06

Number of successes: 6

Number of failures: 0

Operation time to live: Forever

Operational state of entry: Active

Last time this entry was reset: Never

Below are the graphs for network latency, jitter and packet loss taken from NMS.

Saturday, March 5, 2022

Troubleshoot Smart License Call Home in Cisco 4000 ISR

Here's a nice link in troubleshooting the Smart Call Home (SCH) feature in a Cisco device running IOS-XE. I was enabling the Smart License in a Cisco 4K ISR but had an issue with Call Home. I've checked the call-home config and the portal reachability from the router were fine.

R1#show call-home smart-licensing

Current smart-licensing transport settings:

Smart-license messages: enabled

Profile: CiscoTAC-1 (status: ACTIVE)

Destination URL(s): https://tools.cisco.com/its/service/oddce/services/DDCEService

R1#ping tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 198/202/204 ms

I re-applied the Smart License token on the router using the license smart register command (with force keyword) in privilege mode and enable terminal monitor to observe the SCH registration details/error.

R1#terminal monitor

R1#license smart register idtoken <TOKEN STRING> force

Registration process is in progress. Use the 'show license status' command to check the progress and result

Feb 16 01:30:26.300 UTC: %SYS-2-PRIVCFG_ENCRYPT: Successfully encrypted private config file

Feb 16 01:30:20.642 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by crypto-engines

Feb 16 01:30:20.719 UTC: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAMh license status

Feb 16 01:31:17.004 UTC: %CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to: https://tools.cisco.com/its/service/oddce/services/DDCEService (ERR 205 : Request Aborted)

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-AGENT_REG_FAILED: Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message.

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message.

The SCH registration is stuck in REGISTRATION IN PROGRESS and the Smart License Status is still in EVAL MODE. It also mentioned failure is due to a failed Call Home HTTP message

R1#show license status

Smart Licensing is ENABLED

Utility:

Status: DISABLED

Data Privacy:

Sending Hostname: yes

Callhome hostname privacy: DISABLED

Smart Licensing hostname privacy: DISABLED

Version privacy: DISABLED

Transport:

Type: Callhome

Registration:

Status: REGISTERING - REGISTRATION IN PROGRESS

Export-Controlled Functionality: NOT ALLOWED

Initial Registration: FAILED on Feb 16 01:13:35 2022 UTC

Failure reason: Fail to send out Call Home HTTP message.

Next Registration Attempt: Feb 16 01:30:18 2022 UTC

License Authorization:

Status: EVAL MODE

Evaluation Period Remaining: 81 days, 14 hours, 3 minutes, 5 seconds

License Conversion:

Automatic Conversion Enabled: False

Status: Not started

Export Authorization Key:

Features Authorized:

<none>

I tried various "work around" commands but none solved the SCH issue.

R1(config)#ip host tools.cisco.com 173.37.145.8

R1(config)#ip domain lookup source-interface GigabitEthernet0/0/0

R1(config)#ip http client source-interface GigabitEthernet0/0/0

R1(config)#no ip name-server 8.8.8.8

R1(config)#ip name-server 4.2.2.2

R1(config)#call-home

R1(cfg-call-home)#profile "CiscoTAC-1"

R1(cfg-call-home-profile)#destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

R1(cfg-call-home-profile)#

Feb 16 01:40:00.315 UTC: %CALL_HOME-4-HTTP_ADDRESS_NOT_SUPPORTED: Http will be or has been disabled on Smart Call Home Server, please change the address http://tools.cisco.com/its/service/oddce/services/DDCEService to https address for profile CiscoTAC-1. Otherwise, call-home will fail to send messages

R1#show run | section call-home

service call-home

call-home

! If contact email address in call-home is configured as sch-smart-licensing@cisco.com

! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.

contact-email-addr sch-smart-licensing@cisco.com

profile "CiscoTAC-1"

active

destination transport-method http

no destination transport-method email

destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

I tried to use the telnet <FQDN> 443 command and noticed the SCH portal/hostname tried to resolve using an IPv6 address. So I "hard code" the SCH DNS resolution to an IPv4 address instead using http resolve-hostname ipv4-first call-home subcommand.

R1#telnet tools.cisco.com 443 /source-interface GigabitEthernet0/0/0

Trying 2001:420:1101:5::A, 443 ...

% Destination unreachable; gateway or host down

R1(config)#call-home

R1(cfg-call-home)#http ?

resolve-hostname Specify the IP version to resolve server hostname

secure Specify secure settings for http transport method

R1(cfg-call-home)#http resolve-hostname ?

ipv4-first ipv4 first

R1(cfg-call-home)#http resolve-hostname ipv4-first

I removed the "work around" commands and I forced the SCH registration again. This time the SCH finally worked.

R1#license smart register idtoken <TOKEN STRING> force

Registration process is in progress. Use the 'show license status' command to check the progress and result

Feb 16 02:25:42.726 UTC: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage

Feb 16 02:25:43.973 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine

Feb 16 02:25:44.049 UTC: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

Feb 16 02:25:51.895 UTC: %PKI-6-TRUSTPOOL_DOWNLOAD_SUCCESS: Trustpool Download is successful

Feb 16 02:26:20.747 UTC: %CALL_HOME-6-SCH_REGISTRATION_IN_PROGRESS: SCH device registration is in progress. Call-home will poll SCH server for registration result. You can also check SCH registration status with "call-home request registration-info" under EXEC mode.

Feb 16 02:26:20.748 UTC: %SMART_LIC-5-COMM_RESTORED: Communications with the Cisco Smart Software Manager (CSSM) restored

Feb 16 02:26:20.930 UTC: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is allowed

Feb 16 02:26:20.931 UTC: %SMART_LIC-6-AGENT_REG_SUCCESS: Smart Agent for Licensing Registration successful. udi PID:ISR4321/K9,SN:FLM2451ABCD

Feb 16 02:26:33.004 UTC: %SMART_LIC-5-IN_COMPLIANCE: All entitlements and licenses in use on this device are authorized

Feb 16 02:26:33.006 UTC: %SMART_LIC-5-END_POINT_RESET: End Point list reset

Feb 16 02:26:33.006 UTC: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:ISR4321/K9,SN:FLM2451ABCD

R1#show license summary

Smart Licensing is ENABLED

Registration:

Status: REGISTERED

Smart Account: <MY COMPANY>

Virtual Account: <MY VIRTUAL ACCOUNT>

Export-Controlled Functionality: ALLOWED

Last Renewal Attempt: None

Next Renewal Attempt: Aug 15 02:31:59 2022 UTC

License Authorization:

Status: AUTHORIZED

Last Communication Attempt: SUCCEEDED

Next Communication Attempt: Mar 18 02:32:13 2022 UTC

License Usage:

License Entitlement tag Count Status

-----------------------------------------------------------------------------

Boost Performance fo... (ISR_4321_BOOST) 1 AUTHORIZED