Saturday, March 5, 2022

Troubleshoot Smart License Call Home in Cisco 4000 ISR

Here's a nice link in troubleshooting the Smart Call Home (SCH) feature in a Cisco device running IOS-XE. I was enabling the Smart License in a Cisco 4K ISR but had an issue with Call Home. I've checked the call-home config and the portal reachability from the router were fine.

R1#show call-home smart-licensing

Current smart-licensing transport settings:

 Smart-license messages: enabled

 Profile: CiscoTAC-1 (status: ACTIVE)

 Destination  URL(s):  https://tools.cisco.com/its/service/oddce/services/DDCEService

 

R1#ping tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 198/202/204 ms

 

 

I re-applied the Smart License token on the router using the license smart register command (with force keyword) in privilege mode and enable terminal monitor to observe the SCH registration details/error.

 

R1#terminal monitor

 

R1#license smart register idtoken <TOKEN STRING> force          

Registration process is in progress. Use the 'show license status' command to check the progress and result

 

Feb 16 01:30:26.300 UTC: %SYS-2-PRIVCFG_ENCRYPT: Successfully encrypted private config file

Feb 16 01:30:20.642 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by crypto-engines

Feb 16 01:30:20.719 UTC: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAMh license status

Feb 16 01:31:17.004 UTC: %CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to: https://tools.cisco.com/its/service/oddce/services/DDCEService (ERR 205 : Request Aborted)

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-AGENT_REG_FAILED: Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message.

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message.

 

 

The SCH registration is stuck in REGISTRATION IN PROGRESS and the Smart License Status is still in EVAL MODE. It also mentioned failure is due to a failed Call Home HTTP message

 

R1#show license status

Smart Licensing is ENABLED

 

Utility:

  Status: DISABLED

 

Data Privacy:

  Sending Hostname: yes

    Callhome hostname privacy: DISABLED

    Smart Licensing hostname privacy: DISABLED

  Version privacy: DISABLED

 

Transport:

  Type: Callhome

 

Registration:

  Status: REGISTERING - REGISTRATION IN PROGRESS

  Export-Controlled Functionality: NOT ALLOWED

  Initial Registration: FAILED on Feb 16 01:13:35 2022 UTC

    Failure reason: Fail to send out Call Home HTTP message.

  Next Registration Attempt: Feb 16 01:30:18 2022 UTC

 

License Authorization:

  Status: EVAL MODE

  Evaluation Period Remaining: 81 days, 14 hours, 3 minutes, 5 seconds

 

License Conversion:

  Automatic Conversion Enabled: False

  Status: Not started

 

Export Authorization Key:

  Features Authorized:

    <none>

 

 

I tried various "work around" commands but none solved the SCH issue.

 

R1(config)#ip host tools.cisco.com 173.37.145.8

 

R1(config)#ip domain lookup source-interface GigabitEthernet0/0/0

 

R1(config)#ip http client source-interface GigabitEthernet0/0/0

 

R1(config)#no ip name-server 8.8.8.8

R1(config)#ip name-server 4.2.2.2

 

R1(config)#call-home

R1(cfg-call-home)#profile "CiscoTAC-1"

R1(cfg-call-home-profile)#destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

R1(cfg-call-home-profile)#

Feb 16 01:40:00.315 UTC: %CALL_HOME-4-HTTP_ADDRESS_NOT_SUPPORTED: Http will be or has been disabled on Smart Call Home Server, please change the address http://tools.cisco.com/its/service/oddce/services/DDCEService to https address for profile CiscoTAC-1. Otherwise, call-home will fail to send messages

 

R1#show run | section call-home

service call-home

call-home

 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com

 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.

 contact-email-addr sch-smart-licensing@cisco.com

 profile "CiscoTAC-1"

  active

  destination transport-method http

  no destination transport-method email

  destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

 

 

I tried to use the telnet <FQDN> 443 command and noticed the SCH portal/hostname tried to resolve using an IPv6 address. So I "hard code" the SCH DNS resolution to an IPv4 address instead using http resolve-hostname ipv4-first call-home subcommand.

 

R1#telnet tools.cisco.com 443 /source-interface GigabitEthernet0/0/0

Trying 2001:420:1101:5::A, 443 ...

% Destination unreachable; gateway or host down

 

 

R1(config)#call-home

R1(cfg-call-home)#http ?

  resolve-hostname  Specify the IP version to resolve server hostname

  secure            Specify secure settings for http transport method

 

R1(cfg-call-home)#http resolve-hostname ?

  ipv4-first  ipv4 first

 

R1(cfg-call-home)#http resolve-hostname ipv4-first

 

I removed the "work around" commands and I forced the SCH registration again. This time the SCH finally worked.

 

R1#license smart register idtoken <TOKEN STRING> force

Registration process is in progress. Use the 'show license status' command to check the progress and result

 

Feb 16 02:25:42.726 UTC: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage

Feb 16 02:25:43.973 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine

Feb 16 02:25:44.049 UTC: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration

Feb 16 02:25:51.895 UTC: %PKI-6-TRUSTPOOL_DOWNLOAD_SUCCESS: Trustpool Download is successful

Feb 16 02:26:20.747 UTC: %CALL_HOME-6-SCH_REGISTRATION_IN_PROGRESS: SCH device registration is in progress. Call-home will poll SCH server for registration result. You can also check SCH registration status with "call-home request registration-info" under EXEC mode.

Feb 16 02:26:20.748 UTC: %SMART_LIC-5-COMM_RESTORED: Communications with the Cisco Smart Software Manager (CSSM) restored

Feb 16 02:26:20.930 UTC: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is allowed

Feb 16 02:26:20.931 UTC: %SMART_LIC-6-AGENT_REG_SUCCESS: Smart Agent for Licensing Registration successful. udi PID:ISR4321/K9,SN:FLM2451ABCD

Feb 16 02:26:33.004 UTC: %SMART_LIC-5-IN_COMPLIANCE: All entitlements and licenses in use on this device are authorized

Feb 16 02:26:33.006 UTC: %SMART_LIC-5-END_POINT_RESET: End Point list reset

Feb 16 02:26:33.006 UTC: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:ISR4321/K9,SN:FLM2451ABCD

 

 

R1#show license summary    

Smart Licensing is ENABLED

 

Registration:

  Status: REGISTERED

  Smart Account: <MY COMPANY>

  Virtual Account: <MY VIRTUAL ACCOUNT>

  Export-Controlled Functionality: ALLOWED

  Last Renewal Attempt: None

  Next Renewal Attempt: Aug 15 02:31:59 2022 UTC

 

License Authorization:

  Status: AUTHORIZED

  Last Communication Attempt: SUCCEEDED

  Next Communication Attempt: Mar 18 02:32:13 2022 UTC

 

License Usage:

  License                 Entitlement tag               Count Status

  -----------------------------------------------------------------------------

  Boost Performance fo... (ISR_4321_BOOST)                  1 AUTHORIZED