Troubleshoot Smart License Call Home in Cisco 4000 ISR

Here's a nice link in troubleshooting the Smart Call Home (SCH) feature in a Cisco device running IOS-XE. I was enabling the Smart License in a Cisco 4K ISR but had an issue with Call Home. I've checked the call-home config and the portal reachability from the router were fine.

R1#show call-home smart-licensing

Current smart-licensing transport settings:

 Smart-license messages: enabled

 Profile: CiscoTAC-1 (status: ACTIVE)

 Destination  URL(s):  https://tools.cisco.com/its/service/oddce/services/DDCEService

 

R1#ping tools.cisco.com
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 198/202/204 ms

 

 

I re-applied the Smart License token on the router using the license smart register command (with force keyword) in privilege mode and enable terminal monitor to observe the SCH registration details/error.

 

R1#terminal monitor

 

R1#license smart register idtoken <TOKEN STRING> force          

Registration process is in progress. Use the 'show license status' command to check the progress and result

 

Feb 16 01:30:26.300 UTC: %SYS-2-PRIVCFG_ENCRYPT: Successfully encrypted private config file

Feb 16 01:30:20.642 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by crypto-engines

Feb 16 01:30:20.719 UTC: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAMh license status

Feb 16 01:31:17.004 UTC: %CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to: https://tools.cisco.com/its/service/oddce/services/DDCEService (ERR 205 : Request Aborted)

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-AGENT_REG_FAILED: Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message.

Feb 16 01:31:17.005 UTC: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message.

 

 

The SCH registration is stuck in REGISTRATION IN PROGRESS and the Smart License Status is still in EVAL MODE. It also mentioned failure is due to a failed Call Home HTTP message

 

R1#show license status

Smart Licensing is ENABLED

 

Utility:

  Status: DISABLED

 

Data Privacy:

  Sending Hostname: yes

    Callhome hostname privacy: DISABLED

    Smart Licensing hostname privacy: DISABLED

  Version privacy: DISABLED

 

Transport:

  Type: Callhome

 

Registration:

  Status: REGISTERING - REGISTRATION IN PROGRESS

  Export-Controlled Functionality: NOT ALLOWED

  Initial Registration: FAILED on Feb 16 01:13:35 2022 UTC

    Failure reason: Fail to send out Call Home HTTP message.

  Next Registration Attempt: Feb 16 01:30:18 2022 UTC

 

License Authorization:

  Status: EVAL MODE

  Evaluation Period Remaining: 81 days, 14 hours, 3 minutes, 5 seconds

 

License Conversion:

  Automatic Conversion Enabled: False

  Status: Not started

 

Export Authorization Key:

  Features Authorized:

    <none>

 

 

I tried various "work around" commands but none solved the SCH issue.

 

R1(config)#ip host tools.cisco.com 173.37.145.8

 

R1(config)#ip domain lookup source-interface GigabitEthernet0/0/0

 

R1(config)#ip http client source-interface GigabitEthernet0/0/0

 

R1(config)#no ip name-server 8.8.8.8

R1(config)#ip name-server 4.2.2.2

 

R1(config)#call-home

R1(cfg-call-home)#profile "CiscoTAC-1"

R1(cfg-call-home-profile)#destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

R1(cfg-call-home-profile)#

Feb 16 01:40:00.315 UTC: %CALL_HOME-4-HTTP_ADDRESS_NOT_SUPPORTED: Http will be or has been disabled on Smart Call Home Server, please change the address http://tools.cisco.com/its/service/oddce/services/DDCEService to https address for profile CiscoTAC-1. Otherwise, call-home will fail to send messages

 

R1#show run | section call-home

service call-home

call-home

 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com

 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.

 contact-email-addr sch-smart-licensing@cisco.com

 profile "CiscoTAC-1"

  active

  destination transport-method http

  no destination transport-method email

  destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService

 

 

I tried to use the telnet <FQDN> 443 command and noticed the SCH portal/hostname tried to resolve using an IPv6 address. So I "hard code" the SCH DNS resolution to an IPv4 address instead using http resolve-hostname ipv4-first call-home subcommand.

 

R1#telnet tools.cisco.com 443 /source-interface GigabitEthernet0/0/0

Trying 2001:420:1101:5::A, 443 ...

% Destination unreachable; gateway or host down

 

 

R1(config)#call-home

R1(cfg-call-home)#http ?

  resolve-hostname  Specify the IP version to resolve server hostname

  secure            Specify secure settings for http transport method

 

R1(cfg-call-home)#http resolve-hostname ?

  ipv4-first  ipv4 first

 

R1(cfg-call-home)#http resolve-hostname ipv4-first

 

I removed the "work around" commands and I forced the SCH registration again. This time the SCH finally worked.

 

R1#license smart register idtoken <TOKEN STRING> force

Registration process is in progress. Use the 'show license status' command to check the progress and result

 

Feb 16 02:25:42.726 UTC: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage

Feb 16 02:25:43.973 UTC: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine

Feb 16 02:25:44.049 UTC: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration

Feb 16 02:25:51.895 UTC: %PKI-6-TRUSTPOOL_DOWNLOAD_SUCCESS: Trustpool Download is successful

Feb 16 02:26:20.747 UTC: %CALL_HOME-6-SCH_REGISTRATION_IN_PROGRESS: SCH device registration is in progress. Call-home will poll SCH server for registration result. You can also check SCH registration status with "call-home request registration-info" under EXEC mode.

Feb 16 02:26:20.748 UTC: %SMART_LIC-5-COMM_RESTORED: Communications with the Cisco Smart Software Manager (CSSM) restored

Feb 16 02:26:20.930 UTC: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is allowed

Feb 16 02:26:20.931 UTC: %SMART_LIC-6-AGENT_REG_SUCCESS: Smart Agent for Licensing Registration successful. udi PID:ISR4321/K9,SN:FLM2451ABCD

Feb 16 02:26:33.004 UTC: %SMART_LIC-5-IN_COMPLIANCE: All entitlements and licenses in use on this device are authorized

Feb 16 02:26:33.006 UTC: %SMART_LIC-5-END_POINT_RESET: End Point list reset

Feb 16 02:26:33.006 UTC: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:ISR4321/K9,SN:FLM2451ABCD

 

 

R1#show license summary    

Smart Licensing is ENABLED

 

Registration:

  Status: REGISTERED

  Smart Account: <MY COMPANY>

  Virtual Account: <MY VIRTUAL ACCOUNT>

  Export-Controlled Functionality: ALLOWED

  Last Renewal Attempt: None

  Next Renewal Attempt: Aug 15 02:31:59 2022 UTC

 

License Authorization:

  Status: AUTHORIZED

  Last Communication Attempt: SUCCEEDED

  Next Communication Attempt: Mar 18 02:32:13 2022 UTC

 

License Usage:

  License                 Entitlement tag               Count Status

  -----------------------------------------------------------------------------

  Boost Performance fo... (ISR_4321_BOOST)                  1 AUTHORIZED

Exploring AWS Management Console and Mobile App

There are several AWS Support Plan: Basic, Developer, Business and Enterprise. The Basic Support is included for all AWS customers and includes:

• Customer Service and Communities - 24x7 access to customer service, documentation, whitepapers, and AWS re:Post

• AWS Trusted Advisor - Access to core Trusted Advisor checks and guidance to provision your resources following best practices to increase performance and improve security.

• AWS Personal Health Dashboard - A personalized view of the health of AWS services, and alerts when your resources are impacted.

 

The AWS Management Console is the landing page or dashboard once you login to AWS Management Console. It will show the recent AWS services visited and links to various AWS services, tutorials and training videos.

 

There are few ways to search and launch the AWS services. The first one is to use the search bar (magnifying glass icon).

 

In this case I just typed S3 and it showed the services related to S3 (Simple Storage Service).

 

If you're unsure of the type of service, you can click/expand Services (top left) > Storage > S3.

You can pin/bookmark AWS services that are frequently used.

 

In this case, I hover to EC2 and S3 > click the star icon and they will show under Favorites.

 

You can view Alerts and other notifications by clicking on the bell icon (beside the account name).

 

Notice there's a blue dot. Click Open issues to view more information.

 

Go to Event log to view historic event logs or alerts.

 

Click EC2 operational issue with a Status of Open to view more details.

 

Click the Affected resources tab.

Click the second EC2 operational issue (with Status: Closed) to view more details.

Click Amazon CloudWatch Events. It will open a new browser for CloudWatch dashboard.

Click the account name (johnlagura-aws) to view billing information.

Click on the AWS Region to search/deploy AWS services that's operating closest to your customers.

 

In this case I chose Asia Pacific (Singapore) ap-southeast-1.

 

There are also AWS services that are considered Global such as AWS Support Center, Identity and Access Management (IAM), Route 53 (DNS), etc.

Click/expand Support to find various ways to get help in AWS. This is dependent on your support plan.

 

You can raise a AWS support case, access Knowledge Center, Documentation and Forums.

 

Click Create Case (orange button) to raise either an Account and billing support, Service limit increase and Technical support Case.

To monitor the performance of your EC2 Instance (virtual machine), search for EC2 or go the Services > EC2 > Instances.

 

Select the specific EC2 Instance > go to Monitoring tab. You can select the historical trend: 1h, 3h, 12h, etc.

 

You can also download and use the AWS Console mobile app. Just type/search for aws console (App store in this case). It has a limited function but it's a great tool especially when you're outside and didn't bring a laptop.

You can view the CloudWatch alarms and AWS health dashboard.

Click/expand the three lines on the left to get Help and Support, Documentation, Legal Notices and Sign out.

Click/expand the AWS Region to find AWS services deployed in a specific Region.

Under Services > select EC2.

I clicked on Instances. Notice a count of 1 Instance and 1 Running.

This is the Linux EC2 instance I previously deployed. Click on the specific instance to view more details.

You can view the Status, DNS and IP address of the EC2 Instance. Click Actions (orange button)

You can Start, Reboot, Stop and Terminate an EC2 Instance using the app.

Creating an AWS EC2 Instance (Virtual Machine)

You can find the AWS Shared Responsibility Model on this link.

AWS responsibility “Security of the Cloud” - AWS is responsible for protecting the infrastructure that runs all of the services offered in the AWS Cloud. This infrastructure is composed of the hardware, software, networking, and facilities that run AWS Cloud services.

Customer responsibility “Security in the Cloud” - Customer responsibility will be determined by the AWS Cloud services that a customer selects. This determines the amount of configuration work the customer must perform as part of their security responsibilities. 

 

In the AWS Management Console, select the region where the EC2 instance will be deployed.

 

You choose the AWS Region based on the proximity of the users who will frequently access the resource and data compliance. In this case I chose Asia Pacific (Singapore): ap-southeast-1.

You can either create a new EC2 instance under Services > Compute > EC2 or type EC2 in the search bar.

Click Launch instance (orange button) > Launch instance.

There are seven steps in order to create an EC2 instance. The first step is to Choose an Amazon Machine Image (AMI).

 

There are various Linux, MacOS and Windows AMI to choose from. In this case I chose Amazon Linux  2 AMI which is Free tier eligible.

 

The second step is to choose an Instance Type which is the combination of CPU, memory, storage, etc.

 

In this case I chose t2.micro which is 1 vCPU, 1 GB memory and it's Free tier eligible.

 

You can either select Review and Launch to finish or click Next to continue with the EC2 Instance customization.

 

The third step is to configure the Instance Details. I just left it in default settings.

The fourth step is to Add Storage. I just left the default size of 8 GB and Volume Type: General Purpose SSD (gp2). Click Next.

In fifth step, you can optionally add a Tag. Tags are helpful in auditing and billing purposes.

 

In this case I added a Tag Key: Name and Value: test-ec2-apac. Click Next.

 

The sixth step is to configure a Security Group which is the virtual firewall rule set.

 

In this case I chose: Create a new security group > type Security group name: ec2-security-group.

 

Click Add Rule > select Type: SSH > select Protocol: TCP > Port Range: 22 > type Source: ::/0 (all IPv6 address) > click Review and Launch.

The seventh and last step is to Review the summary of the EC2 settings. Click Launch once finished.

A message prompt will ask to either Choose an existing key pair, Create a new key pair or Proceed without a key pair.

 

A key pair is used when accessing the EC2 instance outside the AWS cloud/public Internet.

 

 In this case I chose Create a new key pair > select Key pair type: RSA > type Key pair name: ec2-key-pair > click Download Key Pair.

Save the key pair file (.pem file) > then click Launch Instances.

Click on the EC2 instance ID (hyperlink) to view the status.

The hyperlink will redirect you to the EC2 Instance page. Notice the Instance state: Running.

 

The EC2 instance deployment will take a few seconds or minutes depending on the EC2 Instance type/settings.

 

Select the new EC2 instance to view its details.

 

Click Connect to connect to the EC2 instance within the AWS Management Console.

 

Click Connect (orange button).

 

It will launch a new web browser and open a command terminal.

Use the commands below in the Linux command prompt to run an HTTP web server and display a simple text.

sudo su

yum install -y httpd.x86_64

systemctl start httpd.service

systemctl enable httpd.service

echo "Hello World" > /var/www/html/index.html

 

To test the web server script, copy the Public IPv4 DNS to clipboard (click the page icon).

ec2-3-0-145-251.ap-southeast-1.compute.amazonaws.com

You could either use the DNS or public IP in the web browser.

To SSH the Linux EC2 Instance from outside/public Internet, you'll need to add/load the key pair created earlier.

 

Open PuTTY Key Generator > click Load.

 

 

 

Select All Files (at the bottom) > select the key pair: ec2-key-pair.pem > click Open.

Click OK to continue > click Save private key.

Click Yes to save the key without a passphrase to protect it.

 

Save the new private key file (.ppk).

 

Select the specific EC2 Instance > click Connect > go to SSH client tab.

 

Copy the DNS string to clipboard and paste in PuTTY SSH client.

 

 

 Go to Connection > Data > type Auto-login username: ec2-user

Go to SSH > Auth > browse the .ppk key pair file > Open

It will automatically login to the Linux EC2 command prompt.

You can either Stop, Reboot or Terminate the EC2 Instance. Just select the specific EC2 Instance > click Instance state > choose the Stop.

 

In this case I chose to Stop the instance so it will not incur cost and can be used in the future. Choose Terminate to completely delete the EC2 Instance and its data.

 

Click Stop.

Notice the Instance State changed to Stopped.