Friday, October 6, 2023

Cisco NCS 540 IOS-XR Software Upgrade

The NCS 540 IOS-XR software (.tar) is usually around 2 GB so leave the file transfer running for several minutes. I usually perform this during night time to avoid bandwidth congestion or disruption.

Transfer the new IOS-XR using FTP, TFTP or SCP. You can view the .tar file using dir harddisk: command.

RP/0/RP0/CPU0:NCS540#copy ftp://ftp:ftp123@172.16.5.3 harddisk: vrf Mgmt-intf

Wed Sep 27 14:44:03.768 UTC

Source filename [/ftp:]?NCS540-iosxr-7.4.2.tar

Destination filename [/harddisk:/NCS540-iosxr-7.4.2.tar]?  <PRESS ENTER>

Accessing ftp://ftpuser:*@172.16.5.3;Mgmt-intf/NCS540-iosxr-7.4.2.tar

CCCCCCCCCCCCCCC

 

<OUTPUT TRUNCATED>


CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

1770946560 bytes copied in  21784 sec (    81295)bytes/sec

 

Copy operation success


RP/0/RP0/CPU0:NCS540#dir harddisk:

Fri Sep 29 08:33:50.778 UTC

 

Directory of harddisk:

 522241 drwxr-xr-x. 2       4096 Sep 29 05:49 .sppdc

1305602 drw-rw-r--. 2       4096 Jul 12 18:02 rdsfs_log

     14 -rw-r--r--. 1       7714 Sep 27 05:51 bcm_console_log_output.txt

 261121 drwxr-xr-x. 2       4096 Jul 12 17:38 nvram

1175041 drwxr-xr-x. 3       4096 Jul 12 17:40 ztp

     11 drwx------. 2      16384 Jul 12 17:31 lost+found

 391681 drwxr-xr-x. 3       4096 Jul 12 17:40 mirror

1044481 drwxr-xr-x. 2       4096 Jul 12 17:40 ipodwdm_log

     13 -rw-r--r--. 1          0 Jul 12 17:38 zl30363_trace_file.log

 913923 drwxr-xr-x. 5       4096 Sep 27 06:50 cisco_support

     12 -rw-r--r--. 1      65770 Jul 12 18:02 nvgen_bkup.log

     15 -rw-r--r--. 1        182 Sep 29 07:55 feature_list

 783361 drwxr-xr-x. 2       4096 Jul 12 17:38 shutdown

1305601 drwxr-xr-x. 2       4096 Jul 12 17:38 tftpboot

 522242 drwxr-xr-x. 3       4096 Jul 12 17:42 pam

1044482 drwxr-xr-x. 2       4096 Jul 12 17:42 showtech

 913921 drwxr-xr-x. 2       4096 Jul 12 17:40 .sppdc_new

 130561 drwxr-xr-x. 6       4096 Jul 12 17:40 xr

     16 -rwxr--r--. 1 1770946560 Sep 27 20:47 NCS540-iosxr-7.4.2.tar

 261122 drwxr-xr-x. 2       4096 Jul 12 17:42 dumper

 783362 drwx------. 3       4096 Sep 27 05:50 ima

 

23893884 kbytes total (20877188 kbytes free)


Use the install add source /harddisk:/ <.tar> privilege exec command to extract and install the RPM packages. Note there's a space before .tar file and the operation id is usually 4.

RP/0/RP0/CPU0::NCS540#install add source /harddisk:/ NCS540-iosxr-7.4.2.tar

Sat Sep 30 03:09:13.249 UTC

Sep 30 03:09:15 Install operation 4 started by root:

 install add source /harddisk:/ NCS540-iosxr-7.4.2.tar nooptim

Sep 30 03:09:21 Install operation will continue in the background

 

 

You can view the installation progress using the show install request command.


RP/0/RP0/CPU0::NCS540#show install request

Sat Sep 30 03:09:55.073 UTC

User root, Op Id 4

install add 

NCS540-iosxr-7.4.2.tar

 

The install add operation 4 is 60% complete

Add validate request sent to sysadmin

 

 

RP/0/RP0/CPU0:NCS540#Sep 30 03:12:19 Install operation 4 finished successfully

 

 

It took around 2 mins for the RPM package installation to finish. It's recommended to monitor the upgrade process via the console access.

 

 

RP/0/RP0/CPU0::NCS540#show install request                                

Sat Sep 30 03:12:34.287 UTC

No install operation in progress

 

 

Use the install activate id <ID> privilege exec command to activate the RPM packages. Type yes to proceed.

 

RP/0/RP0/CPU0::NCS540#install activate ?

  WORD             package

  id               The id of the install add operation containing package(s) to activate(cisco-support)

  issu             ISSU operation(cisco-support)

  nooptim          Execute install operation in traditional mode(cisco-support)

  noprompt         do not prompt user.continue assuming default yes(cisco-support)

  process-restart  process-restart(cisco-support)

  reload           reload(cisco-support)

  synchronous      Execute install operation in synchronous mode(cisco-support)

  <cr>            

RP/0/RP0/CPU0::NCS540#install activate id ?

  <1-4294967295>  Activate the package(s) specified in this install id

RP/0/RP0/CPU0::NCS540#install activate id 4

Sat Sep 30 03:13:48.124 UTC

Sep 30 03:13:50 Install operation 5 started by root:

  install activate id 4

Sep 30 03:13:50 Package list:

Sep 30 03:13:50     ncs540-isis-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-mpls-te-rsvp-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-mpls-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-ospf-2.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-mini-x-7.4.2

Sep 30 03:13:50     ncs540-mcast-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-eigrp-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-mgbl-1.0.0.0-r742.x86_64

Sep 30 03:13:50     ncs540-li-1.0.0.0-r742.x86_64

 

 

This install operation will reload the system, continue?

 [yes/no]:[yes] yes

Sep 30 03:15:03 Install operation will continue in the background

 

 

RP/0/RP0/CPU0:NCS540#show install request

Sat Sep 30 03:15:29.157 UTC

User root, Op Id 5

install activate 

ncs540-isis-1.0.0.0-r742.x86_64

ncs540-mpls-te-rsvp-1.0.0.0-r742.x86_64

ncs540-mpls-1.0.0.0-r742.x86_64

ncs540-ospf-2.0.0.0-r742.x86_64

ncs540-mini-x-7.4.2

ncs540-mcast-1.0.0.0-r742.x86_64

ncs540-eigrp-1.0.0.0-r742.x86_64

ncs540-mgbl-1.0.0.0-r742.x86_64

ncs540-li-1.0.0.0-r742.x86_64

 

The install prepare operation 5 is 40% complete

Processing the ISO

 

 

RP/0/RP0/CPU0::NCS540#show install request

Sat Sep 30 03:18:16.789 UTC

User root, Op Id 5

install activate 

ncs540-isis-1.0.0.0-r742.x86_64

ncs540-mpls-te-rsvp-1.0.0.0-r742.x86_64

ncs540-mpls-1.0.0.0-r742.x86_64

ncs540-ospf-2.0.0.0-r742.x86_64

ncs540-mini-x-7.4.2

ncs540-mcast-1.0.0.0-r742.x86_64

ncs540-eigrp-1.0.0.0-r742.x86_64

ncs540-mgbl-1.0.0.0-r742.x86_64

ncs540-li-1.0.0.0-r742.x86_64

 

The install prepare operation 5 is 40% complete

 

 install prepare operation 5 is in progress

Host preparation is in progress

 

Node(s)                 State                   %Completed              Action

--------------------------------------------------------------------------------

Node 0/RP0      :       In Progress                30%                  Host prepare in progress

 

 

RP/0/RP0/CPU0::NCS540#RP/0/RP0/CPU0:Sep 30 03:29:06.098 UTC: fpd-serv[302]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP0: 

 

 

RP/0/RP0/CPU0::NCS540#show install request

Sat Sep 30 03:29:26.665 UTC

User root, Op Id 5

install activate 

ncs540-isis-1.0.0.0-r742.x86_64

ncs540-mpls-te-rsvp-1.0.0.0-r742.x86_64

ncs540-mpls-1.0.0.0-r742.x86_64

ncs540-ospf-2.0.0.0-r742.x86_64

ncs540-mini-x-7.4.2

ncs540-mcast-1.0.0.0-r742.x86_64

ncs540-eigrp-1.0.0.0-r742.x86_64

ncs540-mgbl-1.0.0.0-r742.x86_64

ncs540-li-1.0.0.0-r742.x86_64

 

The install activate operation 5 is 5% complete

 

 install activate operation 5 is in progress

Host preparation is complete

Sysadmin preparation is complete

XR preparation is complete

Host activation is complete

Sysadmin activation is in progress

 

 

Node(s)                 State                   %Completed              Action

--------------------------------------------------------------------------------

Node 0/RP0      :       Completed                  100%                 Sysadmin activation is complete

 

 

RP/0/RP0/CPU0::NCS540#show install request

Sat Sep 30 03:30:45.730 UTC

User root, Op Id 5

install activate 

ncs540-isis-1.0.0.0-r742.x86_64

ncs540-mpls-te-rsvp-1.0.0.0-r742.x86_64

ncs540-mpls-1.0.0.0-r742.x86_64

ncs540-ospf-2.0.0.0-r742.x86_64

ncs540-mini-x-7.4.2

ncs540-mcast-1.0.0.0-r742.x86_64

ncs540-eigrp-1.0.0.0-r742.x86_64

ncs540-mgbl-1.0.0.0-r742.x86_64

ncs540-li-1.0.0.0-r742.x86_64

 

The install activate operation 5 is 20% complete

Updating XR master software profile

 

 

RP/0/RP0/CPU0::NCS540#RP/0/RP0/CPU0:Sep 30 03:31:05.728 UTC: sdr_instmgr[1293]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 5 finished successfully

RP/0/RP0/CPU0:Sep 30 03:31:06.125 UTC: sdr_instmgr[1293]: %INSTALL-INSTMGR-2-SYSTEM_RELOAD_INFO : The whole system will be reloaded to complete install operation 5

 


It took around 20 mins for the RPM install packages to finish. The NCS 540 router will automatically reload afterwards.

 

RP/0/RP0/CPU0:NCS540#LC/0/0/CPU0:Sep 30 03:31:58.195 UTC: ifmgr[332]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Down

LC/0/0/CPU0:Sep 30 03:31:58.196 UTC: ifmgr[332]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/21, changed state to Down

LC/0/0/CPU0:Sep 30 03:31:58.199 UTC: fia_driver[243]: %PLATFORM-DPA-2-RX_FAULT : Interface GigabitEthernet0/0/0/0, Detected Local Fault

LC/0/0/CPU0:Sep 30 03:31:58.203 UTC: fia_driver[243]: %PLATFORM-DPA-2-RX_FAULT : Interface GigabitEthernet0/0/0/21, Detected Local Fault

0/RP0/ADMIN0:Sep 30 03:32:04.663 UTC: shelf_mgr[2604]: %INFRA-SHELF_MGR-4-CARD_RELOAD : Reloading card 0/RP0 

0/RP0/ADMIN0:Sep 30 03:32:10.520 UTC: vm_manager[2637]: %INFRA-VM_MANAGER-4-INFO : Info: vm_manager brought down VM default-sdr--2 

 

Preparing system for backup. This may take a few minutes especially for large configurations.

        Status report: node0_RP0_CPU0: BACKUP INPROGRESS

        Status report: node0_RP0_CPU0: BACKUP HAS COMPLETED SUCCESSFULLY

[Done]

Stoping cgroup-init

Stopping OpenBSD Secure Shell server: sshdinitctl: Unknown instance:

Stopping system message bus: dbus.

Stopping random number generator daemon.

Stopping system log daemon...0

Stopping kernel log daemon...0

Stopping internet superserver: xinetd.

Stopping crond: OK

Stopping rpcbind daemon...

done.

Stopping S.M.A.R.T. daemon: smartd.

Stopping Lighttpd Web Server: stopped /usr/sbin/lighttpd (pid 3963)

lighttpd.

Stopping libvirtd daemon: [  OK  ]

Deconfiguring network interfaces... done.

Sending all processes the KILL signal...

Unmounting remote filesystems...

Deactivating swap...

Unmounting local filesystems...

mount: can't find /mnt/ram in /etc/fstab

Rebooting... [251250.066254] obfl_rb_notify_cb: reboot notifier called

[251250.072019]  writing reset reason 1, (null)

[251250.856170]  Requesting IOFPGA for system wide cold reset operation

[251251.606113] Succesfully wrote mtdoops at 163840 size 32768

[251251.612640] pstore: Successfully logged oops info. Size 32740

 

<OUTPUT TRUNCATED>

 

 

Connecting to 'default-sdr--1' console

ÿûÿûÿûÿýbootlogd: ioctl(/dev/pts/2, TIOCCONS): Device or resource busy

Running postinst /etc/rpm-postinsts/100-dnsmasq...

Running postinst /etc/rpm-postinsts/101-nfs-utils-client...

Running postinst /etc/rpm-postinsts/102-nfs-utils...

update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)

 Removing any system startup links for run-postinsts ...

  /etc/rcS.d/S99run-postinsts

Configuring network interfaces... done.

Starting system message bus: dbus.

Starting OpenBSD Secure Shell server: sshd

  generating ssh RSA key...

  generating ssh ECDSA key...

  generating ssh DSA key...

  generating ssh ED25519 key...

sshd start/running, process 2380

Starting rpcbind daemon...done.

Starting random number generator daemon.

Starting system log daemon...0

Starting kernel log daemon...0

tftpd-hpa disabled in /etc/default/tftpd-hpa

Starting internet superserver: xinetd.

Libvirt not initialized for container instance

Starting crond: OK

SIOCADDRT: File exists

osSignalTask: started

 

 

ios con0/RP0/CPU0 is now available

 

 

Press RETURN to get started.

 

RP/0/RP0/CPU0:Sep 30 03:36:02.641 UTC: fpd-serv[305]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :FPD-NEED-UPGRADE :DECLARE :0/RP0: 

0/RP0/ADMIN0:Sep 30 03:36:17.167 UTC: vm_manager[3160]: %INFRA-VM_MANAGER-4-INFO : Info: vm_manager started VM default-sdr--2 

 

 

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply third-party

authority to import, export, distribute or use encryption. Importers,

exporters, distributors and users are responsible for compliance with

U.S. and local country laws. By using this product you agree to comply

with applicable laws and regulations. If you are unable to comply with

U.S. and local laws, return this product immediately.

 

A summary of U.S. laws governing Cisco cryptographic products may be

found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

 

If you require further assistance please contact us by sending email to

export@cisco.com.

 

 

RP/0/RP0/CPU0:Sep 30 03:37:14.957 UTC: ifmgr[139]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Down

RP/0/RP0/CPU0:Sep 30 03:37:14.976 UTC: ifmgr[139]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Up

RP/0/RP0/CPU0:Sep 30 03:37:15.037 UTC: ifmgr[139]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Down

RP/0/RP0/CPU0:Sep 30 03:37:18.644 UTC: ifmgr[139]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/RP0/CPU0/0, changed state to Up

 

 

It took around 6 mins for the NCS 540 to boot up using the new IOS-XR code.

 

 

RP/0/RP0/CPU0::NCS540#show version

Sat Sep 30 03:38:35.240 UTC

Cisco IOS XR Software, Version 7.4.2

Copyright (c) 2013-2022 by Cisco Systems, Inc.

 

Build Information:

 Built By     : ingunawa

 Built On     : Wed Feb 16 05:24:50 PST 2022

 Built Host   : iox-lnx-025

 Workspace    : /auto/srcarchive15/prod/7.4.2/ncs540/ws

 Version      : 7.4.2

 Location     : /opt/cisco/XR/packages/

 Label        : 7.4.2

 

cisco NCS-540 () processor

System uptime is 3 minutes

 

 

RP/0/RP0/CPU0::NCS540#show install active

Sat Sep 30 03:39:03.701 UTC

Label : 7.4.2

 

Node 0/RP0/CPU0 [RP]

  Boot Partition: xr_lv7

  Active Packages: 9

        ncs540-xr-7.4.2 version=7.4.2 [Boot image]

        ncs540-eigrp-1.0.0.0-r742

        ncs540-isis-1.0.0.0-r742

        ncs540-li-1.0.0.0-r742

        ncs540-mgbl-1.0.0.0-r742

        ncs540-mpls-1.0.0.0-r742

        ncs540-mpls-te-rsvp-1.0.0.0-r742

        ncs540-ospf-2.0.0.0-r742

        ncs540-mcast-1.0.0.0-r742

 

Node 0/0/CPU0 [LC]

  Boot Partition: xr_lcp_lv8

  Active Packages: 9

        ncs540-xr-7.4.2 version=7.4.2 [Boot image]

        ncs540-eigrp-1.0.0.0-r742

        ncs540-isis-1.0.0.0-r742

        ncs540-li-1.0.0.0-r742

        ncs540-mgbl-1.0.0.0-r742

        ncs540-mpls-1.0.0.0-r742

        ncs540-mpls-te-rsvp-1.0.0.0-r742

        ncs540-ospf-2.0.0.0-r742

        ncs540-mcast-1.0.0.0-r742

 

 

The final step is to issue an install commit command.

 

RP/0/RP0/CPU0::NCS540#install commit

Sat Sep 30 03:39:31.523 UTC

2023-09-30 03:39:33 Install operation 6 started by root:

  install commit

2023-09-30 03:39:34 Install operation will continue in the background

RP/0/RP0/CPU0:NCS540#2023-09-30 03:39:46 Install operation 6 finished successfully

RP/0/RP0/CPU0:Sep 30 03:39:46.897 UTC: sdr_instmgr[1296]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 6 finished successfully