Saturday, April 1, 2017

Cisco 3850 Switch IOS Upgrade (via software install command)

I had to upgrade a Cisco 3850 switch due to a high CPU issue via the software install command.


Switch#show processes cpu sorted
Core 0: CPU utilization for five seconds: 77%; one minute: 71%;  five minutes: 71%
Core 1: CPU utilization for five seconds: 28%; one minute: 71%;  five minutes: 78%
Core 2: CPU utilization for five seconds: 96%; one minute: 72%;  five minutes: 67%
Core 3: CPU utilization for five seconds: 99%; one minute: 90%;  five minutes: 91%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
5632   547159      38565649 240    73.91    73.47    73.38    1088  fed
9223   2869678     17955697 98     0.97     1.17     1.15     0     iosd              
6297   1592710     28366670 1186   0.14     0.15     0.14     0     cpumemd           
6278   403412      5609218  837    0.10     0.01     0.03     0     oom_poll.sh       
9219   3878616     89615676 25     0.10     0.10     0.10     0     wcm               
5633   2355637     33902453 13     0.05     0.14     0.12     0     platform_mgr      
5634   918091      19813894 41     0.05     0.06     0.05     0     stack-mgr         
6286   605408      34263960 50     0.05     0.09     0.10     0     pdsd              
6290   905582      23382817 22     0.05     0.01     0.01     0     netd              
1      1650        1056     1562   0.00     0.00     0.00     0     init              
2      0           126      0      0.00     0.00     0.00     0     kthreadd          
3      9620        1171075  8      0.00     0.00     0.00     0     migration/0       
4      0           3        0      0.00     0.00     0.00     0     sirq-high/0       
5      1139468     45134130 2      0.00     0.00     0.00     0     sirq-timer/0      
6      0           6        0      0.00     0.00     0.00     0     sirq-net-tx/0     
7      10          9636     1      0.00     0.00     0.00     0     sirq-net-rx/0     
8      1180        2734379  0      0.00     0.00     0.00     0     sirq-block/0      
9      0           8        0      0.00     0.00     0.00     0     sirq-block-iopo   
10     20          3286     6      0.00     0.00     0.00     0     sirq-tasklet/0

<OUTPUT TRUNCATED>


Once the IOS is copied to the switch's flash memory (via FTP or TFTP), you can issue the software install command from privileged EXEC prompt.

Switch#software install file flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1]: Verifying and copying expanded package files to flash:
[1]: Verified and copied expanded package files to flash:
[1]: Starting compatibility checks
[1]: Finished compatibility checks
[1]: Starting application pre-installation processing
[1]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.02.03.SE.pkg
    Removed cat3k_caa-drivers.SPA.03.02.03.SE.pkg
    Removed cat3k_caa-infra.SPA.03.02.03.SE.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX3.pkg
    Removed cat3k_caa-platform.SPA.03.02.03.SE.pkg
    Removed cat3k_caa-wcm.SPA.10.0.120.0.pkg
[1]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[1]: Creating pending provisioning file
[1]: Finished installing software.  New software will load on reboot.
[1]: Committing provisioning file

[1]: Do you want to proceed with reload? [yes/no]: yes

System configuration has been modified. Save? [yes/no]: yes

Building configuration...
Compressed configuration from 10347 bytes to 4871 bytes[OK]
[1]: Reloading        // REMOTE ACCESS WAS LOST


<OUTPUT TRUNCATED>


The IOS upgrade took me around 15-20 minutes before I started to get ping replies on the switch.

Switch#show version | include INSTALL   
*    1 56    WS-C3850-48P       03.06.04.E        cat3k_caa-universalk9 INSTALL

Switch#show processes cpu     // CPU LOAD FOR THE 4 CORE SIGNIFICANTLY DROPPED
Core 0: CPU utilization for five seconds: 6%; one minute: 8%; five minutes: 46%
Core 1: CPU utilization for five seconds: 0%; one minute: 2%; five minutes: 36%
Core 2: CPU utilization for five seconds: 3%; one minute: 2%; five minutes: 42%
Core 3: CPU utilization for five seconds: 1%; one minute: 4%; five minutes: 36%


<OUTPUT TRUNCATED>


Issue the software clean command to remove unwanted package files after everything is verified. You can't do a software rollback once the unwanted package files are completely removed.

Switch#software clean
Preparing clean operation ...
[1]: Cleaning up unnecessary package files
[1]: No path specified, will use booted path flash:packages.conf
[1]: Cleaning flash:
[1]: Preparing packages list to delete ...
     In use files, will not delete:
       cat3k_caa-base.SPA.03.06.04.E.pkg
       cat3k_caa-drivers.SPA.03.06.04.E.pkg
       cat3k_caa-infra.SPA.03.06.04.E.pkg
       cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
       cat3k_caa-platform.SPA.03.06.04.E.pkg
       cat3k_caa-wcm.SPA.10.2.140.0.pkg
       packages.conf
[1]: Files that will be deleted:
    cat3k_caa-base.SPA.03.02.03.SE.pkg
    cat3k_caa-drivers.SPA.03.02.03.SE.pkg
    cat3k_caa-infra.SPA.03.02.03.SE.pkg
    cat3k_caa-iosd-universalk9.SPA.150-1.EX3.pkg
    cat3k_caa-platform.SPA.03.02.03.SE.pkg
    cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
    cat3k_caa-wcm.SPA.10.0.120.0.pkg
    packages.conf.00-

[1]: Do you want to proceed with the deletion? [yes/no]: yes
[1]: Clean up completed     


The same command is used to upgrade a stacked Cisco 3850 switches but using the switch <SWITCH NUMBER> keyword.

Switch#show switch
Switch/Stack Mac Address : c025.5c85.9123 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State
------------------------------------------------------------
*1       Active   c025.5c85.9123     10     V02     Ready              
 2       Standby  c025.5c85.f456     1      V02     Ready   


Swtich#software install file flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin switch ?
  Valid switch ids: 1,2

  Multiple switch ids may be specified using a
  comma separated list or  '-' separated range

  For example:
    1,3,4     specifies switch ids 1, 3 and 4
    1-3       specifies switch ids 1, 2 and 3

  WORD  Switch id(s)

Swtich#software install file flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin switch 1-2
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1 2]: Finished compatibility checks
[1 2]: Starting application pre-installation processing
[1 2]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-drivers.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-infra.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg
    Removed cat3k_caa-platform.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-wcm.SPA.10.0.111.0.pkg
[2]: Old files list:
    Removed cat3k_caa-base.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-drivers.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-infra.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg
    Removed cat3k_caa-platform.SPA.03.02.02.SE.pkg
    Removed cat3k_caa-wcm.SPA.10.0.111.0.pkg
[1]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[2]: New files list:
    Added cat3k_caa-base.SPA.03.06.04.E.pkg
    Added cat3k_caa-drivers.SPA.03.06.04.E.pkg
    Added cat3k_caa-infra.SPA.03.06.04.E.pkg
    Added cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg
    Added cat3k_caa-platform.SPA.03.06.04.E.pkg
    Added cat3k_caa-wcm.SPA.10.2.140.0.pkg
[1 2]: Creating pending provisioning file
[1 2]: Finished installing software.  New software will load on reboot.
[1 2]: Do you want to proceed with reload? [yes/no]:yes
[2]: Reloading