Saturday, September 12, 2015

Cisco Catalyst 3650 IOS Recovery via USB

I was preparing a new Cisco Catalyst 3650 switch and it's my first time to get a switch: prompt from a newly opened switch. I tried to initialize the it's flash but it didn't work. The IOS recovery via USB on a Cisco 3650 switch is very similar to a Cisco 3850. Below is the procedure that I've performed.


Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@done.
Memory Test Pass!

Base ethernet MAC Address: f4:0f:1b:84:f1:23

Interface GE 0 link down***ERROR: PHY link is down
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.
flash:packages.conf: no such file or directory
flash:packages.conf: no such file or directory

Error loading "flash:packages.conf"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.


switch: flash_init
Initializing Flash...
...The flash is already initialized.

switch: load_helper
Unknown cmd: load_helper

switch: dir flash:
Directory of flash:/

30977  drwx  4096       .
    2  drwx  4096       ..
30978  -rw-  79121160   cat3k_caa-base.SPA.03.03.03SE.pkg
30979  -rw-  6484668    cat3k_caa-drivers.SPA.03.03.03SE.pkg
30980  -rw-  34522096   cat3k_caa-infra.SPA.03.03.03SE.pkg
30981  -rw-  34821452   cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg
30982  -rw-  25131920   cat3k_caa-platform.SPA.03.03.03SE.pkg
30983  -rw-  77312832   cat3k_caa-wcm.SPA.10.1.130.0.pkg

1361833984 bytes available (286060544 bytes used)

switch: boot flash:cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg
flash:cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg: bad/non-bootable Nova bundle file

Error loading "flash:cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

switch: dir usbflash0:
Directory of usbflash0:/

    2  -rw-  62682268   c2900-universalk9-mz.SPA.150-1.M4.bin
    3  -rw-  21890692   c870-advipservicesk9-mz.124-24.T4.bin
    4  -rw-  4968160    TeamViewerQS_en-idch93gk2g.exe
    5  -rw-  310347344  cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin
    6  -rw-  125231421  lms5.1.bin
    7  -rw-  257650008  cat3k_caa-universalk9ldpe.SPA.03.03.05.SE.150-1.EZ5.bin
    8  -rw-  38172672   asa916-4-smp-k8.bin

1200291840 bytes available (821035008 bytes used)

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin    // BOOT IOS FROM USB
Reading full image into memory........................................................................................................................................
....................................................................................................................................................
...............done
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6258928c
Kernel Size       : 0x3f9602/4167170
Initramfs Address : 0x62982890
Initramfs Size    : 0xe04704/14698244
Compression Format: .mzip

Bootable image at @ ram:0x6258928c
Bootable image segment 0 address range [0x81100000, 0x820a0000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin" uncompressed and installed, entry point: 0x81672c70
Loading Linux kernel with entry point 0x81672c70 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=4,mode=600,ptmxmode=000 0 0


FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled


Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...83664 bytes.
Front-end Microcode IMG MGR: Programming device 0...wRrsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

<OUTPUT TRUNCATED>

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss!Front-end Microcode IMG MGR: Microcode programming complete for device 0.
Front-end Microcode IMG MGR: Microcode programming complete in 42 seconds


              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.00.E RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 08-Dec-14 00:20 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3650-24PS (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO1808Q123X
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
1974239K bytes of USB Flash at usbflash0:.
0K bytes of  at webui:.

Base Ethernet MAC Address          : f4:0f:1b:84:f1:23
Motherboard Assembly Number        : 73-15128-05
Motherboard Serial Number          : FDO1810Z456
Model Revision Number              : A0
Motherboard Revision Number        : A0
Model Number                       : WS-C3650-24PS
System Serial Number               : FDO1808Q123X

The System is using a non-recommended Boot mode.
Not all features may be available in this boot mode.
Please check the product configuration guide.


         --- System Configuration Dialog ---

Enable secret warning
----------------------------------
In order to access the device manager, an enable secret is required
If you enter the initial configuration dialog, you will be prompted for the enable secret
If you choose not to enter the intial configuration dialog, or if you exit setup without setting the enable secret,
please set an enable secret using the following CLI in configuration mode-
enable secret 0 <cleartext password>
----------------------------------
Would you like to enter the initial configuration dialog? [yes/no]: no

Would you like to terminate autoinstall? [yes]:
Startup-config is ignored. So, HTTP Secure server configuration not done

Press RETURN to get started!

*Sep  3 02:18:18.201: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
Switch>enable
*Sep  3 02:18:28.433: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server https://devicehelper.cisco.com/pnp/HELLO
Switch#copy usbflash0:cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin flash   // COPY IOS TO FLASH MEMORY
Destination filename [cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin]?
Copy in progress...CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

<OUTPUT TRUNCATED>

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
310347344 bytes copied in 51.920 secs (5977414 bytes/sec)
Switch#dir   
Directory of flash:/

30978  -rw-    79121160  May 28 2014 08:06:16 +00:00  cat3k_caa-base.SPA.03.03.03SE.pkg
30979  -rw-     6484668  May 28 2014 08:06:17 +00:00  cat3k_caa-drivers.SPA.03.03.03SE.pkg
30980  -rw-    34522096  May 28 2014 08:06:16 +00:00  cat3k_caa-infra.SPA.03.03.03SE.pkg
30981  -rw-    34821452  May 28 2014 08:06:17 +00:00  cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg
30982  -rw-    25131920  May 28 2014 08:06:16 +00:00  cat3k_caa-platform.SPA.03.03.03SE.pkg
30983  -rw-    77312832  May 28 2014 08:06:17 +00:00  cat3k_caa-wcm.SPA.10.1.130.0.pkg
46465  drwx        4096   Sep 3 2015 02:15:31 +00:00  virtual-instance
46467  drwx        4096   Sep 3 2015 02:16:13 +00:00  dc_profile_dir
30984  -rw-     2097152   Sep 3 2015 02:17:02 +00:00  nvram_config
46469  drwx        4096   Sep 3 2015 02:17:10 +00:00  wnweb_store
30985  -rw-   310347344   Sep 3 2015 02:19:02 +00:00  cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin

1621966848 bytes total (1048842240 bytes free)

Switch#configure terminal
Switch(config)#boot system switch all flash:cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin
Switch#end
Switch#show run | inc boot
boot-start-marker
boot system switch all flash:cat3k_caa-universalk9.SPA.03.07.00.E.152-3.E.bin
boot-end-marker
diagnostic bootup level minimal
Switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

*Sep  3 02:23:46.882: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.
*Sep  3 02:23:47.526: %STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr:  Received reload request for all switches, reason Reload command
*Sep  3 02:23:47.527: %STACKMGR-1-RELOAD: 1 stack-mgr:  Reloading due to reason Reload command
*Sep  3 02:23:48.028: %IOSXE-3-PLATFORM: 1 process sysmgr: Reset/Reload requested by [stack-manager].
<Thu Sep  3 02:23:48 2015> Message from sysmgr: Reason Code:[3] Reset Reason:Reset/Reload requested by [stack-manager]. [Reload command]
umount: /proc/fs/nfsd: not mounted
Unmounting ng3k filesystems...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.

Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++
Base ethernet MAC Address: f4:0f:1b:84:f1:23

 <OUTPUT TRUNCATED>

Switch>enable
*Sep  3 02:29:47.115: %SSH-5-DISABLED: SSH 1.99 has been disabled
*Sep  3 02:29:47.116: %SSH-5-ENABLED: SSH 1.99 has been enabled
Switch#show version | inc IOS
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.00.E RELEASE SOFTWARE (fc4)
Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
ROM: IOS-XE ROMMON

2 comments:

  1. Many thanks for the posting because of which three dead C3850s of ours are now production ready.

    ReplyDelete
  2. Thanks for this. Documentation on these crappy converged switches can be hard to come by

    ReplyDelete