Monday, January 9, 2017

Password Recovery on a Cisco 3650 Catalyst Switch

I had to reconfigure some used Cisco 3650 switches from a previous deployment but wasn't able to login using known passwords so I had to perform a password recovery. I've been doing password recovery on Cisco 3560 switch which is identical on other switch platforms, but the password recovery for a Cisco 3650 is a bit different. The CONSOLE port is found at the back and it's the top most port (look at LED arrow pointing upward).


The MODE button is found in front and it's a small black button beside the Cisco logo.


Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@done.
Memory Test Pass!

Base ethernet MAC Address: f4:4e:05:57:f1:23

Interface GE 0 link down***ERROR: PHY link is down      // WAIT FOR SYST AND ACTV LED TO BECOME AMBER BEFORE RELEASING MODE BUTTON

The system has been interrupted prior to initializing some
filesystems and loading the operating system software.
Console will be reset to 9600 baud rate, need to change terminal setting first.
The following commands will initialize the remaining filesystems,
and finish loading the operating system software:

    flash_init
    boot

switch: flash_init
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 2 seconds....done Initializing Flash.

switch: SWITCH_IGNORE_STARTUP_CFG=1     // BYPASS STARTUP-CONFIG IN NVRAM

switch: boot flash:packages.conf
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 79121160
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6042f350
Kernel Size       : 0x402ecf/4206287
Initramfs Address : 0x60832220
Initramfs Size    : 0xdb98e6/14391526
Compression Format: .mzip

Bootable image at @ ram:0x6042f350
Bootable image segment 0 address range [0x81100000, 0x82110000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x81653a10 ...Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services

Dec 8 02:21:06 %PLATFORM_MGR-1-PLATMGR_INIT_FAIL: Platform Manager: Failed to set system LEDs after POST.

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found,FIPS Mode Not Enabled

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3650-24PS (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO1837EABC
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
1550272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of  at webui:.

Base Ethernet MAC Address          : f4:4e:05:57:f1:823
Motherboard Assembly Number        : 73-15128-05
Motherboard Serial Number          : FDO18370DEF
Model Revision Number              : D0
Motherboard Revision Number        : A0
Model Number                       : WS-C3650-24PS
System Serial Number               : FDO1837EABC


         --- System Configuration Dialog ---

Enable secret warning
----------------------------------
In order to access the device manager, an enable secret is required
If you enter the initial configuration dialog, you will be prompted for the enable secret
If you choose not to enter the intial configuration dialog, or if you exit setup without setting the enable secret,
please set an enable secret using the following CLI in configuration mode-
enable secret 0 <cleartext password>
----------------------------------
Would you like to enter the initial configuration dialog? [yes/no]: no


Press RETURN to get started!


*Dec  8 02:22:12.388: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Dec  8 02:22:14.285: Registering wireless registries required for roaming

*Dec  8 02:22:14.617: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*Dec  8 02:22:14.617: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Dec  8 02:22:14.619: %NGWC_PLATFORM_FEP-6-FRU_PS_OIR: Switch 1: FRU power supply A inserted
*Dec  8 02:21:32.078: *%INIT-7-SWITCH_BOOTING: 1 wcm:  Switch booting...
*Dec  8 02:22:15.618: %LINEPROT% Generating 1024 bit RSA keys, keys will be non-exportable...O-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Dec  8 02:22:15.618: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Dec  8 02:22:16.507: %MGMTINFRA-3-CFG_PUSH: 1 eicored:  Config push failed.please check wcm provider.
*Dec  8 02:22:16.507: %MGMTINFRA-3-CFG_PUSH: 1 eicored:  Config push failed (rc=10000) for (wcm) on attributes [{ schedulerEnabled@1 : 10000, rtTimeout@1 : 10000, frameBurst@1 : 10000 }, { schedulerEnabled@1 : 10000, rtTimeout@
[OK] (elapsed time was 1 seconds)
1 : 10000, frameBurst@1 : 10000 }]
*Dec  8 02:22:16.893: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Dec  8 02:22:22.669: %SYS-6-STARTUP_CONFIG_IGNORED: System startup configuration is ignored based on the configuration register setting.
*Dec  8 02:22:26.454: %STACKMGR-6-ACTIVE_READY: 1 stack-mgr:  Active switch 1 is ready. System has been configured

*Dec  8 02:22:26.656: %SYS-5-RESTART: System restarted --
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team
*Dec  8 02:22:26.681: %AUTHMGR_SPI-6-START: Auth Manager SPI server started
*Dec  8 02:22:29.100: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Dec  8 02:22:33.460: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Dec  8 02:22:33.511: %PKI-6-AUTOSAVE: Running configuration saved to NVRAM
Switch>enable
Switch#copy startup-config running-config      // LOAD THE START-UP CONFIG FROM NVRAM
Destination filename [running-config]?
% Generating 1024 bit RSA keys, keys will be non-exportable...
[OK] (elapsed time was 1 seconds)
*Dec  8 04:12:14.228: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Dec  8 04:12:14.271: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate
*Dec  8 04:12:20.062: %AAAA-4-NOSERVER: Warning: Server 89.1.2.8 is not defined.
*Dec  8 04:12:20.063: %AAAA-4-NOSERVER: Warning: Server 66.5.3.8 is not defined.
*Dec  8 04:12:20.063: %AAAA-4-NOSERVER: Warning: Server 66.1.3.9 is not defined.
 Warning: The cli will be deprecated soon
 'tacacs-server host 89.1.2.8'       // STARTUP-CONFIG HAS TACACS CONFIGURED
 Please move to 'tacacs server <name>' CLI
 Warning: The cli will be deprecated soon
 'tacacs-server host 66.5.3.9'
 Please move to 'tacacs server <name>' CLI
 Warning: The cli will be deprecated soon
 'tacacs-server host 66.5.3.8'
 Please move to 'tacacs server <name>' CLI
9523 bytes copied in 10.430 secs (913 bytes/sec)
sw02#     // THE HOSTNAME WAS ALSO LOADED
*Dec  8 04:12:20.096: % Multiple self signed certificates in config
    certificate for trust point TP-self-signed-1212499775 ignored
*Dec  8 04:12:20.382: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate        // SAVE AFTER YOU RE-CONFIGURE THE NEW PASSWORDS, ISSUE THE reload COMMAND AND HOLD THE MODE BUTTON AGAIN
sw02#
sw02#delete vlan.dat        // AT THIS POINT YOU CAN EITHER RE-CONFIGURE THE ENABLE, LOCAL USERNAME AND VTY PASSWORDS OR COMPLETELY WIPE OUT THE SWITCH USING THE delete vlan.dat AND write erase COMMANDS (then do a reload).
Delete filename [vlan.dat]?
Delete flash:/vlan.dat? [confirm]
sw02#write erase
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
sw02#reload     // HOLD THE MODE BUTTON AGAIN
*Dec  8 04:17:00.314: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram

System configuration has been modified. Save? [yes/no]: no     // TYPE yes IF YOU RE-CONFIGURED PASSWORDS; TYPE no IF YOU WANT A CLEAN CONFIG
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
*Dec  8 04:18:11.797: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.
*Dec  8 04:18:12.499: %STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr:  Received reload request for all switches, reason Reload command
*Dec  8 04:18:12.500: %STACKMGR-1-RELOAD: 1 stack-mgr:  Reloading due to reason Reload command
*Dec  8 04:18:13.001: %IOSXE-3-PLATFORM: 1 process sysmgr: Reset/Reload requested by [stack-manager].
<Thu Dec  8 04:18:13 2016> Message from sysmgr: Reason Code:[3] Reset Reason:Reset/Reload requested by [stack-manager]. [Reload command]
umount: /proc/fs/nfsd: not mounted
Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.


<OUTPUT TRUNCATED>


Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++
Base ethernet MAC Address: f4:4e:05:51:a9:80

Interface GE 0 link down***ERROR: PHY link is down     // WAIT FOR SYST AND ACTV LED TO BECOME AMBER BEFORE RELEASING MODE BUTTON
The system has been interrupted prior to initializing some
filesystems and loading the operating system software.
Console will be reset to 9600 baud rate, need to change terminal setting first.
The following commands will initialize the remaining filesystems,
and finish loading the operating system software:

    flash_init      // SKIP THE flash_init and boot COMMANDS
    boot

switch: SWITCH_IGNORE_STARTUP_CFG=0      // INSTRUCTS THE SWITCH TO READ/LOAD THE STARTUP-CONFIG

switch: boot flash:packages.conf
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 79121160
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6042d350
Kernel Size       : 0x402ecf/4206287
Initramfs Address : 0x60830220
Initramfs Size    : 0xdb98e6/14391526
Compression Format: .mzip

Bootable image at @ ram:0x6042d350
Bootable image segment 0 address range [0x81100000, 0x82110000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x81653a10 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services

Dec 8 04:22:53 %PLATFORM_MGR-1-PLATMGR_INIT_FAIL: Platform Manager: Failed to set system LEDs after POST.

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found,FIPS Mode Not Enabled

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3650-24PS (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO1837EABC
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
1550272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of  at webui:.

Base Ethernet MAC Address          : f4:4e:05:51:a1:23
Motherboard Assembly Number        : 73-15128-05
Motherboard Serial Number          : FDO18370DEF
Model Revision Number              : D0
Motherboard Revision Number        : A0
Model Number                       : WS-C3650-24PS
System Serial Number               : FDO1837EABC



         --- System Configuration Dialog ---

Enable secret warning
----------------------------------
In order to access the device manager, an enable secret is required
If you enter the initial configuration dialog, you will be prompted for the enable secret
If you choose not to enter the intial configuration dialog, or if you exit setup without setting the enable secret,
please set an enable secret using the following CLI in configuration mode-
enable secret 0 <cleartext password>
----------------------------------
Would you like to enter the initial configuration dialog? [yes/no]: no

Would you like to terminate autoinstall? [yes]:


Press RETURN to get started!


Switch>enable
Switch#configure terminal
Switch(config)#no manual ?     // no manual boot COMMAND IS UNAVAILABLE
% Unrecognized command
Switch(config)#no m?    
mab       mac     macro    map-class
map-list  memory  monitor 

Switch(config)#end
Switch#no m?


I chose to completely wipe out the switch, re-configure it and tested again by rebooting and the startup-config stored in NVRAM remained intact.

3 comments:

  1. Thanks for this. I was going nuts, since password resets on other Cisco models didn't work.

    ReplyDelete
  2. Very useful information, thanks for sharing your knowledge!

    ReplyDelete