Cisco 3900 ISR G2 Power Supply and Fan Module Swap

I got an escalation from our NOC a couple of weeks back saying one of our core router had an alarm. When I checked the logs, I noticed the CPU temperature was high, one of the the power supply and the fan had failed.

Aug  4 00:25:39.937 UTC: %ENVMON-4-ONE_FAN_LOW_RPM: Warning: Fan 2 is running at low RPM.  Rotation speed is now high for all other fans.  Fan Tray replacement is recommended.

Aug  4 00:28:10.119 UTC: %ENVMON-1-CPU_WARNING_OVERTEMP: Warning: CPU temperature 101C exceeds threshold 95C.  Please resolve system cooling immediately to prevent system damage


3945#show environment
SYSTEM POWER SUPPLY STATUS
==========================
 Internal Power Supply 1 Type: AC
 Internal Power Supply 1 12V Output Status: Normal

 Internal Power Supply 2 Type: Absent 

SYSTEM FAN STATUS
=================
 Fan Rotation Alert: Total 1 Fan Low RPM 
 Fan 1 OK, Maximum speed setting
 Fan 2 Low RPM   
 Fan 3 OK, Maximum speed setting
 Fan 4 OK, Maximum speed setting
 Fan 5 OK, Maximum speed setting

SYSTEM TEMPERATURE STATUS
=========================
 Intake Left temperature: 36 Celsius, Normal
 Intake Right temperature: 29 Celsius, Normal
 Exhaust Right temperature: 44 Celsius, Normal
 Exhaust Left temperature: 49 Celsius, Normal
 CPU temperature: 106 Celsius, Over-Temperature  
 Power Supply Unit 1 temperature: 40 Celsius, Normal

REAL TIME CLOCK BATTERY STATUS
==============================
 Battery OK (checked at power up)

SYSTEM POWER
===============
 Motherboard Components Power consumption = 79.0 W
 Total System Power consumption is: 79.0 W

 Environmental information last updated 00:00:07 ago


I immediately contacted Cisco TAC and asked for an RMA.

The power supply (PWR-3900-AC) has screws on the sides which helps in pulling out from the router's chassis.

These are front and back view of the fan module/assembly (3900-FANASSY). The RMA comes with a new faceplate.

According to Cisco, these modules are hot-swappable so you don't need to turn off the chassis. You also got at least two minutes to do the swap in order to avoid further CPU overheating. When I removed the router's faceplate, I've observed that all five fans weren't rotating and blown air was warm. This issue has been going for at least 3 days before I swapped the new modules. Good thing nothing melted inside :)

When you removed the fan assembly, you would see two power supplies.

Aug  7 13:59:32.305 UTC: %ENVMON-2-FAN_TRAY_MISSING: Critical Warning: Fan tray was removed.  Please re-insert fan tray to prevent system from overheating.
Aug  7 14:00:32.380 UTC: %ENVMON-1-POWER_WARNING: : Internal Power Supply Unit 2  AC or DC input source has been removed.
Aug  7 14:01:02.416 UTC: %ENVMON-5-POWER_NOTICE: : Internal Power Supply Unit 2  12V is UP %ENVMON-5-POWER_NOTICE: : Internal Power Supply Unit 2  12V is UP
Aug  7 14:02:32.683 UTC: %ENVMON-6-CPU_TEMP_OK: CPU temperature normal
Aug  7 14:02:32.683 UTC: %ENVMON-6-FAN_TRAY_OK: Fan tray is detected.


3945#show inventory

<OUTPUT TRUNCATED>

NAME: "C3900 AC Power Supply 1", DESCR: "C3900 AC Power Supply 1"
PID: PWR-3900-AC       , VID: V03 , SN: QCS1726abcde

NAME: "C3900 AC Power Supply 2", DESCR: "C3900 AC Power Supply 2"
PID: PWR-3900-AC       , VID: V04 , SN: QCS173abcde

3945#show environment
SYSTEM POWER SUPPLY STATUS
==========================
 Internal Power Supply 1 Type: AC
 Internal Power Supply 1 12V Output Status: Normal

 Internal Power Supply 2 Type: AC
 Internal Power Supply 2 12V Output Status: Normal

SYSTEM FAN STATUS
=================
 Fan 1 OK, Low speed setting
 Fan 2 OK, Low speed setting
 Fan 3 OK, Low speed setting
 Fan 4 OK, Low speed setting
 Fan 5 OK, Low speed setting

SYSTEM TEMPERATURE STATUS
=========================
 Intake Left temperature: 20 Celsius, Normal
 Intake Right temperature: 19 Celsius, Normal
 Exhaust Right temperature: 20 Celsius, Normal
 Exhaust Left temperature: 22 Celsius, Normal
 CPU temperature: 44 Celsius, Normal
 Power Supply Unit 1 temperature: 21 Celsius, Normal
 Power Supply Unit 2 temperature: 24 Celsius, Normal

REAL TIME CLOCK BATTERY STATUS
==============================
 Battery OK (checked at power up)

SYSTEM POWER
===============
 Motherboard Components Power consumption = 123.6 W
 Total System Power consumption is: 123.6 W

 Environmental information last updated 00:00:10 ago