I got an escalation from our NOC a couple of weeks back saying one of our core router had an alarm. When I checked the logs, I noticed the CPU temperature was high, one of the the power supply and the fan had failed.
Aug 4 00:25:39.937 UTC: %ENVMON-4-ONE_FAN_LOW_RPM: Warning: Fan 2 is running at low RPM. Rotation speed is now high for all other fans. Fan Tray replacement is recommended.
Aug 4 00:28:10.119 UTC: %ENVMON-1-CPU_WARNING_OVERTEMP: Warning: CPU temperature 101C exceeds threshold 95C. Please resolve system cooling immediately to prevent system damage
3945#show environment
SYSTEM POWER SUPPLY STATUS
==========================
Internal Power Supply 1 Type: AC
Internal Power Supply 1 12V Output Status: Normal
Internal Power Supply 2 Type: Absent
SYSTEM FAN STATUS
=================
Fan Rotation Alert: Total 1 Fan Low RPM
Fan 1 OK, Maximum speed setting
Fan 2 Low RPM
Fan 3 OK, Maximum speed setting
Fan 4 OK, Maximum speed setting
Fan 5 OK, Maximum speed setting
SYSTEM TEMPERATURE STATUS
=========================
Intake Left temperature: 36 Celsius, Normal
Intake Right temperature: 29 Celsius, Normal
Exhaust Right temperature: 44 Celsius, Normal
Exhaust Left temperature: 49 Celsius, Normal
CPU temperature: 106 Celsius, Over-Temperature
Power Supply Unit 1 temperature: 40 Celsius, Normal
REAL TIME CLOCK BATTERY STATUS
==============================
Battery OK (checked at power up)
SYSTEM POWER
===============
Motherboard Components Power consumption = 79.0 W
Total System Power consumption is: 79.0 W
Environmental information last updated 00:00:07 ago
I immediately contacted Cisco TAC and asked for an RMA.
The power supply (
PWR-3900-AC) has screws on the sides which helps in pulling out from the router's chassis.
These are front and back view of the fan module/assembly (3900-FANASSY). The RMA comes with a new faceplate.
According to
Cisco, these modules are hot-swappable so you don't need to turn off the chassis. You also got at least two minutes to do the swap in order to avoid further CPU overheating. When I removed the router's faceplate, I've observed that all five fans weren't rotating and blown air was warm. This issue has been going for at least 3 days before I swapped the new modules. Good thing nothing melted inside :)
When you removed the fan assembly, you would see two power supplies.
Aug 7 13:59:32.305 UTC: %ENVMON-2-FAN_TRAY_MISSING: Critical Warning: Fan tray was removed. Please re-insert fan tray to prevent system from overheating.
Aug 7 14:00:32.380 UTC: %ENVMON-1-POWER_WARNING: : Internal Power Supply Unit 2 AC or DC input source has been removed.
Aug 7 14:01:02.416 UTC: %ENVMON-5-POWER_NOTICE: : Internal Power Supply Unit 2 12V is UP %ENVMON-5-POWER_NOTICE: : Internal Power Supply Unit 2 12V is UP
Aug 7 14:02:32.683 UTC: %ENVMON-6-CPU_TEMP_OK: CPU temperature normal
Aug 7 14:02:32.683 UTC: %ENVMON-6-FAN_TRAY_OK: Fan tray is detected.
3945#show inventory
<OUTPUT TRUNCATED>
NAME: "C3900 AC Power Supply 1", DESCR: "C3900 AC Power Supply 1"
PID: PWR-3900-AC , VID: V03 , SN: QCS1726abcde
NAME: "C3900 AC Power Supply 2", DESCR: "C3900 AC Power Supply 2"
PID: PWR-3900-AC , VID: V04 , SN: QCS173abcde
3945#show environment
SYSTEM POWER SUPPLY STATUS
==========================
Internal Power Supply 1 Type: AC
Internal Power Supply 1 12V Output Status: Normal
Internal Power Supply 2 Type: AC
Internal Power Supply 2 12V Output Status: Normal
SYSTEM FAN STATUS
=================
Fan 1 OK, Low speed setting
Fan 2 OK, Low speed setting
Fan 3 OK, Low speed setting
Fan 4 OK, Low speed setting
Fan 5 OK, Low speed setting
SYSTEM TEMPERATURE STATUS
=========================
Intake Left temperature: 20 Celsius, Normal
Intake Right temperature: 19 Celsius, Normal
Exhaust Right temperature: 20 Celsius, Normal
Exhaust Left temperature: 22 Celsius, Normal
CPU temperature: 44 Celsius, Normal
Power Supply Unit 1 temperature: 21 Celsius, Normal
Power Supply Unit 2 temperature: 24 Celsius, Normal
REAL TIME CLOCK BATTERY STATUS
==============================
Battery OK (checked at power up)
SYSTEM POWER
===============
Motherboard Components Power consumption = 123.6 W
Total System Power consumption is: 123.6 W
Environmental information last updated 00:00:10 ago