Skip to main content

The Switch Needed a Reboot

· 4 min read

Picture this: you’re a fearless Level‑2 Network Hero, swooping in to rescue a clinic where the phones are dead, the computers are screaming “Network Error,” and the doctors are staring at their screens like a bad horror movie. The culprit? A Cisco switch that decided it was time for a mid‑life crisis and went permanently offline—except for a single, rebellious port that stayed up. Spoiler: the switch is PoE‑centric, and the clinic’s terminals are chained to phones that double as power adapters. Classic case of “If the phone dies, the whole network dies.”

The Setup​

  • Equipment: Dell Wyse Terminals running ThinOS, PoE phones, and networked printers.
  • Connectivity: Each terminal is tucked into a phone’s Ethernet passthrough. The phones don’t support passive passthrough, so lose power on the phone and the terminal gets the boot.
  • Symptoms: Phones dead, computers offline, patients waiting, doctors powerless.

The Investigation​

You open your custom “Network Detective” tool, and the dashboard is a masterpiece of doom:

Interface GigabitEthernet1/0/3 – down
Interface GigabitEthernet1/0/7 – down
...

All the other ports are fine, except for the ones feeding the terminal‑phone combo. Your heart rate spikes; it looks like a layer‑1 cable cut, but why only on one switch? The syslog whispers:

%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/9: PD removed
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/9: Power Controller reports power Tstart error detected

Turns out the switch stopped delivering PoE, and the phones, and therefore the terminals, were left in the dark.

The Dilemma​

You’re at a crossroads: Reboot the switch or wait for the omniscient Network Engineering squad. The switch is partly online, but the PoE is gone. Your manager is in a meeting. The clinic manager is about to lose another hour of patient time. The risk? The switch might not come back or could cause a ripple of outages in the LAN.

You decide: Reboot. Because you’ve never had a switch fail in your career, you’re probably safe.

The Reboot​

You log an emergency change, call the clinic manager, and together you yank the power. The switch lights flicker, a ticket is auto‑generated, and your manager finally checks in. The switch boots, but the phones stay dead. The drama intensifies: “I can’t log in to my computer now either.”

You’re stuck, the switch is still down, and the clinic’s operations are on a cliff edge. The only hope? Manual re‑patching.

The Patch Panel Showdown​

You guide the clinic manager through the patch panel like a game show host. “Follow the blue cable, note the wall plate number,” you say. She does it. Within 10 minutes, three desks are back online. The clinic manager’s confidence surges; the doctors are grateful; you’re breathing a little easier.

Your manager returns, sees the chaos, and says, “You did WHAT?” After you explain the reasoning and the outcome, he’s surprisingly calm and says you’ll be fine. Spoiler: you didn’t get fired. 🎉

The Aftermath​

  • The switch is replaced by “Outeractive” (the vendor, not a real one). PoE returns to life.
  • Free ports on the working switch saved the day.
  • You have a photo of the rack for future reference.
  • The clinic manager sends a positive feedback email that makes you feel like a hero in a medical drama.
  • Network engineers just nod and say they’ll re‑apply the config on the new switch.

The Lessons Learned​

  1. Don’t reboot unless you’re sure the switch is dead. Reboots can be risky if the device is partially online.
  2. Keep a backup plan—manual patching is the unsung hero of network crises.
  3. Document your process so your manager knows what you’re doing.
  4. Don’t let the network engineers take all the credit; a good story ends with a hero and a happy clinic manager.

TL;DR​

A clinic’s PoE‑dependent devices went dark because a Cisco switch stopped delivering power. The hero (our narrator) rebooted the switch, which didn’t fix it, then manually patched terminals via the patch panel, restoring service. The switch was replaced, no one got fired, and the clinic manager was impressed. Lesson: reboot only when necessary, keep a backup plan, and always be ready to play patch‑panel wizard.