A Tale of IT Terror!
|
Who doesn't like a good horror story this time of year? Well, the person to whom the horrible thing happened, that's who! I spoke with just such a horror-scarred IT manager today. In exchange for his candid story of technology run amok, he asked that his name not be used and his company not identified. Prepare to be terrified. Bwa-ha-ha-ha-ha! "A couple of years ago, when we had a certain manufacturer's switches in place, we decided we needed to give our users wireless. We worked hard and long to secure it. We ensured that the configs on both the switches and the APs (notably, from a different manufacturer) were carefully checked and rechecked and decided to schedule the install for 4 p.m. on a Friday because this was a low-risk, low-impact event. Or so we thought." (Cue ominous music.) "Minutes after the first AP was plugged in, someone commented that they couldn't get to e-mail. And then, someone else said they couldn't see the network. More and more reports of outages on our floor came in, and then came the people from other floors, rushing to us like zombies, looking for their e-mail, their file server access, their Internet. (Cue agonized scream.) "We rushed back into the switch room where we'd patched the first AP, and saw, to our horror, that all of the lights on all of the switches in that closet were flashing. When we checked the other floors, we saw the same thing. We couldn't log into the switches remotely, and console access was locked up. Finally, we threw up our hands, crossed our fingers and flipped off the power on one floor. That seemed to work. The floor's access came back and seemed normal, so we went to the other floors and cycled the power there, too. Finally, things seemed to be quiet. We checked all our connections. Everything seemed OK, and then someone in the switch room noticed that the first switch began blinking. Slowly, the rest of the switches in the stack followed suit until they were all down again. It finally took powering everything off and powering everything back on--one at a time, from the core out--to get things stable again. We have since removed those switches, and migrated to a significantly more stable manufacturer's switch platform. But neither the switch nor the AP manufacturer could ever explain what could have caused such a catastrophic interaction between a humble access point and our core switching infrastructure." And here you thought "The Exorcist" was scary. What about you? Can you top this tale of IT terror? |