It's one of the unwritten laws of physics: At some time or another, everybody
screws up.
But when IT pros make mistakes, they don't mess around. Entire buildings go
dark. Web sites disappear. Companies grind to a halt. Because if you're going to
mess up, you might as well make it count.
"I always tell my guys, hey, you're gonna do stupid stuff," says Rich
Casselberry, director of IT operations at Enterasys, a networking systems
vendor. "It's OK to do something stupid if you have the wrong information. But
if you do something stupid because you're stupid, that's a problem. The trick is
to not flip out, which only makes it worse, or try to hide it. You need to
figure out how to keep it from happening again."
[ For more adventures in IT mishaps, check out Stupid user tricks 3: IT admin
follies and Stupid QA tricks: Colossal testing oversights ]
We've gathered up some of the more egregious examples from IT pros brave enough
to share their screwups with us. Backups gone bad, people with admin privileges
who probably shouldn't, what can go south when you unplug the wrong equipment --
in some cases, we've obscured their identities to spare them embarrassment;
other geeks, however, are perfectly willing to own up to their youthful
mistakes.
Sure, some of these mishaps are amusing in retrospect. But don't laugh too hard.
We know you've probably done worse.
True IT confession No. 1: The case of the mysterious invisible backup
Our first tale of misadventure involves a longtime IT pro who doesn't want his
real name used, so we'll just call him Hard Luck Harry.
Harry had his share of mishaps when he started out a decade ago at a major
networking equipment maker in the Northeast. There was the time he changed an
environmental variable that broke everything on his company's financial apps,
earning an e-mail from his boss ordering him to "never hack on this system
again." Or the time he crashed the company's core ERP system by overwriting
/dev/tty. Harry says after he accidentally ripped the company's T1 lines out of
the wall with his pager, he was banned from ever reentering the telecom closet.
But the worst one happened after Harry installed an Emerald tape backup system.
Did he bother to read the manual? Please. This was child's play. Just load
install.exe and let the software do its thing.
It seemed to work perfectly. Four hours later, the first backup completed and
everything looked fine.
Fast-forward six months. Harry gets a call late one night at home from one of
his work pals. That night's backup tape is completely blank, the friend tells
him. Worse, the last four weeks of backups are also blank.
As Harry soon discovered, that particular backup program installs in demo mode
by default. Demo mode looked exactly like real mode and even took the same
amount of time as an actual backup, but nothing ever got written to tape -- a
fact that was noted in the manual, which Harry might have seen had he read it.