A recent article on Network World talks about the top 10 dumbest mistakes network managers make. It's interesting that at least 5 of the top 10 mistakes documented by Network World are solved and purely automated with NetMRI. They also forgot to add the #1 cause of network problems - people making changes and not realizing their mistakes - which is also automatically picked up by NetMRI, showing who made the change, where, when, what the change was, and what the impact of that change was on the overall health of the network.
So, what are some of the results of these mistakes? I recently talked to some of our system engineers and gathered some interesting stories about organizations that decided to purchase NetMRI because of a blunder. Here are some of the interesting blunders we've seen in the past (names not shared to protect the innocent):
Large chocolate manufacturer: They had to call the plumbers once because the guy that took out the network also took out the applications that controlled the factory machines. The applications were down long enough that the chocolate had hardened inside the pipes...hence the plumber needed to be called.
Large Investment Bank: A network admin made a minor change that took out their stock trading floor for hours. Ouch! Since then, no one is allowed to touch any part of the network during business hours.
One of the top biggest banks in the US failed to process the minimum required number of checks during a given 24 hours because someone had made a network change and took out the check processing applications. They faced hefty fines from the US government. The poor guy who made the change is no longer there.
African Telecom company: In an African country the network went out for hours because they didn't bother to secure the generator that powered the datacenter from thieves who stole the gasoline from the generator. Doh!
And another company noticed that a particular router would reboot at about the same time every night. No one could figure out why it would do that, until one of the network guys decided to stay late and see what happens to it physically. That's when he noticed that the cleaning people would come in late at night, unplug the router and plug in their vacuum cleaner. And then plug it back in when done. :)
What are some of the interesting networking blunders you've seen in your career? Please share!
- yh