Pessimistic Engineering
I don’t know about you, but I am regularly told “You worry too much” or “You don’t need to worry about that”. Sometimes its “What are the chances of that ever happening?”. These are things that Ive heard from many people over the years and the best I can come up with is:
“That’s what you pay me for! Im here to think of the worst case scenario and then mitigate against that.”.
This is usually followed by confused looks from those around me who do not seem to grasp what I am getting at here.
The way I see it, it is my job to constantly be thinking about worst case scenarios.
- “What happens if we lose this device/site/cable?”
- “What happens if we all backups are lost?”
- “What if…”
These sorts of questions are exactly the reason why I fell in love with Networking as a discipline within IT. The very fact that I have the ability to build redundant systems that take serious effort to bring down draws me deeper in (assuming Im given appropriate budget 😉 ),
Why do I build many of my networks like a Service Provider network? Because I have found that these basic design principles are usually the most robust. Configurations utilising OSPF to carry core routing information and BGP to provide end user routes stands up to some serious beating – and it is extensible too!
Why do I cry when I hear vendors pronouncing “With our new Wonder Fabric Technology you can now turn of Spanning Tree”? I cry because I feel that this is sending the wrong message. I have a whole other post coming on that topic, but please people don’t just turn off spanning tree. Are all your edge ports protected? Can you ensure that nobody will ever mis-cable? (And don’t even get me started on VMWare’s view about filtering BPDUs!)
Why do I prefer two stand alone systems providing redundant network services over a single HA unit? Devices redundant power, RE’s and line cards, but with a shared management plane are still susceptible to risk of incorrect configuration causing a service interruption. Switch Stacks, Virtual-Chassis, VSS and what ever other similar technology all suffer from this problem. I would rather a technology such as Multi-Chassis Link Aggregation, Virtual Port Channels, or even utilising VRRP/HSRP or anycasted Services to provide the desired network redundancy. Sometimes this is “harder”, but again – This is what you pay me for 🙂
Mop and bucket
While I know that I am pre-disposed to the negative and pessimistic tendencies and views, but am I the only one who feels that “Worst Case Scenario Thinking” is one of the prime reasons people pay us? My wife could easily plug in a couple of cables and “make intarwebz happen”, as is proven by the millions of home users CPE, but true network design and redundancy comes from thinking about the worst that can happen and how to mitigate against these risks.
Im curious as to the thoughts of those of you out there.