Breaking the network – badly

Ideally you want to avoid breaking the network, but we don’t live in an ideal world.   Hopefully you have change management policies that help reduce the risk, but  you work on extremely complex systems with imperfect information.  You will, eventually, break the network.  Sometimes  you can resolve the issue quickly without much fuss.   Every once in a while though,  you mess up really badly.  You know how this goes:
You are just completing a router config and you type in your final command…..

(Good stuff,  finished, will head home now, it’s been a long day)
hit return
(no response from terminal. that’s weird)
return…return.
(oh no)
return…..return…return.
(no, no, no ….. )
return…..return…return….. return….return
(nooooooo!)
Yes.  That feeling;  like you’ve taken a punch in the stomach;  when you realize you took a branch office off the air or isolated a datacenter from the wrong end of the link.    You may even need to drive for an hour and reboot the router to restore service.  You want to hide, or run away.   Hopefully you realise that the right thing to do is:

Own it

Whether you like it or not, moments like these are pivotal in your network career.   Your reputation will depend on what happens next.   You know most about what happened, and you are in the best position to help get the network back on the air.  Here’s some suggestions;

• Own the issue. Don’t try to hide it or you will prolong the outage.
• Get in touch with your on call engineer or bring in another colleague immediately.  Forget about the slagging you’ll get later.    Letting your colleagues bust their nut troubleshooting an issue you cause while you stay silent is completely unacceptable.
• Tell the other engineer exactly what happened, exactly where you were in the procedure.
• Spin up a con call / trouble ticket etc… as per your company policy.
• Start troubleshooting   [We’ll do a troubleshooting in a later post, we’ll keep it high-level for now].
• Before you go any further…. either you or your partner (preferably you!) should contact your manager.

Management

It’s common for network engineers to downplay the role of executive management when there is a network related crisis.  We’re the techies and we’ll fix this.. Management should stay out of our way, stop asking for updates, and order us pizza .. right?
Ermm…. no.  Well….except the pizza bit, that is actually really helpful.    It’s true that execs can get a little out-of-hand when asking for updates, but they have a job to do too.    If your manager hears about this incident from her own higher-ups before they’ve been briefed by you,  then your situation has worsened considerably.
Your manager will not be happy,  but crisis management is part of their job.   Give your boss time to prepare their message and hopefully your boss will be the one to call their VP or your customers, rather than the other way round.  There is nothing better as a manager to be able to utter the following lines to their VP;   “Yes….our mistake… we’re fixing it now….10 minutes ago….will be fixed in 20 minutes”.   Don’t let your boss be blind-sided by a screaming VP.    There will be a post-mortem and you want your boss on your side.
Provide a timely but brief update to your boss.   If you’re in a crunch then just send a page or a text message.   Return to your fix, and leave your manager  inform their higher-ups. Do whatever it takes, but please don’t ignore your management team.
Your career will thank you later.