CAREER – Managing your mistakes

CAREER – Managing your mistakes

Breaking the network – badly

Ideally you want to avoid breaking the network, but we don’t live in an ideal world.   Hopefully you have change management policies that help reduce the risk, but  you work on extremely complex systems with imperfect information.  You will, eventually, break the network.  Sometimes  you can resolve the issue quickly without much fuss.   Every once in a while though,  you mess up really badly.  You know how this goes:
You are just completing a router config and you type in your final command…..

(Good stuff,  finished, will head home now, it’s been a long day)
hit return
(no response from terminal. that’s weird)
(oh no)
(no, no, no ….. )
return…..return…return….. return….return
Yes.  That feeling;  like you’ve taken a punch in the stomach;  when you realize you took a branch office off the air or isolated a datacenter from the wrong end of the link.    You may even need to drive for an hour and reboot the router to restore service.  You want to hide, or run away.   Hopefully you realise that the right thing to do is:

Own it

Whether you like it or not, moments like these are pivotal in your network career.   Your reputation will depend on what happens next.   You know most about what happened, and you are in the best position to help get the network back on the air.  Here’s some suggestions;

  • Own the issue. Don’t try to hide it or you will prolong the outage.
  • Get in touch with your on call engineer or bring in another colleague immediately.  Forget about the slagging you’ll get later.    Letting your colleagues bust their nut troubleshooting an issue you cause while you stay silent is completely unacceptable.
  • Tell the other engineer exactly what happened, exactly where you were in the procedure.
  • Spin up a con call / trouble ticket etc… as per your company policy.
  • Start troubleshooting   [We’ll do a troubleshooting in a later post, we’ll keep it high-level for now].
  • Before you go any further…. either you or your partner (preferably you!) should contact your manager.


It’s common for network engineers to downplay the role of executive management when there is a network related crisis.  We’re the techies and we’ll fix this.. Management should stay out of our way, stop asking for updates, and order us pizza .. right?
Ermm…. no.  Well….except the pizza bit, that is actually really helpful.    It’s true that execs can get a little out-of-hand when asking for updates, but they have a job to do too.    If your manager hears about this incident from her own higher-ups before they’ve been briefed by you,  then your situation has worsened considerably.
Your manager will not be happy,  but crisis management is part of their job.   Give your boss time to prepare their message and hopefully your boss will be the one to call their VP or your customers, rather than the other way round.  There is nothing better as a manager to be able to utter the following lines to their VP;   “Yes….our mistake… we’re fixing it now….10 minutes ago….will be fixed in 20 minutes”.   Don’t let your boss be blind-sided by a screaming VP.    There will be a post-mortem and you want your boss on your side.
Provide a timely but brief update to your boss.   If you’re in a crunch then just send a page or a text message.   Return to your fix, and leave your manager  inform their higher-ups. Do whatever it takes, but please don’t ignore your management team.
Your career will thank you later.

7 thoughts on “CAREER – Managing your mistakes

  1. Great post and great blog! Just found it today!
    I hope you don’t mind, I just emailed a link of this post to my entire team…
    I sit in the techs corner, but I do have a nice perspective of the management task, and you really nailed it on the need to keep the manager informed and on your side… If you don’t, sooner or later he’ll get an angry call from VP/customer and if he doesn’t have the information, we will feel betrayed and will resent it….
    However, if he had the time to prepare himself for the speech, his primary role will be to support and defend it’s subordinates…

    1. João,
      Thanks so much for the kind words, it’s great to hear from fellow engineers who have had the same experiences. I too have been on the other side of the management ‘divide’. Great to hear your feedback.
      John Harrington.

    1. Thanks Nina. Yeah, you see a few people trying to hide their mistakes and it’s generally doesn’t work. I’ve found it to be less hassle to just fess up.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.