SeniorTroubleshooting

The network team applied a change last night. Since then roughly five percent of connections between your web subnet and your data subnet time out, the other ninety five percent are perfectly fine. It is intermittent and nobody can reproduce it on demand. Walk me through how you attack this.

What they are really testing: They want to see if you can debug a partial, non-reproducible failure without flailing, and whether you know that an intermittent network fault usually means one node or one rule is different from the rest.

A real interview question

The network team applied a change last night. Since then roughly five percent of connections between your web subnet and your data subnet time out, the other ninety five percent are perfectly fine. It is intermittent and nobody can reproduce it on demand. Walk me through how you attack this.

What most people say

drag me

Five percent is quite low so I would add retries in the application and keep an eye on it to see if it gets worse.

Retries hide a real fault, the five percent is a signal that something is genuinely different about one path, and burying it means the next change makes it a hundred percent.

The follow-ups they ask next

  • Flow logs show nothing denied, yet connections still time out. Where do you look next?

    Look beyond NSGs: user defined routes, an appliance in the path, SNAT port exhaustion, or the destination refusing connections.

  • The application team wants to add retries and move on. How do you respond?

    Agree retries buy time, but insist the fault is found, because five percent today is a hundred percent after the next change.

  • How do you make this class of change safe next time?

    Network changes go through infrastructure as code with a plan diff, a canary subnet, and a synthetic connectivity test in the pipeline.

What the interviewer is listening for

  • Treats five percent as a clue rather than acceptable noise
  • Actively converts an intermittent fault into a deterministic one
  • Knows subnet and NIC level rules combine into effective rules
  • Leaves behind a synthetic probe so the next drift is caught early

What sinks the answer

  • Papers over the failure with retries and moves on
  • Asks what the change was supposed to do instead of reading flow logs
  • Checks only one side of the connection
  • Chases three hypotheses at once and cannot say which was disproved

If you genuinely do not know

Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.

I have not debugged this exact NSG issue, but here is how I would reason about a five percent failure: find what is different about that five percent, because intermittent almost always means one path is not like the others.

Keep going with troubleshooting

All 336 cloud engineer questions

Knowing the answer is not the same as recalling it under pressure

Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.

Start free