What makes a good alerting strategy, and how do you avoid alert fatigue?
What they are really testing: Whether you alert on user-facing symptoms rather than every cause, keep alerts actionable, and distinguish paging from ticketing. Bad alerting is a top real-world failure.
A real interview question
What makes a good alerting strategy, and how do you avoid alert fatigue?
What most people say
drag me
“I would set up alerts on CPU, memory, and disk so I know when something is wrong.”
Alerting on every resource metric is exactly how you get fatigue: most fire without user impact and get ignored. Good alerting pages on user-facing symptoms (SLO burn), and each alert must be actionable.
The follow-ups they ask next
Why alert on symptoms rather than causes?
Symptoms (SLO/error/latency) map to user impact and are far fewer; causes are many and most do not affect users. Cause-based alerting floods the pager with noise, while symptom alerts catch what actually matters and you investigate the cause after.
What is a burn-rate alert?
An alert on how fast you are consuming the error budget. Fast burn pages urgently; slow burn can be a ticket. It ties alerting to real, sustained user impact instead of momentary threshold blips.
What the interviewer is listening for
- Alerts on user-facing symptoms/SLOs
- Insists every alert is actionable
- Page vs ticket + burn-rate + runbooks to fight fatigue
What sinks the answer
- Alerts on every resource metric
- Non-actionable/noisy alerts
- No notion of fatigue or page/ticket split
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Alert on [symptoms users feel (SLO/error/latency), not every internal cause]. Make [every alert actionable, if no action, it is a dashboard/ticket, not a page]. Split [page (urgent, user-impacting) vs ticket (non-urgent)]. Fight fatigue with [tuned thresholds, SLO burn-rate alerts, grouping/dedupe, and a runbook per alert]. Test: [if it pages at 3am, must a human act].”
Keep going with observability
Foundation
What is the difference between monitoring and observability?
Foundation
What are the three pillars of observability, and what is each good for?
Foundation
What should you measure first for a service? Explain the golden signals (or RED/USE).
Junior
What are good logging practices for a distributed system?
Junior
What are the main metric types (counter, gauge, histogram), and when do you use each?
Junior
What is distributed tracing, and how does it work?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free