How should a service retry a failed call to a dependency, and what is a retry storm?
What they are really testing: Whether you know retries need exponential backoff plus jitter and caps, and that naive retries amplify an outage into a retry storm (thundering herd). A classic self-inflicted outage.
A real interview question
How should a service retry a failed call to a dependency, and what is a retry storm?
What most people say
drag me
“I would retry the request a few times until it succeeds.”
Immediate, uncapped retries are exactly how a brief dependency blip becomes a retry storm: every client retries at once, multiplying load on an already-struggling service and keeping it down. You need backoff, jitter, caps, and idempotency.
The follow-ups they ask next
Why is jitter important in addition to exponential backoff?
Backoff alone still lets many clients retry at the same moments (synchronized waves). Jitter randomizes the timing so retries spread out, preventing coordinated spikes that re-overload the recovering dependency.
Why only retry idempotent operations?
A retry may duplicate the request. If the operation is not idempotent (a payment, an order), the duplicate causes real damage. Either make it idempotent (idempotency keys) or do not blindly retry it.
What the interviewer is listening for
- Exponential backoff + jitter + caps
- Retries only idempotent/transient
- Names retry storm/thundering herd + circuit breaker
What sinks the answer
- "Just retry until it works"
- No backoff/jitter
- Unaware retries can amplify an outage
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Retry [only transient failures and idempotent operations]. Use [exponential backoff (1s, 2s, 4s) so the dependency can recover], [add jitter so clients do not retry in lockstep], and [cap retries + pair with a circuit breaker to fail fast when it is down]. Naive immediate retries cause [a retry storm/thundering herd that turns a blip into an outage].”
Keep going with reliability
Foundation
What is a Region versus an Availability Zone, and why does the difference matter?
Foundation
What is the difference between vertical scaling and horizontal scaling, and when would you reach for each?
Foundation
What does "highly available" actually mean, and how do you achieve it?
Foundation
Why are stateless services easier to scale and make reliable than stateful ones?
Junior
What is idempotency, and why does it matter for reliable distributed systems?
Junior
What is graceful degradation, and when would you fail open versus fail closed?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free