Survive flaky upstreams: timeouts, retries and breakers
Continues from the last build: Order events flow through an outbox and webhooks deliver with signatures and retries.
The outbox and webhook work from the last rung is solid, merchant systems get their order events, signed, deduped, retried with backoff.
What you'll build
The API never hangs on a slow payment provider again. Every outbound call has a connect budget and a total budget, retries only happen for calls that are provably safe to repeat and carry the provider's idempotency key, a circuit breaker opens after repeated failures and fails fast instead of queueing requests behind a dead upstream, and when the provider is genuinely down the API degrades honestly to pending_payment instead of hanging or lying about success. A chaos test with an injectable stub provider proves the p95 stays under 500ms at a 30% failure rate.
See how we teach, before you sign up
You don't just get code dumped on you. Every starter file and every solution is explained line-by-line, in plain English. Here's one real file from this project:
import { Agent, fetch as undiciFetch } from 'undici';
const PAYMENT_PROVIDER_URL = process.env.PAYMENT_PROVIDER_URL ?? 'http://localhost:4100';
// TODO milestone 1: tune these budgets
const paymentAgent = new Agent({
connect: { timeout: 2000 },
bodyTimeout: 3000,
headersTimeout: 3000,
});
const TOTAL_BUDGET_MS = 5000;
export interface ChargeRequest {
orderId: string;
amountCents: number;
currency: string;
idempotencyKey: string;
}
export async function callProvider(path: string, body: unknown) {
const signal = AbortSignal.timeout(TOTAL_BUDGET_MS);
return undiciFetch(`${PAYMENT_PROVIDER_URL}${path}`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
signal,
dispatcher: paymentAgent,
});
}
// TODO milestone 2: wrap callProvider in chargeWithRetry with backoff + jitter
// TODO milestone 3: wrap chargeWithRetry in a CircuitBreaker instance and export itReading this file
connect: { timeout: 2000 },Connect budget, separate from the total call budget below.const TOTAL_BUDGET_MS = 5000;Hard ceiling on the entire call, adjust and justify the number in milestone one.dispatcher: paymentAgent,Required for the Agent's timeouts to actually apply to this fetch call.// TODO milestone 2: wrap callProvider in chargeWithRetry with backoff + jitterPoints at exactly where the retry wrapper from milestone two plugs in.
Timeouts are already wired, the retry wrapper and breaker are the TODOs this rung fills in.
That's 1 of 8 explained code blocks in this single project.
The build, milestone by milestone
- 1
Put a clock on every call to the payment provider
5 guided stepsOne unbounded call does not just make one request slow, it ties up a connection and an event loop turn while it waits, so the next requests queue behind it and the whole API looks down even though your own code and database are fine. A hard budget turns an unbounded hang into a bounded, predictable failure you can then retry or fail fast on.
- 2
Retry only the calls that are safe to retry
6 guided stepsRetrying a non-idempotent charge with a new idempotency key each time is how customers get charged twice. Retrying every kind of failure, including a 400 for insufficient funds, wastes calls on something that will never succeed. The safe rule is narrow: retry only transport failures and known-transient provider errors, and always send the same idempotency key so the provider can deduplicate on its side even if your retry and the original charge both landed.
- 3
Wrap the payment client in a circuit breaker
6 guided stepsWithout a breaker, an outage means every single request still pays for maxAttempts timeouts before it gives up, so your API stays slow even though you already know the provider is down. A breaker remembers recent failures and, once the threshold is crossed, stops calling the provider entirely for a cooldown window, failing in microseconds instead of seconds, then cautiously lets one probe through to see if the provider has recovered.
- 4
Degrade honestly instead of hanging or lying
6 guided stepsA customer who gets a fake 'paid' response when the charge never happened will ship a product you never got paid for. A customer stuck waiting 30 seconds for a response you already know will fail is a worse experience than an honest 202 with a URL they can poll. pending_payment is the truth: you accepted the order, the charge is unresolved, here is where to check.
- 5
Chaos-test it: prove p95 holds at 30% failure
6 guided stepsCode review cannot prove a resilience story, only a running test against real failure injection can. A chaos test that dials the stub to 30% failures and multi-second latency, then measures your own p95, is the only artifact that proves the timeout budgets, retry rules, and breaker actually compose correctly under load rather than just passing unit tests in isolation.
What's inside when you start
You'll walk away with
This is portfolio-grade. Build it free.
Sign up to unlock every milestone step-by-step, the code skeletons, full reference solutions, and checkable tasks, with your progress saved as you build.
Start building