Count clicks with an async analytics pipeline
Continues from the last build: The rate limiter is up and abuse is contained, but every redirect still pays a write to the clicks table before it can send the 302, or worse, that write got dropped from the critical path months ago and nobody has trusted the click counts since.
Two ways this rung starts, and you have probably lived both.
What you'll build
Redirects are decoupled from counting: the app tier fires a lightweight click event onto a message queue and returns the 302 immediately (sub-5ms added latency), a stream consumer processes events at-least-once with idempotent, deduplicated aggregation into a click-counts store, a batch job reconciles from a durable raw-event log for correctness, and the queue has bounded depth with backpressure and a dead-letter path so a slow consumer degrades gracefully instead of taking down the producer.
See how we teach, before you sign up
You don't just get code dumped on you. Every starter file and every solution is explained line-by-line, in plain English. Here's one real file from this project:
# ADR-003: Move click counting off the redirect path ## Status Proposed ## Context The redirect handler currently [describe the synchronous write and the pain: latency spike, lock contention, or a dropped try/except that silently loses data]. Redirect p99 must stay under 10ms. Analytics only needs to be correct within [X minutes] of the actual click, not instantly. ## Decision drivers - Redirect latency budget: <= 10ms p99, click counting must add close to zero. - Delivery semantics: at-least-once is acceptable IF counting is idempotent. - Durability: a raw record of every click must survive a consumer crash. - Backpressure: a slow consumer must never block or crash the producer. ## Options considered 1. Synchronous DB write on redirect path (status quo, rejected: latency + lock contention) 2. Fire-and-forget HTTP call to an analytics service (rejected: no durability, silent loss on network blip) 3. Message queue (Kafka or SQS) + stream consumer + reconciliation batch (chosen) ## Decision Use a message queue between the redirect service and counting logic. The app tier appends a small click event and returns immediately; it never waits on an ack from the counting side beyond the queue's own bounded enqueue call. ## Consequences - Counts are eventually consistent (seconds to low minutes behind real clicks). - At-least-once delivery means the consumer MUST be idempotent or counts will drift high. - A poison message (malformed event) must land in a dead-letter queue, not block the partition. - Queue depth becomes a new metric to alert on; an unbounded queue is a new failure mode we are choosing to accept the risk of and must guard against explicitly.
Reading this file
Redirect p99 must stay under 10ms.The latency budget that the whole design must respect.3. Message queue (Kafka or SQS) + stream consumer + reconciliation batch (chosen)States the chosen option explicitly, not just implied by the rest of the doc.At-least-once delivery means the consumer MUST be idempotent or counts will drift high.Names the consequence that the next milestone's idempotent consumer directly addresses.Queue depth becomes a new metric to alert on; an unbounded queue is a new failure mode we are choosing to accept the risk of and must guard against explicitly.Flags the backpressure risk this rung must design against, not ignore.
Fill in each section; do not skip 'Consequences', that is where the tradeoff actually lives.
That's 1 of 5 explained code blocks in this single project.
The build, milestone by milestone
- 1
Prove the synchronous write is the bottleneck, with numbers
3 guided stepsEvery added component in this ladder has had to earn its place with math. A queue is not free: it adds moving parts, an ops surface, and eventual consistency. If the synchronous write only costs 0.3ms and never contends, you do not need this rung's complexity yet. If it costs double-digit milliseconds under load and occasionally locks a hot row, the queue pays for itself immediately. Do the math first so the rest of the rung is a defended decision, not a reflex.
- 2
Design the click event and the producer side of the queue
3 guided stepsThe producer side is where most 'async analytics' designs quietly reintroduce the synchronous bottleneck they were trying to remove, by making the enqueue call itself block on an ack or retry synchronously with no timeout. The event schema also decides whether idempotent counting is even possible later, since the dedup key has to be created once, at the point of the click, not regenerated on retry.
- 3
Design an idempotent stream consumer and the aggregate store
3 guided stepsAt-least-once delivery is the industry-standard, achievable guarantee for a queue like this; exactly-once end to end is expensive and mostly a myth once you count from a queue into a separate store. The real answer is at-least-once delivery plus an idempotent consumer, so a duplicate delivery is a no-op instead of a double count.
- 4
Add backpressure and a dead-letter queue so a slow consumer cannot take down the pipeline
3 guided stepsAn unbounded queue is not a safety net, it is a deferred outage: if nobody notices consumer lag for an hour during a traffic spike, the queue can grow until it exhausts broker disk or memory, and then the producer side (the redirect path) starts failing too, which is exactly the coupling this rung was built to remove. A dead-letter queue solves a different problem: one malformed or unprocessable event should not block every event behind it on the same partition forever.
- 5
Design the batch reconciliation job and verify counts stay trustworthy
3 guided stepsIdempotent stream counting reduces double-counting risk but does not make the fast aggregate store infallible; a bug, a Redis restart that loses uncommitted state, or a dedup TTL misconfiguration could still cause drift. The raw event log is the source of truth precisely because it is dumb, append-only, and easy to reason about; a batch job that recomputes from it is the safety net that makes the fast path trustworthy.
What's inside when you start
You'll walk away with
This is portfolio-grade. Build it free.
Sign up to unlock every milestone step-by-step, the code skeletons, full reference solutions, and checkable tasks, with your progress saved as you build.
Start building