Design a CI/CD pipeline that lets a team deploy many times a day safely. How do you make production changes low-risk with progressive delivery?

Question

Accepted Answer

Deploying many times a day is only safe if each deploy is low risk, so I design for blast-radius control, not just automation. The foundation is a trusted pipeline: every change runs the same gate, unit and integration tests, dependency and security scanning, and produces one immutable artifact. I keep that pipeline under about 15 minutes, because if it takes an hour people batch changes and batch risk. Then I never flip everyone at once. I roll out progressively, a canary at 5 percent of traffic for 10 or 15 minutes, then 25, then 100, and I gate each step on health, watching error rate and p99 latency against the baseline and automatically halting or rolling back on regression. A bad release then reaches 5 percent of users for 10 minutes instead of everyone for an hour. I also decouple deploy from release with feature flags, so code ships dark and a feature is turned on for 1 percent, then 10, and killed instantly without a redeploy. Trade-offs I would name: canary keeps blast radius tiny but is slower and only works if the metrics are good, while blue-green gives an instant cutover and rollback at the cost of running 2 full copies of the infrastructure during the switch, and flags are powerful but accrue config debt, so I put an expiry on each one and clean them up. The point is that fast rollback and a small blast radius make high deploy frequency a safety feature, not a risk.

Answer

A canary looks healthy on its own metrics but is quietly corrupting data that only shows up later. How does your pipeline help, and where does it fall short?

Canary catches fast, observable regressions (errors, latency) but not slow or downstream effects, so health-gating is necessary but not sufficient. The mitigations are good integration tests before the canary, longer bake times for risky changes, monitoring downstream/data-quality signals, and feature flags so you can disable the behavior instantly even after it is fully deployed.

Answer

When would you choose blue-green over a canary?

Blue-green shines when you need instant, all-or-nothing cutover and instant rollback (for example a change that is hard to run in a mixed-version state, like certain schema or protocol changes), and you can afford to run two full environments briefly. Canary is better when you want to limit blast radius and have the per-request metrics to judge a partial rollout.

Design a CI/CD pipeline that lets a team deploy many times a day safely. How do you make production changes low-risk with progressive delivery?

Design a CI/CD pipeline that lets a team deploy many times a day safely. How do you make production changes low-risk with progressive delivery?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with architecture & design

Knowing the answer is not the same as recalling it under pressure