SeniorSystem Design

Secrets are scattered across environment variables, config files, and a spreadsheet somewhere. Design secrets management for the whole organisation.

What they are really testing: They want to see if you can design for rotation and revocation, not just storage. The real test is whether you have a migration story for secrets that are already leaked into git history and CI logs.

A real interview question

Secrets are scattered across environment variables, config files, and a spreadsheet somewhere. Design secrets management for the whole organisation.

What most people say

drag me

I would put everything in a secrets manager like Vault or AWS Secrets Manager, and the applications read from it instead of environment variables. That way nothing is hardcoded.

It solves storage and stops there. It never mentions how the application authenticates to the store in the first place, how a leaked secret is revoked, what happens when the store is unavailable, or what to do about the secrets already sitting in git history.

The follow-ups they ask next

  • A credential leaked into a public repo an hour ago. Walk me through the next 30 minutes.

    Rotate first, then assess exposure from audit logs, then remove from history. Order matters, and saying rotate first is the signal.

  • How do you rotate a database password without dropping live connections?

    Dual credential rotation, two valid users or a grace window, application reconnect logic, and rotate in a low traffic slot.

  • How do CI pipelines get secrets without storing them?

    OIDC federation from the CI provider to the cloud, short lived tokens per job, no long lived keys in the CI settings.

What the interviewer is listening for

  • Leads with workload identity rather than with a product name
  • Treats rotation and revocation time as the real success metric
  • Has a plan for secrets already in git history and CI logs
  • Names the tier zero dependency they accepted and how they bounded it

What sinks the answer

  • Stops at store the secret in a vault with no auth story for the workload
  • No rotation plan, so a leaked secret lives forever
  • Assumes the secret store never fails
  • Ignores CI and build systems, which is where most leaks actually happen

If you genuinely do not know

Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.

I have not rolled this out organisation wide, but here is how I would reason about it: the hard part is not storage, it is how a workload proves who it is and how fast we can revoke, so I would start with workload identity and short lived credentials.

Keep going with system design

All 336 cloud engineer questions

Knowing the answer is not the same as recalling it under pressure

Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.

Start free