Secrets Management: Keeping Keys, Passwords, and Tokens Out of the Wrong Hands

The 3 a.m. cloud bill

A developer pushes a quick fix on a Friday evening. Buried in the diff is a config file with an AWS access key, committed "just to get it working." The repo is public. Within minutes, automated bots scraping GitHub find the key. By Saturday morning the account has spun up hundreds of GPU instances mining cryptocurrency, and the bill reads five figures. The developer did not get hacked through some clever exploit. They handed over the key themselves and pushed it to the internet.

This is not a rare horror story. Scanners watch public Git pushes in real time, and a leaked credential is often abused before the developer has even noticed it is gone. Secrets are the single highest-leverage thing to protect, because one leaked key can bypass every other control you built.

What counts as a secret

A secret is any value that grants access or proves identity and would cause harm if a stranger held it: API keys, database passwords, OAuth tokens, TLS private keys, signing keys, SSH keys, webhook signing secrets. The defining trait is simple, if it leaks, someone can act as you. Treat anything matching that test as a secret, not as ordinary config.

A secret is only secret while exactly the systems that need it can read it, and nothing else can.

That sentence is the whole discipline. Everything below, where you store secrets, how you hand them to apps, how often you change them, who can read them, is just making that one sentence true and keeping it true.

How secrets should actually flow

The goal is that a secret never lives in your codebase, your container image, or your Git history. Instead the app fetches it at startup or runtime from a dedicated secret manager, holds it in memory, and never writes it down. Behind the scenes a rotation loop keeps swapping the stored value for a fresh one.

1. 1

   App boots with no secret

   The image and the repo contain zero credentials. There is nothing to leak even if the image is pulled or the repo is cloned.

2. 2

   App proves who it is

   It authenticates to the secret manager using its workload identity, an IAM role, a Kubernetes service account, or a short-lived token, not a stored password.

3. 3

   Manager checks the policy

   The secret manager confirms this identity is allowed to read this specific secret, then returns it over an encrypted channel.

4. 4

   Secret is injected at runtime

   The value lands in the process's memory (or a tmpfs-backed env var). It is never written to disk, the image, or version control.

5. 5

   Rotation runs in the background

   On a schedule, the manager issues a fresh value and expires the old one. Apps re-fetch and keep working, a leaked old copy becomes useless.

Where NOT to put secrets (and where to)

Most leaks come from picking the wrong storage location, not from sophisticated attacks. Here is how the common options actually compare on the three things that matter: how exposed the secret is, whether you can rotate it cleanly, and whether you can see who used it.

Reading a secret the right way

The code that consumes a secret should fetch it from the environment or a manager client, never embed the literal. The example below reads a database password from a secret manager with an env-var fallback for local dev, and fails loudly if it is missing rather than silently using a blank value.

import os
import boto3
from functools import lru_cache


@lru_cache(maxsize=1)
def get_db_password() -> str:
    # Local dev: read from an env var that is NEVER committed.
    local = os.environ.get("DB_PASSWORD")
    if local:
        return local

    # Production: fetch from AWS Secrets Manager at runtime.
    secret_id = os.environ["DB_SECRET_ID"]  # just a name, not the value
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=secret_id)
    return resp["SecretString"]


def connect():
    password = get_db_password()
    # ... open the connection; never log `password`
    return password  # used in-memory only


# NEVER do this:
# DB_PASSWORD = "hunter2-prod-9f3a"   # hardcoded, leaks on first commit

Notice three things: the value of the secret never appears in the file (only the secret's *name*, DB_SECRET_ID), the result is cached so you do not hammer the manager on every call, and there is no print(password) anywhere, secrets in logs are one of the most common silent leaks.

Rotation and least-privilege

Two practices turn "we have a secret manager" into "we are actually safe." Rotation means every secret has an expiry; the manager issues a new value and retires the old one on a schedule, so a copy stolen six months ago is already dead. The gold standard is *dynamic secrets*, credentials minted on demand that live for minutes, not months.

Least-privilege access means each identity can read only the exact secrets it needs. The billing service has no business reading the email provider's API key. Scope policies per-service so a single compromised workload exposes one secret, not the whole vault. Pair this with the audit log: if you cannot answer "who read this secret and when," you cannot tell whether a leak has been used.

Detecting leaked secrets

Assume a secret will eventually slip through. Your job is to catch it fast. Run a secret scanner in pre-commit hooks and in CI so a credential is blocked before it ever reaches the remote, and enable your Git host's push protection (GitHub and GitLab both scan pushes for known token formats).

# Block secrets before they are committed (run as a pre-commit hook)
gitleaks protect --staged --verbose

# Scan full history for anything already leaked
gitleaks detect --source . --report-path leaks.json

# If a real secret is found in history, ROTATE it first,
# then purge from history (e.g. git filter-repo), deleting the
# file in a new commit does NOT remove the old value.

Common mistakes that cost hours (or accounts)

1. Committing secrets to Git. Even a deleted .env lives forever in history. Once pushed, treat the secret as compromised and rotate it, do not just remove the file.
2. Never rotating. A secret that has not changed in two years has likely been copied into someone's notes, a Slack message, or a screenshot. Set an expiry on everything.
3. Granting broad access. One "admin" policy that can read every secret turns any single compromised service into a full breach. Scope per-service.
4. Leaking secrets in logs. print(token), verbose error traces, and crash dumps quietly ship credentials to your log aggregator, which often has far weaker access controls than your vault.
5. Putting secrets in container image layers or CI variables in plaintext. Images get pushed to registries; CI logs get shared. Inject at runtime instead.

Takeaways

Where to go next

Secrets management is one pillar of a secure delivery pipeline. Build the surrounding picture next:

• Start with the fundamentals: What Is Application Security? frames where secrets fit among the broader threats.
• Then extend the discipline to your build inputs: Securing the Software Supply Chain covers dependencies, provenance, and signing.
• Practice the delivery mechanics in the CI/CD lab, where injected secrets and pipeline variables come to life.
• See how this fits a full role on the DevOps Engineer path.