PII & Privacy in LLM Apps

The transcript that should never have left the building

A customer messages your support bot: "Hi, I think I was double-charged. My name is Maria Alvarez, SSN 482-19-7733, and the card ending 4412." Your app does the obvious thing, it stitches the whole conversation into a prompt and ships it to a hosted model to draft a reply. In that one HTTP call, a government identifier and a partial card number just crossed your trust boundary into a vendor you do not control.

Nobody attacked you. There was no breach. You simply forwarded raw user input to a third party, and depending on that vendor's settings, the SSN may now sit in their request logs for 30 days, or worse, in a training pipeline. The scary part: the code that did this looks completely normal. It is a one-line string concatenation.

The principle: minimize, redact, control where data goes

Send the model the least data it needs, strip the identifiers it does not, and always know exactly where that data lands and how long it stays.

Privacy is not a feature you bolt on at the end, it is a property of your data flow. The model is a black box you rent. Once a prompt leaves your process, you have given up direct control over it. So the work is to make sure that whatever leaves is already safe: minimized to what the task needs, with personal identifiers removed or replaced, sent to a destination whose retention and residency you have verified.

The privacy layer, in one picture

The fix is a thin layer that sits between your application and the model. Raw input goes in; a redacted prompt goes to the model; the response comes back; and only sanitized text reaches your logs. The real identifiers never leave your boundary, they stay in a vault keyed by the placeholder tokens you sent instead.

1. 1

   Detect

   Scan the incoming text for PII, names, emails, phone numbers, SSNs, card numbers, addresses, using a recognizer library or a dedicated detection service.

2. 2

   Redact & tokenize

   Replace each hit with a stable placeholder like [PERSON_1] or [SSN_1], and store the real value against that token in an in-memory or short-lived vault.

3. 3

   Call the model

   Send the redacted prompt. The model reasons over placeholders just fine, it does not need the real SSN to write a refund reply.

4. 4

   Re-insert

   When the response comes back, swap any placeholders the model echoed for the real values from the vault, so the user sees a natural reply.

5. 5

   Log safely

   Persist only the redacted prompt and redacted response. The vault mapping is dropped after the request, it never hits durable storage or your log pipeline.

What actually counts as PII

PII (personally identifiable information) is any data that can identify a person on its own or when combined with other data. The obvious ones are direct identifiers; the dangerous ones are the quasi-identifiers that feel harmless alone but pinpoint someone in combination.

• Direct identifiers, full name, SSN / national ID, passport number, email, phone, credit card, bank account, government case numbers.
• Quasi-identifiers, date of birth, ZIP code, gender, job title, employer. Studies show DOB + ZIP + gender alone re-identifies most of a population.
• Sensitive categories, health conditions, biometric data, religion, sexual orientation, political views. These carry stricter legal duties under GDPR Article 9 and HIPAA.
• Indirect / contextual, an account ID, an order number, or even a very specific free-text detail ("the only left-handed pilot in our Reykjavik office") can identify someone.

How to handle the data: four options

Not every app needs the same level of protection. The right choice depends on the sensitivity of the data, your regulatory exposure, and how much the model genuinely needs the real values. Here is the spectrum, from cheapest-and-riskiest to safest-and-heaviest.

Code: detect, redact, call, re-insert

Here is a minimal but real privacy layer in Python. It uses a PII recognizer to find entities, replaces each with a stable token, keeps the mapping in a per-request vault, calls the model with the redacted prompt, then re-hydrates the response. Swap the toy detector for a library like Presidio in production.

import re
from dataclasses import dataclass, field

# --- 1. Detection -----------------------------------------------------------
# In production use a real recognizer (e.g. Microsoft Presidio) that handles
# names, addresses and context. These patterns are illustrative only.
PII_PATTERNS = {
    "EMAIL": r"[\w.+-]+@[\w-]+\.[\w.-]+",
    "SSN": r"\b\d{3}-\d{2}-\d{4}\b",
    "CARD": r"\b(?:\d[ -]*?){13,16}\b",
    "PHONE": r"\b\+?\d[\d ().-]{7,}\d\b",
}


@dataclass
class Vault:
    """Per-request store mapping placeholder tokens -> real values.
    Lives only for the lifetime of one request; never persisted."""
    mapping: dict[str, str] = field(default_factory=dict)
    _counters: dict[str, int] = field(default_factory=dict)

    def tokenize(self, label: str, value: str) -> str:
        # Reuse the same token if we have already seen this exact value.
        for token, stored in self.mapping.items():
            if stored == value:
                return token
        n = self._counters.get(label, 0) + 1
        self._counters[label] = n
        token = f"[{label}_{n}]"
        self.mapping[token] = value
        return token


def redact(text: str, vault: Vault) -> str:
    """Replace every detected PII span with a stable placeholder token."""
    for label, pattern in PII_PATTERNS.items():
        text = re.sub(
            pattern,
            lambda m: vault.tokenize(label, m.group(0)),
            text,
        )
    return text


def rehydrate(text: str, vault: Vault) -> str:
    """Swap any placeholder tokens the model echoed back for real values."""
    for token, value in vault.mapping.items():
        text = text.replace(token, value)
    return text


# --- 2. The guarded model call ---------------------------------------------
def ask_model_safely(user_text: str, call_llm) -> tuple[str, str]:
    vault = Vault()

    # a) scrub before anything leaves the process
    safe_prompt = redact(user_text, vault)

    # b) only the redacted prompt crosses the trust boundary
    raw_reply = call_llm(safe_prompt)            # e.g. an Anthropic / hosted call

    # c) re-insert real values locally so the USER sees a natural reply
    user_reply = rehydrate(raw_reply, vault)

    # d) return BOTH: the real reply for the user, the redacted one for logs
    safe_reply = raw_reply                       # still tokenized -> safe to log
    return user_reply, safe_reply


# --- 3. Logging without leaking ---------------------------------------------
def handle_request(user_text, call_llm, logger):
    vault = Vault()
    safe_prompt = redact(user_text, vault)
    raw_reply = call_llm(safe_prompt)

    # Log ONLY the redacted prompt + redacted reply. The vault is never logged
    # and goes out of scope (garbage-collected) when this function returns.
    logger.info("llm_call", prompt=safe_prompt, reply=raw_reply)

    return rehydrate(raw_reply, vault)

Retention, training opt-out, and data residency

Redaction handles the prompt. But three vendor-side settings decide what happens to whatever data does reach the model, and these are the ones auditors ask about.

Retention

By default many APIs keep request/response data for a window (often ~30 days) for abuse monitoring. Check whether the vendor offers zero data retention for your tier, and whether it must be requested. If your data is sensitive, zero-retention plus redaction is the belt-and-suspenders combination you want.

Training opt-out

Reputable API providers do not train on business API traffic by default, but consumer products and some tiers do. Confirm in writing (the DPA or terms) that your prompts are not used to train models. "We don't train on your data" should be a contractual statement, not a blog post you half-remember.

Data residency

Regulations like GDPR restrict where personal data of EU residents can be processed. If you serve regulated regions, you may need an endpoint that guarantees the data stays in-region (an EU or in-country deployment). Cross-border transfer of PII without the right legal basis is a compliance violation regardless of how good your redaction is.

Common mistakes that cost hours (or a fine)

1. Logging raw prompts. The single most common leak. Your APM, your error tracker, your debug logs, all of them happily archive the SSN you forgot to strip. Redact before you log, never after.
2. No redaction at all. Forwarding user text straight to the model because "the vendor is trusted." Trust plus a contract is not the same as the data never leaving. Minimize first.
3. Training on customer data. Quietly feeding real conversations into a fine-tuning job or eval set without consent or de-identification. Customer prompts are not your training corpus.
4. Ignoring residency. Calling a US endpoint with EU customer PII because it was the default region. The default is almost never the compliant one.
5. Tokenizing but never expiring the vault. A reversible token map that gets persisted forever just relocates the risk, now your vault is the breach target. Keep it ephemeral.
6. Trusting the model to keep secrets. Telling the model "do not repeat the SSN" is not a control. If it never sees the SSN, it cannot leak it.

Takeaways

Where to go next

Privacy is one pillar of building LLM features responsibly. The natural next steps are securing the prompt path against attackers and fitting all of this into a governance program.

• Prompt Injection & LLM Security, how attackers manipulate prompts, and why a leaked PII context makes injection far more damaging.
• Responsible AI: Governance & the EU AI Act, where retention, opt-out, and residency live in a real compliance program.
• Follow the full AI Engineer career path to build LLM apps that are private, secure, and production-ready end to end.