Responsible AI: Governance & the EU AI Act

The feature that hit a wall

Your team shipped it on a Friday. A slick AI feature that scores loan applicants, flags the risky ones, and routes the rest to instant approval. Conversion went up. The demo got applause. Then, three weeks later, a compliance officer asked a deceptively simple question: "Can you show me why this applicant was rejected, and prove the model wasn't biased against them?"

Silence. There were no decision logs. No record of which model version ran. No bias evaluation. No human in the loop on rejections. The feature wasn't broken, it was *unaccountable*. And under the EU AI Act, credit scoring is high-risk, which means "we'll add logging later" is not a roadmap item, it's a legal exposure. The feature got pulled.

Build for accountability from day one

Build for accountability from day one, because you cannot retrofit a paper trail onto a decision that already happened.

Compliance is not a gate you pass at the end. It's a property you design in, the same way you design in security or observability. The EU AI Act doesn't ask you to make your model *perfect*, it asks you to make it explainable, supervised, tested, and traceable. Those are engineering concerns, and they're cheap when built in and brutally expensive when bolted on.

The picture: from system to audit

Every regulated AI system follows the same pipeline. You classify the system by risk, that classification dictates a set of required controls, and those controls feed an audit trail that proves, after the fact, that you did the right things. Here's the shape of it:

1. 1

   Classify the system

   Before writing a line of feature code, decide which risk tier the use case lands in. Credit, hiring, biometric ID, and critical infrastructure are high-risk by default.

2. 2

   Derive the required controls

   The tier dictates obligations: a chatbot needs disclosure; a hiring filter needs bias testing, human oversight, logging, and documentation.

3. 3

   Wire transparency

   Tell users they're interacting with AI, and ship a model card describing intended use and limitations.

4. 4

   Insert human oversight

   For high-risk decisions, a person must be able to review, interpret, and override the model's output before it takes effect.

5. 5

   Log every decision

   Capture input, model version, output, confidence, and the human action taken, immutably, with retention that matches the regulation.

6. 6

   Make it auditable

   Ensure all of the above is queryable. An auditor (or your future self) must reconstruct any single decision months later.

The EU AI Act risk tiers

The Act is risk-based: the obligations scale with how much harm the system can do. Most software falls in the bottom two tiers and barely needs to change. The work concentrates in high-risk, which carries the full weight of the regulation. Know your tier before you estimate the engineering cost.

Compliant-by-design: logging + the model card

Two artifacts do most of the heavy lifting in practice: a decision log (the audit trail, per-inference) and a model card (the documentation, per-model). Here's a minimal decision-log record, note that it captures the model version and the human action, not just the model's output:

# One immutable record, written on every high-risk inference.
decision_id: dec_7f3a9c21
timestamp: 2026-06-06T09:14:22Z
system: loan-risk-scorer
model:
  name: credit-risk
  version: v3.2.1          # exact artifact that ran, never "latest"
  card_uri: s3://models/credit-risk/v3.2.1/MODEL_CARD.md
subject:
  id_hash: sha256:9b1c...   # pseudonymized, never raw PII
input_features:
  income_band: "40-60k"
  employment_months: 28
  existing_debt_ratio: 0.31
output:
  decision: "refer"         # approve | refer | decline
  score: 0.62
  confidence: 0.81
  top_factors:              # explainability: why this output
    - { feature: existing_debt_ratio, contribution: 0.34 }
    - { feature: employment_months, contribution: -0.19 }
human_oversight:
  required: true
  reviewer: agent_4471
  action: "overridden"      # accepted | overridden | escalated
  final_decision: "approve"
  rationale: "Recent promotion not reflected in income_band"
retention_until: 2036-06-06  # match the regulated retention window

The model card is the human-readable companion. Keep it in the repo next to the model so it versions with the artifact. A minimal template:

# Model Card, credit-risk v3.2.1

## Intended use
Scores consumer loan applications to **assist** human underwriters.
Not for automated final decisions. Not validated for business lending.

## Risk classification
EU AI Act tier: **High-risk** (Annex III, creditworthiness assessment).

## Training data
- Source: 2019-2025 internal loan book (820k records)
- Protected attributes excluded from features; retained for bias eval only
- Known gap: under-represents applicants under 25 (~4% of data)

## Performance
- AUC 0.84 overall
- Evaluated across age, gender, and region cohorts (see Bias eval below)

## Bias evaluation
| Cohort | Approval rate | Disparate impact ratio |
|--------|---------------|------------------------|
| Group A (reference) | 41% | 1.00 |
| Group B | 36% | 0.88 |  <!-- > 0.80 = passes 4/5ths rule -->

## Limitations & failure modes
- Degrades on thin-file applicants (< 6 months history)
- Do not use outside EU consumer-credit context

## Human oversight
Every `decline` and `refer` is reviewed by a trained underwriter
who can override. Overrides are logged with rationale.

## Contact / owner
team-lending-ml@company.example · last reviewed 2026-06-01

Bias testing and human oversight, explained

Bias testing: measure, don't assume

A model trained on historical data inherits historical bias. Bias testing means slicing your evaluation by protected cohorts (age, gender, ethnicity, region) and checking whether outcomes differ in ways you can't justify. A common yardstick is the four-fifths rule: if a group's favorable-outcome rate is below 80% of the reference group's (a disparate-impact ratio under 0.80), you have a flag worth investigating.

• Slice on outcomes, not just accuracy. A model can be 90% accurate overall and still reject one group at twice the rate.
• Test at training time and in production. Drift reintroduces bias as the input distribution shifts.
• Keep protected attributes for evaluation only, pull them out of the feature set, but retain them so you *can* measure fairness.
• Record the result in the model card. An unmeasured model is, for audit purposes, a biased one.

Human oversight: a real override, not a rubber stamp

For high-risk systems, the Act requires meaningful human oversight, a person who can understand the output, interpret it correctly, and override it before it has real-world effect. "Human-in-the-loop" only counts if the human can actually intervene and has the information and authority to do so. A dashboard that shows a score but offers no way to reverse it is theater, not oversight.

• **Give the reviewer the *why*.** Surface the top contributing factors, not just a number.
• Make override the path of least resistance, and log the rationale every time.
• Guard against automation bias, if reviewers approve 100% of model outputs, your oversight isn't working.
• Define the escalation route for cases the reviewer can't resolve alone.

Common mistakes that cost hours (or millions)

1. No audit trail. You logged the request, but not the model version, the input features, or the human action. When the auditor asks "why this decision," you can't reconstruct it. Log per-inference, immutably, from day one.
2. No human oversight on high-risk. A fully automated reject/approve loop on credit or hiring is the single fastest way into the penalty bracket. High-risk needs a real override.
3. No bias evaluation. Shipping without slicing outcomes by cohort. "We don't use race as a feature" does not mean the model is unbiased, proxies leak. Measure it.
4. "Latest" instead of a pinned version. If the log says model: latest, you can never prove which model made a past decision. Pin the exact artifact and store the card URI.
5. Model card as a one-time doc. Written once, never updated, drifted from the deployed model. Version it with the artifact and regenerate the metrics in CI.
6. Treating disclosure as optional. Limited-risk chatbots and AI-generated content must tell the user. It's a one-line UI change, just do it.
7. Classifying late. Discovering you're high-risk *after* you've built the feature means re-architecting under deadline. Classify before you design.

Takeaways

Where to go next

Governance is one pillar of shipping AI responsibly; it sits alongside privacy and operations. These pair directly with this article:

• PII & Privacy in LLM Apps, how to keep personal data out of prompts, logs, and training sets (the data-governance half of compliance).
• LLMOps: Productionizing LLM Apps, versioning, evaluation, and the deployment pipeline where you enforce these controls.
• Follow the AI Engineer career path to build the full skill set end to end.

Start small: pick one AI feature you've shipped, classify its risk tier, and ask whether you could reconstruct a single decision from six months ago. If the answer is no, you've found your first sprint.