Security Incident Response and Digital Forensics: What to Do When You Are Breached

When the alert is an attacker, not a bug

At 02:14 a GuardDuty finding fires: an IAM role you have never seen calling AssumeRole from an IP in a country you do not operate in. Your first instinct, honed by years of on-call, is to fix it: kill the session, rotate the key, restart the service, go back to bed. That instinct is exactly wrong. This is not a service that broke. This is a person who got in, and the actions you take in the next ten minutes either preserve the truth or destroy it.

SRE incident response asks 'how do we restore service?'. Security incident response asks a harder set of questions first: 'is the attacker still here? what did they touch? what did they take? and can I prove it?'. Restoring service too fast can re-expose you, tip off the adversary, or wipe the very logs that answer those questions.

The mental model: you are a first responder at a crime scene

An incident is a confirmed adverse event with impact; an event is anything observable. Most events are noise. The job is to decide which events are incidents, and to not contaminate the scene while you do.

Hold this frame for the whole article. Every controversial call, should I isolate or watch? reboot or snapshot? notify or wait?, gets easier when you ask 'what would a careful first responder do?'.

The IR lifecycle (NIST 800-61) is a loop, not a line

NIST splits incident handling into phases. The trap is reading them as a one-way checklist. In a real breach you bounce between detect/analyze and contain/eradicate repeatedly as you discover more, and every incident feeds prepare for the next one. Draw it as a loop.

1. 1

   Prepare

   Before anything happens: centralized logs you cannot tamper with, a documented IR plan, contact lists, pre-staged forensic tooling, and rehearsed tabletops. The work you skip here is the work you regret at 02:14.

2. 2

   Detect & Analyze

   Triage the event. Is it real? What is the scope and blast radius? Establish a timeline and an initial set of indicators of compromise. Declare an incident and assign an incident commander.

3. 3

   Contain

   Stop the spread. Short-term containment buys time (isolate a host); long-term containment is a sustainable holding pattern while you prepare to eradicate. This is where the isolate-vs-observe trade-off lives.

4. 4

   Eradicate

   Remove the attacker's access entirely: revoke credentials, kill persistence (cron jobs, new IAM users, backdoored AMIs), patch the entry vector. Half-eradication means they come back.

5. 5

   Recover

   Rebuild from known-good, restore service, and watch closely, heightened monitoring on the affected systems for days, because attackers often return to test whether you actually closed the door.

6. 6

   Lessons Learned

   Within a week or two, run a blameless post-mortem. What let them in, what slowed detection, what was missing in prepare. Feed every gap back into the loop.

Containment is a trade-off: isolate now vs watch the attacker

The most contested decision in a breach is *when* to pull the plug. Isolate immediately and you stop damage, but you tip off the attacker, who may have other footholds you have not found, and they may detonate (wipe, ransom, exfil-dump) on the way out. Observe quietly and you map the full intrusion, but every minute the attacker is live is more risk. There is no universally right answer; there is a right answer for *this* incident.

Forensic evidence: what each source actually reveals

Reconstructing an attack is correlation across sources, because no single log tells the whole story. CloudTrail shows you the API calls but not the packets; flow logs show the connections but not the payload; EDR shows process behavior on the host; memory shows what disk never will. Know what each gives you before you need it.

Indicators of compromise (IOCs) are the breadcrumbs you pivot on across all of these: an attacker IP, a malicious file hash, a rogue IAM principal, a suspicious domain, a user-agent string. Find one IOC in CloudTrail, then go hunt it in flow logs and EDR to expand the picture.

Hunting an IOC and preserving evidence (hands on)

Suppose your IOC is a suspicious external IP that performed an AssumeRole. Start in CloudTrail (queried via Athena) to find every API call from that source and which roles it touched.

-- Find all AssumeRole and credentialed calls from a suspect IP
-- (CloudTrail logs queried via Athena over the partitioned S3 table)
SELECT
  eventtime,
  useridentity.arn       AS principal,
  eventname,
  sourceipaddress,
  requestparameters,
  errorcode
FROM cloudtrail_logs
WHERE sourceipaddress = '203.0.113.66'
  AND eventtime >= '2026-06-05T00:00:00Z'
ORDER BY eventtime ASC;

-- Then pivot: which roles were assumed, and what did the
-- resulting temporary sessions do afterward?
SELECT
  useridentity.arn AS assumed_role,
  eventname,
  count(*)         AS calls
FROM cloudtrail_logs
WHERE useridentity.type = 'AssumedRole'
  AND useridentity.sessioncontext.sessionissuer.username = 'deploy-bot'
  AND eventtime >= '2026-06-05T00:00:00Z'
GROUP BY 1, 2
ORDER BY calls DESC;

Once you have identified a compromised EC2 instance, your goal is to capture evidence before touching the host. The cardinal rule: snapshot, then hash, then analyze a *copy*, never the original. Snapshotting an EBS volume is read-only and does not disturb the running instance.

#!/usr/bin/env bash
set -euo pipefail

INSTANCE_ID="i-0abc123compromised"
CASE="IR-2026-0605"

# 1. Tag and isolate the network WITHOUT terminating.
#    Swap to a quarantine SG that denies all egress but keeps
#    the instance running so memory stays intact.
aws ec2 modify-instance-attribute \
  --instance-id "$INSTANCE_ID" \
  --groups sg-0quarantinedenyall

# 2. Find the attached volumes.
VOLS=$(aws ec2 describe-volumes \
  --filters "Name=attachment.instance-id,Values=$INSTANCE_ID" \
  --query 'Volumes[].VolumeId' --output text)

# 3. Read-only snapshot of each volume (does NOT modify the source).
for VOL in $VOLS; do
  aws ec2 create-snapshot \
    --volume-id "$VOL" \
    --description "$CASE forensic image of $VOL" \
    --tag-specifications \
      "ResourceType=snapshot,Tags=[{Key=case,Value=$CASE},{Key=evidence,Value=true}]"
done

# 4. When you later restore a snapshot to a forensic volume,
#    hash it immediately to anchor the chain of custody.
#    sha256sum /dev/xvdf  >  ${CASE}_xvdf.sha256
echo "Snapshots created. Capture memory via SSM/EDR before any reboot."

Chain of custody and breach notification

Chain of custody is the documented, unbroken record of who handled each piece of evidence, when, and what they did to it. It is what makes evidence trustworthy, to your own analysts, to leadership, to regulators, and possibly to a court. Hash every image at capture (and re-verify the hash later to prove it was not altered), log every access, and store originals write-once. If you cannot say who touched the snapshot and when, its evidentiary value collapses.

Breach notification is a legal obligation with hard clocks, and engineers are usually the ones who surface the facts those clocks depend on. You do not have to memorize the law, but you must know the timelines exist so you escalate fast enough.

Blameless post-mortems and tabletop exercises

After recovery, run a blameless post-mortem, same discipline as an SRE outage review, with a security lens. The point is never 'who clicked the link'. Humans will always click links; the system should survive that. Ask instead: why did one phished credential grant this much access? why did detection take six hours? why were the logs we needed not retained? Blame hides the systemic gaps; safety surfaces them.

Tabletop exercises are how you find those gaps *before* a real breach. Gather the team, narrate a scenario ('an engineer's laptop is compromised and an AWS access key is now active from Eastern Europe'), and walk through your response out loud. You will discover the runbook is stale, nobody knows who can revoke prod keys at 3am, and your CloudTrail logs roll off after 14 days. Far cheaper to learn that in a conference room than during the real thing.

• Run tabletops at least quarterly, rotating scenarios: ransomware, insider, cloud key leak, supply-chain compromise.
• Include non-engineers, legal, comms, leadership, because notification and messaging are part of the response.
• End every tabletop with concrete action items and owners, then feed them into prepare.

Common mistakes that cost you the case

1. Rebooting or terminating the host to 'clean it up' before capturing memory and disk, you delete the evidence you most need.
2. Logging into the compromised box with admin tools and poking around, which overwrites timestamps, drops new artifacts, and contaminates the scene.
3. Containing too early on a sophisticated actor and tipping them off before you have mapped their other footholds, or containing too late on ransomware.
4. Treating it like an SRE outage: rushing to restore service re-exposes the same vulnerability and wipes the trail.
5. No centralized, tamper-resistant logs, attackers delete local logs, so if CloudTrail/flow logs are not shipped to a locked, separate account, you are blind.
6. Skipping chain of custody, un-hashed, undocumented evidence may be worthless when it matters most.
7. Forgetting the notification clock until day four, turning a contained breach into a compliance failure.
8. A blameful post-mortem that scapegoats the person who clicked, so nobody reports the next incident and the real gaps stay open.

Takeaways and where to go next

Incident response is downstream of preparation. The single highest-leverage investment is making sure you *can* answer 'what happened?' when the time comes, which is a logging and detection problem long before it is a forensics problem.

• Build the visibility that makes forensics possible: security logging and monitoring.
• Reduce the surface attackers exploit in the first place: threat modeling.
• Borrow the operational muscle, commander, comms, timelines, from incident management and on-call.
• Practice the mechanics on the Linux lab and networking lab so log-hunting and host triage are reflexes, not first-time fumbles.