Digital Forensics and Evidence Preservation

At 02:17 UTC, the response team has enough evidence to know that an unapproved release reached production. A deployment identity created it, a newly issued credential invoked it, and one tenant-read event followed. The affected pod is still running. Its temporary filesystem, connections, mounted identity, and process memory may explain whether the read was an accident, an intrusion, or the beginning of something larger.

The fastest containment action is to delete the pod and revoke everything associated with it. That action may also erase the only volatile record of what ran. Waiting, however, may allow another read. The forensic decision is therefore not “evidence or containment.” It is which facts can be captured safely before the next change, which change must happen now, and which uncertainty the team accepts.

Engineers do not need to conduct the whole investigation. They do need to keep the first response from destroying it.

Decide from volatility and harm

The commander gives one person authority over preservation. That evidence owner asks two questions before each destructive action: what disappears if we do this, and what harm continues if we wait? The answers determine the order.

In the unapproved-release incident, identity and cloud control-plane records already exist outside the pod, but their ordinary retention and administrative protections still need confirmation. The pod’s process tree, connections, open files, temporary data, and memory will disappear on deletion. The deployed image, manifest, admission decision, registry record, and source commit are less volatile, provided the attacker cannot alter their stores.

The team blocks new traffic and dangerous outbound connections, prevents the workload from receiving replacement credentials, and stops the deployment controller from recreating it. Those actions reduce harm while leaving the running object available for a bounded collection. If isolation cannot prevent another tenant read, deletion wins. The evidence owner records that the volatile state was sacrificed to protect customers; the gap is part of the case, not something to conceal.

This is the working discipline:

• preserve before changing when the delay is safe;
• minimize contact with the original system;
• record every collection and containment action;
• protect the integrity and access boundaries of the copies;
• involve specialists before the team crosses its authority or competence.

Begin with questions, not a log inventory

Collection should answer the incident’s open questions. For this release, the first questions are who authorized the deployment, what artifact ran, what authority it obtained, which tenant objects it reached, what left the service boundary, and whether the path persists elsewhere.

The CI/CD record connects the source revision, approving identity, build, artifact digest, and deployment request. Registry and provenance records establish whether the digest deployed is the digest built. Kubernetes events and admission records show how the workload entered the cluster; its manifest, service-account binding, secrets references, network policy, node placement, and runtime metadata describe the authority it received. Application and database audit events connect that authority to the tenant read. Identity logs explain credential issuance and use. Network, DNS, gateway, and cloud flow records may show whether data left through another service or destination.

Each source has limits. Application logs may name the tenant yet omit the underlying database query. Database audit may show a query yet lose the application actor. Network records can show a connection without its purpose. Endpoint telemetry can reveal process execution and persistence, while cloud audit records usually describe control-plane change rather than application behavior. AI prompt, retrieval, policy, and tool traces may connect an agent action to data access, but they can also contain credentials and customer content that should not be copied into the general incident channel.

Corroboration is the work. No single event has to tell the whole story, but timestamps need a common reference and identifiers must join across sources. Preserve the source timezone, clock offset, query, export range, and any known ingestion delay. “02:14” is not a sequence until the team knows which clock produced it.

Capture the state that will vanish

Volatile collection happens before reboot, rebuild, failover, rescheduling, autoscaling, credential refresh, or process termination when the resulting delay is safe. Depending on the system and the collector’s authority, it may include running processes, active connections, open files, mounted volumes, temporary files, runtime metadata, shell history, logged-in sessions, and memory.

Memory collection is specialist work unless the organization has a tested procedure and trained operators. A hurried dump can stall a critical host, gather secrets and personal data far outside the incident scope, or alter the very state under examination. The same caution applies to live endpoint acquisition and packet capture. More data is not automatically better evidence.

Ephemeral platforms shorten the decision. For a container workload, record the pod and node identifiers, image digest, start time, restart count, owner, events, runtime state, service account, effective network policy, deployment history, and allowed environment metadata. Preserve the actual image version rather than relying on a mutable tag. For a serverless function, preserve its deployed package or image digest, configuration version, execution identity, triggers, invocation records, environment settings allowed by policy, and concurrent deployment history.

Do not explore a suspect system through several responder shells. Assign a collector, use a tested read-only or low-impact procedure where possible, and have the rest of the team work from controlled copies. Every login and command creates new state.

Preserve the cloud control plane

In cloud incidents, resource configuration is evidence alongside stored data. A disk or database snapshot cannot explain exposure if the role trust policy, bucket policy, security group, route, key policy, or service configuration that permitted access has vanished.

Export or protect the relevant audit, identity, object-access, network-flow, DNS, key-management, load-balancer, registry, backup, and security-service records. Capture configuration with provider-generated identifiers and timestamps. Preserve IAM policy versions and trust relationships, not merely the current rendered policy. Record recent changes to logging, retention, trails, sinks, backup policy, and encryption keys.

Then test whether the suspected identity can alter those sources. If it can disable logging, delete evidence storage, shorten retention, change encryption, or destroy backups, containment of that control plane takes priority over collecting another endpoint artifact. Move or copy evidence into an account or project outside the suspected blast radius, using an approved path whose audit trail survives the move.

Snapshotting is not the only answer. Export append-only audit records before their window expires. Retain the vulnerable image, dependency lockfile, build output, infrastructure plan, prompt or policy version, and retrieval-corpus identifier needed to reproduce the path. Quarantine a host or detach a workload from production instead of deleting it when isolation genuinely contains the risk. The least destructive effective action usually leaves the investigation more options.

Build an evidence index as you collect

The evidence index is the investigation’s map. Give every item an identifier and record:

• the incident identifier and the question the item may answer;
• the original source, resource identifier, and owner;
• the time range, source timezone, and collection time;
• the collector, authority, method, command or query, and tool version;
• the original format, storage location, hash where meaningful, and access restriction;
• any transformation, redaction, failed collection, handoff, retention decision, or known limitation.

Store originals in a write-restricted case repository. Analyze copies. Preserve source timestamps and metadata where the collection method permits it. A cryptographic hash can show that a file has not changed since collection; it cannot prove that the source was truthful, the query was complete, or the collector had lawful authority. Those claims come from the surrounding record.

Notes belong in the same disciplined system but must distinguish observation, hypothesis, and decision. “The deployment identity assumed the production role at 02:14 UTC” is an observation tied to a source. “The identity was stolen” is a hypothesis until corroborated. “Revoke it before memory collection because tenant access continues” is a decision with an explicit trade-off. Mixing all three into a confident timeline corrupts later impact analysis and recovery work.

Know when engineering collection stops

Call the forensic owner before invasive collection when any of these conditions is true:

1. The system may contain volatile evidence and the team lacks a tested collection procedure.
2. Criminal activity, destructive malware, ransomware, litigation, legal hold, or law-enforcement contact is plausible.
3. Employee conduct, insider misuse, personal devices, or monitoring beyond ordinary operations may be involved.
4. Regulated data, substantial customer content, cross-border transfer, or material customer impact may change collection or retention duties.
5. The suspected actor can alter the evidence systems, or the integrity of the control plane itself is uncertain.

The forensic owner directs acquisition and integrity handling. Legal, privacy, or HR decides the authority and restrictions for its domain. Cloud, identity, endpoint, data, and platform owners provide safe source-specific collection. These responsibilities should already be named in the incident plan; the middle of an incident is a poor time to invent them.

Escalation does not suspend protection. If people, customers, data, or critical systems face immediate harm, contain first. Capture the safest readily available state, record why waiting was unsafe, name what the action destroyed or changed, and preserve the audit trail of the containment itself.

The mistakes that cannot be undone

Rebooting a host, deleting a pod, rebuilding an instance, or failing over a service before checking its unique state can close an investigative path permanently. So can leaving short-retention logs untouched, allowing an autoscaler to replace the only suspect workload, deleting an artifact behind a mutable tag, or rotating a credential without preserving its issuance and use history.

The opposite failure is indiscriminate collection. Copying every prompt, mailbox, database row, packet, or employee file “just in case” increases exposure, contaminates the case with irrelevant material, and makes the useful evidence harder to find. Scope follows questions and authorized time windows.

At the end of the first preservation pass, the team should be able to point to the state captured before containment, the records protected from expiry or tampering, the controlled originals, the evidence index, the facts lost to necessary action, and the specialist who now owns deeper acquisition. Chapter 47 can then ask whether eradication and recovery are trustworthy. Without this record, “clean” means only that the team can no longer see what made the system suspect.