Detection Engineering

A release identity deploys an artifact whose digest was never approved. Four minutes later, the new workload reads an export credential and begins scanning tenant records. The deployment event exists. The secret read exists. The database and network events exist. Yet no responder sees the sequence because each source reaches a different queue, under a different name, with a different clock.

Chapter 42 left this transition deliberately uncertain: would a stolen release identity look ordinary? Detection engineering turns that uncertainty into a maintained claim about observable evidence. The work is larger than saving a query. A production detection needs versioned logic, a reliable data contract, tests, an owner, and a response decision that can be made in time.

Begin with the decision

The first design question is not “Which query language do we use?” It is “What should a responder be able to decide?” For the release scenario, the decision is whether to suspend the release identity and lane, quarantine the artifact, and isolate the workload before a tenant export completes. That decision determines how quickly the signal must arrive and which evidence must accompany it.

Write expected behavior before attacker behavior. A normal production deployment uses an approved identity, an accepted artifact digest, a known pipeline definition, and a bounded release window. The workload then follows its usual secret, data, and network paths. The suspicious sequence differs in ways the system can observe: the accepted digest has no matching review record, the workload reaches an export credential it has not recently used, and read volume crosses both its service baseline and an absolute safety threshold.

This is a correlation detection. None of the three events proves compromise by itself. Their shared artifact, workload, environment, and time window create a case strong enough to act on. The response does not need attribution. It needs evidence that the production state no longer agrees with the state the organization approved.

Choose a simpler form when it answers the decision. A rule or signature is often right for MFA being disabled on a privileged account, a public storage policy, use of a break-glass identity, or an unauthorized privileged-role assignment. Integrity monitoring can compare deployed artifacts, pipeline definitions, files, and policies with an accepted state. A behavioral rule can follow API-key creation with use from an unusual environment, or an unexpected model or agent tool invocation with access to sensitive retrieval data. A canary credential or honeytoken provides a high-confidence signal because legitimate work should never touch it.

Statistical anomaly detection is useful only when the baseline is stable enough to defend. “Impossible travel” is especially easy to overclaim: corporate egress, mobile networks, shared infrastructure, clock error, and concurrent sessions can make distance arithmetic look decisive when it is not. Treat it as supporting evidence unless local identity and network behavior make the inference reliable. A plain production-export threshold with a narrow allowlist may protect the release scenario better than an opaque anomaly score.

Make the evidence dependable

The release correlation needs four sources: deployment acceptance, artifact provenance, credential access, and tenant-data reads. Each event needs a stable name; event and ingestion timestamps; actor and workload identity; target resource; environment and tenant where applicable; action, decision, and result; and correlation values such as deployment, request, session, artifact, and workload identifiers.

Document whose clock supplies the event time. A five-minute correlation window fails if a buffered source arrives seven minutes late or if one parser substitutes ingestion time. Specify acceptable latency and completeness for every source, along with retention, sampling policy, parser owner, and the signal that tells operators when the source is missing. Evidence required for a decision must not be optional telemetry.

Normalization is a contract, not a cosmetic rename. If one source calls the release actor principal, another calls it subject, and a third resolves the workload only after enrichment, the normalized fields must preserve the original values while presenting one stable meaning to the detection. Enrichment can add service ownership, criticality, data classification, approved release records, and active exception context. It must also expose its own age and failure state; stale ownership data can route a correct alert to an abandoned queue.

Monitor the path that feeds the rule. Count source events, parsing failures, unmapped values, enrichment misses, ingestion delay, and the fraction of expected workloads reporting. A silent parser break creates a reassuring absence of alerts. Loss of a required source should produce a coverage-degraded signal owned with the same urgency as the detection it weakens.

Prove the path from event to action

Start with small fixtures. One unit case contains an approved digest followed by ordinary workload reads and must remain quiet. Another changes the digest without changing the review record and must match the artifact branch. Further cases exercise missing fields, late arrival, duplicated events, reordered events, known maintenance, and the exact boundary values of thresholds and windows. These tests make grouping, joins, suppressions, and severity mapping reviewable without needing a live incident.

Then replay a sanitized historical window. Replay answers a different question: how the proposed logic behaves amid real volume, seasonality, automation, and imperfect data. Record which alerts would have fired and inspect representative matches and non-matches. Historical silence does not establish recall if the attack behavior never occurred in that window.

A safe simulation or atomic test should create one controlled behavior at a time: deploy an inert unapproved digest in a non-customer environment, request a test credential, or issue synthetic export-shaped reads. “Atomic” describes bounded behavior, not authorization to run destructive steps. Name the permitted system, test identity, time window, safety observer, stop conditions, cleanup, and evidence that must be preserved.

Finally, exercise the sequence end to end. The purple-team run asks whether production-equivalent events survive collection and normalization, the rule creates one correctly enriched case, routing reaches the on-call responder, and the runbook supports a timely decision. Capture action time, source-event time, ingestion time, alert time, acknowledgement, decision, and containment result. A green query with a dead route is a failed detection product.

Keep each test as a small record with scenario and rule version, fixture or replay window, expected result, actual result, relevant event identifiers, observed noise, and a link to the response exercise. Sanitize customer data and credentials. Run the suite again after changes to schemas, parsers, enrichment, windows, thresholds, suppressions, severity, or routing. The regression most worth catching is the one that turns a meaningful field into a null while the rule continues to compile.

Ship the detection as code

Keep logic, configuration, fixtures, tests, and documentation in version control, or in a detection system with equivalent history and review. The specification should travel with the implementation. It names the threat scenario and response decision; expected and suspicious behavior; source events and required fields; joins, grouping, windows, thresholds, and enrichment; severity and routing; suppression boundaries; owners; test evidence; runbook; rollout and rollback; quality expectations; and review triggers.

Review a change according to what it can alter. A parser edit can remove coverage. A wider time window can multiply cases. A new suppression can conceal the behavior the rule exists to find. A lower severity can delay containment as surely as a broken query. Those are production changes even when no application binary moves.

Observe-only rollout is useful when the team needs a volume baseline and can tolerate missing the response during that period. It is not a universal safety step. A high-confidence signal for use of a honeytoken or disabled logging may deserve immediate routing because suppressing it is the more dangerous experiment. Define the observation period, reviewer, promotion evidence, and rollback condition before deployment.

Tune against the original decision. Separate actual false positives from accepted risk, expected but important noise, authorized maintenance, benign automation, duplicate cases, and behavior that lacks enough context. Each category calls for a different repair: narrow the logic, enrich the case, schedule maintenance context, deduplicate, or redesign the source event. “This fires often” is not a sufficient reason to suppress privileged activity.

A suppression needs a reason, owner, exact scope, creation and expiry times, approval, and a test proving that the remaining rule still catches the threat path. An indefinite suppression on a privileged action should receive scrutiny comparable to a permission grant.

Retire a detection when the scenario is no longer plausible, the architecture removed the path, stronger coverage replaced it, or its data source cannot be restored. Retirement includes removing routes and stale dashboard claims, preserving the decision record, and testing the replacement when one exists.

Judge quality by the decisions it supports

Precision is the share of generated cases that represent the behavior the rule claims to find. Recall is the share of relevant behavior the detection finds. Precision can often be estimated from reviewed alerts. Recall usually cannot: the organization rarely possesses a complete set of attacks it missed. Use labeled fixtures, simulations, incident evidence, and sampled retrospective searches to bound it, and state what remains unknown.

Volume, duplicate rate, stale cases, time to acknowledge, and time to contain show whether the signal can be operated. Source freshness, parser failure, field completeness, enrichment age, and reporting coverage show whether it can be trusted. Review these by detection and threat scenario. A portfolio can have excellent aggregate precision while one critical export path has no healthy source.

Severity expresses required urgency and plausible impact, not confidence alone. The correlated release and export sequence is severe because delay can increase tenant harm and because responders have bounded containment actions. A vague anomaly with no validating context should not page as critical merely because its score is numerically high.

The health dashboard should therefore show more than alert counts. For every production detection, show deployment and rule version, enabled or observe-only state, owner and route, last successful end-to-end test, source freshness and completeness, parser and enrichment failures, case volume and duplicates, acknowledgement age, active suppressions and their expiries, and the threat scenario whose coverage is degraded when something fails. Page on conditions that invalidate a critical claim; ticket slower quality drift with a named review deadline.

Give the responder a usable case

The release alert should arrive with the accepted and approved digests, identity and workload, relevant timestamps, environment, credential target, export volume, source-health state, known maintenance, and links to immutable evidence. The responder should not have to reconstruct the rule before deciding whether the production state is trustworthy.

Its runbook begins with safe validation: confirm the records came from healthy sources, compare accepted provenance, establish whether the identity and change were authorized, and check whether export activity continues. It then names bounded containment options—suspend the identity or release lane, quarantine the artifact, isolate the workload, block the export route—along with who may authorize them and what service risk each creates. It identifies evidence to preserve before state changes, escalation triggers for privileged access or tenant-data exposure, communication and incident-opening criteria, reversal conditions, and what proves closure.

This structure transfers to the common detection backlog: MFA disabled, privileged assignment, API key created and then used from an unusual context, public storage exposure, suspicious CI/CD token use, production database export, disabled logging, backup deletion, and unexpected AI tool invocation. Each needs local expected behavior, evidence, timing, ownership, and a decision. A copied query fragment supplies none of those.

When an incident or exercise reveals a miss, keep the feedback specific. Was the behavior outside the scenario, the source absent, the field lost, the correlation too narrow, the severity wrong, the route dead, or the containment step unusable? Change the scenario, event contract, logic, test, route, or runbook that actually failed, then preserve the case as regression evidence.

The release detection is ready only when the team can demonstrate the whole claim: controlled behavior produces dependable events; versioned logic creates the intended case without swallowing routine releases; a healthy route reaches an owner; and the runbook supports an authorized decision before the export completes. When that decision opens a coordinated security incident, Chapter 44 supplies the roles, evidence discipline, and command model that carry it forward.