Automated Security Testing

The invoice-correction pull request from Chapter 31 has changed. Tenant identity now comes from the authenticated session, the invoice query is tenant-scoped, policy failure denies the action, and logs omit the correction payload. Its unit tests prove cross-tenant denial and safe behavior when the policy service times out.

The pipeline is green. That fact is useful, but smaller than it sounds.

A static analyzer can recognize a dangerous logging call; it cannot know whether billing-editor is sufficient authority for this invoice. A dependency scanner can identify a vulnerable parser; it cannot establish that production requests reach the affected function. A dynamic scanner can send hostile input to a deployed route; it cannot discover an invoice belonging to another tenant unless its test world contains both tenants and the scanner knows what distinction to observe.

Automated testing becomes assurance when a team can say which claim each test might disprove, where its result changes a decision, and who must act on it. Without those connections, a scanner produces findings. With them, it makes risky changes harder to ship silently while leaving human judgment responsible for what the signal cannot see.

Start with a claim the test can challenge

Choose a test because of the failure it can reveal, not because a security program is expected to own one of every scanner category.

Static application security testing (SAST) inspects source or compiled forms for recognizable dangerous flows and APIs. It is a good place to catch the invoice payload reaching a general-purpose logger, an unparameterized query, or an unsafe cryptographic primitive. Its model of data flow is not the application’s model of authority. A clean result says nothing about whether an account manager should be allowed to correct a settled invoice.

Software composition analysis (SCA) compares dependency and package metadata with policy and known advisories. It can expose a vulnerable parser anywhere in the resolved dependency graph, but a package match alone does not establish reachability or impact. Container scanning extends that inventory into operating-system and application packages in an image. It still needs deployment context: an image in a registry, an image running on an internet-facing service, and an image reachable only in an isolated build job do not present the same decision.

Dynamic application security testing (DAST) probes a running service from the outside. Instrumented application security testing (IAST) adds runtime code context in a test environment. Both depend on the paths, roles, tenants, data states, and feature flags exercised during the run. If the invoice scanner knows only a happy-path editor account, it cannot observe cross-tenant denial or a policy-service timeout. Runtime instrumentation does not repair an impoverished test world.

Fuzzing is strongest where structured input meets a parser, protocol, state machine, or native boundary. It can discover crashes, hangs, resource exhaustion, and surprising transitions that examples did not anticipate. A crash becomes a security decision only after the team maps the input path to exposure, privilege, data, and recoverability.

Infrastructure-as-code scanning can reject an object store made public or a role with a wildcard action before deployment. Policy-as-code testing can express the organization’s required properties for manifests, identities, networks, and release evidence. Neither proves that the policy itself reflects the right risk decision, and neither detects runtime drift unless deployed state is checked separately.

Secrets scanning catches credential-shaped material in commits, build output, images, and other configured stores. Its scope is literal: scanning source history does not prove that logs, prompts, backups, tickets, or external tools contain no secrets. Treat a valid leaked credential as an exposure to revoke and investigate, not merely a string to delete.

AI abuse-case evaluation replays hostile instructions and contexts against model behavior, retrieval boundaries, output handling, resource limits, and tools. It can preserve a known cross-tenant or tool-misuse regression. It cannot establish general model safety, and its result can change when the model, prompt, retriever, tool schema, policy, or surrounding application changes. The evaluated configuration must therefore be part of the evidence.

These signals overlap, but they are not interchangeable. The useful inventory is not “we run SAST, SCA, and DAST.” It is “these are the claims we challenge, these are the assets and artifacts in scope, and these are the blind spots we still review by hand.”

Put the signal beside a decision

The earliest useful check is usually the cheapest place to learn. A focused unit security test, linter, or custom rule on a workstation can shorten the correction loop. A pre-commit secrets check can stop an obvious credential from entering shared history. These local checks should be fast enough to remain enabled, but the central pipeline must repeat any control whose evidence or enforcement cannot depend on a developer’s machine.

The pull request is the natural gate for checks tied to changed code and configuration: SAST, dependency diffs, IaC and policy rules, image diffs, and regression tests for known abuse cases. A result here can block merge or request security review while the author still has the change in mind.

After merge, scan the complete resolved dependency set and the artifacts actually produced. This catches risks hidden by diff-only analysis and creates a central record tied to the commit and build. Generated clients, vendored code, compiled assets, lockfiles, and generated deployment manifests need an explicit inclusion or exclusion decision. “The source directory was scanned” does not prove that the release artifact was.

Near release, test what only an assembled environment can reveal: external behavior, deploy-time configuration, image provenance, and a small set of consequential abuse cases. A release gate should examine the same immutable artifact that will advance. Rebuilding after the scan breaks the evidence link.

Runtime checks observe what delivery-time analysis cannot: deployed exposure, configuration drift, newly disclosed dependency risk, and telemetry conditions. Their result arrives after code has shipped, so the decision is different. It may open a ticket, page an incident owner, remove exposure, or trigger an emergency change. Calling a runtime alert a “gate” obscures that the system is already at risk.

Placement is a trade-off among speed, context, coverage, and cost. Move a check earlier when it can make the same judgment cheaply. Keep it later when it requires the release artifact or a real environment. Repeat it only when the second placement protects a distinct boundary.

Onboard a tool as a control

A scanner is not ready when it first produces output. Before allowing it to block work—or citing it as assurance—write a short onboarding record:

1. Name the security claim and failure classes the tool is expected to detect, along with what it cannot establish.
2. Name the owner for rules, upgrades, baselines, integration failures, and escalation, plus the engineering owner who resolves each finding.
3. Define scope in concrete terms: repositories, branches, paths, generated artifacts, packages, images, environments, accounts, roles, and exclusions.
4. Choose each execution stage and the decision at that stage: advise locally, request review, block merge, block release, create backlog, or start incident handling.
5. Calibrate the initial rules against representative true findings and known-safe code. Record how exposure, reachability, exploitability, and data sensitivity modify the tool’s reported severity.
6. Define where raw results and human decisions live, how long they are retained, and how they link to the commit, build, immutable artifact, environment, rule version, and system.
7. Define suppression authority, required evidence, maximum duration, renewal, and escalation when an exception expires.
8. Test the control itself with a safe seeded finding. Confirm that the expected owner receives it and that the intended gate actually stops the path.

Noise is not merely inconvenient. A high-volume, unowned signal trains engineers to route around the control and conceals the finding that deserved attention. Tuning should narrow a rule to the code and contexts where it means something, improve its model, or change the gate. Quieting a correct rule because it interrupts delivery is a risk decision and should be recorded as one.

A baseline can make a new tool adoptable without forcing an unsafe all-at-once cleanup. Freeze the known findings, assign owners and review dates, and prevent new instances. A baseline with no burn-down decision or expiry becomes a permanent exception under another name.

Follow one finding to a decision

Suppose SCA reports a critical vulnerability in a compression library used by the invoice service. The label is a reason to investigate quickly, not yet a release verdict.

The triage record should answer, in order:

• Identity: Which dependency version, artifact, service, environment, rule or advisory version, and finding instance are involved?
• Reachability: Which production path loads the affected component, and can an attacker-controlled invoice or attachment reach the vulnerable function? Preserve a call trace, configuration, minimal reproduction, or equally inspectable evidence.
• Exploitability: Which permissions, input shape, timing, platform, or chained weakness does exploitation require? Separate confirmed facts from assumptions.
• Exposure and impact: Which tenants, data, credentials, actions, build systems, or production controls are reachable if exploitation succeeds?
• Compensating controls: Which verified boundary prevents the path or limits the blast radius? A control counts only if its configuration and operating evidence apply to this artifact and environment.
• Fix risk: Does the upgrade change parsing behavior or a critical workflow? What validation and rollout are needed?
• Decision: Block merge, block release, remediate on a dated backlog, accept a time-bound exception, or treat the report as a false positive. Name the owner and deadline.

If production accepts attacker-supplied compressed attachments through the vulnerable function, the release should normally stop until the dependency is fixed or an approved temporary control removes that path. If the library exists only in an unreachable development tool, the team may record the proof, schedule removal, and avoid blocking the service release. The scanner supplied the lead; system context supplied the decision.

Define these defaults before the uncomfortable finding arrives. A valid leaked secret blocks the path, triggers revocation, and prompts an exposure investigation. A reachable critical dependency or high-impact infrastructure exposure blocks release unless the authorized risk owner accepts a bounded alternative. A medium finding with demonstrated lack of reachability may enter a dated backlog. Missing regression coverage for a known abuse case blocks the change that could reopen it. A proven false positive may be suppressed without pretending the underlying rule is useless everywhere.

Make suppression expire

A suppression says that a particular result should not trigger its usual decision within a particular scope. It does not erase the result. Record:

• the finding, rule, affected paths or artifacts, and environments covered;
• whether the result is false, irrelevant to the deployed path, or an accepted exposure;
• the evidence for that judgment and any residual risk;
• the compensating control and monitoring, when risk remains;
• the approving owner, implementation owner, creation date, and expiration date;
• the fix or re-evaluation condition, tracking link, and behavior when the record expires.

Keep scope narrow. Suppressing one generated fixture is different from disabling a rule across a repository. If the scanner supports only broad suppression, treat that limitation as additional exposed surface and use another check or review control where necessary.

Renewal is a new decision. Reconfirm the artifact, reachability, control, owner, and deadline instead of extending the date mechanically. An expired suppression should restore the gate or escalate visibly; silent expiry makes the date decorative.

Keep a one-page testing strategy

The invoice system’s strategy can remain short if every line changes practice. Name the assets and boundaries: public API, workers, tenant data, policy service, dependencies, images, infrastructure modules, release artifacts, and AI prompt and tool configuration. For each important security claim, name the automated test, its stage, scope, owner, and decision. State the blocking rules and the triage path. Point to retained results and show how they bind to commits, builds, artifacts, environments, and rule versions. List current baselines and exceptions with their expirations.

End with what automation does not prove. For this system, those gaps include business authorization for invoice state transitions, meaningful separation between tenant roles, abuse paths outside the crawler’s test world, recovery from an uncertain side effect, and attacker chains that cross services. Assign those questions to security code review, design review, or the manual testing work in the next chapter.

Review the strategy when architecture, deployment, test coverage, data sensitivity, or tool behavior changes. Also review it after an incident or manual test finds a path the automated controls missed. The right response to such a miss may be a new custom regression, a better fixture, a rule change, or an explicit decision that human review remains the better control.

Return to the green invoice pipeline. Its evidence now supports bounded claims: the tested cross-tenant request was denied; the seeded logging violation triggered the pull-request gate; the release artifact contained no dependency finding above the defined threshold; the deployed abuse test observed safe policy-timeout behavior. None of those claims means “the invoice service is secure.” Together, with named blind spots and owners, they mean the release decision was informed by repeatable evidence rather than scanner ceremony.