Secure CI/CD and Release Engineering

The invoice export from Chapter 26 is ready to ship. Its server-side authorization tests pass, its hostile customer name remains text in the browser, and its pull request has two approvals. The release workflow produces image sha256:8c1…, but the deployment record names only invoice-export:1.7. That tag can be moved. The production job also rebuilds the image after approval, using a runner that handled pull-request code earlier in the day.

The team has reviewed one thing, tested a second, and is about to deploy a third.

CI/CD security must prevent those identities from drifting apart. A trustworthy release is a chain of custody: the reviewed source revision enters an expected builder, the builder emits one immutable artifact and evidence about how it did so, an authorized deployment promotes that same artifact, and production records what is running. Every transition either preserves identity or creates a new object whose parentage can be verified.

Give source changes the authority they deserve

The protected branch is the first custody boundary. Require review and passing checks before merge, prevent ordinary users and automation from bypassing the rule, and review repository permissions as production permissions. A signature on a commit or tag can bind an identity to bytes; it does not show that the signer was authorized, the review was adequate, or the branch rule could not be bypassed. Use signing where that additional identity evidence serves the threat model, then verify it as part of the release rather than displaying a badge nobody consumes.

Ownership must follow consequence. Changes to authentication, authorization, cryptography, release policy, infrastructure, and the invoice export’s approval contract need the reviewers who understand those boundaries. CODEOWNERS can request the right review, but the branch rule must require that review and must cover the ownership file itself.

Workflow definitions are executable production policy. A change to a trigger, third-party action, runner label, cache restore rule, token permission, artifact upload, environment mapping, or deploy script can bypass controls without touching application code. Protect those paths with specific owners and inspect the effective workflow, including reusable workflows and organization-level defaults. Pin third-party workflow components to an immutable identifier the platform actually honors; a movable release tag preserves convenience, not artifact identity.

For the invoice export, reviewers should see one pull request that changes application code and a workflow. The workflow diff deserves a separate question: does code from this pull request gain authority it did not have before?

Let untrusted code prove itself without giving it power

A pull-request job executes code chosen by the contributor. Tests, compiler plugins, package installation hooks, build scripts, fuzz targets, and even a test name interpolated into a shell command can all become execution paths. Run that work with read-only repository access, no production secrets, no package-publishing token, no signing identity, and no route to sensitive internal services.

Disposable runners reduce persistence between jobs, but “ephemeral” is a lifecycle claim that must be proved. Confirm that each job receives a clean machine or isolated workload, that no second job can overlap it, and that workspaces, process state, credentials, containers, and local disks disappear afterward. Self-hosted capacity needs particular care because a compromised job may reach the host, other jobs, cloud metadata, attached networks, or runner registration material.

Caches and artifacts cross job boundaries even when machines do not. Let an untrusted job read only caches whose poisoning is harmless, and do not allow it to populate a cache later consumed by a privileged release. Bind cache keys to the relevant lockfile, toolchain, platform, and trust context. Treat every downloaded artifact from an earlier job as hostile until its producer, digest, and expected contents are checked.

Constrain the build inputs as far as the ecosystem permits. Use a versioned builder image and toolchain, committed lockfiles, controlled registries, and immutable references. Remove network access when the build can be hermetic; otherwise record the allowed sources and retrieved materials. Chapter 28 decides whether a dependency is acceptable. This chapter ensures that the release build cannot silently resolve a different one.

Logs sit on both sides of the boundary: they are evidence, and they are a disclosure channel. Do not print environments, command traces containing credentials, signing material, customer data, or private package metadata without purpose. Redaction is a last defense against accidental output. It can miss encoded, transformed, derived, split, or newly issued values, so a privileged value should never enter a job that does not need it.

Issue authority at the last responsible moment

The invoice export’s validation job needs to read source and publish test results. The build job may write an artifact to a registry. Signing needs authority over one digest. Staging deployment needs one environment. Production deployment needs another. Rollback may need to change live traffic while being unable to build or sign anything.

Represent these as separate identities rather than one “CI service account.” Set the workflow’s default token to no permissions or read-only, then grant each job only the operations it performs. Prefer federation that exchanges a verifiable workload identity for a short-lived credential. Bind the trust policy to the repository, protected branch or tag, workflow, environment, and any other claims the provider can reliably enforce. A short lifetime limits reuse; claim restrictions decide who can obtain the credential in the first place.

Environment protection should prevent a pull-request job from merely renaming itself “production.” Restrict which refs and workflows may request the environment, require approval when human judgment genuinely changes the decision, prevent self-approval for sensitive releases, and release environment credentials only after protection rules pass. Review bypass privileges and break-glass paths explicitly. A mandatory approval that reveals no artifact digest, evidence, change, or rollout plan asks a person to perform ceremony, not control.

A pipeline permission review should answer concrete questions:

• Which events can start each job, and can attacker-controlled data become shell, expression, path, or configuration syntax?
• Which source revision and workflow revision execute, especially for pull-request and privileged follow-on triggers?
• What repository, registry, signing, cloud, cluster, network, secret, cache, and artifact access does the job receive?
• Which claims allow a workload to obtain short-lived credentials, and could another repository, ref, workflow, or runner satisfy them?
• What can approve or bypass an environment rule, and where are issue, use, denial, and revocation recorded?
• After compromise, can the identity be disabled without stopping unrelated builds or destroying the evidence needed to investigate?

Build one artifact, then name it exactly

After merge, the protected release workflow checks out the recorded commit and builds the invoice export once. It computes a digest, stores the artifact in an immutable repository, and passes the digest—not a mutable tag—to every later job. Staging and production promote those bytes. A friendly version such as 1.7 may remain for people, but the release and deployment records resolve it to sha256:8c1….

That invariant removes an entire class of ambiguity. Rebuilding for production, even from the same commit, creates another object: time, registries, base images, toolchains, generated files, and network inputs may have changed. If policy requires a second build, compare independently produced outputs or treat the result as a new artifact with new evidence. Never inherit approval from bytes that were merely intended to be equivalent.

The builder emits provenance describing the artifact, source, build process, and builder identity. Generate that statement in a control plane the build cannot freely forge, and bind it to the artifact digest. Sign the artifact or attestation with an identity whose use is limited to the protected builder. Verification must check the signature, trusted issuer and identity, subject digest, expected source and workflow, and policy—not simply that some valid signature exists.

The SBOM answers a different question: which components are in these bytes? Generate it for the final artifact, bind it to the same digest, retain it with the release, and index it so Chapter 28’s vulnerability response can find deployed exposure. A signature protects integrity and identity. Provenance describes parentage. An SBOM describes composition. None proves that the software is safe.

For a critical release, the evidence bundle should let an investigator recover:

• the repository, commit, protected ref, review decision, and effective workflow revision;
• the builder identity, runner class, toolchain, resolved dependencies, build parameters, and permitted external inputs;
• the unit, integration, security, and policy checks that ran, with their results and any owned exception;
• the artifact digest, storage location, signature, provenance, and digest-bound SBOM;
• the deployment identity, approver where required, target environment, rollout events, observed health, and final production digest.

Retention must outlive the incident and rollback windows it supports. Evidence that expires before the deployed artifact does cannot establish how production was made.

Make gates decide, not decorate

The invoice export needs its ordinary unit and integration suite, the server authorization tests from Chapter 25, the hostile rendering paths from Chapter 26, and checks on its deployment configuration. Static analysis may catch injection and dangerous sinks; software-composition analysis may identify vulnerable packages; infrastructure and container checks may expose unsafe configuration; dynamic testing may exercise a deployed boundary; fuzzing may be justified for a complex parser. Run the controls that correspond to the release’s threats and changed surfaces.

Every required check needs a stable identity and an enforceable result. Protect the configuration that defines it. Prevent a contributor from replacing a real check with a successful job that has the same display name. Record the exact check result against the source and artifact it covers.

Thresholds need the context that scanner severity alone lacks. Known exploitation, reachability, internet exposure, privilege, sensitive data, and available mitigation all affect the decision. State which findings block automatically, which require named security judgment, who may accept residual risk, how long an exception lasts, and what evidence closes it. A flaky gate that is routinely rerun until green teaches the organization that failure carries no meaning; fix it, narrow it, or remove it from the required path.

Release readiness also reaches beyond scanners. Confirm that threat-model obligations affected by the change are complete, operational owners can observe the new path, alerts and runbooks exist for its likely failures, capacity and dependency assumptions hold, and data changes have a recovery posture. The gate should reject an operationally blind release even when every code scan is quiet.

Promote carefully and preserve a route back

Production receives the approved digest through a protected deployment identity. Roll out to a small cohort or slice, observe security and service signals, then increase exposure. Record each traffic or workload change so an operator can tell which digest was live when a signal moved. Concurrency control should prevent two deployments from interleaving into a state neither release plan anticipated.

Define rollback before approval. For a stateless service it may mean restoring the previous digest. For the invoice export, a schema migration, queued job, changed authorization rule, or external side effect may survive old code. Use backward-compatible migrations, expand-and-contract changes, feature controls, queue versioning, or a forward repair where reversal would corrupt data. Test the selected path under production-like conditions and name the person or automation authorized to invoke it.

Rollback signals should be chosen, not improvised: authorization denials changing unexpectedly, cross-tenant access alarms, elevated export failures, integrity verification failure, unusual signing or deployment identity use, and ordinary service health can each stop or reverse the rollout. Preserve failed artifacts, attestations, logs, and decisions for investigation. An emergency release may shorten the path, but it must still produce an immutable artifact, an attributable deployment, and a retrospective record of every bypass.

Read the release record before approving it

A useful release gate is one compact record, not a summary table assembled for ceremony. It should name the source commit and review, the effective workflow and builder, and the one artifact digest being proposed. It should link the required check results, provenance, signature verification, SBOM, open exceptions, threat-model changes, and operational readiness. It should identify the deployment identity and environment rule, the rollout and monitoring plan, the prior production digest, the data-migration posture, and the tested rollback or forward-repair action.

The approver’s questions then become answerable:

1. Are these the reviewed source and workflow revisions?
2. Did an expected, isolated builder produce this digest from controlled inputs?
3. Do the checks and exceptions match this change’s real threats?
4. Do signature, provenance, and SBOM verification all resolve to this same digest?
5. Can only the intended pipeline place it in this environment?
6. Will the rollout reveal harm early, and can the team stop or reverse it without guessing?

Return to invoice-export:1.7. The release is not ready while that label can resolve to changing bytes, the production job rebuilds, or pull-request residue can reach the builder. Rebuild once in the protected path, record the resulting digest, verify its provenance and composition, run the risk-based gates against it, and promote those exact bytes. Production should be able to report the same digest back.

NIST SP 800-218 supplies a broader secure-development practice frame, while SLSA provenance gives precise vocabulary for artifact parentage and build integrity. Their value here is practical: the same release record that makes deployment defensible can support incident response and assurance without asking the team to reconstruct custody after the fact.

The chain is trustworthy only if a reviewer can start with the production digest and walk backward—through deployment identity, verified evidence, builder, workflow, and review—to the source that was intended to run. Rollback completes the chain in the other direction: it proves the team can act when trustworthy production turns out not to be safe production.