Secrets Management

The payment provider from Chapter 14 signs each refund webhook with a shared secret. The API verifies the signature before it changes the refund state. The cryptographic operation is simple; the path that delivers the secret is not.

The provider issues a value in its console. An administrator can reveal it. A deployment pipeline must make it available to the verifier. Two application versions may overlap during release. A support bundle can capture process configuration. A debug statement can print the request headers it was meant to protect. Rotation changes the provider and every verifier without creating a window in which forged or legitimate webhooks are accepted incorrectly.

A secret is therefore more than a value to hide. It is temporary authority moving through a system. Manage its entire path: creation, custody, delivery, use, observation, replacement, revocation, and retirement. Every copy and every reader enlarges the attack surface.

Begin with the authority, not the label

API keys, database passwords, OAuth client secrets, CI/CD credentials, TLS private keys, signing keys, break-glass credentials, and AI provider keys are all secrets, but their consequences differ. A database password may reveal or destroy customer data. A signing key may manufacture trusted tokens or releases. A CI credential may alter source, artifacts, and production in one chain. An AI provider key may expose submitted data, invoke connected tools, or spend against an account.

Classify a secret by the authority it grants and the boundary it crosses. The type name alone does not tell an incident commander whether exposure permits a read, a destructive action, impersonation, deployment, cross-tenant access, or unbounded cost. Nor does it reveal whether the same credential has been shared across production, staging, developer machines, and a vendor integration.

For each secret, the inventory must record:

• a stable name that does not contain the value;
• the credential type and the exact actions, resources, account, tenant, and environment it authorizes;
• an accountable owner and an incident contact;
• the issuing system, approved store, and every workload, pipeline, person, or third party allowed to consume or administer it;
• how it reaches each consumer and whether the consumer can reveal or export it;
• creation, expiry, normal rotation, last rotation, emergency revocation, and retirement state;
• access logs, anomaly signals, scanner coverage, and review cadence;
• downstream credentials, cached sessions, signed artifacts, or retained data whose trust may survive its revocation.

This is an incident index, not a spreadsheet of names. If the webhook secret leaks, the responder should be able to find every verifier, the provider account, the owner, the replacement procedure, the relevant logs, and the longest replay window without searching chat history.

Remove the secret when identity can do the job

The safest durable secret is one the application never receives. When a platform can authenticate a workload directly, bind a workload identity to the specific environment and action. Let that identity exchange for a short-lived, audience-restricted credential when the target system requires a token. A database connection, cloud API call, queue client, or deployment job that can use identity-native access should not inherit a long-lived shared password for convenience.

This does not make the system secretless in an absolute sense. The platform still protects signing material and issues credentials. It moves custody out of application configuration, shortens useful lifetime, and gives the caller an identity that can be revoked and audited.

Some integrations still require a shared value. Put it in a managed secrets store or managed key service that provides encryption, access policy, versioning or controlled replacement, and audit events. Separate read access from administrative access. Bind retrieval to the workload, environment, and purpose; a production verifier should not be able to enumerate unrelated database, deployment, or vendor credentials.

The store is only the beginning of the delivery path. Runtime injection can place a value in a protected file, an in-memory response, or an environment variable. Choose deliberately. Environment variables are widely supported, but they can escape through process inspection, diagnostics, crash reports, debug endpoints, and commands that dump configuration. A mounted file can be over-permissive, copied into a support archive, or left readable after the process no longer needs it. A client library can cache a value indefinitely or log the response on error. Trace the path on the actual platform and restrict the file permissions, process readers, diagnostic output, cache lifetime, and cleanup behavior it creates.

Never put plaintext secrets in source, repository history, container image layers, build arguments, tickets, chat, documentation, screenshots, analytics events, logs, or CI output. Encryption is not an automatic exception: an encrypted value stored with a broadly available decryption path is still a shared credential with an obscured copy.

Break-glass credentials require a separate path because ordinary identity may be unavailable during recovery. Store them apart from routine application secrets, require named retrieval, alert on access, limit their authority and lifetime, and test the retrieval and replacement procedure. A credential that has never been exercised is a hope, not a recovery control.

Rotation is a distributed change

Changing a value in the secrets store is not rotation. Rotation is complete when every legitimate consumer uses the replacement, the old value no longer works, and the evidence can distinguish both versions.

Consider the webhook secret. If the provider can sign with only one value but the application rollout spans several instances, replacing the provider first causes rejected events; replacing the application first causes the same failure in reverse. The design needs a transition protocol. Where the provider supports overlapping secrets, add the replacement, deploy verifiers that accept both named versions, switch signing, observe use of the new version, then remove trust in the old one. Verification should identify which version matched without logging the signature or secret.

Where overlap is impossible, use a coordinated maintenance window, a proxy or adapter that can bridge the transition, or a provider-specific replacement mechanism. Do not quietly preserve the old secret forever as the rollback plan. The inventory must state the constraint before the emergency.

A normal rotation runbook should:

1. Name the owner, issuer, consumers, dependencies, expected service behavior, overlap capability, rollback boundary, and evidence to collect.
2. Create a distinct replacement in the approved system and grant only the intended consumers.
3. Prepare consumers for overlap or the coordinated cutover, then canary the change where the protocol permits it.
4. Switch issuance or use and observe successful traffic by credential version across every declared consumer.
5. Look specifically for old-version use, authentication failures, queued work, offline clients, and undeclared consumers.
6. Revoke the old value once legitimate dependency has ended; prove that a controlled old-version attempt now fails.
7. Remove obsolete copies and access paths, update the inventory, and retain the rotation record without retaining the value.

Schedule is only one trigger. Rotate when the issuer or algorithm requires it, when a consumer or owner changes, when scope can be reduced, and whenever exposure is suspected. Prefer credentials whose expiry bounds mistakes, but do not mistake expiry for revocation: an exposed credential may remain useful until its clock runs out, and downstream sessions or artifacts may outlive it.

Monitor the lifecycle itself. Alert on retrieval by an unexpected caller, unusual rate or location of use, policy changes, export or reveal operations, failed rotation, use of an old version, access to a break-glass credential, and continued use after revocation. Retire a credential when its integration ends, its owner disappears, or its authority moves elsewhere. Deleting the inventory row before revocation and dependency checks destroys the map while leaving the power behind.

Scan the places where custody fails

Secret scanning is a set of tripwires around prohibited paths. Put fast feedback on the developer path, block high-confidence findings before a push or merge, and scan the full repository history rather than only the current tree. Extend coverage to container layers, packages, generated documentation, CI logs, build artifacts, support bundles, runtime logs, object storage, and other places configuration is copied.

A scanner finding needs an owner and an incident path. Record the location and credential type without reproducing the value. Validate whether the candidate is real through a protected mechanism. A known live secret should be revoked, not merely deleted from the latest commit. Removing a line does not erase clones, caches, forks, build artifacts, or history already in another party’s possession.

Patterns have limits. Some secrets have recognizable prefixes; others look like arbitrary random strings or ordinary configuration. Encoded, split, encrypted, or newly introduced formats may evade detection, while fixtures can produce false positives. Maintain organization-specific patterns where they are reliable, control scanner bypasses, and audit accepted exceptions. Scanning reduces dwell time and prevents some leaks; it does not make an unsafe delivery path acceptable.

The first response is loss of trust

Suppose a support engineer finds the webhook secret in a diagnostic archive. The archive is access-controlled and there is no obvious misuse. Those facts help estimate blast radius, but they do not restore secrecy. Anyone who could read or copy the archive may possess the same authority as the provider.

Use this exposure incident playbook:

1. Open an incident and preserve the finding’s location, time, access history, relevant logs, build or commit identifiers, and the identity of the person who reported it. Restrict the artifact; do not paste the value into the incident channel or ticket.
2. Identify the credential by protected fingerprint or issuer metadata, then use the inventory to find its authority, environments, consumers, dependencies, and revocation owner.
3. Stop further disclosure. Disable the logging path, quarantine the artifact, restrict downloads, or pause the affected job while preserving evidence.
4. Revoke or disable the exposed value through the prepared emergency path. When abrupt revocation would create a serious safety or availability risk, the incident owner must make that trade-off explicitly, add containment, and set the shortest defensible cutover.
5. Issue a replacement with no broader scope, deliver it through the approved path, and verify every legitimate consumer. Do not reuse a neighboring environment’s credential to recover quickly.
6. Search for use before and after the earliest possible exposure: webhook verification results, provider activity, administrative reveals, archive access, network source, affected data or actions, spend, and any credentials or sessions the secret could create.
7. Test that the old value fails, remove exposed copies according to evidence and retention requirements, repair the escape path, and add a detection or preventive control that addresses the actual failure.

Emergency rotation compresses the normal sequence; it does not discard it. The team still needs consumer discovery, replacement, verification, revocation, blast-radius assessment, and a record. For high-consequence credentials, exercise the runbook with a disposable credential before an incident. Measure time to identify all consumers, time to revoke, time to restore legitimate use, and time to prove old-version rejection.

Review the complete handling path

Before release, choose one real credential and follow it rather than reviewing the secrets store in isolation. The handling review should establish:

• who creates the credential, with what scope, expiry, and environment;
• why a durable secret is necessary instead of workload identity or a short-lived exchange;
• where the value may exist, every principal that can retrieve or administer it, and how plaintext is kept out of prohibited paths;
• how each consumer receives it without embedding it in source, images, or deployment output;
• how normal rotation crosses old and new versions without an outage or an indefinite overlap;
• which events prove retrieval, use, policy change, rotation, revocation, and rejected old-version use without recording secret material;
• how scanners cover code, history, images, logs, and artifacts, and who owns a finding or bypass;
• how the emergency runbook contains exposure, replaces authority, searches for misuse, preserves evidence, and repairs the path;
• when the credential expires or retires and how obsolete copies and access grants are removed.

The webhook secret is under control only when the team can change it and lose trust in it on demand. The same test applies to a database password, deployment token, provider key, certificate private key, or break-glass credential. A managed store can protect a value at rest. Secrets management protects the authority through time—and hands Chapter 16 a precisely identified workload and network path to constrain.