Vulnerability Management and Patching

A dependency scanner reports the same critical image-processing vulnerability in two repositories. One is a developer-only conversion tool that has not been built in months. The other is the invoice-attachment worker: it parses files uploaded by customers, runs with access to tenant documents, and is deployed in every production region. A public exploit is available.

The package name and severity are identical. The decisions should not be. The dormant tool needs an owner and cleanup; the worker needs immediate containment and a controlled patch. Vulnerability management is the work of establishing that difference, acting on it, and proving that the dangerous condition has ended.

Turn reports into findings

Vulnerabilities arrive through dependency, container, host, code, and cloud-configuration scanners; penetration tests and bug bounty reports; vendor advisories; customer reports; threat intelligence; and incident investigations. These sources observe different things. A source-code scanner sees a declared dependency. A runtime scanner sees what was deployed. A penetration tester may show the path by which a weakness becomes access. An incident may show that the path has already been used.

Known-exploitation catalogs deserve their own intake path. CISA’s Known Exploited Vulnerabilities catalog records vulnerabilities with evidence of exploitation in the wild; a match should raise urgency and trigger exposure analysis rather than wait for the next routine scan. It still does not answer whether the affected product is present or reachable in this environment.

Normalize those reports without erasing their provenance. A finding should retain its source and confidence, the vulnerable condition, affected asset or artifact and version, environment, first-seen time, detection method, and available exploit evidence. Then associate duplicates with one underlying condition. Ten scanners rediscovering the same deployed library should improve confidence and coverage; they should not create ten remediation projects.

Asset and deployment identity are indispensable. “Image library version 4.1 is vulnerable” is not yet an operational finding. “Build invoice-worker@8f31c2, running in production regions A and B, contains version 4.1 and accepts customer uploads” can be assigned, contained, replaced, and verified. If the program cannot connect a report to a deployed owner, its first vulnerability is inventory.

Incident findings bypass ordinary backlog intake until responders decide whether the exposed path is contained. A weakness discovered during compromise may require credential rotation, isolation, evidence preservation, or new detection before anyone attempts a patch. Moving it into a thirty-day queue would confuse software maintenance with incident response.

Trace the reachable risk

Start triage by asking whether the reported condition exists, then whether an attacker can reach it. For the invoice worker, the team confirms the affected function is included in the built image and invoked on customer-controlled files. The upload API requires a tenant session, but any ordinary tenant user can submit an attachment. Successful exploitation could reach documents from other tenants because the worker’s service identity has broad storage access. Those facts make a far stronger priority argument than the critical label alone.

Work through the path in order:

1. Identify the deployed artifact, asset, environment, and accountable owner.
2. Confirm that the vulnerable version, setting, or behavior is present and enabled.
3. Find the entry condition: public network, tenant account, internal foothold, local access, or physical access.
4. Determine whether the affected function is actually invoked along that path.
5. Follow the data and privilege available after successful exploitation, including lateral movement and control-plane access.
6. Check for known exploitation, credible exploit code, incident evidence, and attacker interest in this kind of target.
7. Test which controls interrupt the path and what evidence shows that they work.

Asset criticality, exposure, exploitability, known exploitation, data sensitivity, required privilege, and compensating controls now have a place in the same argument. CVSS is useful input about a vulnerability’s general characteristics. It does not know that one copy is dormant, another parses hostile files, or the second workload can read every tenant’s documents.

A compensating control earns credit only where it breaks the path. Disabling attachment uploads prevents new hostile files from reaching the parser. Restricting the worker’s storage identity limits consequence. A tested filter may reject the exploit form. Low traffic, an undisclosed endpoint, or a policy that forbids malicious uploads changes no attacker capability.

For the worker, the team disables new attachment processing, preserves queued files without parsing them, narrows the service identity, and adds monitoring for suspicious parser failures and storage access. These measures reduce exposure while the patched release is prepared. They do not close the finding.

Choose a remediation lane

The triage decision must produce an owner, a due time, an escalation path, and a closure method. A useful policy defines five lanes:

• Emergency begins now when exploitation is active or credible against an exposed, consequential system. Contain first when necessary, patch or otherwise remove the path, verify, and keep incident and business owners informed.
• Expedited takes precedence over planned work when reachability, sensitive data, privilege, or likely exploitation creates high risk but immediate emergency action is not warranted.
• Standard enters normal engineering delivery with a fixed deadline when the weakness is reachable and important but neither urgent nor safely deferred for a maintenance window.
• Scheduled receives an owner and a named change window when exposure is low or the change requires coordinated downtime.
• Accepted risk is a temporary, accountable decision used when remediation cannot be completed by the applicable deadline. It is not a fifth kind of backlog.

Teams need explicit service targets behind those words. A defensible starting policy is immediate action for emergency findings, seven calendar days for expedited work, thirty days for standard work, and ninety days for scheduled work or risk-exception expiry. Critical systems and active campaigns may need tighter limits; systems with hazardous change constraints may need a different scheduled window. What matters is that any variation names the authority, rationale, and escalation behavior rather than quietly redefining “urgent” ticket by ticket.

The invoice worker enters the emergency lane. The developer tool enters scheduled removal after its owner confirms it is neither built nor distributed. If that tool returns to use, a build control must prevent the vulnerable dependency from shipping; a dormant repository is not permission to reintroduce the condition later.

Patch the running system

A patch is a release, not a version-number edit. The worker team must update the dependency, rebuild the artifact, test the parsing behavior and security regression, deploy through a bounded rollout, watch both exploit signals and service health, and establish which version is running in every affected environment.

Dependency compatibility makes this harder than the ticket suggests. The fixed library changes a decoding API used by the worker. The team adds malformed and oversized file cases, validates ordinary invoice attachments, and exercises resource limits. It can canary the new worker on preserved benign samples before reopening uploads to a small tenant cohort.

Rollback also needs security judgment. Returning to the known-vulnerable image may restore availability by restoring exposure. Prefer roll-forward, disable the affected feature, or retain the contained state while repairing the patch. If a vulnerable rollback is the least harmful emergency choice, an accountable incident decision should set its narrow scope, monitoring, and expiry.

The same rule takes different forms elsewhere. Rebuild container images from a patched base and replace running workloads. Prove a library has left both the built artifact and the deployed process. Capture the resulting patch or configuration level for hosts and managed services. For a SaaS or third-party product, retain the vendor advisory, affected service boundary, mitigation status, expected fix, and the residual-risk decision; “the vendor owns it” does not transfer the consequence.

An emergency patch runbook should tell an operator how to move without inventing the process under pressure. It names the incident or finding commander, affected assets and owners, containment options, emergency-change approver, minimum security and service tests, deployment and cohort sequence, stop conditions, safe fallback, monitoring queries, communications, and the evidence required before restoration. It ends with removal of temporary controls and a review of any exposure that existed before containment.

Make exceptions expire

Suppose a legacy exporter shares the vulnerable parser but cannot take the fixed version until a plugin is replaced. An exception begins with that technical fact, not with a request to “accept the CVE.” Record the exact artifact and environments, reachable path and consequence, reason the deadline cannot be met, remediation plan and owner, accountable risk owner, compensating controls and their tests, monitoring, expiry, and triggers for earlier reassessment.

The exporter might stop accepting external files, process only objects admitted by the patched worker, run with a narrower identity in an isolated queue, and alert on unexpected file origins or parser crashes. Each control needs an evidence source and an owner. The exception should state that public exploitation, control failure, material exposure change, or an incident reopens the decision immediately.

The risk owner must have authority over the operational or business consequence. Security can challenge weak evidence and refuse to describe an unbounded deferral as controlled risk. On expiry, the finding reopens automatically. Renewal requires current proof that the asset still exists, the exploit path has not worsened, the controls still operate, and remediation still has a credible date.

Verify the claim that closed it

The worker ticket is not complete when the pull request merges, the registry receives a patched image, or the scanner stops reporting the source repository. Closure must answer the original risk claim in the relevant environment.

The team records the patched artifact digest, deployment records for every production region, dependency evidence from the built image, and a rescan tied to those running workloads. It repeats the attachment-path security test and confirms that temporary storage restrictions did not get removed before the safe worker was serving traffic. Monitoring shows no unexplained parser failures or cross-tenant storage access during the exposure window. Only then does the owner close the finding and reopen normal attachment processing.

Choose proof that matches the condition: a rescan for presence, a version or artifact attestation for deployment, a configuration export for state, an exploit-path retest for reachability, or a compensating-control test for an active exception. Reopened findings deserve investigation. They may expose incomplete rollout, stale assets, rollback, an incorrect inventory relationship, or a closure rule that proved paperwork rather than production.

Read the program as risk flow

Backlog size mostly describes intake. A new scanner can double the count while making the program healthier; one unowned, exploited path into a critical service can make a small backlog dangerous.

Track time to triage and time to verified remediation by lane, then inspect the distributions and oldest cases rather than relying on an average. Keep exposed or known-exploited findings, expired exceptions, findings without owners, and reopened findings visible. Measure how often due dates move, how much of the affected production population has actually received a fix, and how long emergency containment remains in place. These views reveal stuck risk and false closure.

For any consequential finding, a leader should be able to follow one chain: source report, affected production identity, reachable path, priority decision, owner and deadline, patch or mitigation, rollout population, verification, and residual risk. That chain is the vulnerability triage worksheet. It can live in a ticket or incident record, but none of its decisions should depend on chat memory.

The two identical scanner alerts no longer look identical. One led to controlled removal of dormant code. The other led from containment through a tested release to evidence from the running worker. Severity started the investigation; exposure, consequence, ownership, and verified change finished it.