Bug Hunt and Incident Exercises

Two captures, one checkout

The exercise begins with a support message and an alert:

A customer clicked checkout once. The payment request timed out, the order later completed, and the customer was charged twice. Duplicate captures have risen since this morning’s release.

You have a repository, a test suite, partial logs, and twenty-five minutes. There is no tidy failing assertion pointing at the broken line.

The first question is not, “Where is the retry code?” It is, “Can this still happen?” A live money-movement failure changes the order of work. Before you explain the mechanism, you may need to stop it.

A useful opening is brief:

I am treating this as an active duplicate-charge risk. First I want to bound the affected path and see whether automatic retry can be disabled or the release rolled back. Then I will reconstruct one order across the first call and its retry. I will not call anything the root cause until the evidence distinguishes it from the other plausible explanations.

That response reveals production judgment without performing incident-command theatre. It names the harm, a reversible protection, the first unit of investigation, and the limit of what is currently known.

Stabilize what you do not yet understand

A rollback, feature flag, queue pause, or traffic shift is not an admission that diagnosis has failed. It is a way to buy a safer investigation window. The appropriate move depends on what the exercise gives you.

Suppose the release changed only the automatic retry worker, duplicate captures began soon afterward, and capture can be retried manually from the operations console. Disabling automatic retry will create some delayed orders, but it reduces the chance of charging more customers twice. That is a defensible mitigation while evidence is incomplete.

Say the trade-off plainly:

I would disable automatic capture retry, not all checkout traffic. Some orders will need reconciliation or manual retry, but a temporary delay is preferable to another duplicate charge. I would record the cutoff time and retain the queued commands so we can reason about what remains in flight.

The last sentence matters. Pausing work is not the same as erasing it. A queue may contain requests created under the old behavior; a rollback may leave corrupted rows; a disabled feature may not affect calls already at the payment provider. Mitigation reduces the rate of harm. It does not repair history or prove a cause.

If the prompt is a local failing test rather than a live symptom, this step is smaller. Run the focused test before editing and preserve its failure output. The protected thing is then the evidence itself: do not change the test, the fixture, and the implementation together until the original failure has become impossible to interpret.

Define the broken promise

“Checkout failed” and “the customer was charged twice” are symptoms. Neither states the rule that the system violated.

For this exercise, assume an order has one payment intent and the system may retry a capture when the provider’s first response is uncertain. The governing promise is:

For one capture command, every delivery attempt presents the same external
idempotency identity. A retry may repeat the command; it must not create a
second capture.

That formulation does useful work. It allows retries, so “never call the provider twice” is not the rule. It locates identity at the capture command, not at a network attempt. It also leaves room for a different order or a different legitimate capture to have a different identity.

Expected behavior gives the investigation a fixed point. Without it, a patch can make the example pass while quietly changing the product rule. It is also how you decide which evidence is close to the failure: command identity, delivery attempts, provider requests, and persisted results matter more than a generic spike in HTTP errors.

Reconstruct one order through time

Do not open every dashboard and search every file that contains payment. Choose one affected order and build the shortest chronology that can separate the plausible mechanisms.

The available logs might yield this:

09:14:02.118 order=ord_1042 command=cap_804 attempt=att_771
             key=capture:att_771 provider_result=timeout

09:14:07.406 order=ord_1042 command=cap_804 attempt=att_772
             key=capture:att_772 provider_result=success provider_id=pay_A

09:14:11.039 order=ord_1042 command=cap_804 attempt=att_773
             key=capture:att_773 provider_result=success provider_id=pay_B

One artifact now carries several pieces of the argument. The order and command remain stable. The delivery attempt changes. The idempotency key changes with it. The provider reports two successful captures under different identities.

The chronology does not yet prove why three attempts were scheduled. A worker could have redelivered the same message, two workers could have claimed it, or an operator could have retried it manually. Those are real questions, but they are not equally urgent. Even if delivery occurs more than once, the system’s stated contract requires repeated attempts of cap_804 to share an external identity.

This is where a good investigation becomes selective. The changing key is evidence at the broken boundary. A queue-depth chart may explain how often the bug manifests; it cannot explain why the provider accepts the same command as new work.

Make a hypothesis risk being wrong

“The retry logic is buggy” is not a hypothesis. It predicts almost anything. A useful hypothesis exposes itself to failure:

The provider key is derived from the per-delivery attempt ID instead of the stable capture-command ID. If that is true, the request builder will receive or construct att_771, att_772, and att_773 as three different keys even though every attempt belongs to cap_804.

Now choose the smallest experiment. Trace the code from the retry worker to the provider adapter, or write a focused test that delivers the same command twice and records the outgoing requests. The test double need not simulate a payment platform. It only needs to reveal the boundary contract:

given one persisted capture command
when delivery times out and the worker retries it
then both provider requests carry the same idempotency key

Suppose the test fails with:

expected: ["capture:cap_804", "capture:cap_804"]
actual:   ["capture:att_771", "capture:att_772"]

The evidence now connects production behavior to an executable failure. The logs showed unstable identity on an affected order. The focused test shows how the request builder produces that identity. If the request builder had already used cap_804, the hypothesis would be wrong, and the next investigation would move outward: perhaps two distinct commands were created for one order, or provider results were persisted under the wrong order.

A wrong first hypothesis is ordinary. A hypothesis that cannot lose is the problem. During the interview, keep the reasoning visible at the moment it changes:

The key is stable in the request builder, so my first hypothesis does not hold. I am moving one boundary earlier to see whether the timeout path creates a second capture command for the same payment intent.

That is more credible than quietly abandoning a guess or defending it after the evidence turns.

Repair the owner of identity

The tempting patch is in the retry worker: reuse the first attempt’s key when creating the next attempt. It can make the focused example pass, but it gives one caller responsibility for a command-wide invariant. A manual retry, a second worker, or another delivery path could still invent a new key.

The capture command should own the identity that every adapter call needs. It can generate the key when the command is created and persist it with the command, or derive it from a stable command identifier if that identifier is immutable and unique. In this modeled system:

capture command
  id: cap_804
  order_id: ord_1042
  payment_intent_id: pi_615
  provider_key: capture:cap_804

delivery attempt
  id: att_772
  command_id: cap_804
  retry_number: 1

The provider adapter receives provider_key; it does not derive payment identity from the attempt. Attempts may multiply. The command does not.

This repair assumes the provider honors an idempotency key for repeated capture requests and that one command represents one intended capture. State those assumptions in an interview. If the provider lacks that contract, the system needs a stronger local state machine and reconciliation boundary; a string key cannot create a guarantee the dependency does not offer. If partial or multiple captures are valid, each intended capture needs its own stable command identity rather than one key for the whole order.

The code fix is only part of the incident. Already duplicated captures require reconciliation and, where appropriate, refund handling. Commands queued before the change may need inspection or replay under the new identity rule. A schema change that persists provider_key may need a backfill for commands that can still be retried. None of those consequences belongs inside the request builder, but a senior close names them.

Prove more than disappearance

When the symptom disappears, ask what the result actually proves.

The focused regression test should deliver the same command more than once and assert a stable outgoing key. A nearby integration test should cover the timeout-then-retry path. Then test the contrast: a different capture command must receive a different key. Without that contrast, a constant key would pass the first assertion and make every customer’s capture collide.

Verification for the live incident has a different shape. After mitigation or rollout, watch successful provider captures per capture command, not merely the overall checkout error rate. Confirm the retry backlog drains without creating new duplicates. Reconcile the affected time window because a falling alert does not repair previous charges.

A precise close separates knowledge from remaining work:

The duplicate captures came from one capture command being presented to the provider under a new idempotency key on each delivery attempt. The logs show the changing keys, and the focused test reproduces the request-builder behavior. Automatic retry remains disabled while the command boundary is changed to own a stable provider key. The regression test covers timeout and retry, with a contrasting command to prove keys remain distinct. We still need to reconcile the affected orders and inspect commands already queued under the old representation.

Notice what the close does not say. It does not claim the entire payment system is safe. It names the mechanism demonstrated, the protection in place, the proof attached to the repair, and the residual exposure.

The same investigation at smaller scale

A failing unit test can exercise the same judgment without dashboards or incident roles. Suppose users retain premium access for one extra day after a refactor. Begin with the exact rule: access ends at the expires_at instant, not at the end of its calendar day. Preserve the failing test. Predict that a date-only conversion or inclusive comparison will appear near authorization. Inspect that boundary, change one variable, and add cases immediately before, at, and after the expiration instant.

Do not inflate a local bug into a fictional outage. There may be no mitigation beyond reverting the patch. But do ask whether the fix has consequences beyond the assertion: cached entitlements may retain the old result, clocks should be injected for deterministic tests, and a timezone conversion may have created other boundary failures. Production judgment is the habit of following the rule far enough, not the performance of severity.

Practise the investigation, not the vocabulary

Use the payment incident once with the full chronology. Then change one constraint at a time:

• The idempotency key is stable, but two capture commands exist for one payment intent. Find the uniqueness boundary and decide how to handle existing duplicates.
• The provider times out after completing the capture and offers no idempotency contract. Design the state and reconciliation path without pretending the uncertainty can be removed locally.
• Duplicate captures occur only in one region after a configuration rollout. Decide which evidence would distinguish application behavior from routing, secrets, and regional provider configuration.

For each variation, produce a five-line investigation note before touching code:

Broken promise:
Immediate exposure and protection:
Hypothesis and predicted observation:
Smallest decisive evidence:
Repair owner and regression guard:

You are ready when those five lines determine your next action rather than summarize it afterward. You should be able to discard a plausible hypothesis without losing momentum, explain why mitigation precedes diagnosis in one case but not another, and distinguish the boundary that exposed the failure from the boundary that owns the rule.

A bug hunt starts with damaged trust. Restoring it requires more than finding a suspicious line. Protect what is at risk, reconstruct one failure through time, let a hypothesis fail if it must, repair the violated promise where it is owned, and attach proof that would expose the same mistake again.