Case Study 3: Fraud Detection Model

A Better Model, a Worse Friday

This modeled case begins with an apparently successful replacement. A payment platform’s new fraud classifier ranks known fraud above legitimate transactions more effectively than the rules and older model it will replace. Its precision-recall curve is better on the final test set. The release team chooses a threshold that also improves recall and opens the traffic gate.

By Friday evening, the manual-review queue contains twice its normal volume. Investigators start with the highest scores, so ordinary cases age while legitimate customers wait for purchases to clear. Some retry with another card or account, making their behavior look more suspicious. Customer operations receives complaints but cannot tell which model and threshold produced each hold. Fraud losses have not yet matured, so the dashboard still shows the offline improvement that justified the release.

The classifier may be better. The deployed decision system is worse.

The use case must therefore be stated as an operating decision: rank transaction risk well enough to allocate allow, step-up, review, and reversible block actions within explicit limits on fraud loss, customer harm, investigative capacity, and unequal error. The score contributes evidence. It does not own the action.

Give the Model a Baseline It Must Beat

Before the replacement, the platform uses deterministic blocks for a small set of known attacks, step-up verification for ambiguous cases, and investigator review for rules that combine velocity, device, account, and transaction signals. The baseline is blunt. It misses novel attacks and inconveniences some legitimate customers. It is also observable: the team knows how many cases each rule creates, how long they wait, which rules investigators overturn, and how customers recover.

Those operational measurements matter as much as the old model’s predictive results. The candidate must beat the same baseline under the same traffic, action definitions, staffing, and outcome window. Comparing a new classifier with generous review capacity against rules operating under a fixed queue would attribute an operational advantage to the model.

The affected people extend beyond the fraud team. Cardholders and merchants bear delays and declines; investigators absorb the queue and make judgments that become future labels; customer-support agents explain decisions and restore access; fraud operations carries loss; security tracks changing attack campaigns; model risk challenges the evidence; and engineering keeps features, scoring, case routing, and version lineage intact. A decision that optimizes only fraud loss quietly spends everyone else’s time and trust.

The Threshold Writes the Operating Policy

The model emits a score. If the platform describes that score as a probability, it must show that the score is calibrated for the relevant population and period; otherwise it remains a ranking signal. Policy converts it into action bands:

• low-risk transactions proceed;
• an intermediate band receives a low-friction challenge when one is available;
• cases worth human judgment enter review only while the queue can meet its service limit;
• the highest-risk cases receive a reversible hold or block under separately approved rules.

Known security and compliance rules remain deterministic and auditable. The model cannot override a hard block, silently expand the populations in scope, or turn an unavailable review path into an automatic decline.

This last constraint changes the architecture. A policy service combines score, rule results, channel, available recovery paths, and current review capacity. It records the model, feature set, threshold set, and policy version with every decision. The case service presents investigators with the underlying evidence and keeps their disposition separate from the model recommendation. Notice and appeal services connect a challenged customer to the original decision and its available remedy. When the model or a required feature is unavailable, the router returns to a tested rules-and-step-up policy rather than improvising from partial data.

The threshold record names the assumed losses, friction limits, safe queue capacity, segment floors, model and feature versions, approving owner, effective period, stop conditions, and fallback. A threshold edit is a production release because it changes who proceeds, who waits, and who must prove legitimacy.

The Queue Failure Changes the Design

The Friday incident is not repaired by asking investigators to work faster. The release threshold was chosen from an average daily review count. It did not model hourly arrivals, case-handling time, shift coverage, or the fact that suspicious campaigns cluster. Once arrivals exceeded completions, the backlog grew. The nominal review band became a delay band, and investigator triage by score created a second threshold that no one had approved.

The team first returns traffic to the previous policy. It identifies every decision made by the candidate bundle, applies the approved recovery path to holds that can no longer receive timely review, preserves affected records, and gives customer operations a route for notice and correction. Only then does it analyze whether the model can return.

The revised policy has a capacity envelope rather than one static review threshold. Release evidence covers ordinary and peak arrival rates, handling-time distributions, aging, abandonment, and recovery—not just the number of alerts in a day. As the queue approaches its safe bound, the system may send eligible cases to step-up verification or revert selected traffic to the baseline. It may not resolve overload by silently converting review cases into blocks. If no safe action exists, exposure contracts.

This is the case’s central reversal: review capacity is not a resource assumption outside the model evaluation. It is one of the policy’s safety constraints.

Labels Remember the Policy That Created Them

Fraud outcomes arrive on different clocks. Investigator decisions appear quickly but may be influenced by the score and evidence the interface showed. Customer reports and appeals reveal some false positives. Chargebacks and confirmed fraud arrive later. A declined transaction may never reveal whether it was legitimate, and an allowed transaction without a reported loss is not certain proof of innocence.

The platform therefore records both the outcome and how it became observable. Training and evaluation data retain the policy version, action, review selection, evidence available to the investigator, outcome source, observation window, and later correction. Investigator notes distinguish observed evidence from final disposition. Appeals repair labels through a governed process rather than entering the next training set as informal exceptions.

This provenance exposes selective labels: reviewed cases are overrepresented precisely because the old policy considered them suspicious. Training only on investigator outcomes would teach the new model from a population chosen by its predecessor. Time-based splits reduce leakage across changing campaigns, but they do not cure this selection. The team needs matured outcomes from allowed traffic, carefully designed exploration where it is safe and authorized, and explicit uncertainty where ground truth remains unavailable. Post-event fields such as chargeback status cannot enter features used at transaction time.

Proxy review also begins with a harm pathway rather than a list of forbidden columns. Device, location, channel, account history, and transaction behavior can correlate with vulnerable or protected status, yet deleting a named attribute does not remove the encoded relationship. The team asks which customers experience extra challenges, holds, and failed recovery; then it examines the features and mechanisms that produce that burden.

Evaluate the Whole Decision Path

Evaluation proceeds in the order a transaction moves through the system. First, data and label checks must show that features exist at decision time, outcome windows are sufficiently mature, training and test periods are separated, and important observation gaps are documented. A critical leakage or lineage failure ends the evaluation before model comparisons begin.

Next, the team compares the candidate with the rules baseline and current model on the same periods. Ranking measures such as PR-AUC help determine whether the score separates known fraud from legitimate activity in this rare-event setting. They do not choose the operating point. At each candidate threshold, the report publishes confusion-matrix counts alongside precision, recall, alert yield, expected loss under stated assumptions, and calibration where probability language will influence action.

Then the policy is evaluated. A replay sends historical event sequences through allow, challenge, review, and block actions while simulating arrivals, investigator handling, queue aging, and fallback. A shadow deployment checks live feature availability, score distribution, latency, rule disagreement, and projected workloads without changing customer outcomes. Neither predicts an adaptive attacker perfectly, but together they can expose the capacity failure that the first release missed.

Finally, error cases and aggregate results are examined for predeclared customer, transaction, channel, geography, device, accessibility, and plausible intersectional segments. Counts and uncertainty accompany rates, especially where fraud is rare. The question is not whether every group has an identical metric. It is whether a consequential disparity exists, what mechanism creates it, whether customers can recover, and whether a proposed repair reduces rather than relocates the harm.

The candidate returns only if it improves on the baseline under the same capacity, stays within the peak queue envelope, meets the declared fraud-detection and false-positive floors, introduces no critical leakage, preserves a usable appeal path, and passes a rollback rehearsal. A higher offline score cannot compensate for failure of one of these conditions.

Operate Against a Moving Opponent

Fraudsters learn from declines, challenges, and successful paths. Customers and merchants also change behavior in response to friction. Monitoring must therefore preserve unlike signals rather than blending them into one health score: feature availability, score and calibration drift on matured labels, rule-model disagreement, review arrivals and age, investigator consistency, step-up completion, block reversal, appeals, customer contacts, fraud loss, and segment effects. Every signal is attributable to a complete model-and-policy version.

The incident playbook distinguishes feature corruption, scoring outage, abrupt distribution shift, coordinated attack, excessive false positives, missed-loss spike, and capacity collapse because their safe responses differ. Depending on the evidence, responders may disable a suspect feature, narrow a channel, widen step-up verification, revert the complete release bundle, or return to deterministic rules. Moving a threshold in the middle of an attack may be necessary, but it still requires an owner, recorded rationale, bounded duration, consequence review, and an exit condition.

Ownership follows the action path. Fraud operations owns thresholds, queue policy, and investigator practice. Engineering owns data, serving, routing, and lineage. Security owns adversarial intelligence. Model risk independently challenges model and policy evidence. Customer operations owns notice and recovery. Legal and compliance specialists review jurisdiction-specific obligations; they do not become a vague final approver or turn the model into the policy owner.

The Threshold Review

Before approving the candidate again, the decision group must be able to answer:

• Which loss, friction, capacity, and segment assumptions select these action bands?
• Which outcomes are delayed, unobserved, or selected by an earlier policy?
• What happens to review cases when arrivals exceed safe capacity?
• Can a customer understand, challenge, and recover from each restrictive action?
• Which complete model-and-policy version produced a decision?
• What signal stops the release, and which tested policy replaces it?

The strong system makes score, policy, workload, and remedy inspectable as one decision path. The weak one celebrates a better curve, lets an overloaded queue invent policy in production, and then trains on the decisions that failure produced.

The support case before this one placed meaningful human review between generated text and a customer. Here, human review is scarce capacity inside an adaptive decision system. The next case raises the evidentiary burden again: a clinical draft must preserve what the source said, what it omitted, and who can safely sign the result.

See Metrics for Predictive Models, Robustness, Fairness, and Segment Performance, and Change Management.