Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 32

Online Experiments, Pilots, Shadow Mode, and Launch Decisions

Earn production exposure through staged evidence, comparable cohorts, explicit stop conditions, and tested rollback.

The Pilot That Quietly Became Production

A claims team connects a new triage model to live cases. For two weeks, specialists are supposed to compare its recommendations with their own. Accuracy appears promising, so managers tell more teams to “try it.” The recommendation becomes preselected. Turnaround targets begin to assume its use. Reviewers learn which option the model prefers before recording an independent judgment.

No launch meeting occurred, yet the model acquired production influence. The pilot spoiled its own comparison, changed employee behavior, and expanded without a tested rollback or a clear owner for harmful routing errors.

The team needs more than a cautious rollout calendar. It needs a sequence in which every increase in authority answers a particular question, exposes only people and data the team is prepared to protect, and can stop without leaving the workflow in an unknown state.

A progressive exposure ladder moves from shadow mode through reviewed pilot, canary, limited release, and general release; every stage has pass and stop gates and a rollback path to a known-good baseline while evidence requirements increase.
Exposure and evidence rise together. A stage is safe only when it has an explicit stop gate and a fast path back to a known-good configuration.

Freeze the Candidate Before Asking What It Proves

The team begins again, this time with one release bundle: model, code, input pipeline, threshold, explanation view, routing policy, interface, and monitoring configuration. If any of these changes during the trial, the evidence no longer describes one candidate.

It also writes the decision before opening production data. The candidate may prioritize claims for specialist review, but it may not deny, approve, or change payment. The non-AI queue is the baseline. Urgent-case recall, queue delay, specialist capacity, segment floors, and unauthorized actions govern advancement. One unauthorized action stops exposure immediately. A drop below the urgent-case floor or a sustained capacity breach pauses the stage for review. The observation window must finish before an ordinary go decision, even if the first results look good.

That record is an evidence contract. It names eligible and excluded claims, the comparison, allocation method, expected volume, label delay, user notice and remedy, data retention, and the owners who may advance, hold, redesign, or stop the release. It also sets the ceiling on the next stage. “The pilot went well” cannot silently authorize a new region, a new claim type, or automatic decisions.

The stages that follow are often given overlapping names. Their names matter less than their questions:

  • A technical trial asks whether the bundle can operate within its service and control limits.
  • An online experiment asks whether the candidate caused a better outcome than a credible comparison.
  • A pilot asks whether people can use it safely in the actual workflow.
  • A canary or phased release asks whether the bundle can assume more production load without unacceptable failure.

One stage may answer more than one question, but only when its design supports each answer. A canary may reveal a latency regression without establishing long-term benefit. An A/B test may estimate an average effect while missing a severe failure in a small, high-impact segment.

Let the System See Reality Before It Can Change It

Before any live input reaches the model, the team runs synthetic and historical claims through a production-like stack. It checks schema changes, missing documents, permissions, load, cost, logging, and failure paths. Internal dogfood confirms that the interface and support route function, but employees are not representative claimants or claims specialists. Replay confirms only what the recorded data contains; it cannot show how people adapt when a recommendation enters their work.

Shadow mode is the first encounter with live inputs. The candidate scores each eligible claim in a separate environment while the existing queue continues to determine the outcome. This reveals production coverage, latency, score distribution, capacity, and differences from the baseline without granting the model routing authority.

The first shadow run finds a problem that replay missed. Scanned documents arrive after structured fields, so early scores systematically under-prioritize claims from one submission channel. The model itself did not change. The timing of the live system changed the meaning of its input.

Shadow mode contained the routing error, but it was not risk-free. Live claims copied personal data into a new processing path and created sensitive logs. The team uses read-only or simulated dependencies, suppresses side effects, limits access and retention, and verifies deletion. It reviews disagreements with independently labeled cases rather than treating historical routing as truth; the existing process may be noisy or biased too.

Introduce Influence Without Surrendering the Comparison

After fixing event timing and repeating the shadow test, the team begins a human-reviewed pilot with two specialist teams and reversible routing only. Specialists record urgency before seeing the model score. They can then inspect its evidence, accept or override the route, and escalate uncertain cases.

The order matters. Revealing the recommendation first would turn the reviewer into a witness to the model’s preference and corrupt the independent comparison. A reviewer also cannot serve as a meaningful safety control without enough time, evidence, authority, and a workable exception path.

The pilot measures more than adoption. It follows disagreement, override quality, review time, queue movement, escalation, recovery, and affected-party outcomes. A high acceptance rate may indicate a useful model, automation bias, or a workflow that makes disagreement too costly. Interviews with specialists and downstream teams help distinguish them.

This is also where an online experiment may become appropriate. Random allocation can create a credible counterfactual, but it is not automatically ethical or operationally possible. The team could instead use matched cohorts, a stepped introduction across teams, an interrupted time series, or a bounded before-and-after comparison. Each leaves different bias. Whatever the design, the record must account for contamination between treatments, learning and novelty effects, interference between workers, delayed claim outcomes, volunteer selection, repeated peeking, and averages that conceal segment failures.

Statistical significance does not decide the launch. Effect size and uncertainty sit beside practical value, service reliability, safety, privacy, fairness, staff capacity, and reversibility. An average reduction in queue time cannot compensate for missing urgent claims.

Increase Load Only When the Stop Path Is Ready

Once the workflow evidence is credible, the team sends 5 percent of eligible work to a canary. Candidate and control telemetry carry separate release labels and cover the same time window. Overall service dashboards are insufficient: a badly failing candidate can disappear inside the other 95 percent of traffic.

The first canary signals are those that reveal catastrophic failure quickly—unauthorized actions, unavailable service, disclosure, severe safety violations, broken escalation, or runaway cost. Slower outcomes such as claim quality, complaints, fairness, and downstream harm require a longer stage. The team chooses the first cohort for observability and recoverability, not because its members have less power to object, and excludes claims whose harm cannot be contained or remedied.

For ordinary software, blue-green deployment can keep a complete known-good environment ready while traffic moves to the candidate. A feature flag can disable the model’s influence without replacing the rest of the application. These mechanisms help only if the release bundle is compatible with the old path and the people on duty have authority to use them.

The team rehearses the stop. The kill switch disables model-directed routing. The queue returns to the known-good policy. Alerts remain active through the transition, and an operator verifies that code, schema, threshold, interface, and dependencies agree with the restored bundle.

Stopping the software is only half the rollback. A triage decision may already have delayed a claim, taught a reviewer to rely on the score, or written model output into a case record. Those effects require compensating action: reprocess affected claims, correct records, notify the right people, remove unjustified data, or route cases for human review. A rollback plan that ends at redeployment confuses restored software with restored reality.

Decide from the Full Window

At the launch review, the team can explain what each stage established and what it did not. Synthetic and replay tests established bundle behavior against known cases. Shadow mode exposed live input timing without changing routes. The pilot tested human use and oversight. The canary tested the candidate under limited production authority. None proved permanent safety or usefulness under every future condition.

The decision record answers a compact set of questions:

  • Is the exact release bundle fixed, and have the offline, segment, adversarial, privacy, security, and workflow gates passed?
  • Did the stage use a credible comparison, complete its planned window, and separate candidate evidence by release, cohort, and consequential segment?
  • Were stop rules written before exposure, connected to owned alerts, and exercised against a known-good path?
  • Can technical rollback and compensation address both system state and effects on people?
  • Do the observed benefit, uncertainty, harms, reliability, capacity, and residual risk justify the next exposure ceiling?
  • Will a change to the model, prompt, policy, data, retrieval, tool, interface, or scope trigger proportionate reevaluation?

The claims release advances only for the claim types, teams, and routing authority actually tested. General availability remains an operating state, not graduation from evaluation. Change-aware monitoring, periodic evaluation, incident response, correction and appeal, and reapproval triggers continue after launch.

Now transfer the method to a generative writing assistant. Choose one real workflow and write the first stage that exposes the system to live inputs. Name the question that stage can answer, the comparison it preserves, one fast stop signal, one slow outcome, and one effect that redeployment alone cannot reverse. Then state the maximum authority the next stage could earn.

A launch is credible when increased exposure has produced stronger evidence, and when the organization can still explain—before it needs to—how it will stop.

Source Notes