Batch, Streaming, Reprocessing, and Reconciliation

The Dashboard Is Fixed. The History Is Not.

Consider a commerce team that discovers its revenue pipeline has counted refunded shipping fees as revenue for three weeks. The streaming consumer is patched within minutes. New refunds now produce the right number, the lag graph is green, and every job is succeeding.

The revenue dashboard is still wrong.

Three weeks of warehouse partitions contain output from the old rule. Finance has already exported some of those totals. A delayed settlement file will arrive tomorrow carrying events whose business dates fall inside the affected period. If the team simply reruns the job, the current lookup tables may differ from the ones used originally. If it appends corrected rows, readers may count both versions. If it overwrites the old partitions immediately, it loses the rollback point before proving that the correction is sound.

This is the work that the batch-versus-streaming argument obscures. Streaming shortens the time between a new fact and a new answer. It does not by itself correct answers already produced. Batch can recompute a bounded history, but a scheduled job is not automatically reproducible or safe. Trust comes from a data path that can establish a baseline, apply changes, revisit old inputs, replace derived meaning, and produce independent evidence that source and destination agree.

The unit of design is therefore not a batch job or a stream processor. It is a correction-capable data path.

Every Answer Has Two Clocks

The refund has an event time: when the business event occurred. It also has a processing time: when the pipeline learned about it. Those clocks diverge when a mobile client reconnects, a partner sends a delayed file, a payment settles later, an operator repairs a record, or a source system accepts a backdated change.

The pipeline cannot resolve that divergence with timestamps alone. It needs a business rule. Should a refund received after monthly close reduce revenue on the original purchase date, appear as an adjustment in the current period, or wait for approval before reopening the closed period? Each answer can be valid in a particular domain. Letting the stream processor’s default watermark choose is not a policy.

Different facts may also demand different correction speeds. A late refund may wait for the next controlled restatement. An access revocation may need to disappear from every serving path within seconds, even if historical audit reports are rebuilt later. A privacy deletion may need an exhaustive propagation check rather than the sampling used for an ordinary analytical correction.

For each derived answer, name both clocks and the rule that connects them: the allowed lateness, the treatment of closed periods, the authority to backdate, and the response when a fact arrives outside the ordinary window. That rule determines what the machinery must do.

Batch and Streaming Do Different Work

Return to the revenue pipeline. Its streaming path exists because product and operations want intraday visibility. It consumes committed order, refund, shipping, and settlement facts, then updates provisional revenue continuously. Stable event identity makes delivery retries harmless. Ordering is required within an order or settlement, not across the entire business. A watermark says when an interval is probably complete enough to display, while late events remain possible.

The batch path closes daily partitions. It reads a named input range, applies the same revenue meaning, validates whole-day totals, and publishes a result suited to reporting. It also provides the practical route for initial history, scheduled cost control, and bounded recomputation. None of those strengths requires batch to be the only path.

The two paths make different promises. The stream answers, “What do we know now?” The close answers, “What result are we prepared to stand behind for this period?” If both implement the revenue rule separately, they can drift even while each remains internally healthy. Shared transformation code reduces that risk, but shared code is not independent proof. The paths still need versioned inputs and outputs, side-by-side comparison, and reconciliation against business invariants.

Initial loads reveal the same complement. A new warehouse table begins from a named snapshot or manifest while live facts continue to arrive. Its movement contract supplies the handoff: a log position, high-water mark, or intentionally overlapping window with stable identity. The longer-lived promise is that the baseline and every later update remain rebuildable when old input arrives or old logic proves wrong.

Reprocessing Creates a Candidate, Not the Truth

The revenue repair begins by freezing what the words “three weeks” mean. The team records the order, refund, shipping-fee, and settlement inputs for the affected dates, including the delayed-input policy. A file manifest, source snapshot, event-offset range, partition set, or immutable object version can provide that identity. “Whatever the source contains when the job runs” cannot.

Next it names the transformation. Revenue rule v2 includes the corrected refund classification, but code is only part of the identity. Schema versions, configuration, exchange-rate data, tax rules, merchant mappings, model versions, and other lookups can change an output. A rebuild is explainable only when the team can say which versions produced it.

The job writes to shadow partitions rather than the serving partitions. That separation turns reprocessing into a candidate build. It gives validation somewhere to work and preserves the old answer for rollback. Depending on the destination, the final replacement might be an atomic partition exchange, table or index alias swap, versioned serving view, immutable correction entry, or controlled overwrite. The choice should prevent readers from accidentally seeing old and new output together.

The rebuild must also separate replaceable state from irreversible effects. A partition can be replaced, a search document overwritten, and a cache flushed. A payment instruction, notification, partner email, or webhook cannot safely be emitted again merely because an old event was replayed. Replay mode must suppress those effects or rely on durable idempotency and compensation designed for them.

Determinism makes the candidate auditable. Given the same input identity and transformation identity, the replaceable output should be the same. Current time, random sampling, mutable reference data, unversioned configuration, and live external calls are common sources of accidental variation. If nondeterminism is necessary, record and control it rather than pretending repeated runs are equivalent.

Idempotency applies to the rebuild itself. A worker crash, repeated partition, or overlapping range must not duplicate business meaning. Output replacement by stable key and version is usually safer than blind append. Checkpoints should say which bounded work is complete without making a half-built candidate eligible for serving.

Reconciliation Must Be Able to Disagree

A successful reprocessing job proves that code ran. It does not prove that the new revenue is right. Reconciliation needs an independent route from authoritative facts to the promise being checked.

For the revenue repair, row counts can reveal a missing partition, but equal counts can conceal a wrong classification. Canonical partition hashes can reveal changed values, but they do not say whether the change is legitimate. Sampled record traces can expose a broken join or an unexpected transformation, but a sample can miss a rare duplicated payout. Domain totals by day, currency, refund reason, merchant, and settlement status test the meaning finance relies on. An exhaustive check for a high-severity invariant—such as one settlement contributing at most once—covers what sampling must not be trusted to find.

These checks complement one another because they fail differently. Their queries and code should not simply repeat the transformation under test. If both sides reuse the same faulty classification, they can agree on the same wrong answer. Prefer direct source controls, independently computed aggregates, known control records, and comparisons at the grain where the business promise lives.

Reconciliation must end in a decision. For this repair, the result names the affected range, old total, candidate total, expected delta, unexpected residual delta, first bad partition, failed invariant, and owner. It then says whether to publish, investigate, repair automatically, suppress serving, or roll back. A report that no one is obliged to interpret is delayed observability, not a control.

Sampling remains valuable for broad field-level drift, especially when an exhaustive comparison would be costly. Its limit should be explicit. One missed revocation, one duplicated payout, or one retained deletion may be unacceptable even when a random sample looks clean. Match the strength of the check to the consequence of a miss.

Restatement Is a Product Event

When people have acted on an answer, replacing it is more than a storage operation. Finance may have closed a period. Customers may have downloaded reports. Support may have explained a charge. An external partner may have imported an extract.

The commerce team therefore promotes revenue v2 with an effective time, a record of affected periods, and the reconciliation evidence. The serving view changes only after approval. Version v1 remains available through the rollback window. The dashboard marks the restatement; finance receives the delta and decides whether downstream statements need replacement or adjustment entries.

Late and backdated facts follow the same declared policy. Some reopen historical output. Some create a current-period correction. Some enter an exception queue for human approval. Deletions and redactions propagate through every relevant copy, including exports and cached or modeled data, and receive a stricter completion control when retention would violate policy or law.

The incident closes only when the historical candidate reconciles, the serving version changes safely, new streaming output continues to reconcile under v2, downstream consumers receive the required correction, and the failed control has been improved. Finishing the backfill is one event inside that sequence.

Hybrid Architecture Is a Repair Choice

A team can keep separate streaming and batch implementations. This lambda-style shape may be justified when immediate estimates and authoritative close have different latency or computational needs. Its cost is semantic duplication: every rule change can make the paths disagree. Version alignment and continuous comparison become first-class operations.

A team can instead retain a durable event history and use one transformation for live processing and replay. This kappa-style shape reduces duplicate logic but does not remove correction work. Retention must cover the rebuild range, processors must remain reproducible, replay capacity must coexist with live traffic, and unsafe side effects must be fenced off.

In practice, both often rest on snapshot plus stream: a baseline, an incremental tail, and periodic reconciliation. Choose among them by asking how the system repairs old meaning. Can it identify the inputs? Can it reproduce the transformation? Can it build output without corrupting the serving version? Can it prove the candidate against the source? Can it afford the replay while production remains live? Pattern names are secondary to those answers.

Record One Operable Correction Plan

The plan should be specific enough for an operator to run and for a reviewer to stop. For the revenue repair, it might read:

Affected answer and authority:
  Daily recognized revenue is derived. Orders, refunds, shipping fees, and
  settlement records remain authoritative.

Input and time boundary:
  Rebuild 2026-06-08 through 2026-06-28 from manifest rev-2026-07-18-01.
  Book events by business date. Refunds received after close become labeled
  adjustments unless Finance approves reopening the period.

Transformation identity:
  Revenue rules v2, schema v7, merchant mapping 2026-06-30, currency-rate
  snapshot fx-2026-06-final, and configuration digest 91c4... produce output v2.

Build and side-effect safety:
  Write shadow partitions revenue_v2; replace by partition only after approval.
  Replay writes derived state and emits no invoices, notifications, or partner calls.

Validation and reconciliation:
  Compare counts and canonical hashes by partition; compare revenue by day,
  currency, refund reason, merchant, and settlement status; exhaustively verify
  settlement uniqueness; trace refund-heavy merchants separately.

Publication and rollback:
  Finance and the data owner approve the residual delta. Swap the serving view,
  retain revenue_v1 for seven days, mark restated dates, and send the control
  report to downstream owners. Roll back on failed invariants or unexplained delta.

Operations:
  Cap rebuild I/O at 20 percent of measured source headroom. Pause on source p99,
  replica lag, or live-consumer age thresholds. The data-platform on-call owns the
  run; the revenue-data owner owns meaning and publication.

A permissions index, search catalog, training set, or partner export needs a different plan because its authoritative facts, intolerable misses, replacement mechanics, and communication duties differ. The form is useful only when its answers expose those differences.

Rehearse the Repair Before You Need It

Take one derived answer your system already serves and introduce a bounded error: a three-day transformation bug, a late source file, a missed delete, or a changed reference table. Name the exact input and transformation versions, build corrected output beside the current version, and decide which checks can genuinely disagree with the code under repair. Then trace publication and rollback as if users had already consumed the old answer.

Repeat the exercise with one irreversible side effect in the path. If replay would send it again, the correction design is incomplete. Repeat it once more while live traffic continues. If the rebuild can consume all available source, broker, or destination capacity, the system has a correction mechanism that may be unusable when production is under pressure.

Freshness is only one dimension of trust. A durable data path can show what it knows now, revisit what it believed before, replace an answer without losing control, and prove why the new answer deserves to serve. The next constraint is physical: reprocessing, reconciliation, shadow output, and rollback retention all consume capacity. Repair work must fit inside the production system it is meant to restore.