Ingestion, CDC, Outbox, and Data Movement

The Order Exists, but Search Cannot Find It

An orders service commits a new order, then calls the search service before returning to the user. The search call times out. The order is visible in account history but absent from search, and the request handler cannot tell whether the index accepted the document before the connection failed.

Retrying the entire request is dangerous: payment, inventory, or another side effect may run twice. Retrying only the search call is safer, but the retry exists only in the memory of one process unless the application recorded durable work. A reconciliation job may eventually notice the missing document, but until then the index is available and wrong.

This is the ordinary dual-write failure. One user action tried to change two systems without one authority able to commit both changes. Faster calls, longer timeouts, and a more reliable search cluster can make the window rarer. They cannot close it.

The repair begins by treating movement as part of the data system. A production path must say which fact is moving, where its authority remains, what order the destination needs, where work can restart, how duplicates and deletes behave, and what independent evidence can reveal drift. Those are not additions to the pipe. They are the pipe’s correctness contract.

The Destination Defines What Must Survive the Trip

The same order can travel to several destinations, but it should not carry one generic promise.

The search index serves retrieval. It needs a stable document identity, tenant-scoped visibility, monotonic updates for each order, prompt removal when an order becomes hidden, and a rebuild route. A few seconds of lag may be acceptable; showing a revoked or cross-tenant document is not.

A finance export serves audit and settlement work. It may move only once an hour, yet require a named snapshot, stable grain, duplicate control, a manifest and checksum, and evidence that every expected order is represented. A partner fulfillment feed needs durable delivery evidence and an idempotency key because an external shipment instruction is not safely replayed like an index document.

Other destinations sharpen different parts of the contract. A regional replica must preserve the consistency and recovery promise for which it exists. An archive must balance retrieval, retention, redaction, and deletion. An analytical model needs lineage and restatement rules. Online and offline feature stores need versioned transformations and a way to detect training-serving skew. A cache may sacrifice history while requiring rapid invalidation.

Start with that destination promise, then choose a movement mechanism. Starting with a favorite queue, connector, or processing mode hides the very failures the mechanism must control.

Choose the Capture Boundary

A batch extract is often enough when freshness can be measured in hours. It is strongest when the source snapshot is identifiable, files or partitions are versioned, retries replace output deterministically, and a manifest records what was produced. Its pressure points are long source scans, invisible deletes, and jobs that cannot resume below the whole-run boundary.

Incremental polling can work without access to a database log. Its change marker must be stable and monotonic enough for the promise being made. Pagination needs a deterministic tie-breaker; windows usually need intentional overlap and deduplication; deletes need their own representation. A query for rows “updated since the last run” loses facts when timestamps collide, clocks disagree, or a page boundary moves.

Change data capture, or CDC, reads committed changes from a database log. It is useful when a legacy application cannot publish events, when several derived stores need a consistent feed of row changes, or when the storage-level change itself is the right contract. CDC does not automatically create a public data interface. Raw table mutations can expose private schema, transaction structure, and implementation churn that product consumers should never depend on.

A transactional outbox is a better boundary when the owning service can name a durable business fact. The service writes its state and an outbox record in the same local transaction; a relay publishes the record after commit. The database decides whether both durable records exist, so the request no longer depends on the broker or destination being available at commit time.

Domain-event publication can then carry facts such as order_created, account_suspended, or access_revoked rather than every internal column mutation. File drops remain appropriate for large exports and partner feeds when schema, partitioning, encryption, manifest, checksum, snapshot time, retention, and deletion behavior are explicit.

These mechanisms can coexist. The orders service may publish supported domain events through an outbox, use CDC to rebuild search, and produce signed batch files for finance. Each path still needs its own destination promise and owner.

Follow One Outbox Record to Search

Suppose the order transaction writes two rows: order o_8421 at version 1 and outbox event evt_9930. The event names tenant t_123, the order, its version, event type, schema version, creation time, and a stable idempotency key. If the transaction rolls back, neither row exists. If it commits, the event remains available even while search is down.

The relay reads unpublished outbox rows and sends them to the broker. It marks progress only after the broker acknowledges the publish. A crash after publication but before that progress update sends evt_9930 again. That duplicate is normal. Marking the row first would create the more dangerous failure: a crash could lose an event that was never published.

The search consumer uses (tenant_id, order_id) as document identity and stores the latest applied order version. It creates version 1 once and ignores a duplicate of version 1. If intermediate states matter, it holds a later version until the missing predecessor arrives or the gap is repaired. If only the latest projection matters, a conditional destination write accepts a higher version and rejects stale ones. Unrelated orders can move independently; the contract requires per-order order, not an expensive global sequence.

The consumer advances its broker checkpoint only after the index has durably accepted the update or after retry safety is otherwise established. A crash before the checkpoint may repeat work. A checkpoint advanced before the destination commit may skip it. The event identity, destination write, and checkpoint rule therefore describe one restart story, not three independent settings.

Deletion must use the same path. An order_hidden event names the tenant, object, and new visibility version so the destination can remove the document or apply current policy. A movement path that handles creates beautifully but loses deletes cannot claim eventual consistency; it is a growing disclosure and correctness defect.

The outbox table also becomes production state. Relay lag, poison records, retention, archiving, backpressure, and table growth need owners. A stuck event should be quarantined with its error class and checkpoint position, not silently skipped to make the lag graph green.

CDC Is Snapshot Plus Log, Joined at a Boundary

An empty destination needs history as well as new changes. CDC therefore has two phases: capture a snapshot of existing rows and tail committed changes from a known log position. The difficult part is the join.

Assume the bootstrap records log offset 1050 and begins a consistent snapshot. Orders can continue changing while the snapshot is copied. When the copy finishes, the consumer starts the log after the recorded boundary and applies every later change. Depending on the source’s snapshot semantics, rows changed near the boundary may appear in both phases; stable identity and version handling make that overlap harmless. Starting the tail from an unrecorded “now” can leave a permanent gap.

Large snapshots need bounded chunks, source-pressure limits, pause conditions, and durable progress. The log offset is operational state, not disposable cache. Losing it can cause excess replay, skipped changes, or a full rebuild, so it needs backup and monitoring tied to the identity of the consumer.

The source log may provide commit order, while a broker or destination preserves only per-partition order. The contract must name the order the destination actually needs: per order, account, tenant, file, or another business key. Timestamps alone are weak ordering keys when clocks differ and retries arrive late.

Schema changes also cross the boundary. Adding a nullable field is unlike renaming a status, changing an enum’s meaning, splitting an entity, or changing its identity. A consumer needs compatibility rules, version tests, notification for semantic changes, and a quarantine route for input it cannot safely interpret. The fact that bytes deserialize does not prove that meaning survived.

Deletes need an explicit log representation such as a tombstone, purge event, or retained key. The destination may require more context than the database log keeps after deletion, which is a reason to enrich the change before the source state disappears.

Backfill Without Taking the Source Down

The search repair now needs all historical orders. That backfill is a production workload competing with live transactions, replicas, caches, workers, and the ordinary CDC tail.

Before it starts, name the source snapshot, log handoff, chunk key, rate ceiling, restart point, validation query, owner, and stop condition. Pause automatically when user-facing latency, lock waits, replication lag, error rate, or live-event age crosses its declared threshold. Reserve capacity for current changes so the destination does not become fresher historically while falling further behind now.

Tenant context from the source must survive the copy. A row written under an old placement or policy may need historical interpretation, while access control at the destination usually needs the current authority. The plan should say which meaning is preserved, which is upgraded, and how suspended, moved, or deleted tenants are handled. Bulk work is not exempt from the tenant boundary merely because it runs offline.

Cutover should compare old and new paths while rollback is still cheap. Recent-document samples, missing-delete probes, partition counts or checksums, and domain totals each detect different defects. Rate limits and comparisons must be scoped so one tenant’s import or reindex cannot consume everyone’s recovery margin.

Replay uses the same machinery but raises a sharper question: which outputs are safe to repeat? An index document can usually be replaced by identity and version. A payment instruction, notification, or partner call may need an idempotency contract, compensation, or a replay mode that rebuilds state without re-emitting the side effect. Retention determines how far back the team can replay; deterministic transformation determines whether the result can be explained.

Observe Truth, Not Only Motion

Lag shows whether the destination is current enough for its promise. Throughput shows whether it can keep up. Error rate, dead-letter volume, and oldest backlog age expose poison input and stalled work. Operators should also be able to map a source checkpoint to each destination’s accepted checkpoint; a generic green job status is not enough.

None of those signals proves that the facts are right. A consumer can keep pace while dropping deletes or misreading a schema change. Every important path therefore needs an independent drift check chosen for the destination promise: source-to-index samples with visibility state, partition checksums, missing-delete probes, totals by business dimension, or exhaustive checks for high-severity facts such as access revocations.

Ownership follows the contract. The source team owns event meaning and compatibility. A platform team may own log capture, broker health, and relay infrastructure. The destination team owns projection semantics and repair. Incident routing should already know who can pause a backfill, reject a schema, replay a range, drain quarantine, or degrade the destination.

Record the Movement Contract

The contract should be specific enough that an operator can restart the path and an architecture reviewer can challenge it. For the orders-to-search path, a useful record looks like this:

Purpose and promise:
  Search orders within 10 seconds; never return a cross-tenant document.
  Remove revoked visibility within 5 seconds and enforce newer policy at query time.

Authority and boundary:
  Orders database is authoritative. The search index is rebuildable.
  The outbox transaction is the publication boundary.

Identity and ordering:
  Document key = (tenant_id, order_id).
  Event ID and idempotency key are stable across retries.
  Apply monotonically by order_version; no global order is required.

Checkpoint and replay:
  Advance the consumer checkpoint only after an idempotent index commit.
  Retain replayable events for the declared rebuild window.
  Replays update search state and do not repeat external side effects.

Bootstrap and backfill:
  Name the source snapshot and log handoff.
  Chunk by stable order key, reserve capacity for live changes, and pause on
  source latency, lock-wait, replication-lag, or live-event-age thresholds.

Schema, deletes, and rejection:
  Version the supported event meaning.
  Quarantine unknown semantic changes.
  Propagate hidden, deleted, and tenant-revoked states through explicit events.

Evidence and ownership:
  Expose source and destination watermarks, oldest event age, errors, and
  quarantine volume. Compare recent documents and exhaustively probe revocations.
  Name owners for event meaning, transport, index correctness, and incident repair.

Another destination should produce a different answer. A finance file may substitute a snapshot manifest and checksum for a broker checkpoint. A partner feed may make irreversible side-effect safety the center of the design. The fields remain useful only when the answers reveal the actual risk.

Rehearse the Failures

Take one cron job, dual write, CDC consumer, export, or file feed and write its contract. Then work two repairs far enough to expose the mechanism:

1. Replace a request-time dual write with either an outbox or CDC path. Trace a crash before source commit, after source commit but before publication, after publication but before relay progress, and after destination commit but before consumer checkpoint. State where duplication can occur and why business meaning remains singular.
2. Design a historical backfill while live changes continue. Name the snapshot and handoff, chunk and rate controls, tenant boundary, restart point, delete behavior, source-protection thresholds, validation evidence, cutover, and rollback.

Finally, force an access revocation or delete through every derived copy. If the checkpoint, replay range, ordering scope, schema behavior, deletion route, or drift check becomes vague, the path is still relying on ordinary days.

A trustworthy movement path can be launched, stopped, restarted, and observed without guessing what happened to a fact. That is enough to move data safely. It does not yet answer the harder question of how to replace derived meaning after old inputs arrive or a transformation proves wrong; that requires a correction and reconciliation policy of its own.