Data and Workflow Platforms

The partition that finished twice

At 09:00, the analytics catalog points readers of orders/2026-07-14 to generation 41. At 09:05, a source system delivers a file that should have been in that partition. An operator starts a repair. At 09:12, the ordinary daily schedule starts the same transform. One worker then loses its lease after writing output but before reporting completion.

By 09:20, object storage contains three plausible results:

attempt 771: transform v18, original inputs + late file
attempt 772: transform v18, original inputs only
attempt 773: retry of 771 after its worker lost the lease

Which result may become visible? Can the abandoned worker still report success? If consumers already read generation 41, what does replay mean for them? And if the platform cannot answer those questions, what exactly did its green “job complete” badge certify?

This is the center of a data or workflow platform design. The platform accepts work, but acceptance alone is not the promise. It must say who owns the work, which evidence makes an attempt complete, how pressure is bounded, what output is authoritative, and how a human repairs the contract when automation cannot.

Define the work before drawing the machinery

The unit of work might be a log record, metric sample, event, source file, dataset partition, scheduled job, workflow step, or long-lived timer. Do not allow the noun “platform” to hide the differences among them. For each unit, establish:

• the producer and consumer, including tenant or domain ownership;
• the identity and schema carried with the work;
• the freshness, ordering, delivery, and retention promise;
• the side effects that make retries dangerous;
• the evidence of completion;
• the path for rejection, quarantine, replay, cancellation, and manual repair.

For the late-partition case, assume product services and approved connectors produce events and files. Analytics tables, operational dashboards, feature jobs, and finance reports consume derived datasets. Minute-level dashboards may accept provisional aggregates. A finance partition may arrive later, but it must not silently omit input or change beneath a reader without a new published generation.

The platform retains immutable raw input long enough to perform stated backfills. It processes at least once and earns duplicate-safe outcomes at specific sinks. It never promises global exactly-once behavior across object storage, a warehouse, an email, and an external API. Every such boundary needs its own idempotency or compensation story.

A compact opening to the design is therefore:

I will define the unit of work, its owner, completion evidence, and repair
path. Then I will separate the control plane that authorizes and records work
from the data plane that ingests and executes it. Delivery, backpressure,
checkpointing, publication, and replay will be explicit parts of that contract.

Keep authority out of the volume path

The control plane owns definitions, schema policy, credentials, schedules, dependency graphs, retry rules, tenant quotas, run state, lineage, and manual actions. These records change relatively slowly and need controlled writes, audit, and clear ownership.

The data plane authenticates producers, admits input, buffers it durably, partitions it, executes transforms or activities, and writes results. It must absorb burst, skew, slow consumers, worker loss, and replay without turning the metadata store into the high-volume event path.

The two planes meet through small, explicit records:

input manifest
  source, owner, schema_version, checksum, event_time_range, record_count

run
  run_id, definition_version, input_manifest_ids, target_partitions, state

task
  task_id, run_id, partition, attempt, lease_owner, fencing_token, checkpoint

output candidate
  attempt, object_uri, checksum, row_count, quality_results, lineage

publication
  dataset, partition, generation, chosen_candidate, published_at

Those records make the late file discussable. The repair is not “run the job again.” It is a new run with declared inputs, transform version, target partition, and publication intent.

Let scale reveal the dangerous dimension

Request rate is rarely the only useful estimate. Suppose event ingestion averages 800,000 events per second at 700 bytes each. That is about 560 MB/s, or 48 TB per day before replication and compression. The number affects partitions, network capacity, raw retention, and replay time. It does not tell you whether the system will survive one producer that owns half the traffic or one consumer that has been offline for six hours.

For a metrics system, series cardinality can break cost and query performance while sample throughput still looks ordinary. For logging, indexed bytes and query concurrency may dominate retained bytes. For a scheduler, pending timers, dependency fan-out, the longest lease, and deadline pressure matter more than API calls. For ETL, the largest corrective backfill may be the event for which the platform must reserve capacity.

Name the dimension and let it change the design. Partition event streams by a key that gives the required local ordering, while detecting hot keys. Isolate interactive work from bulk replay. Put per-tenant and per-producer limits at admission, not only on a billing dashboard. Reserve some execution capacity for repair so a saturated live workload does not make recovery impossible.

Admit input as evidence, not truth

An ingestion gateway authenticates the producer, resolves tenant and owner, checks quota, validates the envelope and schema version, and assigns a durable position before acknowledging acceptance. A file lands with a manifest and checksum. An event carries a stable event id, producer identity, schema version, event time, and an ordering key where one exists.

Validation has several outcomes. A malformed or unauthorized record is rejected. A temporarily unprocessable record may enter a bounded quarantine with an owner and expiry. Compatible input is appended to durable storage. The dead-letter path is not an infinite second queue; it needs reason codes, volume limits, access control, redaction, and a decision to repair, replay, or discard.

Backpressure begins here. Admission can throttle one producer, return a retry signal, spill a bounded buffer to durable storage, sample low-value telemetry, or reject work whose contract permits loss. It cannot accept without limit and call the growing queue “durability.” Queue age, oldest unprocessed position, and bytes retained show pressure more honestly than ingestion throughput.

Event time and processing time must remain distinct. A watermark is an operational statement about how much lateness the computation currently expects, not proof that no older event will arrive. The product still needs a policy for input beyond the watermark: ignore it, correct a later view, or open a backfill.

Make execution survive an uncertain worker

The orchestrator expands run 771 into tasks only after storing the run and its dependency state. A worker acquires a time-bounded lease and a fencing token. It heartbeats during long work, reads the declared input manifests, and writes periodic checkpoints where resuming is safe.

A lease is not a guarantee that the old worker stopped. After a network pause, that worker may continue computing while a replacement owns a newer token. Any control-plane state transition must reject the older token. Side effects need the same protection through a destination-supported conditional write, an idempotency key bound to the task and parameters, or an explicit compensation.

For the failed task in run 771, each attempt writes to an immutable attempt location. The original worker’s output can remain as an orphan for later collection; writing bytes does not publish them. Attempt 773 resumes from a verified checkpoint or recomputes. Only the current fenced owner may register its candidate as complete.

This is where retry policy becomes part of correctness. Retry a transient read, but quarantine a deterministic poison input. Limit attempts and elapsed time. Preserve the last useful error. Do not let a retry storm consume the workers needed by healthy tenants. A workflow activity that charged a card or sent a notification needs recorded idempotency or compensation, not a larger retry count.

Publish a generation, not a directory full of files

Completion of every task gives run 771 a candidate output. Before publication, the platform checks checksums, expected partitions, schema compatibility, row counts, domain quality rules, and lineage. These checks can reject suspicious output; they cannot prove that the transform expresses the business rule correctly. Ownership and review remain part of the contract.

Publication writes data to an immutable generation, then atomically changes a catalog record or manifest pointer. Readers either see generation 41 or the new generation; they never discover half of each. The publication record names the input manifests, transform version, quality results, and chosen candidate.

Run 772 cannot overwrite the repaired partition merely because it finished later. It began without the late-file manifest. A per-partition publication coordinator compares the candidate’s declared base generation and publication intent with current state. The ordinary run must wait, rebase onto the accepted inputs, or be discarded. Run 771’s repaired output becomes generation 42 only after its current attempt passes those guards.

“Exactly once” is now a bounded claim. The catalog makes one candidate visible for this dataset partition and generation. A downstream consumer can still receive notification twice or replay generation 42 twice. Give the consumer a stable publication id and require its sink to deduplicate, transact its checkpoint with its write, or tolerate replacement.

Replay must not erase the present

Raw retention makes correction possible, but a replay is a production workload with a different risk profile. Declare its source range, transform version, target partitions, expected cost, publication policy, and downstream effect. Run it in isolated queues or worker pools so it does not starve current input. Throttle its writes to shared sinks and expose progress by event time or partition, not only by task count.

Consumers need to know whether replay emits historical events again, creates a new dataset generation, or sends a correction stream. A consumer that sends email cannot safely treat replay like a table rebuild. A feature store may replace offline partitions while leaving online features under a separate freshness contract. An audit stream may allow a correcting event but not mutation of old evidence.

Lineage answers which inputs and definition produced an output. It does not, by itself, repair anything. The operator surface should show the late source, blocked dependency, stale dataset, failed task, current lease, quarantined records, affected consumers, and available actions. Manual retry, skip, override, and publication require scoped permission, reason capture, and audit because they can change customer-visible truth.

Follow the failure until a user can see it

If consumers fall behind, measure lag in both positions and time. Scale them when work is parallelizable, but also inspect a hot partition, slow sink, poison record, or exhausted quota. Preserve a bounded replay window and decide which low-priority work may be shed. Unbounded lag is a delayed outage.

If a schema changes, compatibility policy belongs at registration and ingestion. Route incompatible data to an owned quarantine; do not let every consumer discover the break differently. A schema that parses may still change meaning, so semantic changes need versioned producers, transforms, and rollout evidence.

If the control plane is unavailable, existing data-plane workers may continue only within already issued leases and policy. New definitions, manual overrides, credential changes, and publication should fail closed or queue behind a clearly chosen boundary. If the data plane is unhealthy, the control plane should remain available enough to show state, stop admission, cancel runs, and coordinate repair.

Measure the promise rather than the component list: acknowledgement latency, rejected and quarantined input, oldest queue age, consumer lag, watermark delay, lease loss, retry amplification, backfill interference, dataset freshness, publication conflicts, and time to repair. Attribute volume, storage, compute, replay, and high-cardinality cost to producer, tenant, dataset, or workflow so operators can act on it.

Protect producer and consumer credentials, connector secrets, workflow inputs, logs, and manual controls. Enforce tenant isolation through topics, object paths, task queues, lineage, observability, and exports—not merely in the API database. Retention and deletion policy must cover raw input, derived output, quarantine, checkpoints, logs, and backups according to the promises actually made.

Transfer the work contract

A logging platform accepts high-volume, variably valuable records. Its decisive policy is how to protect production when the logging path is slow: bounded local buffering, quotas, redaction, sampling where loss is allowed, hot indexed storage for recent search, and colder storage for retention or replay. Parser bugs should be repairable from retained raw records without blocking the service that emitted them.

A metrics system accepts numeric samples whose label sets create series. Its decisive policy is cardinality admission. Bound unsafe labels, aggregate or drop according to an explicit telemetry contract, protect alert evaluation, and publish missing-data behavior. Downsampling and retention should preserve the resolutions that actual alerting and capacity decisions need.

An analytics pipeline accepts events and files and publishes datasets. Its decisive policy is the boundary between provisional freshness and corrected truth: event time, watermarks, immutable raw input, quality gates, versioned partitions, and observable backfills.

An event-ingestion service accepts producer records and offers ordered, retained positions to consumers. Its decisive policy is the ordering boundary. A partition can provide useful local order; global order across all producers is expensive and often meaningless. Schema compatibility, hot keys, consumer lag, replay isolation, retention, and compaction follow from that choice.

A distributed scheduler owns when runnable work receives capacity. Its decisive policy is assignment under failure: dependencies, priority, fairness, leases, fencing, deadlines, cancellation, and stuck-task detection. A workflow engine adds durable state across long waits and external side effects, so definition versioning, idempotent activities, timers, compensation, and operator visibility become central.

An ETL platform accepts source windows and publishes destination versions. Its decisive policy is repeatability. Connectors need isolated credentials, incremental checkpoints, schema-drift handling, raw landing manifests, quality checks, lineage, and a way to rerun one window without exposing partial or older output.

The components change, but the questions hold: what was accepted, who owns it, what can be retried, which state is authoritative, how pressure is bounded, what evidence proves publication, and who can repair the promise?

Rehearse one broken promise

Take the late-partition trace and give yourself twelve minutes. Draw only the input manifest, run, task lease, candidate output, and publication record. Decide what attempt 771 may do after losing its lease, how attempt 773 avoids duplicating an external side effect, and why run 772 cannot replace the repaired partition.

Then change the prompt to a workflow that calls a partner API. Identify which parts of atomic dataset publication no longer apply and where idempotency, side-effect evidence, or compensation must replace them. Finally, change it to a metrics system and name the pressure that should be rejected before durable admission.

When reviewing the design, ask:

1. Can a producer tell whether work was rejected, accepted, delayed, or completed?
2. Does every retry have an idempotency, fencing, or compensation boundary?
3. Can live traffic survive the largest plausible replay or backfill?
4. Can a consumer identify the authoritative generation and its inputs?
5. Can an operator find the owner, failure, affected output, and safe action?
6. Do security, observability, and cost follow the chosen contract rather than appear as a closing recital?

The short scratchpad sequence is: define the work; separate authority from volume; admit with bounds; execute under leases; publish immutably; replay in isolation; expose repair.

Collaboration platforms inherit these asynchronous-work problems but add a sharper question: whether tenant and permission context survives every queue, index, notification, export, and administrative action.

Related links

• Databases and Storage Engines
• Transactions and Isolation
• Queues, Logs, and Stream Processing
• Distributed Systems Fundamentals
• Observability, Performance, and Cost
• Consistency, Idempotency, and Transactions
• Reliability and Failure Engineering