Case Study: A Burst-Tolerant Event Pipeline

Northstar accepted every telemetry record during the product launch. Producers received acknowledgements, brokers retained their replicas, and the daily reconciliation found no missing event identifiers. The incident review still opened with a failed objective: the p99 age of the near-real-time aggregate reached more than six minutes.

Both statements were true. Nothing was lost, and the product was late.

That distinction changes the design review. An ingest counter proves that a boundary accepted work. It does not prove that the work reached a correct materialized view while it was still useful, that a hot partition remained serviceable, or that the system could replay retained history without displacing live demand. A queue stores bytes, but it also stores promises about time, state, and recovery.

This case builds those promises for a fictional telemetry platform. Northstar receives 180,000 records/s in steady operation and 420,000 records/s during a ten-minute launch burst. It produces minute aggregates for operational dashboards, keeps source history for correction and audit, and supports historical reprocessing. The numbers are deterministic modeled teaching evidence, not measurements of Kafka, Flink, or a cloud streaming product.

The design lesson is: a streaming system’s central SLO is often the age and recoverability of queued work, not raw ingestion throughput.

Start with five dimensions, not one rate

The overloaded word event hides several units. Northstar’s capacity packet names them before drawing a topology.

One logical record can produce several physical operations: a replicated broker append, deserialization, validation, state reads and writes, a sink mutation, checkpoint bytes, and telemetry. Northstar reports accepted logical records, processing attempts, and materialized effects separately. A replayed record is another processing attempt, not another logical event. A sink write that times out may be an ambiguous attempt until reconciliation proves whether its effect exists.

The freshness clock begins at source event time when the producer clock is trustworthy within a declared bound. It also records broker append time. Comparing the two separates upstream collection delay from pipeline delay. The terminal timestamp is the moment a versioned aggregate is queryable, not the moment a processor emitted an RPC. If the sink commits at 10:03 but its serving replica becomes visible at 10:05, the user-facing age includes those two minutes.

Northstar uses four objectives for this packet:

• p99 event age below 10 seconds in steady state;
• maximum modeled event age below 420 seconds during the declared ten-minute burst;
• burst backlog drained within 1,500 seconds after arrivals return to steady state;
• zero accepted-record loss and zero duplicate materialized effects within the fixture’s reconciliation boundary.

The last line is intentionally scoped. It is not a claim that a record crosses every network exactly once. It states what the source, state, and sink protocol must make true at the materialized-effect boundary.

The workload envelope includes the burst’s shape

Northstar’s average payload is 640 bytes for this workload version. The distribution still matters: the admission layer rejects records above the declared maximum, and capacity tests include the p95 and maximum encoded sizes. The 640-byte value is only the input to this worked example.

The arrival envelope has three modes:

The launch burst is not represented as “2.3× average.” Duration determines stored work. Correlation determines partition heat. Payload size determines bytes. A ten-second peak and a ten-minute plateau can share the same maximum rate while requiring radically different buffers and drain plans.

The live processor has a tested boundary of 260,000 records/s with checkpointing, schema validation, state mutation, and sink acknowledgement enabled. That is lower than the broker’s append boundary. Northstar deliberately allows the broker to absorb work faster than processors can complete it for a bounded interval. If the team advertised only the broker’s 500,000-record/s ingest result, it would confuse acceptance capacity with end-to-end useful capacity.

Partitioning is a correctness and skew decision

The initial design used tenant_id as the partition key because all telemetry “belonged to a tenant.” During the burst, the largest tenant produced 18% of records:

420,000 records/s × 0.18 = 75,600 records/s

One modeled partition remains inside its service-demand and checkpoint boundary at 7,000 records/s. Ninety-five idle partitions cannot help the one receiving 75,600 records/s. Topic-wide utilization would look comfortable while that tenant’s event age grows without bound.

The ordering requirement is narrower than tenant order. Northstar needs per-entity sequence for state changes to the same device and metric series. It does not need a total order across every device owned by a tenant. The selected key is therefore (tenant_id, stable_entity_bucket), where the bucket is derived from an entity identifier. The top tenant uses sixteen stable buckets:

75,600 records/s ÷ 16 buckets = 4,725 records/s per bucket

That favorable calculation assumes even bucket distribution and no correlated super-entity. The production gate checks the actual maximum, p99, and Gini-like concentration across partitions; it does not accept the arithmetic as proof. A tenant with one device producing most of its traffic may still create a hot bucket. Such a device needs a product decision: can its operations be split by metric family or time window, or is strict order important enough to cap its admitted rate?

Changing a partition function is a state migration. Old and new producers cannot freely hash the same entity into unrelated partitions while consumers assume one ordered stream. Northstar introduces a partitioning epoch in the envelope. During migration, a router sends one entity to one epoch, state is checkpointed and transferred, and the old epoch is fenced before the new owner advances the entity’s sequence. A consumer rejects an older epoch even if its record arrives later.

Apache Kafka’s design documentation is a useful implementation reference because it makes the partition boundary visible: consumer position is an offset per partition, and ordering is retained within a log. That does not select Northstar’s key or prove its end-to-end semantics. It supports the narrower transfer that ordering and parallelism must be reasoned about at the partition boundary. See Apache Kafka design.

One picture carries four operating artifacts

The diagram is not a vendor deployment map. A real broker cluster adds replicas, leaders, network paths, controllers, and storage tiers. A real processor adds task slots, state backends, and sink connectors. The analytical topology intentionally exposes only the boundaries required for the capacity and recovery decisions.

Size the backlog before sizing the disk

During the burst, arrivals exceed live processing by:

420,000 arrivals/s − 260,000 completions/s
  = 160,000 records/s of backlog growth

Over 600 seconds, the backlog reaches:

160,000 records/s × 600 s = 96,000,000 records

At 640 application payload bytes per record, that is 61.44 GB of incremental payload. Northstar applies a 1.25 storage factor for the declared envelope, indexes, and modeled encoding overhead, then a replication factor of three:

96,000,000 records × 640 bytes/record × 1.25 storage factor
  × 3 replicas = 230.4 GB

The reservation adds a 1.5 safety factor, producing 345.6 GB of incremental replicated buffer. This value is not the total topic size. Baseline retention, segment fragmentation, other topics, compaction workspace, broker repair, filesystem reserve, and a replica rebuilding while the burst is active all require separate capacity. Compression is also not deducted from the safety case until its distribution is measured under the burst’s actual data.

Time-based retention and byte capacity must agree. Kafka’s topic configuration reference, for example, notes that retention.bytes is enforced per partition and that time and byte retention operate independently. A system can declare six hours of retention and still delete earlier under a byte bound, or provision aggregate bytes while one partition reaches its local bound first. Northstar checks minimum readable time for the hottest partition under the maximum declared payload distribution. See Apache Kafka topic configurations.

Buffering is safe only if the post-burst system has positive drain capacity. At steady arrivals, the live pool drains at:

260,000 completions/s − 180,000 arrivals/s
  = 80,000 records/s of drain capacity

The 96 million-record backlog therefore needs:

96,000,000 records ÷ 80,000 records/s
  = 1,200 s = 20 minutes

If the burst recurs every fifteen minutes, this design is unstable even though it survives one isolated burst. If steady load grows to 250,000 records/s, drain falls to 10,000/s and the same backlog takes 160 minutes. A buffer postpones overload; it does not create service capacity.

Record lag and event age answer different questions

At the burst peak, the consumer group is 96 million records behind. That is useful for storage and work accounting, but it does not directly state how late the oldest record is. Under the fixture’s FIFO and homogeneous-service approximation:

96,000,000 records ÷ 260,000 records/s
  ≈ 369.23 s of modeled oldest-event age

The modeled oldest event is about 6.15 minutes late, inside the declared 420-second burst objective. The backlog takes twenty minutes to disappear because new steady arrivals continue consuming 180,000/s of the processor boundary. Drain time and maximum waiting age are related but not identical.

That approximation breaks under skew and event-time disorder. A hot partition can contain an old record while the group-wide record lag falls. A poison record can block one ordered lane. A late producer can append an event whose source timestamp is hours old even when broker lag is zero. Northstar therefore reports distributions of:

• source-to-broker delay;
• broker residence age for the oldest unread record per partition;
• processing queue and service time;
• event-time watermark delay, with idle-partition handling stated;
• sink acknowledgement and serving-visibility delay;
• end-to-end materialization age by tenant, schema version, and bounded workload class.

It also reports records and bytes behind per partition, not only group totals. The operational alert combines age and slope: “oldest unread age above 300 seconds and still increasing” is more actionable than a static count whose meaning changes with record size and processing cost.

Checkpoints bind state to replay position

Northstar aggregates per entity and minute, deduplicates event identifiers, tracks schema identity, and emits versioned sink mutations. Restarting from a broker offset without restoring the corresponding state would mix two timelines. Restoring state without the corresponding source position would omit or duplicate work.

The processor takes a consistent checkpoint every thirty seconds. At the 260,000-record/s processing boundary, the simple upper bound between checkpoints is 7.8 million processed records. It is an exposure estimate, not a guarantee that every failure replays exactly that many: alignment delay, in-flight buffers, per-partition position, checkpoint completion, and the point of failure determine the actual replay.

On recovery, the system restores keyed state and source positions from one completed checkpoint, then replays retained records. A replay may execute transformation code again. Correctness depends on what the sink does with repeated attempts.

Northstar writes each materialized result with (aggregate_key, window, result_version) and a source-position vector or equivalent monotonic generation. The sink accepts a newer version, returns the existing result for the same version and content hash, and rejects conflicting content for a reused identity. An external side effect that cannot participate in this protocol goes through an idempotent outbox consumer whose effect identity outlives the replay horizon.

Apache Flink’s fault-tolerance documentation draws an important boundary: “exactly once” managed-state semantics do not mean each event physically flows only once, and end-to-end exactly-once behavior requires replayable sources plus transactional or idempotent sinks. Northstar uses that boundary rather than applying the phrase to the whole topology. See Apache Flink fault tolerance and connector fault-tolerance guarantees.

The deduplication horizon is twenty-four hours, longer than the declared online retry and ordinary replay window. That horizon does not make an arbitrary historical replay safe. A six-month archive may contain event identifiers whose online dedup state expired. Historical correction therefore writes to a versioned shadow output or uses a replay generation, then reconciles and promotes results. It does not inject old records blindly into the live effect identity.

Schema evolution is part of replayability

Retaining bytes does not guarantee that current code can interpret them. Northstar stores an immutable schema identifier with each record and retains the corresponding writer schema. A reader declares which writer versions it accepts, how defaults are applied, and which changes require migration rather than implicit resolution.

The compatibility rules are behavioral:

• adding an optional field with a valid reader default may be compatible, but the default represents missing historical information rather than a value the old producer actually observed;
• renaming requires an explicit alias or a migration; deleting and recreating a similar field is not automatically the same fact;
• changing units from milliseconds to seconds is semantically incompatible even if both fields are integers;
• changing enum values needs an unknown-value policy;
• changing event identity, partition key, or event-time semantics requires a coordinated state migration.

The Apache Avro specification distinguishes writer and reader schemas and defines schema resolution, defaults, and aliases. It provides a concrete serialization mechanism, not a substitute for Northstar’s semantic review. See Apache Avro 1.12 specification.

Every release runs a replay corpus containing each retained schema generation through the candidate reader. It checks parse success, semantic invariants, partition-key derivation, event-time behavior, and materialized result differences. A schema registry saying “backward compatible” is necessary evidence for some encoding changes, but it cannot prove that a unit change or business meaning remains safe.

Poison data is isolated rather than allowed to pin a partition. At steady load, the fixture injects 0.05%, or 90 records/s, into a quarantine lane with capacity for 1,000/s. The lane retains original bytes, schema identity, partition, offset, validation error, and a privacy-safe diagnostic fingerprint. Advancing the main partition past a poison record is an explicit semantic choice: the affected aggregate is marked incomplete until correction, and replay later fills the gap with a newer result version. Silently dropping the record would make the pipeline look fresh by redefining correctness.

Live, backfill, and recovery are different workloads

Historical reprocessing is valuable precisely when production logic or data was wrong. It is therefore likely to be large, cold, and urgent. Treating it as spare live work invites the correction to cause a second incident.

Northstar declares three resource pools:

The pools have separate consumer groups, process slots, state I/O budgets, sink admission, network queues, and observability. Merely assigning different process names is not isolation if they share an unbounded sink connection pool or saturate the same checkpoint store. Storage read IOPS, network egress, decompression CPU, schema service, and destination write capacity all need partitions or admission controls.

The six-hour replay in the fixture contains:

180,000 records/s × 6 hours × 3,600 s/hour
  = 3.888 billion records

At the isolated 60,000-record/s recovery boundary, it takes 64,800 seconds, or eighteen hours. That may be acceptable for a historical correction and unacceptable for a disaster recovery objective. The design makes the conflict visible. To shorten recovery, Northstar must reserve more end-to-end recovery capacity, reduce the recovery set through a valid checkpoint or snapshot, or change the objective. It may borrow capacity only through an explicit controller that verifies live age, sink headroom, checkpoint health, and a rapid revocation path.

A reprocessing plan names:

1. immutable input range, schema set, and replay generation;
2. reader and transformation versions;
3. partition function and ordering requirements;
4. expected records, bytes, state size, and destination writes;
5. backfill and recovery admission rates at every shared dependency;
6. shadow-output identity and reconciliation checks;
7. pause, resume, rollback, and promotion conditions;
8. cleanup ownership and evidence retention.

Live output is not overwritten in place until the shadow generation reconciles counts, keys, aggregates, late-data rules, and sampled source-to-result traces. Promotion is a metadata or serving-authority change with a reversible pointer when possible. If the destination cannot support versioned generations, the plan needs a compensating restore path before replay begins.

Three capacity strategies make different promises

The selected buffer-and-drain design is not the only plausible architecture. The review compares it with two alternatives under the same launch, sink, and replay states.

Provision end-to-end for the full burst

Northstar could provision processors and sinks to complete 420,000 records/s without intentional backlog. In the ideal fluid model, burst-induced queue age approaches zero. The design also needs enough capacity during one broker loss, a slow sink, checkpoint I/O, and repair; “420,000 workers” is not an end-to-end boundary if the sink remains at 260,000 useful mutations/s.

Full-burst provisioning is attractive when freshness is strict, bursts are frequent, or queued data loses value quickly. Its costs include idle steady reserve, larger state and connection populations, checkpoint fan-out, and the risk that autoscaled workers arrive too late or cold. Northstar would select it if launch event age had to remain below seconds rather than seven minutes. The acceptance test would hold the full path at 420,000/s with checkpointing and one declared failure, not extrapolate from a single operator benchmark.

Buffer and drain within an age objective

The selected design provisions 260,000/s of live completion, uses the durable log to absorb a known ten-minute envelope, and reserves an 80,000/s post-burst drain margin. It is economically useful when bursts are bounded and the product accepts the calculated 369-second age.

Its central risk is ratcheting backlog. Growth in steady demand, longer bursts, repeated launches, or a concurrent sink slowdown can remove the drain margin. The controller continuously recomputes predicted peak records, oldest age, and drain completion from current arrivals and useful completions. Crossing the forecast boundary triggers admission or product degradation before disk fills. Disk-free percentage is a late signal because time can become unacceptable while plenty of storage remains.

Aggregate or sample before durable central admission

Edge agents could aggregate repetitive telemetry, sample optional diagnostic events, or send sketches rather than raw records. This reduces records and bytes before the central bottleneck. It is a semantic redesign, not transparent compression. A sampled trace cannot later answer an exact per-device audit question; an edge aggregate can conceal distribution shape or prevent a corrected historical computation.

Northstar applies this only to explicitly optional diagnostics with versioned sampling policy. Contracted billing, safety, and audit events retain individual identity. The producer reports considered, emitted, aggregated, and dropped populations so central success rates cannot pretend the missing work never existed. During overload, policy changes are signed, time-bounded, and reversible; a device does not invent its own sampling rate.

The decision is conditional:

Northstar records buffer and drain as the current decision because it meets the declared age and cost boundary while preserving raw source history. It records full-burst capacity and semantic reduction as revisit options, not rejected ideas that can never become correct.

The economic comparison includes the cost of being late. Buffering saves steady compute only if six-minute launch data remains useful. For an operational dashboard, that delay may slow an incident decision; for billing, lateness may be acceptable while loss is not; for automated safety control, minutes may make the data worthless. Northstar attaches a product consequence and owner to each freshness class. Optional diagnostics can share the 420-second burst objective, while alert-triggering telemetry receives a smaller reserved lane and a stricter age gate. Priority does not permit reordering events within one entity state machine; it selects capacity and admission before that boundary.

Cost per retained payload byte is also incomplete. The team prices broker storage, replica and repair network, checkpoint storage, processor and sink service demand, quarantine, replay, and the operational reserve needed during failure. A cheap archive with an untested 60,000-record/s restore path may be expensive when an eighteen-hour correction blocks a product decision. Conversely, provisioning every stage for 420,000/s can waste capacity if the burst occurs twice a year and the product accepts bounded age. The decision record therefore carries both steady cost and the modeled cost and duration of one recovery.

The revisit trigger is quantitative: steady arrivals above 210,000/s, burst duration above 600 seconds, a second burst before drain, hottest bucket above 6,300/s for two windows, predicted event age above 360 seconds, or recovery finish outside its declared objective opens a new capacity decision. Those thresholds sit below hard failure boundaries to leave time for admission or scaling. They are policy inputs, not universal streaming rules.

Recovery is a controlled mode, not “resume consumers”

The operational sequence prevents enthusiasm for catch-up from overwhelming the path that is still healthy.

1. Name the failed boundary. Determine whether acceptance, one partition, processor state, checkpoint storage, sink goodput, or serving visibility is impaired. Freeze unrelated partition and schema migrations.
2. Preserve retained truth. Confirm the minimum readable source position, writer schemas, completed checkpoint, and sink generation. Stop deletion or compaction changes that could erase the recovery range.
3. Protect live work. Enforce live, backfill, and recovery admission at processor, storage, network, state, and sink boundaries. Pause planned backfills before borrowing from live reserve.
4. Choose a consistent start. Restore one completed checkpoint with its source positions, or start a new shadow generation from a declared source range. Never combine the newest state snapshot with unrelated newer offsets.
5. Prove one partition. Replay a bounded canary partition through schema resolution, state restore, sink versioning, and reconciliation. Include a duplicate and an ambiguous sink attempt.
6. Expand by age and goodput. Increase recovery only while live p99 age, checkpoint duration, sink queue, and useful completion remain inside gates. Record predicted finish time after each stage.
7. Reconcile before authority changes. Compare logical identities, state keys, aggregate versions, completeness markers, and sampled source-to-result traces. A matching record count alone is insufficient.
8. Return borrowed capacity deliberately. Reduce recovery rate, restore ordinary admission, resume backfills, and re-enable optional producers one control at a time.
9. Retain the evidence. Preserve input range, schemas, code and policy versions, checkpoints, rates, faults, reconciliation, and operator decisions for the next restore test.

The exit condition is not zero consumer lag. Recovery is complete when the intended generation is authoritative, logical records reconcile to correct effects, live age is steady, borrowed capacity is returned, quarantine has an owner, and another restart can use a known checkpoint. A queue can be empty because work was skipped; emptiness without reconciliation is not recovery.

Backpressure must reach an admission decision

A slow sink first appears upstream. Processors spend more time waiting for output buffers, checkpoints take longer, broker lag grows, and producers may still receive acknowledgements because the durable log has space. The system is functioning as designed only while that state remains inside the declared envelope.

Northstar does not respond by adding unbounded processor concurrency. More in-flight records can consume memory and sink connections without increasing completion capacity. The controller distinguishes:

• source pressure: broker append, replication, or disk is constrained;
• compute pressure: transformations or keyed state exhaust CPU or memory;
• coordination pressure: checkpoint barriers or state upload delay progress;
• sink pressure: admitted mutations exceed useful sink completion;
• skew pressure: one partition is hot while aggregate capacity is idle.

Flink’s backpressure documentation offers a concrete set of observable mechanics: downstream inability to consume propagates upstream, and task time can be separated into busy, idle, and backpressured portions. Northstar’s exact runtime may differ, but the causal transfer is valuable: measure where output stops being accepted rather than interpreting backlog alone. See Apache Flink backpressure monitoring.

Admission policy uses the earliest safe boundary. Optional debug telemetry is sampled before durable append when the platform is in a declared degraded mode. Contracted telemetry is admitted up to the tested broker envelope and then receives a bounded failure response; it is not accepted and silently discarded. Producer retries have jitter, a maximum age, and a stable event identifier. The system counts retry attempts separately so a network fault cannot masquerade as new logical demand.

Four fault trials establish recoverability

The test harness drives arrivals from a fixed schedule independent of completions. Otherwise a slow consumer can make a closed-loop generator send less work and produce a flattering result. Each trial reconciles accepted records, broker positions, checkpoint state, quarantine entries, sink versions, and visible aggregates.

Broker loss during the burst

For ninety seconds, broker repair and leadership movement reduce effective processing to 200,000 records/s while arrivals remain 420,000/s. The incremental backlog is:

(420,000 arrivals/s − 200,000 completions/s) × 90 s
  = 19.8 million records

After normal live capacity returns and arrivals are steady, the 80,000-record/s drain margin removes that increment in 247.5 seconds. The test fails if acknowledged records disappear, the hottest remaining partition exceeds its disk or network boundary, leaders flap repeatedly, or repair traffic consumes the storage reserve assumed by the burst calculation.

Replication acknowledgement policy is tested, not inferred from “three replicas.” If the system acknowledges before the declared durable replica condition, the loss model must include records that existed only on the failed node. If it refuses writes during insufficient replication, producer retry and admission behavior must remain bounded.

A sink slows for twelve minutes

The sink boundary falls to 140,000 records/s under steady 180,000/s arrivals. Backlog grows by 40,000/s for 720 seconds, reaching 28.8 million records. Once the sink recovers and live capacity returns to 260,000/s, that backlog drains in 360 seconds.

The important evidence is not merely “caught up.” The trial checks that checkpoint duration remains bounded, processor memory does not grow with pending sink calls, cancellation releases connections, sink idempotency survives ambiguous timeouts, and live event age declines monotonically after recovery. If the sink completes only 140,000 useful mutations while clients launch 260,000 attempts and time out the rest, reported attempt throughput is not goodput.

Poison data enters an ordered partition

The generator emits 90 malformed or semantically invalid records/s, including one record in a hot entity sequence. The processor writes a bounded quarantine record, advances according to the incomplete-aggregate policy, and emits a visible completeness marker. Operators correct the schema mapping and replay the quarantined identity into a newer aggregate version.

The test fails if one poison record blocks unrelated entities in the same broker partition indefinitely, if automatic retries spin without a maximum, if quarantine loses the original provenance, or if the dashboard presents an incomplete aggregate as complete. A dead-letter queue is not success; recoverability includes the route back.

A full six-hour replay runs beside live traffic

The recovery group reads 3.888 billion records at 60,000/s for eighteen hours while live traffic continues. The gate asserts that live capacity, p99 event age, checkpoint success, and sink admission remain within their steady objectives. Recovery age is tracked separately. The replay writes a shadow generation and performs no authority switch until reconciliation passes.

The trial injects a worker restart halfway through. Recovery resumes from its own checkpoint without resetting the live consumer group. It also introduces an old schema and a duplicated source segment. The reader resolves the supported schema; the result-version protocol suppresses duplicate effects; unsupported semantics stop the replay generation rather than corrupting live output.

The evidence packet reproduces the arithmetic

Run the dependency-free packet:

cd examples/performance-engineering-system-design-handbook/part-08/burst-tolerant-event-pipeline
node analyze.mjs
node verify.mjs

It calculates the 160,000-record/s burst excess, 96 million-record peak backlog, 61.44 GB payload, 230.4 GB replicated modeled storage, 345.6 GB reserved increment, 80,000-record/s drain margin, twenty-minute drain, 369.23-second age approximation, partition-key alternatives, 7.8 million-record checkpoint exposure, and all four fault/replay cases.

The packet does not simulate a broker, checkpoint algorithm, filesystem, network, or sink. Its job is to make units and internal claims executable. Production acceptance still requires open-loop load, real payload distributions, broker failure, state restore, sink ambiguity, schema corpus replay, and source-to-materialization reconciliation in the target environment.

Failure modes that remain after the design

None of these is repaired by increasing a topic partition count alone. More partitions can improve parallelism, but they increase metadata, connections, state shards, checkpoint fan-out, and rebalance work. The right count follows the ordering boundary, hot-key distribution, per-partition service limit, recovery behavior, and expected growth.

Design exercise: the burst becomes a daily replay

Northstar signs a contract requiring a rolling twenty-four-hour correction to finish within four hours whenever a reference-data bug is found. Steady traffic grows to 220,000 records/s, while the tested live boundary remains 260,000/s. The sink can accept at most 340,000 useful mutations/s across all workload classes. Historical records use the same average payload but produce 1.4 sink mutations per input after enrichment.

Redesign the live, backfill, and recovery plan. Quantify the replay input rate, sink demand, live drain margin, and minimum retained data. State whether a four-hour objective is feasible without changing code, capacity, or output semantics. Include one response if the correction is urgent during a launch burst.

Answer guide

Twenty-four hours at 220,000 records/s contains 19.008 billion records. Finishing in four hours requires about 1.32 million replay records/s:

19.008 billion records ÷ 14,400 s
  = 1.32 million replay records/s

At 1.4 sink mutations per replayed record, replay alone demands about 1.848 million sink mutations/s, far above the 340,000/s total sink boundary. The objective is impossible under the stated mechanism and capacity. Borrowing the live pool cannot solve a downstream shortfall and would reduce live freshness.

A defensible redesign changes at least one premise. It might compute compacted corrections by affected key rather than replaying every record, increase isolated transformation and sink capacity, materialize an intermediate state that reduces replay work, relax the four-hour objective, or promote a shadow dataset through a bulk-load path with separately tested semantics. The team must quantify the new boundary rather than label it “batch.”

Live drain margin has fallen to 40,000 records/s before replay. During the original 420,000/s burst, backlog grows at 160,000/s and later drains twice as slowly as the original case. The launch and urgent correction cannot both assume the same reserve. A safe response can pause backfill, preserve the immutable input and correction generation, keep live admission and checkpointing inside the burst envelope, then resume correction when event age and sink reserve recover. If contract language forbids pausing, the architecture needs dedicated correction capacity across compute, storage, network, and sink—not only more consumers.

Retention must exceed detection time, correction preparation, four-hour execution, retries, and an evidence margin. Twenty-four hours of source data is insufficient if the bug can be detected near the end of that window and the replay then needs four more hours. The plan declares a minimum readable horizon and verifies it per hot partition under byte limits.

Field review card

When reviewing a burst-tolerant event pipeline:

• Name logical records, attempts, bytes, partitions, state, effects, and freshness separately.
• Describe the burst rate, duration, recurrence, payload distribution, and correlation.
• Put ordering at the narrowest correct key; test actual partition heat.
• Treat partition-function changes as fenced state migrations.
• Distinguish broker acceptance from correct materialization.
• Calculate backlog growth, physical bytes, safety reserve, drain rate, and drain time.
• Track oldest event age and per-partition age, not only record lag.
• Couple checkpoint state with source positions and test restore.
• Scope “exactly once” to a named state or effect boundary.
• Give replay identities and deduplication horizons that cover the intended recovery.
• Retain writer schemas and test semantic evolution with a replay corpus.
• Quarantine poison data with provenance, completeness semantics, and a return path.
• Isolate live, backfill, and recovery at every shared constrained dependency.
• Size recovery by restore rate and objective, not retained bytes alone.
• Drive fault tests from fixed arrivals and reconcile logical records to visible effects.
• Test broker loss, slow sink, poison data, and full replay while live work continues.
• Record what the arithmetic cannot establish about a real runtime or storage system.

Northstar succeeds when operators can say not only how quickly it accepts work, but how old the oldest correct result can become, how much work can wait, how the queue drains, which state is replayed, and what recovery is allowed to consume. The next case moves from queued time to a much shorter clock: a mobile search request has 180 milliseconds, and every additional retrieval or ranking stage must justify the portion it spends.