Serialization, Compression, and Protocol Shape

Trace one 96 KiB Pulsepipe payload. A producer materializes fields, validates them, encodes a row message, allocates an output buffer, optionally compresses it to a modeled 24 KiB, wraps it in a length-delimited frame, copies or gathers the frame into transport buffers, and sends it. The receiver identifies the frame, bounds its declared length, decompresses, decodes, validates again at its trust boundary, allocates an application object, and reads 12 KiB of fields. Seven-eighths of the uncompressed bytes were never useful to that consumer.

At 100 Mbit/s, modeled compression reduces transfer time enough to repay codec work. At 10 Gbit/s, the same one-core codec path costs more latency than the bytes it removes. At 1 Gbit/s it improves per-message latency modestly, raises network capacity fourfold, then makes one compression core the modeled throughput bottleneck. “Compression is faster” and “compression is slower” are both incomplete. The answer depends on payload, entropy, hardware, concurrency, link, access pattern, and the boundary being timed.

Representation is therefore part of architecture. It decides how much work crosses every boundary, which consumers can evolve independently, when partial data becomes useful, what must be allocated or copied, and how corruption or hostile input is contained.

Follow the payload cost stack, not the file size

For one useful operation, account for these stages:

source values
  -> projection / materialization
  -> encode + validate
  -> optional compress
  -> frame + checksum / authenticate
  -> copy / gather + queue
  -> transport or storage
  -> deframe + authenticate / verify
  -> optional decompress
  -> decode + validate
  -> application representation
  -> useful fields consumed

The end-to-end service demand and latency are related but not identical. If stages overlap across messages, throughput approaches the constraining stage while one message still traverses the critical-path stages. Queueing appears when arrival exceeds any stage’s sustainable service.

For a simple sequential model:

[ L_{payload} = L_{materialize} + L_{encode} + L_{compress} + L_{queue} + L_{transport}

• L_{decompress} + L_{decode} + L_{validate} + L_{consume}. ]

Define (U) as bytes before compression, (C) as compressed bytes, (B) as link bits/s, (R_c) and (R_d) as compression and decompression bytes/s per allocated core, and (L_f) as fixed transport latency. A teaching bound is:

[ L_{raw} = L_f + \frac{8U}{B} ]

[ L_{compressed} = L_f + \frac{U}{R_c} + \frac{8C}{B} + \frac{U}{R_d}. ]

This omits queueing, memory bandwidth, framing, copies, and overlap, so it narrows a decision rather than proves one. The chapter fixture uses (U=98{,}304) bytes, (C=24{,}576) bytes, (R_c=400) MB/s, and (R_d=800) MB/s as modeled inputs:

The fixed 2 ms term appears in both designs. Removing bytes cannot remove distance, handshake, scheduling, or unrelated service. Conversely, a throughput decision must compare stage capacities. At 1 Gbit/s, raw network capacity is about 1,272 payloads/s. Four-to-one compression raises the link limit to about 5,086 payloads/s, but a single 400 MB/s compression core handles about 4,069 payloads/s. The bottleneck crosses from network to codec CPU even while total capacity increases about 3.2×.

Measure every stage in CPU-time, wall-time, allocations, bytes read/written, and queue wait. A codec may appear cheap in CPU profiles because threads block on memory allocation, page faults, or downstream queue credits. A zero-allocation decoder can still touch every input byte and build expensive indexes. A compact payload can increase total CPU enough to reduce goodput.

The useful denominator matters. Bytes per request rewards omitting required fields; CPU per decoded object rewards decoding unused fields. Prefer cost per correct, useful field-set or business operation, with rejected, corrupt, incompatible, and partial messages reported separately.

Format labels describe several independent choices

“JSON versus binary” compresses multiple design dimensions into one slogan. Separate them:

Text versus binary describes lexical representation. Text often improves direct inspection, generic tooling, and public interoperability. Binary can encode numeric and structural information more compactly and avoid number-to-text conversion. Neither guarantees speed: a binary format with reflection, copying, and allocation can lose to a specialized text parser; a text format can compress well because field names repeat.

Row versus columnar describes adjacency and access. A row message keeps fields for one event together, favoring per-event processing and mutation. Columnar batches keep values of one field together, favoring projection, scans, compression, and vectorized operations. Transposing individual events into columns has fill, buffer, null, offset, and latency costs. Chapter 20’s batch boundary and Chapter 17’s locality model determine whether those costs amortize.

Tagged versus positional describes how a decoder associates data with fields. Tagged formats carry field identifiers or types, supporting omission and some evolution at byte cost. Positional formats can be compact and fast but require an exact schema/version agreement or external metadata. A schema ID does not solve schema distribution, compatibility, retention, authorization, or recovery when the registry is unavailable.

Self-describing versus schema-driven describes how much meaning travels with the payload. Self-description improves generic handling but may repeat names and types. Schema-driven formats move meaning into an agreed definition. They can be smaller, yet a receiver without the right definition cannot interpret bytes safely. Some formats combine a schema message with many data batches.

Whole-message versus streaming describes when work and values become available. A whole-message API can simplify ownership but requires enough memory and time to receive and validate the complete object. Incremental parsing can emit bounded records or fields earlier, but the application must not commit unsafe effects before message-level integrity and semantic validation are satisfied.

Fixed-width versus variable-length, canonical versus non-canonical, and compressed versus uncompressed are further axes. Choose a combination from workload and invariants. Do not infer columnar layout from “binary,” random access from “compressed,” or canonical bytes from “schema-driven.”

Encoding and decoding spend more than instructions

The encode path may traverse an application object graph, calculate sizes, allocate a contiguous buffer, copy strings and nested values, convert numbers, normalize text, validate ranges, write tags/lengths, and grow buffers after underestimated sizes. The decoder reverses only part of that work; it also handles bounds, unknown fields, duplicates, defaults, presence, malformed input, and ownership.

Account for:

• passes: size-first plus encode, validation plus encode, or parse plus materialize;
• allocations: one arena/buffer, per-field objects, temporary strings, decompression destination, and growth copies;
• copies: application to codec buffer, codec to frame, kernel or runtime buffers, decompression, and object materialization;
• memory traffic: input read, output write, metadata, indexes, and cache-line disruption;
• branching: tags, variable lengths, escaping, nulls, rare fields, and error paths;
• validation: syntax, bounds, schema constraints, authorization-relevant semantics, and cross-field invariants; and
• lifetime: whether decoded views may borrow the input buffer and for how long.

“Zero copy” must name the copy avoided. An Arrow IPC receiver can reconstruct supported arrays from buffers using offsets rather than copying each value, but bytes still arrived, metadata was parsed, buffers must remain alive, alignment and trust must be handled, and application conversion may copy. TLS, compression, kernel boundaries, language ownership, or a mismatched layout can add other copies. State exactly which buffer ownership and representation make borrowed access safe.

Allocation strategy can dominate tails. Per-message object graphs pressure an allocator and managed heap; retaining a small slice of a large input buffer can pin the whole buffer. Arenas reduce per-field allocation but can inflate lifetime when one long-lived field keeps the arena. Pooling reduces allocation only if buffers are bounded, cleared safely, and not retained across slow consumers. Measure retained bytes and lifetime, not allocation count alone.

Validation is not optional codec overhead. A parser that accepts a length prefix before checking a maximum can reserve attacker-controlled memory. A decoder that creates objects before checking nesting depth can exhaust stack or heap. Put hard limits on frame length, decompressed size, compression ratio, nesting, field count, string length, collection count, recursion, dictionary size, and total work. Return a machine-actionable rejection without retry amplification.

Compatibility is a mixed-version state machine

Schema evolution is not “old readers ignore new fields.” Write the producer/consumer matrix across deployed versions and transformations:

Protocol Buffers uses field numbers and wire types rather than serialized field names. Its official guidance permits certain additions and removals when identifiers are reserved and types remain compatible. The encoding is not canonical, and unknown-field behavior depends on the runtime and transformations. Those facts illustrate the distinction: wire compatibility is a property of specific changes and toolchains, not permission to change meaning invisibly.

Test at least producer (N) to consumers (N-1), (N), and (N+1) where rolling policy requires it; and old producer to new consumer. Include gateways that decode and re-encode, storage replay, dead-letter inspection, cached payloads, signed bytes, dictionaries, and schema-registry outage. Preserve golden messages for every supported version and assert semantics after round trips, not only parse success.

Defaults are API decisions. If an old message omits priority, does the new consumer infer ordinary, reject ambiguity, or derive from another field? If zero means both “not supplied” and a valid value, a patch can erase intent. Use explicit presence or versioned semantics where the distinction matters.

Evolution also affects performance. Adding a commonly populated field raises payload size, compression state, allocation, cache residency, egress, and downstream storage. A new optional field can be cheap for old readers that skip it and expensive for a gateway that materializes every field. Add schema size and decode-cost budgets to compatibility review.

Compression trades bytes for CPU, memory, and access granularity

Compression ratio (U/C) is only one point on a frontier. Measure compression and decompression throughput, setup, block size, working memory, latency distribution, output variance, and effect on downstream access. A codec level that produces 5% fewer bytes while halving throughput may be wrong for an interactive path and right for retained archival data.

Small payloads pay headers, dictionaries, checksums, and codec setup over few bytes. Combining them into a block can improve ratio and throughput, but Chapter 20’s formation wait, fairness, memory, and partial-failure obligations return. One large block delays first output, increases retry unit, and makes random access decompress unrelated data. Many small independent blocks improve parallelism and seekability at ratio and metadata cost.

For the modeled 96 KiB payload, the codec service is 0.246 ms compress and 0.123 ms decompress. Those means are insufficient for a deadline. Measure p50/p95/p99 by input-size and entropy class, plus queue wait at allocated codec workers. At 4,000 payloads/s, a one-core compression stage is near its modeled 4,069/s service limit; variability will create queueing before the arithmetic maximum.

The compression frontier moves with link rate:

saved wire time = 8(U - C) / B
codec time      = U/Rc + U/Rd

compress for latency only when saved wire time exceeds codec time
after adding queue, copy, memory, and block-formation costs.

At 10 Gbit/s in the fixture, saved wire time is about 0.059 ms while codec time is about 0.369 ms. Compression loses about 0.310 ms before queueing. At 100 Mbit/s it saves about 5.898 ms on wire and wins after codec work. Recalculate for the actual path: a nominal 10 Gbit/s interface can still traverse a throttled tunnel or paid egress link, while CPU may have vectorized/multicore capacity or may share a saturated tenant quota.

Compressibility depends on entropy and representation. Repeated text field names may compress well; already compressed images and encrypted bytes usually do not. High-cardinality identifiers and random values reduce reuse. Sorting or columnar grouping can improve locality and ratio but changes formation, latency, and ordering. Never use a single lorem-ipsum or zero-filled corpus to rank production codecs.

Compressed input is an amplification boundary. Bound declared and actual decompressed size, nesting, window/dictionary memory, and CPU. Reject or quarantine malformed frames without allocating from untrusted lengths. A checksum can detect accidental corruption; it does not authenticate an adversarial payload.

Block size and random access must match the consumer

A consumer that reads an entire event benefits from row framing. A query that projects timestamp, tenant, and status across a million events should not decode every message body merely to discard most fields. Columnar record batches can store adjacent values and apply per-column encoding/compression; they introduce batch materialization, null/offset metadata, and more expensive point mutation.

Apache Arrow’s current columnar specification defines contiguous physical layouts, record batches, schema/dictionary messages, a stream form, and a file footer for record-batch random access. Those are precise representation choices. They do not imply that arbitrary compressed content is randomly accessible or that every consumer can borrow buffers without conversion.

For compressed archives or analytical partitions, create independent blocks with an index mapping logical ranges to byte offsets. The block should be large enough to amortize metadata and find repetition, and small enough to bound decompression amplification, first-row latency, memory, and repair. Evaluate point lookup, short range, scan, and recovery separately.

Index and block metadata become correctness state. A checksum failure may invalidate one block, a partition, or the entire object depending on framing. Recovery needs source lineage or replicas. If a writer publishes the index before all blocks are durable, readers can follow offsets into incomplete data. Define atomic publication and versioning.

Framing makes a byte stream into bounded messages

Transport preserves bytes according to its own contract; it may not preserve application message boundaries. A decoder needs framing such as fixed-size records, delimiter with escaping, length prefix, type-plus-length envelope, or self-delimiting structure. HTTP/1.1, for example, has explicit message-body framing rules; method semantics alone do not determine arbitrary application message boundaries.

A safe incremental timeline is:

receive fixed header
  -> validate magic/version/type
  -> parse bounded header length and payload length
  -> reserve only within per-connection and global credits
  -> stream payload blocks; update checksum/authentication state
  -> decode complete bounded fields or records
  -> validate required semantics
  -> publish effects allowed at this validation boundary
  -> release buffers / return credits

Length-prefixed framing is efficient only after validating the length against protocol and resource limits. A delimiter requires unambiguous escaping and a cap when the delimiter never arrives. Incremental parsers must distinguish “need more bytes” from invalid input and end-of-stream. A zero-byte read, partial multibyte code point, truncated varint, or frame ending mid-field needs a defined outcome.

Streaming reduces time to first useful record and peak buffer size when the application can consume bounded units. It does not automatically reduce total work. Backpressure must propagate from the consumer to decoder, decompressor, socket, and sender. Otherwise the parser eagerly materializes an unbounded stream and recreates the queue Chapter 19 warned about.

Early effects require care. If a message-wide signature or checksum is verified only at the end, acting on early records can commit effects from a frame later rejected. Options include per-record authentication, chunk-level verified frames, staging reversible work, or waiting for whole-message verification. The protocol’s integrity unit and the business effect unit must align.

Canonicalization, integrity, and confidentiality are separate jobs

Serialization can produce multiple byte strings for the same semantic value: object member order, map iteration, insignificant whitespace, alternate numeric forms, Unicode normalization, default omission, and unknown-field ordering can vary. If a signature, hash, cache key, deduplication ID, or content address covers bytes, define a canonical representation or sign a separately canonical semantic structure.

Protocol Buffers documentation explicitly warns that serialization is not canonical and byte output can change across builds or implementations. Comparing serialized bytes is therefore not a general semantic-equality test. Deterministic output in one library may stabilize a build without becoming a cross-version canonical protocol.

A checksum detects accidental changes under its threat model. A message authentication code or authenticated-encryption tag proves integrity and authenticity to holders of the key. A digital signature adds a different trust and verification model. Encryption hides content but can remove repetition that compression needs. A common pipeline compresses plaintext before authenticated encryption, while avoiding secret-dependent compression contexts that expose information through length. The correct order depends on protocol threat analysis; never disable validation or authentication to make a codec benchmark win.

Integrity work has CPU, buffering, and failure costs. Whole-object signatures can delay streaming effects. Per-chunk authentication increases metadata and key/nonce bookkeeping. Retries need nonce and replay rules. Key rotation and mixed-version verification belong in compatibility tests. Report rejected-authentication work separately so hostile traffic cannot consume unlimited decompression or parsing CPU before authentication where the protocol could verify earlier.

Project before encoding and compression

The cheapest byte is often the field not selected. Mercury’s public API might store 96 KiB of product and diagnostic state while a list view needs 12 KiB. Projecting to the list contract reduces encode, validation, allocation, transfer, decode, cache, and client work. Four-to-one compression of all 96 KiB still sends 24 KiB—twice the 12 KiB useful projection before that projection’s own encoding or compression.

Projection must preserve contract semantics. A caller needs a stable way to request or negotiate fields, and the server must bound arbitrary query shape. Omitting a field can mean “not requested,” “unknown,” “not authorized,” or “default”; encode that distinction where it matters. Do not leak sensitive field existence through size or timing without considering the threat model.

Overfetch can move the bottleneck across layers: database reads, object materialization, serializer CPU, network, client parse, memory, and battery. Measure server and client boundaries. A backend that sends a compact payload after fetching and constructing a huge object has optimized only the wire.

Underfetch creates chatty protocols and round trips. A projection that forces ten follow-up calls can cost more than one bounded representation. Chapter 29 develops the API granularity decision; here, compare total useful fields, calls, bytes, and critical-path time for representative journeys.

Dictionaries are shared state with a lifecycle

Content-specific dictionaries replace repeated tokens or byte sequences with compact references. They can materially help small related records that do not contain enough local repetition. Columnar dictionaries can encode repeated categories as integer indexes; compression dictionaries can prime a codec from a representative corpus.

The dictionary ID in a payload is not the dictionary. Producer and consumer need the exact bytes or logical entries, version, distribution, retention, authorization, and failure behavior. RFC 8878’s Zstandard format includes dictionary identifiers but leaves dictionary acquisition out of band—a useful reminder that the protocol still owns delivery.

Design:

• immutable dictionary versions and collision-resistant identity where required;
• publication before first dependent payload;
• retention at least as long as replayed/stored payloads;
• bounded cache/admission for dictionaries themselves;
• fallback when a dictionary is missing, corrupt, or unauthorized;
• tenant and confidentiality boundaries for training data;
• rollout with old/new readers and writers; and
• metrics for ratio, lookup misses, memory, and decode failures by dictionary version.

A global dictionary can leak cross-tenant vocabulary or let one tenant poison effectiveness. A per-tenant dictionary improves isolation but multiplies state and cold starts. A rapidly changing dictionary can erase compression gains through distribution and cache churn. Train on representative data, validate on a held-out time window, and include drift/retraining cost.

Benchmark the distributions that drive the decision

A format benchmark needs a claim. Examples: “projection plus binary row encoding keeps Pulsepipe ingress below 0.5 Gbit/s and one encode core at 200,000 events/s,” or “compressing the 96 KiB response improves p99 on the constrained link without making codec CPU the saturation point.” “Codec A is fastest” lacks a system boundary and useful unit.

Build a corpus stratified by:

• payload size distribution, including empty, median, p95, p99, and configured maximum;
• entropy, repetition, cardinality, string length, numeric range, null density, and nesting;
• field presence, unknown fields, mixed schema versions, and malformed inputs;
• row/column batch size, block size, dictionary state, and projection selectivity;
• compressible text, already compressed media, encrypted/random fields, and adversarial expansion;
• tenant/operation mix and hot/cold phases; and
• result classes: accepted, rejected, partial, incompatible, corrupt, and unauthenticated.

Record wall and CPU distributions for materialize, encode, compress, frame, transfer, deframe, decompress, decode, validate, and consume. Capture allocations, retained memory, copies/bytes touched where observable, output bytes, queue age, and useful goodput. Pin runtime, library, codec options, hardware, thread count, affinity policy, and transport. Warm code and dictionaries appropriately, but also run cold and mixed-version states.

Verify semantic round trips against a canonical object comparison, required presence/default behavior, unknown-field expectations, and authorization/integrity checks. Fuzz parsers and size/depth bounds. Preserve corpus generators, sampled protected fixtures, raw observations, and exact commands. Compare at equal correctness and durability boundaries.

Microbenchmarks isolate stage demand. End-to-end tests determine whether reducing that stage changes the objective. Use both. A decoder benchmark on an in-memory buffer cannot claim network p99; a full service test cannot explain whether improvements came from payload, queueing, or unrelated cache warmth without stage evidence.

Applied choice: telemetry and public APIs need different shapes

Pulsepipe ingests 200,000 telemetry events/s. The deterministic worksheet models 960-byte JSON events, 300-byte schema-driven binary row events, and 150-byte compressed binary events. Their payload rates are 1.536, 0.480, and 0.240 Gbit/s respectively. Modeled per-event stages consume 0.24 encode cores, 0.60 compression cores, 0.30 decompression cores, and 0.20 decode cores at that rate.

For this workload, choose a length-delimited, schema-driven binary row envelope at ingress:

• one event is the routing, validation, retry, and ownership unit;
• field identifiers support compatible optional additions under tested rules;
• a schema/version ID is carried, with definitions distributed and retained before dependent producers deploy;
• frame, decompressed-size, field, and nesting limits are enforced before allocation;
• compression is enabled by measured payload class or bounded microbatch, not universally;
• events remain row-oriented for per-event validation and partition routing, then convert into columnar batches only at the analytical boundary; and
• raw event identity and source schema survive conversion for replay and diagnosis.

The 0.60 modeled compression cores look affordable, but that is not proof. Validate entropy, small-message setup, batch formation delay, p99 CPU queueing, and cold dictionaries. If the network has headroom and ingress CPU is saturated, the uncompressed 300-byte form may deliver more goodput.

Mercury’s public API has different constraints: diverse clients, inspectability, long compatibility horizons, selective fields, and response sizes ranging from small errors to large product documents. A UTF-8 JSON response governed by a versioned schema can be reasonable when:

• projection removes unrequested fields before object materialization where possible;
• numeric, Unicode, duplicate-member, unknown-field, and presence semantics are constrained beyond the base grammar;
• HTTP content negotiation and compression vary safely by representation and cache key;
• large eligible responses use compression only after size/entropy and client capability checks;
• streaming endpoints use explicit record framing rather than concatenated ambiguous values; and
• clients receive bounded errors for unsupported versions or fields.

JSON is not chosen because text is universally slower or safer. It is chosen because interoperability and public evolution value outweigh extra bytes for this boundary after projection. Pulsepipe’s controlled high-rate path values compact tagged rows and predictable schema deployment. The two decisions use the same cost model and reach different formats.

Did compression move the bottleneck?

At 1 Gbit/s, the modeled raw 96 KiB path is network-limited to about 1,272 payloads/s. Four-to-one compression raises the network limit to about 5,086/s, but one compression core supplies about 4,069/s. Compression improved capacity and moved the constraint to CPU.

That is acceptable only if CPU headroom, queueing, and economics fit. Options include allocating more independent codec workers, using a faster/lower-ratio setting, projecting before compression, reserving compression for larger/highly compressible payloads, or leaving traffic uncompressed on faster local links. More codec workers can make memory bandwidth or downstream decode the next constraint.

Prove the crossing with offered-load sweeps. Before compression, link utilization and transport queue age should rise near the knee while codec CPU is absent. After compression, link bytes fall, compression service/queue grows, and the saturation knee approaches codec capacity. If origin object construction or receiver validation remains dominant, neither signature will explain end-to-end goodput.

Format selection matrix

The decision rule is end-to-end: optimize total cost per useful field, not encoded size or codec speed in isolation.

Field checklist

• Which useful fields and business operation define the denominator?
• Where are projection, materialization, encode, compress, frame, copy, queue, transfer, decode, validation, and consumption costs paid?
• Are text/binary, row/columnar, tagged/positional, schema-driven/self-describing, and streaming/block choices evaluated independently?
• What payload size, entropy, cardinality, null, nesting, and version distributions must the design handle?
• Which additions, removals, defaults, unknown values, and transformations are compatible semantically—not only parseable?
• What hard limits bound frame, decompressed size, depth, fields, dictionaries, memory, and CPU?
• At what link and offered load does compression repay codec service, and which resource becomes next?
• How do block size and framing affect first-value latency, random access, retry, repair, and memory?
• Which copies are actually avoided, and which buffer owns borrowed data for how long?
• Does canonicalization match hashing/signature/deduplication semantics across versions and implementations?
• How are dictionaries published, retained, isolated, rolled back, and recovered?
• Do fixtures and experiments preserve raw data, correctness, mixed-version, malformed-input, and transfer-limit evidence?

Design exercise: two protocols, one review standard

Produce two design records.

For Pulsepipe telemetry, define event schema, framing, maximums, binary row encoding, optional compression threshold, dictionary policy, schema rollout, unknown-field behavior, partition-routing fields, validation boundary, retry identity, row-to-column conversion, and replay compatibility. Calculate bytes/s and CPU cores at 200,000 events/s using the supplied worksheet, then replace at least three modeled inputs with ranges you would measure.

For Mercury’s public API, define text schema constraints, projection, pagination/bounds, content negotiation, compression eligibility, cache-key variation, streaming record framing, canonical signature needs, mixed-client compatibility, and error behavior. Compare one large projected response against several smaller calls on useful bytes and critical path.

A valid pair may choose different formats. The review standard remains common: useful work, end-to-end costs, correctness, evolution, overload, recovery, evidence, and transfer limits.

Diagnostic exercise: compression helped, then the service collapsed

A deployment reduces payload bytes by 75%. Link utilization falls from 88% to 31%. Goodput rises from 1,250 to 3,700 payloads/s, then p99 climbs sharply. One compression worker reaches full service occupancy; its queue age grows; application CPU averages 62% across eight cores; decompression is 0.123 ms/payload; failures trigger whole-block retries.

Explain why host-average CPU does not refute a codec bottleneck. Estimate the one-core compression capacity from the fixture, identify the saturation knee, and propose discriminating changes: more bounded workers, projection, lower codec work, smaller retry units, or conditional compression. Include memory bandwidth, block formation, receiver, and correctness evidence before selecting one.

Review questions and durable conclusions

1. For 96 KiB compressed to 24 KiB over 1 Gbit/s, calculate raw and compressed wire time. Which fixed and codec terms determine the latency result?
2. Why can compression improve capacity while moving the bottleneck to CPU? Name the telemetry signature before and after.
3. Give one workload where row representation is favored and one where columnar batches are favored. Include conversion and latency costs.
4. Why does a schema ID not by itself provide compatibility or availability?
5. Describe a change that remains wire-parseable but breaks semantic compatibility.
6. Which bounds must an incremental decoder check before allocating from a frame or compressed payload?
7. Why can deterministic serialization still fail to be a canonical cross-version signing format?
8. When can projection produce a larger end-to-end gain than a higher-ratio codec?

The durable model is a stage pipeline around useful fields. Format families are combinations of independent representation choices. Encoding cost includes traversal, allocation, copies, memory traffic, branching, validation, and lifetime. Compatibility is a mixed-version semantic state machine. Compression is justified only on the end-to-end frontier of bytes, codec service, latency, memory, and access granularity. Framing and incremental parsing require hard resource and effect boundaries. Canonicalization, checksums, authentication, encryption, and signatures solve different problems. Projection and representative evidence usually matter more than format fashion.

Evidence and transfer limits

• RFC 8259: JSON defines JSON grammar and interoperability considerations. An application still must constrain schema, duplicates, number ranges, Unicode, sizes, and semantics.
• Protocol Buffers encoding documents tags, field numbers, wire types, and encoding properties; Proto serialization is not canonical scopes deterministic versus canonical output. Library/version behavior and application semantics require their own tests.
• Protocol Buffers proto3 guide gives current message-update guidance, including reserved fields and unknown fields. It is one schema-driven format’s contract, not a universal evolution rule.
• Apache Arrow columnar format specifies physical layouts, record batches, dictionaries, streaming, file framing, and supported buffer reconstruction. Actual conversion, instruction selection, and workload performance are implementation-dependent.
• RFC 8878: Zstandard format specifies frames, blocks, checksums, and dictionary identifiers for one lossless format, while leaving dictionary acquisition out of band. No codec rate or ratio in this chapter is attributed to it.
• RFC 9112: HTTP/1.1 is a current primary example of explicit message syntax and body framing. It does not define every application record boundary.
• Every numerical payload, codec, link, and telemetry result is deterministic modeled evidence from examples/performance-engineering-system-design-handbook/part-03/protocol-shape/. The fixture assumes fixed sizes, ratio, codec throughput, and transport latency with no queueing, copies, memory contention, or overlap. It is not a benchmark of JSON, Protocol Buffers, Arrow, Zstandard, HTTP, hardware, or production traffic.

Once representation costs are explicit, work placement can account for bytes, codec CPU, data locality, cache warmth, and queue state rather than distributing requests by count alone. That is the next mechanism.