Media Systems

Eight missing seconds

At 20:03, eight seconds disappear from a live concert. The broadcaster is still connected, but one ingest server has stopped producing segments. Some viewers freeze; others skip forward; a third group receives an advanced manifest and keeps asking the CDN for an object that was never written.

The event has 800,000 concurrent viewers. The product promises playback within roughly six seconds of the stage, a replay twenty minutes after the concert, podcast audio the next morning, and still images for social posts. The creator also expects to upload a higher-quality master after the show without breaking old links.

“Put the stream behind a CDN” does not answer the prompt. A CDN can distribute an object that exists; it cannot decide whether an incomplete segment is safe to publish, recover source material that was never preserved, or keep a replay manifest consistent while transcoders finish at different times. The design needs an account of what became authoritative at each step.

This event gives the case its path. We will keep the live stream moving, turn its archive into on-demand video, extract a podcast, generate images, and then accept the creator’s replacement master. The products differ, but the source, derivative, publication, and delivery boundaries remain visible in all of them.

Make the promises measurable

Before drawing components, ask what the user is allowed to notice. “Low latency” is not yet a requirement. Ask how much glass-to-glass delay the interaction permits, how much rebuffering viewers will tolerate, and whether a short gap is better than freezing the player. For the replay, ask when the first watchable rendition must appear and whether higher qualities may arrive later. For the podcast and images, ask whether old URLs must continue to serve the old bytes after a correction or should move to a new version.

Then establish the source and authority boundaries:

• Who may broadcast, upload, replace, publish, unpublish, or delete the media?
• Is the source a file, a continuous stream, an import, or output from another service?
• Which derivatives are required, and which may be delayed or dropped under load?
• Must private playback be reauthorized, and how quickly must a takedown reach manifests, caches, downloads, and feeds?
• How long must originals, replay segments, and rarely requested variants be retained?

Scale the work in the units that consume resources. Concurrent viewers and delivered bitrate drive streaming egress. Source duration multiplied by the number and complexity of renditions drives encoder work. Segment duration and latency target drive manifest churn and player buffer margin. Original size, derivative count, retention, and cache hit ratio drive storage and origin load.

For example, 800,000 viewers averaging 3 Mb/s imply about 2.4 Tb/s of edge delivery during the event. That number argues for CDN capacity and origin shielding; it does not imply that the application servers need to carry 2.4 Tb/s. A two-hour source encoded into six video renditions represents roughly twelve rendition-hours before accounting for codec complexity. That estimate belongs in the processing-capacity discussion, not in a generic request-rate total.

In an interview, a useful framing is:

I will separate ingest, durable source, processing, publication, and delivery.
The live path is controlled by continuity and latency; the replay path by
processing delay and version consistency; both are constrained by egress cost
and the need to recover from a preserved source.

Give every asset a source and a version

The control plane stores small, queryable records. Object storage holds the large bytes. A media record might contain:

Media
  id, owner_id, policy_state
  source_version, source_kind, source_object, source_hash
  state, created_at, retention_class

Derivative
  media_id, source_version, recipe_version
  kind, codec, dimensions, bitrate, object_key, checksum
  state, duration, created_at

Publication
  media_id, publication_version
  source_version, manifest_object, status, published_at

source_version changes when the creator deliberately replaces the canonical input. recipe_version changes when encoding, watermark, thumbnail, caption, or moderation rules change. A derivative is therefore reproducible and addressable: “720p produced from source 3 with recipe 7” is not confused with an older 720p file left in a cache.

The states should describe product truth rather than worker activity. An uploaded file may move through receiving, source_ready, processing, preview_ready, published, blocked, and failed. Individual derivatives have their own attempts and results. A single failed caption job should not make a playable video indistinguishable from a source that never completed.

A compact API sketch makes the boundary concrete:

POST /media/uploads
  -> upload_id, media_id, resumable_part_instructions, expires_at

POST /media/{media_id}/uploads/{upload_id}/complete
  {parts, expected_hash, idempotency_key}
  -> source_version, state

POST /broadcasts
  -> broadcast_id, short_lived_stream_credential, ingest_endpoints

GET /media/{media_id}/playback
  -> authorized_manifest_url, publication_version, expires_at

Completing an upload is idempotent. The server verifies the expected parts, size, checksum, owner, quota, and content signature before promoting the assembled object to a source version. A live broadcast performs the same logical promotion repeatedly: accepted segments enter a durable rolling archive before or as they become eligible for publication.

Follow one live segment

The broadcaster connects to a nearby ingest endpoint with a short-lived credential bound to the event. The ingest service validates the stream, tracks source timestamps and a monotonically increasing ingest sequence, and monitors bitrate, keyframe cadence, and reconnects. It sends the media to a packager and, independently, to durable archive storage. Those two outputs serve different promises: one keeps viewers near the present; the other makes replay and repair possible.

The packager creates short immutable segments and a live manifest that names only segments known to exist. It writes a segment under a versioned key, verifies the write, and only then advances the manifest. Publishing the manifest first creates exactly the 20:03 failure: clients are told to fetch an object the origin cannot serve.

The CDN caches manifests briefly and segments much longer. An origin shield coalesces regional misses so that a popular new segment does not become hundreds of thousands of simultaneous reads against object storage. The player fetches the manifest, chooses an initial rendition conservatively, builds a small buffer, and adapts quality as measured throughput and buffer health change.

Latency is accumulated, not bestowed by a protocol name:

capture + encoder + network to ingest + package interval
        + manifest availability + CDN fetch + player buffer

Reducing segment or chunk duration can reduce part of that delay. It also increases request and manifest frequency, leaves less buffer to absorb jitter, and makes ingest or packaging stalls visible sooner. A live auction or watch party may accept that fragility. A large concert whose audience mostly watches may prefer several extra seconds of delay to repeated playback interruption.

Repair the eight-second gap

At 20:03, health monitoring sees that an ingest connection remains open but the next sequence number has not arrived by its deadline. The response depends on where the bytes still exist.

If the broadcaster is sending redundantly to a second ingest path, the control plane can fence the unhealthy primary and let the packager continue from the standby at the next safe timestamp. It must deduplicate overlapping segments and avoid publishing two objects for one sequence. If only one path exists, the service cannot reconstruct arbitrary missing video. It can hold the live manifest briefly, mark a discontinuity and resume, or terminate the stream. Which choice is right follows the promised latency and the player’s tested behavior.

The CDN should never be asked to hide missing origin state with long error caching. A player that receives a segment miss retries within a bounded policy, refreshes the manifest, switches rendition when the gap is isolated to one variant, and eventually skips or reports a discontinuity. Endless retries turn eight missing seconds into a frozen session.

Recovery also produces evidence. Keep the ingest endpoint, source timestamps, sequence range, archive result, packaging attempt, manifest version, origin status, CDN response class, and player outcome. “CPU was normal” does not explain whether the broadcaster stopped producing bytes, the packager failed to commit them, the manifest advanced too early, or the edge served stale state.

After the live path is stable, ask the uncomfortable follow-up: what if the primary and standby both derived from one failed encoder at the venue? Regional redundancy cannot repair a shared source failure. The system can make server-side loss rare and recoverable; it cannot promise to recreate media it never received.

Turn the archive into a replay

When the concert ends, the rolling archive is closed as a canonical source. Workers probe its container and streams, validate duration and timestamps, and enqueue renditions, audio, captions, thumbnails, and packaging. A job carries media_id, source_version, recipe_version, an idempotency key, and the expected output key. Retrying after a crash either confirms the completed output or replaces the same versioned result; it does not create an ambiguous second replay.

The scheduler separates work by consequence. A low-resolution replay and poster may be needed within twenty minutes. High-bitrate renditions, waveform generation, extra thumbnails, and archival formats can wait. Per-owner quotas and distinct interactive, publication, and bulk queues prevent one creator’s back catalog from starving every new replay.

Publication is a commit over a coherent set of derivatives. Workers may finish in any order, but clients receive manifest version 12 until publication version 13 names a valid set of segments and renditions. The service writes the new immutable manifest, verifies its referenced objects, changes the publication pointer, and then allows caches to discover it. A partial-quality publication is honest if the product labels it and can later advance atomically to the full ladder.

The creator’s higher-quality master becomes a new source version. Old derivatives remain addressable while source 2 is probed and processed. Once the replacement is ready, a new publication version switches playback. This avoids overwriting a popular object in place, where CDN nodes, manifests, and players can disagree about which bytes an unchanged URL means.

Codec migration uses the same machinery. Generate the new recipe beside the old one, publish it to capable clients or a small cohort, compare startup, rebuffering, quality, errors, and cost, then expand or roll back by publication version. Deleting the old recipe before the new path proves itself destroys the easiest recovery route.

Transfer the lifecycle to the other prompts

For a pure video-upload prompt, spend more time on the resumable session. Clients upload numbered parts directly to constrained storage destinations; completion verifies part identity and a whole-object checksum. Expired sessions and orphaned parts are reclaimed. Treat file extensions and declared content types as hints: probe the bytes, cap size and duration, quarantine malformed input, and isolate risky decoders. Preserve the accepted original before starting destructive work.

For video-on-demand streaming, the source is already packaged. The difficult path runs from manifest through CDN and player. Discuss startup time, first-frame failure, rebuffer ratio, seek behavior, quality switches, origin shielding, private playback tokens, and hot versus long-tail objects. Signing each highly personalized URL may weaken shared caching; separating the authorization decision from the stable cache key can preserve both access control and edge reuse when the threat model permits it.

For a podcast, the publication contract includes both media bytes and feed metadata. Normalize and validate the audio, preserve stable episode identity, render a versioned feed, and account for clients and platforms that poll, prefetch, retry, and cache on their own schedules. Replacing an episode may require a new media version without changing its feed identity. Download analytics should state what they count; raw requests mix listeners with range requests, crawlers, retries, and prefetches.

For images, arbitrary transformation parameters are a disguised compute and cache attack. Validate signature, dimensions, decompression bounds, ownership, and policy before processing. Precompute common thumbnails when first-view latency matters; create uncommon variants lazily when long-tail storage would cost more. Normalize every allowed crop, size, density, quality, and format into a bounded recipe key. Expensive decoders run in isolated workers, and the CDN never receives an unbounded query-string vocabulary from clients.

These are not five copies of one architecture. Live media has an irreversible present; uploaded media can wait for validation. A podcast feed is observed by caches the service does not control. Image processing faces extreme transform fan-out. The transferable discipline is narrower: identify the source, name the derivative version, define the publication event, and follow the client that experiences failure.

Make security, cost, and deletion reach the edge

Media systems accept large, opaque inputs and can amplify them into expensive global outputs. Authenticate upload sessions and broadcasters; bind temporary credentials to owner and event; enforce byte, duration, concurrency, transform, and egress quotas before costly work; probe file signatures; scan or moderate according to product policy; and keep untrusted parsers and encoders away from broad storage credentials.

Private playback authorization needs an expiry and a revocation story. A signed token may authorize a manifest or path prefix, while the origin and CDN must agree on which token fields affect caching. Stream keys and download URLs do not belong in logs. Administrative replacement, takedown, legal hold, and deletion are privileged actions with durable audit evidence.

A takedown is a distributed workflow, not a metadata flag. Stop issuing new playback authorization, unpublish manifests and feeds, purge or expire edge objects according to the promise, cancel queued processing, and prevent an old job from republishing forbidden derivatives. Deletion must follow the source version through variants, transform caches, archives, and indexes while respecting any legitimate retention hold. The system should report what it cannot retract from devices that already downloaded the bytes.

Cost pressure belongs in the architecture because the largest bill may sit outside the application database. Measure stored bytes by source and recipe, encoder time by media minute and codec, CDN and origin egress by region, long-tail hit ratio, abandoned upload bytes, and unused derivatives. Then make specific choices: archive cold originals, prune obsolete recipe versions after the rollback window, restrict low-latency modes to products that need them, and stop generating renditions that devices or viewers rarely select.

Observe the promise the viewer sees

Instrument the boundaries with identifiers that survive the trip: broadcast, media, source version, publication version, rendition, segment sequence, processing job, and request correlation. Operators should be able to move from a player error to the CDN response, origin object, manifest, packaging attempt, and ingest sequence without logging private media content.

Watch the pipeline in product terms:

• upload completion and checksum failure, plus the age of abandoned sessions;
• archive gaps, ingest reconnects, late segments, and time to publication;
• queue age by job class, processing duration, deterministic failures, retries, and partial-publication time;
• playback startup, first-frame failure, rebuffering, quality switches, manifest errors, CDN hit ratio, and origin load;
• takedown propagation, unauthorized delivery attempts, storage, processing, and egress cost.

Return to 20:03. A credible answer can now say whether sequence 8421 reached either ingest region, whether it entered the archive, whether its rendition objects committed, which manifest version named it, what the CDN returned, and how players recovered. It can also say whether the replay should contain a gap, be repaired from the redundant source, or wait for the creator’s master.

That is the useful rehearsal. Draw the source, archive, packager, manifest, CDN, and player. Lose one segment. Then follow the same media record into a replay, podcast, thumbnail, replacement master, and takedown. If every step can name its authoritative version and its user-visible failure, the architecture is doing more than moving files.

Related links

• Networking and the Web
• Caching and Content Delivery
• Queues, Logs, and Stream Processing
• Observability, Performance, and Cost
• Real-Time and Location Systems
• Data and Workflow Platforms