Real-Time Fan-out, Presence, and Collaboration

At 14:03:11, Relay Rooms begins accepting 60 900-byte announcements each second into a room with 800,000 subscribed sessions. At 14:03:12, the publish API is still healthy and every connection owner is reachable. At 14:03:14, aggregate socket egress is 49 GiB/s, several owners have crossed their soft queue limits, and clients switching from mobile data to Wi-Fi begin reconnecting. At 14:03:18, replay competes with new delivery. The broker has not failed; the design has allowed each accepted event to become unbounded recipient state.

“Four million open connections” did not predict this incident. The useful units are:

• accepted publishes per second by room/topic and event class;
• eligible recipient deliveries per publish, including audience skew;
• encoded bytes and CPU per delivery;
• queued bytes and oldest queued event per connection;
• durable log and replay work per reconnect;
• routing-directory and subscription operations; and
• connection establishment, migration, heartbeat, and drain work.

The primary decision is where fan-out becomes concrete and what happens when a recipient cannot keep up. The boundary starts when a publish or session operation is accepted. It ends when an eligible client has either applied the event in order, learned that it must repair a gap, or been disconnected under an explicit slow-consumer policy. A successful enqueue into an internal broker is not end-user delivery.

A connection is owned state, not a file descriptor

A persistent transport such as WebSocket gives the server a bidirectional byte channel, control frames, and a closing handshake. The application still owns identity, authorization, subscriptions, sequence positions, liveness, flow control, and resumption. Treating the transport as the session contract leaves recovery and duplicate semantics undefined.

A connection owner should know at least:

session_id and authenticated principal
connection generation and owner epoch
transport handle and protocol/version
authorized subscriptions with versions
last accepted client sequence
last sent and last acknowledged server position by stream
queued bytes/events and slow-consumer state
heartbeat/liveness timestamps
regional placement and drain status

Not every field must be durable. The socket handle is inherently local and ephemeral. A resumable session identity and last acknowledged position may live in a shared or replicated session store. Subscription authority may belong to a room membership service, with a versioned derived copy at connection owners. Name the authority for each field so migration does not merge two owners’ guesses.

The owner epoch fences stale processes. When a directory maps session s to owner B at epoch 42, owner A at epoch 41 must not continue to accept publishes or acknowledgments after transfer. A lease, generation, or compare-and-swap can establish one active owner; the exact mechanism must survive delayed messages and partitions. Without fencing, migration produces duplicates, reordering, or two buffers billed to one user.

Heartbeat policy is a failure detector, not proof of death. Ping/pong or application heartbeats detect an unresponsive path after a timeout, but mobile suspension, proxy timeouts, packet loss, and scheduler stalls can look the same. Choose interval and timeout from desired detection, battery/network cost, proxy behavior, and false-positive tolerance. Jitter reconnect attempts so a regional network recovery does not become an authentication and replay storm.

Rooms, topics, and presence have different truth

A subscription says which events a session is eligible to receive under a policy and version. A room often adds membership, roles, history, and ordering. A topic may be a looser routing name. Audience size is a distribution: private rooms, workgroups, and celebrity broadcasts should not share one assumed fan-out.

Presence is usually a time-bounded inference such as “this user has an active session seen within 45 seconds,” not durable truth about a human. One user can have several devices and regions. A disconnect may be delayed or never observed cleanly. Model presence as session observations aggregated into a product state with expiry, privacy, and visibility rules. Do not put every heartbeat into the durable message history merely because both are “events.”

Presence and cursor movement are often overwriteable state. If ten updates for the same (room, user, cursor) are queued and only the newest matters, coalesce them. Chat messages, financial instructions, and membership changes are not equivalent; dropping an intermediate event can violate their contract. Classify event semantics before choosing compression or loss.

Membership checks must occur at a defined version. A publish accepted before a member removal but delivered afterward needs policy. Sensitive removals may require immediate directory invalidation and queue purge; ordinary state updates may follow room-log order. Cache authorization with bounded staleness only where the consequence is acceptable.

Follow one event to its connection owners

A distributed message path commonly resembles:

publisher -> authenticated publish service
          -> room authority / durable room log
          -> fan-out router by room shard
          -> connection-owner directory
          -> owners with eligible local sessions
          -> per-connection scheduler and socket
          -> client apply and optional acknowledgement

The durable append establishes the event identity and room position. The router does not need one message per recipient if many recipients reside on the same owner; it can send one room event plus a local recipient set or subscription index. Owners must avoid scanning all local connections. Maintain a room-to-local-session index whose update and migration semantics are explicit.

The path has two fan-outs: room shard to connection owners, then owner to local connections. Measure both. Broker publish rate can remain low while owner delivery rate and socket bytes explode. Include envelope, encryption, compression, framing, copying, serialization, kernel buffering, and retry in per-delivery cost.

Ordering is scoped. A per-room sequence can order committed room events without imposing one global order across every room. A per-user notification stream may merge several sources using a separate position. State the key, sequencer, and failure boundary. Arrival on one socket does not prove causal completeness if events came from independent streams.

Give each durable event an immutable identity and stream position. Clients remember the last contiguous applied position, deduplicate repeated event IDs, and detect received_position > expected_position. A gap triggers bounded repair, not silent continuation. If the missing range has expired or is too large, the client obtains a snapshot at position p, replaces scoped derived state, then applies deltas after p.

Fan-out on write, on read, or at a boundary

Fan-out on write materializes recipient work when the event is accepted: per-user inbox entries, owner deliveries, or both. Reads are cheap and personalized filtering can be precomputed, but a large audience creates write amplification, storage, and queue pressure. Membership changes and deletion can require updating many derived entries.

Fan-out on read stores one event in a room/topic log and has clients or gateways fetch or merge it later. A celebrity publish stays cheap, but every active and reconnecting reader pays selection and merge cost. Low-traffic readers may repeatedly scan shared state; personalized order or filtering can become expensive.

Hybrid fan-out chooses a boundary. Materialize private and medium rooms into recipient inboxes; keep very large broadcasts as shared logs and send active sessions a lightweight invalidation or room position. A client read then fetches a bounded range or snapshot. The threshold should depend on recipient count, active fraction, event rate, payload, personalization, storage/write cost, read/reconnect rate, and deadline—not a folklore follower count.

The fixture contrasts 12-recipient private rooms at 20 publishes/s with an 800,000-recipient broadcast at 60 publishes/s. At 1,100 encoded bytes per delivery, the private class creates 240 deliveries/s, about 0.252 MiB/s. The broadcast creates 48 million deliveries/s and about 49.17 GiB/s. Its audience multiplier is about 66,667× the small room. These are modeled steady inputs; compression, owner aggregation, active fraction, network framing, and acknowledgement shape can move the result dramatically.

A decision table should preserve semantics:

Do not switch strategies invisibly at a threshold if it changes ordering, offline availability, or deletion. Keep one client protocol—position, gap, snapshot, delta—even when the server changes materialization.

The routing directory must survive movement

The directory answers “which owner currently holds session s?” or more efficiently “which owners currently have subscribers for room r?” Cache it, but bind entries to epochs and expiry. A stale positive can route to an old owner; a stale negative can drop an event. The old owner should redirect or reject with the newer epoch, never accept indefinitely.

A connection cannot literally migrate between processes without transport support and shared endpoint semantics. Operational migration usually means:

1. mark owner A draining and stop assigning it new sessions;
2. retain existing sessions while clients receive a reconnect hint or naturally reconnect;
3. establish session generation on owner B through the directory;
4. resume from last acknowledged durable position;
5. fence A, expire its directory entries, and release buffers; and
6. terminate remaining connections at a bounded deadline with jittered retry advice.

During regional movement, choose whether the user’s home log stays in the original region, follows the user, or is replicated. Each choice changes publish latency, read latency, residency, and failover. The connection can be near the user while room authority remains elsewhere. Make that extra hop visible rather than claiming all state is local.

A directory outage needs a safe mode. Existing owners may continue already authorized subscriptions for a short lease, while new sessions or membership changes fail closed. Broadcasting every event to every owner “until discovery recovers” converts control failure into network collapse and may violate isolation.

Delivery semantics need positions, not slogans

At-most-once transport can lose events on disconnect. Retrying can duplicate. A durable log plus idempotent event IDs and sequence-aware clients can provide effectively-once application effects within a declared stream and retention window, but the system still transmits duplicates and must handle ambiguous acknowledgements.

Separate stages:

• accepted by publish API;
• durably committed to room authority;
• selected for an eligible session;
• enqueued at an owner;
• written to a transport;
• received, persisted, or applied by a client; and
• acknowledged at a named position.

Only promise the stage you can prove. A socket write completion says bytes entered a local transport buffer; it is not client application. Per-event acknowledgements for 800,000 recipients may be more expensive than delivery. Cumulative acknowledgements of the last contiguous position reduce work, with explicit gap and timeout handling.

Ephemeral events may be dropped, superseded, or expire before replay. Durable events remain in a log or snapshot path for a declared window. One envelope should carry event class, stream identity, sequence, event ID, expiry or retention class, and schema version. A client that does not understand a required durable schema must stop and upgrade or use a compatible snapshot; silently skipping it would advance the position past unknown state.

A slow consumer is a state machine

Every per-connection buffer must have byte, event-count, and age bounds. Bytes protect memory and egress; count protects per-event overhead; age protects the product’s temporal meaning. A 100-byte presence queue can be useless when it is 30 seconds old.

Use explicit states:

NORMAL
  queue below soft byte/count/age limits
  -> SOFT_LIMIT when any soft limit is crossed

SOFT_LIMIT
  coalesce overwriteable state; stop optional events; lower scheduler weight
  -> NORMAL after bounded recovery
  -> SNAPSHOT_REQUIRED when durable gap exceeds delta policy
  -> HARD_LIMIT when hard byte/count/age limit is crossed

SNAPSHOT_REQUIRED
  discard superseded deltas in the scoped stream; send snapshot marker/position
  -> NORMAL after snapshot acknowledgement
  -> HARD_LIMIT on deadline or continued growth

HARD_LIMIT
  record reason and last durable position; close transport with retry guidance
  -> reconnect and bounded repair, never unlimited queue retention

The teaching policy uses a 256 KiB soft byte limit and 1 MiB hard limit. If 0.5% of an 800,000-session broadcast audience reaches the hard limit, 4,000 connections alone hold about 3.91 GiB of queued payload, before object, allocator, index, and transport overhead. The safe hard limit therefore depends on owner memory reserve and simultaneous slow fraction, not only what one client can tolerate.

Backpressure must propagate to the correct boundary. One slow recipient should not stop a whole room. The owner may drop that recipient’s ephemeral updates, require a snapshot, or disconnect it. If most recipients are slow, the system may be regionally impaired; continuing to accept unlimited durable publishes only grows replay debt. Admission can reject optional publishes, lower presence frequency, switch massive rooms to log-notification mode, or declare delayed delivery.

Fair scheduling prevents one connection with a deep queue or one large room from monopolizing an owner’s write loop. Use bounded work per connection/room per turn, separate control traffic from bulk deltas, and charge CPU/bytes to tenant and room. A “writable socket” signal does not guarantee the downstream network will drain at the modeled rate; track actual queue residence and progress.

Coalesce state; snapshot history when deltas stop paying

Coalescing replaces several pending states with the newest state for the same semantic key. The fixture models 120,000 cursor/presence updates/s reduced 20:1 to 6,000 published states/s before recipient fan-out. That is valid only if intermediate values are not required. Coalescing after fan-out may save socket bytes but not broker/owner work; move it before the expensive boundary when semantics permit.

Deltas are efficient while the receiver has the correct base. A snapshot establishes a complete scoped state at position p; later deltas apply after p. Include schema and membership boundary. Generate snapshots at a sustainable cadence, cache immutable snapshots, and prevent a reconnect wave from causing one fresh snapshot build per client.

Compression trades CPU, dictionary state, and head-of-line risk for bytes. Batch several tiny events only within a latency allowance and without mixing authorization scopes. A compression context shared across messages can leak information or make recovery depend on missing history; reset boundaries and security need review. Measure CPU and tail latency at realistic room skew, not only compression ratio.

Reconnect repair has a bounded decision tree

A reconnecting client presents session identity, stream identity, last contiguous applied position, and supported schema. The server checks:

1. Is the session still authorized and resumable?
2. Does the position name the same stream generation?
3. Is every required event after that position retained?
4. Is the delta count/bytes/age within replay limits?
5. If not, which snapshot supersedes the missing range?

The fixture gives durable replay a 900-second and 5,000-event bound. An ordinary stream at five events/s accumulates 4,500 events and fits. A celebrity stream at 60 events/s accumulates 54,000 and does not; it must use a snapshot/shared-log recovery or shorter outage. The 5,000 cap is not a silent truncation. The server returns “snapshot required,” including a stable snapshot identity and position, so the client cannot mistake incomplete history for completeness.

Deduplicate by event ID within retention and by sequence application. If the client receives position 106 after 104, it does not simply set its cursor to 106; it records a gap at 105, buffers or discards later data under a bound, and repairs. If the server returns a limited timeline, the protocol must indicate the gap. Sequence rollover, stream reset, account switch, and room rejoin need different generation identities rather than ambiguous reuse.

Offline catch-up should not compete without limit with foreground delivery. Give replay separate concurrency, byte, and CPU budgets; prioritize recent interactive repair; cap per-session work; and cache shared snapshot/log ranges for popular rooms. Jitter clients and admission after regional recovery. Observe resume success, replay versus snapshot choice, missing positions, replay bytes/events/age, repair latency, and reconnect cause.

Connection capacity is multi-dimensional

The fixture models four million sessions, a full-load owner limit of 100,000 connections, and a 70% target. The fleet needs ceil(4,000,000 / 70,000) = 58 owners before zone/failure and deploy reserve. At 24 KiB base plus 32 KiB average queued state per connection, resident connection state alone is about 213.62 GiB across the fleet. Runtime, TLS, kernel, subscription-index, allocator, and page overhead remain uncounted.

Use this worksheet:

Connections and churn
  concurrent by region/device/protocol, establish/resume/disconnect rate,
  heartbeat traffic, session duration, TLS/auth CPU, owner utilization

Subscriptions and audience
  rooms per connection, active/inactive fraction, recipient distribution,
  owner-level room index size, membership and presence update rate

Delivery
  publishes/s by class, recipients/publish distribution, encoded bytes,
  serialization/compression CPU, broker-owner and socket egress, acks

Buffers and slow consumers
  base/queued bytes per connection, queue count/age, soft/hard limits,
  coalesce/drop/snapshot/disconnect rates, allocator and kernel reserve

Durability and reconnect
  log/snapshot write rate, retention, positions, replay count/bytes/age,
  resume success, gap/snapshot rate, reconnect admission capacity

Placement and operations
  owner/directory failure domains, regional authority hops, headroom,
  deploy batch, drain deadline, zone loss, reconnect-wave reserve

Evidence and decision
  observed/model scope, versions/date, skew and failure tests,
  cost, uncertainty, admission/degradation, rollback trigger

File descriptors can be the first limit, but they are not the capacity model. Check memory, event-loop or scheduler delay, TLS/auth CPU, heartbeat work, directory operations, subscription index, broker fan-out, serialization, network packets/bytes, buffers, replay storage, and reconnect admission. Validate at connection age and churn representative of production; one million idle loopback sockets prove little about 800,000-recipient delivery.

Deploy draining is a capacity event. Four owners with 70,000 live sessions each displace 280,000 connections. At a protected reconnect admission of 10,000/s, the arithmetic floor is 28 seconds, excluding client detection, jitter, TLS, authentication, directory updates, subscription restoration, and replay. A safe batch interval must be longer and must preserve zone-loss headroom. Stop a rollout when resume latency, replay work, owner utilization, or slow-consumer rate crosses its trigger.

Applied work

Design one chat service for private groups and massive broadcasts. Keep one client envelope with event ID, stream generation, sequence, class, and expiry. Use a durable room log for messages and membership. Push private-room events through owner-level local indexes; switch massive rooms to a shared log plus active notification or hierarchical owner fan-out. Coalesce presence/cursor state, never durable messages. Define the threshold inputs, membership-version check, slow-consumer state machine, reconnect path, regional placement, origin/fan-out budgets, and evidence that strategy changes do not alter client completeness.

Specify reconnect gap repair. Write the exact request and responses for contiguous replay, duplicate event, gap within retention, gap beyond count/byte/age bounds, stream-generation change, authorization removal, and snapshot fallback. Include the position a snapshot represents and the first legal delta afterward. Test disconnect after server send but before client acknowledgement, owner loss, delayed duplicate delivery, retention expiry, and reconnect storms.

Rehearse a celebrity event during a deploy. Drain one owner batch while publishing at the protected large-room envelope. Inject 0.5% slow clients and a regional reconnect surge. Verify owner-level rather than per-recipient broker traffic, queue bounds, ephemeral coalescing, durable gap semantics, reconnect admission, directory epochs, zone reserve, and rollout stop. Success means bounded memory and honest degradation, not zero disconnects.

Sources and transfer limits

• RFC 6455, The WebSocket Protocol defines WebSocket framing, ping/pong, close behavior, and reconnect considerations at the transport protocol boundary. It does not define application sessions, ordering, replay, or slow-consumer policy.
• Matrix Client-Server API synchronization specification v1.19 provides one current example of since/next_batch positions and explicit limited timelines across gaps. Matrix room semantics are not a universal chat protocol; the transferable principle is that truncation and recovery position are explicit.
• Discord Gateway documentation documents one product’s sequence-based session resume and lost-event replay. Its limits and close codes are version-specific and are evidence of a concrete design, not requirements for other systems.
• Slack’s real-time messaging architecture describes persistent client WebSockets and channel-to-message-server routing in one production architecture. Scale, topology, and implementation claims do not transfer without current local measurement.

The chapter’s numeric examples are reproduced by examples/performance-engineering-system-design-handbook/part-05/realtime-fanout/. They are modeled, not measurements of Matrix, Discord, Slack, WebSocket, or another service. A production choice needs actual audience distributions, active fraction, payload/envelope sizes, owner aggregation, queue residency, runtime/TLS costs, reconnect causes, replay retention, regions, and failure/deploy evidence.

Decision rule

Bound state per recipient by bytes, count, and age; make coalesce, snapshot, drop, and disconnect transitions explicit; and capacity the system by deliveries, queued bytes, replay, and churn as well as connections. Choose write, read, or hybrid fan-out from audience and recovery distributions while preserving one position-and-gap contract for clients.

Fan-out cost varies by room and recipient state even when every event has the same byte size. The next archetype adds another source of heterogeneity: inference requests whose model, input shape, batching opportunity, memory footprint, and quality target can differ by orders of magnitude.