Communication Systems

When has a message been sent?

Mara posts a message to a project room from a train. The connection drops before the client receives a response. She retries from her phone. One teammate is online, another opens the room that evening, and a third has muted it but asked to receive direct mentions by email.

Has the message been sent?

A diagram containing an API, a queue, a database, and a fleet of WebSocket gateways cannot answer. The word sent may mean that Mara’s device emitted bytes, that the service accepted responsibility, that one gateway attempted delivery, that another device received the event, or that a person read it. Those are separate facts, and a credible design refuses to exchange one for another.

The distinction travels across this whole family of prompts. Chat promises recoverable conversation history and intelligible ordering. Presence promises a fresh-enough hint. Push promises a wake-up attempt, not proof of attention. Email promises durable work and a reconciled provider outcome. Notification preferences promise that user intent is enforced before a channel becomes noisy.

Give one room a precise promise

Suppose the prompt is to design group chat with presence and notifications. Clarify the product before the machinery:

• Does an accepted message have to survive a disconnected sender and an offline recipient?
• Is ordering required within a conversation, within one sender’s messages, or across the entire service?
• Can new members read old messages, and does a departing member retain history?
• Which states does the interface expose: accepted, delivered to a device, read, edited, deleted, or failed?
• How large can a room become, how many devices can one user have, and how long may presence be stale?
• Do mutes, blocks, quiet hours, mentions, and unsubscribes change delivery by channel?

Assume an ordinary collaboration product. Once the service accepts a message, authorized members can recover it in conversation order. Online members should usually see it within a second or two. Offline members catch up later, and may receive notifications according to current preferences. Presence and typing are hints. A delivery receipt refers to a device; a read receipt is an optional product claim about a user.

The opening contract can be spoken in less than a minute:

Accepted: authorized and durably recorded with a conversation position.
Ordered: by server sequence within this conversation, not globally.
Delivered: offered to a connected device or available through later sync.
Confirmed: an optional device or read receipt, never implied by acceptance.
Safe retry: sender + conversation + client_message_id returns one message.

This is the answer’s center. The architecture now has something to protect.

Follow the message through acceptance

Mara’s client submits the conversation id, a client-generated message id, body metadata, and attachment references. The chat service authenticates her, authorizes current membership, checks blocks and moderation policy, and looks up the retry key (conversation_id, sender_id, client_message_id).

If the key is new, the service assigns a conversation position and commits the message. A compact record might contain:

Message
  conversation_id, sequence, message_id, sender_id
  client_message_id, body_ref, created_at
  visibility_state, edit_version, deletion_state

MembershipInterval
  conversation_id, user_id, joined_sequence, left_sequence, role

The service acknowledges only after the durable commit. If the response is lost and Mara retries, it returns the original accepted message rather than appending a duplicate. The WebSocket connection is not part of this truth; it is one delivery path from it.

A single monotonically increasing sequence per conversation makes pagination, offline replay, unread positions, and support investigation coherent. It can also become a serialization point for a very hot room. Most rooms can be partitioned by conversation. An exceptional public room may need an append log with ordered merge, several substreams with weaker display order, or a product decision that not every event deserves one total order. The answer should weaken the promise explicitly before weakening the mechanism.

Membership intervals give ordering a privacy consequence. A member who joins at sequence 900 may or may not see earlier history; a member who leaves at 1200 must not receive 1201 through a stale gateway subscription. Store the chosen policy rather than reconstructing it from today’s membership row. Edits and deletions also need ordered events or version checks so an old payload cannot reappear after a tombstone.

Let delivery fail without losing the conversation

After commit, an outbox or log event tells routers that sequence 1042 is ready. Gateways holding authorized room subscriptions offer it to connected devices. A device advances its sync cursor after receiving the event; after a gap or a reconnect it asks for messages after its last known sequence.

This path tolerates several ordinary failures:

• If an event is routed twice, the stable message id and sequence let a client deduplicate it.
• If sequence 1043 arrives before 1042, the client can buffer briefly or fetch the missing range rather than trusting arrival order.
• If a gateway dies, another gateway resumes from durable history; no message migration between socket servers is required.
• If routing falls behind, acceptance may remain available until a declared queue-age or capacity bound forces backpressure. The service should not turn an unbounded promise into an unbounded queue.

Small rooms can receive online fan-out on write. Sending every message into a million member inboxes is a different product. For a very large room, keep one durable history, push real-time events to active viewers, notify mentioned or explicitly subscribed members, and let inactive members read on return. This hybrid spends work where immediacy is visible.

Unread counts are derived state. Exact per-user counts are convenient for small groups but expensive under large fan-out and membership churn. A last-read sequence plus approximate badges may be enough. Deleting or hiding a message can make counts briefly disagree with the visible page; decide whether that is a cosmetic lag or a product defect before paying for stronger coordination.

Presence belongs to another clock

The teammate shown as online may have closed a laptop without sending a clean disconnect. Treating connection teardown as truth leaves ghosts whenever a device sleeps or a network partitions.

Instead, gateways write session heartbeats with an expiry. The product derives online or away state from recent authorized sessions, and stale state ages out. Typing indicators are weaker still: short-lived, rate-limited events that need not be replayed. A newer heartbeat version must win over a delayed older one, but presence does not need the conversation’s durable ordering.

Visibility is part of the model. Membership, blocks, privacy settings, and tenant boundaries decide who may observe last-active or typing state. Regional aggregation can reduce watcher fan-out, provided the displayed freshness bound admits the added delay. “Online within the last thirty seconds” is an honest contract; globally consistent presence is usually an expensive fiction.

A notification is not a second copy of the message

The offline teammate needs a durable in-product notification event that points to sequence 1042. A notification planner evaluates the event against recipient, room, workspace, event type, mention status, device, channel, quiet hours, digest rules, and explicit overrides. The decision should be explainable:

event 7f3... -> user 28
in-app: recorded
push: suppressed (room muted)
email: queued (direct mention; critical override allowed)
policy_version: 381

Preferences are therefore part of delivery, not a settings page added after fan-out. Cache their read-heavy state, but invalidate or recheck user-protective changes such as block, unsubscribe, and workspace removal before a slow external send. Otherwise the system can act on an old preference minutes after the user revoked consent.

For push, choose eligible devices, discard invalid tokens, collapse replaceable events, and enqueue provider-specific attempts. The payload should reveal no more on a lock screen than the privacy policy allows. Provider acceptance means only that the provider accepted an attempt; the chat history remains the source from which the app recovers the message.

For email, preserve a send intent containing tenant, recipient, template and version, payload reference, priority, and idempotency key. Transactional and security mail should not wait behind a bulk campaign. Workers enforce suppression, hand off to a provider, then reconcile delayed delivery events, bounces, complaints, and unsubscribes. Provider failover is safe only when the previous outcome is known or duplicate delivery is acceptable; an unknown timeout is not permission to send the same receipt again.

Abuse and operations follow the promise

Communication systems provide attackers with distribution. Rate limits should cover senders, recipients, rooms, tenants, device tokens, and external channels, not merely API requests. Large-volume tenants need reputation and quota controls. Blocks and moderation decisions must reach online routing as well as history reads. Operators need a bounded way to stop a tenant, room, campaign, template, token, or provider credential without disabling unrelated traffic.

Useful signals follow a message’s changing status:

• acceptance latency, durable-write failures, idempotency hits, and rejected authorization;
• routing age, missing-sequence recovery, active connections, reconnects, and delivery attempts;
• heartbeat age, expiry rate, watcher fan-out, and stale-presence reports;
• notification decisions by reason, queue age by priority, provider throttles, invalid tokens, bounces, complaints, and duplicate suppression;
• blocks, reports, tenant volume, moderation age, and emergency shutdown use.

A single “messages sent” counter erases the very distinctions the design depends on. Support tools should be able to trace one message from acceptance through routing and notification decisions without exposing its contents to every operator. Audit access to history, recipient addresses, tokens, preference overrides, and delivery records.

The same boundaries expose cost. Online fan-out consumes connections and egress; durable history consumes storage and index work; exact unread state multiplies writes; heartbeats consume regional write capacity; provider sends cost money and reputation. Coalesce low-value repetition, expire hints, archive history under the product’s retention policy, and measure cost by tenant and channel. Do not save money by silently weakening unsubscribe, privacy, or recoverability.

Transfer the contract to neighboring prompts

The group-chat design supplies a method, not a component recipe.

One-to-one chat: synchronize devices, not just people

The conversation promise stays durable and ordered, but multiple devices make delivery and read state more visible. A send retried from a phone and laptop still produces one message. Each device needs a sync cursor; a user-level read position may advance from one device and appear later on another. If end-to-end encryption is required, key distribution, membership changes, device removal, backup, abuse reporting, and server-visible metadata become first-class parts of the prompt rather than a sentence appended to the design.

Presence: promise bounded staleness

There may be no durable communication record at all. Define heartbeat interval, expiry, aggregation, watcher authorization, and the UI’s stale-state language. Design for missing disconnects and delayed heartbeats. Typing events should disappear rather than enter a replay queue.

Push notifications: preserve the underlying event

Persist the product event or notification history, evaluate current preferences, select devices, and treat providers as fallible edges. Token rotation, invalid tokens, rate limits, collapse identifiers, delayed attempts, and private lock-screen content are the useful deep dives. Losing an attempt must not lose the event the app would show after opening.

Email delivery: reconcile custody

Define whether acceptance means durable enqueue or provider handoff. Version templates, isolate priorities, make retries duplicate-aware, process bounces and complaints, enforce suppression, and retain enough audit to explain an outcome. The system owns its decision to send even when a provider owns the last transport leg.

Notification preferences: make user intent executable

Model defaults and overrides by event type, channel, scope, and urgency. Mutes, quiet hours, digests, blocks, and unsubscribe differ: some delay, some coalesce, and some prohibit. Store a policy version or decision reason so support can explain why a notification fired. Decide which changes require immediate cache invalidation and a final pre-send check.

Rehearse the lost response

Give a twenty-minute group-chat answer beginning with Mara’s message. Carry it through authorization, retry, conversation order, online routing, reconnect, presence, mute evaluation, and one external notification. Do not add components until one of those transitions requires them.

Then change one condition:

• the room grows from twelve members to one million;
• leaving a room must stop every new delivery within five seconds;
• one region accepts messages but cannot reach the routing plane;
• users require end-to-end encryption across several devices;
• a push provider returns timeouts after accepting some attempts;
• an unsubscribe arrives while ten thousand email jobs are already queued.

For the chosen change, say which promise moved. Follow the consequence through acceptance, stored state, ordering, fan-out, retry, user-visible status, operating signals, and cost.

The design is ready when the word sent is no longer doing hidden work. The service may accept a message, attempt a route, receive a device acknowledgement, record a read, or reconcile a provider event. Each transition has its own evidence and failure. Once those facts are kept separate, queues and sockets become implementation choices instead of substitutes for a product promise.

Related links

• Queues, Logs, and Stream Processing
• Distributed Systems Fundamentals
• Security and Privacy
• Observability, Performance, and Cost
• High-Level Architecture and Data Flow
• Reliability and Failure Engineering
• Content and Storage Systems
• Feeds, Search, and Recommendations