Collaboration and Multi-Tenant Platforms

The edit submitted after access ended

At 14:00, a contractor opens a product roadmap in a customer’s workspace. The document service authorizes the read, the collaboration service admits the session, and the client downloads the current document. Its network connection then drops, and the client continues accepting edits offline.

At 14:03, a workspace administrator removes the contractor. The membership service commits the change and emits a revocation event. One collaboration server receives it promptly; another has a stale membership cache. A search index still contains the document title. A mention notification is waiting in a queue. An export requested before the revocation has not yet produced a download.

At 14:05, the client reconnects and submits three operations created while it was offline. What should happen to them?

“Check permissions” does not settle the design. Which permission, checked when, by which service, against which resource version? May the user recover a local copy of rejected work? Should an export authorized before revocation be allowed to finish afterward? How quickly must open sessions disappear? The answers are product and security promises, not properties of a database column.

This case gives the chapter its route. Define the boundary, follow the edit through revocation, and then reuse the same reasoning for calendars, permissions, identity, tenant isolation, and audit systems.

Put a clock on the authorization decision

Begin with the nouns required by a protected operation:

tenant + actor + resource + action + context
    -> decision at policy version and time
    -> effect or denial
    -> audit evidence when the action warrants it

The tenant is the administrative and data-ownership boundary: perhaps an organization, workspace, account, or customer contract. The actor may be a person, service account, worker, tenant administrator, or support operator. The resource may be a document, calendar, group, policy, export, or audit record. Use specific actions such as read, edit, invite, share, export, and impersonate; a generic access permission conceals too much.

Context contains only facts that can legitimately change the answer: session state, resource status, sharing grant, device posture, policy version, break-glass case, or request origin. Derive the resource and its tenant from trusted server state. A browser that belongs to several workspaces may propose a workspace, but it cannot prove the scope of a stored document by sending a tenant_id.

Time belongs in the model because a decision licenses an effect, not every future continuation of a request. A read decision licenses the response being constructed. Joining a live-editing session licenses participation under the session’s revalidation rules. It does not grant an indefinite right to submit new mutations. Creating an export request may authorize creation of the job, but the product may require another check before reading rows or releasing a download.

State the freshness promise in terms a user can understand. “Revocation takes effect within 30 seconds for connected sessions and before any newly started export” is a design. “We cache permissions” is an implementation detail with an unspoken exposure window.

Decide what the revocation changes

The administrator’s action first updates authoritative membership state. That write receives a new policy or membership version and emits an invalidation event. Session gateways, collaboration servers, policy caches, and other interested services consume the event. Because delivery can be late, mutation paths still enforce a maximum cache age or verify that the decision grant’s membership and policy versions remain current before committing sensitive work.

For the contractor’s open document, choose explicit behavior:

• New reads, comments, shares, and edits are denied after the revocation becomes authoritative.
• Connected sessions are closed or changed to a non-authoritative local view within the promised propagation window. Presence expires even if the close message is lost.
• The reconnecting client may show its three rejected edits and offer to copy them locally. It must not upload them into the tenant, preserve them in a server-side draft, or reveal newer document state without authorization.
• Search results are candidates, not authority. Before returning a title, snippet, count, or document, search applies current visibility or calls a service that does.
• A queued mention rechecks that the recipient may still know the document exists before placing its title or link in email or push delivery.
• A sensitive export rechecks authority before reading data and again before releasing the artifact if generation is long-running. Its output is private, tenant-scoped, short-lived, and independently revocable.

There is an honest limit. The system cannot recall plaintext already displayed, downloaded, copied, or cached on a device it does not control. Revocation prevents future service and synchronization according to the stated contract; it is not retroactive erasure. Product language and incident response should not promise otherwise.

The same reasoning distinguishes harmless and dangerous continuations. An already accepted telemetry write may finish under the service’s authority. A notification that discloses a private title needs current recipient visibility. A billing record for the completed export request may need to survive even though the export itself is denied. Do not solve all delayed work with one rule.

Keep convergence separate from authority

The collaboration service manages document sessions, operation order, deduplication, presence, snapshots, and history. Operational transformation or a CRDT can reconcile concurrent edits; version-checked writes may be enough for a simpler note. None of those techniques decides whether an actor may submit an operation.

Each submitted operation should carry a stable operation id, document id, base version or causal context, actor and session identity, and the tenant scope established by the server. The service checks authority, rejects a duplicate operation idempotently, integrates an accepted operation, advances document state, and durably records enough history to recover. Snapshots bound replay time; compaction removes obsolete mechanism history only after the required product and audit history is preserved elsewhere.

For an online session, checking every operation against a local session grant may be practical if that grant has a short lifetime and prompt invalidation. For an offline client, there is no valid live grant to extend: every batch is a new request when it reconnects. If part of the batch predates revocation, the timestamp supplied by the client does not restore authority. Client clocks and queued bytes are not permission evidence.

Conflict behavior also belongs to the product. Two authorized editors may converge without losing text, yet disagree semantically. Comments may tolerate eventual ordering; a document title may use a last-writer rule; deleting a document while another user edits it may require a recoverable tombstone. Explain the behavior users receive before naming the algorithm that implements it.

Follow tenant scope through every copy

The canonical document row is only one residence of tenant data. The same content or metadata can reach cache entries, search postings, presence state, queue messages, notification templates, object keys, exports, analytics, backups, support tools, logs, metrics, and traces. Each path needs the minimum trusted context required to keep its promise.

Cache keys include tenant and resource scope. Queue schemas make tenant, resource, originating actor or service, action, and correlation id explicit; workers reject malformed jobs rather than treating absent context as system authority. Object storage uses tenant-scoped prefixes and narrow credentials. Logs and traces keep identifiers and policy outcomes without copying document bodies, invitation secrets, or session tokens. Metrics expose per-tenant harm without making unbounded tenant labels collapse the observability system.

Search forces a visible trade-off. Permission-aware postings can make reads fast, but group and policy changes create invalidation work. Fetching broad candidates and authorizing each result keeps decisions fresh, but adds latency and may leak through counts or timing if filtering is careless. A design may combine coarse tenant partitioning in the index with current resource checks at retrieval. The correct choice follows corpus size, sharing patterns, revocation target, and acceptable disclosure—not a universal search recipe.

Delayed jobs should carry a snapshot of why they were admitted and still know when to ask again. The snapshot supports debugging and audit; it is not automatically a transferable grant. Decide separately for indexing, notifications, retention, deletion, import, and export where current authority is required.

Isolation is a property of paths, not a database count

Many small tenants can safely share tables, queues, and compute when tenant scope is structurally present in keys, queries, jobs, indexes, object paths, credentials, tests, and operator tools. Pooled infrastructure improves utilization and makes fleet operations simpler. It also enlarges the blast radius of a missing predicate, a hot tenant, or an overpowered support tool.

Dedicated databases, keys, queues, or compute pools can reduce that blast radius for regulated, unusually large, or high-consequence tenants. They add provisioning delay, uneven capacity, fleet upgrades, cross-tenant analytics work, and migration machinery. A hybrid design commonly begins pooled and moves selected tenants behind a routing layer to dedicated cells.

Make that move credible. Give each tenant placement a routing generation. Create the destination, copy a consistent snapshot, replay changes, verify counts and checksums, fence or briefly pause writes, advance the routing generation, and retain a deliberate rollback window. Jobs and caches carrying an old generation must be redirected or rejected; otherwise the migration can split one tenant across two homes. Audit identity remains stable across the move.

Quotas defend shared paths before work becomes expensive. Limit API calls, open sessions, edits, storage, search fan-out, notifications, exports, and background concurrency by the resource each operation consumes. Put bulk exports behind separate concurrency and queue budgets so one customer cannot starve interactive edits. Measure latency, errors, queue age, conflicts, storage, and cost by tenant where an operator may need to isolate or move one customer.

Calendars make ownership plural

A calendar event resists the idea that one row has one simple owner. The organizer owns the meeting definition, attendees receive participant-specific state, delegates may act for another person, and a room or shared calendar has its own policy. A recurring series also has a rule plus exceptions, not merely many independent events.

Model the organizer, calendar, tenant, attendees, visibility, recurrence rule, timezone, exceptions, and delegate grants. Give invitations stable ids so a retry does not invite everyone twice. Keep each participant’s response separate from the organizer’s event. Expand recurrence within bounded query or reminder windows, preserving the event’s timezone so daylight-saving changes do not silently move local meetings.

Revoking a delegate stops future administration but does not erase changes already committed while the grant was valid. Removing an attendee cancels their future visibility and reminders according to policy. Free/busy queries return the least revealing answer the caller may receive—often occupied time without private title, location, notes, or attendees. Notification jobs check the current participant view before rendering sensitive details.

This is the collaboration boundary in another form: decide who owns each piece of state, which actor may change it now, which copies follow the change, and what earlier effects remain true.

Permissions and identity are control planes

A permissions service answers a narrow but demanding question: may this principal perform this action on this resource under the current policy? Represent principals, groups, roles, resource hierarchy, actions, conditions, tenant scope, and policy versions. Stable roles are useful for broad duties; resource grants and conditions handle the differences that roles alone cannot.

Decision rate, hierarchy depth, group churn, and revocation latency drive this design more than total user count. Cache keys must include every policy input that can change the result, or the cache is unsound. Policy writes need controlled rollout and audit. A decision response should expose a safe reason and policy version so operators can distinguish a real denial from stale or misconfigured policy.

An identity platform establishes accounts, credentials, sessions, federation, provisioning, and deprovisioning. It does not own every resource permission. External identity providers can assert who authenticated and selected attributes; the application still maps that identity into its tenant and resource policy. Group synchronization, account suspension, session expiry, key rotation, and identity-provider outages all need an explicit freshness and failure policy.

Privileged support access is a separate product path. Bind it to a case or reason, give it narrow and short-lived scope, require approval where the consequence warrants it, and make impersonation visible. An internal tool with a global database credential can defeat otherwise careful tenancy.

Audit records preserve decisions, not secrets

Operational logs help run a service. Audit records answer later questions about sensitive changes: who acted, through which identity and tool, on which tenant and resource, what they attempted, whether it succeeded, why it was allowed or denied, and which policy version participated.

Producers authenticate to the audit service and send a stable event id, timestamp, actor, tenant, resource, action, result, reason code, policy version, and correlation id. The event omits content, tokens, credentials, and other secrets not needed as evidence. Ingestion validates schema, appends durably, deduplicates stable ids or exposes duplicates honestly, and sends records to protected raw storage plus tenant-scoped query indexes.

“Immutable” needs an operating meaning. Application identities cannot rewrite accepted events; retention changes and privileged reads are controlled and themselves audited; integrity measures make unauthorized alteration detectable to the degree required by the threat. Redaction, legal hold, and privacy obligations can conflict with indefinite retention, so store references or minimized fields where possible and verify the governing requirement rather than promising that nothing can ever change.

Audit ingestion must have a failure contract. Security-sensitive actions may fail closed when no durable audit acceptance is possible. Less critical paths may write to a bounded local or regional buffer and reconcile later. Silent drops are not an option. Track rejected events, duplicate rate, producer lag, oldest unindexed event, query latency, export volume, retention execution, and all privileged access to the audit system.

Audit is not authorization. A perfect record of a cross-tenant export is still evidence of a failure.

Put pressure on the boundary

Return to the contractor at 14:05. A defensible answer now has observable state: authoritative membership version, session grant and expiry, invalidation lag, document version, rejected offline operation ids, queued notification state, export authorization checkpoints, search-index policy version, and audit events for the revocation and denials.

Then change one condition at a time:

1. The policy service is unavailable. Private reads and mutations fail closed after their valid cached grant expires. Existing sessions may remain read-only for a short, stated window if they reveal no new data. The admin surface shows that revocation propagation is impaired.
2. A tenant floods exports and reminders. Per-tenant admission limits and separate bulk queues preserve interactive collaboration for other tenants. Optional notifications can be delayed; permission checks cannot be skipped.
3. A group change affects a million documents. Version the group or policy, invalidate coarse grants promptly, repair indexes asynchronously, and reauthorize returned candidates until repair catches up.
4. The stale collaboration server accepts an edit. The authoritative write boundary rejects its expired grant or old policy version. If a forbidden mutation committed, quarantine further propagation, preserve evidence, repair document state deliberately, and identify every derived effect.
5. A regulated tenant moves to a dedicated cell. Isolate replay from live work, fence the routing switch, verify every store and index, and keep support and audit paths tenant-scoped through the move.

Good observability names the broken promise. Measure revocation propagation, open-session closure, denied stale grants, cross-tenant access attempts, authorization latency, cache age, search-repair lag, notification suppression, export release denial, noisy-tenant throttling, audit-ingestion lag, and time to contain a tenant-specific incident. Global CPU does not answer whether one customer’s former contractor can still edit.

Transfer the design without copying it

For a collaborative document editor, center the discussion on concurrent operations, snapshots, offline work, session revalidation, comments, presence, search, and revocation. The difficult promise is convergence among actors who remain authorized while authority itself can change.

For a calendar, center it on organizer and participant state, recurrence, timezone, delegation, free/busy privacy, idempotent invitations, reminders, and removal. The difficult promise is coordination without turning shared time into shared private detail.

For a permissions service, center it on resource decisions, group and policy versions, hierarchy, decision rate, cache soundness, denial behavior, and revocation latency. The difficult promise is a predictable answer to “may this actor do this now?” under constant policy change.

For an identity platform, center it on federation, credentials, sessions, provisioning, deprovisioning, tenant membership, and emergency access. The difficult promise is fresh identity state without confusing authentication with application authority.

For a multi-tenant SaaS platform, center it on trusted tenant routing, pooled and dedicated cells, quotas, noisy-neighbor containment, tenant-aware observability, export and deletion, and migration. The difficult promise is that shared infrastructure does not create shared fate beyond the contract.

For an audit-log platform, center it on authenticated producers, minimized events, durable acceptance, protected retention, tenant-scoped query and export, integrity evidence, and ingestion lag. The difficult promise is that a later investigation can reconstruct a sensitive decision without the evidence becoming another disclosure.

The architecture changes across these prompts. The discipline does not: bind each effect to a tenant, actor, resource, action, context, and time; then follow that boundary wherever the effect travels.

Rehearse the five-minute gap

Draw only the membership record, session grant, collaboration service, document store, search index, notification queue, export job, and audit stream. At 14:03, revoke the contractor. At 14:05, reconnect the offline client.

Explain which component holds authoritative state, how connected sessions hear about the change, where a current decision is required, what work may finish, and what evidence remains. Be precise about the user’s rejected edits and about data already delivered before revocation.

Then make two changes. First, let the policy service fail during propagation. Second, let the workspace contain ten million documents and an export already be halfway through. A credible design will not answer both variations with “invalidate the cache.” It will name freshness deadlines, effect boundaries, partial artifacts, containment, and recovery.

The useful closing question is not whether every component has a tenant id. It is whether the system can explain, at 14:05, why each surviving effect is still authorized.

Related links

• Security and Privacy
• Consistency, Idempotency, and Transactions
• Security, Abuse, and Privacy
• Observability and Operability
• Data and Workflow Platforms