Case Study: Multi-Region SaaS and Tenant Isolation

The Contract Has a Geography

The collaboration product began in one North American region. Its ordinary architecture had served it well: one primary database cluster, object storage for attachments, a search index, caches, background queues, a warehouse feed, and a support console. Small customers shared tables keyed by tenant_id. Backups restored cleanly. The on-call team knew the system.

Then a European public-sector buyer put three promises into the same contract review. Its users needed lower interactive latency. Customer content, attachments, audit records, support material, and backups had to remain inside an approved geography. If the home region failed, service had to return within four hours with no more than five minutes of acknowledged work at risk.

Sales summarized the request as “multi-region.” That phrase concealed three different jobs: move the request path nearer to users, control where every copy of tenant data may exist, and recover from a regional loss. None requires every region to accept writes. Treating them as one active-active database project would add conflict resolution before the team had even decided what belonged to the tenant.

The design therefore begins with one placement record, not a cloud map. The buyer receives a home region, an isolation cell, a set of permitted replica destinations, a residency boundary, and named recovery objectives. That record becomes an input to routing, jobs, exports, support tools, deletion, and backups. The architecture is global because it can place and operate tenants deliberately, not because every data store spans the globe.

Follow One Document Until the Boundary Is Honest

The buyer’s placement record lists more than a database region. A tenant owns workspaces, users, permissions, documents, comments, audit events, integration tokens, webhook state, files, search documents, and product configuration. It also participates in global identity, billing, plan catalogs, abuse controls, feature rollout, operational telemetry, and support workflows.

Take one restricted document through the existing system. Its authoritative row lives in the application database. Its body is also present in an object version, a search index, a cache, perhaps a queue payload, and a backup. Its title may appear in an application log or trace. A support engineer may paste an excerpt into a case. A warehouse export may carry its workspace or author identifier. Moving only the authoritative row leaves most of the promise behind.

The team classifies each copy by its job. Tenant-owned operational data—documents, permissions, workflow state, attachments, audit events, and secrets—stays under the tenant’s home-region write authority. Global control data carries the minimum facts needed to coordinate the business, such as tenant ID, billing reference, plan, domain ownership, and feature eligibility. Derived data includes search, caches, projections, and reporting copies; being rebuildable does not exempt it from access and residency rules. Operational evidence includes logs, traces, support notes, screenshots, exports, and backups, the copies most likely to escape an architecture diagram.

This tracing exercise changes the design. The European placement record now governs database and object storage, but also search collections, queue execution, log sinks, backup and restore staging, warehouse exports, support access, and deletion propagation. Analytics may receive approved aggregates with identifiers transformed according to contract and policy. Billing may hold the account reference and invoice state without receiving document content or unrestricted activity. Support access from outside the boundary requires a separate approved and audited path.

Residency is therefore not a storage setting or a conclusion drawn from a region name. It is a claim about storage, processing, inspection, replication, recovery, and export that the team must be able to test. The useful audit question is always the same: where can this tenant be found now? New pipelines, debug logs, and temporary restore locations deserve the question as much as the primary database.

Give the Tenant a Cell

The original shared tables remain suitable for ordinary commercial tenants. They are efficient and familiar, provided authorization is enforced on every path, indexes and cache keys include tenant identity, cross-tenant tests are adversarial, and the data can be extracted without improvised surgery.

The restricted buyer receives a cell: a bounded, repeatable unit containing an application database shard, object-storage namespace, search collection, queue and scheduler workers, cache namespace, secrets, and operational dashboards. The cell is not synonymous with a database. Its boundary follows everything the team may need to restore, move, throttle, delete, encrypt, or inspect together.

That choice buys more than contractual separation. A noisy reindex, bad migration, restore, or runaway tenant affects a known population. Capacity and error budgets can be observed per cell. New restricted tenants can be placed into another instance of a pattern the operators already know.

A dedicated database would narrow the storage blast radius further, but it would not automatically isolate a shared job queue, search index, support console, or backup pipeline. The team reserves that additional burden for a tenant whose size, restore granularity, contract, or operational risk earns it. Isolation follows the required operation; it does not improve merely because the database count increased.

Keep One Writer Until a Fact Earns More

Under normal operation, the buyer’s home region accepts writes for its tenant-owned state. Other regions may terminate sessions, serve static assets, route traffic, cache non-sensitive material, or answer reads whose staleness budget allows a replica. They do not independently update the same permissions, documents, workflow transitions, audit sequence, or deletion state.

Those facts resist casual merging. If two disconnected regions both change a permission, attach the same custom domain, send a webhook, or act on a deletion request, the team must define which result is valid and which external effects may be repeated. Active-active replication does not supply those business semantics.

Narrower facts can earn multiple writers. Presence may tolerate last-writer-wins. Region-local rate counters may merge into an approximate global view. Append-only telemetry with stable identities may reconcile safely. A collaborative editor may use a purpose-built conflict model. Each exception names its invariant, merge rule, duplicate behavior, and repair path; it does not confer active-active semantics on the rest of the tenant.

Read replicas need the same precision. After changing a permission, the user must not be sent to a replica that still grants the old access. Sensitive flows can return to the home region, carry a freshness token, or pin subsequent reads until a replica reaches the required point. Search and reporting may lag if the interface exposes that state. “Local read” is not one consistency contract.

Rehearse the Region Loss in Time

The recovery promise becomes credible only when the team can narrate a failure without hand-waving. In the rehearsal scenario, normal traffic creates documents, attachment metadata, permission changes, audit events, webhook deliveries, and search updates inside the restricted cell. At 09:07 the home region becomes unreachable. The disaster-recovery copy has applied commits through 09:05.

The first action is not promotion. It is fencing. Routing rejects writes, schedulers stop, queue consumers lose their authority, and operators record the last accepted and last replicated positions. A stale region returning later must not be able to resume writes simply because its processes restarted.

At 09:14 the team can account for the two-minute uncertainty window. The database copy is inside the five-minute recovery-point objective, but that answer is incomplete. Operators also compare object uploads with metadata, identify webhook attempts whose provider outcome is unknown, check the audit sequence, and freeze warehouse exports. Search remains marked stale because it is a derived view, not evidence that recovery is complete.

The recovery lead then promotes the allowed in-boundary copy, records the new authority in the placement control plane, and routes a small verification cohort. Tenant counts, recent-write samples, permissions, attachment downloads, and audit continuity are checked before broad traffic returns. Queue work resumes from durable checkpoints with idempotency controls; uncertain webhooks are reconciled before retry; caches are discarded; search rebuilds from the recovered authority.

Service becomes customer-usable well inside the four-hour recovery-time objective, but degraded state remains visible until search, exports, and support tooling catch up. The measured recovery time ends when the promised user journeys work, not when the database accepts its first connection. The achieved recovery point is the latest authoritative state the team can prove, not the replica timestamp displayed on a dashboard.

This sequence also reveals the promise the system cannot make. If a tenant requires zero acknowledged-data loss and the design cannot synchronously preserve every relevant write inside an allowed failure domain, promotion is not honest. The safer product behavior may be write unavailability until authority is restored. RPO, RTO, and availability are constraints to reconcile, not slogans to stack.

Recovery Follows the Data Class

One target for the whole product would flatten important differences. Application records and attachment metadata must recover together; restoring one without the other produces broken objects. Audit events need strong continuity because they explain access and operational action. Billing entitlements may need a separate continuity path because they decide whether the tenant can use the product. Search can be rebuilt and may return later. Warehouse exports can wait if the delay and checkpoint are visible.

The recovery plan names an authority, objective, method, verifier, and owner for each class. Application state is checked with counts, hashes or other integrity evidence, recent-write samples, and representative flows. Attachment recovery checks object existence against metadata and exercises upload and download. Audit recovery reports gaps and verifies tamper evidence where the design provides it. Search recovery tests permissions as well as result presence. Export recovery proves its checkpoint and duplicate behavior.

The numbers belong to the customer and business promise. The proof belongs to engineering. A four-hour target without a start event, a customer-usable end state, and a rehearsed verifier is only a hopeful duration.

A Placement Change Is a Migration

The same design must handle a tenant that becomes restricted after years in the default region. The team first creates its target placement and cell, then snapshots authoritative records and large objects. Change capture closes the gap while search and other derived state rebuild in the target. The final cutover fences writes, drains or stops tenant-scoped jobs, validates counts, permissions, object references, audit continuity, and representative flows, then switches routing and worker ownership.

Rollback remains available until the target has proved stable and the old copy has a declared disposition. It may be removed, retained temporarily as an approved backup, or held for a policy-defined period, but it cannot remain an accidental second authority.

Again, byte copying is the easy part. Failed-job queues, webhook cursors, scheduled exports, object lifecycle rules, log sinks, support views, deletion requests, and backup catalogs must move or be consciously excluded. The placement record gives each subsystem one source for the answer instead of relying on a migration checklist that will become stale.

The Decision the Team Can Operate

The architecture record for the buyer is short enough to use. Its goal is local interactive service, contractual placement, and a four-hour recovery time with a five-minute recovery point. General active-active writes are explicitly outside scope. The tenant receives an EU home region and restricted cell; all normal writes occur there, and promotion is permitted only to a prepared copy inside the allowed boundary. Read replicas may serve flows with declared staleness budgets, while permission and other read-your-writes paths return to authority.

The same record names the minimal global facts, the destinations allowed for each derived copy, outside-region support restrictions, backup and restore locations, the cell owner, and the authority that may declare failover. Exit requires a placement audit, a tenant-move rollback rehearsal, and a live regional-failure drill that meets both objectives. The record has a revisit trigger too: measured latency, tenant size, restore behavior, a changed contract, or an invariant that genuinely needs independent regional writes.

This is less visually impressive than a diagram full of bidirectional arrows. It is also much easier to reason about during an incident. The design makes the smallest distribution move that satisfies the actual pressure and leaves room to add cells or regions when evidence demands them.

Case Review

Use the buyer’s request without naming a product. Write the placement decision in ordinary prose: which facts belong to the tenant, which minimal facts remain global, where each derived copy may go, what the cell contains, who holds normal write authority, which reads may be stale, and what support and backup paths the boundary permits.

Then vary the rehearsal. Suppose the old region returns while the promoted region is accepting writes, an object upload completed without its database metadata, and one webhook attempt has an unknown outcome. Decide how fencing is proved, which side remains authoritative, how the attachment is reconciled, whether the webhook may be retried, and what evidence establishes the achieved recovery point. If the plan cannot answer those questions, the contract is ahead of the architecture.

Multi-region SaaS is not the act of spreading infrastructure. It is the discipline of placing tenants, limiting authority, tracing every copy, and proving recovery under the same boundary the customer was promised. Only the facts that can survive independent writers should receive them. Everything else needs an owner, a fence, and a rehearsed way home.