Multi-Tenancy, Identity, Authorization, and Data Isolation

The Import That Found the Tenant Boundary

A collaboration product stores every customer in shared tables. Each workspace row has tenant_id; every ordinary API query supplies it. For two years, that looks like a tenancy model.

Then a large customer imports ten years of documents. Its jobs occupy most worker slots and drive a hot index range. While other customers wait, a support engineer uses an old troubleshooting console that can search by email without first selecting a tenant. The customer also asks the team to demonstrate recovery of one accidentally deleted workspace. The only rehearsed procedure restores the whole database.

Three different failures have found the same missing design. The product can label a row, but it cannot consistently contain a tenant’s authority, resource use, or recovery. Moving the customer to a separate database might reduce some contention. It would not repair the support console, discover the tenant’s files in object storage, or make background jobs obey suspension.

A tenant boundary is the set of properties the system can keep true for one customer while other customers share some of the machinery. The required properties usually include:

• facts cannot be read, changed, inferred, or exported across the boundary without explicit authority;
• one tenant’s work cannot consume more shared capacity than policy allows;
• placement, migration, suspension, export, deletion, and recovery can be scoped to the tenant;
• operators can reconstruct where the tenant’s facts went and who accessed them.

The physical model—shared tables, schemas, databases, clusters, or cells—changes how those properties are enforced. It does not replace them.

One Tenant Contract, Two Planes

Before changing storage, decide what a tenant is. The collaboration product may call the enterprise a customer, billing may call it an account, identity may call each workspace an organization, and storage may use a project ID. If those grains disagree, a valid identity can be routed to the wrong facts even when every query contains a tenant column.

Write the tenant contract in operational terms. Give the tenant a stable internal ID and define parent and child boundaries. Name which unit owns access policy, billing, residency, retention, support approval, export, deletion, recovery, and incident notification. Shared objects need a deliberate sharing model; they cannot be quiet exceptions to tenant scoping.

The contract has a control plane and a data plane.

The control plane is a tenant registry. For each tenant, it records lifecycle state, parentage, placement, isolation tier, residency and retention policy, limits, support-access rules, backup coverage, restore route, operating owner, and policy version. Routers, job schedulers, provisioning, authorization, data movement, and support tooling should use that authority rather than reconstructing placement or policy from billing flags and local configuration.

The data plane carries tenant-owned facts through tables, indexes, caches, files, events, search documents, analytical rows, and exports. Each path needs enough tenant context to enforce the contract. The registry says that tenant t_123 is suspended and placed in cell us-east-2a; the data path must prevent a queued job with an old payload from continuing writes somewhere else.

Treat changes to the registry as production changes. A move, suspension, residency update, or isolation-tier change needs a version, an audit trail, and behavior for components that still hold the previous value. Otherwise the control plane can disagree with the paths it is meant to control.

Carry Context Before Reading Data

A request should resolve tenant, actor, placement, policy version, and applicable limits before it reads tenant-owned data. That context must remain attached as the request crosses service interfaces. Chapter 25 established that an owner must control the meaning exposed through an interface; in a multi-tenant system, the interface must also preserve who is entitled to ask and which boundary constrains the answer.

In shared tables, tenant identity belongs wherever identity or access is tenant-scoped. A document slug may require (tenant_id, workspace_id, slug) rather than a global constraint. An object ID may be globally unique, but a lookup should still include tenant scope so the system does not fetch first and reconstruct authority afterward. Secondary indexes should begin with the tenant keys used by common access paths unless workload evidence calls for a different partitioning strategy. Cache keys, object paths, search documents, event partitioning, dead-letter records, and export manifests need the same scrutiny.

Background work is a frequent escape route. A job should carry an immutable tenant ID and resolve current placement and lifecycle policy before execution. It should stop safely when placement cannot be resolved or the tenant is suspended. Events should identify the tenant and the ordering scope consumers must preserve. Reindexers, migration scripts, repair tools, and analytical transformations are data paths, not exemptions from the product boundary.

The collaboration product’s import therefore enters through a tenant-aware scheduler. Its work items name the tenant and import operation, while the scheduler resolves current cell, policy version, and remaining budget. Moving the tenant does not require trusting a stale database address embedded in millions of queued jobs.

Choose Separation by the Failure You Need to Contain

Shared tables concentrate enforcement. They are efficient when many small tenants have similar requirements, but a missing predicate, broad credential, unscoped cache key, or full-system restore procedure can affect everyone. They demand strong query construction, database guardrails, workload controls, and an explicit tenant extraction and repair path.

Separate schemas create namespaces within one database environment. They can make tenant-specific enumeration, migration, and export clearer, while introducing schema-count limits, migration fan-out, metadata overhead, connection handling, and many nearly identical objects for tooling to manage.

Separate databases create clearer credential, backup, restore, and migration surfaces. Ask what remains shared: the instance, administrator role, connection pool, storage volume, maintenance window, automation, and support console may still couple failure and authority. Cross-tenant reporting also becomes an intentional data-movement path.

Separate clusters or accounts can contain more security, performance, and operational failure. They also multiply upgrades, monitoring, backup verification, capacity planning, incident response, and idle cost. Dedicated infrastructure is a product promise only when the organization can operate its variance.

Cells group tenants into bounded shared environments. A cell limits blast radius and offers a route between one large shared system and an environment per tenant. It works only when placement is explicit and tenants can move. Routing, jobs, streams, caches, search, analytics, backup lineage, and audit continuity all have to survive the move.

Choose the least costly model that can contain the failures in the tenant contract. A regulated customer may require a dedicated account; a large but ordinary customer may fit a cell with stricter limits; thousands of small customers may fit shared tables. Mixing tiers is reasonable if the control plane represents the difference and every operational path can honor it.

Authorization Must Survive the Unusual Path

The old support console reveals why a tenant predicate is insufficient. Authorization must answer which tenant, which actor, which object, which action, and which purpose. A support operator who may inspect one workspace under an approved case does not gain authority to search every customer by email. A migration job may read more objects than a user, but only for a named migration and only while its approval is valid.

Enforce these decisions at more than the polished API edge. Service methods, database access, caches, search filters, exports, analytics, repair scripts, and support tools should receive scoped credentials or policy decisions. High-harm operations should fail closed: an export with no tenant context does not run; a job with unknown placement goes to repair; support access without a reason and approval is denied before data retrieval.

Row-level security can add a valuable database-side guardrail in shared tables. Its strength depends on the surrounding system: connection pools must set and clear tenant context safely, ordinary application roles must not bypass policy, migrations need negative tests, privileged roles need narrow use and logging, and policy changes need query-plan and behavior review. A policy that production code can silently evade is decoration.

The console repair is therefore larger than adding a search filter. The operator first selects an authorized tenant and support case. The console obtains time-limited access for the allowed purpose and object scope. Search enforces the tenant filter itself, and the audit record captures actor, tenant, purpose, approval, policy version, path, and outcome without copying sensitive content into general logs.

Capacity Is Part of Isolation

The import is not generic slowness. One tenant is converting a shared resource into delay for others. The boundary must extend through API concurrency, database work, worker slots, queues, cache occupancy, search indexing, storage growth, file operations, exports, and backfills.

Set budgets at the place where work becomes expensive. Entry-point rate limits cannot stop one accepted request from starting a huge scan. The product may need per-tenant request concurrency, query time or rows scanned, import and export throughput, worker concurrency, storage growth, and reindex limits. Interactive work can receive priority over bulk jobs, and paid tiers can receive different budgets, provided the policy is explicit rather than improvised during an incident.

The scheduler now admits only a bounded number of this customer’s import tasks. Database governors stop pathological queries; workers reserve capacity for interactive traffic; the import pauses when latency, lock waits, replication lag, or queue age cross declared thresholds. The tenant can buy more headroom, but it cannot unknowingly take everyone else’s.

Operators need tenant-attributed load and denials to enforce this. Avoid placing raw customer names or unbounded tenant identifiers on every metric. Tier aggregates, top-N views, sampling, protected drill-downs, and stable internal IDs can preserve operational visibility without turning the metrics system into a sensitive, uncontrolled index.

Restore the Tenant, Not Just the Database

The customer’s restore request asks a narrower question than disaster recovery: can the team recover tenant t_123 without rewinding everyone else or erasing valid changes made after the accident?

For shared tables, the answer commonly begins with a point-in-time backup restored into an isolated environment. The team extracts the tenant’s authoritative rows in dependency order, including join records and child objects. It compares them with current production, then applies the selected facts through idempotent repair paths or controlled patches that recheck invariants. Derived search documents, caches, and projections are rebuilt from authoritative data where possible.

The database is only one item in the recovery inventory. Object storage, streams, analytical snapshots, exports, support copies, and third-party deliveries may contain tenant facts. Some are rebuildable; some have already crossed an external boundary; some are retained under an operating or policy requirement. The restore record must say what was recovered, rebuilt, left unchanged, or could not be recalled, and who approved those decisions.

Separate databases make the first extraction easier, not the whole recovery automatic. A tenant can still share files, indexes, queues, analytics, credentials, and operator tools. Restore granularity is a property of the end-to-end data path.

Rehearse this route and give it its own recovery objective. A full-system recovery time does not imply a single-tenant repair time. The rehearsal for the collaboration product should discover foreign-key ownership, derived copies, conflicting new writes, and irreversible exports before an incident does.

Lifecycle Operations Exercise the Same Boundary

Provisioning creates more than a row: it assigns placement, keys or namespace, default roles, limits, residency policy, backup coverage, audit configuration, and operating ownership. Suspension defines whether reads continue, writes stop, queued jobs pause, integrations run, exports remain available, retention clocks advance, and support access changes.

A tenant move changes routing while work may still be in flight. The move needs source and destination identity, a copy or replay boundary, comparison evidence, a cutover condition, rollback, cache handling, job behavior, and one declared authority for new writes. Dedicated placement is safe only after these mechanics exist; otherwise it relocates the ambiguity.

Export needs a tenant-scoped snapshot, manifest, lineage, approval, encrypted destination, expiry, and deletion behavior. Deletion must distinguish authoritative facts, rebuildable derived facts, externally delivered copies, logs retained for system integrity, and backups governed by retention policy. The architecture should describe the technical route precisely and leave legal interpretation to the applicable policy owners.

Incidents then become answerable. For a suspected leak, operators can identify the path, actor, affected tenants, objects, policy version, derived copies, and revocation action. For a noisy neighbor, they can see admitted work, capacity consumed, limit decisions, and customer impact. For a failed move, they can identify where new writes landed and which checkpoint or route version governs rollback.

Write the Isolation Plan

Record the enforceable design before approving storage topology:

Tenant grain and stable identity:
Parent, child, and explicit sharing rules:
Lifecycle states and control-plane owner:

Placement model and isolation tier:
Shared substrate and failures it can transmit:
Tenant move boundary, cutover, and rollback:

Keys, constraints, indexes, caches, files, events, and derived copies:
Tenant, actor, object, action, and purpose enforcement points:
Privileged access, negative tests, and audit evidence:

Per-tenant budgets and admission points:
Interactive versus bulk-work policy:
Tenant-attributed telemetry and safe drill-down:

Authoritative-data inventory:
Single-tenant restore objective, procedure, and last rehearsal:
Rebuildable, retained, and externally delivered copies:

Provision, suspend, export, delete, and incident behavior:
Named owners, approvals, and unresolved limits:

Apply the plan to the collaboration product before deciding where the enterprise tenant should live. Disturb it with the import, the support search, a queued job after suspension, an object shared across workspaces, an accidental deletion followed by valid new writes, and a move that fails halfway through. The answers should identify an enforcement point, an owner, and recovery evidence—not merely another component to add.

The tenant boundary is real when ordinary requests, unusual tools, heavy work, recovery, and lifecycle changes all preserve it. Chapter 27 follows the hardest extension of that promise: keeping the boundary intact when CDC, outbox relays, backfills, exports, and replay carry tenant facts away from their first write.