Service Topology and Critical-Path Design

Part III treated queues, caches, protocols, admission, retries, partitions, replicas, consistency, and interfaces as mechanisms with bounded contracts. Distributed architecture asks a different question: where should those mechanisms live, and which of their interactions must a user wait for?

A network boundary is not a decomposition by itself. It is a new contract for latency, partial failure, version skew, authentication, cancellation, retries, observability, capacity, and ownership. It may be exactly the right price for independent scaling or fault isolation. It may also turn one in-process invariant into a distributed transaction and one function call into a failure-amplifying dependency.

Model a successful operation as a dependency graph (G=(V,E)). Nodes perform work or own state; edges transfer calls, messages, configuration, or data. The user-visible latency is governed by the longest required chain after parallel completion rules, not by the number of boxes. The operation’s availability is constrained by every mandatory node and edge, including shared infrastructure that the main diagram omitted. Recovery adds a second graph: discovery, credentials, configuration, placement, state catch-up, caches, and traffic movement.

The design task is therefore not “monolith or microservices?” It is: which boundary provides enough ownership, state integrity, independent scaling, failure containment, security, or change autonomy to justify the extra graph? Create a network boundary only when ownership, isolation, or independent evolution is worth the added latency, failure modes, observability burden, and coordination cost.

Topology has several boundaries, and they need not coincide

Architecture discussions often overload service. Separate these dimensions:

• Code boundary: module, package, crate, library, or component with an enforced API.
• Repository boundary: source, review, and release-history ownership.
• Build boundary: independently compiled or packaged artifact.
• Deployment boundary: independently released and rolled back unit.
• Runtime boundary: process, address space, container, host, or execution pool.
• Network boundary: serialized interaction subject to partial failure.
• State boundary: authority, transaction, consistency, retention, and migration ownership.
• Failure boundary: components expected to fail or degrade together for a named cause.
• Team boundary: people accountable for design, operation, and change.
• Tenant/security boundary: trust, policy, keys, quotas, and data-isolation scope.

A monolith usually means one deployment unit, not necessarily one module, process, database, or team. A modular monolith enforces internal ownership and dependency direction while retaining local calls and transactions. A service adds at least a runtime/network contract and commonly an independent deployment owner. A cell repeats a bounded slice of the serving stack for a tenant or shard population so one cell’s overload, bad deployment, or state fault has a limited scope. A control/data-plane split separates changes to desired configuration from the high-volume path that uses already-provisioned state.

These forms compose. A cell can contain a modular monolith, several services, or both. A service can own several modules. Several services can still share one database, credential authority, queue, or deployment controller and therefore one failure boundary. Draw the dimensions that affect the decision; do not infer them from box count.

A boundary earns its cost on explicit axes

Code size is weak evidence for a network split. Evaluate the candidate on six axes:

The Azure boundary guidance recommends validating domain boundaries against chatty calls, independent deployment, scaling, consistency, and pragmatic consolidation. That is useful scoped guidance, not a rule that every bounded context must be a process. Begin coarse when the domain is uncertain. A local module boundary can preserve a future extraction point without paying network cost today.

Make each claimed benefit falsifiable. “Independent scaling” needs demand and resource curves showing that scaling the whole unit wastes material supply or prevents an objective. “Failure isolation” needs a fault test showing the other side remains useful. “Team autonomy” needs release records showing fewer coordinated changes, not a diagram with team names.

The critical path is a completion rule over a graph

Mercury checkout has this modeled path:

user
  -> edge                 8 ms
  -> checkout API         7 ms
  -> identity             9 ms
  -> cart                14 ms
  -> parallel:
       pricing           18 ms   required
       inventory         22 ms   required
       recommendations   38 ms   optional by product contract
  -> order commit        24 ms
  -> response             5 ms

When the aggregator waits for every branch, the modeled no-queue path is:

[ 8+7+9+14+\max(18,22,38)+24+5=105\ \text{ms} ]

The maximum is correct only because the three branches run in parallel and the response waits for all three. Adding their latencies would overstate that stage. Adding component p99s would not produce the end-to-end p99 either; correlation, conditional routing, queueing, and distribution shape matter.

Recommendations are useful but not part of the declared success rule: a correct checkout needs authenticated identity, priced cart, reserved inventory, and durable order identity. Mercury removes recommendation from the mandatory response. It emits a bounded asynchronous signal after commit and lets the confirmation screen retrieve recommendations independently. The path becomes:

[ 8+7+9+14+\max(18,22)+24+5=89\ \text{ms} ]

The model saves 16 ms and removes recommendation availability from checkout completion. This is not “make everything asynchronous.” Inventory remains synchronous because acknowledging an order without its capacity decision would change correctness. Pricing remains required because the accepted amount is part of the order contract. The architecture follows the success definition.

Applied decision: remove one synchronous dependency completely

Moving recommendation behind a queue is insufficient if checkout still waits for enqueue through a shared broker whose overload blocks the transaction. Mercury uses a local transactional outbox entry written with the order. The response depends on the order store, not on the broker. A publisher later delivers the event. If recommendation is delayed, checkout is complete and the confirmation screen renders without it.

The change needs evidence:

• traces show recommendation spans no longer parent the checkout response;
• fault injection blackholes recommendation and its broker while valid checkout goodput remains within objective;
• outbox backlog age and publication retry are visible;
• the optional UI has a missing/stale state rather than an endless spinner;
• cancellation of the checkout request does not delete a committed order; and
• recommendation consumers use the order/event identity idempotently.

The dependency is pruned only when no synchronous lookup, service discovery, configuration fetch, token exchange, logging sink, or queue admission reintroduces it.

Synchronous depth spends latency and failure budget

Every synchronous hop adds some combination of serialization, queueing, connection acquisition, proxying, network transit, remote scheduling, execution, response transfer, and retry. It also creates outcomes that local calls do not normally expose: timeout with unknown remote state, caller cancellation while callee continues, incompatible versions, stale discovery, authentication failure, partition, and retry duplication.

Set a dependency budget per operation:

Budgets are hypotheses, not entitlement. Trace actual serial depth and remaining deadline at every edge. A deep call graph often hides because each team sees only its immediate dependencies. Preserve a logical-operation ID and parent/linked trace context across sync and async edges, but remember sampling can omit the exact slow paths. Combine traces with server-side call counts and dependency inventories.

Avoid cycles on a request path. Service A calling B, which calls A through a different route, can deadlock pools, multiply retries, and make ownership unknowable. Google SRE’s cascading-failure guidance warns about intra-layer communication, shared-resource exhaustion, retry amplification, cancellation, and “noncritical” backends that blackhole. Dependency direction should move toward authority or durable handoff, with redirection handled explicitly rather than arbitrary peer proxy chains.

Fan-out trades elapsed time for tail and resource exposure

Parallel fan-out is not free speedup. If a query contacts (n) shards and waits for all, its completion time is the maximum branch latency. More branches increase the chance that at least one exceeds the deadline and multiply connections, buffers, CPU, and retry work.

The fixture gives each of 24 shards a modeled 99.8% probability of meeting the branch deadline. Under an explicitly independent teaching approximation:

[ P(\text{all 24 meet})=0.998^{24}\approx0.9531 ]

About 95.31% is not a forecast. Real shards share hosts, networks, deployments, query shapes, and hot keys, so failures are correlated. The calculation demonstrates why a good per-shard number can still produce a poor all-shard outcome.

Choose a completion rule from correctness:

• all: required for exact distributed predicates or complete results; invest in pruning, co-location, partitioning, and strict branch budgets;
• quorum or threshold: valid only when the result semantics define how partial participants establish correctness;
• first valid: useful for equivalent replicas or hedged reads with cancellation and duplicate-work budgets;
• best effort until deadline: valid for explicitly partial search, enrichment, or analytics with coverage metadata;
• hierarchical aggregation: limits connection and merge concentration by combining within topology levels.

Bound fan-out cardinality at the API and partition planner. Propagate one deadline; reserve merge and response time rather than giving every branch the full caller budget. Cancel unnecessary branches, but assume cancellation is advisory and account for residual work. Retry the missing branch only if the operation identity and remaining budget make it safe.

Shared infrastructure defines the real failure graph

Separating application processes does not isolate them when they depend on the same:

• database cluster, schema, transaction log, connection pool, or lock authority;
• cache fleet, queue partition, object store, DNS, service registry, or secrets service;
• identity provider, certificate authority, policy engine, time source, or configuration store;
• cluster scheduler, ingress, service mesh control plane, deployment system, or quota;
• network link, NAT port pool, egress gateway, filesystem, host, zone, or region; or
• operator workflow, library rollout, credential, and emergency control.

Draw hidden dependencies beside the business call graph. State whether they are on the steady data path, change path, recovery path, or observation path. A telemetry sink should not block serving. A feature flag lookup should not require a remote call on every request when a bounded local snapshot is acceptable. A certificate or policy cache needs expiry and fail behavior; “cached” does not mean independent forever.

Availability products such as (A_1A_2A_3) can teach that mandatory dependencies consume budget under independence. Do not present them as measured reliability when common causes dominate. Use fault injection, incident histories, correlated change records, and recovery exercises to discover the actual graph.

Control planes should not be surprise data-plane dependencies

A control plane changes desired state: placement, routes, policy, schemas, certificates, limits, or configuration. A data plane performs the system’s recurring useful work from applied state. The distinction is about job and failure behavior, not necessarily a product or machine.

The AWS Builders’ Library article on static stability describes a transferable principle: existing data-plane work can continue from local applied state during a control-plane impairment, even when new changes cannot propagate. That is usually safer than making each request synchronously retrieve current configuration.

For Mercury:

• the control plane assigns tenants to cells, distributes routing epochs, rolls policy, and provisions capacity;
• the data plane authenticates against locally usable trust material, routes by an applied cell map, serves checkout, and records order state;
• loss of cell-assignment updates prevents safe new moves but does not stop existing assignments;
• expired credentials or policies have explicit grace and fail behavior rather than an unlimited stale mode;
• recovery does not assume the global control plane can immediately create replacements while the environment is impaired.

Static stability is scoped. A revoked credential cannot remain valid forever. A newly discovered safety fault may require halting old configuration. Record maximum configuration age, revocation channel, local fallback, reconciliation after recovery, and which changes are safety-critical.

Keep control-plane load away from data-plane overload. A million failing requests must not trigger a million route recomputations, certificate renewals, or autoscaling writes. Conversely, a configuration storm should not consume the CPU, memory, or connections reserved for serving.

Cells and bulkheads limit scope only when dependencies and spillover obey the boundary

A cell is a repeated serving unit assigned a bounded tenant, key, or traffic population. It commonly includes compute, queues, caches, and state partitions sized together. A router maps requests to a cell using durable assignment. The cell provides a unit for capacity, rollout, evacuation, and incident containment.

global change plane
  tenant -> cell assignment, software/config release, capacity policy
              |
       applied routing epoch 57
              v
cell router (no per-request control-plane lookup)
  |--------------------|--------------------|
  v                    v                    v
cell A                 cell B               cell C
tenants 0-1249         tenants 1250-2499    tenants 2500-3749
API + queue            API + queue          API + queue
cache + authority      cache + authority    cache + authority
own limits/pools       own limits/pools     own limits/pools

The fixture models eight equally assigned cells, so a single-cell tenant population is 12.5%. But all eight synchronously call one global identity service. If that service fails and checkout cannot use a bounded verified local context, the effective checkout blast radius is 100%, not 12.5%. Cell diagrams must include global critical dependencies.

AWS Well-Architected bulkhead guidance describes cells as isolated workload instances. The transferable design questions are concrete:

• What demand and state choose a cell, and can one tenant dominate it?
• Which resources are physically or logically separate: queues, pools, state, quotas, credentials, deployments, and operators?
• Does a bad binary or global configuration roll to every cell simultaneously?
• Can an unhealthy cell spill traffic into healthy cells and overload them?
• Is evacuation capacity reserved, and is movement fenced by an assignment epoch?
• Which global services remain mandatory, and what is their failure behavior?
• Can one cell be restored, backfilled, or rolled back without global control?

Bulkheads can exist without full cells. Separate worker pools, connection pools, queues, cache partitions, rate limits, and schedulers by traffic class or tenant. The boundary is useful only if resources cannot be silently borrowed until every compartment fails. Borrowing needs a ceiling and a stop rule. During an overload, fail the affected cell or optional class cheaply rather than turning isolation into global retry traffic.

Co-location, sidecars, and proxies move cost; they do not erase it

Co-locating two components can reduce network transit and improve cache or state locality, but it couples scheduling, resource contention, deployment, and host failure. A sidecar places a helper process beside an application instance. A node proxy amortizes helper cost across workloads but enlarges its sharing boundary. A remote gateway centralizes policy and connections but adds another network path and possible concentration point.

Service meshes can standardize identity, telemetry, routing, retries, and policy. Istio’s architecture documentation explicitly places proxies on the data path and the control plane on configuration. Its performance guidance publishes workload-specific measurements rather than claiming zero overhead. Envoy’s threading documentation separates configuration work on a main thread from traffic handled by workers in one implementation.

Model the actual path. The fixture assumes five network hops, two proxies per hop, and 0.7 ms per proxy traversal:

[ 5\times2\times0.7\ \text{ms}=7\ \text{ms} ]

This 7 ms is teaching data, not an Istio or Envoy claim. Real cost depends on protocol, payload, connection reuse, encryption, filters, telemetry, CPU limits, topology, and tail state. Measure application-only and full-path distributions with the same workload and correctness checks.

Audit proxy behavior as architecture:

• who owns retry, deadline, hedging, circuit-breaking, and outlier ejection policy;
• whether retries preserve operation identity and respect admission budgets;
• how many connection pools, buffers, and queues exist per hop;
• what happens when configuration is stale, rejected, or partially applied;
• whether a proxy failure restarts or drains the application;
• whether telemetry cardinality or log flushing competes with requests; and
• whether policy creates asymmetric paths that traces or diagrams hide.

“The mesh handles it” is not a completion or failure contract.

Data ownership prevents accidental distributed joins

A service boundary is incomplete when callers bypass it to query the service’s tables. Shared database schemas let a local join look fast while coupling migrations, locks, resource demand, and correctness across owners. Conversely, forbidding all derived data can force expensive runtime joins across services.

State the authority for each fact. Other services may hold:

• an immutable identifier and fetch current state on a required path;
• a bounded cached representation with version/freshness evidence;
• a purpose-built derived view updated through durable events;
• a snapshot for one workflow with explicit reconciliation; or
• no copy, when policy or correctness requires the authority.

Suppose checkout needs customer display name, risk class, and shipping eligibility. A runtime join among identity, risk, and logistics on every request creates three dependencies and mixed-version semantics. If display name is optional, omit or cache it. If risk approval is a required decision, expose a versioned approval result owned by risk. If shipping eligibility changes slowly and has a safe freshness bound, maintain a derived checkout view with source position and fallback. Do not copy authoritative payment credentials into the view because locality is convenient.

Azure’s microservices data guidance highlights private state ownership, consistency challenges, and chatty boundaries. The important transfer limit is that private stores alone do not solve end-to-end invariants. Chapter 28’s transaction, saga, rights, and process choices still apply. Architecture must name the cross-owner state machine rather than hiding it in an orchestrator.

Dependency inversion and graceful isolation keep optional work optional

Dependency inversion means core policy depends on a stable capability contract, while adapters depend on external implementations. In a distributed system it also provides a seam for local fallback, recording intent, simulation, and consolidation. It does not mean adding an interface to every function.

Classify dependencies by success semantics:

Graceful degradation preserves a named success rule while removing optional work or reducing quality. Returning an order without recommendation is graceful. Returning an order without inventory authority is corruption. Serving a bounded stale catalog may be graceful for browsing but unsafe for final price acceptance. Test blackholes as well as fast failures; a dependency that returns an error in 2 ms can look optional while one that consumes the full 30-second timeout exhausts every worker.

Architecture entropy is measurable drift between intended and actual topology

Topology changes through new calls, libraries, shared stores, feature flags, queues, proxies, migrations, and emergency paths. A diagram from design review becomes fiction unless runtime evidence checks it.

Maintain a topology ledger with:

• service/module owner, repository, deployment, runtime and state authority;
• inbound/outbound edges by protocol, sync/async form, operation and completion rule;
• observed call rate, fan-out, payload, latency, errors, retries, deadline and cancellation;
• queues, connection pools, rate limits, credentials, discovery and configuration dependencies;
• tenant, zone, region, cell and control/data-plane placement;
• critical, optional, recovery and migration path classifications;
• compatibility versions and last consumers of deprecated edges; and
• evidence source and as-of time.

Compare declared and observed graphs. Alert or review on an unknown synchronous edge, new cycle, increased call depth, fan-out tied to result count, cross-cell traffic, shared-state access, retry-owner duplication, or a control-plane call in the request path. Distributed traces help, but use service access logs, DNS/flow records, broker metadata, database clients, configuration, deployment manifests, and code analysis to cover unsampled or non-request edges.

An architecture decision record should include a revisit trigger: p99 edge cost, coordinated release rate, cross-owner incident count, cell spillover, data-view staleness, service utilization divergence, or repeated changes that cross the boundary.

Consolidation can restore a boundary that distribution obscured

Splitting is reversible only with discipline, and so is consolidation. Merge services when the promised autonomy never materialized and the edge imposes recurring cost:

• most changes and releases are coordinated;
• one invariant or transaction crosses the boundary on nearly every operation;
• calls are hot and chatty, with no useful asynchronous or cache boundary;
• independent scaling is theoretical because state or demand is shared;
• incidents and on-call ownership remain inseparable;
• version skew creates more risk than independent release removes; or
• one service has no useful degraded mode when the other fails.

Consolidation need not recreate a tangled monolith. Preserve modules, internal APIs, ownership tests, dependency direction, separate resource accounting, and a state migration plan. Replace network calls with local capability calls only after reconciling retry, timeout, idempotency, and partial-failure behavior that callers may have learned to depend on. A local call that now participates in one transaction changes semantics; record that improvement and its coupling.

Applied decision: should pricing split from checkout?

Mercury’s pricing capability has separate policy ownership and changes more often than checkout. That favors a boundary. Yet every checkout requires an accepted price, the workloads scale together, and a network failure cannot be degraded safely. Before splitting, the team asks for evidence:

1. Can pricing precompute signed, versioned quotes so checkout consumes local verified state rather than a live call?
2. Does pricing need independent CPU/hardware or a different scaling curve?
3. Can pricing deploy independently while supporting old quote versions through the retry horizon?
4. Can a pricing incident avoid blocking already quoted checkouts?
5. Is the added serialization/proxy/queue budget inside the end-to-end objective?

If signed quote production creates a genuine asynchronous authority boundary, a service can buy policy autonomy without a mandatory live hop. If every request still requires a synchronous pricing transaction and releases remain lockstep, Mercury keeps pricing as an owned module inside the checkout deployment until the contract changes. The decision follows observed autonomy, not an aspiration to have more services.

Topology decision record

operation and success rule: ____________________________________
population / mode / objective: _________________________________

candidate boundary:
  capability and owner: ________________________________________
  code / repo / build / deploy / runtime / network: ____________
  authoritative state and invariants: __________________________
  scaling/resource difference: _________________________________
  failure/security/tenant boundary: ____________________________
  change-rate and compatibility evidence: ______________________

critical path:
  required nodes/edges and completion rule: ____________________
  serial depth / parallel fan-out / merge: _____________________
  latency, deadline, proxy and queue budgets: __________________
  retry, cancellation, pending and unknown owner: ______________
  optional edge that can be pruned: _____________________________

isolation and recovery:
  shared infrastructure and correlated causes: _________________
  cell/bulkhead assignment and spillover stop rule: ____________
  control-plane loss and maximum applied-state age: ____________
  state catch-up, cache warm-up and recovery dependencies: _____

evidence:
  traces/calls/bytes/utilization/deployments/incidents: _________
  fault and blackhole test: ____________________________________
  extraction or consolidation plan: ____________________________
  revisit trigger and owner: ___________________________________

Approve the network edge only when at least one autonomy or isolation claim has evidence, every added failure/completion state has an owner, and the critical path still fits its objective under skew, overload, failure, and recovery.

Design drills

Prune the Mercury path. Reproduce the 105 ms modeled path. Remove recommendation from the response and show the 89 ms path. Then blackhole identity, inventory, recommendation, the outbox publisher, and the global configuration service one at a time. For each, state checkout correctness, response, deadline/cancellation behavior, queued work, useful goodput, and recovery. Reject any design that calls recommendation “optional” while waiting on its network, broker, or policy path.

Justify a service split. Select one module in a modular monolith. Produce evidence for domain ownership, state authority, independent resource curve, failure isolation, security boundary, and release rate. Annotate the new call, proxy, queue, compatibility, and recovery path. Design the old/new mixed deployment and rollback. If the best evidence is “the code is large,” keep a module boundary and name the condition that would justify extraction.

Audit a cell claim. Draw eight cells, assignment routing, cell-local queues/pools/state, global identity, configuration, deployment, telemetry, and recovery dependencies. Begin with the nominal 12.5% tenant share. Inject one cell overload, identity outage, bad global policy, router epoch skew, and emergency spillover. State which cases remain 12.5%, which become 100%, and the control that prevents healthy cells from accepting an unbounded retry storm.

Durable topology rules

1. Separate code, repository, deployment, runtime, network, state, team, tenant, and failure boundaries before calling something a service.
2. Require a network edge to buy evidenced ownership, isolation, scaling, security, or evolution autonomy.
3. Define success first, then place only correctness-required work on the synchronous critical path.
4. Budget serial depth, fan-out completion, queueing, proxies, retries, cancellation, and merge work end to end; do not add percentiles as if they were deterministic.
5. Draw shared infrastructure and recovery dependencies; box placement does not prove fault isolation.
6. Keep data-plane work useful from bounded applied state during scoped control-plane impairment when correctness permits it.
7. Build cells and bulkheads from separate resources, assignments, deployments, and spillover stop rules—not labels.
8. Give every authoritative fact one owner; use versioned caches or derived views deliberately rather than performing accidental distributed joins.
9. Test optional dependencies by blackholing them and verify that useful goodput, resource bounds, and correctness survive.
10. Compare intended topology with runtime evidence and consolidate when the edge adds cost without delivering autonomy.

The next architectural choice replaces some synchronous edges with durable transport. Chapter 31 examines queues, publish/subscribe systems, and logs as storage, ordering, flow-control, and consumer-state mechanisms—not as arrows that magically make work reliable or complete.

Evidence and transfer limits

• Microsoft Azure Architecture Center’s service-boundary guidance and data considerations discuss bounded contexts, chatty calls, independent deployment, state ownership, consistency, and pragmatic merging. Product/platform examples do not mechanically determine another workload’s boundaries.
• Google SRE’s Addressing Cascading Failures documents deep dependencies, overload feedback, retries, cancellation, shared resources, cycles, and blackholed noncritical backends. Its examples motivate failure tests; they are not probability inputs for Mercury.
• AWS Well-Architected’s bulkhead guidance and the Builders’ Library article on static stability support scoped cell and control/data-plane reasoning. Cloud availability-zone claims do not prove application-cell independence.
• Istio’s architecture and performance and scalability documentation make data-plane placement and workload-specific measurement visible. Envoy’s threading model describes one proxy implementation. Neither supplies a universal per-hop latency.
• The executable fixture in examples/performance-engineering-system-design-handbook/part-04/service-topology/ reproduces the 105 ms all-branch and 89 ms pruned checkout paths, 16 ms saved, an independent 24-shard deadline approximation of about 95.31%, 7 ms of modeled proxy traversal, and a 12.5%-to-100% blast-radius change through shared identity. These are deterministic teaching calculations, not observed system performance or reliability predictions.