Security, Privacy, and Performance Trade-offs

The proposed latency patch is impressive: p99 falls by 19%. It is also invalid.

The patch returns a cached customer record before evaluating the caller’s current authorization. Its benchmark contains only authorized users, so every answer is correct inside the fixture and the missing control appears to be dead weight. In production, a revoked support role can read a cached record until expiry. The benchmark improved a different system—one with a weaker trust boundary.

Security mechanisms consume CPU, memory, network round trips, queue slots, key-service capacity, storage, and operator attention. They belong in a performance model. Security invariants are not optional terms in that model. The engineering decision is where and how to perform a required check, what evidence may be safely reused, how freshness and revocation work, and what happens under attack or partial failure.

The rule is: optimize the implementation and placement of required controls; do not trade away an unstated security invariant for a local latency win.

Put controls on the actual work path

A request does not become application work merely because it reached a handler. Before and after the business operation, the system may need to:

1. establish an authenticated transport and validate its peer;
2. parse and structurally validate bounded input;
3. authenticate an identity or workload;
4. authorize the exact action on the exact resource;
5. enforce tenant, quota, and isolation policy;
6. decrypt protected state and encrypt the response;
7. produce an integrity-protected audit event; and
8. propagate only permitted telemetry.

Each control has a fast path, slow path, failure path, and recovery path. Token verification may be local until a signing key rotates. Authorization may hit a compiled local policy until revocation state is stale. Audit emission may append to a bounded buffer until its sink is unavailable. Encryption may use hardware acceleration until traffic moves to a different instance class. A useful trace labels these paths separately instead of hiding them inside “middleware.”

For request class c, model security service demand as:

D_security,c = D_transport + D_validation + D_authentication
             + D_authorization + D_isolation + D_audit

The terms are resource demands, not merely elapsed times. Some overlap; some queue behind different constrained resources. Adding their latency percentiles would be wrong. Measure or model the causal path, then ask which term dominates at the relevant load and state.

Applied decision: amortize authentication without moving the boundary

Mercury’s modeled baseline spends 2.300 ms of service time per customer-read request:

The optimized design caches verified session evidence locally. A 95% hit costs 0.050 ms; a miss costs 0.420 ms, so expected verification demand is:

0.95 × 0.050 ms + 0.05 × 0.420 ms = 0.0685 ms

Compiling policy reduces authorization demand, and batching audit writes behind a bounded durable handoff reduces request-path demand. The modeled reduction is 20.07%. None of those changes makes authorization optional.

The cache key includes issuer, audience, subject, credential version, tenant, and policy version. Entries expire before their source evidence, and revocation must invalidate them within the stated 30-second bound. A policy-store outage may use a still-valid local snapshot only for operations whose failure policy explicitly permits it; it must not silently become “allow.” High-risk writes can require online authorization even when reads use reusable evidence.

This is why “cache authorization” is too vague. Cache authenticated evidence and a scoped decision only when the invalidation, freshness, context, and fail-closed behavior are designed. A role name cached without resource, tenant, policy version, or revocation semantics is a latent privilege escalation.

Handshakes, sessions, and keys

Connection reuse and TLS session resumption can amortize public-key and round-trip work. In the teaching packet, a 3 ms full handshake with 98% reuse contributes an expected 0.06 ms per connection opportunity. That is a capacity and latency benefit, not permission to stretch keys or sessions indefinitely.

Record:

• which endpoint authenticates which peer;
• termination and re-encryption boundaries;
• session lifetime, ticket protection, and invalidation;
• key source, rotation, overlap, and emergency revocation;
• behavior when the key service or clock is unavailable; and
• whether resumed or early data is legal for each operation.

TLS 1.3 0-RTT data has replay considerations. An operation being nominally idempotent is not a complete defense: replays can consume resources or interact with application retries. Restrict early data to an application profile that defines safe messages and replay behavior. Never put a side-effecting authorization or payment operation into 0-RTT merely to save a round trip.

Acceleration and batch verification

Cryptographic acceleration changes the resource curve. Measure the actual library, algorithm, key size, message sizes, batch shape, hardware, isolation mode, and failure fallback. Acceleration that improves throughput can enlarge the queue in front of a serialized key handle or exhaust pinned buffers.

Batch verification amortizes setup and can exploit vector hardware, but it adds collection delay and failure ambiguity. If a batch fails, the system may need individual verification to identify the invalid element. Put a deadline and maximum size on the batch; reserve capacity for fallback; keep tenants separate where cross-tenant timing or fairness matters. Do not report only successful all-valid batches.

Benign traffic is not the capacity envelope

Production capacity tests usually sample expected users. An attacker chooses inputs that maximize work, memory, coordination, downstream cost, or information leakage.

Requests per second therefore has a missing dimension. Let w(x) be bounded server work for request x. Admission must control both request rate and aggregate work:

sum(w(x) for x admitted during T) <= B_tenant,T <= B_system,T

At 100 requests/s, Mercury’s benign median query costs 2 work units and benign p99 costs 8. A crafted query costs 70 because it expands nested selections and triggers many downstream fetches. A conventional 100 requests/s limiter admits 7,000 work units/s—far above the 1,600-unit worker budget. The request rate is legal while the work rate collapses the service.

The corrected API applies authenticated tenant and system budgets before expensive planning:

• maximum query depth 8;
• maximum resolved fan-out 20;
• 50 ms CPU budget with cooperative cancellation;
• 1 MiB response bound;
• at most four concurrent expensive requests per tenant;
• bounded database rows, decompression ratio, regex complexity, and downstream calls; and
• separate quotas for paid or irreversible side effects.

The admitted crafted request is capped at 12 work units. At 100 requests/s that is 1,200 units/s, leaving 400 units/s of modeled reserve. This arithmetic is not a universal quota. It demonstrates why a work estimator, concurrency limit, execution deadline, response bound, and downstream budget must complement a request counter.

Benign versus adversarial distributions

Do not append one “attack load” point to a normal histogram. Maintain at least three populations:

Plot work units, CPU, allocation, fan-out, response bytes, lock hold, paid calls, and correctness—not only latency. Random payload fuzzing may find parser bugs but miss a low-rate request that triggers an expensive authorized workflow. Conversely, a volumetric denial-of-service event may saturate network capacity before application admission can help. State the protection layer and its limit.

Rate limiting is itself a security control when it slows guessing, enumeration, resource exhaustion, and paid-side-effect abuse. Choose identity keys carefully: an IP-only limit can punish a shared network and can be distributed around; an account-only limit may be applied too late to protect unauthenticated work. Use layered source, credential, tenant, operation, work, and system budgets. Protect the limiter’s state store and define behavior when it fails.

Isolation has a price because it buys a boundary

Tenant cells, processes, sandboxes, virtual machines, language runtimes, and memory-safe components impose different startup, translation, copying, scheduling, and memory costs. Those costs purchase fault, privilege, or information boundaries. Evaluate them against the threat and failure model.

Pooling across tenants can improve utilization while creating contention and timing signals. Dedicated pools improve isolation but may strand capacity. A hybrid can reserve a minimum per tenant, permit bounded borrowing, partition sensitive key material, and prevent a noisy tenant from occupying every worker. Measure normal demand, skew, attacker-controlled work, cell loss, and recovery. The cheapest steady-state placement may have the largest breach or blast radius.

Authorization and isolation must agree. A request correctly authorized for tenant A can still cross a boundary if an object cache omits tenant identity, a worker retains ambient credentials, or an asynchronous task loses its principal. Carry the security context only as far as required, reconstruct it from verifiable evidence, and prevent shared-state keys from aliasing tenants.

Side channels are performance observations used against you

Performance engineering deliberately observes time, cache hits, branch behavior, resource use, and contention. Side-channel analysis asks what secret-dependent information those observations reveal.

Potential channels include:

• different response time or size for “account absent” and “wrong credential”;
• shared caches whose hit state reveals another tenant’s access;
• deduplication or compression ratios that reveal content similarity;
• CPU caches, branch predictors, speculative execution, and shared accelerators;
• lock contention or queue position correlated with protected operations; and
• error, retry, and audit volume that exposes sensitive workflow state.

Constant-time primitives are necessary in some cryptographic boundaries, but an application can restore a timing difference with parsing, cache access, allocation, database lookup, or error formatting around the primitive. “Add random delay” is usually weak: repeated samples reduce noise, while legitimate tail latency worsens.

Review the complete observable distribution and co-residency model. Partition state when the threat justifies it, use vetted constant-time implementations, normalize externally visible errors, avoid secret-dependent cache keys and compression contexts, and reduce cross-tenant sharing for sensitive work. Hardware and runtime behavior changes; implementation-specific claims need versioned evidence.

Privacy-safe performance evidence

A trace can contain user IDs, query text, URLs, headers, database statements, stack memory, network topology, and business events. A profiler can sample sensitive values. Treat telemetry as a data product with collection purpose, access policy, retention, lineage, and deletion—not as harmless exhaust.

Build the benchmark pipeline in this order:

1. define the performance claim and minimum fields required;
2. classify fields at instrumentation, before export;
3. drop secrets, payloads, raw identifiers, and unnecessary attributes;
4. tokenize or aggregate the join key when a join is genuinely required;
5. sample by a documented rule that preserves the population needed for the claim;
6. apply access control, encryption, integrity, region, and retention policy;
7. freeze a versioned, synthetic or de-identified fixture; and
8. publish provenance, exclusions, uncertainty, and deletion date with the result.

The teaching pipeline starts with one million raw-shaped rows and samples 10%, producing 100,000 rows. It retains 7 of 19 candidate fields, a 63.16% field reduction, and sets 14-day retention. Those numbers demonstrate reproducibility; they are not proof of legal anonymization. Tokenized identifiers may remain personal or linkable data. Rare latency outliers can also re-identify a workflow. Privacy and legal owners must define the boundary.

Data minimization can improve performance directly: fewer attributes reduce serialization, network, indexing, storage, scan, and deletion work. The benefit is earned only when the removed data was not required for security, debugging, billing, accessibility, or the stated benchmark. Minimize by purpose, not by blindly deleting the evidence needed to explain user harm.

Forensics and retention pull in opposite directions

Incidents need sufficient, trustworthy history to reconstruct authentication, authorization, configuration, control actions, and attacker behavior. Privacy and breach exposure argue against indefinite detailed retention. Separate streams by purpose:

• short-lived high-cardinality performance detail;
• integrity-protected security audit with a stricter schema;
• aggregated long-horizon capacity trends; and
• a declared legal hold process, not an engineer’s private archive.

Audit loss must be observable. Backpressure policy should distinguish operations that may proceed with a local durable spool from operations that must stop if attribution cannot be preserved. Cap the spool, encrypt it, test recovery, and prevent audit recovery from overwhelming the service after an outage.

Performance-security review matrix

Use the matrix before approving an optimization:

A secure default must remain the default under overload, rollback, regional failover, expired configuration, and operator pressure. A diagnostic bypass guarded only by convention will be used during the worst incident. If exceptional access is necessary, make it narrow, time-bound, attributable, independently authorized, and automatically revoked. Its path should be tested more carefully than the ordinary path.

“Disable certificate verification,” “skip authorization for cache hits,” “turn off audit,” and “log raw payloads temporarily” are not optimizations. They change the system’s contract. The correct response is to find the cost, improve its mechanism, reduce unnecessary data or round trips, or change an explicitly governed requirement through the proper authority.

Applied work

Run the packet:

cd examples/performance-engineering-system-design-handbook/part-07/security-performance
node analyze.mjs
node verify.mjs

First, audit the authentication change. Reproduce 2.300 ms, 0.0685 ms expected verification demand, 1.8385 ms optimized total, 20.07% reduction, and 0.06 ms expected handshake contribution. Draw the trust boundaries and list every cache-key field, expiry, revocation path, key rotation state, failure action, and 0-RTT-eligible operation. Reject any design in which a hit avoids current resource authorization.

Second, attack the query budget. Show why 100 requests/s becomes 7,000 work units/s unbounded, then verify the 1,200-unit bound and 400-unit reserve. Add one memory-amplification input, one paid-downstream action, and one low-rate timing probe. Decide which limit activates first, which principal owns the budget, how retries are charged, and what a legitimate large tenant can do instead.

Finally, construct a privacy-safe benchmark manifest: purpose, population, field allowlist, transformations, sample rule, access, region, retention, raw-data deletion, fixture provenance, and transfer limits. Explain which incident question cannot be answered after minimization and how a governed temporary escalation would collect only the missing evidence.

Evidence and limits

• RFC 8446 defines TLS 1.3, session and early-data mechanics, and 0-RTT replay considerations. It does not select Mercury’s session lifetime or authorize application operations.
• OWASP API4:2023 Unrestricted Resource Consumption inventories missing time, memory, operation, response, and spending limits. Mercury’s work-unit estimator and quotas remain local design evidence.
• OpenTelemetry security guidance warns that telemetry may contain sensitive information and requires secure collection. It does not certify this privacy pipeline.
• NIST Privacy Framework supplies risk-management vocabulary. Applicable law, consent, retention, and identifiability decisions require the relevant organizational authorities.
• Every Mercury number in this chapter is modeled teaching evidence reproduced by examples/performance-engineering-system-design-handbook/part-07/security-performance/, not a production measurement, cryptographic benchmark, or universal limit.

The final question is organizational: can a reviewer see these invariants, assumptions, bounds, and tests before the optimization ships? Chapter 64 turns that question into a design-review gate and a decision record that survives the meeting.