Production Data Systems Handbook / Chapter 56
Appendix B: Workload Fingerprint Worksheet
A reusable worksheet for turning vague data-system requirements into concrete workload facts before product comparison.
Purpose
Use this worksheet before naming a database, queue, cache, stream, warehouse, index, or storage product. The goal is to describe the work the system must do, the facts it must preserve, the failures it must survive, and the operating burden the team is accepting.
A useful fingerprint is specific enough that two engineers can disagree about trade-offs without retreating to product preference. It does not choose a system by itself. It reveals the capabilities that matter, the failure modes the team cannot accept, and the promises that still need evidence.
When to Use the Short Version
Use the short version for routine design reviews, new service proposals, feature work that adds a data path, and early architecture discussions. Keep it to one page by naming the dominant operations and constraints rather than cataloging every possibility.
Complete it from facts outward: first the business outcome and source of truth, then the data shape and operations, then the guarantees and operating boundary. Give every measurement a date and source. If a field cannot be answered, write down the missing evidence, who will obtain it, and what could go wrong while it remains unknown.
| Dimension | Questions | Record |
|---|---|---|
| Business goal | What user, customer, regulatory, operational, or revenue outcome depends on this data? | Outcome, affected journey, consequence of wrong or missing data. |
| Source of truth | Which component owns the authoritative fact? Which copies are derived? | Owner, authoritative system, derived stores, rebuild source. |
| Data shape | Is the data entity state, event history, document, time-series, graph, vector, blob, log, or aggregate? | Primary shapes and lifecycle for each shape. |
| Size now | How many rows, documents, events, files, vectors, tenants, partitions, or bytes exist today? | Current cardinality and storage size with measurement date. |
| Growth | What is expected daily, monthly, and annual growth? How long is data retained? | Growth rate, retention window, archive behavior. |
| Read patterns | Are reads point lookups, range scans, joins, search, aggregation, traversal, similarity search, subscriptions, or exports? | Named read operations, result sizes, callers, access filters. |
| Write patterns | Are writes inserts, updates, upserts, appends, imports, deletes, backfills, corrections, or side effects? | Named write commands, transaction boundary, retry behavior. |
| Latency | What p50, p95, and p99 targets matter for reads and writes? Under what load? | Percentile targets, concurrency, peak window, timeout policy. |
| Correctness | What invariants must hold even under concurrency, retries, replay, and partial failure? | Invariant names and enforcement locations. |
| Consistency | Which stale, missing, duplicated, reordered, or inconsistent states are acceptable? For how long? | Staleness budget, read-your-writes needs, ordering scope. |
| Durability | Which acknowledged data may be lost, if any? Which side effects must be recoverable? | RPO, commit boundary, replay or reconciliation source. |
| Availability | Which user journeys must continue during store, region, dependency, or consumer failure? | Critical journeys, degraded modes, operations to reject. |
| Recovery | What are RPO, RTO, restore granularity, and proof of restore? | Recovery targets, restore owner, last drill evidence. |
| Security | What classification, access model, audit trail, encryption, residency, or abuse risk applies? | Data classes, access boundary, audit and retention requirements. |
| Operations | Who owns dashboards, alerts, runbooks, cost, access, migrations, and incidents? | Team, on-call maturity, escalation path, support hours. |
| Cost | Which dimensions drive cost, and what ceiling or alert threshold exists? | Storage, reads, writes, indexes, replicas, retention, egress, queries. |
| Evolution | Which schema, traffic, tenancy, region, access, retention, or product changes are likely? | Expected changes, compatibility constraints, reversal plan. |
The Short-Version Decision Gate
Before comparing products, the review should be able to answer five questions from the worksheet:
- What fact is this data path responsible for preserving?
- Which operation is latency-critical, and which operation is correctness-critical?
- Which failure would create user-visible harm, financial harm, privacy harm, or unrecoverable operator work?
- Which data can be rebuilt, and which data must be protected as the source of truth?
- Who is accountable when the data is slow, wrong, missing, duplicated, expensive, or inaccessible?
Extend the Fingerprint Where Risk Demands It
The long version is a set of extensions, not a second form to complete mechanically. Add the sections that expose a binding risk: tenant skew, burst load, deletion, derived-data drift, regional failure, an unproven performance claim, or a costly exit. Payments, entitlements, health or safety data, compliance-sensitive datasets, high-volume event paths, multi-region systems, large migrations, and shared platforms will usually need several of them.
Distribution and Tenancy
| Dimension | Questions | Record |
|---|---|---|
| Tenant count | How many tenants or customer partitions exist now and at expected scale? | Current count, projected count, tenant creation rate. |
| Largest tenant | How large is the largest tenant compared with the median tenant? | Largest-to-median ratio for storage, writes, reads, and exports. |
| Hot tenant risk | Can one tenant dominate traffic, locks, partitions, queues, indexes, or support operations? | Hot-tenant scenarios and isolation controls. |
| Partition key | Which key distributes storage or work? Does it match read and write paths? | Candidate keys, skew evidence, rebalancing plan. |
| Cross-tenant operation | Which operations scan, aggregate, export, delete, or repair across tenants? | Frequency, authorization, throttling, audit requirements. |
Load Shape
| Dimension | Questions | Record |
|---|---|---|
| Average load | What is the normal read, write, and background-job rate? | Measured baseline and date. |
| Peak load | What peak rate must the system absorb? | Peak multiplier, seasonality, campaign or batch windows. |
| Burst tolerance | How long can bursts last before callers, queues, or users are harmed? | Burst duration, buffering limit, backpressure behavior. |
| Fan-out | How many derived writes, index updates, messages, cache invalidations, or notifications follow one source write? | Write amplification map and failure handling. |
| Concurrency | Which records, keys, tenants, ranges, or aggregates receive concurrent writes? | Contended resources and lost-update protection. |
Lifecycle, Retention, and Deletion
| Dimension | Questions | Record |
|---|---|---|
| Lifecycle | Is each data class hot, warm, cold, archived, disposable, or evidentiary? | Lifecycle stage and transition trigger. |
| Retention | How long must data be kept for product, legal, analytical, audit, or debugging needs? | Retention by data class and storage location. |
| Deletion | Which deletes are user-requested, policy-driven, legal, operational, or corrective? | Deletion type, propagation path, proof requirement. |
| Tombstones | Are deletion markers needed so derived systems observe deletion? | Tombstone retention, compaction timing, replay behavior. |
| Export | Can users, regulators, customers, or operators request export? | Export scope, format, authorization, time limit. |
Derived Data and Freshness
| Dimension | Questions | Record |
|---|---|---|
| Derived stores | Which caches, indexes, read models, warehouses, streams, feature tables, or exports depend on this data? | Store list, owner, source, rebuild path. |
| Freshness target | How stale may each derived store become before users or systems are harmed? | Freshness SLI, alert threshold, user-facing effect. |
| Rebuild time | How long does a full or partial rebuild take at current and projected scale? | Rebuild duration, capacity limit, verification method. |
| Replay model | Can changes be replayed safely? Are operations idempotent? | Replay source, offset or checkpoint, duplicate handling. |
| Drift detection | How will the team detect divergence between source truth and derived copies? | Reconciliation query, sample audit, checksum, business total. |
Multi-Region and Failure Boundaries
| Dimension | Questions | Record |
|---|---|---|
| Region role | Is each region active, passive, read-only, disaster-recovery only, or tenant-owned? | Region map and traffic routing policy. |
| Write ownership | Where are writes accepted, and how are conflicts prevented or resolved? | Ownership rule, consensus boundary, conflict policy. |
| Read locality | Which reads must be local, and what staleness do local reads accept? | Locality requirement and freshness measurement. |
| Failure mode | What happens during network partition, provider region outage, replica lag, or partial dependency failure? | Allowed behavior, fencing, failover trigger, rollback trigger. |
| Data residency | Where may data be stored, processed, backed up, logged, and accessed? | Residency rule including derived systems and observability. |
Benchmark and Prototype Plan
| Dimension | Questions | Record |
|---|---|---|
| Risk to test | Which assumption would make the design wrong if false? | Top risks ranked by impact. |
| Representative data | Does the test use realistic size, skew, indexes, payloads, and history? | Dataset shape and known gaps. |
| Load model | Does the test include peak, burst, background jobs, retries, and failure? | Load profile and stop conditions. |
| Correctness test | Does the test exercise concurrency, duplicate delivery, stale reads, delete propagation, and recovery? | Invariants tested and expected evidence. |
| Cost observation | Which cost driver will the test expose? | Metrics for storage, compute, I/O, egress, query spend. |
| Decision rule | What result would accept, reject, or force a redesign of the option? | Thresholds and the decision each threshold triggers. |
Reversibility Plan
| Dimension | Questions | Record |
|---|---|---|
| Exit trigger | What evidence would show this choice is wrong? | Metrics, incidents, cost thresholds, product constraints. |
| Migration path | How would data leave this system? | Export format, dual-write or backfill path, compatibility plan. |
| Lock-in surface | Which APIs, query language, operational features, data formats, or managed-service capabilities are hard to replace? | Lock-in inventory and mitigation. |
| Cleanup | What must be retired after reversal? | Derived stores, queues, indexes, permissions, dashboards, contracts. |
Worked Example: Checkout Reservations
This hypothetical fingerprint is deliberately incomplete: its figures are design assumptions, not measured production results. It is enough to expose the decisions and the evidence the team still owes.
| Dimension | Example answer |
|---|---|
| Business goal | Prevent overselling scarce inventory while keeping checkout responsive. |
| Source of truth | Inventory reservation table owns active holds; order ledger owns completed sale; search and catalog availability are derived. |
| Data shape | Mutable entity state for SKU inventory, append-only reservation events, short-lived reservation records. |
| Size and growth | Assume 8 million SKUs, up to 40 million active reservations during peak sale windows, and 18 months of reservation-event retention; validate against the next campaign forecast. |
| Read patterns | Point lookup by SKU and warehouse, customer cart summary, operator audit by reservation ID. |
| Write patterns | Create hold, extend hold, release hold, convert hold to order, expire hold, correct inventory. |
| Latency | Checkout hold create p95 under 80 ms during sale peak; operator audit p95 under 2 seconds. |
| Correctness | Available quantity never goes below zero for committed reservations; a reservation converts to at most one order. |
| Consistency | Catalog availability may lag source truth by 60 seconds; checkout must read current reservation authority. |
| Durability | An acknowledged active hold may not be lost before expiration or conversion. Recovery must reconstruct current holds from durable reservation records and the order ledger. |
| Availability | Browsing may continue with stale availability; checkout hold creation may fail closed during authority outage. |
| Recovery | Rebuild from the latest consistent snapshot plus durable reservation records, then reconcile against the order ledger and expiration events. |
| Security | Reservation data is internal operational data; customer identifiers in audit views require support-role access. |
| Operations | Commerce platform owns source truth and on-call; search team owns derived availability index freshness. |
| Cost | Hot-sale write throughput, reservation-event retention, and search-index update rate dominate cost. |
| Evolution | Expected split by region and warehouse; reservation API must support dual reads during migration. |
The fingerprint has already ruled out several vague designs. Checkout cannot rely on the eventually updated catalog index for availability, because that copy may be sixty seconds stale. The reservation authority needs an atomic or conditional write that protects available quantity and conversion uniqueness at the SKU-and-warehouse boundary. During an authority outage, browsing can continue but new holds fail closed.
It has not yet selected a product. The peak cardinality is still an assumption; the contention distribution by SKU and warehouse is missing; and the no-loss durability requirement needs a tested recovery design. Those gaps define the prototype: replay realistic hot-sale skew, race hold creation against expiration and conversion, interrupt the authority, then prove both the invariants and reconstruction time.
Review Guidance
A workload fingerprint should not force premature precision. It should make uncertainty visible. If the team cannot answer a field, the decision record should name the missing evidence, the prototype or measurement that will produce it, and the operational consequence of being wrong.
The strongest fingerprints produce fewer arguments about products and better arguments about facts: which invariant matters, which workload dominates, which failure is acceptable, which recovery promise is real, and which team can operate the result.