Data as Product, Evidence, and Liability

The Refund Label

A retailer wants to predict customer dissatisfaction. The data team joins support contacts, refunds, delivery events, and app ratings into a tidy training table. Identifiers reconcile, required fields are populated, and the label—refund within fourteen days—is pleasantly balanced. The first model performs well.

Then an analyst reads twenty records. One customer accepted store credit. Another gave up after the returns form failed on a screen reader. A third received a proactive refund for a late parcel without ever complaining. Several dissatisfied customers kept inexpensive items because returning them took too much time. The label is consistent as a database event and inconsistent as evidence of dissatisfaction.

The same table contains contact text, order histories, and derived behavior. Copies now sit in a feature store, experiment workspace, evaluation set, and vendor-managed labeling tool. Even if the model is abandoned, the exposure remains.

A production dataset has three jobs at once: serve its consumers as a maintained product, support only the claims its observations can bear, and keep the liabilities created by collection and use under control. The work belongs in one dataset record because the three jobs constrain one another.

One Record, Three Questions

As a product, the refund dataset needs an owner, named consumers, stable semantics, quality objectives, change notice, and a support path. A pipeline that arrives on time but silently changes the meaning of refund_reason is not dependable.

As evidence, it needs a defensible account of what produced each observation: who had an opportunity to appear, what the interface or policy allowed them to do, how cases were selected, what was measured, and what remained invisible. A representative sample of a poor proxy is still poor evidence.

As a liability, it needs enforceable boundaries around authority, access, vendors, retention, correction, deletion, incident response, and prohibited uses. “Internal data” is not a control, and deleting the source table is not deletion if derived copies remain.

Where those answers overlap lies the approved use boundary. The refund events may be a dependable product for financial reconciliation and useful evidence for forecasting return volume. That does not make them valid evidence of dissatisfaction. A carefully collected study might support that claim yet still be too restricted for a general-purpose model endpoint.

Follow a Record Backward Before Using It

Take one row labeled refund_14d = true. It looks like a fact. In reality it is the end of a causal history: a delivery occurred, a customer encountered a product and a return policy, an interface offered particular choices, a payment system recorded an event, and several pipelines joined and transformed those events.

Trace the row backward through six stages:

1. Phenomenon: the real condition or event of interest.
2. Opportunity: who or what could be observed.
3. Instrument: form, sensor, reviewer, transaction, log, or proxy.
4. Selection: which observations entered the dataset.
5. Transformation: cleaning, joining, aggregation, redaction, and labeling.
6. Consumption: feature, prompt, retrieval result, metric, decision, or audit evidence.

At each stage, ask what systematic distortion could enter. The retailer’s form makes “changed my mind” easy to select but hides delivery complaints behind a second screen. Fraud review determines which refunds are delayed. Customers without an accessible return path are less likely to create the event at all. A policy introduced midway through the year gives premium customers automatic refunds.

Cleaning cannot remove those conditions because they created the observations. Documenting the generating process does not disqualify the dataset; it establishes which conclusions the records can and cannot support.

Do Not Let the Label Rename the Construct

A construct is the concept the team cares about: satisfaction, risk, quality, urgency, abuse, or readiness. A proxy is an observable stand-in. A label is the encoded target used for training or evaluation.

The chain should be explicit:

construct -> observable proxy -> collection process -> label rule -> model target -> operational action

Weak chains hide value judgments. “Productivity” becomes keyboard activity. “Potential” becomes manager rating. “Risk” becomes prior investigation. “Quality” becomes complaint absence. The model can predict the label accurately while the system fails the construct.

The retailer can test the chain with domain experts and people who understand the affected customer journeys:

• Does the proxy capture the concept or only one convenient aspect?
• Which groups experience the proxy differently?
• Can system behavior change the proxy without improving the construct?
• What plausible alternative explanations produce the same observation?
• Which decisions are safe at the available evidence strength?

The last question controls the action. A weak proxy may still support a narrow operational forecast. Predicting “likelihood of a recorded refund under the current policy and interface” is more honest than predicting dissatisfaction, and it should not be used to decide which customers deserve service recovery.

Missingness Describes the Process Too

The blank rating field also has a history. Distinguish values that were not applicable, lost through ordinary failure, absent for a known device or region, unobserved because an earlier decision prevented follow-up, and removed by retention, correction, consent withdrawal, or access policy.

Imputation can make the column complete without making the evidence complete. Preserve a missingness indicator when it has legitimate analytical value, test performance across the important missingness patterns, and ask whether production will create the same patterns. Never reconstruct a sensitive value that was removed to satisfy a rights or retention decision merely to make a feature pipeline convenient.

A Labeling Queue Is Part of the Measurement Instrument

Suppose the team asks reviewers to read contacts and label dissatisfaction directly. Human judgment may improve the proxy, but it does not create ground truth automatically. The instructions, interface, source context, workload, language competence, incentives, and adjudication policy become part of the measurement instrument.

A production labeling operation therefore needs:

• a versioned ontology with inclusion, exclusion, and ambiguity rules;
• representative examples and hard counterexamples;
• source context sufficient for the judgment but minimized for exposure;
• trained annotators with appropriate domain and language competence;
• independent overlap samples and agreement analysis;
• an adjudication path that records why disagreement occurred;
• quality monitoring for fatigue, drift, throughput pressure, and systematic error;
• a process to revise prior labels after taxonomy changes;
• confidentiality, worker-wellbeing, and access controls.

Agreement alone does not establish truth. Reviewers may agree because an oversimplified rule is easy to apply. Disagreement may reveal an ambiguous concept, missing context, or two legitimate readings. Inspect where and why reviewers differ. Preserve uncertain, insufficient evidence, and multiple valid labels when the operational problem genuinely contains them; forced certainty is a measurement error with tidy formatting.

Make Reliability a Producer-Consumer Promise

A dataset becomes a product when a producer makes promises its consumers can test. Identify the authoritative source, owner, version, keys, units, allowed values, and null semantics. Define event meaning and time basis, not only field type. State delivery cadence, freshness, completeness, segment coverage, reconciliation rules, support path, and what happens when an objective is missed.

The contract also governs change. When the retailer introduces proactive refunds, the field types do not move, but the event’s meaning does. Schema compatibility is not semantic compatibility. Policy, interface, population, and instrumentation changes need a producer change log, notice period, consumer impact assessment, migration or backfill decision, and a way to deprecate the old version.

Rights and lifecycle limits belong in the same contract. A consumer needs to know whether a dataset is approved for training, evaluation, analytics, or only reconciliation; whether it may cross a region or vendor boundary; and how corrections, deletions, archival, and retirement will reach its use. These are observable properties of the product, not footnotes for a governance review.

Preserve the Release, Not Just the File

The retailer’s model release cannot be reproduced from a checksum of the final table. Its manifest must identify:

• source snapshots and query or extraction logic;
• schema and semantic version;
• filters, exclusions, joins, deduplication, and preprocessing code;
• label taxonomy, instructions, annotator cohort, and adjudication version;
• train, validation, test, and evaluation split logic;
• synthetic-data generator and filters, if any;
• rights and access-policy version;
• known corrections and deletions;
• downstream features, indexes, models, evaluations, and releases.

This manifest is both an evidence record and a propagation map. When a customer corrects a record or a source must be deleted, lineage should reveal the affected extracts, features, indexes, caches, evaluations, models, logs, and vendor copies. The appropriate legal or risk owner decides what action is required; engineering must make the impact knowable and the decision executable. Not every correction requires retraining, but “we cannot tell where it went” is not a lifecycle strategy.

Write the Card So Another Team Can Say No

The retailer’s first useful data card is not a claim of universal quality. It is a compact argument for a bounded use:

Identity and ownership. refund-events/3.1 is produced by commerce data engineering from the order, payment, return, and service systems. The customer-experience analytics lead owns meaning and approved use; the privacy and security owners approve boundary changes.

Motivation and approved use. The release supports refund-volume forecasting, reconciliation, and research into return journeys. It is not approved as a measure of satisfaction, customer trustworthiness, or service eligibility.

Composition and generating process. Each unit is an order-item journey joined to refund events and permitted service interactions. Return policy, interface accessibility, agent discretion, fraud review, shipping region, and proactive-refund rules affect whether and how an event appears. Contact attachments and full payment credentials are excluded.

Measurement and limitations. refund_14d means a recorded refund within fourteen days of delivery under the policy in force at that time. It is not a validated dissatisfaction label. Customers who keep an unwanted item, abandon an inaccessible process, or receive non-refund remedies are not captured consistently. Analysis must separate proactive and requested refunds and report coverage by return channel, region, device, and accessibility-assisted contact where appropriate and permitted.

Processing and releases. The card links to extraction logic, joins, exclusions, deduplication rules, missingness treatment, policy calendar, label instructions, split logic, and release manifest. Changes to return policy, interface, source population, or event semantics trigger a major version and consumer review even when the schema remains compatible.

Rights, security, and lifecycle. Access is role-based; approved analytics and model environments are enumerated; raw contact text has a shorter retention period than aggregate refund events. The card names processor and region restrictions, correction and deletion targets, incident owner, archival rule, and the method for finding downstream copies. New uses require approval rather than inheriting permission from availability.

Maintenance and rejection path. Monthly review covers objective failures, missingness shifts, segment coverage, source-policy changes, consumer inventory, access, and overdue copies. Consumers receive change notice and have a support contact. The card lists known limitations, prohibited uses, approval scope, expiry date, and re-review triggers so another team can reject the dataset without reverse-engineering its history.

The card is useful because it makes limitation and rejection possible. A reusable dataset is not one that can be reached by many teams; it is one whose fitness for a proposed use can be judged without folklore.

Draft a Card for Support Tickets

A software company wants to reuse support tickets for queue routing and evaluation of draft replies. Before copying the retailer’s headings, choose one real ticket and trace how it came to exist. The customer selected an entry category, a routing rule assigned a queue, an agent may have reassigned it, and the final resolution may have happened outside the platform. Final route therefore records both the issue and the organization’s queue design.

Write the card another team would need. It should identify the dataset and owner; motivation, consumers, and prohibited uses; composition and exclusions; collection, selection, and preprocessing; construct, proxies, label instructions, disagreement, and uncertainty; missingness, coverage, measurement error, and known limitations; authority, licensing, confidentiality, access, vendors, and geography; split and leakage controls; service objectives and change notice; retention, correction, deletion, archival, and retirement; downstream lineage; and approval scope, expiry, and re-review triggers.

Now test the card with two proposed consumers. A routing team wants to predict the correct destination, while a manager wants to score agent performance. Does the record give each team enough information to accept, narrow, or reject the use? If both receive the same answer, the approved-use boundary is probably still too vague.

Only after that boundary is explicit does model selection become an engineering decision. The next chapter asks which system is simple enough to satisfy it.

Source Notes

• Datasheets for Datasets, the foundational proposal for documenting dataset motivation, composition, collection, preprocessing, distribution, uses, and maintenance; verified 2026-07-10.
• NIST AI RMF Core, guidance on documenting data selection, suitability, representativeness, construct validation, privacy risk, and lifecycle risk; verified 2026-07-10.
• Hidden Technical Debt in Machine Learning Systems, analysis of data dependencies, feedback loops, undeclared consumers, and configuration debt in production ML; verified 2026-07-10.