Data Quality, Governance, Lineage, and Ownership

The Dashboard Moves but the Business Does Not

The revenue dashboard rises after a billing deployment, but the ledger does not. No column disappeared. No pipeline failed. The invoice_paid event still arrives on time and still passes its schema checks. Only its meaning changed: it used to be emitted after successful capture; now it is emitted when payment is authorized. Some authorizations never become captured payments.

The software is green because every component accepted the new event. The number is wrong because the data crossed several boundaries without anyone protecting what the event meant.

This is the work of governance. Its purpose is not to collect descriptions of tables. It is to keep a fact usable as it moves from a source service through events, transforms, semantic models, dashboards, exports, and decisions. That requires an owner who can decide, a contract that includes meaning, lineage that exposes consumers, quality gates tied to the decision, and a change path that reaches everyone who will bear the consequence.

Begin at the Decision, Not the Table

The warehouse table containing invoice_paid is not important by itself. It matters because finance uses a revenue model derived from it, customer success receives a renewal export, and product analytics measures conversion. Those uses do not share one tolerance for error. Finance needs a definition that reconciles with its control totals. The renewal export must include the intended customer population and only the fields its audience may receive. Product analytics may accept a documented delay but cannot silently switch from captured payments to authorizations.

Govern the decisions, then work backward to the data that supports them. For the revenue metric, ask what business event the number claims to represent, at what grain and time boundary, which corrections are allowed, how fresh it must be, and what independent fact can disprove it. Those answers determine which sources matter and which controls deserve to block publication.

This keeps governance proportional. A temporary table used inside one bounded repair does not need the same ceremony as a shared revenue model. A source fact that drives money, access, compliance, or customer communication deserves stronger ownership, compatibility, and incident behavior because the cost of ambiguity travels downstream.

Ownership Follows the Fact

The first response to the dashboard discrepancy is often a chain of referrals. Finance points to analytics because analytics owns the model. Analytics points to billing because billing emits the event. Billing points to the warehouse because its deployment succeeded. Everyone owns a component; nobody owns the decision path.

Ownership needs both authority and duty. The billing system owner operates the source service, deploys schema and event changes, and responds when the producer fails. The data product owner defines the supported meaning, grain, compatibility promise, and deprecation path of the shared revenue data. A business steward decides domain rules that code alone cannot settle: whether authorization, capture, settlement, refunds, trials, tax, and backdated corrections belong in the metric. The on-call owner has authority to stop publication, quarantine suspect data, and coordinate repair. A platform owner supplies contract checks, lineage capture, quality infrastructure, access review, and the standard route for changes.

One team may perform several of these roles. The organizational chart matters less than the absence of gaps. For each important handoff, someone must be able to answer three questions: Who may change this? Who decides what it means? Who acts when it is wrong?

A catalog entry containing a team name answers none of them unless the team accepts those duties. Observable ownership means approving breaking changes, maintaining the contract, responding through a known escalation route, reviewing quality incidents, and retiring consumers deliberately. Ownership stops being metadata when it changes what happens during a deploy or incident.

A Contract Protects Meaning

The producer’s schema contract did not catch the invoice_paid change because the event kept the same fields and types. A production contract must cover more than shape.

For an event, it should define the real-world action represented, the moment the event becomes true, its identity and grain, required fields, ordering and duplication behavior, sensitivity, replay semantics, and compatibility policy. If invoice_paid means successful capture, emitting it at authorization is a breaking change even when every consumer can deserialize it. If historical events can be replayed, the contract must also let consumers distinguish replayed history from new business activity where that distinction affects behavior.

The semantic model has its own contract. It identifies source facts, inclusion and exclusion rules, time and currency treatment, correction behavior, refresh expectation, material consumers, and the people who approve definition changes. This prevents an ordinary code review from changing a business number before the people who use it know what moved.

Compatibility therefore includes interpretation, freshness, and access as well as schema. Changing metric grain, delaying a feed beyond its promise, narrowing an export audience, or redefining an enum can break a consumer with no parser error. The contract should say which changes are additive, which require a migration window or dual run, and what evidence permits the old path to be removed.

Quality Is a Promise to a Use

Teams sometimes compress data quality into one score. That hides the failure that matters. A revenue dataset can be complete and fresh while every row is classified at the wrong business moment.

Name the promise instead. Completeness asks whether the expected invoices, tenants, and required fields arrived. Accuracy asks whether the values represent the controlling source or real-world fact. Freshness asks whether the data arrived soon enough for the decision. Consistency asks whether related representations—billing state, ledger controls, warehouse rows, and the dashboard—agree. Validity asks whether values satisfy domain rules and relationships. Uniqueness asks whether duplicate events or rows change totals or side effects.

Each dimension needs a boundary and a response. A missing optional product attribute may only warrant a warning. A revenue model that does not reconcile with its control total should stop publication. An entitlement feed beyond its freshness promise may need to page because stale state can deny paid access or overgrant restricted capability. Quality thresholds should reflect the distribution of harm, not only an aggregate percentage: a tiny missing segment can contain the customers or transactions that dominate the decision.

Place gates where meaning can break. At the source contract, schema and domain checks reject malformed or incompatible events. During ingestion, completeness, uniqueness, and referential checks catch missing invoices, duplicate event identities, and unknown tenants. Around transforms, version and freshness checks show which code produced which interval of data. Statistical checks reveal volume drops, cardinality shifts, duplicate spikes, and segment changes that valid rows can still create.

The final gate must be a business control. Revenue should reconcile against an independently maintained billing or ledger control appropriate to the organization’s accounting model. Entitlements should reconcile with subscription state and explicit exceptions. An export should reconcile its population and prove that restricted fields are excluded. These controls are valuable because they challenge the conclusion, not merely the machinery that produced it.

Lineage Turns Suspicion into a Bounded Search

Once the mismatch is detected, responders need to know where the changed event went. Useful lineage follows the fact from billing service to event stream, ingestion job, warehouse table, transformation version, semantic model, dashboard, scheduled export, and any service that consumes the result. It records owners, schedules, contracts, quality gates, sensitivity boundaries, and consumers along that route.

That detail changes the incident. Responders can identify the first affected event, the transform versions that consumed it, the dashboard intervals that became unreliable, and the export runs that used the changed definition. They can notify known consumers instead of broadcasting a vague warning. They can also see that rolling back the producer will stop new damage but will not correct data already transformed or exported.

Version and time belong in the graph. Source schemas, contract definitions, transform code, semantic models, quality rules, and backfill jobs all change. Without their effective intervals, lineage can show where a number came from but not why Tuesday’s number differs from Monday’s. With them, the team can separate business movement from definition movement and choose the exact interval to replay.

Lineage earns its upkeep before planned changes too. A team proposing to change event timing can discover the revenue model and renewal export, invite their owners into review, dual-publish old and new semantics, and compare outcomes before cutover. Governance then shortens a risky search rather than lengthening a safe deploy.

The Catalog Is an Operating Surface

The catalog should expose what responders and consumers now need: the dataset’s meaning and grain; system, data product, business, and on-call ownership; its source and versioned lineage; freshness promise and current state; quality controls and history; sensitivity and allowed uses; access route; retention and deletion behavior; material consumers; limitations; deprecation path; and incident contact.

Metadata is most trustworthy when production workflows maintain it. A contract release records its version and owner approval. A deployment records the active source and transform versions. Quality jobs publish current results. Access workflows record purpose, approver, and expiration. Deprecation records the replacement and removal date. Manual descriptions remain necessary for business meaning and caveats, but a catalog that depends on someone remembering to refresh every operational field will soon lose to private messages.

The practical standard is interruption avoided. A consumer should be able to decide whether the dataset fits a use, obtain the right access, see current health, and find escalation without scheduling a meeting. An owner should receive fewer questions whose answers already exist. A searchable inventory of names is not yet governance.

Governance Lives in the Change Path

The event-timing change should have entered a breaking-change review even though its schema was stable. That review would identify affected consumers, compare old and new semantics, define a compatibility period, choose dual-publish or versioned events, specify reconciliation, name rollback conditions, and require approval from the owners of meaning and operation.

The same discipline appears in three other recurring workflows. A new-dataset review establishes purpose, source facts, owners, contract, expected consumers, quality gates, freshness, sensitivity, access, retention, and retirement before the dataset becomes depended upon. The depth of review should grow with sharing and consequence.

An access review verifies that people, services, jobs, notebooks, business-intelligence tools, and exports still need the data they can reach. Sensitive access needs a purpose, owner, review or expiration, and audit evidence. Export access deserves explicit attention because it moves data beyond many of the controls represented in lineage.

A retention review asks whether each copy still has a justified business, operational, privacy, or legal purpose. It follows the path through sources, streams, warehouse tables, semantic outputs, logs, backups, exports, and incident artifacts. Legal and privacy owners decide the applicable obligations; engineering makes the chosen lifecycle executable and testable.

These workflows should not make every change political. Local, reversible, low-consequence work can take a light path. Shared facts with material consumers need evidence and coordination. The point of the review is to expose the cost of a change while there is still time to design it, not to reward the production of forms.

Repair the Decision Path

When the dashboard and control total diverge, containment comes before explanation. Mark the revenue dataset unreliable for the affected interval, stop scheduled publication or exports where wrong data would cause harm, and halt or roll back the semantic change that created new bad outputs. Preserve enough scoped evidence to identify the bad interval without creating uncontrolled copies.

Correction starts from an authoritative fact. Restore the intended event semantics or introduce a new version, identify authorizations that did and did not become captured payments, and replay only the bounded interval through idempotent transforms. Then reconcile the corrected result against the independent control. A green backfill is execution evidence; reconciliation is correctness evidence.

Communication names which data was wrong, for what interval, which decisions and consumers may have been affected, what has been corrected, and what uncertainty remains. The durable repair is larger than a row update. The event contract now treats timing semantics as compatibility. Lineage includes the export. The business control blocks publication. The runbook names who may quarantine and republish. The change workflow requires consumer review for future meaning changes.

Governance has succeeded when the next change is safer and the next failure is easier to bound.

A Dataset Readiness Review

Before publishing a shared dataset or relying on it for money, access, compliance, or customer-visible behavior, walk its path in order:

1. Name the decision and the consequence of wrong or late data. Define the fact, grain, identity, timing, and authoritative source.
2. Assign authority for operation, meaning, domain policy, incident response, and platform mechanisms. Record an escalation route that a consumer can actually use.
3. Trace sources, events, transforms, versions, semantic models, exports, and material consumers. Make sensitivity boundaries and derived copies visible.
4. Put quality gates at the handoffs where shape, completeness, accuracy, freshness, consistency, validity, or uniqueness can fail. End with an independent business control where the consequence warrants one.
5. Define compatibility, rollout, rollback, correction, and communication for changes. State when publication stops and how repaired data earns trust again.
6. Record access purpose and review, retention and deletion behavior, known limitations, examples, freshness, current quality state, and deprecation policy where consumers will find them.

The review is ready when another engineer can judge the dataset, change it safely, and operate a failure without reconstructing its history from chat.

Governance Review Drill

Trace one revenue metric from the source transaction to the dashboard. Mark every owner, contract, transform version, quality gate, freshness boundary, access decision, retention rule, consumer, and stop-publication condition. Then propose changing the source event from successful capture to authorization. Decide whether to version the event, dual-publish, or migrate consumers, and specify the evidence that would permit cutover and rollback.

Repeat the trace for an entitlement path or customer export. The new path should force different priorities: tighter freshness and repair for entitlement, or sensitivity, population reconciliation, retention, and revocation for the export. If either exercise becomes archaeology, that is the finding. Governance begins with making the path knowable.