Data Platform and Analytics Playbook

A product team wants to understand why customers cancel. The source application already restricts billing records and support conversations, but the analysis creates a new dataset: customer identity joined to payment history, product activity, and support text. An analyst samples it in a notebook. A dashboard caches the result. A contractor receives a spreadsheet for qualitative review. Six weeks later, a customer asks for deletion.

Which system now owns the promise to delete the customer’s data?

Every useful step created another copy, another reader, and another expiration problem. Source-system authorization did not travel automatically into the warehouse, notebook output, dashboard cache, or spreadsheet. The platform is secure only when each copy remains tied to its classification, purpose, identity, access policy, retention, monitoring, and a workable way to revoke or delete it.

Classify what the join creates

Classification begins at the source, then changes with the data. The cancellation dataset is more revealing than any one input: it connects a named person to finances, behavior, and private support messages. Aggregation may reduce sensitivity; a join can restore identity or create a new business-sensitive view. Treat derived tables, feature sets, extracts, and materialized views as data products with their own classification, rather than inheriting the least restrictive label among their inputs.

A usable dataset record names the owner, sources, classification, allowed purpose, retention rule, access model, downstream consumers, deletion behavior, and incident contact. “Analytics” is too broad to justify access. Churn analysis, fraud investigation, customer support, product experimentation, executive reporting, and model training have different purposes even when they begin with the same rows.

The analysis does not need full support transcripts, complete payment records, or an unlimited history. Select the smallest date range and columns that answer the question. Tokenize direct identifiers when stable linkage is needed, aggregate when individual detail is not, and keep exceptional high-sensitivity fields in a more restricted zone. Apply the same decision to rejected records, pipeline logs, dead-letter queues, samples, validation reports, and temporary files. These are copies, not harmless exhaust.

Give the pipeline only one route

The ingestion job needs a workload identity allowed to read the named source objects and write the named destination. It does not need a shared extractor credential that can inspect every production database or populate every analytics zone. Separate identities for extraction, transformation, quality checks, and export make both containment and attribution possible.

At ingestion, verify the source, schema, classification label, destination, freshness, and plausible record volume. Quarantine an unexpected support-text column instead of allowing permissive schema evolution to carry it into a broadly readable table. Record the dataset, partition, row count, pipeline identity, run identifier, transformation version, and disposition of rejected data without copying sensitive values into operational logs.

Prefer workload identity or short-lived credentials supplied at runtime. Where a connector still requires a secret, retrieve it from the managed secret system, scope it to one route, rotate it, and make its owner and consumers discoverable. Scheduler variables, environment dumps, notebook cells, copied configuration, and source repositories are poor secret stores because they multiply both readers and residues.

Pipeline secret handling checklist

Before a pipeline is released, its owner should be able to show:

• a distinct runtime identity or, when unavoidable, a managed secret with a named owner and rotation path;
• the exact sources, destinations, operations, and network routes that identity can use;
• separation between production execution, human debugging, and emergency access;
• logs that identify use and policy changes without recording the credential or sensitive payload;
• a tested revocation procedure and a list of jobs, connectors, and schedules that will fail when access is removed;
• removal of obsolete credentials after migration, connector retirement, or ownership change.

Enforce policy below the dashboard

Grant access by identity, purpose, dataset, environment, and sensitivity. The analyst may need pseudonymous customer histories; a support operator may need identified records for assigned accounts; the contractor may need a narrow sample with direct identifiers removed. Those are different policies, not three memberships in one “data users” group.

Enforce row and column restrictions at a layer that governs every supported query path. A dashboard filter is presentation logic if a user can bypass it through direct SQL, an API, a notebook, or a cached extract. Test the policy through each path and test denied cases, including cross-tenant rows, restricted columns, service identities, exports, and newly derived tables.

Administrative capabilities deserve their own review. Creating connectors, scheduling jobs, installing packages, changing dataset policy, impersonating users, sharing reports, or approving exports may bypass ordinary read permissions. Keep those roles rare, individually attributable, time-bounded where practical, and subject to recertification. Review service readers as closely as humans: BI tools, reverse-ETL jobs, data-quality monitors, training jobs, and export services often possess the broadest access in the platform.

Keep notebook work inside a boundary

A notebook combines code execution, package installation, credentials, data access, stored output, and collaboration. For the churn analysis, use an environment tied to the analyst’s identity, the approved datasets, controlled egress, an allowed package source, and a bounded runtime. Do not turn the notebook server into a convenient bridge from sensitive tables to arbitrary internet destinations.

Secrets should not appear in cells, outputs, hidden metadata, checkpoints, or shared files. Data can remain in output even after the query that produced it has been removed. Sharing the notebook therefore requires a review of code, output, embedded files, widgets, checkpoints, permissions, and destination—not merely clearing the visible cells.

Notebook security checklist

Before granting or sharing notebook access, verify:

• the user, project, approved datasets, purpose, and expiration;
• runtime isolation, network egress, package-installation policy, and access to local or mounted storage;
• runtime identity or managed secret delivery without credentials embedded in the notebook;
• limits on sharing, publishing, downloading, scheduling, and creating persistent services;
• treatment of cell output, metadata, checkpoints, exported formats, logs, and snapshots as possible data copies;
• session, kernel, temporary-storage, and workspace cleanup when the work expires;
• query, export, package, identity, and network evidence available for investigation.

Dashboards and reporting tools create related copies. Cached queries, scheduled reports, embedded charts, shared links, email subscriptions, screenshots, and CSV buttons all need an audience, purpose, and lifecycle. A view-only interface is not a no-export control when the underlying tool permits subscriptions, API retrieval, or unrestricted queries.

Make an export an expiring grant

The contractor’s spreadsheet crosses from governed query controls into a weaker environment, so approval must describe the copy rather than merely record that somebody clicked “approve.” Start with the dataset and its derived sensitivity. State the business purpose and why a smaller, masked, or aggregated result is insufficient. Name the recipient and storage location; enumerate the columns, rows, date range, and aggregation level; then name the transfer protection, access restriction, expiration, deletion owner, and evidence that will confirm both delivery and disposal.

That record is the analytics export approval form. It should also name the approver, data owner, requestor, incident contact, and a unique export identifier. Approval is invalid if the destination cannot enforce the promised access or deletion, or if nobody can locate the copy later. Recurrent exports need a reviewed pipeline and continuing policy, not a chain of informal exceptions.

Retention follows every branch of the lineage: raw zones, curated tables, materialized views, caches, backups, snapshots, notebook output, exports, model inputs, and logs. Deleting the source row is incomplete while identified copies remain elsewhere. For the cancellation dataset, test deletion with a seeded record: locate it through the governed lineage, trigger the workflow, verify removal or approved irreversible transformation at each supported copy, and record any backup delay or legal hold that prevents immediate deletion. A policy document alone cannot prove propagation.

Detect a copy leaving its expected path

Useful detection combines behavior with business context. Watch for unusually large results, rare sensitive-table reads, restricted-column spikes, new destinations, access outside a user’s normal pattern, interactive use of service identities, broad joins, repeated denied queries, notebook egress, and changes immediately followed by export. Capture the actor, role, dataset, query class, columns, approximate volume, tool, destination, source, policy decision, and recent permission changes while avoiding a second sensitive dataset inside the security logs.

The quarter-end report may legitimately read millions of rows. A forgotten dashboard loop may do the same. An attacker may imitate either. Route the alert to people who understand the dataset and the requesting workflow, and preserve enough query and policy evidence to distinguish them.

Containment must work at the identity and copy boundaries. Responders may need to suspend a user, revoke a connector, stop a job, disable a share or export, quarantine a derived dataset, block notebook egress, rotate a pipeline credential, and preserve query, lineage, and approval records. Record downstream recipients before erasing volatile state. If the platform can detect a questionable export but cannot locate or revoke it, detection arrives one control too late.

Data platform security checklist

Release review should follow the copy through the system:

• Source and derivation: owners, purposes, classifications, minimization decisions, and lineage cover both inputs and newly sensitive joins.
• Movement: each pipeline has narrow identity, validated input and destination, controlled secret delivery, bounded error paths, and attributable runs.
• Access: row and column policy holds across SQL, APIs, notebooks, BI tools, caches, service identities, and exports; denied paths are tested.
• Workspaces and tools: notebook, dashboard, connector, scheduling, package, sharing, and administration boundaries match the data’s sensitivity.
• Copies: exports have named recipients, destinations, scope, protection, approval, expiration, deletion ownership, and evidence.
• Lifecycle: retention and deletion propagate through derived tables, caches, snapshots, outputs, logs, and known downstream systems.
• Defense: anomaly signals reach data-aware owners, and responders can contain identities, jobs, connectors, datasets, egress, and exports.

When the customer asks for deletion, the answer should not depend on an analyst remembering a spreadsheet. The dataset record identifies the join, lineage identifies the notebook and dashboard, the export record identifies the contractor’s copy and its expiration, and the deletion run supplies evidence from each governed branch. Following the copy is no longer an investigative slogan; it is how the platform keeps a source-system promise after the data becomes useful.