Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 24

Privacy, Confidentiality, and Data Protection by Design

Trace personal and confidential data through prompts, retrieval, models, reviewers, logs, embeddings, evaluation sets, vendors, retention, and deletion.

The Conversation That Would Not Disappear

A company uses an assistant to draft performance-review summaries. A manager pastes a note about an employee’s medical leave, deletes the conversation a few minutes later, and asks support to confirm its removal.

The chat database no longer contains the thread. The request still cannot be closed.

The prompt was copied into an application trace. A quality reviewer marked the answer for later analysis. That example entered an evaluation set. The draft was embedded so similar language could be found. The model provider retained request data under the deployed account settings. Each copy has a different owner, access path, retention rule, and deletion mechanism.

“The provider does not train on our prompts” was true. It answered one question about one destination. The employee’s data had traveled through an entire system.

Privacy by design makes every use and retained copy earn its place. It follows personal and confidential data through collection, inference, human review, monitoring, reuse, suppliers, and deletion—not only through model training.

A privacy data-flow map follows data from collection through prompt, retrieval, generation, review, and monitoring, with checkpoints for necessity, permission, minimization, retention, and deletion; logs, embeddings, and evaluation sets are shown as persistent storage pockets with deletion propagation.
Ask the privacy questions at every stage and every derived store. A deletion promise is real only when it propagates through logs, embeddings, evaluation data, caches, and suppliers.

Three Questions, Not One

Security, confidentiality, and privacy overlap, but they do not make the same decision.

Security asks whether the system resists unauthorized access, alteration, and disruption. Confidentiality asks whether information reaches only authorized parties for authorized purposes; it covers trade secrets and internal records as well as information about people. Privacy asks whether collecting, inferring, using, sharing, retaining, or deleting data about people is appropriate in the first place and what control those people should have.

Encryption can protect a prompt that the product never needed to collect. A reviewer can be authorized to open a queue that should not contain full conversations. A notice can describe processing while a weak access boundary discloses the data. The architecture needs all three lenses.

This is engineering and governance guidance, not legal advice. Applicable duties depend on jurisdiction, sector, organizational role, data, purpose, contracts, and current law. Qualified privacy and legal reviewers must determine the required authority or basis, notices, rights processes, transfer controls, and records for a particular deployment.

Draw the Data Lineage Before Choosing Controls

Start with the employee’s sentence, not a component inventory. Follow it through state and time:

source -> prompt -> context -> provider -> output -> review
       -> trace -> evaluation example -> embedding -> deletion result

At every transition, record the data, the person or organization concerned, the source, the approved purpose, the processing, the destination, who can gain access, how long it persists, how it can be corrected or deleted, and what evidence proves the claim.

The drawing must include routes that ordinary architecture diagrams omit:

  • uploaded files, retrieved passages, metadata, tool results, hidden system context, and generated output;
  • traces, analytics, support cases, incident captures, abuse-review queues, and human quality review;
  • chunks, indexes, embeddings, caches, feature stores, evaluation sets, feedback datasets, and fine-tuning data;
  • model providers, telemetry vendors, review platforms, subprocessors, exports, and backups; and
  • inferred attributes and profiles, including those about people who never use the product.

A manager may paste information about an employee, a customer record may mention a family member, and a retrieved case file may identify a witness. The account holder is not the complete population affected by the system.

Purpose Must Survive the Transformation

The performance-review assistant needs enough employment evidence to help a manager draft a fair summary. That purpose does not automatically authorize product analytics, general model improvement, an indefinite evaluation corpus, or a search feature over old conversations.

For each new use, ask whether the same job can be done with less data. A category may replace free text. A local identifier may replace a name. A short-lived lookup may replace a durable copy. Aggregate counts and outcome codes may replace raw conversations. An authored fixture may replace a production example in routine testing.

Then make the approved purpose executable. Carry identity, tenant, role, purpose, and classification through retrieval and transformation. Filter fields before they enter model context. Retrieve only passages required for the current request. Keep employee cases out of a general policy corpus. Give reviewers the minimum content needed for their assigned decision.

Consent or another authority identified by the responsible reviewer does not excuse overcollection. Nor does pseudonymization settle the question. Replacing a name with an identifier can reduce casual exposure, but linked records, distinctive facts, or external information may restore identity. Anonymization is a claim that must withstand testing against the data and realistic access available to an attacker or recipient.

The strongest privacy control is often the absent field, copy, or integration. It cannot leak later, drift into a new purpose, or complicate a deletion request.

Storage Pockets Turn One Request into Many Records

AI systems create useful derivatives. Parsing produces chunks. Retrieval produces indexes and embeddings. Operation produces traces. Evaluation produces labeled examples. Human review produces annotations. None becomes harmless merely because it no longer looks like the source sentence.

An embedding is designed to preserve useful relationships in source data. Whether text can be reconstructed under a particular attack is only one concern. The embedding still belongs to a source, purpose, tenant, access model, retention schedule, and threat model. Classify it accordingly, preserve provenance, isolate tenants, and test relevant inference or extraction routes.

Logging presents the same trade-off. Operators need enough evidence to diagnose latency, policy decisions, failed retrieval, and abuse. Full bodies are rarely the necessary default. Prefer request identifiers, component versions, timing, decision codes, and aggregate counts. Allowlist structured fields. Redact before persistence and export. When content is genuinely required for an incident, place it in a restricted capture path with explicit access, short retention, and an owner.

The quality-review queue also needs a purpose decision, not merely an employee with permission. Specify which cases qualify, whether reviewers need full context, how examples are de-identified where appropriate, what may move into an evaluation set, and when the record expires. Access control limits who can see a corpus; purpose limitation decides whether that corpus should exist.

Deletion Is a Distributed Operation

The visible conversation is only the first deletion target. For the employee’s request, the team must locate the source record, parsed cache, trace, review item, evaluation copy, embedding, result cache, export, supplier copy, and relevant backups. Some systems support direct deletion; others require index rebuilds, cryptographic erasure, expiry, or removal from the next backup rotation.

A workable deletion design records four things for every store: the link back to the source, the deletion action, the maximum propagation time, and the evidence of completion. It also defines what the product does while propagation is pending. Search and generation should not keep serving a record simply because one derivative has not yet been rebuilt.

Exceptions require equal precision. A security record, support dispute, or legally required record may follow a different schedule, but “retain as necessary” is not executable. Name the category, purpose, owner, access boundary, triggering event, expiry rule, and review path. Do not promise immediate erasure if backups or supplier systems cannot perform it.

Correction deserves the same lineage. If the employee’s record is amended, decide whether stale chunks, embeddings, generated summaries, evaluation labels, and reviewer notes must be replaced, invalidated, or preserved with an explicit history. A source-of-truth correction that leaves the assistant serving the old claim has not reached the system.

A Supplier Boundary Is a Runtime Boundary

A contract can allocate obligations, but deployed configuration determines behavior. For each provider and subprocessor, establish which data crosses the boundary, for what purpose, from which locations, under which account and product settings, and with what human-review, improvement, retention, backup, incident, and deletion behavior.

Verify the path with technical evidence. Inspect configured settings and request payloads. Test that excluded fields do not leave the service. Exercise export and deletion. Confirm how material provider or subprocessor changes are detected. Plan how the organization will retrieve its data and verify deletion when the relationship ends.

A favorable procurement answer does not prove that production uses the reviewed account, endpoint, region, or logging configuration. Provider, model, purpose, corpus, review practice, logging, and retention changes must reopen the privacy decision when they alter processing.

Close the Employee’s Request

The team redesigns the assistant around a narrower record. The manager selects approved evidence categories before adding bounded text. Health and family details are excluded from general drafting. The system sends the provider only the context needed for the current generation. Routine traces contain identifiers, versions, timing, and outcome codes rather than bodies.

Production conversations no longer enter evaluation by default. Reviewers see sampled cases under a separate purpose, access boundary, and expiry. Approved examples retain lineage to their source. Embeddings are tenant-isolated and removable through a source-to-chunk manifest. The deletion service fans out to the chat store, trace index, review tool, evaluation registry, vector index, caches, and supplier API, then records each result and any timed exception.

The interface explains the material processing in plain language before collection and gives the manager a route for correction or deletion. It does not claim that clicking delete makes every copy vanish instantly. Support can now answer the employee with the systems reached, the actions completed, the bounded exceptions, and the expected propagation time.

The redesign improves more than the notice. It changes what the system is allowed to remember.

AI Privacy Impact Checklist

Use this record before release and whenever the purpose, population, model, provider, corpus, logging, review, or retention design changes:

  • State the user job, approved purpose, prohibited reuse, and less-data or non-AI alternative.
  • Identify affected users and non-users, including people described in prompts, documents, or inferred profiles.
  • Record the applicable authority or basis, notice, rights, contract, and review decisions for the deployment context.
  • Trace prompts, outputs, retrieval, tools, logs, embeddings, caches, review queues, evaluation data, suppliers, exports, and backups.
  • Give every field and free-text route a necessity decision before collection or transfer.
  • Preserve identity, tenant, role, purpose, classification, and source lineage across transformations.
  • Give every retained store an owner, access model, retention trigger, deletion method, propagation bound, and completion evidence.
  • Verify supplier configuration and request behavior against contractual terms, including improvement, human review, retention, subprocessors, and termination.
  • Test cross-tenant access, log and review exposure, deletion and correction propagation, and relevant inference, memorization, or extraction risks.
  • Make human review necessary, scoped, trained, auditable, and time-bounded.
  • Give people plain-language notice and usable correction, deletion, appeal, and incident routes where applicable.
  • Name the accountable release owner, residual risks, monitoring signals, and events that require re-review.

Now vary the design. Suppose the manager pastes the same sentence into an internal search assistant, then into a public model account used outside the approved workflow. Redraw the lineage for each case. If only the provider box changes, the review has missed purpose, identity, retention, and control.

Privacy work is ready for a release decision when the declared purpose can be traced into deployed boundaries and operational evidence—and when the team can explain what happens to one real record from collection through deletion.

Source Notes