Retiring, Archiving, and Deleting Data Systems

The Index Nobody Wants to Delete

A catalog team has two search indexes for the same product documents. The newer one serves customer search. The older one belonged to an internal tool that was retired six months ago, yet it still receives every catalog update, appears in the on-call dashboard, and accepts a credential shared by support. Its monthly bill is easy to see. What nobody can see is whether anything still needs it.

Deleting it today might break a quiet fraud notebook or an emergency support path. Keeping it indefinitely preserves seller notes in an unnecessary store, gives privacy deletion one more destination to reach, and makes responders reason about an index that should no longer shape production. Both instincts—delete the obvious waste, keep the possible dependency—are reactions to missing evidence.

Chapter 32 ended with a dependency known well enough to retire. Knowing a consumer, however, is only the beginning. Retirement must separate what the old system does from what anyone is still entitled to receive from it. A search endpoint can disappear while a retained record survives elsewhere. An export job can stop while a legal hold preserves a bounded subset of its files. A database can be deleted while tombstones remain long enough to prevent a restore from resurrecting deleted users.

The question is therefore larger than “Can we delete it?”

After this system disappears, which consumers, retention duties, deletion duties, recovery paths, audits, and operators still need something from it?

That question turns decommissioning into the last phase of system design. The work is to preserve the obligations that remain, remove the mechanisms that do not, and leave proof strong enough that a future operator need not rediscover the old system.

Why Systems Linger

Systems linger for reasons that are rational one decision at a time and expensive when combined.

Unknown consumers are the usual blocker. A table may be read by dashboards, fraud jobs, notebook exports, support scripts, incident playbooks, partner feeds, feature training pipelines, or one monthly compliance report. The owning team may see no traffic from the main application and still be unable to prove the system is unused.

Fear of deletion is often justified. The cost of breaking a quiet dependency is immediate and personal. The cost of leaving an obsolete copy online is distributed across storage, privacy exposure, access review, on-call confusion, and future migrations. Teams keep the old system because nobody has built the evidence needed to remove it safely.

Retention ambiguity encourages hoarding. Teams keep data because they do not know what is legally required, contractually required, useful for operations, subject to legal hold, requested for deletion, or safe to aggregate. “Keep it just in case” feels conservative until a breach, subpoena, deletion request, or audit asks why the data existed.

Ownership decay makes retirement invisible. The team that created the system may have dissolved. The service account may be shared. The dashboard may be important but unowned. The budget may sit with one group while operational risk sits with another. If nobody is accountable for cost, access, retention, deletion, and consumer support, the system becomes nobody’s priority.

Operational habit completes the trap. Old queues stay in dashboards. Backup jobs continue to run. Credentials keep rotating. Incident diagrams still show paths that no longer serve users. The application may be product-dead while every surrounding control treats it as alive.

Retirement Candidates

Age is a poor retirement test. The useful candidates are systems whose serving job has ended, whose obligations are unclear, or whose continuing cost and risk exceed their value.

Unused indexes are a common first target. They add write overhead, storage cost, backup cost, query-planner complexity, and migration drag. Before dropping one, inspect query history, slow-query samples, explain plans, emergency scripts, migration tools, reporting jobs, and low-frequency workflows. An index unused by ordinary traffic may still support an end-of-month export or a support escalation path.

Duplicate stores deserve scrutiny when they no longer have a distinct serving job. A duplicate search index, cache, read replica, or reporting table should have a named freshness promise, consumer set, owner, and recovery plan. If it has none, it is carrying risk without a product contract.

Old exports are high-risk because they often bypass the controls of the source system. File buckets, partner feeds, scheduled emails, downloaded CSVs, and warehouse extracts can leak sensitive data long after the integration that created them has ended. They need retention, access, and deletion review, not only cost review.

Dead queues and topics often survive migrations. They may still receive messages from old producers, accumulate poison messages, retain sensitive payloads, trigger replay jobs, or confuse incident responders who cannot tell whether the backlog matters.

Legacy replicas and regional copies can preserve stale security boundaries. If a copy is no longer part of serving, failover, analytics, or recovery, it should have a retirement plan.

Derived tables and model features deserve the same scrutiny. A denormalized table, feature-store value, or materialized view may no longer power a product surface but may still preserve personal data, bad semantics, or access grants. The common test is not whether a component looks idle. Ask which serving, analytical, audit, recovery, or retention obligation would fail if it vanished. If nobody can name one, uncertainty becomes the first fact the retirement must resolve.

The Decommission Control Loop

Retirement needs a control loop because the team begins with incomplete knowledge. Each step should reduce uncertainty before the next step increases irreversibility.

Start with a candidate record. Name the system, owner, purpose, data classes, known consumers, current cost, alerting surface, retention class, access grants, backup behavior, and suspected replacement. The record makes retirement reviewable. “Old search index” is not enough. “Product document index products_v1, written by catalog-service, queried by legacy support search, contains public product metadata plus seller notes, retained in snapshots for 14 days” is a starting point.

Discover consumers through more than one signal. Use query logs, API logs, service-account activity, lineage metadata, dashboard metadata, job schedules, orchestration history, access grants, tracing, network flow logs, repository search, runbooks, and owner interviews. Discovery should produce accountable owners, not only technical identifiers. A service account named analytics-reader is a clue, not a consumer.

Freeze new use before removing old use. Mark the system deprecated, remove it from provisioning templates, block new credentials, reject new dashboards, stop adding producers, and route new integrations to the replacement. Without a freeze, the retirement project races fresh dependencies.

Announce the retirement with enough detail for a real consumer to act. Name the system, impact, replacement path, freeze date, observation window, disable date, archive or deletion date, owner, escalation path, and testing instructions. A vague announcement teaches consumers to ignore retirement notices. A useful announcement lets a dashboard owner, partner engineer, or support lead verify whether they are affected.

Observe access during a defined window. Monitor reads, writes, credentials, network connections, dashboard refreshes, scheduled jobs, support scripts, partner calls, and operational tools. The window should cover the system’s rhythm: daily traffic, weekly reporting, month-end jobs, billing cycles, seasonal jobs, and known incident procedures. Any access should be classified as expected, migrated, blocked, or exception-approved.

Disable before deleting when possible. Make the system read-only, remove a route, revoke a credential, pause a topic, disable a dashboard, stop a writer, or remove DNS while rollback is still cheap. A reversible disable step turns hidden dependency discovery into a controlled exercise instead of a destructive surprise.

Archive only what has a retention reason and a retrieval procedure. An archive without owner, index, encryption, access control, restore steps, and deletion rules is not a compliance asset. It is a slower production liability.

Delete with proof. Record what was deleted, from which systems, under which retention or deletion rule, when, by whom or by which job, how derived stores were handled, how backups are treated, and what verification evidence exists.

Verify after deletion. Check that access fails as intended, alerts and runbooks no longer reference the system, credentials are removed, cost actually drops, derived stores stop receiving stale writes, backups expire on schedule, and architecture diagrams no longer imply an active path.

Data Deletion

Deletion is not one operation. It is a set of propagation rules across the source of truth, derived stores, logs, backups, archives, third parties, and future restores.

Legal retention and user deletion obligations can point in different directions, so classify records before acting. Some must be retained for a contract, audit, tax rule, legal hold, or operational obligation; preserve those in controlled storage with an owner, access path, and expiration rule. Some must be deleted because of a user request, policy, expired retention period, contract, or risk decision; remove those from source and derived stores and verify the propagation. Raw data may sometimes be replaced by an allowed aggregate, redaction, or anonymized form, but only after considering reidentification risk. Everything else is merely eligible to retain. “We might need it” becomes a decision only when a named owner accepts its cost, access, and eventual deletion.

User deletion requests rarely stop at the source of truth. Caches, search indexes, materialized views, analytics tables, exports, logs, feature stores, ML training sets, backups, and third-party sinks may contain derived copies. The deletion design needs propagation, verification, and exception handling.

Tombstones help asynchronous deletion. A tombstone records that an entity was deleted so downstream systems can remove or ignore stale copies. Tombstones need enough identity to match derived records, enough metadata to drive routing, and enough retention time for late consumers, replays, and disconnected sinks. Tombstones also need access controls; a deletion marker can itself reveal that a user or account existed.

Backups complicate deletion because their purpose is to preserve old state. A practical policy should name backup retention windows, restore procedures that reapply deletion events, and cases where deletion from historical backup media is infeasible or handled through expiration and restricted access. A restore drill should include the question: after restoring an old snapshot, how are deletions replayed before the system serves traffic?

Derived stores need deletion semantics that match their role. A cache can evict. A search index can delete a document. A denormalized table may need row deletes or partition rewrites. An aggregate may need recomputation. A feature store may need point-in-time rules so training data does not reintroduce deleted personal data. A log may need redaction, encryption-key destruction, shortened retention, or a documented exception.

Deletion proof should be specific enough for an audit and useful enough for an incident. Useful proof includes the entity scope, system list, deletion job version, run identifier, start and finish times, counts, failures, retries, exception approvals, backup policy reference, and verification query results. “Job succeeded” is not enough if nobody can say which stores it touched.

Archival Design

Archiving is not the polite word for keeping everything. It is a design choice for data that no longer belongs in active serving paths but still has a legitimate future retrieval need.

An archive needs a retrieval contract. Define the data classes, retention class, retention clock, encryption, access approval, index, owner, expected retrieval time, restore environment, and deletion behavior. If a support, legal, finance, or incident response workflow depends on the archive, that workflow should know how to request and validate retrieval.

The retention clock must be explicit. Some clocks start at collection. Some start at account closure, contract termination, transaction completion, invoice issuance, consent withdrawal, legal hold release, or final customer interaction. Ambiguous clocks create permanent archives because no job can prove when deletion is allowed.

Archive metadata should outlive the retired system. Store schema version, source system, ownership, retention class, data classification, encryption key reference, access approval route, legal holds, deletion exceptions, and restore steps somewhere the old system does not own. Otherwise the archive becomes unreadable precisely when the old service, repository, and team vanish.

Retrieval procedures must be tested. If restoring an archive requires tribal knowledge, expired credentials, unknown schemas, hand-edited scripts, or a service that was decommissioned with the source system, the archive will fail when it matters. A small retrieval drill is cheaper than discovering during litigation or an incident that the archive is a pile of encrypted bytes nobody can interpret.

Access should narrow over time. Retired data usually has fewer legitimate users than active data. Archive credentials, keys, approval paths, audit logs, and break-glass procedures should reflect that. Moving data to cold storage while preserving broad production access only changes the storage tier, not the risk.

Archive format matters. Prefer durable, documented formats over formats that require a retired application stack to read. Preserve schema documentation, encoding, partitioning, checksums, and sample validation queries. For analytical archives, include enough indexing or manifest data to retrieve a bounded record set without scanning the entire archive for every request.

The Search Index Goes Dark

Return to the old catalog index. Its retirement record names the writers, credentials, data classes, snapshots, dashboards, cost, and replacement API. Discovery finds three read paths: an old support script, a weekly dashboard that still refreshes but has no viewers, and one fraud notebook that uses a copied credential. The team migrates the support script, deletes the dashboard after owner approval, and gives fraud a supported endpoint with narrower fields.

New use is frozen by removing the index from templates, denying new credentials, and adding a deprecation warning to internal docs. Access logs run for two business cycles, including month end. The team watches reads, writes, credential use, dashboard refreshes, and failed connection attempts.

The disable step happens before deletion. Writes are stopped first so the index can no longer drift as an accidental serving path. Reads are blocked for normal credentials while an emergency rollback switch remains available for a short window. No unexpected access appears. The old index is not archived because the source database, current index, and documented snapshots satisfy retention and recovery requirements. Keeping a searchable archive of seller notes would increase exposure without adding an obligation.

Deletion then removes the index, snapshots after their retention window, credentials, dashboards, alerts, runbook references, and architecture-diagram edges. The proof packet records deleted resources, owner approval, access-log evidence, replacement path, verification queries, expected monthly cost reduction, and the decision not to archive.

The important result is not only lower spend. Incident responders now have one search path to reason about. Privacy deletion has one fewer derived store. Security has fewer credentials to review. Future schema changes no longer have to update an index that no user should reach.

Retention Ambiguity in a Customer Export

A customer success team asks whether an old nightly export can be removed. The export writes account activity to object storage for a vendor integration that ended six months ago. The bucket still contains years of files. Nobody knows whether finance, support, or legal uses it.

The unsafe answer is either “delete the bucket” or “keep it forever.” The production answer is classification. The team separates the files by customer, date, contract, data class, and known legal holds. They confirm that the vendor integration has ended, revoke vendor access, and inspect access logs. They find one internal notebook reading the files for a churn analysis that can move to the warehouse, and one legal hold covering a small customer set.

The retirement plan deletes files outside retention, preserves the legal-hold subset in a controlled archive, records the retention clock, and creates a retrieval procedure that does not depend on the old export job. It also removes the scheduler, service account, bucket write policy, partner credentials, monitoring, and documentation. The archive is not a general-purpose substitute for the old export; it is a bounded record set with an owner, approval path, and expiration rule.

This is the pattern to copy when retention is unclear: narrow the obligation before preserving data. Uncertainty should trigger classification, not permanent storage.

Post-Retirement Review

Retirement is complete only after the old system stops shaping production behavior. The review should answer four questions.

Did consumers move cleanly? Check error rates, support tickets, dashboard failures, partner escalations, query logs, and failed authentication attempts after deletion. Hidden consumers often show up as noise in places outside the owning team’s dashboard.

Did obligations survive? Confirm that retention records, archive retrieval, deletion propagation, legal holds, restore behavior, and audit evidence match the plan. If the system held personal data, verify that deletion workflows no longer reference a removed target without replacing it.

Did risk actually decrease? Remove stale alerts, credentials, IAM roles, firewall rules, network routes, backup jobs, dashboards, runbooks, diagrams, ownership records, and cost tags. A deleted table with live credentials and stale runbooks is an incomplete retirement.

What should future systems learn? Many retirements reveal design debt: no owner metadata, shared service accounts, missing query attribution, unclear retention clocks, no deletion manifest, or archives tied to application code. Capture those lessons in platform templates, data contracts, provisioning policy, and review checklists so the next system is easier to retire.

A Record That Can Stop the Deletion

A useful decommission record is filled with evidence, not prompts. The old index’s record might look like this just before approval:

candidate: products_v1 search index; catalog-service writer; seller notes present
owners: catalog platform (operation); support and fraud (consumers); privacy (review)

known consumers: support script moved to catalog-support API
                 weekly dashboard deleted with owner approval
                 fraud notebook moved to restricted fraud-search endpoint
unknown access: none during two business cycles including month end
freeze: removed from templates; new credentials denied; replacement documented

disable: writes stopped 2026-06-02; ordinary reads denied 2026-06-09
rollback: break-glass read credential held by incident commander until 2026-06-16
archive decision: none; source database and current index satisfy the serving duty
                  snapshots expire after 14 days; retaining seller notes adds no duty

deletion scope: index, snapshots at expiry, credential, alerts, dashboard,
                runbook entry, architecture edge, privacy-deletion destination
proof: access-log query; credential inventory; resource deletion event;
       failed-read probe; cost report; privacy propagation test

The record should be allowed to say “stop.” An unidentified credential, an observation window shorter than the business cycle, an untested archive, a legal hold without a bounded record set, or a restore path that can resurrect deleted data is not paperwork to complete later. It is the reason deletion is not yet safe.

For personal or regulated data, attach a propagation trace rather than a generic claim that deletion is handled. Begin at the source of truth: identify the authoritative deletion state and how it is ordered against concurrent writes. Follow the entity into caches and search indexes, naming eviction, document identifiers, retry behavior, and protection against refill from stale sources. Follow it through streams and CDC, including the tombstone’s retention and the way late consumers receive it.

Then account for stores whose removal is less direct. Analytics may require row deletion, partition rewrite, aggregate recomputation, or an allowed transformation. Feature systems must exclude the entity from both serving values and point-in-time training data. Logs may require redaction, shorter retention, key destruction, or a documented exception. Backups need an expiration window and a restore procedure that reapplies deletion before serving. Archives need a valid retention rule and expiry. Third parties need instructions, acknowledgements, and retry follow-up.

The trace is complete only when a verifier can start with one deletion request and find its result or explicit exception at every destination. A system inventory without entity-level evidence describes intent; it does not prove deletion.

Retirement Drill

Pick one duplicate store, old export, unused index, dead topic, or legacy replica. Write a retirement plan that names the owner, discovers consumers, freezes new use, observes access, disables safely, classifies retention, archives only what deserves retention, deletes derived copies, verifies proof, and removes operational residue.

Before accepting the plan, walk through three failures:

1. A monthly job reads the system after three quiet weeks.
2. A user deletion request arrives for data already moved to an archive.
3. A restore from an old backup reintroduces data that had been deleted.

A retirement plan that cannot detect, prevent, or repair those failures is still a deletion script. The discipline is not deletion for its own sake. The discipline is reducing production risk while preserving the facts and obligations that still matter.