Error Tracking Checklist

A Full Error Inbox Can Still Leave You Blind

An error tracker can receive every stack trace and still fail at its job. Ten thousand events may describe one defect. One event may represent the only design partner being unable to finish the workflow they pay for. A handled exception may be ordinary control flow, while a job that returns success may have silently produced the wrong result.

The useful question is not “Are errors being collected?” It is “When a customer capability fails, will I learn soon enough, see enough to act safely, and know whether the repair worked?”

Run this review for one customer journey at a time: sign-in, checkout, import, report generation, delivery, or another capability whose failure has a visible consequence. A solo founder needs a small number of trustworthy signals more than a comprehensive stream of interruptions.

Define Failure at the Product Boundary

Describe the journey in customer terms before opening the tracking tool. Name the successful outcome, an expected rejection, a recoverable failure, and a failure that leaves the customer blocked or uncertain.

This prevents ordinary behavior from becoming noise. An invalid file rejected with a clear explanation is not necessarily an application error. A transient dependency failure that is retried successfully may deserve a metric, but not three unresolved issues. By contrast, a worker that exhausts its retries and strands an accepted job has failed even if no request returned a 500 response.

Trace the journey across every place it can end: browser or mobile client, application server, background worker, scheduled task, webhook, and third-party boundary. Capture a failure where it becomes an operational outcome, and capture it once. Preserve the original cause when wrappers or retries would otherwise replace it with a generic final exception.

Then cause one safe failure in a non-production environment, or with synthetic production data when that is the only honest route. Confirm that the event reaches the intended project and environment with the correct release, operation, and source location. A configured SDK is not proof of a working signal.

Keep the Context That Changes the Next Action

A useful event should let you distinguish a bad release from bad input, one failed attempt from a terminal job, and a single account from a broad regression. Depending on the journey, that may require:

• the application, environment, release, and feature-flag state;
• a low-cardinality operation name such as report_import rather than a raw URL or file name;
• an opaque account or workspace reference when customer scope matters;
• a request, trace, job, or operation identifier that joins the event to authorized logs;
• retry count, dependency result, and the last completed state transition; and
• the exception type, relevant owned stack frames, and time of failure.

More context is not automatically better. Exception messages, stack traces, URLs, headers, request bodies, database statements, file paths, and user-supplied text can contain secrets or personal data. Decide what the tracker is allowed to receive before enabling broad capture. Remove or mask access tokens, session identifiers, credentials, payment data, uploaded content, and direct personal identifiers as close to the application as practical. Restrict access and retention as you would for production data.

Test the scrubber with conspicuous synthetic values. Search the resulting event, breadcrumb trail, attached logs, and replay data for every value that should have disappeared. A privacy rule that has never faced a representative payload is only an intention.

Group by a Cause You Can Repair

The grouping key should be stable enough to collect repetitions and specific enough to preserve distinct remedies. Exception type plus relevant owned stack frames is often a sound beginning. A message containing a customer name, object key, timestamp, or generated identifier is not: it can split one defect into hundreds of issues and leak data into the title.

Customize grouping only after inspecting real collisions and splits. Two failures that share a low-level timeout may need separate groups when they belong to different customer operations and demand different action. Conversely, retries of the same failed job should not create separate work merely because their event identifiers differ.

Tag events with release and operation so a new regression becomes visible without making the release itself part of every grouping key. When the grouping rule changes, check recent events again; a tidy dashboard is not useful if unrelated causes have been collapsed into one issue.

Let Customer Consequence Set the Response

Severity is a response policy, not a synonym for exception class or event count. Decide what happens while calm.

• Interrupt now only when an immediate human action can reduce active customer harm, data loss, corruption, security exposure, or failure of a critical promise.
• Take the next work block when customers are blocked, a release is regressing, or the affected population is growing, but a short delay will not materially worsen the outcome.
• Queue deliberately when the product degrades but has a credible recovery or workaround. Give the issue an owner and a review condition.
• Observe or suppress expected, unactionable, or fully recovered events. Preserve a count elsewhere when the trend still matters.

Use affected operations, affected accounts, failure rate, duration, and customer importance alongside raw volume. Three failures out of four attempts require a different response from three out of a million. A single failure can still be urgent when it blocks the product’s only active pilot or risks irreversible state.

Every interruption needs a destination and an action. If the notification arrives while no useful response is possible, route it to a review queue instead. Repeated alerts that do not change behavior train the founder to ignore the one that should.

Copyable Error-Tracking Review

Use plain language for the customer consequence and exact names for technical fields. “Fix quickly” is not a response target.

# Error-tracking review: [one customer journey]

Customer promise:
Successful outcome:
Expected rejection that should not become an error:
Recoverable failure and visible customer behavior:
Terminal or dangerous failure:

## Capture path
Client, server, worker, schedule, webhook, and dependency surfaces:
Boundary where each terminal failure is recorded:
Known retries or wrappers that could duplicate or hide the cause:
Synthetic failure used to prove delivery:
Observed event link or evidence location:

## Actionable context
Environment and release:
Low-cardinality operation name:
Opaque customer or workspace reference, if justified:
Request, trace, job, or operation identifier:
State transition and retry context:
Exception type and useful owned frames:
Context intentionally excluded:

## Privacy and access
Secrets and personal data that could enter the event:
Scrubbing location and rules:
Synthetic values used to test the scrubber:
Who can access the tracker:
Retention and deletion path:

## Grouping and impact
Cause-oriented grouping rule:
Known causes that must remain separate:
Volume, rate, and affected-customer views:
Interrupt-now condition and immediate action:
Next-work-block condition:
Queued-review condition and date:
Expected events to suppress or count elsewhere:

## Ownership and closure
Owner:
Acknowledgment or response target:
Customer communication trigger:
Mitigation or safe degraded mode:
Regression check:
Evidence that the fix reached production:
Condition for resolving the issue:
Condition for reopening it:
Next signal-quality review:

Do not call an internal target an SLA unless it is actually part of a service commitment. The record exists to constrain action, not to create an impressive label.

Worked Review: The Import That Failed Three Times

An agency reporting product accepts a source file, starts an import job, and later produces a report for approval. The tracker currently emits one ObjectNotFound exception for every retry. Each event title contains a storage key, so one failed import becomes three separate issues. The key includes the original file name, and the event also carries the uploader’s email address. The tracker is noisy, the grouping is useless, and the captured context is more sensitive than the investigation requires.

The founder first defines the product boundary. A malformed file rejected before acceptance is an expected validation result. A temporary storage read that succeeds on retry is counted as recovered dependency trouble. An accepted import that exhausts its retries without producing a report is a terminal failure and must reach the error tracker once.

The terminal event uses the operation name report_import, an opaque workspace reference, job identifier, release, attempt count, and the transition accepted -> processing. It records a low-cardinality cause such as source_missing_after_acceptance; it does not send the file name, object key, email address, source contents, or raw job payload. The founder verifies the scrub with a synthetic file name and email chosen to be easy to find.

Events group by the cause and relevant worker frame rather than the storage key. Release and workspace remain filters. The immediate notification is tied to customer impact: an accepted import has become stranded and no automatic retry remains. Its first action is to place the product in a safe, visible failure state and contact an affected design partner when necessary, not merely to open the stack trace.

The joined job history reveals that a new cleanup path can delete the source after acceptance but before the worker’s final attempt. The repair changes that state transition and adds a regression test that holds the worker, runs cleanup, resumes processing, and proves that accepted input remains available until the job reaches a terminal state. After release, the founder runs a synthetic import through acceptance, processing, approval, and retrieval. Only then is the issue resolved.

The same review also exposes what the error tracker cannot prove. It can show exceptions, but not the denominator of all imports, a silently incorrect report, or a job that remains pending forever without throwing. The founder pairs it with an import success-rate metric, a stuck-job age check, and a synthetic journey. Error tracking is one view of product reliability, not its complete measurement.

Keep the Signal Worth Trusting

Review the top unresolved groups and the notifications from the last operating cycle. For each one, ask whether it represented customer harm, led to a useful action, duplicated another signal, lacked context, or captured data it did not need. Delete obsolete alerts, repair bad grouping, and improve the capture point before adding another dashboard.

The review is complete when a representative failure arrives once, contains enough privacy-safe context to choose an action, reaches the right destination at the right urgency, and can be closed only by evidence from the repaired customer journey. The goal is not an empty inbox. It is a signal the founder still believes when it interrupts the day.