Fairness, Harm, and Impact Assessment

The Average That Hid the Harm

An education platform recommends advanced courses. After a ranking change, both acceptance and completion improve. The product team prepares to expand it.

A regional counselor objects. Learners who share low-bandwidth devices are receiving fewer advanced recommendations. Interrupted sessions look like weak engagement, the ranking system treats weak engagement as lack of readiness, and counselors cannot see why a course disappeared. The aggregate dashboard celebrates improvement while an infrastructure constraint becomes an opportunity constraint.

No single fairness metric discovered the problem. Someone close to the affected workflow did.

Fairness is a contextual governance judgment supported by evidence. An impact assessment makes that judgment inspectable by tracing who benefits, who bears risk, how harm can occur, which alternatives exist, and who may accept what remains.

Follow the Missing Recommendation

The team first needs to reconstruct what happened to one learner. A network interruption changed an engagement feature. That feature lowered a ranking score. The interface withheld an advanced course without showing the counselor the decisive evidence. The learner never saw an adverse decision, so there was nothing obvious to question.

Write this as a causal chain:

condition or design choice → system behavior → human or institutional action → affected outcome → detection or remedy

“The model may be biased” names a concern but gives engineers little to investigate. “Interrupted sessions lower an engagement feature, which suppresses advanced-course recommendations for low-bandwidth learners, while counselors cannot see or override the factor” identifies a pathway that can be tested and redesigned.

Now map the people around that pathway:

• direct users who operate or receive the system;
• decision subjects whose opportunities, resources, rights, or treatment may change;
• indirectly affected people such as coworkers, families, communities, creators, or customers;
• operators and reviewers whose workload, discretion, safety, or employment changes.

Within each group, look for differences in language, disability, age, geography, connectivity, socioeconomic conditions, and other facts relevant to this use. The team should not collect sensitive attributes merely because they might be analytically useful. It needs a stated purpose, authority, protection, retention period, and a safe way to proceed when collection would itself create risk.

The counselor’s report reveals an allocation harm: some learners lose access to an opportunity. Following it further exposes other burdens. The service may work poorly with assistive technology or local languages. More manual review may fall on already stretched counselors. A new connectivity field may create privacy risk. An appeal that exists only in English may preserve the disparity under the appearance of remedy.

This is the practical value of a harm taxonomy: it keeps the inquiry from stopping at the first measurable disparity. Ask about allocation and opportunity; quality of service and accessibility; dignity and representation; privacy and autonomy; safety and security; labor and institutional burden; and effects on the information environment. Use each family to find a plausible pathway, not to fill a row.

For every material pathway, judge severity, scale, duration, reversibility, likelihood, detectability, and the reality of remedy. A rare irreversible loss can deserve stronger treatment than a frequent inconvenience. Weak evidence does not turn risk into zero; it may justify a smaller pilot, better research, or no deployment.

Let the Decision Choose the Measure

The team cannot repair the recommendation system by browsing a catalog of fairness metrics. Equal selection rates, equal true-positive rates, equal false-positive rates, calibration, individual consistency, and equal error cost answer different questions and can conflict. Before choosing among them, define:

1. the decision and affected interest;
2. the favorable and harmful outcomes;
3. the validity of labels and reference decisions;
4. which errors matter to whom;
5. which groups and intersections are decision-relevant;
6. the intervention available if a disparity appears.

Suppose the label for “ready for an advanced course” is completion of a previous online course. That label already contains access to devices, time, connectivity, language support, and earlier recommendations. A historical outcome is not neutral ground truth merely because it sits in a database. Examine how each label was produced and compare the AI-enabled workflow with the actual non-AI process, not with an imaginary perfect system.

Report uncertainty and sample coverage. A small subgroup estimate may be too unstable for a precise conclusion while still revealing that the evidence cannot support broad deployment. In the education case, selection and completion rates by connectivity context are useful, but so are case review, counselor reversals, accessibility tests, complaints, appeals, and what learners say about missing recommendations. The numbers and the lived workflow test different parts of the pathway.

Treat Procedure as Part of Fairness

Outcome metrics do not capture whether people were informed, heard, or able to obtain correction. Procedural fairness includes:

• understandable notice before or when AI materially affects a process;
• a chance to supply relevant information and correct data;
• a qualified human who can disagree with the system;
• consistent reasons for decisions and exceptions;
• accessible appeal, service targets, and protection from retaliation;
• records sufficient to investigate patterns and individual cases.

Participation should influence scope, requirements, test cases, launch limits, monitoring, and remedy. The counselor’s observation arrived before expansion and changed the investigation; that is more consequential than a survey after the design is irreversible. Include learners who were never shown a course, not only users who accepted one. Compensate community or domain expertise where appropriate, explain what changed, and preserve reasoned disagreement.

Make the Assessment Change the Release

Impact assessment is useful at intake, before launch, after material change, following an incident, and during periodic review. Scale the depth to consequence, autonomy, reach, novelty, evidence uncertainty, and difficulty of remedy.

For each significant pathway, record:

• affected people and the benefit or harm pathway;
• existing evidence and material uncertainty;
• baseline and reduced-scope or non-AI alternatives;
• preventive, detective, corrective, and remedial controls;
• evaluation measures, segment coverage, and acceptance thresholds;
• residual severity, likelihood or uncertainty, and exposure;
• owner, decision authority, conditions, expiry, and review trigger.

The assessment is complete only when it can change the release. Its decision may be to proceed, limit, redesign, pilot, reject, pause, or retire. A total score that always reduces those choices to green, amber, or red can hide a severe harm behind several tidy controls.

Repair the Pathway Before Polishing the Metric

Use a hierarchy:

1. avoid: remove the AI role or prohibited purpose;
2. reduce scope: narrow population, action, autonomy, data, or exposure;
3. redesign: change objective, data, model, interface, workflow, or incentive;
4. control: add thresholds, oversight, monitoring, security, and fallback;
5. remedy: correct outcomes, restore access, compensate where appropriate, and learn;
6. accept: authorize only bounded residual risk with conditions and expiry.

The weak response is to add device type to a dashboard and tune a threshold until selection rates converge. The engagement proxy remains, counselors still cannot inspect the recommendation, and learners still have no route to correction. The metric improves while the pathway survives.

The stronger response removes network-interruption events from the engagement feature and asks whether engagement is a defensible measure of readiness at all. It tests recommendation and completion outcomes by relevant connectivity context, exposes the recommendation evidence to counselors, and gives them recorded authority to override. Learners can inspect prerequisites, request reconsideration, and choose a non-personalized route.

The first release stays limited. Monitoring covers missing recommendations, counselor reversals, time to remedy, course outcomes, and new workload on staff. If appeals rise because the redesign merely moved the burden to counselors, the assessment reopens.

Fine-tuning or threshold adjustment can shift a metric without fixing the pathway. Every mitigation needs a side-effect test: lowering one group’s false-negative rate may increase delay, privacy collection, manual workload, or another error. Name who receives the benefit and who absorbs the cost.

AI Impact Assessment Record

• System and decision: purpose, AI role, workflow, boundaries, autonomy, scale, regions, versions, and owners.
• Stakeholders: users, decision subjects, indirect groups, operators, representatives, and participation method.
• Benefits and harms: causal pathways, distribution, severity, scale, duration, reversibility, detectability, and remedy.
• Evidence: data provenance, label validity, segment and intersection results, qualitative findings, incidents, and uncertainty.
• Alternatives: current process, non-AI, reduced scope, different architecture, delay, or no deployment.
• Controls and mitigations: design changes, tests, owners, thresholds, monitoring, appeal, and remediation.
• Residual risk: decision, authority, rationale, dissent, conditions, exposure limit, expiry, and stop triggers.
• Reassessment: material changes, production signals, stakeholder feedback, cadence, and publication or disclosure plan.

Disturb the Decision

Complete a lightweight impact assessment for the education recommender, then disturb it three times.

First, the selection disparity narrows, but learners using screen readers abandon the prerequisite review page. Next, counselors use overrides frequently in one region, but completion there improves. Finally, collecting connectivity context would improve measurement while creating a sensitive record the platform cannot yet protect.

For each change, redraw the harm pathway. Decide which evidence is missing, whose knowledge should influence the decision, what may ship, who bears the cost of the mitigation, and which authority can accept the residual risk. A copied assessment should fail this exercise: the changed facts must change the record or the release.

Fairness becomes governable when the organization can follow a benefit or burden from system design into a person’s opportunity, test that account against evidence and affected-party knowledge, and alter the system in response. The record does not certify that the choice is morally clean. It preserves the reasoning, authority, limits, and reopening conditions that the next chapter’s legal and policy review must evaluate against duties outside the system itself.

Source Notes

• NIST AI RMF Core calls for beneficial and harmful impacts to individuals, groups, communities, organizations, and society to be characterized and for relevant external feedback to inform lifecycle risk management; voluntary guidance verified 2026-07-20. NIST states that AI RMF 1.0 is being revised.
• NIST AI RMF Playbook provides voluntary suggested actions for impact assessment, affected-community engagement, disaggregated analysis, measurement, and continuing risk management; verified 2026-07-20.
• Fairness definitions, protected classes, impact-assessment duties, consultation, and remedies vary by jurisdiction, sector, role, and contract. Qualified reviewers should determine applicable obligations.
• See Robustness, Fairness, Bias, and Segment Performance for measurement design and Accountability, Transparency, and Explainability for notice and challenge routes.