Skip to content
PracticalHandbooksSoftware Handbooks
Library

AI Systems Handbook / Chapter 46

Environmental and Societal Considerations

Assess compute, labor, information, inclusion, and power effects proportionately across an AI-enabled system's lifecycle.

The Assistant That Saved Minutes and Added a Night Shift

A logistics company gives dispatchers an AI assistant that summarizes disruptions and recommends schedule changes. The pilot reports faster case handling and lower inference cost than the original design. Three months later, dispatchers are correcting poor recommendations after their shifts, supervisors are using activity logs to compare workers, and a single external model provider has become difficult to replace. None of those effects appears in the product dashboard.

The model metric was real. The system accounting was incomplete.

Broader-impact review is disciplined system accounting: identify material effects on compute, work, information, inclusion, and power; compare feasible alternatives; mitigate what the organization can influence; assign decisions; and revisit them with operating evidence.

An AI-enabled system is examined through four lenses: compute, work, information, and power. A response sequence moves from measurement to comparison, mitigation, and recurring review.
Broader impacts become governable when teams examine a bounded system through explicit lenses, compare alternatives, assign mitigation, and reopen the decision as evidence changes.

Bound the Decision Before Counting Effects

Every digital service has external effects. That does not justify the same review for a private meeting summary and a national benefits screen. The dispatch assistant deserves more than a paragraph because it reaches an essential service, changes workers’ authority, runs continuously, and can fail during disruptions when recovery is hardest.

Start with the operated service: dispatchers, schedules, retrieved data, model calls, deterministic checks, review queues, telemetry, provider, and fallback. Then name a credible baseline. For this team, the alternatives are the existing manual workflow, rules and search, a smaller model that drafts summaries, and a larger model that also recommends schedule changes. Comparing models while holding none of the surrounding work constant would answer the wrong question.

Scale the inquiry by consequence, reach, duration, resource intensity, novelty, uncertainty, and affected people’s ability to refuse or recover. A bounded internal tool may need a short reflection in its use-case record. A high-impact public system may require measured resource data, worker and community participation, independent review, contractual controls, and an executive risk decision. Imperfect measurement calls for stated uncertainty and a plan to learn; it does not make the effect disappear.

Follow One Completed Dispatch Through the System

The first pilot report divides inference spend by API calls. That denominator rewards cheap calls even when retries and unusable answers multiply. The team instead traces a completed dispatch case from disruption alert to an accepted schedule update.

The trace includes model development and repeated experiments; production calls, retries, and discarded outputs; retrieval, storage, networking, observability, and safety checks; the dispatcher and exception queue; and, where material and knowable, hardware, region, energy source, cooling, and water context. It also records the activity the service avoids. A remote coordination tool may replace travel; a new drafting feature may stimulate far more messages. Both the avoided baseline and the rebound belong in the comparison.

Decision-useful measures can include compute-hours, measured energy, estimated emissions with method and location, water where material, utilization, and requests per completed task. The denominator should connect resources to an accepted outcome. Energy per token can improve while total energy rises because the service creates more calls, longer outputs, or new demand.

Now architecture becomes part of the impact decision. The team tests rules and search, a smaller drafting model, and the larger recommendation model against the same quality, safety, accessibility, and recovery thresholds. It also tests caching, batching, shorter context, lower precision, regional scheduling, and fewer speculative calls. An efficiency change that removes an accessible interface or pushes more judgment into an understaffed review queue has merely moved the cost.

The Night Shift Is Part of the Architecture

The same case trace exposes what the latency dashboard omitted. Routine recommendations are quick, but rare disruptions produce plausible schedules that dispatchers must untangle after their shifts. Supervisors have begun using detailed activity logs to rank individuals, although those logs were collected to diagnose the service. “Productivity” says nothing about who received time, who inherited correction work, or who acquired power.

Map tasks before and after deployment. Which tasks disappear, grow, or move? Who checks failures, at what hour, with what staffing and service target? Which telemetry is necessary to operate the system, who can see it, and which managerial uses are prohibited? Can a dispatcher override or challenge an output without retaliation, and does the challenge get resolved? Where do the gains appear—in safer work, customer outcomes, headcount reduction, or an unmeasured queue?

These questions require worker participation while the workflow can still change. Dispatchers know which exceptions are safety-critical and which recommendation will be treated as compulsory despite an “advisory” label. Give them system facts, paid time to test realistic disruptions, a protected route to raise concerns, and a recorded response. A demonstration followed by a survey does not supply that influence.

The redesign removes individual productivity rankings from telemetry, limits access to operational debugging and aggregate capacity planning, and staffs a daytime exception queue. The intervention is architectural, operational, and managerial because the harm crossed all three boundaries.

Follow the Output Beyond Its First User

The assistant’s summaries also travel to drivers and depot staff. Compression may obscure uncertainty, a poor translation may change an instruction, or a correction may reach the dispatcher but not the copied message. In other systems the same path can flood a channel with low-cost content, amplify false claims, enable impersonation, or make reliable material harder to find. It can also improve translation, description, search, and access for people previously excluded.

Map creation, distribution, reliance, and correction. Test source support, identity misuse, disclosure comprehension, volume, moderation load, language quality, accessibility, and whether a correction reaches every material destination. Synthetic media and public-interest content need provenance and labeling joined to an operational withdrawal route; see Intellectual Property, Content, and Provenance.

Accessibility claims need evidence from the conditions of use: assistive technology, relevant languages, varied literacy, noisy environments, intermittent connections, and people with different sensory, cognitive, and motor needs. A spoken summary may help a driver who cannot look at a screen and fail one who cannot hear it. Inclusion is a design question with more than one affected group, not a credit that cancels other weaknesses.

For systems that materially affect a community, participation must extend beyond employees and direct customers. Identify who receives the service, its errors, its resource demands, and its infrastructure burden. Give affected groups enough information and time to influence siting, access, fallback, remedy, or the decision not to deploy. Consultation after procurement can document opposition; it cannot recover a choice that has already vanished.

Dependence Changes the Decision

The dispatch team relies on one provider for the model, safety policy, hosting, evaluation access, and incident information. A price or model change would affect operations immediately, yet the pilot treated the provider as an interchangeable API.

The review therefore records which prompts, evaluations, logs, and workflow components are portable; what a model, policy, region, price, or service change would break; whether an alternative meets minimum requirements; how long migration would take; and who would fund it. Dispatchers retain a workable non-AI route during outages and a way to challenge decisions without appealing to the provider.

Exit planning is not a promise to avoid vendors. It reveals bargaining asymmetry and gives the organization a response when the provider’s incentives diverge from the service’s purpose. Those dependencies pass naturally into the operating model in the next chapter: someone must own the evidence, fund the fallback, and have authority to reopen the decision.

Choose the Complete Workflow

The original review chose the cheaper of two model APIs. The revised review compares the manual baseline, rules and search, drafting with a smaller model, and automatic recommendations with a larger one. Each option is evaluated on completed cases, serious errors, correction time, energy per accepted case, accessibility, after-hours work, surveillance exposure, portability, and fallback.

Rare disruptions decide the result. Automatic recommendations save the most time in routine cases but create the heaviest correction burden when schedules are already unstable. The team chooses drafting plus deterministic checks, retains human schedule authority, narrows telemetry, staffs exceptions, and sets a six-month provider-dependence review.

This workflow is not impact-free. It is preferable on declared criteria, under present evidence, with named conditions that can invalidate the choice.

Broader Impact Reflection

  • System and baseline: bounded service, purpose, scale, locations, lifecycle, affected groups, and credible alternatives.
  • Compute: experiment and production resources, measurement method, uncertainty, normalized outcome, efficiency options, and rebound risk.
  • Work: task changes, hidden labor, skills, surveillance, participation, benefit distribution, transition, and recourse.
  • Information and inclusion: access, accessibility, languages, integrity, provenance, misuse, correction, and distribution effects.
  • Power: provider concentration, portability, bargaining asymmetry, affected-person choice, community impact, and exit plan.
  • Decision: material impacts, mitigations, owners, accepted residual effects, dissent, evidence gaps, stop conditions, and review date.

Use the reflection on three disturbances before approving the dispatch design: demand triples, the provider retires the selected model, and dispatchers report that correction work has moved into unpaid hours. For each disturbance, decide which evidence is now stale, who bears the new cost, who can change or stop the service, and which alternative must be reconsidered. If the record cannot route those decisions, it is a description of impacts rather than a governable commitment.

Source Notes