OT, IoT, and Embedded Systems Playbook

An engineer discovers that a vendor maintenance account can reach every conveyor controller at a packaging plant and proposes quarantining the fleet at the network switches. The operations lead stops the change: several cells enter a hard stop when supervisory communication disappears, clearing a jam requires a lockout procedure, and the night shift has no controls technician on site. A fast security action could turn a reachable controller into stopped machinery that people must approach and reset.

That disagreement is the beginning of an OT security review, not evidence that safety and security are opposed. Systems that sense or act on the physical world require a different order of work: establish the physical consequence and the authority to change operating state, then constrain digital authority without defeating the safety design. Every identity, update, network rule, log, incident action, and retirement decision has to survive that test.

Draw the physical boundary first

The conveyor controller does not act alone. It receives recipes from an engineering workstation, status polls from a supervisory system, maintenance commands through a vendor jump path, firmware from an update service, and signals from field sensors. Its outputs affect motors and actuators. Independent safety relays may stop motion, but only if they are truly independent, correctly wired, tested, and not bypassed during maintenance.

Before discussing malware or controls, name four things:

• the physical states the system can create, including unsafe, degraded, stopped, and manually controlled states;
• the interlocks and fallback procedures that bound those states;
• the people authorized to change operating state or declare it safe;
• the digital paths that can alter commands, logic, firmware, configuration, or the operator’s view.

This changes the usual priority list. Confidentiality still matters: device data may reveal production, occupancy, location, or personal behavior. Integrity and availability, however, can become immediate safety properties. A forced reboot, automatic quarantine, credential rotation, or network block that is routine in enterprise IT may remove operator visibility, strand a device between states, or trigger hazardous manual work.

The review therefore needs an operations owner and a safety owner as well as security and engineering owners. It needs approved maintenance windows, stop authority, manual fallback, recovery criteria, and a record of which actions may be taken remotely. A control that cannot be used during the conditions it was designed for is a wish, not a control.

Give every device a revocable identity

The plant’s inventory lists controller serial numbers, but the vendor account authenticates a technician to the whole fleet. Neither fact answers the incident question: which device accepted which authority?

A managed device needs a cryptographic identity bound to an inventory record. That record should connect model and hardware revision, current firmware, owner, site and cell, expected network segment, certificate or key, update channel, support status, and the physical function affected. Provisioning should reject factory-default passwords, shared fleet credentials, cloned certificates, and undocumented installer accounts.

Identity has to be replaceable without replacing the machine. Plan how a certificate or key is enrolled, rotated, revoked, and recovered when a device is offline. Moving a controller to another cell, returning it for repair, taking it from a spares cabinet, or transferring ownership should require a new authorization decision. Otherwise inventory records describe where a device used to belong while its credentials continue to work anywhere.

For the conveyor fleet, responders can first close the shared vendor entrance and issue individual, expiring access through the jump path. They should not yet revoke every controller certificate: some certificates also authenticate telemetry or update retrieval, and mass revocation could remove the evidence and recovery path they need. The identity map makes selective containment possible.

Make startup and updates part of one trust story

Secure boot answers a narrow but consequential question: will this device start software authorized by the device owner or manufacturer? Where the platform supports it, verification should extend from a protected root of trust through the bootloader and operating image to the application and security-relevant configuration. Rollback protection matters because a correctly signed old image may restore a known vulnerability.

Verification failure is also a physical-state decision. A controller that simply halts may leave a process in a worse state than one that enters a constrained recovery mode while independent safety functions remain available. The intended behavior must be designed with operations and safety engineers, then tested with tampered images, broken configuration, interrupted power, and failed recovery media. “Fail closed” is incomplete until the design names what closes and what the physical process does next.

The update service extends this boot trust across the fleet. It should accept only signed, versioned packages authorized for the exact model, hardware revision, site or population, and allowed version range. Separate building, signing, release approval, distribution, and field deployment where the organization can do so. Protect signing keys separately from ordinary build credentials and retain the build inputs, signing event, release manifest, compatibility decision, rollout approval, and observed result.

Suppose the controller vendor now reports that an update agent accepts a replayed older package. The fix itself must travel through the vulnerable mechanism. The plant does not solve this by choosing between “patch now” and “wait for the next outage.” It narrows the update path, blocks unapproved sources, verifies whether rollback prevention exists at boot, tests the fixed image on the correct hardware revision, and stages one low-consequence cell during an approved window. The rollout stops if boot verification, safety-state signals, operator visibility, command response, or telemetry reconciliation departs from the agreed baseline.

Firmware update security checklist

Before authorizing a rollout, require a release record that answers these questions:

• What source, build inputs, and approval produced the package, and which protected key signed it?
• Which models, hardware revisions, sites, and current versions may accept it? Which must reject it?
• Does the device verify the signature and package integrity before changing persistent state, and does it prevent unauthorized downgrade?
• What happens after interrupted power, lost connectivity, duplicate delivery, failed verification, or a device that has missed several releases?
• Which cell or cohort receives the update first, what observations permit expansion, and what conditions stop the rollout?
• Can the device return to a known safe image without bypassing rollback protection or the physical safety design?
• Do update results reconcile against the device inventory, including devices that never reported success or failure?

A signing log alone proves that somebody used a key. The release manifest, device-side verification test, staged observations, exception list, and inventory reconciliation together show that the intended fleet received the intended software.

Reduce reachability without losing control

The safest network is not automatically the one with the fewest packets. Operators may need current process state; controllers may need tightly bounded peer communication; an update gateway may need to buffer packages for disconnected devices. Segmentation should preserve those required flows while making accidental or hostile reachability explicit.

Separate enterprise users, engineering workstations, supervisory systems, controllers, safety domains, vendor access, update infrastructure, and management services according to consequence as well as convenience. Default-deny rules should name source, destination, protocol, direction, and operating condition. Vendor maintenance should pass through a controlled path with an individual identity, strong authentication, approval, expiry, and session evidence. A permanent vendor VPN into a flat controller network turns one external account into fleet authority.

Test the denied paths. A diagram is a claim; connection tests and rule telemetry show whether an office laptop, compromised engineering workstation, unmanaged wireless bridge, or vendor session can actually cross the boundary. Also test the allowed path during degraded operation. A firewall change that blocks direct controller access but silently breaks time synchronization or update status can erase evidence at the moment it matters.

Physical access forms another network edge. Field devices may expose storage, reset controls, serial consoles, debug headers, removable media, labels, sensors, or maintenance ports. Locked enclosures, protected keys, disabled or authenticated debug access, encrypted storage, tamper evidence, and inventory inspections should match the consequence of access. Do not place fleet secrets in firmware, QR codes, manuals, service applications, or shared removable media. A technician who needs local access should receive site-specific, time-bound authority with a recorded procedure and a review when the work ends.

Spend scarce telemetry on decisions

Small devices cannot retain every event. Remote sites may connect intermittently, and raw sensor data may be sensitive or too expensive to centralize. The response is not to accept blindness; it is to decide which later questions the evidence must answer.

For each device, preserve enough information to connect identity and time to boot verification, firmware version, update attempt and result, configuration change, authentication, administrative command, network summary, fault, and safety-state transition. Buffer critical events during disconnection, protect their integrity where the platform permits, and define what lives on the device, gateway, supervisory system, historian, update service, or cloud control plane. Stable identifiers and synchronized time are more valuable than a flood of unjoinable messages.

In the conveyor incident, the investigation needs to distinguish an exposed path from a used path. Remote-access sessions, controller authentication, update manifests, firmware measurements, configuration changes, safety events, and network records should form a timeline. A controller that stopped reporting is an uncertainty to resolve, not evidence that nothing happened. High-volume telemetry that displaced the last boot or update result has consumed storage while weakening the decision.

Coordinate disclosure, containment, and safe recovery

A useful vulnerability report identifies affected models and versions, reproduction conditions, required physical or network access, plausible safety effect, available mitigation, evidence-handling needs, and a communication owner. Triage combines exploitability with physical consequence and current exposure. An unauthenticated management service reachable through a vendor path deserves a different response from a local debug weakness on a locked, segmented device, even when both receive the same severity label.

Patch windows should exist before disclosure creates urgency. Name who can approve an emergency change, how operations may stop or degrade safely, which devices can be staged, how rollback or recovery works, and which observations permit return to service. When field visits or production shutdowns make the final fix slow, use explicit interim controls: close network paths, remove shared access, increase inspection, constrain commands, or isolate unsupported devices. Give the remaining risk an owner and an end date instead of promising a patch schedule the system cannot meet.

During an incident, security coordinates with operations and safety from the first consequential action. Establish the incident commander, safety stop authority, site contact, evidence owner, vendor contact, containment options, manual fallback, recovery criteria, and return-to-service approver. Possible actions include closing remote access, blocking a command path, placing equipment in maintenance mode, revoking a particular identity, swapping a device, reflashing trusted firmware, or stopping a site. Their order depends on the physical process.

For the exposed conveyor fleet, the team can close the shared remote path while operators supervise the affected cells, preserve the access and update records, inspect the few devices that accepted maintenance traffic, and move individual devices into a safe maintenance state before re-establishing trust. Pulling power or imaging storage is not automatically forensic best practice when it can destroy volatile evidence, remove visibility, or provoke unsafe manual work. Safety wins when evidence collection conflicts with it; the record should say what was lost and why.

Return to service requires more than “the device is back online.” Confirm the expected hardware and identity, authorized boot state and firmware, known configuration, required network paths and blocked paths, functioning safety controls, trustworthy telemetry, and operator acceptance. Record residual uncertainty. The same parties who can restore connectivity should not silently decide that physical operation is safe.

Retire trust, not only hardware

Long-lived devices eventually outlast update support, cryptographic capability, monitoring integration, or the protocol assumptions around them. An unsupported controller should appear as a dated risk decision with an owner, constrained network position, compensating controls, replacement dependency, and retirement date. A forgotten exception is an unplanned operating model.

Decommissioning is the final identity operation. Before a controller leaves service:

Device decommissioning checklist

• move the physical process to its approved replacement or safe state and record the operations approval;
• revoke device certificates, keys, tokens, local and remote accounts, vendor access, cloud enrollment, and update-channel membership;
• remove network allowlists, management routes, telemetry acceptance, monitoring expectations, support records, and spares-pool eligibility;
• wipe or destroy storage and removable media according to the information and key material they held;
• mark the inventory record retired, record custody and disposal, and prevent automatic re-enrollment;
• test from the former device’s network position that its identity can no longer authenticate or receive trusted commands and updates.

A controller on a warehouse shelf can still hold a fleet credential. A replacement project is complete only when the old trust has been withdrawn.

OT/IoT threat model worksheet

Start the review with a process sketch rather than an asset list. Name the physical output, unsafe and degraded states, safety interlocks, manual fallback, stop authority, and return-to-service authority. Then annotate the sketch with:

1. Devices and identity. Which controllers, gateways, sensors, actuators, engineering stations, and safety systems participate? How is each provisioned, authenticated, rotated, and revoked?
2. Command and update paths. Who can change logic, configuration, firmware, set points, or operating mode? Through which network and physical paths? What approves and records the change?
3. Trust at startup. What verifies boot components and configuration? What does the device do after failed verification, downgrade, interrupted update, or lost connectivity?
4. Exposure. Which enterprise, internet, wireless, vendor, removable-media, service-port, and supply-chain paths cross the boundary? Which are necessary, and which have been proved blocked?
5. Evidence. Where can responders recover identity, time, boot, update, command, configuration, network, fault, and safety-state evidence? What disappears during power loss or isolation?
6. Response and lifecycle. Who may contain, stop, recover, and approve operation? How are unsupported devices constrained, and how will credentials and data be removed at retirement?

Finish by walking one hostile remote action, one local physical action, one failed update, and one lost-telemetry event from cause through physical consequence. The useful output is not a filled template. It is agreement about which action is safe, who may take it, and what evidence will show that it worked.