Embedded and Systems Engineering

The motor has one millisecond. The cloud can wait.

An embedded system-design prompt might begin like this:

Design the firmware for a connected motor controller on a packaging line.

The controller samples current and position at 1 kHz and must update its output
before the next sample. It receives commands and sends diagnostics over CAN.
It has 512 KiB of flash and 128 KiB of RAM. A stale command must cause a
controlled stop. Devices receive remote firmware updates and may lose power at
any point. A separate hardware circuit handles the emergency stop.

Nothing in the prompt is exotic. That is why it works well in an interview. The components are small enough to sketch, but they make ordinary software assumptions visible. Work cannot expand until the host catches up. A pointer does not explain who owns the memory behind it. An average execution time does not prove a deadline. A successful download does not prove that a device can boot after power loss.

The strongest opening is not a recital of buses, schedulers, or language features. It is a short account of what must remain true:

• each control iteration uses one coherent sample and publishes at most one output;
• the control path finishes inside its budget even when communication and diagnostics are busy;
• command, sample, and buffer ownership cross execution contexts explicitly;
• malformed, late, or incompatible input cannot become an actuator command;
• loss of software health moves the equipment toward its defined safe state;
• an interrupted update leaves a bootable image or a known recovery path.

These are constraints, not implementation choices. They give the rest of the answer something to protect.

The same discipline applies beyond physical devices. For a runtime, database engine, kernel component, or performance-sensitive service, replace the motor deadline with a scheduler, allocator, tail-latency, ABI, or crash-recovery constraint. The interview still asks whether you can reason when an abstraction stops absorbing mistakes.

Turn the prompt into budgets

The one-millisecond period is not one millisecond of application code. Timer jitter, interrupt handling, sensor transfer, computation, output transfer, and scheduling delay all spend the same interval. State that decomposition before choosing an RTOS priority or optimizing arithmetic. Then ask for the missing facts: permissible jitter, worst-case sensor and bus latency, actuator behavior on a missed update, clock accuracy, interrupt load, and the evidence behind any existing timing claim.

The distinction between average and worst case is central. A loop that usually takes 80 microseconds but occasionally waits behind a flash erase has not met a one-millisecond requirement. A senior answer identifies every unbounded or poorly bounded operation on the critical path: allocation, logging, retries, lock contention, cache maintenance, a peripheral wait, or a loop whose length comes from input.

Classify the work by consequence. Sampling and control belong to the periodic critical path. Command parsing may be time-sensitive without sharing the same deadline. Telemetry, cloud retry, verbose logs, and update download are best effort. They may be delayed, sampled, or dropped. The architecture becomes clearer once those classes stop pretending to have equal urgency.

This is also where resource limits become design facts rather than atmosphere. Account for static data, task stacks, driver and DMA buffers, queues, the update workspace, and a margin for measured peaks. If dynamic allocation is allowed, say where it is allowed, what happens on failure, and why fragmentation or allocator latency is acceptable there. “No heap” is not inherently senior; an allocation policy with a measured failure story is.

Flash needs a similar budget. Two complete images make rollback simple but may not fit beside the bootloader, configuration, calibration, and diagnostic storage. That tension should appear early, because it changes the update mechanism rather than merely changing a deployment checklist.

Follow one sample through the machine

Now trace state through time. A timer or peripheral event begins a sample. DMA may move sensor data without keeping the CPU occupied. Completion makes a buffer available to the control task. The task validates freshness, computes the next output, writes through the actuator interface, records a compact health observation, and releases the buffer. A separate task packages selected observations for the bus.

The execution boundaries matter more than the boxes. An interrupt service routine should acknowledge the source, capture the minimum state that would otherwise be lost, and notify deferred work through a mechanism that is safe from interrupt context. Formatting logs, allocating memory, waiting for a mutex, parsing a long message, or retrying a device transaction in that context makes interrupt latency depend on unrelated work.

Give buffers visible ownership. For a receive buffer, the legal movement might be:

free -> DMA-owned -> complete -> control-owned -> free

Only the owner may mutate the payload. The DMA completion path must publish the completed state before the consumer can observe it, and the buffer cannot return to DMA while the consumer still reads it. Use an RTOS queue, an interrupt-safe handoff primitive, or a short critical section when one of those gives a simple, documented guarantee. Atomics are not a decoration. If you choose them, name the producer and consumer, the data protected by the ordering, the target architectures, and the argument that prevents a partial observation.

This state trace exposes several questions that a block diagram hides. Can a new sample overwrite one still in use? What happens when control misses a release? Can telemetry retain a pointer after the control task recycles the buffer? Does cache maintenance occur at the CPU/DMA boundary on hardware that needs it? Which context owns the actuator driver?

Those questions are the design. “Use DMA” or “use a real-time operating system” is only the beginning of an answer.

Let the small coding prompt reveal the same design

A practical round may shrink the controller to a bounded ring buffer carrying samples from one producer to one consumer. Before writing modulo arithmetic, make the contract explicit:

capacity is fixed and greater than zero
head identifies the next readable slot
tail identifies the next writable slot
full and empty have distinct representations
exactly one producer publishes tail
exactly one consumer publishes head
overflow returns an error; it never overwrites unread control data

If the interviewer has not required concurrent access, implement the single-threaded structure correctly first. If they add an interrupt producer and task consumer, explain which synchronization primitive is available on the target. A solution copied from a lock-free queue is weak when neither the memory-ordering contract nor the interrupt behavior can be defended.

Test the first fill, the first drain, wraparound, alternating operations, minimum capacity, full writes, empty reads, and counters near their integer wrap boundary. Then test the stated handoff. If overflow must drop samples in another product, change the invariant deliberately and make the loss visible; do not let accidental overwrite become a policy.

A parser prompt deserves the same treatment. Bound the frame before copying it. Validate length, version, command, range, and freshness before changing control state. Define how partial input, an unknown version, a failed integrity check, repeated garbage, and resynchronization behave. Parser code sits at a reliability and security boundary even when the job title does not contain either word.

Hardware is a contract with variation

The controller does not talk to “the sensor” in the abstract. It talks to a particular hardware revision over a particular electrical and protocol boundary. The contract includes units, ranges, byte order, register or message version, timing, reset state, calibration, tolerances, timeout behavior, and known errata. It should also say which layer owns each decision.

A hardware abstraction layer can make register access and board differences manageable. It cannot erase them. Let the driver own device-specific transfer and error details; let the control layer own whether a missing or implausible sample permits continued operation. Keep that policy out of scattered driver callbacks where it becomes difficult to test.

For CAN commands, the application protocol still needs a contract even though the bus handles framing and its own error detection. A command can carry a version, sequence or freshness information, bounded values, and an explicit type. The receiver needs a policy for unknown commands, delayed traffic, bus silence, repeated frames, and recovery after a controller or bus error. A checksum is not a freshness rule, and a retry is not safe until the operation is known to be repeatable.

Hardware also changes over the life of the product. A second sensor vendor may have a different startup delay. A board revision may invert a signal or change a pull-up. Oscillator tolerance can expose a timing assumption that worked on the bench. Good project and design answers mention revision identity, capability discovery where appropriate, manufacturing tests, calibration ownership, and fleet slices that can reveal correlated failures.

Safety is a state, not an adjective

“Fail safe” is incomplete until the safe state and the transition into it are named. For the packaging-line controller, stale command data might require a controlled stop, while an emergency-stop circuit independently removes drive authority. Whether that is the correct behavior depends on the hazard analysis for the actual machine. Firmware should not quietly appoint itself as the sole safety mechanism when a fault in that firmware is one of the hazards.

The software design still has substantial work to do. Define which faults permit degraded operation, which inhibit motion, which latch until an operator acts, and which can be retried. Validate actuator outputs at the final boundary. Keep the last fault and reset evidence across reboot when the platform allows it. Make manual recovery comprehensible to the people servicing the device.

A watchdog is one mechanism in that design, not proof of safety. Refresh it only after the health conditions it is meant to supervise have occurred. A low-priority watchdog task that runs while the control task is dead may certify the wrong thing; refreshing from a stuck high-priority loop may do the same. Also decide what repeated watchdog resets do. Endless rebooting can be worse than entering a diagnosable service state.

Testing should match the failure model. Unit tests can protect parsers and state machines. Simulation can sweep more timing and input combinations than the bench. Hardware-in-the-loop can exercise real interfaces and outputs. Fault injection can interrupt transfers, corrupt frames, delay tasks, exhaust buffers, and remove power. Soak and environmental testing can expose slow leaks and hardware variation. None is a universal substitute for the others.

An update is not complete until the next boot is trusted

For remote update, start with the states in which power can disappear: during download, verification, metadata change, image selection, first boot, and health confirmation. The mechanism must make each transition either repeatable or recoverable.

With enough flash, an A/B arrangement can download into an inactive slot, verify authenticity, integrity, hardware compatibility, and version policy, then ask a small bootloader to try the candidate. The application marks the image healthy only after essential startup checks. Failure to boot or confirm within the defined policy selects the known-good image. Configuration changes need compatible versioning or their own rollback story; restoring old code does not help if the new image irreversibly rewrote its data.

When two images do not fit, say so. A recovery image, external staging storage, or a carefully designed in-place scheme may be appropriate, but each spends a different combination of flash, update time, hardware cost, and recovery risk. The senior move is to preserve the recovery invariant under the actual budget, not to insist on A/B slots after the numbers have ruled them out.

Roll out in cohorts. Observe download completion, boot confirmation, rollback, reset, control-fault, and support rates by hardware and firmware revision. Keep update work out of the critical control path, define battery or power preconditions, and make a paused update an ordinary persistent state rather than an improvised exception.

The reboot that looked like a power problem

Suppose resets increase after the new firmware reaches five percent of the fleet. They correlate loosely with high motor load. It is tempting to declare a brownout and tune a threshold.

Resist that answer. First hold the rollout and establish whether the equipment is reaching its defined safe state. Compare reset reason, fault status, last control-cycle high-water mark, stack watermark, supply-voltage observation, hardware revision, firmware version, command history, and the last entries in a bounded trace buffer. Preserve affected devices when possible instead of reflashing away the evidence.

The evidence changes the leading hypothesis. Resets report a processor fault, not a brownout. The last trace entry is often a newly requested diagnostic frame. A gateway update began sending the optional CAN FD form with a larger payload, while the receiver copied the reported payload length into an eight-byte legacy buffer before validating the application message. The copy corrupted adjacent state; motor load was correlated only because busy periods requested more diagnostics.

Now the repair can match the mechanism. Reject unsupported frame forms before copying, bound every copy by the destination and protocol contract, version the diagnostic message, and keep decode state separate from control state. Reproduce the fault with the recorded frame sequence, then add truncated, oversized, unknown-version, repeated, and high-rate cases. Run the sequence under the same interrupt load and on each relevant hardware revision. Release first to a diagnostic cohort and watch both resets and rejected-frame counters.

The incident does more than supply a debugging story. It tests whether the candidate can distinguish correlation from cause, preserve evidence, cross a hardware/software ownership boundary, mitigate before certainty, and improve the system that will diagnose the next unknown failure.

Carry the controller through the interview loop

The loop may enter through different doors, but it need not produce five unrelated performances.

In a coding or code-review round, use the ring buffer or frame decoder. State ownership, bounds, overflow policy, error behavior, and concurrency assumptions before polishing syntax. In C or C++, make lifetime, initialization, integer conversion, and cleanup visible. In Rust, explain the unsafe boundary rather than implying that ownership types define the peripheral or DMA contract for you.

In system design, draw execution contexts and state handoffs as well as components. Spend the timing, RAM, flash, bus, power, and recovery budgets. Let the interviewer change the sample rate, hardware revision, fleet size, or update constraint and show which decision moves.

In a debugging round, keep observations, hypotheses, experiments, mitigation, and proof distinct. “It is probably a race” is not a diagnosis. Name the evidence that would raise or lower each hypothesis and avoid fixes that erase the only useful state.

For a project deep dive, choose work in which a constraint changed the design: memory pressure, deadline misses, an unsafe interface, hardware variability, an ABI migration, a performance regression, or a field failure. Show the artifact that made the judgment inspectable—a trace, benchmark, timing budget, fault tree, test fixture, rollout gate, or support procedure. Impact without a mechanism sounds remote; a mechanism without a consequential decision sounds junior.

Behavioral questions often put the same facts under delivery pressure. A hardware team disputes the fault boundary. Product wants the full update rollout before rollback evidence exists. Manufacturing needs a diagnostic mode that could bypass normal limits. A strong answer states the hazard or failing mechanism without theatrics, proposes a bounded path, identifies who must decide the residual risk, and names the evidence that would reopen the broader release.

Practice until the constraints alter your answer

One prompt can support a useful week of preparation:

1. Implement the bounded ring buffer. Then add an interrupt producer and task consumer only after you can explain the handoff available on the target.
2. Write the CAN command decoder with fixed bounds, versioning, freshness, and resynchronization. Fuzz or generate malformed frames rather than testing only named examples.
3. Allocate the 128 KiB RAM budget across static state, stacks, DMA buffers, queues, diagnostics, and margin. State which measurements would replace your estimates on hardware.
4. Draw a one-millisecond timing budget and add a flash operation, a burst of bus interrupts, and a missed sample. Decide what moves out of the path and what the controller does when the budget is lost.
5. Design the update twice: once with room for two images and once without it. Interrupt power at every persistent-state transition.
6. Rehearse the reboot investigation. Start with only the symptom, ask for evidence, and do not reveal the oversized frame to yourself until your hypothesis tree has earned that clue.
7. Tell one real project story without product names. If the listener cannot identify the constraint, the rejected alternative, the evidence, the risk, and what remained after the work, refine the story rather than adding more low-level vocabulary.

Preparation is adequate when you can reason precisely without a particular microcontroller or operating system doing the talking for you. You can explain who owns a byte, what can preempt a state change, how a deadline is measured, what a peripheral may return, how a device fails, and how an update recovers. When a prompt changes, those questions should lead you to a different design.

That is the specialty signal. Senior systems engineers do not merely work closer to the machine. They make the machine’s limits part of the argument, then leave enough evidence for someone else to verify it.

Related links

• Processes, Threads, and Memory
• Concurrency and Synchronization
• Working in an Unfamiliar Codebase
• Bug Hunt and Incident Exercises
• Reliability and Failure Engineering