Concurrency-Safe Design

The last unit of stock

An inventory service has one unit left. Two orders try to reserve it. Each request runs code that is locally reasonable:

item = items[sku]
if item.total - item.reserved >= quantity:
    item.reserved += quantity
    reservations[id] = Reservation(...)

Both requests can read reserved = 0. Both pass the availability check. Both then create a reservation for the same last unit. No line is malformed, and neither request makes an illegal state transition. The failure lives between the lines:

                    Order A             Order B
read reserved       0
                                         0
check available     yes
                                         yes
write reserved      1
                                         1
create reservation  A
                                         B

The final counter may even look plausible while two active reservations exist. This is why “the map is thread-safe” or “the counter update is atomic” is not a complete answer. The rule spans a read, a decision, a counter, and a reservation record.

The rule is:

For each SKU, the sum of active reservation quantities never exceeds total.

That invariant is the center of the design. A concurrency primitive is useful only if it makes the invariant true under every allowed overlap.

Protect the command, not its fields

Locking getReserved() and setReserved() separately still leaves the check between them exposed. The protected unit must be the whole read-check-write command. For an in-memory service, a per-SKU lock gives one small, intelligible boundary:

reserve(order_id, sku, quantity, idempotency_key):
    lock locks_by_sku[sku]:
        existing = reservation_by_key.get((sku, idempotency_key))
        if existing:
            return reservations[existing]

        item = items[sku]                         // read while protected
        if item.total - item.reserved < quantity:
            return insufficient_stock

        reservation = Reservation(
            new_id(), order_id, sku, quantity, active
        )
        item.reserved += quantity
        reservations[reservation.id] = reservation
        reservation_by_key[(sku, idempotency_key)] = reservation.id
        return reservation

The lock has a domain-shaped key. Orders for different SKUs proceed independently; orders contending for one SKU serialize at the point where they could violate its invariant. The lock registry itself must be safe to access, but it does not require a single global lock around every reservation.

The boundary also owns duplicate behavior. If a client times out and repeats the request with the same idempotency key, it gets the committed reservation rather than consuming stock again. That lookup cannot happen before the lock: two duplicate requests could both miss it and then create separate records.

Release uses the same ownership boundary. A reservation’s SKU is immutable, so it identifies the lock; the mutable state is then re-read after the lock is acquired:

release(reservation_id):
    sku = reservation_sku(reservation_id)

    lock locks_by_sku[sku]:
        reservation = reservations[reservation_id]
        if reservation.state == released:
            return reservation
        if reservation.state != active:
            return conflict(reservation.state)

        reservation.state = released
        items[sku].reserved -= reservation.quantity
        return reservation

Two releases may both arrive, but only the first crosses active -> released and changes the count. The second returns the already-released result. Chapter 55 supplied the legal transition; this boundary makes the transition and its stock effect indivisible with respect to competing commands.

The boundary is a policy choice

A lock is one way to serialize a decision, not the definition of concurrency safety. Keep the invariant and the conflict policy fixed while choosing the mechanism that the actual state owner can enforce.

An atomic operation fits when the entire invariant lives in one atomic value: claim a flag if it is still clear, increment a metric, or advance a small state word with compare-and-swap. It does not by itself protect a rule spanning an inventory count and several reservation records.

Optimistic concurrency moves the same reasoning to a versioned persistence boundary:

current = read_item_and_reservations(sku)
next = apply_reservation(current, command)

if save_if_version(sku, expected=current.version, value=next):
    return next.reservation

return conflict                         // or re-read and retry safely

Only one writer can commit against a given version. A losing writer must re-read; retrying the write against its stale snapshot merely repeats the bug. This approach suits rare conflicts and commands that are safe to recompute. A database transaction, conditional update, uniqueness constraint, or exclusion constraint may enforce the persisted invariant more directly. The application still needs to handle the database’s conflict result deliberately.

A single-owner queue takes the other common path: all commands for a SKU are routed to one partition and processed in order. This can simplify a hot stream of state changes, but the routing key is part of the proof. If commands for the same SKU can reach two owners, queue ordering does nothing for this invariant. One very hot SKU can also build a long backlog, so the design needs backpressure and a policy for stalled or poisonous commands.

There is no universally strongest boundary. A useful choice answers three questions:

• Can this mechanism cover every value that participates in the invariant?
• Does it serialize only work that can actually conflict?
• When contention occurs, should the caller wait, retry from a fresh read, receive a conflict, or join an ordered queue?

Those answers carry more engineering information than the name of the primitive.

Keep outside work outside

Suppose reservation also charges a card and sends a receipt. Putting those calls under the SKU lock makes the code look sequential:

lock sku:
    reserve_stock()
    charge_card()
    send_receipt()

It also makes correctness depend on network time. Every order for the SKU waits for the payment provider. A timeout leaves the service unsure whether the card was charged, and a process crash after the charge may leave no local record of it.

The local boundary should commit local truth and a durable promise of outside work together:

transaction:
    re-read inventory and idempotency record
    validate availability
    create reservation
    update reserved quantity
    append payment_requested(reservation_id, payment_key) to outbox

An outbox worker later calls the provider with payment_key, records the outcome, and retries according to policy. The external call cannot be made atomic with a local database commit, so stable identities and observable lifecycle states do the work that a wider lock cannot. A retry may repeat the delivery attempt; it must not create a second charge.

This separation also clarifies cancellation. There are two materially different moments:

• Before the reservation commits, cancellation can abandon the command with no shared mutation.
• After it commits, cancellation is another command. It releases an active reservation once and records what should happen to any payment attempt.

Code that catches a cancellation signal halfway through several unrelated writes is much harder to reason about than code with a visible commit point. After that point, recovery or compensation must be explicit; pretending the command never happened would erase useful truth.

Concurrent tasks need an owner

Not every concurrency bug corrupts a counter. Some work simply escapes. Consider request code that starts email, push, and analytics tasks and then returns. If push fails, nobody awaits its error. If the request is cancelled, the tasks may keep writing. If one task observes an old reservation version, it may publish a result after the reservation has been released.

Request-scoped work should normally live in a structured lifetime: the parent awaits its child tasks, propagates cancellation, and collects their errors. If the work must survive the request, give it durable ownership instead—a job record, queue message, workflow instance, or outbox item with retry state. Detached work with only a log line is neither reliably request-scoped nor reliably durable.

Cancellation is a request to stop, not permission to leave an invisible half result. At every commit point, ask:

• Who owns this task now?
• Where will its failure be observed?
• What durable fact says whether it committed?
• Can stale work still write, or must it present the expected version?

A task that wakes after cancellation should either find that it still owns the expected version or refuse to commit. That version check connects async lifetime safety back to the same stale-state problem as the last-unit race.

When one command has two owners

A wallet transfer makes the per-key story harder because one command changes two accounts. If transfers acquire source and then destination, opposite transfers can deadlock:

Transfer A holds account 1 and waits for account 2.
Transfer B holds account 2 and waits for account 1.

Acquire the pair in a stable order unrelated to transfer direction:

first, second = sort(source_account_id, destination_account_id)

lock first:
    lock second:
        re-read both accounts
        reject if source.balance < amount
        update both balances
        append one ledger entry

All transfers agree on lock order, so the circular wait cannot form. The balances and ledger entry belong in the same commit boundary; otherwise a crash can leave money moved without evidence, or evidence of money that did not move.

Stable ordering prevents this deadlock, but it does not prove good operational behavior. A global lock can be correct and still destroy throughput. A retry loop can be correct in isolation and still amplify load until no writer makes progress. A queue can preserve order while starving ordinary keys behind one hot partition. Concurrency design must account for contention, fairness, backlog, and bounded retry as well as safety.

Make the dangerous schedule happen

A stress test that passes once proves little. The useful test forces the interleaving that would break the invariant.

Start with the unsafe implementation and place a barrier after its read:

Order A reads available = 1, then pauses.
Order B reads available = 1 and commits.
Order A resumes and commits from its stale decision.

The test should observe two reservations for one unit. After the command is moved behind the per-SKU boundary, the same test should observe one reservation and one insufficient_stock result. The assertion is about the invariant, not which order wins.

Then press on the edges created by the design:

• submit the same idempotency key concurrently and require one reservation ID;
• release the same reservation concurrently and require one stock change;
• cancel on both sides of the commit point and require either no mutation or a visible reservation that can be released;
• resume stale async work after the reservation version changes and require its commit to fail;
• for a two-owner command, start opposite-direction transfers and require both completion and preserved total balance.

Deterministic barriers, a controllable repository, or a fake scheduler often teach more than adding thousands of iterations and hoping the operating system chooses the revealing timing. The test should put the stale assumption on the page.

Reasoning under interview time

In a practical exercise, begin with the threatened rule rather than a survey of locks and queues. For the reservation service, a concise design explanation is enough:

Invariant:
  active reserved quantity never exceeds total for a SKU

Owner and boundary:
  a per-SKU lock covers idempotency lookup, availability check,
  reservation creation, and reserved-count update

Conflict policy:
  different keys compete for stock; duplicate keys return one result

Outside work:
  payment is an idempotent outbox task, never a call under the SKU lock

Proof:
  controlled last-unit race, duplicate reserve, double release,
  and cancellation on both sides of commit

If the exercise requires persisted state, replace the in-memory lock with the enforcement the storage layer can guarantee and say how conflicts surface. If it requires many processes, an ordinary process-local mutex is insufficient. If it requires more throughput, partition by the narrowest ownership key and name the hot-key risk. Each changed assumption should produce a changed design decision, not a longer list of primitives.

For practice, take the reservation example through three variations. First, allow one order to reserve several SKUs and decide whether partial reservation is legal. Next, move the service to two processes and replace its local lock. Finally, let payment confirmation arrive after release and decide which version or lifecycle state prevents stale work from reviving the reservation. For each variation, write the invariant and the failing interleaving before choosing a mechanism.

Concurrency becomes tractable when overlap is made concrete. Find the shared rule, show the stale schedule that breaks it, and put every participating read and write behind one boundary that the real owner can enforce. Then make retries, cancellation, and escaped work present their identities before they commit. The test is no longer “is this thread-safe?” It is whether the one dangerous interleaving still has somewhere to hide.