Processes, Threads, and Memory

When the heap looks healthy and the process dies

An upload service is killed after traffic spikes. The managed heap rises, garbage collection runs, and the heap settles near 700 MiB. Yet the container’s memory usage continues toward its 2 GiB limit until the operating system terminates the process.

“The heap is stable” is true and insufficient. The process also has thread stacks, native compression buffers, mapped files, allocator bookkeeping, runtime metadata, and resident pages that the heap profiler may not show. A burst can multiply several of those terms at once. The first useful question is therefore not “Which garbage collector should we tune?” but “What memory does this process own, and which measurement reached the limit?”

That question leads to a compact runtime model:

The model is deliberately smaller than an operating-systems course. Its job is to let you explain a crash, size a worker pool, challenge an unbounded cache, or choose a process boundary without treating the runtime as invisible.

A process draws the first boundary

A process has a virtual address space and an operating-system identity. It is also a practical unit of lifecycle, permissions, open resources, signals, and resource accounting. Ordinary code in one process cannot follow a pointer into another process and mutate its heap. Data must cross through an explicit mechanism such as a pipe, socket, file, message broker, or deliberately shared memory region.

That separation is useful when failure or privilege must be contained. An API server might send image conversion to supervised worker processes because the converter calls a native library that can crash or retain memory. A failed worker can be killed and replaced without taking the request-serving heap with it. The price is visible: request serialization, data transfer, startup, timeouts, abandoned-work cleanup, and supervision.

The boundary does not confer correctness on everything the processes touch. Two isolated workers can still overwrite the same file, race on a database row, or consume the same message. A process isolates address spaces; a shared external fact still needs its own consistency rule.

Process isolation also does not imply that every physical page is unique. Shared libraries, mapped files, and copy-on-write pages may be backed by the same physical memory. After a process forks, parent and child can initially share clean pages. A write causes a private copy of the affected page, so a preloaded service may look economical at startup and grow as workers mutate their heaps.

Threads share the consequential state

A thread is an execution path inside a process. It has its own CPU register state and call stack, but it normally shares the process’s heap, globals, mapped regions, and open file or socket resources with peer threads.

Cheap access to shared memory makes threads useful. It also means that a thread boundary provides little fault containment: a bad pointer in native code can corrupt the process, and an unhandled fatal failure can end every thread. Two threads mutating the same heap object need an ownership or synchronization rule that preserves the object’s invariant.

A coroutine, future, promise, or async task is usually a runtime-level unit, not a new address space. Many tasks may take turns on one event-loop thread; other runtimes schedule them across a thread pool. Either way, async syntax does not make shared state private, CPU work free, or child work automatically owned.

This distinction prevents a common design error. Moving unsafe conversion to another thread keeps communication cheap, but it does not protect the API process from a native crash or runaway heap allocation. Moving it to another process creates a kill and accounting boundary, at the cost of inter-process coordination.

Follow one request through memory

Suppose one upload request enters the service. Its handler runs on a thread. Function calls add frames to that thread’s stack: return addresses, saved state, parameters, and some local values. The request object, decoded metadata, and dynamically sized buffers are likely on a heap. A compression library may allocate another buffer through a native allocator. Streaming code may map a temporary file. The kernel also owns socket and file state on the process’s behalf.

“Memory used by the request” is therefore not one number.

Stack

Each thread needs a stack. Stack allocation is cheap because entering and leaving a function usually moves a stack pointer, but the stack is bounded. Deep recursion and large stack-local values can exhaust it. A thread’s configured stack size may reserve virtual address space without making every page resident immediately, so multiplying thread count by stack size is a useful conservative budget, not always a measurement of physical use.

For input whose depth is not trusted, recursion hides an unbounded term in the call stack. Enforce a depth limit or use an explicit heap-backed work stack whose growth can be inspected and bounded.

Heap and native allocations

The heap holds objects whose lifetime is not tied to one call frame: request graphs, caches, queues, connection pools, and service state. Native libraries and runtimes may maintain additional heaps outside the one reported by a managed-language profiler.

Languages reclaim heap memory in different ways. Manual allocation requires explicit release. Reference counting reclaims an object when its reference count reaches zero, with special handling needed for cycles. Tracing garbage collectors discover which managed objects remain reachable. Arenas and regions release groups of allocations together.

Those mechanisms change how memory is reclaimed; they do not choose what the program is allowed to retain. A cache with no limit can keep every entry reachable forever while the garbage collector behaves exactly as designed. The same is true of an event-listener list, a global map, an unconsumed queue, or a background task that still holds its request context.

This is the managed-runtime form of a leak: memory remains reachable through the wrong long-lived owner even though it no longer performs useful work. Other growth is not a leak at all. An allocator may keep freed arenas for reuse instead of returning pages promptly to the operating system, or a burst may expand a legitimate cache to its configured budget. Retention profiles and allocation traces distinguish these cases better than the word “leak.”

File handles, sockets, native buffers, and mapped files also need lifecycle ownership. Garbage collection may provide a last-resort cleanup path in some runtimes, but prompt release cannot depend on when a collector happens to run.

Virtual addresses and resident pages

Virtual memory gives a process its own address space. The operating system maps virtual pages to physical frames, file contents, or no physical backing until a page is touched. A large address reservation can therefore consume little physical memory.

A page fault occurs when the current mapping cannot satisfy an access. Some faults are cheap bookkeeping; others require loading file data or recovering a page from slower storage. The phrase does not by itself mean that the machine is out of memory.

Resident set size, or RSS, estimates the process’s pages currently resident in physical memory. It is valuable, but it is not a perfect container total: shared pages complicate addition across processes, and a memory control group can account for page cache and kernel memory that one heap view omits. When a container approaches its limit, compare the runtime heap, process RSS, control group usage, native allocations, mapped regions, page cache, and thread count rather than expecting one chart to explain them all.

Put the bytes into a budget

A design becomes more honest when every load-dependent term appears in one rough equation:

memory pressure ~= baseline runtime
                + active requests * memory per request
                + worker stacks
                + cache budget
                + queue depth * bytes per item
                + native buffers and mapped pages

Consider the upload service with these planning numbers:

• a 280 MiB baseline;
• 120 concurrent uploads using about 3 MiB of request and decode state each;
• 48 worker stacks budgeted at 1 MiB each;
• a 512 MiB cache;
• 4,000 queued records averaging 48 KiB;
• a 256 MiB allowance for native buffers and mapped pages.

The total is about 1,644 MiB. A 2 GiB limit leaves roughly 404 MiB for burst error, runtime metadata, page cache, and measurements that were too optimistic. If upload concurrency doubles without backpressure, the estimate rises by another 360 MiB and nearly consumes that margin.

The arithmetic does not need false precision. It reveals which choices own the risk. Streaming can reduce per-upload state. Admission control can cap active uploads. A bounded queue prevents delayed work from becoming an unbounded heap. A cache needs an explicit budget, admission rule, and eviction policy. A separate conversion process can receive its own limit and restart policy.

Resource limits extend beyond bytes. File descriptors, process and thread counts, connection pools, socket buffers, temporary disk, queue capacity, CPU quota, and downstream concurrency can fail a service first. “The machine has enough RAM” says little if one worker exhausts its descriptor limit or a container is restricted to a fraction of the host.

Scheduling decides whether workers help

The operating-system scheduler decides which runnable threads receive CPU time. A context switch saves one execution context and resumes another. That cost includes scheduler work and often lost cache locality, not merely a fixed instruction count.

More runnable workers can therefore reduce throughput. CPU-bound work usually needs a bounded number of workers chosen near the CPU capacity available to the process, then adjusted with measurement. I/O-bound work can sustain more concurrency because tasks spend time waiting, but memory, sockets, downstream limits, and queueing delay still impose a ceiling.

An event loop makes waiting economical only while handlers yield quickly. A large synchronous parse, compression job, or hot calculation blocks unrelated connections on the same loop. Move that work to a bounded CPU pool or worker process; do not replace one blocked loop with an unbounded work queue.

Data layout matters once profiling points to memory access. A contiguous array scan often uses caches and prefetching better than a pointer-heavy walk, even when both algorithms have similar asymptotic complexity. Batching can improve locality and amortize per-item overhead. These are consequences of real access patterns, not reasons to complicate a correct design before it is measured.

Diagnose the upload service

Return to the process killed with a stable managed heap. A useful investigation separates the measurements before proposing a fix.

First, align the timeline for managed heap, RSS, control-group memory, thread count, active uploads, queue depth, and restarts. If heap and RSS rise together, inspect retained managed objects and allocation rate. If heap is flat while RSS grows, inspect native buffers, mapped files, thread stacks, allocator arenas, and runtime-specific direct memory. If process RSS looks stable while control-group usage rises, inspect charged page cache and other processes in the same group.

Next, reproduce the shape under a bounded load. Heap retention paths may lead to completed requests held by a listener. Native allocation tracing may show one compression buffer per active upload. A file inspection may reveal temporary mappings that are never closed. The evidence identifies the owner; the owner determines the fix.

Finally, make the failure policy explicit. Bound active uploads and queue depth, reject or defer excess work, close native resources on success, failure, timeout, and cancellation, and alert on the measurement that approaches the actual limit. Raising the limit may buy time, but it does not repair an unbounded term.

Make the model do some work

Before reading the answer, budget this process:

container limit:           2,048 MiB
baseline:                    300 MiB
cache budget:                600 MiB
80 worker stacks:             80 MiB
200 active uploads:        1,000 MiB
10,000 queued items:          313 MiB
native and mapped allowance:  250 MiB

The planned peak is about 2,543 MiB, already 495 MiB beyond the limit. No garbage-collector setting can make those simultaneous budgets fit. A credible response chooses a product policy: stream smaller chunks, admit fewer uploads, reduce or externalize the cache, shorten the queue, or give the work a separately limited process. It also says what happens at the bound—a deadline, a retryable rejection, or durable deferral—rather than allowing the kernel to choose by killing the process.

Now change one condition: the native decoder can crash on malformed input. Memory arithmetic alone no longer chooses the boundary. Put the decoder in a supervised process with a narrow input and output contract, then budget how many such workers may run. The new process costs startup, communication, and duplicated memory, but it gives the system a failure and kill boundary.

Answer from boundary to evidence

When runtime behavior appears in a coding, design, or production problem, move through the model in order:

1. Name the process, thread, task, request, and job boundaries that actually exist.
2. Say what each boundary isolates and what state remains shared.
3. Account for stack, managed heap, native allocation, mapped files, handles, and the terms that multiply with load.
4. Give long-lived memory an owner, a budget, and a cleanup event.
5. Bound workers and queues by CPU, I/O, memory, and downstream capacity, then name the metric and failure policy at the limit.

The value of this sequence is not vocabulary. It is that every proposed fix must attach to a mechanism. Garbage collection can reclaim unreachable managed objects; it cannot shrink a cache the program still owns. Threads can share data cheaply; they cannot contain a native crash. Async work can wait efficiently; it cannot make CPU work disappear. A larger container can delay a kill; it cannot bound a queue.

Once the boundaries and budgets are visible, the next question is precise: when work overlaps inside or across them, which invariant, capacity, visibility, ordering, or lifetime rule must survive? That is the work of concurrency and synchronization.