Why fast code still waits

A modern CPU core can start several simple operations in each clock cycle. Yet profiles of real programs often show the processor doing nothing useful for long stretches. The code is "fast" in the sense that each instruction is cheap. The program is slow because instructions keep waiting—for data, for earlier results, or for their turn.

This article separates four reasons for waiting: memory latency, dependency chains, poor locality and queues.

Waiting for memory

Arithmetic happens on values in registers. Values come from memory, through a hierarchy of caches: small, very fast caches close to each core; larger, slower ones further out; and main memory beyond that. A load that finds its data in the nearest cache completes quickly. A load that must go all the way to main memory can take far longer than many arithmetic operations combined. Intel optimization reference manual (opens in a new tab)

Exact figures vary by processor, so this article does not quote them. The relationship is what matters: an add is cheap, a cache hit is cheap, and a trip to memory is expensive enough to dominate a loop.

Waiting for the previous answer

CPUs hide some memory latency by working ahead. If upcoming instructions do not depend on a pending load, the core can execute them while the load is in flight. Several misses can be outstanding at once.

That only works when independent work exists. Compare two loops:

# A: sum an array
for i in 0..n:  total += a[i]

# B: follow a linked list
node = head
while node:     total += node.value; node = node.next

In loop A, the address of a[i+1] is known before a[i] arrives. The hardware can request many elements ahead of time. In loop B, the address of the next node is inside the current node. A simple pointer chain limits how many independent requests the core can issue: it cannot ask for the next node until the previous load completes. If every node misses the cache, the loop tends toward one memory latency per element, no matter how fast the core is. Some prefetchers can recognise certain indirect patterns, but scattered layouts generally make access harder to predict.

This is a dependency chain, and it is the same idea that limits GPU parallelism in how a GPU works and token generation in how AI inference works.

Waiting because of layout

Caches move memory in fixed-size lines, not individual bytes. Reading one value brings its neighbours along.

Sequential access uses that well: one line fetch serves several consecutive elements, and the hardware prefetcher can predict the next lines. Scattered access wastes it: each fetched line may deliver one useful value, and the next address is less predictable.

The same data, arranged differently, can therefore cost very different amounts. A table stored as an array of records, when you only need one field, drags every other field through the cache. Storing that field contiguously means each line is full of values you will use. The memory and data movement section works through locality and layout with runnable models.

Waiting in line

The final kind of waiting is not about one core at all. A request arrives at a service—a database, a network interface, a GPU inference server—and finds others ahead of it.

Queues behave non-intuitively. When arrivals stay below capacity, a short burst creates a queue that drains afterward. When arrivals approach capacity, small bursts cause long waits, because there is little spare capacity to catch up. When the average arrival rate stays above capacity, the backlog keeps growing—unless admission control, backpressure or drops intervene. A finite burst above capacity can still drain once it ends.

Take an illustrative fixed-service model: eight requests arrive together at one worker, and each takes 200 ms. The first finishes at 200 ms; the last starts at 1,400 ms and finishes at 1,600 ms. Its 200 ms of work came with 1,400 ms of waiting. The server's work per request is unchanged; the added time is waiting. The public AI serving demo lets you change the number of requests, worker slots and service time for exactly this burst model.

Reading a slow profile

When code is slower than its instruction count suggests, ask four questions:

1. Is it waiting for memory? Large working sets, cache misses, scattered access.
2. Is it waiting for itself? Each step needs the previous result—pointer chasing, serial reductions.
3. Is the layout wasting fetched bytes? Unused fields, strided access.
4. Is it waiting in line? Utilisation near capacity, bursty arrivals, a shared resource.

These have different fixes: improving locality, restructuring to expose independent work, changing data layout, or adding capacity and smoothing load. Treating every slowdown as "the code needs to be faster" misses most of them.

For the CPU-side model, continue with how programs execute. The AI serving demo applies the same queueing ideas to a burst of model requests.