How AI inference works from prompt to first token

"Inference" just means using a trained model to produce an output. Many models do that in one step: an image classifier returns a label, an embedding model returns a vector. This article is about one specific and very visible kind: autoregressive transformer language models, the type behind chat assistants, which generate text one token at a time.

For those models, the time to the first word you see and the speed of the words after it are governed by different parts of the work. Separating them is the key to explaining why a response feels slow.

Step one: text becomes tokens

These models receive token IDs rather than raw text. A tokenizer splits the prompt into tokens and maps each to an integer. Tokenizers vary: depending on the model, tokens may be bytes, characters, word fragments or whole words, so the number of tokens for a given text differs between models. Hugging Face text generation guide (opens in a new tab)

Tokenisation is part of the path the user waits for; how much it costs depends on the implementation and where it runs.

Step two: prefill reads the whole prompt

The model is a stack of transformer layers. Each layer contains attention, which lets each token draw on earlier tokens, followed by further computation. Attention Is All You Need (opens in a new tab)

During prefill, the model processes every prompt token through every layer. Because the whole prompt is already known, tokens can be processed together as large matrix operations—the kind of work GPUs handle well. Along the way, each attention layer computes a key and value for every token. Prefill ends by producing the probabilities for the next token.

Prefill work grows with prompt length. A long document pasted into a prompt means more prefill before anything can be generated.

Step three: decode, one token at a time

Generation is a loop. Choose a token from the probabilities (greedily or by sampling), append it, and run the model again to get the next probabilities. Each new token depends on the one before it, so the tokens of a single response cannot all be computed at once.

Here the KV cache matters. Without it, every step would recompute keys and values for the entire sequence. With it, those values from earlier tokens are stored and reused; each step computes them only for the new token and attends over the cached ones. Hugging Face KV cache (opens in a new tab)

The cache trades computation for memory. It grows with sequence length, number of layers and model width, and it is held for every active request. Running out of cache memory limits how many conversations a server can serve at once.

Weights and the cost of each decode step

The model's weights—its learned parameters—are needed at every step. In a decode step for one sequence, the arithmetic per weight is small, but the weights still have to be read from memory into the compute units. That is why decode for a single request is often limited by memory bandwidth rather than raw arithmetic.

Servers improve this by batching: running decode steps for many requests together, so each read of the weights serves many sequences. Batching raises throughput; it can also mean a request waits for a slot in the batch.

Where the first token's time goes

Put the pieces in order for one request:

time to first token ≈ queueing + tokenisation + prefill + first sampling + delivery
time per later token ≈ one decode step (shared with the batch) + delivery

This is an accounting model, not a measurement. Real servers overlap stages and schedule requests in sophisticated ways. But it gives you vocabulary.

Consider two invented situations. In the first, a short prompt arrives at an idle server; most of the wait is prefill and setup. In the second, the same prompt arrives behind many others; prefill is unchanged, but the request waits in a queue first. The model did not get slower. The system got busier.

What to look at when it feels slow

Treat the symptoms as hypotheses, not diagnoses. A long delay before the first token, with fast tokens afterward, suggests queueing or a long prefill. A quick first token followed by slow streaming suggests decode—cache pressure, batch size or memory bandwidth. But networking, response buffering, scheduling and cold starts can produce either pattern too. Measure time to first token and time between tokens separately, under realistic load, with the same prompts.

The public AI serving demo models the queueing part interactively. For why memory reads dominate so often, read why fast code still waits; for the hardware side, start with how a GPU works.