How an LLM works

The LLM call contained a sentence that was deliberately left unproven there: the voice at the other end is "technically a huge, clever random generator with no memory whatsoever". This page makes good on that sentence. Before Block III compares models, downloads them and puts them on your own hardware, we look into the machine itself - because every later question of this block (Which model? How much VRAM? What do benchmarks measure?) is a follow-up question to what happens here.

The only job: the next token

A language model can do exactly one thing, and it is smaller than the name "artificial intelligence" suggests: it receives a sequence of tokens - text pieces as produced by the tokenizer - and computes from it, for every token in its vocabulary, the probability of being the next one. "The cat sits on the" yields a list with a good 150,000 entries: sofa likely, roof possible, fridge unlikely, photosynthesis practically ruled out.

That is all. There is no second mode, no hidden plan, no intention. Even the longest, smartest answer is assembled from exactly this one operation.

Writing means: guess, append, repeat

The single operation becomes text through a loop called autoregression: the model computes the distribution, one of the likely tokens is drawn, appended to the end of the sequence - and then everything runs again from the top, now with one token more. Token by token, until a stop signal falls.

Two things about this are decisive for building agents:

• The model reads its own answer. Every token it has written is part of the input at the next step. An early mistake is therefore not corrected but continued - it is context now.
• There is no inner state between steps. No notepad, no working memory beside the text. Everything the model "knows" while answering is in the token sequence itself. The context is not one input among several - it is the entire working state.

You do not have to take the distribution on faith - you can look at it. And the dials you know from the LLM call (temperature, top_p) act exactly here - after the computation, on the finished distribution:

Live experiment

Loading distributions …

Why guessing is enough

That leaves the real question - and it is the most important one on this page: how can something like translating, programming or arguing emerge from "predict the next token"?

The answer: whoever wants to predict the next token well in arbitrary text cannot get by on memorisation. "The capital of France is" is only reliably followed by Paris if something in the weights behaves like that fact. Half a Python function is only followed by the correct second half if something has learned the rules of the language. And hitting the next word in the middle of a crime novel requires tracking the state of the plot - who the culprit is decides which continuation is plausible.

Good prediction forces understanding. Not because anyone programmed understanding into the model, but because understanding is the cheapest strategy for lowering the prediction error. Grammar, facts, style, a piece of logic - all of these are by-products of a single, mercilessly simple training objective. How that training runs, and why it first produces a continuer and not yet an assistant, is what the deeper tiers below show.

An LLM is a function: token sequence in, probability distribution over the next token out. Text comes into being by running this function in a loop - guess, append, repeat.

Everything else follows from that: the context is the entire working state, a written mistake becomes input, and "knowledge" is whatever improved the prediction - not a database anyone looks things up in. Whoever has understood this one loop reads every model quirk for what it is: a property of the task, not a bug in the product.