Every large language model in production today is a transformer, and the architecture is more boring than the results suggest. It is a stack of identical blocks that repeatedly refine a list of vectors, trained on one task: guess the next chunk of text. There is no reasoning module, no knowledge base, no planner. Understanding the machine is mostly a matter of following what happens to one list of numbers as it moves through the stack.
Start at the input. Text is a string; the model needs numbers. So the text is cut into tokens — not words, but common fragments. Frequent words are single tokens, rarer ones split into pieces, and a rough rule of thumb for English is about three-quarters of a word per token. This is why models trip over character-level questions like counting letters in a word. They never see the letters. The word arrived as one indivisible symbol.
Each token has a row in a lookup table, and that row is a vector of a few thousand numbers. That is the embedding. At this point the vector for “bank” is the same whether the sentence is about rivers or money, because nothing has looked at the sentence yet. Fixing that is the job of everything downstream. If you want the practitioner’s view of embeddings as a standalone tool, there is a separate piece on that; here they are just the entry point.
Position has to be added, because the model has no sense of order
A subtlety worth pausing on. The stack processes all token vectors at once, with no notion of first or last. Feed it “dog bites man” and “man bites dog” and, without help, it sees the same bag of vectors.
So position is encoded and added into the embeddings before the stack begins. The original design used sine and cosine waves of different frequencies; current models mostly use rotary embeddings, which encode relative distance inside the attention step. The mechanism varies, the point does not: order is supplied as data, not as structure. It is a bolt-on. Extending a model’s context window is therefore a real engineering problem rather than a config change.
The stack: the same block, many times
Now the list of vectors passes through a sequence of identical blocks — dozens in a small model, a hundred or so in a large one. Each block does two things in order.
Attention. Every vector looks at the other vectors and pulls in what is relevant. This is where “bank” finds out that “river” is three tokens away and adjusts accordingly. It is the only step in the whole architecture where tokens communicate. How it works is its own subject.
A feed-forward network. Each vector is then passed, independently, through a small two-layer network — the ordinary weighted-sum-and-nonlinearity kind. No token talks to any other here. This is where most of the parameters live, typically two-thirds or more of the model, and where most factual knowledge is believed to sit.
Both steps are wrapped with a residual connection — the block’s output is added to its input rather than replacing it — and a normalisation layer. Those two additions are what make a hundred-block stack trainable at all. The residual gives the error signal a direct path back to the early layers, which is the fix for the vanishing gradient described in backpropagation.
The shape of the data never changes. One vector per token goes into every block, one vector per token comes out. The stack does not build a tree or a parse or a summary. It refines the same list, over and over, until each vector carries enough context to be useful.
The training objective is one line
At the end, take the final vector for the last token, multiply by one more weight matrix, and get a score for every token in the vocabulary. Softmax turns the scores into probabilities. Sample one, append it, run again.
That is the entire machine, end to end:
One list of vectors, refined in place. The block never changes shape — one vector per token in, one per token out — so it can be stacked as many times as the budget allows. The residual path is what keeps a hundred-block stack trainable.
Training is: take real text, hide what comes next, ask the model to predict it, measure how much probability it gave the correct token, and adjust every weight to raise that probability. Cross-entropy loss, gradient descent, backprop. Nothing in the training procedure is specific to language.
Two consequences follow directly from this objective, and they explain most of what people find surprising about these models.
- There is no fact-checking step. The model is optimised to produce plausible continuations. Text that is fluent and false scores well on fluency. Hallucination is not a bug that slipped in; it is what the training objective permits.
- Everything else is downstream of prediction. Translation, summarising, code, arithmetic — all of it is learned because it appeared in text, and predicting text well required getting it right. Instruction-following and refusals are added afterwards by fine-tuning, on top of a model that was only ever taught to continue a document.
Why this replaced recurrent networks
Before 2017, sequences were handled by recurrent networks, which read one token at a time and carried a hidden state forward. The design has an inherent problem: step 500 cannot be computed until step 499 is done. Training could not be parallelised along the sequence. On top of that, information from early tokens had to survive hundreds of updates to the hidden state, and it usually didn’t.
The transformer’s attention step removes both limits at once — the argument Vaswani and colleagues made in Attention Is All You Need, whose title is a fair summary of the paper: drop the recurrence, keep the attention, and the sequence still gets modelled. Every token attends to every other in a single matrix multiplication, so a whole sequence trains in one parallel pass, and the distance between any two tokens is one step rather than five hundred. That is the reason GPUs could suddenly be filled to capacity, and the reason model sizes jumped by orders of magnitude within a few years. The architecture did not become smarter than an RNN. It became trainable at a scale where the same objective started producing very different behaviour.
The cost is quadratic: doubling the context length quadruples the attention work. Most research on long contexts is an attempt to soften that curve.
One sentence is walked through the full stack, matrices drawn out, in Transformers, the tech behind LLMs by 3Blue1Brown.
The practical value of holding this picture is that it makes the model’s limits predictable rather than spooky. Tokenisation explains the spelling failures. The fixed context window explains the forgetting. The prediction objective explains the confident fabrication. What the picture will not give you is any way to say in advance what a model of a given size, trained on a given pile of text, will turn out to be able to do. After several years and a great deal of money, that is still found out by training one and testing it.