Take the word “left”. On its own it is close to useless. In “he left the building” it is a verb about departing. In “the left lane” it is a direction. In “there is one slice left” it is about what remains. Same three letters, three unrelated meanings, and the only thing that distinguishes them is the company the word keeps.
A language model has the same problem in a sharper form. It starts with one vector per token, pulled from a lookup table, identical every time that token appears. Before it can predict anything useful, those context-free vectors have to become context-aware. Attention is the mechanism that does it, and it is the only part of a transformer where tokens are allowed to influence each other. Everything else in the architecture processes each token in isolation.
The mechanism itself is a soft lookup. Each token asks a question, every other token advertises what it can answer, the good matches get weighted heavily, and the asker pulls in a blend of what those matches offer.
Queries, keys and values, without the matrices
The three names are unhelpfully abstract, so ground them in the analogy they came from: a database lookup.
In a normal lookup you have a query, you compare it against keys, and you get back the value stored at the key that matched. Attention does the same thing but softly — instead of one key matching, every key matches to some degree, and you get back a weighted average of all the values.
Each token produces three vectors from its current representation, using three learned weight matrices:
| vector | the question it answers |
|---|---|
| query | what am I looking for in the rest of the sequence? |
| key | what do I have to offer to tokens looking for something? |
| value | if someone attends to me, what information do they get? |
The concrete story for the phrase “the left lane”: the token “left” emits a query that means roughly “is there a noun near me that could tell me what kind of word I am?”. The token “lane” emits a key that means roughly “I am a noun about roads”. Those two vectors point in similar directions, so their dot product is large. That is the match.
Now compare “left” against every token in the sequence, one dot product each. Scale the results, push them through a softmax so they are positive and sum to one, and you have a set of weights — a distribution of attention across the sequence. Multiply each token’s value vector by its weight, add them all up, and add the result back onto the vector for “left”. Drawn out, one token’s attention looks like this.
The softmax forces the weights to be positive and to add to 1, so attention is always a blend, never an extra dose. Over half of what “left” absorbs here comes from “lane” — which is how “left” ends up meaning a direction rather than a departure.
That vector is no longer the generic dictionary “left”. It has absorbed the roadness of “lane”. Two blocks later it will have absorbed more. By the top of the stack, a token’s vector is less a word than a summary of a position in a specific sentence.
Keys and values are separate for a real reason. What makes a token findable is not the same as what makes it useful. A token may want to advertise its grammatical role to be found, while the information worth passing on is its meaning. Splitting the two lets the model learn each independently.
Many heads, because words need several things at once
One attention operation produces one pattern of who-looks-at-whom. That is not enough. In a real sentence a verb needs to find its subject, a pronoun needs its referent, an adjective needs its noun, and none of those searches want the same weighting.
So the operation runs many times in parallel with different weight matrices — typically a few dozen heads per layer — and the results are concatenated. Each head is free to specialise. In practice some do specialise legibly: heads that track the previous token, heads that link pronouns to names, heads that match brackets. Many others do something nobody has managed to name.
That last point deserves emphasis, because attention maps are frequently presented as explanations of model behaviour. They are not. A head attending strongly from one word to another tells you information flowed along that edge; it does not tell you the model’s answer depended on it, and interventions regularly show it did not. This is the same trap as reading a feature importance chart as a causal account, covered in feature importance is not explanation. Attention weights are a mechanism, not a justification.
Why this fixed long-range dependencies
Before transformers, sequences were handled by recurrent networks, which read one token at a time and carried a hidden state forward. For a word at position five to influence a word at position five hundred, its contribution had to survive 495 successive updates to that state. It generally didn’t. Signals faded, and the network’s effective memory was much shorter than its input. Attention was invented as a patch on exactly that failure: Bahdanau, Cho and Bengio’s 2014 translation paper Neural Machine Translation by Jointly Learning to Align and Translate let the decoder look back at every source word instead of squeezing the whole sentence through one fixed vector. The transformer kept the patch and threw away the recurrence it was patching.
Attention removes the distance entirely. The dot product between token five and token five hundred is computed directly, in one step, exactly as cheaply as between two adjacent tokens. There is nothing to fade. Position matters only because positional encoding put it there deliberately, not because the architecture imposes a decay.
That single change is what made long-range coherence possible, and it is the reason a model can resolve a pronoun to a name mentioned three paragraphs ago.
The bill arrives as compute. Every token compares against every token, so the work grows with the square of the sequence length. Doubling the context quadruples the attention cost. This is why context windows expanded slowly and expensively, and why a large fraction of current research is about approximating attention without paying the full quadratic price.
One more detail matters for generation. A model predicting the next token must not see the future, so the attention scores for later positions are set to negative infinity before the softmax — they get zero weight. This is causal masking, and it is the only structural difference between a model that generates text and one that only reads it.
To watch the score matrix fill in as a sentence is processed, see Attention in transformers, step-by-step by 3Blue1Brown.
The word “attention” oversells the mechanism and undersells the result. Nothing is attending to anything; there are dot products, a softmax, and a weighted sum. But that arrangement solved a problem — letting every part of a sequence reach every other part in one cheap, parallel step — that a decade of clever recurrent designs had not. Most of the last few years in language modelling is the consequence of running it at scale.