What Attention Actually Does Inside a Large Model
A plain-language tour of queries, keys, values, causal masking, multi-head attention, induction heads, and why long prompts get expensive.
After text has been turned into tokens and given position-aware vectors, the core of a Transformer layer does one job: let each token look at the tokens it is allowed to see, and decide which of them matter for what comes next. That mechanism is attention — the part that gave the Transformer its name.
Three roles for every token: Q, K, and V
Each token is transformed into three new vectors: Query, Key, and Value. A useful everyday picture is: Query asks “what am I looking for,” Key answers “what can I offer,” and Value carries the information that actually gets passed along when a match is strong. The projection matrices that create Q, K, and V are learned in training, so the model discovers for itself what to seek and what to provide.
Matching is scored by comparing each Query with the Keys it can see, then turning those scores into weights that sum to one. High scores get large weights; low scores nearly disappear. The new representation of a token is a weighted mix of Value vectors — information pulled in from the tokens that mattered.
Take the sentence “The cat that I saw yesterday was sleeping.” When the model reaches was, it needs to know what is sleeping. Attention can give cat a high weight even though several words sit in between, because training taught verbs like was to look for subject-like Keys — and cat produces one that fits.
Causal masking: no peeking at the future
Autoregressive models such as GPT-style systems generate text left to right. A token at position five may attend to positions one through five, but not to six, seven, or eight — those tokens do not exist yet. Causal masking implements that rule by driving future match scores so low that their weights become zero after softmax.
Many heads, not one fixed slice
Language needs many kinds of links at once: subject–verb agreement, pronoun reference, long-range callbacks, local phrase structure. Multi-head attention runs several attention channels in parallel. An important correction to a common textbook mistake: each head does not receive a literal fixed slice of the original vector. Each head has its own learned projections into a smaller Q/K/V space. After the heads finish, their outputs are concatenated and mixed by another learned layer.
Heads often specialize without being told to. Researchers have found heads that track grammar, resolve pronouns, follow position patterns, and — in interpretability work highlighted by Anthropic — induction heads that copy a prior A…B…A pattern by looking back to what followed A last time. That pattern-matching story is one of the clearest known mechanisms behind in-context learning.
Why long prompts cost so much — and GQA
In full attention, each token compares itself with every visible token. Double the prompt length and the comparison work grows roughly fourfold. That cost is why FlashAttention, sparse attention, and related efficiency work exist. At inference time, models also keep a KV cache of past Keys and Values so generation need not recompute the whole prefix — and that cache is a major memory expense for long context.
Grouped-query attention (GQA) is a practical response: many query heads share a smaller set of key/value heads. Models such as LLaMA-2 70B and Mistral 7B use this pattern so quality stays close while memory and inference cost drop. Attention is not one magic trick; it is a family of trade-offs between what tokens can exchange and what you can afford to compute.