Where Do Large Models Store Their Facts?
Attention lets tokens talk. Feed-forward layers do much of the deep processing — and hold a surprising share of factual knowledge, plus the scaffolding that makes deep stacks trainable.
Attention gets most of the headlines. Inside each Transformer layer, though, there is a second step that is quieter and no less important: the feed-forward network, or FFN. If attention is tokens exchanging notes across the sequence, the FFN is each token closing the door and doing deep processing on its own vector — with no mixing across tokens at that step.
Expand, bend, compress
A typical FFN widens the token vector, applies a nonlinearity, then projects back to the original width. The original Transformer used a four-times expansion; modern SwiGLU-style models vary the ratio, but the big shape stays. Nonlinearity matters because stacked linear layers collapse into a single linear map. Without a bend in the middle, the FFN cannot compute much richer functions than one matrix multiplication.
Activation functions themselves evolved: ReLU in the original Transformer, later GELU in GPT and BERT eras, then SwiGLU in many LLaMA-, Mistral-, and PaLM-era models. The outer expand-and-compress pattern stayed; the middle nonlinearity kept changing.
Much of the “knowledge” lives here
In dense Transformers, most parameters sit in the feed-forward layers, not in attention. Interpretability work has linked certain neurons to concepts or facts — the Eiffel Tower, programming languages, past-tense verbs. The claim that Paris is the capital of France is not a sentence stored in a database; it shows up as weight patterns and activations inside particular FFNs.
That storage story has a striking corollary. Methods such as ROME (rank-one model editing) can change an association — for example, editing “Eiffel Tower in Paris” toward “Eiffel Tower in Rome” — with a targeted low-rank change to FFN weights, without full retraining. The research is still early, but it shows that our picture of where facts live is concrete enough to edit by hand in limited settings.
MoE: many experts, few activated
Some frontier models replace a single dense FFN with several parallel expert FFNs and a small router that sends each token to a few of them. Mixtral 8x7B is a clear example: eight experts per layer, two activated per token — large total parameter count, much smaller active compute per token. MoE raises new engineering problems around routing and training stability, but it changed how teams think about scaling parameters without scaling cost one-for-one.
Residual streams and normalization: unglamorous, essential
Without residual connections and layer normalization, very deep networks barely train. Residuals add a sub-block’s output to the existing vector instead of replacing it, so early embeddings keep a direct additive path through dozens or hundreds of layers. The idea comes from ResNets in computer vision (He et al., 2015); Transformers inherited the same shortcut. Modern interpretability often treats the residual stream as a highway that every head and FFN reads from and writes back to.
Normalization keeps those running sums from exploding or collapsing. Two modern defaults matter: pre-norm (normalize before the sub-block, as in GPT-2 and many open models) trains deep stacks more reliably than the original post-norm design; and RMSNorm often replaces full LayerNorm by scaling without recentering — cheaper, and empirically enough for much of the benefit. Facts may live in FFN weights, but residual paths and normalization are why those deep stacks can exist at all.