How Large Language Models Store Facts

Figure 1: Parameter counts of major LLM generations. GPT-4's architecture is estimated from indirect sources; OpenAI has not officially disclosed its total parameter count.

01 The Mystery Inside the Weights

A large language model is, at its core, a collection of matrices filled with numbers. GPT-3 has 175 billion parameters; GPT-4 is estimated to have over a trillion. Each parameter is a single real number, typically stored in 16-bit floating point format. When you ask an LLM a factual question and it answers correctly, that answer emerged from a series of matrix multiplications that transformed your input tokens into output tokens. But where, exactly, was the factual knowledge stored? Which of those billions of numbers encode the fact that Paris is the capital of France?

This question is the domain of mechanistic interpretability, a research field that aims to reverse-engineer neural networks the way one might reverse-engineer a compiled binary: by identifying the specific components, circuits, and computations that produce observable behavior. The field is young, but it has already produced surprising findings about how neural networks organize information internally.

02 The Transformer Architecture: Where Facts Could Live

To understand where facts are stored, we must first understand the structure that stores them. A modern Transformer-based LLM consists of an embedding layer, a stack of transformer blocks, and an output layer. Each transformer block contains two main components: a multi-head self-attention mechanism and a multilayer perceptron (MLP). The attention mechanism moves information between positions in the input sequence, enabling the model to relate words to each other. The MLP processes each position independently, transforming the representation at each token through two matrix multiplications with a nonlinear activation function in between.

The MLP layers are where most of a model's parameters reside. In GPT-3, the MLP layers account for roughly two-thirds of the total parameter count. Each MLP layer has an "up-projection" matrix that expands the representation into a higher-dimensional space and a "down-projection" matrix that compresses it back. The intermediate dimension is typically four times the model's hidden dimension. This bottleneck structure has led researchers to hypothesize that the MLP layers function as an associative memory, storing facts in the expanded representation and retrieving them through the down-projection.

Figure 2: The MLP layers within each transformer block are the primary suspected site of factual storage, containing roughly two-thirds of total model parameters.

03 Knowledge Neurons: Individual Units That Encode Facts

In 2021, researchers at the AI research institute DAIR identified what they called knowledge neurons: specific neurons in the MLP layers that activate strongly when the model processes particular factual relationships. By identifying which neurons fire when the model sees "Paris is the capital of" and then suppressing those neurons, they could make the model forget that specific fact while leaving other knowledge intact. Conversely, stimulating different neurons could cause the model to produce a different capital city.

This finding was significant because it suggested that factual knowledge is not distributed uniformly across the entire network but is at least partially localized to identifiable neurons. The picture is not clean, however. A single fact may be spread across multiple neurons, and individual neurons may participate in representing multiple facts. The relationship between neurons and facts is many-to-many, not one-to-one. Still, the existence of identifiable knowledge neurons provides a foot in the door for interpretability research: if we can find where a fact lives, we can study how it got there and how it is retrieved.

04 Superposition: More Facts Than Dimensions

A central puzzle in mechanistic interpretability is the problem of superposition. A transformer model might have a hidden dimension of 4,096, but it can store far more than 4,096 distinct features. How? The answer, proposed by researchers at Anthropic in 2022, is that neural networks represent features not as individual dimensions but as directions in a high-dimensional space. Multiple features can share the same dimensions as long as they are nearly orthogonal, allowing the network to store more features than it has dimensions.

This is analogous to how a radio can carry many stations on different frequencies through a single antenna. In neural networks, the "frequencies" are directions in activation space, and the "antenna" is the vector of activations. Superposition explains why simply looking at individual neurons does not reveal a clean mapping to concepts: a single neuron may participate in many superposed features, and a single feature may be distributed across many neurons. Unpacking this superposition requires techniques like sparse autoencoders, which attempt to decompose dense activations into interpretable sparse features.

05 Attention Heads as Information Routers

If the MLP layers store facts, the attention layers route them. Attention heads are the individual components within the multi-head attention mechanism that determine which parts of the input sequence should influence each output position. Researchers have identified specific attention heads that perform identifiable functions: induction heads that copy patterns from earlier in the sequence, name-mover heads that copy token identities to the output, and previous-token heads that attend to the immediately preceding token.

The interaction between attention and MLP layers is what makes factual recall possible. When the model encounters a prompt like "The capital of France is," attention heads route information about "France" to the MLP layers, where it activates the stored representation of the France-to-Paris relationship, and then subsequent attention heads route the resulting information to the output position where "Paris" is generated. This is a simplified picture -- the actual circuit involves many layers and many heads working in concert -- but it captures the division of labor: attention routes, MLP stores.

06 Memorization Versus Generalization

A fundamental question about LLM fact storage is whether the models are truly learning or merely memorizing. If an LLM can answer questions about the Eiffel Tower, did it learn a generalizable representation of what the Eiffel Tower is, or did it memorize a specific association between the string "Eiffel Tower" and a set of related strings?

Research suggests the answer is both. Small models trained on limited data tend to memorize: they can reproduce training examples but fail to generalize to novel questions. Large models trained on vast datasets exhibit grokking, a phenomenon where the model initially memorizes but later, with continued training, transitions to a more generalizable representation. The transition from memorization to generalization is visible in the model's internal representations: the weight matrices reorganize, and features that were initially entangled become more structured and interpretable. This finding suggests that scale and training duration matter not just for performance but for the quality of the internal representation the model builds.

07 Implications for AI Safety and Alignment

Understanding how LLMs store facts has direct implications for AI safety. If factual knowledge is localized in identifiable neurons, it may be possible to surgically edit a model's beliefs: removing a dangerous fact, correcting a false one, or inserting a new one. Early experiments with model editing using techniques like ROME (Rank-One Model Editing) have shown that targeted modifications to MLP weights can change what the model believes about specific subjects, though the edits can have unintended side effects on related knowledge.

Interpretability also matters for alignment: if we cannot understand what a model knows or how it will use that knowledge, we cannot verify that it will behave safely in novel situations. The ability to inspect a model's internal representations, identify stored facts, and trace how those facts are retrieved and used is a prerequisite for trustworthy deployment of increasingly capable AI systems. The research described here is in its early stages, but it points toward a future in which neural networks are not opaque black boxes but systems whose internal workings can be inspected, understood, and controlled.