Inside the Black Box: Mechanistic Interpretability

01Interpretability is reverse engineering

Mechanistic interpretability treats a trained neural network less like a table of scores and more like a piece of software whose implementation was never written down. The weights are the artifact; the researcher’s job is to recover useful algorithms from them.

That ambition distinguishes it from a saliency map that merely highlights input pixels. The question is causal: which internal components combine to produce this output, and what happens if one of them is changed?

02Features are not always one-neuron concepts

A tempting picture is one neuron per idea: one unit for “dog,” one for “syntax,” one for “Paris.” Real networks often reuse dimensions. Multiple concepts can be packed into the same directions, a phenomenon usually described as superposition.

The result is polysemanticity: a feature may respond to several things that look unrelated until the surrounding circuit is examined. Finding a strong activation is a clue, not an explanation.

03Circuits connect features to behavior

A circuit is a causal chain of feature activations. In a language model, an input pattern can activate an early representation, route through attention or a feed-forward block, and influence a later token distribution. The useful unit of analysis is therefore often a pathway, not an isolated neuron.

Chris Olah’s circuits framing made that goal concrete: identify small, interpretable mechanisms, test them with interventions, and build upward toward larger computations.

04Sparse autoencoders separate the mixture

Sparse autoencoders, or SAEs, are trained to reconstruct a model’s activation while using only a small number of latent features. The hope is that these sparse features align better with human concepts than the original, densely mixed coordinates.

The technique is not a magic decoder. Reconstruction quality, feature splitting, dead features, and the choice of layer all matter. A feature that has a neat label still needs causal validation.

05Scale makes the map harder

The GPT-2 family is a useful ruler: 12, 24, and 36 layers across its small, medium, and large variants, with hidden widths from 768 to 1,280 dimensions. More layers and dimensions give a model room to represent richer computations, but they also increase the search space for researchers.

That is why mechanistic work often begins with toy models or a narrow behavior. A complete map of a frontier system is not currently a realistic assumption; targeted circuits are.

06Causal tests beat attractive stories

Interpretability can fail in a familiar scientific way: a researcher notices a pattern, gives it a compelling name, and mistakes correlation for mechanism. Activation patching, ablation, feature steering, and counterfactual prompts are ways to ask whether the proposed component actually changes the output.

A good explanation should predict what happens under intervention. If disabling a supposed “planning” circuit does nothing, the label was premature—or the circuit is redundant.

07The safety case is a measurement problem

Mechanistic interpretability is valuable for safety because behavior tests reveal what a model did, while internal analysis may reveal what it is trying to do across contexts. Neither view is sufficient alone. A model can contain a dangerous capability that a benchmark never elicits, and a circuit map can be incomplete.

N43’s conclusion is modest but important: interpretability is not mind reading. It is experimental science for learned computation—hypothesize, intervene, measure, and publish the uncertainty around the map.