The Attention Mechanism: A Visual Guide to How AI Reads Text

01 What Attention Looks Like

When a transformer reads the sentence 'The cat sat on the mat because it was tired,' different attention heads focus on different things. One head connects 'it' to 'cat' (coreference resolution). Another connects 'sat' to 'mat' (semantic roles). Another connects 'because' to 'tired' (causal reasoning). Each head learns a different type of relationship. Together, they form a web of connections that constitutes the model's 'understanding' of the sentence.

02 The Heads That Don't Matter

Not all attention heads are important. Research from 2023 showed that you can prune 40-60% of attention heads with minimal performance impact. The remaining heads do most of the work. This suggests that transformers are overparameterized — they have more capacity than they need. This is good news for efficiency (you can run smaller models) but raises a question: if 40% of the model does nothing, are we wasting compute training it? The answer is probably yes, but identifying which 40% is useless without running the model, which defeats the purpose.

03 Attention Is Not Explanation

There's a debate in the AI research community about whether attention weights are explanations. When a model pays 0.8 attention to 'cat' when processing 'it,' does that mean the model 'thinks' 'it' refers to 'cat'? Maybe. But attention weights can be misleading — a model might attend to a word for reasons unrelated to the task (e.g., syntax processing). The consensus: attention is a clue, not an explanation. For true understanding of model behavior, we need interpretability techniques (like sparse autoencoders) that go beyond raw attention patterns.