How Language Models Learned to See Math: Why Vector Spaces Made LLMs Visual

01 THE PROBLEM WITH WORDS AS SYMBOLS

For most of computing history, a word was an opaque token. Programs treated "bank" as a lookup key, a string to match against other strings, with no notion that it might sit near "river" in one sentence and near "deposit" in another. Language models, on the other hand, cannot afford that innocence. To predict the next word they must somehow weigh every plausible continuation against every other, and that weighing requires a notion of relatedness that no dictionary of discrete symbols supplies.

The breakthrough, and the thing that eventually made this technology explainable to a mass audience, was a change of representation. Instead of storing a word as a symbol, store it as a list of numbers: a vector. Once words are vectors, the vague question "are these words related?" becomes the crisp question "how close are these vectors?" The problem stops being linguistic and becomes geometric.

02 FROM TOKENS TO ARROWS: THE EMBEDDING SPACE

Every large language model begins by mapping each token in its vocabulary to a point in a space with thousands of dimensions: 12,288 of them for GPT-3, to name a publicly reported example. Each dimension is a learned coordinate, adjusted during training by gradient descent until words that behave similarly end up with similar coordinates. Nobody assigns the meanings by hand; they crystallize out of the statistics of prediction.

What emerges is structure that looks suspiciously semantic. The vector for "king," minus the vector for "man," plus the vector for "woman," lands in the immediate neighborhood of "queen." Country names cluster with other country names, verbs with verbs of the same kind. Similarity is measured as the cosine of the angle between two vectors, so relatedness becomes literally a matter of direction, and a sentence becomes a path traced through a landscape of meaning.

03 WHAT THE ATTENTION HEADS DO WITH THAT GEOMETRY

Embeddings are the raw material; attention is the machine that works it. Each attention head projects every token's vector into queries, keys, and values, then lets tokens compare themselves against their neighbors. In geometric terms, each head is asking a directional question of the sentence: which earlier tokens point the same way I do, right now, along the dimensions this head cares about?

That is why the same word behaves differently in different sentences. The vector for "light" is the same, but attention re-weighted against its neighbors pulls it toward physics or toward nutrition, depending on context. It is worth separating the measured from the interpreted here: the query-key dot products are hard mechanics, reproducible from the weights. What each head "means" by its question is an interpretation, and the field's maps of which heads track which relations are still approximate.

04 SIMILARITY AS DISTANCE, MEANING AS DIRECTION

The mental model that 3Blue1Brown's visual explainer made famous is worth stating precisely, because it is the source of both its power and its risks. Words become arrows from the origin. Sentences become regions. Related concepts sit near one another, and certain consistent directions in the space encode consistent semantic properties, like tense or gender or, in famous demonstrations, king-to-queen offsets.

Once you accept that picture, a model's "understanding" becomes something you can inspect rather than something you must take on faith. You can compute the nearest neighbors of any embedding, or subtract and add meaning arithmetically. The intuition demystifies precisely because it replaces an unanswerable question, does the model really understand, with an operational one: where in the space does this token sit, and what does its neighborhood look like?

05 WHERE THE VISUAL INTUITION BREAKS DOWN

Every honest account has to mark the boundary of the metaphor. A 12,288-dimensional space does not fit on a screen; the 2-D and 3-D projections that make the demos work are distortions in which distances are not faithfully preserved. Cosine similarity in the full space is not closeness on the printed page.

More importantly, the geometry is not a semantic theory. It is a compressed record of training data statistics, and it inherits the biases of that data: the direction that encodes gender also encodes the stereotypes the corpus carried. The famous analogy arithmetic works for celebrated examples and fails or offends for many others. The map is real, but it is a map of what the model read, not of what is true.

06 IMPLICATIONS: A FIELD THAT CAN FINALLY SHOW ITS WORK

The visual framing changed how the field communicates with everyone outside it. Papers now ship with annotated UMAP and t-SNE projections of embedding spaces; interpretability researchers probe for feature directions and publish the clusters they find; product decisions, about retrieval, about moderation, about clustering, are made by people reasoning about vector distance as if it were a normal engineering quantity.

There is a subtler consequence. When behavior is framed as geometry, it stops being a black box in a psychological sense, even though the weights remain uninterpretable in detail. The model becomes a machine that occupies positions in a space, moves through that space as it reads, and lands on a distribution over what comes next. Whether or not that deserves the word understanding, it is something a person can reason about with their eyes.

07 LEGACY: THE PICTURE THAT OUTLIVED THE HYPE

The embedding-space picture has outlasted several cycles of hype about language models because it is not attached to any one model. It applies equally to systems that came before, like Word2Vec and GloVe, and to whatever comes after the current generation: as long as meaning is stored as vectors, the geometry is the interface to what the system knows.

That is the durable lesson of the visual turn. The math was always there, in linear algebra that undergraduates learn. What changed is that the public got a way to see it, and once seen, the geometry is hard to unsee. The model stops being an oracle and becomes a place, which is exactly the demystification that was needed.