From Words to Vectors: How Embeddings Give Language Models Meaning

01Words Become Numbers

A computer cannot read the word river; it can only store and transform numbers. The naive fix, assigning each word an arbitrary ID number, tells the machine nothing about what the word means. The breakthrough idea of embeddings is to represent each word as a point in a high-dimensional space, a long list of floating-point values called a vector, learned from data rather than assigned by hand. When a model is trained on large amounts of text, words used in similar contexts drift toward similar coordinates. King and queen end up near each other; king and sandwich do not. Meaning becomes measurable distance, and every downstream capability of modern language models, from translation to retrieval, rests on that single move.

02Word2vec and Static Vectors

The technique went mainstream in 2013, when a Google team led by Tomas Mikolov published word2vec. The system learned one fixed vector per word, most commonly with 300 dimensions, by predicting which words tend to appear near which other words. It famously captured analogies as arithmetic: the vector for king, minus man, plus woman, lands close to queen. Stanford's GloVe, published in 2014, reached the same destination by counting word co-occurrences across entire corpora. The limitation was rigidity. A static vector assigns bank one location, even when a sentence is about rivers rather than money. Polysemy, the everyday reality that most words carry several senses, could not be expressed by any single point.

03Contextual Embeddings Inside Transformers

Transformers solved that problem by making embeddings dynamic. In models such as BERT, published by Google researchers in 2018, each token does not get one fixed vector; instead, the model computes a fresh representation for every occurrence, shaped by the full sentence around it. In she sat on the bank of the river, the vector for bank is pulled toward water-related neighbors; in a finance sentence, it drifts elsewhere. BERT-base used a hidden size of 768 dimensions, and those internal vectors, produced layer by layer, are what actually carry meaning through the network. Modern large language models continue this design: an embedding layer converts input tokens into vectors, and every subsequent operation is linear algebra on them. Today, dedicated embedding services such as OpenAI's text-embedding-3-small return vectors of 1536 dimensions.

04Measuring Meaning with Cosine Similarity

Once words are vectors, comparing meaning reduces to comparing directions. The standard measure is cosine similarity: take two vectors, compute the cosine of the angle between them, and get a score from minus one to one. Two words used in similar contexts point roughly the same way and score high; unrelated words point in different directions and score near zero. Because the measure uses angle rather than raw distance, it ignores how long the vectors are, which matters when comparing texts of different lengths. This one calculation powers nearly everything built on embeddings. Search engines rank documents by it, clustering algorithms group neighbors with it, and retrieval pipelines use it to decide which chunks of text are relevant to a question. The dot product, its cheaper cousin, serves the same role inside model internals.

05Semantic Search and RAG

Embeddings turned search from keyword matching into meaning matching. A query like cheap flights to Japan shares almost no keywords with a page reading budget airfare to Tokyo, but their embeddings point in similar directions, so a similarity search finds it anyway. Retrieval-augmented generation, or RAG, extends the idea: an application embeds its document collection, embeds the user's question, finds the nearest chunks by cosine similarity, and pastes them into the language model's context as grounding material before it answers. The model is no longer limited to what it memorized during training; it can cite fresh, private, or domain-specific text on demand. That is why RAG became the default architecture for enterprise AI systems, and why the quality of an application often depends less on the model than on the retrieval layer feeding it.

06Why Vector Databases Exist

Here is the engineering problem that birthed an industry: similarity search over millions or billions of vectors does not fit normal database indexes. Comparing a query against every stored vector, brute force, is linear in collection size, and at web scale it becomes too slow and too expensive. Vector databases such as Pinecone, Weaviate, Milvus, and pgvector for PostgreSQL store embeddings and answer nearest-neighbor queries using approximate algorithms such as HNSW, which trade a sliver of accuracy for orders-of-magnitude speedups. Their secondary job is housekeeping: updating vectors as documents change, filtering by metadata, and sharding across machines. As the Fireship source video puts it, the entire "so hot right now" moment for vector databases happened because RAG pipelines needed somewhere to put their embeddings, and nothing already installed could search them fast enough.

07Why This Architecture Won

Looking back, embeddings succeeded because they convert the fuzzy problem of meaning into problems engineers already knew how to solve: distance, angle, and nearest-neighbor search. Static vectors from the word2vec era proved the concept; contextual vectors inside transformers made meaning sensitive to usage; cheap embedding APIs made the technology a utility; and vector databases made it searchable at production scale. The stack is now so standardized that a competent developer can assemble semantic search over a private corpus in an afternoon. What has not standardized is semantics itself: embeddings inherit the biases and gaps of their training text, and similarity to a query is not the same as truth. The geometry is powerful, but it is only as good as the corpus that shaped it.