Retrieval-augmented generation: how LLMs learn to cite their sources

01Why a fluent LLM can still be confidently wrong

A large language model is trained on a vast amount of text and becomes the basis for many modern chatbots — ChatGPT, Claude, Gemini, Grok, DeepSeek and their peers. Fluency is what such a model is optimized to produce, and fluency is exactly what makes its failure mode dangerous: when the model does not know something, it does not fall silent. It completes the pattern in the most plausible voice available, which reads as authority even when the underlying facts were thin, outdated, or never in the training data at all.

Three structural gaps drive most of these errors. Knowledge has a cutoff — anything that happened after training began simply is not in the weights. Private facts — your contracts, tickets, or internal documentation — were never there to learn. And provenance is absent: even a correct answer arrives with no citation anyone can check. Retrieval-augmented generation, first proposed in 2020 and since widely adopted, attacks all three at once by making the model look things up before it answers rather than trusting a frozen internal summary.

02The RAG loop: retrieve, augment, generate

The core loop has three moves. First, retrieve: the user's question is transformed into a search and fired against an external corpus — a document store, a wiki, a ticket database — returning the passages most relevant to the query. Second, augment: those passages are spliced into the prompt alongside the question, so the model reads its answer material instead of recalling it. Third, generate: the model writes its response grounded in the retrieved text, ideally with quotes or references pointing back at the source documents.

The pipeline is usually split into an offline half and an online half. Offline, documents are chunked into passages, converted into vectors, and loaded into an index — work that runs whenever the corpus changes. Online, the query path runs end to end in milliseconds, and the retrieval step acts as a filter that the generation step can be audited against. That auditability is the quiet product feature: when an answer is wrong, engineers can replay the retrieval and see exactly which documents the model was looking at.

03Embeddings and vector search explained

The trick that makes semantic retrieval work at scale is the embedding: a text passage mapped to a point in a space of hundreds or thousands of dimensions, arranged so that passages about similar topics land near each other. An embedding model is itself a neural network — typically a transformer variant — trained so that distance in vector space approximates similarity in meaning. A question about refund windows lands close to the policy paragraph that answers it, even if the two share no keywords.

Search then becomes geometry. The corpus's vectors live in an index built for approximate nearest-neighbor lookup, and the query vector's neighbors become the retrieved passages. Practical systems rarely stop there: keyword or BM25 retrieval runs alongside the vector search and the results are merged, because lexical matching still catches exact identifiers — error codes, product names, statute numbers — that embeddings blur. A reranking model often rescores the merged candidates, keeping the small handful the generator will actually read.

04RAG versus fine-tuning: when to use which

Fine-tuning changes the weights; RAG changes the context. That distinction decides nearly everything about when to use each. Fine-tuning teaches a model to behave differently — new tone, output formats, domain vocabulary, task-specific judgment — and the change is permanent and expensive to revise. RAG hands the model information it was missing, and the information can be corrected, extended, or deleted by editing the index, with no retraining at all.

As a rule of thumb: if the knowledge changes — prices, policies, documentation, anything with a date — it belongs in a retrieval corpus, because the index updates in minutes and each answer can cite its source. If the skill changes — the model should write in your house style or parse your bespoke log format — fine-tuning is the better tool, because no amount of retrieved text reliably teaches a behavior. The two compose: plenty of production systems fine-tune a model for domain style and then retrieve over domain facts, which is why the choice is rarely exclusive.

05Where RAG already ships: search, support, enterprise

RAG stopped being a research demo years ago; it is now the default architecture for products whose answers must be right about things that change. Web search assistants retrieve and summarize live results rather than answering from memory. Support bots retrieve from knowledge bases and past tickets, with citations that let agents verify the bot before escalating. Enterprise assistants sit on top of internal wikis, contracts, and code, scoped by access control so the retrieval layer only ever surfaces documents the asking user may read.

One quiet consequence of the trajectory above deserves note: as context windows grew from a few thousand tokens to a million, some predicted retrieval would become obsolete — just stuff everything into the prompt. In practice long contexts raised, rather than lowered, the value of good retrieval: attention degrades over huge windows, costs scale with every token sent, and a focused set of retrieved passages still beats a haystack for both accuracy and latency. Big windows changed how much can be retrieved; they did not change whether retrieval is needed.

06The failure modes: bad retrieval, stale indexes, security

A RAG system inherits the weaknesses of a search engine and a language model at once. Bad retrieval is the most common: the vector search misses the right passage, or retrieves near-misses that are superficially similar, and the generator confidently builds an answer on the wrong foundation — garbage in, eloquence out. Chunking choices quietly set the ceiling: chunks too small lose context, chunks too large dilute the signal, and neither mistake announces itself in the output.

Staleness and security follow. An index that reflects last quarter's policy produces answers that were correct once, which is often worse than an admitted gap; keeping embeddings fresh with the source documents is unglamorous operational work that never stops mattering. On the security side, the retrieval layer is an attack surface: an attacker who can plant text in a retrievable source — a poisoned document, a malicious ticket, a public web page — can steer what the model reads and, through it, what it says. Prompt-injection via retrieved content is now a standard threat model, and the defense is to treat retrieved text as untrusted input, never as instructions.

07Agentic retrieval and what comes next

The latency profile above explains the newest design pressure: because generation dwarfs every retrieval stage, spending more compute on retrieval is nearly free — and agentic systems spend it. Instead of a single retrieve-then-answer pass, an agentic pipeline decomposes the question, issues several searches, reads the results, decides what is still missing, and loops — retrieving again with refined queries until the evidence is sufficient. The model stops being a one-shot answer machine and becomes a controller orchestrating its own lookups.

What comes next is less a new architecture than a maturing discipline. Graph-augmented retrieval connects facts instead of ranking passages in isolation. Cache layers reuse embeddings so repeated questions skip the search entirely. Evaluation suites treat retrieval quality as a first-class metric rather than a footnote under generation scores. The 2020 paper's core idea — let the model consult an external source and say what it read — has hardened into infrastructure, and the systems being built on it are judged by an old standard that fluency alone never met: is the answer true, and can you check.