Model Collapse: What Happens When AI Trains on Its Own Output

01 The Feedback Loop Problem

For most of the internet's history, the raw material of language-model training was a byproduct of human activity: forum posts, blog entries, news articles, code repositories, and encyclopedia edits, all written by people for people. That assumption is quietly dissolving. Generative systems now produce an enormous share of new public text, and much of it flows back into the crawls, scrapes, and archives that feed the next generation of models.

The result is a feedback loop with no natural exit. A model generates text, that text is published, repackaged, spun into derivative articles, and eventually swept up by a crawler that cannot reliably distinguish it from human writing. The next model trains on a corpus contaminated by the previous model's output. Each pass around the loop compounds the problem, because the synthetic share of the corpus grows while the provenance of any given document becomes harder to verify.

What makes this loop genuinely new is scale. Earlier eras of machine learning drew from curated, well-documented datasets assembled by institutions that knew exactly what they contained. Today's frontier models are trained on web-scale corpora assembled opportunistically, where nobody can say with confidence how much of the material is machine-written. The loop is not a hypothetical scenario; it is a description of the default data pipeline for modern AI.

02 What Model Collapse Actually Is

The degradation that results has a name. In artificial intelligence, model collapse, also known as "AI inbreeding", "AI cannibalism", "Habsburg AI", and "model autophagy disorder" or "MAD", is the degradation of machine learning models from uncurated synthetic data, or from training on the outputs of another model, such as prior versions of itself. The colorful nicknames all point at the same structural flaw: a system consuming its own byproducts instead of fresh external nourishment.

The core failure is a loss of variety. A language model does not memorize the world; it approximates the statistical distribution of its training text. Every real-world distribution has a body of common patterns and a long tail of rare ones: unusual words, atypical phrasings, minority viewpoints, edge cases. When a model is trained on synthetic data that approximates a distribution, it reproduces the body faithfully but lets the tail thin out. Train on that thinned output again, and the tail thins further. Variance shrinks generation by generation until the model can only produce a narrow set of highly probable strings.

The foundational study of this dynamic is the 2023 Nature paper by Ilia Shumailov and colleagues, "The curse of recursion: training on generated data makes models forget". The paper formalized what recursion does to distributions and demonstrated the outcome experimentally. It frames the risk starkly: without careful curation, each round of training on model output is a lower-fidelity copy of the last, a statistical photocopy of a photocopy.

03 Why Tails Disappear

The mechanism behind collapse is subtle, because the early generations look fine. A model trained on a mixture of real and synthetic data will often perform adequately on common queries, exactly the territory the synthetic body of the distribution captures well. The damage is hidden in the tail: rare-but-real patterns that appeared a handful of times in the original corpus but were never prominent enough for the model to reproduce them reliably.

The asymmetry is the key. When a model samples from its learned distribution, it over-represents high-probability patterns and under-represents rare ones. The synthetic data it emits is therefore a biased summary of its training set, weighted toward the average. Rare-but-real data carries disproportionate information precisely because it is uncommon; an averaged synthetic summary strips it out. Feeding that summary back in as training data compounds the averaging. With enough recursion, the model converges toward a narrow band of generic output, competent at the center and blind at the edges.

The photocopy analogy is imperfect but useful. Each generational copy loses resolution, and the losses are not random: they fall first on whatever was faintest in the original. What survives is the loudest, most common signal. A corpus treated this way for many generations becomes a smoothed caricature of the human text it started from, and a model trained on it inherits the caricature. The tail does not vanish in one step; it dissolves in fractions, generation after generation.

04 The Evidence So Far

The strongest experimental evidence remains the Shumailov et al. work published in Nature in 2023. In the paper's experiments, a language model fine-tuned on data that was 99 percent synthetic text degraded measurably within a few generations of recursive training. By later generations the model had lost most of the distributional diversity of the original corpus, a quantitative demonstration that unmanaged recursion is a slow-motion failure, not a stable practice. The finding was central enough that the paper itself frames the phenomenon as a curse of recursion.

Evidence from applied domains has followed. A 2025 study by researchers at Oxford and Lancaster examined what happens when models absorb large volumes of LLM-generated medical content, finding that training on such material reduced model reliability in downstream tasks. The medical domain is a useful probe because errors are costly and the true distribution has a long tail of rare conditions, drug interactions, and edge cases that averaged synthetic text handles badly. It also illustrates the practical stakes: collapse is not only about stylistic blandness but about correctness in the tail where correctness matters most.

It is worth separating what is measured from what is interpreted. The Nature experiments measure diversity loss under controlled recursion with specific models and ratios; they do not prove that any particular production model has collapsed, because frontier labs do not publish their data mixes. What the evidence establishes is a mechanism, demonstrated under controlled conditions, whose preconditions, large synthetic fractions and weak curation, are increasingly satisfied in the wild. That distinction between demonstrated mechanism and inferred field prevalence is the honest reading of the literature.

05 Mitigations and Guardrails

The obvious mitigation is the expensive one: keep verified human data in the loop as an anchor. Rather than letting recursion run unconstrained, labs can interleave fresh human-written material into every training cycle, so the model is continuously re-grounded in the real distribution rather than its own approximation of it. The hard part is verification, because on an open web the label "human-written" is increasingly a claim rather than an observable fact.

Provenance tracking is the second pillar. If datasets recorded where each document came from, when it was collected, and whether it passed through a generative system, synthetic content could be quantified, capped, or excluded at training time. In practice, provenance on web-scale corpora is patchy, and detection of machine-generated text is unreliable at the margins, so provenance works best where the pipeline is controlled: licensed archives, editorial feeds, and purpose-built corpora with known histories.

It is also important not to overcorrect into the belief that all synthetic data is poison. The counterexample is game-playing systems such as AlphaGo and AlphaZero, which trained on vast volumes of self-play data and improved beyond anything human data could teach. The difference is ground truth: those systems generated synthetic experience inside a simulator with hard rules and a scoring function, so every generated sample carried reliable information about the world. Synthetic data improves models when it is grounded in a verifier; it degrades them when it is merely an average of what the model already believes. Alongside provenance, practical guardrails include filtering, deduplication, and careful curation of synthetic fractions, all of which slow the thinning of the tail even if none eliminates the underlying mechanism.

06 What It Means for 2026

The market has already priced in part of this risk. Frontier labs are now paying for human data through licensing agreements with publishers, archives, and content owners, effectively bidding for the scarce resource that recursion cannot manufacture: verified, high-variance, human-written text. What was once a free byproduct of the open web has become a commodity with a price and a contract, which is itself an acknowledgment that undifferentiated web crawls are degrading in quality.

Two curves are colliding. The industry's demand for training data keeps growing as models scale, while the share of fresh, unambiguously human text on the open web keeps shrinking under the flood of generated content. Commentators have called the first curve the data wall; the second is the slop flood. Where they meet, the economics of AI training change: data acquisition becomes a competitive moat rather than an afterthought, and the labs with the best-licensed, best-provenanced corpora hold an advantage that raw compute cannot buy back.

The open questions are the ones worth watching. Nobody outside the frontier labs knows how large the synthetic fraction in any production corpus actually is, or how much deliberate curation is already offsetting the recursion effect. The mechanism is proven, the preconditions are visible, but the state of the field is uncertain. If the mitigations hold, model collapse will remain a cautionary laboratory finding; if they slip, it becomes the quiet structural failure of the decade. Either way, the era of treating the open web as an inexhaustible well of human text is over.