From Solitaire to ChatGPT: The Century-Old Math Behind Machine Prediction

01 A Card Game That Broke the Exact Math

In 1946, the mathematician Stanislaw Ulam was recovering from an illness at Los Alamos, whiling away the hours with patience - the card game most people know as solitaire. He found himself wanting to know how often a randomly dealt hand actually resolves to a win. The natural instinct of a mathematician is to enumerate: count every arrangement of the deck, classify each one, divide. But a standard deck admits roughly 52 factorial orderings, a number so large that enumerating it is not merely impractical; it is inconceivable. Ulam's insight, recalled in his memoir and retold in the Veritasium video that anchors this piece, was to abandon exactness. He asked instead: what if I just play the game a few hundred times, record the outcomes, and count the fraction of wins?

The idea sounds almost dishonorable - replacing proof with a tally of coin-flips. Its legitimacy rests on a theorem from the previous century: the law of large numbers, which guarantees that the average of many independent random trials converges on the true underlying probability. Ulam had grown up studying under Banach in Lwow, where probability questions of exactly this shape were daily currency. He mentioned the scheme to John von Neumann, who immediately saw its fit for neutron diffusion calculations then stalled at Los Alamos, and the two, with Nicholas Metropolis, carried the approach into the Manhattan Project's computing machinery. The technique needed a suitably glamorous code name for a paper, and Metropolis, drawing on a uncle's fondness for Monte Carlo's roulette tables, supplied one.

What matters for the present moment in artificial intelligence is the shape of the move Ulam made. Faced with a system too complex to compute exactly, he did not simplify the system - he sampled it. Every Monte Carlo simulation since, from weapons work to weather ensembles to the statistical reasoning that now sits inside chatbots, is a descendant of a sick man playing cards. The 1949 paper by Metropolis and Ulam, registered with the American Mathematical Society, formalized the method: draw random inputs, run the deterministic machinery, average the outputs. That pattern - exactness traded for distribution - is the intellectual ancestor of everything else in this article.

02 Markov's Chains: States Without Memory

The mathematics that makes sampling and prediction work traces back to Andrey Markov, a Russian mathematician working in Saint Petersburg at the turn of the twentieth century. Markov's celebrated 1906 contribution was a class of stochastic processes that now bear his name: a Markov chain is a sequence of possible events in which the probability of each event depends only on the state attained in the previous event, and not on the longer history that produced it. This property - the Markov property, or memorylessness - is the structural simplification that makes the whole edifice tractable. Instead of tracking an entire trajectory, you track a single state and a table of transition probabilities.

Markov himself was a combative, atheist contrarian who delighted in quarrels with the church and the academy alike, and he chose a pointed demonstration for his mathematics. In a 1913 analysis, he applied his chains to the text of Alexander Pushkin's verse novel Eugene Onegin, counting how often a vowel follows a vowel versus a consonant across twenty thousand letters. The choice of text was a jab - probability was then regarded as fit for gambling and insurance, not for high literature - but the exercise proved the point: letter sequences in Russian verse behave like a Markov process, with each letter's identity conditioned on its predecessor in measurable, stable proportions. It was, arguably, the first statistical model of language, and it was built forty years before electronic computers existed.

The Wikipedia summary of Markov chains in this article's reference list states the definition plainly: a stochastic process describing a sequence of possible events where the probability of each event depends only on the current state. That dry sentence is the load-bearing wall of modern machine prediction. When a language model decides what the next word should be, it is, structurally, doing what Markov's vowel-counting did: conditioning a prediction on the state of the sequence so far. The state has grown enormously richer, and the conditioning is learned rather than counted by hand, but the skeleton is his.

03 Monte Carlo: Prediction by Repeated Sampling

The Monte Carlo method that Ulam and Metropolis formalized in 1949 is the practical engine that grew on Markov's theoretical root. The Wikipedia summary describes it as a broad class of computational algorithms based on repeated random sampling to obtain numerical results, and the essential trick is that the randomness lives in the input, not the output. The physical process being studied - neutron diffusion, card shuffles, molecular motion - is treated as deterministic given a configuration; what the analyst does not know is which configuration reality has chosen. So the analyst draws configurations at random, runs the process for each, and lets the law of large numbers grind the sample average toward the true answer.

Two refinements turned the method from a clever hack into a pillar of applied mathematics. The first was importance sampling, which biases the random draws toward the configurations that matter most and re-weights the results accordingly, buying accuracy per sample. The second was the realization, due largely to Metropolis and colleagues in 1953, that you can construct a Markov chain whose long-run distribution is exactly the distribution you want to sample from - the birth of Markov chain Monte Carlo, MCMC. That fusion is precisely what its name advertises: Markov's memoryless sequences, driven by Monte Carlo randomness, converging on the statistics of systems too complex to touch any other way. Modern Bayesian statistics is built on it.

The line to contemporary AI is direct and often under-appreciated. When a language model generates text, it samples from a probability distribution over its vocabulary at every step - temperature, top-p, and the other sampling knobs exposed by inference APIs are literally Monte Carlo controls. And the reasoning-style inference that emerged in the mid-2020s, where a model works through long chains of intermediate steps before answering, can be read as Monte Carlo over chains of thought: the model explores possible solution paths stochastically, and the answer is whichever branch the sampling process commits to. The 1946 solitaire problem and the 2026 chatbot are doing the same kind of arithmetic: trading exact computation for repeated random trials, and trusting the theorem that the average will hold.

04 From Letters to Tokens: Shannon's Sequence Bet

In 1948, Claude Shannon published A Mathematical Theory of Communication, the founding document of information theory, and buried inside it was the second half of the ancestry. Shannon wanted to measure the redundancy of English, and to do so he built generative models of text: start with a table of letter frequencies and produce letters at random; then upgrade to tables of which letter follows which - a Markov chain, directly in Markov's own sense; then upgrade again to which word follows which word. As the conditioning window widens, the output swings from gibberish to eerie quasi-English, and Shannon's famous n-gram samples read like a seance with the language itself.

Shannon's deeper contribution was a quantity: entropy, the number of bits per symbol that a source actually carries once its statistical structure is accounted for. Entropy tells you how predictable a sequence is, and therefore how much of it could be guessed by a well-informed observer. That framing converts prediction from an act of understanding into an act of statistics, and it set the terms for everything that followed. The perplexity metric used to grade language models today is a direct descendant - entropy measured per word - and the entire training objective of the modern large language model, next-token prediction, is Shannon's exercise run at industrial scale: given the preceding context, estimate the distribution over what comes next.

The Veritasium video at the head of this article makes the connection vivid: what separates Shannon's hand-counted tables from a transformer is not the goal, which is identical - predict the next symbol - but the capacity of the table. An n-gram model conditions on the last few words exactly; a transformer conditions on the entire context, every token in the window, with learned attention deciding which parts matter. The jump in fidelity between Shannon's two-letter tables and a modern model's distribution is a matter of the size of the conditioning state, not of a change in the underlying question.

05 PageRank: A Markov Chain the Size of the Web

The next milestone arrived in 1998, when Sergey Brin and Larry Page, then at Stanford, published The Anatomy of a Large-Scale Hypertextual Web Search Engine, the paper that described Google. PageRank's core idea is usually explained in the language of reputation: a page is important if important pages link to it. But the mathematical object underneath is a Markov chain, and it is worth seeing it in those terms. Model a hypothetical web surfer who at each step clicks a random outbound link from the current page, occasionally teleporting to a random page anywhere on the web. The surfer's position is the state; the link structure defines the transition probabilities; the teleportation - the damping factor - guarantees the chain is well-behaved and converges. A page's PageRank is the long-run fraction of time the surfer spends there.

This is the stationary distribution of a Markov chain, the equilibrium toward which the process settles regardless of where it started, and computing it for the actual web graph was one of the largest applications of Markov's mathematics ever attempted. The scale is the story. Markov analyzed twenty thousand letters of Pushkin; PageRank analyzed a web of hundreds of millions of pages, with the transition matrix so vast that it could never be materialized - only iterated against, using the power method, exactly the sort of repeated operation Monte Carlo thinking had made routine. The random surfer is Ulam's card player and Markov's letter-counter wearing a new costume: a stochastic process too large to enumerate, sampled and iterated into a usable answer.

The commercial consequences need no retelling here - a search engine built on this computation became one of the most valuable companies on earth. What matters for this article's argument is that the ranking of the entire web, in 1998, was formulated as a prediction problem about a memoryless random process, a decade before anyone spoke of deep learning. The intellectual continuity is unbroken: the same mathematical machinery that counted vowels in Eugene Onegin was, ninety-two years later, deciding which pages a hundred million people would read each morning.

06 Transformers: Same Question, Bigger Table

The 2017 paper Attention Is All You Need, by Vaswani and colleagues at Google, introduced the transformer, the architecture behind every large language model in current service, from the earliest GPT releases onward. The technical novelty was the attention mechanism: instead of compressing context into a fixed-size hidden state, as earlier recurrent networks did, a transformer lets every token directly consult every other token in its window, learning per-context which relationships matter. This made models trainable at unprecedented scale and gave them a context-sensitivity that n-grams could never approach. Wikipedia's summary of large language models notes that they are trained on vast text corpora for generation tasks, and the training objective is the old one: predict the next token.

But strip away the engineering and the activity is recognizably Markovian in structure. The model maintains a state - the context window - and produces a probability distribution over the next token conditioned on that state; sampling from that distribution and appending the result advances the state; repeat. A purist will object that the transformer's conditioning on the full window violates strict memorylessness, and the objection is fair: a transformer is not a Markov chain in the textbook sense. The accurate statement is subtler. The generation loop is a Markov process whose state is the entire accumulated context, and the model's job is to compute the transition probabilities of that process - a table far too large to ever write down, learned instead from sampling an appreciable fraction of the human-written web. The Monte Carlo instinct - you cannot enumerate it, so learn it from samples - is doing the same work Ulam did in his hospital bed.

The lineage, then, is not a metaphor but a lineage. Markov supplied the structure: states, transitions, conditional probability. The law of large numbers supplied the license: samples converge on truth. Ulam and Metropolis supplied the method: when exact computation fails, sample. Shannon supplied the objective: predict the next symbol, and measure yourself by entropy. Brin and Page supplied the proof of industrial scale: a stochastic model of the whole web is worth building. The transformer assembled these into a single artifact. Nothing in that artifact's mathematical ancestry postdates 1953.

07 What Changed, and What Refuses To

The honest reading of this history cuts against both the hype and the dismissal. The dismissal says large language models are stochastic parrots, mere table lookups, and therefore nothing new. The history shows that description is simultaneously true and irrelevant: Markov's Pushkin analysis was a table lookup, PageRank was a table lookup, and both changed the world. The table's size and the fidelity of its conditioning are what matter, and on those axes the modern model is separated from Shannon's n-grams by something like nine orders of magnitude in parameters and an unbounded conditioning window. The hype, conversely, says transformers represent a discontinuity, a new kind of machine. The mathematics says otherwise: the objective is Shannon's, the structure is Markov's, the scale is PageRank's, and the randomness is Ulam's.

The shared ancestry also explains the shared failure modes, because the limits of these methods were discovered alongside the methods themselves. A Markov chain has no knowledge outside its transition table; sampled text can be locally plausible and globally wrong, a property Shannon's own n-gram demonstrations exhibited in miniature and which the industry now calls hallucination. Distribution shift - the world changing underneath a table estimated from the past - was a known affliction of every statistical model from Markov's era onward. And Monte Carlo methods converge in probability, not in certainty: Ulam's solitaire estimate was a fraction, not a proof, and a language model's output is a sample, not a guarantee. None of these pathologies arrived with neural networks. They are the family diseases of prediction-by-statistics, and they have been documented for a century.

That is the useful frame for anyone building on these systems in 2026. The century-old mathematics tells you what a next-token predictor can and cannot become, and the boundary is not moved by scale alone. What scale did buy is the transition from counting letters to conditioning on entire documents, which turned a parlor trick into a general-purpose instrument. But the instrument still answers the question Markov asked in 1906: given where we are, what is likely to come next? The strange math that predicts almost anything, as the source video puts it, is strange mostly because it is so old - and because it keeps working each time someone finds a bigger table to fill.