The 300-Millisecond Brain: Attention, Sampling, and What an LLM Actually Computes

01 THE MISCONCEPTION IN THE WAY

Most people's first mental model of a chatbot is wrong in a specific, load-bearing way. They imagine a database with opinions attached: something that looks up stored facts and animates them with a persona. The model that Andrej Karpathy spends his introductory lecture building, plank by plank, is almost the opposite. It is a function that runs for a few hundred milliseconds, emits one token, and then stops, owning nothing and remembering nothing until it is run again.

Getting that model right matters more than any technical detail, because every downstream question people ask, whether it can reason, whether it lies, whether it knows what it said a minute ago, is really a question about the loop: what goes in, what comes out, and what happens in between.

02 TOKENS IN, A DISTRIBUTION OUT

The mechanism is austere. Text is chopped into subword tokens drawn from a fixed vocabulary of, typically, tens of thousands of entries. Those tokens pass through the network in a single forward pass, a stack of transformer layers mixing information across positions via attention. At the top sits one job description: produce a probability distribution over the entire vocabulary for the next token.

That is all the model outputs. Not a sentence, not an answer, not a thought. One column of probabilities, perhaps 30 percent for "the," 12 percent for "a," fractional percentages for tens of thousands of alternatives. Everything a language model ever writes is assembled from those columns, one draw at a time, with each sampled token appended to the input for the next pass.

03 ATTENTION: THE ONLY PLACE CONTEXT IS REMEMBERED

Inside the forward pass, attention is the mechanism that lets a token know what its neighbors are. Every position broadcasts a query, every earlier position offers a key, and the match scores decide how much of each neighbor's information flows into the current token's representation. Run in parallel across dozens of heads and dozens of layers, this is the entire act of reading: context does not accumulate in some external memory, it is rebuilt from scratch on every single pass.

This is the precise sense in which a language model has no memory. The conversation you had with it an hour ago is not stored anywhere in the network. What you paste into the context window is the only world it has ever seen, and attention is the only faculty it has for looking at it.

04 THE MILLISECOND CLOCK: SAMPLING IN A FEW HUNDRED MILLISECONDS

The "300-millisecond brain" framing refers to inference latency. A modern forward pass over a modest context completes in tens to hundreds of milliseconds on production hardware, a figure consistent with the throughput of current inference stacks and with the pace at which deployed chat systems visibly emit text. The model samples from its distribution, appends the drawn token, and re-runs. At twenty to fifty tokens per second, the loop feels like typing, which is exactly why streaming interfaces show it that way.

Two consequences follow from the clock. First, the system's apparent continuity, the way a paragraph flows, is an illusion of composition: each token was chosen with only the preceding text in view, in a few hundred milliseconds, with no plan for the rest of the sentence. Second, randomness is a dial, not a defect. The temperature setting reshapes the distribution before sampling, from near-deterministic at the coldest setting to adventurous at the highest. The model itself is fixed; the sampling strategy decides what kind of writer it appears to be.

05 EVIDENCE: SIMPLE LOOP, COMPLEX BEHAVIOR

The strongest evidence for this mental model is how much behavior it predicts. Trained on next-token prediction over internet-scale text, the same loop writes code, drafts contracts, and passes professional exams, all without a mechanism for any of those tasks being added by hand. The capabilities arrive as a byproduct of scale, the trend documented across the GPT series from roughly a hundred million parameters to hundreds of billions.

Alongside capability comes a failure mode that the loop explains exactly. Because the model's only world is the context window, it can be made to "remember" anything by supplying it, including false things, a fact with a short step to the jailbreaks and injection attacks Karpathy lists among deployed systems' known liabilities. The lecture's inventory, hacking, misinformation, and roleplay misuse, is not a list of mysteries; each is the loop running on inputs someone else chose.

06 LIMITS: WHAT THE MODEL DOES NOT HAVE

The mental model also enforces honesty about absences. The network has no fact database to consult, only statistical patterns compressed into weights, so its errors, the confident fabrications the field calls hallucinations, are the loop operating exactly as designed on thin evidence. It has no goals of its own, no ongoing experience between invocations, and no verified notion of truth, only distributions over plausible continuations.

None of this means the system is trivial. It means the interesting questions have moved. The engineering frontier today, sampling strategies, speculative decoding, batching and caching, is all about running the same bare loop faster and more reliably; and the alignment frontier is about which tokens the loop should decline to emit. The loop never got more complicated; the scaffolding around it did.

07 LEGACY: WHY THIS LECTURE ENDURES

Years after it was recorded, Karpathy's lecture remains the reference introduction because it resists the two symmetrical temptations: mystifying the model into an artificial mind, and dismissing it as a mere autocomplete. The truth it teaches is that these are the same machine, described at different levels of detail: a function, run every few hundred milliseconds, whose iterated output has come to occupy a central place in daily work.

That is the durable skill it hands its audience. Anyone who internalizes the loop, context in, distribution out, sample, repeat, can reason about new capabilities, new failure modes, and new products without waiting for the field's next announcement. The forward pass will not change; only what surrounds it will.