Inside the Black Box: What Actually Happens When You Send a Prompt to an LLM

01 From keystroke to context: tokenization and why models count in tokens

Before a single weight fires, your text is chopped up. A tokenizer runs a learned vocabulary — commonly 32,000 to 256,000 pieces called tokens — over the prompt, splitting words into frequent chunks, subwords, or single characters. A common word may survive whole; rarer strings shatter into several pieces. Numbers, code, and non-English text are notorious for consuming more tokens than their apparent length suggests, because training vocabularies skew toward frequent English.

Three consequences follow. First, context limits are denominated in tokens, not words: a 128,000-token window holds far less prose than a naive character count implies. Second, pricing is denominated in tokens, so an invoice reflects the tokenizer's view of your text, not yours. Third, spelling is fragile: a model asked to count the letters in a word sees a token ID, not a letter sequence, which is why such tricks trip otherwise capable systems. Everything downstream — embeddings, attention, sampling — operates on this token stream, so the tokenizer quietly defines the granularity of everything the model can reason about.

02 Embeddings and the forward pass: how a prompt becomes geometry

Each token ID indexes a lookup table and comes back as a vector — a point in a space with thousands of dimensions. Directions in that space encode meaning: the same geometry that places queen near king minus man plus woman is what lets a model generalize from a handful of examples. Positional information is added so the network can tell where each piece sat in the sequence.

The forward pass then carries these vectors through dozens of stacked blocks. A 7B-class model typically has around 32 such blocks, each containing an attention operation and a feed-forward network that holds most of the parameters. After the final block, the vector at the last position is projected onto the vocabulary to produce one score per possible token: the logits. Nothing in this pass is mysterious in isolation. The apparent magic is what billions of fitted parameters do to geometry after training on trillions of words — compression so aggressive that statistical regularities start to look like reasoning.

03 Attention layers: the context window as working memory

Attention is the mechanism that lets each token consult every other token in the context. For each position the model computes queries, keys, and values; dot products between a query and every key produce weights, and the weighted mixture of values becomes the new representation. Early layers tend to pick up local patterns such as syntax and adjacent-word relations, while later layers assemble longer-range structure: topic, coreference, discourse.

This is why the context window behaves like working memory. Everything the model can use about your instructions must fit inside the window, and relevance is recomputed at every layer rather than stored once. Long prompts therefore do not accumulate understanding linearly: material buried in the middle competes for attention with everything else, and recall of mid-context facts is often weaker than at the edges. When a model seems to forget a constraint stated twenty paragraphs earlier, the honest description is not amnesia but attention that distributed its weight elsewhere. The window is not a recording; it is a rehearsal that must be continuously re-earned.

04 Sampling: why the same prompt never gives the same answer twice

The forward pass ends with logits — a score for every token in the vocabulary. A deterministic argmax would always pick the top one, and identical prompts would get identical answers. Instead, the sampler converts scores into a probability distribution and draws from it. Temperature rescales the logits before the softmax: low values sharpen differences toward predictability, high values inflate the tails toward surprise. Truncation rules such as top-p and top-k cut the tail so genuinely implausible tokens stay excluded even at high temperature.

That draw is why the same prompt never gives the same answer twice, and it is a feature rather than noise. Sampling is where diverse phrasings, brainstorm alternatives, and code variants come from. It is also where instability comes from: a long chain of individually reasonable draws can wander into confident nonsense, and two samplings of one question can disagree. Temperature near zero recovers near-determinism for extraction tasks; creative work usually wants the variance left in.

05 Prefill versus decode: the two very different phases of inference

Inference runs in two phases with very different economics. Prefill processes your entire prompt at once: all tokens are known, so the work is a large batch of matrix multiplications that saturates accelerator compute. It happens once per request and produces the first token — so time-to-first-token scales roughly with prompt length. A ten-word question feels instant; a twenty-page document stuffed into context makes the user wait before a single word appears.

Decode then generates one token at a time. Each new token must attend over the whole prefix, so every step reads the full model state from memory — yet each step is a small matrix operation that cannot use compute efficiently. Output speed is bounded not by floating-point throughput but by memory bandwidth, which is why tokens per second stays roughly flat no matter how long the answer grows. Serving stacks such as those described in the ORCA and vLLM papers exist largely to keep both phases busy at once, batching prefills from one request against decodes from others.

06 Why speed varies: memory bandwidth, KV cache, and batch effects

Three hardware-level factors explain most of the variance in response speed. First, memory bandwidth: decoding reads every weight and the accumulated cache on each step, so token production tracks how fast bytes move, not how fast the chips multiply. Second, the KV cache: to avoid recomputing attention over the whole prefix, systems store keys and values for every layer and every past token. A 7B-class model with 32 layers reserves roughly half a mebibyte per token in fp16 precision, so a 32,000-token context parks about 16 GiB on the accelerator before a word is emitted — charted below. Third, batching: servers juggle many requests per accelerator, and your latency depends on how much company you have. Heavy load stretches queue time and time-to-first-token; a long shared context inflates memory pressure and can slow everyone. This is also why long conversations feel progressively slower: the cache grows with every turn, and each new token pays a slightly larger tax.

07 What the pipeline explains: prompt length, cost curves, and hallucination pressure

The pipeline is not trivia; it predicts behavior. Prompt length maps to prefill cost, so context dumps raise latency and price in near-linear proportion — charted above — and trimming redundancy is the cheapest optimization available. Because billing counts tokens, formatting and examples are not free: a verbose template pays on every call. Because context is working memory rather than storage, instructions stated once and buried get out-attended by recent text, which is why restating critical constraints at the end of a long prompt works so reliably.

Hallucination pressure also has a mechanical reading. Sampling guarantees fluent output even when the next-token distribution is spread thin over genuinely unknown territory; the model will still emit a confident-looking draw. Nothing in the detokenizer checks facts — its only job is turning token IDs back into text. Treat fluency as a property of the pipeline, not evidence of verification, and spend effort where leverage sits: short, specific prompts; constraints restated near the end; temperature matched to the task; and skepticism calibrated to how far an answer sits from anything the context could have grounded.