Why the Same Prompt Never Answers Twice: Sampling Is the Quiet Product Decision

01 The Same Prompt, A Different Answer

Every team that ships a language model feature meets the same surprise eventually: a prompt that produced a polished answer on Tuesday produces a different one on Wednesday, and a third variant on Friday. Nothing in the prompt changed. The model did not get updated. The difference came from a layer most product reviews never discuss, the step where the system chooses which words to actually emit.

A large language model, at the end of every processing step, does something very specific: it assigns a probability to each token in its vocabulary as the plausible next piece of text. Those probabilities are the mechanism, and they are the same every time for the same input under the same weights. What changes between runs is the selection rule applied afterward. Engineers call that rule sampling, and its parameters, temperature, top-p, and the random seed, are configuration switches sitting in an API call. This article takes the deployment view of that step: not how the probabilities are computed, but what happens when a product decides how to draw from them, why the same prompt never answers twice, and why auditing, evaluation, and agent reliability all inherit the consequences.

02 What Temperature Does To A Distribution

The model ends each step with logits, a raw score for every candidate token. A softmax function converts those scores into probabilities that sum to one. Temperature acts before that conversion: it divides the logits, which rescales how competitive the runners-up are allowed to look. At temperature values below 1, gaps between candidates widen and the distribution sharpens toward its favorite. As temperature approaches zero, sampling collapses into greedy decoding, always taking the single highest-probability token. Above 1, gaps compress, the distribution flattens, and unlikely tokens get a real chance of being drawn.

Two companion settings trim the tail before sampling. Top-p keeps only the smallest set of tokens whose cumulative probability reaches p, say 0.9, and renormalizes. Top-k keeps only the k highest candidates. Temperature then reshapes whatever survives. The chart below shows how one hypothetical top-5 distribution changes shape across three temperature settings; the favorite absorbs most of the mass at 0.2 and gives most of it back at 1.2.

03 The Decode Loop, One Token At A Time

Sampling does not happen once per answer. It happens once per token, inside a loop. The prompt goes in, the model produces a distribution, temperature and top-p reshape it, one token is drawn, and that token is appended to the sequence. The whole sequence, original prompt plus everything generated so far, then feeds back through the model for the next step. The loop exits when a stop token is drawn or a length limit is hit. What reads as a fluent paragraph is a chain of these small decisions, each one conditioning all the ones after it.

This loop structure explains why variation compounds. A slightly adventurous draw at step three changes the context for step four, which changes the distribution at step five, and soon the answer has wandered somewhere visibly different. Two runs of the same prompt are not two rolls of one die; they are two long chains of coupled draws. The schematic below traces one pass of the loop, and the repetition arrow is where product-visible nondeterminism actually lives.

04 Seeds, Reproducibility, And The Audit Trail

Run-to-run differences come from a random draw, so the obvious fix sounds simple: fix the seed. A seed pins the pseudo-random number stream, and with the same seed, the same weights, and the same inputs, the sampled tokens should match. Several APIs expose a seed parameter for exactly this reason, and it works well enough for test harnesses. What it does not guarantee is bit-for-bit reproducibility in production. Inference providers batch many requests onto the same accelerators, reorder work, and run matrix arithmetic whose summation order can vary; those effects can shift a probability by a hair, and a hair is enough to flip a near-tie draw. Providers are generally explicit that seeded sampling is best-effort, not a contract.

The enterprise consequence is procedural. An audit trail for a model-assisted decision should record the prompt, the model version, the sampling configuration, and the seed, and should still treat the recorded answer as reproducible-in-class rather than reproducible-to-the-token. Evaluation pipelines chasing perfect determinism fight the infrastructure; pipelines that fix the seed, run each case several times, and score the aggregate get stability they can actually defend in an incident review.

05 Defaults Are Product Design

Most teams never set temperature, which means most products ship on the provider default, typically around 1.0. That default is a decision somebody else made about how adventurous your assistant should sound, and users experience it as personality. Lower the temperature and a support bot becomes terser, more repetitive, and visibly safer. Raise it and the same weights produce livelier copy with more wrong turns. When a model gets rebranded or a provider quietly retunes defaults, users often report that the model got smarter or dumber when, in decoding terms, it merely got more or less daring.

This is where public discourse routinely confuses capability with sampling regime. A model that flubs a question at temperature 1.2 may answer the identical question correctly under greedy decoding, and a model praised for creative range is often just running hot. Neither observation says anything about capability; both describe the draw. The practical takeaway for product teams is to treat sampling configuration the way they treat typography or tone of voice: named, versioned, reviewed, and changed deliberately, because it is part of what the user experiences as the product.

06 Agents Need Quieter Decoding

Agentic workloads lean harder on sampling than chat does, because every step has to be machine-readable. An agent that must emit a JSON tool call does not benefit from an adventurous draw; a stray token breaks the parse, triggers a retry, and burns latency and budget. Teams running tool loops converge on low temperature, tight top-p, or constrained decoding modes that restrict output to a valid grammar, precisely because the goal is to make one correct answer more likely rather than to diversify candidates. Structured-output features in major APIs are, in effect, sampling policy expressed as a schema.

The contrast is instructive rather than absolute. Ideation, ad copy, and search diversification genuinely benefit from hotter settings, and retrieval-augmented pipelines sometimes sample several candidates in parallel to widen coverage. The failure mode is applying one regime everywhere: enterprises that force a single global temperature tend to discover that their summarizer is dull, their agent is flaky, or both. Sampling policy should follow the task, and evaluation harnesses should fix it, so that scores measure the model rather than the dice.

07 Limits And Outlook

The honest limits first. Sampling parameters govern selection, not ability; no temperature setting teaches a model a fact it does not know, and the strongest configuration cannot rescue a weak base model. Reproducibility claims are bounded by shared infrastructure, so a seed is an aid to testing rather than a legal guarantee. And the knobs themselves are drifting out of user reach: several reasoning-oriented models expose few or no sampling controls because the provider tunes decoding internally to make long chains of thought reliable, and speculative decoding techniques change the mechanics under the hood while preserving the statistical intent.

The outlook is that sampling config becomes boring in the best sense, an explicit, versioned line in every feature spec, logged next to prompts and model versions, reviewed when behavior reviews happen. Teams that already write it down debug faster, reproduce regressions more often than not, and can explain to auditors why the system answered as it did. The same prompt will still never answer exactly the same way twice. The difference between a toy and a product is knowing why, and having decided on purpose how much that should matter.