Same prompt, different answer: testing a model that won’t sit still

You run the same prompt twice and get two different answers. Reasonable response: set temperature to 0. You run it twice more and still get two different answers — not wildly different, but not identical either, and now the tidy mental model is gone.

This is the fact that makes people give up on measuring LLM systems, and it shouldn’t be. It’s not a mystery and it’s not a defect. It’s variance, and variance is the thing statistics was invented for.

Temperature 0 is not determinism#

Temperature controls the sampling step: at 0, decoding is greedy, and the model takes the highest-probability token at each position rather than sampling from the distribution. That removes the deliberate randomness. It does not make the system deterministic, because several other things vary underneath it:

• Floating-point addition isn’t associative. GPU kernels sum partial results in whatever order the batch and the hardware schedule produces, and (a + b) + c does not always equal a + (b + c) in floating point. Tiny differences in the logits, occasionally enough to change which token is highest.
• Batching changes the computation. Your request is processed alongside whatever else arrived at the same moment, and batch composition affects the reduction order above.
• Mixture-of-experts routing can depend on the batch. Where routing decisions are made across a batch rather than per-sequence, the other requests in flight influence which experts see your tokens.
• Hardware and kernel versions differ. The same model served from two GPU generations, or across a library upgrade, rounds differently.
• The model behind the name changes. Providers update served weights under stable aliases. Your “same model” in March and in July may not be.
• Ties have to be broken. When two tokens have equal probability, something picks, and that something is an implementation detail.

The honest summary: greedy decoding removes the randomness you chose and leaves the incidental kind, which is usually small and is entirely outside your control. If your test plan depends on bit-identical outputs, it depends on a guarantee nobody is offering you.

The reframe: n = 1 is not a result#

Here’s the part that changes what you do on Monday. The problem isn’t that the model is unpredictable. The problem is that one output compared against one output is a sample of size one, and nobody would accept that anywhere else in engineering.

You would not benchmark a database by running one query against each version and shipping whichever was faster. You would not accept a latency regression report built on a single request. But it is completely normal to change a prompt, run it once, read the two answers side by side, and decide.

That decision has the same evidential weight as the single-query benchmark. It feels stronger because you read both outputs and formed a judgement about them — but attention to one sample doesn’t make it two.

Non-determinism doesn’t make LLM systems unmeasurable. It makes them measurable only in aggregate, which is the same condition as every other noisy system in production.

What variance costs you, in traffic#

Variance is not just a philosophical inconvenience — it’s a term in the sample-size formula. Required sample scales with the variance of your metric, so a noisy metric needs more observations to resolve the same difference.

This has a practical ordering consequence that’s worth internalising: the low-variance metrics resolve first. Format compliance is binary and nearly deterministic, so a change in it becomes visible quickly. Cost per request varies with output length but within a fairly tight band. A judge score on an open-ended answer, sampled across genuinely different questions, is the noisiest thing you measure — and therefore the slowest.

That’s why the advice in how much traffic do you need to A/B test a prompt? is to measure the cheap metrics first. It isn’t a shortcut for small teams. It’s a direct consequence of where the variance lives.

Measure the variance before you fight it#

Before spending any effort reducing non-determinism, find out how much you have. The experiment takes twenty minutes:

Take twenty representative inputs. Run each one five times through your current configuration. Score all hundred outputs on the metric you care about. Now you have two numbers — the spread across repeats of the same input, and the spread across different inputs.

The second is almost always much larger than the first, and that ratio is the most useful thing you’ll learn. It tells you that the variation you’ve been worrying about — the same prompt giving different answers — is small compared to the variation you can’t avoid, which is that your users ask different questions. Teams that run this check usually stop trying to eliminate non-determinism and start collecting more data instead, which is the correct move.

It also gives you a floor. If repeated runs of one input differ by more than the effect you’re hoping to detect, no amount of careful reading of individual outputs will ever show you that effect. Only aggregation will.

What to actually do about it#

Ranked by how much they help:

Aggregate instead of inspecting. Twenty outputs summarised into a number beats two outputs read closely. This is the whole discipline in one line.

Pin what you can pin. Model version, prompt version, temperature, top-p, seed where the provider offers one, judge configuration. You cannot make the model deterministic, but you can make sure the variation you observe is the model’s and not your own configuration quietly drifting underneath the comparison. This matters most across time: a result from three weeks ago is uninterpretable if you can’t say what produced it.

Lower temperature where diversity isn’t the point. Extraction, classification, structured output — none of these benefit from sampling. Reducing temperature genuinely narrows the spread, which buys you statistical power. It won’t reach zero, and on open-ended generation it can cost quality, so don’t apply it reflexively.

Never compare across time without a control. If arm A ran in June and arm B ran in July, you cannot separate your change from a provider-side model update. This is the failure mode described in the silent regression, and it’s why a concurrent split beats a before-and-after every time — both arms experience the same model, the same traffic, and the same week.

Record what produced each result. A quality number without the configuration that generated it is an anecdote with a decimal point. LLMJury stamps the model, prompt version, and judge configuration onto every event, so a verdict says what it was measuring — the reasoning behind getting your prompts out of your codebase applies to every other knob too.

Where non-determinism is the feature#

One counterweight, because the goal is not to stamp it out. Retries only work because the second attempt differs from the first. Self-consistency methods sample an answer several times and take the most common one, converting variance into accuracy. Creative and brainstorming features are worse when every run is identical.

So the aim isn’t a deterministic model. It’s a deterministic measurement of a non-deterministic model — a pipeline where the configuration is pinned, the sample is large enough, the arms run concurrently, and the number at the end means the same thing this month as it did last month.

That’s achievable, and it’s a lower bar than the one people give up at.

Start free, or read why prompt changes deserve A/B tests, not vibes for the argument this post is the mechanism behind.