Offline evals aren't enough: why prompts need testing on live traffic

2026-07-16 · The LLMJury team

If you maintain a golden dataset and run your prompts through an eval harness before shipping, you're ahead of most teams — keep doing it. But an offline eval answers a narrower question than the one you actually care about. It tells you "does the new prompt do well on the examples we wrote down?" The question that decides whether you should ship is "does it do better for the users we serve?" Those come apart more often, and more expensively, than teams expect.

Your golden set is a photograph of last quarter's traffic

Eval sets are built from the failures you've already seen and the use cases you already know. Meanwhile your real traffic drifts every week: new user segments, new phrasings, new topics, the seasonal question spike you forgot existed. The prompt that scores 94% on your frozen set is being graded against a distribution that no longer matches production — and the gap is invisible precisely because the examples that would expose it aren't in the set.

Live A/B tests don't have this problem by construction. The sample is the traffic. Whatever your users are asking this week, that's what both variants get graded on.

Offline scores overfit the way you iterate

There's a subtler failure mode: you tune the prompt until the eval passes. After enough iterations, the prompt is optimized for the eval set — the same way a model overfits its training data. The eval number goes up; whether real quality went up is exactly the question the process stopped being able to answer. An experiment on held-out live traffic is the eval you can't overfit, because you never saw the questions coming.

The metrics that decide the business don't exist offline

An offline harness can score correctness and format. It cannot tell you:

Latency under production load — the longer prompt costs 300ms at p95; your eval never noticed because it doesn't measure time, and definitely not on production infrastructure.
Cost at your real token distribution — sampled test questions are short; your users paste entire documents.
What users do next — do they accept the answer, rephrase in frustration, convert, come back? Task success, retention, and revenue only exist where real users are.

These are routinely the deciding metrics. A variant that's 3% "better" offline and 20% slower in production is a bad trade you'd ship blind. LLMJury reads quality, speed, cost, and downstream business events in one experiment, so the trade-off is a verdict, not a surprise.

Judges drift too — grade both variants at the same moment

A wrinkle that bites careful teams: if you evaluated the old prompt in March and the new one in July, you likely changed judge model, rubric, or temperature in between. Your "improvement" may be a grading change. An A/B test grades both variants with the same judge, on the same days, over the same traffic mix — the comparison is fair because everything except the prompt is held constant. (LLMJury also versions every judge and rubric, so a result always records exactly what graded it.)

Keep the harness — change what it's for

The right mental model is the one you already use for code:

Offline evals are unit tests. Fast, cheap, run on every edit. They catch outright breakage — format violations, obvious regressions, safety failures — before anything ships. A prompt that fails offline never needs an experiment.
Live A/B tests are production verification. Slower and heavier, but they measure the thing itself: real questions, real infrastructure, real users, real money.

Nobody argues unit tests make production monitoring unnecessary. Shipping a prompt on offline evals alone is the same category error — you've verified the parts you thought to check, on the inputs you thought to write down.

So: gate on the harness, then let the experiment decide. Route 10% of traffic to the new prompt, let the judge grade a sample, and read the verdict with latency, cost, and conversions beside it. If it wins, promote it with a number attached. If it doesn't, you just avoided shipping a regression your eval set was never going to catch.

Start free — every plan includes unlimited experiments — or read the companion piece on why prompt changes deserve A/B tests at all.