“Build an eval set” is advice everyone gives and almost nobody follows through on, because the next question — from what? — has no obvious answer. So the team books an hour, writes forty test cases between them, and ships an eval suite that will quietly mislead them for the next year.
We’ve argued that offline evals aren’t enough, and that’s still true. But “not sufficient” isn’t “not useful”: an offline set is the cheap filter that runs before the expensive experiment. Whether yours is worth running comes down almost entirely to where the examples came from.
The two bad origins
Invented examples. Written by the team, in a room, from memory. They encode what you imagine users ask, and they are systematically cleaner than reality: better spelled, better punctuated, better scoped, phrased in your product’s own vocabulary. Real users type three words and a typo, paste an entire email thread, ask two questions at once, and use the wrong name for your feature. An eval set of invented examples measures how well your system handles a user who has read your documentation.
Model-generated examples. Worse, and in a specific way worth naming. Ask a model to generate a hundred test questions and you get a hundred questions that model finds plausible. Your eval set and your system now share a prior about what a question looks like — so the queries your model handles badly are, disproportionately, the queries it didn’t think to generate. The blind spot is in both places at once, and the eval cannot see it by construction.
Neither is worthless as a supplement. Neither should be your foundation.
The composition that works
Production traffic is the foundation, deliberately re-weighted. Roughly:
| Stratum | Share | Why it’s there |
|---|---|---|
| Representative traffic | ~50% | Sampled uniformly from real requests, so the set reflects the actual distribution and a change that helps the common case is visible |
| Known failures | ~20% | Anything that drew a complaint, a thumbs-down, an escalation, or a rollback. The highest information per row in the set |
| Edge cases | ~20% | Empty input, enormous input, wrong language, adversarial phrasing, ambiguous requests, and questions your product genuinely cannot answer |
| Rare but critical | ~10% | The high-stakes query types that are a rounding error in traffic and the whole of your risk profile |
| Frozen regression set | additive | The specific cases behind past bugs. Never rebalanced, never refreshed, only appended to |
Two notes on that table.
It is deliberately not a representative sample, and it shouldn’t be. You’re optimising for information per example, not for an unbiased estimate of production performance. If you want the unbiased estimate, run an experiment on live traffic — that’s what live traffic is for. The offline set exists to catch breakage quickly and cheaply, and known failures and edge cases catch far more breakage per row than average traffic does.
Include questions your product can’t answer. A correct refusal is a passing case. An eval set with no unanswerable questions in it silently rewards a system that always produces an answer, which is precisely the behaviour you were trying to avoid. This is the single most commonly missing stratum.
The frozen regression set is the part that pays for itself. Every bug you fix contributes its triggering input, permanently. It never gets rebalanced away, and it’s the reason a fix from March doesn’t quietly come undone in September.
Somebody has to say what good means
There’s no way around the labelling, and pretending otherwise is how eval sets end up unused.
Label a small set by hand — a hundred examples is a real afternoon and enough to start. And do it with the rubric open, editing it as you go, because the rubric is the more valuable output of the exercise. The labels tell you how the system did on a hundred inputs. The rubric tells you what your team actually means by “good”, which turns out to be less agreed-upon than anyone expects, and every disagreement you resolve while labelling is a disagreement that would otherwise have surfaced as an argument about whether to ship.
Then use those labels to calibrate a judge: run the judge over the same hundred, look at where it disagrees with you, fix the rubric, repeat. That’s the only path from a hundred labelled examples to a thousand scored ones, and the reliability question it raises is answered in can you trust an LLM to grade an LLM?.
Two things that will go wrong
Contamination. Someone needs few-shot examples for a prompt, the eval set is right there and nicely curated, and now your system has seen the test. This is not hypothetical — it’s the path of least resistance, and it happens without anyone deciding to do it. Keep the eval set in a separate place from the prompt assets, and treat “where did this example come from” as a question with an auditable answer.
Overfitting. Six months of tuning against 200 examples produces a system that is excellent at those 200. Every change is evaluated against the same fixed set, so changes that help those cases get shipped and changes that help the rest of the distribution don’t. The set stops being a measure and becomes a target — the failure mode that offline evals aren’t enough is fundamentally about.
The defence isn’t a better eval set. It’s remembering what an eval set is for: a filter that catches obvious breakage before you spend traffic on it. The decision about whether a change is better still belongs to an experiment on live traffic, for the reasons in why prompt changes deserve A/B tests, not vibes.
Maintenance, and the obligation attached
An eval set is a living artefact with an owner. Traffic drifts — users start asking different things, and a set assembled last spring gradually stops describing your product. Refresh the representative stratum quarterly, append to the frozen set continuously, and re-examine the edge cases whenever you ship a new capability.
And be clear-eyed about what you’re holding: production examples are real user data. Names, addresses, account details, and whatever else people paste into a text box are now sitting in a curated file that gets copied to laptops and pasted into notebooks — which is a materially different risk profile from a log with a 30-day retention. Redact before curating, not after, and read logging LLM traffic without creating a privacy incident before you build the pipeline rather than after.
The sequence is: eval set catches what’s broken, experiment decides what’s better. Skipping the first wastes traffic on changes that were never going to work. Skipping the second means you never actually find out.