Someone opens a benchmark table, sorts by the composite score, picks the top row, and ships it. Three weeks later the feature is slower than the interface can absorb and refuses a quarter of the support tickets it was built for.
The benchmark wasn’t lying. It was answering a different question — which model is best in general — and nobody ships in general.
Why a leaderboard can’t answer your question
A public benchmark measures the model. Your product is the model plus your prompt plus your traffic plus your latency budget plus your error tolerance, and only the first of those is public. Every one of the other four can reverse a ranking.
That isn’t an argument against benchmarks. They’re a coarse filter, and a useful one: they tell you which three or four models are plausible, which saves you the cost of properly evaluating twelve. Treat a leaderboard as a shortlist generator and it does its job well. Treat it as a decision and you’ve outsourced your product’s most consequential dependency to a test suite that has never seen your users.
There’s a second problem with published numbers, and it’s structural rather than anyone’s fault: popular benchmarks leak into training data over time, and a score on a contaminated benchmark measures memorisation. You can’t audit that from outside. Your own traffic, by definition, isn’t in anyone’s training set yet.
The criteria that actually decide it
Nine things determine whether a model works in your product. Most shortlists consider two.
| Criterion | Why it decides the outcome | How to measure it on your traffic |
|---|---|---|
| Task quality | The thing you think you’re choosing on, and the only one needing a judge or human labels | Sampled real inputs, scored against a rubric — never invented examples |
| Instruction-following at length | Degrades differently by model and is invisible on the short prompts benchmarks use | Your actual system prompt, not a trimmed version; count violated constraints |
| Structured output compliance | A well-formed object with an invented enum member is a production incident, not a quality issue | Parse rate against your real schema, over a few hundred outputs |
| Latency shape | Time to first token and total completion diverge, and your interface picks which one is felt | Measure both at p95 on your own prompt lengths |
| Cost per request | A verbose model at a cheap per-token rate loses to a terse model at a dear one | Mean tokens in + out on real traffic × current rates — recompute, don’t assume |
| Refusal and over-caution | Domain-specific and generic benchmarks never show it | Refusal rate on legitimate inputs from your own product |
| Context window vs what you send | Most teams shop for a limit ten times what they use | p99 of your actual prompt token count |
| Tool-calling reliability | For agents, the difference between a demo and a product | Valid-call rate and correct-tool rate per task type |
| Operational fit | Rate limits, regions, retention terms, and concentration risk kill deals after the tech is fine | Read the terms before the evaluation, not after |
The row people skip is cost per request. Per-token pricing is what providers publish and what comparison posts tabulate, and it’s the wrong unit. A model that answers in 200 tokens at twice the rate of one that answers in 700 is cheaper per answer and faster to read. You cannot get that from a price page — you get it by running your own inputs and counting.
The row people regret skipping is refusal rate. If your product touches security, medicine, finance, legal text, or anything a safety filter finds interesting, models differ enormously here and no general benchmark reports it. It surfaces on day one of production and it is expensive to discover then.
The selection sequence
Four steps, and the order matters because each one is cheaper than the next.
- Shortlist three from public evidence. Benchmarks, provider docs, and what people you trust report. Ten minutes, not a week — this step is only allowed to eliminate the obviously unsuitable.
- Build a set of fifty real inputs. Sampled from your traffic if you have it, from your beta or your support queue if you don’t. Fifty is enough to catch gross failures. Building an eval set out of production traffic covers how to stratify it so those fifty aren’t all the easy case.
- Run all three offline. Check format compliance, refusal, latency, cost per request, and outright breakage. This round is for eliminating — it is not sensitive enough to rank two models that both work.
- Take the survivors to live traffic. One or two arms against your incumbent, with quality as primary and cost and latency alongside it. This is the only step that can rank, because it’s the only one measuring the model on the distribution you actually serve.
Most teams do step one and step four’s decision without steps two, three, or the experiment. The offline round is worth its afternoon: it’s where you find that the model you were about to ship emits markdown fences around JSON that your parser chokes on.
Your prompt is not a neutral instrument
Here’s the trap that makes a naive bake-off dishonest.
Your current prompt was written against your current model. It contains months of accumulated workarounds for that model’s specific failure modes — the sentence that stops it rambling, the example that fixes its formatting, the emphatic instruction it needed and another model doesn’t. Run that prompt against a challenger and you are measuring the challenger’s performance on someone else’s crib sheet.
That bias is systematic and it favours the incumbent. So do one of two things, and say which in the write-up:
- Normalise. Give each candidate a fair re-tune — the same effort, ideally by someone without a favourite — and compare tuned against tuned.
- Accept it, and label it. A like-for-like swap is a legitimate test of "can we switch cheaply today", as long as nobody later quotes the result as "model X is worse than model Y".
Upgrading to a new model without breaking your product covers the re-tuning step in depth, including why "the new model is worse" so often turns out to be "our prompt is old".
The mirror-image error is changing two things in one arm. A challenger model with a freshly written prompt against your incumbent with its old one gives you a number you can’t attribute to either. If you want both answers, run both arms.
Nobody chooses once
The uncomfortable part: this is not a decision you make and defend. Providers update models behind stable names, prices move, new entrants land every few months, and your traffic drifts underneath all of it. The model you picked correctly in March can be the wrong one in September without anyone changing a line of your code.
Which means the useful output of a selection process isn’t a choice — it’s a comparison you leave running. A second arm on a small share of traffic, with the same metrics, permanently. It costs a slice of traffic and it turns "should we re-evaluate?" from a quarterly project into a dashboard you already have.
In LLMJury a model swap is an ordinary variant: assignment is sticky, so a user doesn’t flip mid-conversation, and cost and latency per arm arrive alongside quality, which is what makes the trade-off readable on one screen rather than three.
The honest summary is short. Use the leaderboard to pick who gets evaluated. Use your own fifty inputs to eliminate. Use live traffic to decide. And don’t let the prompt you wrote for the incumbent be the instrument you judge the challenger with.
Start free, or try the interactive demo to see what a two-model comparison reads like when the numbers land.