The rubric says “rate helpfulness from 1 to 5”. The scores come back, ninety per cent of them are a 4, and the metric can’t distinguish last week’s prompt from this week’s.
That isn’t the judge failing. An unanchored scale doesn’t measure quality — it measures the model’s prior about what a 4 looks like, which is stable enough to feel real and unstable enough to move when you change the judge, the temperature, or a word in the instructions.
Anchor every level in something observable
The single change that fixes most rubrics: every level of the scale needs an anchor describing what the output did, not how good it was.
“Very good” is not an anchor — it hands the definition back to the model. “Answers the question and states the one caveat that applies” is an anchor, because two people reading the same output would agree on whether it happened.
Here’s the shape, for answer helpfulness in a support assistant:
| Score | Anchor |
|---|---|
| 5 | Answers the question directly, with the specific steps or values the user needs to act, and flags any condition that would change the answer |
| 4 | Answers the question correctly, but the user must infer a step or look something up to act on it |
| 3 | Addresses the topic without resolving the question — general guidance where a specific answer existed |
| 2 | Responds to a related but different question, or gives an answer that is partly incorrect |
| 1 | Refuses, fails to engage with the question, or gives an answer that is wrong in a way that would cause harm if followed |
Notice what the anchors are made of: whether the user can act, whether the answer is specific, whether it’s the right question. All observable. None of them says “good”.
Write these for your product, not from a template. The half hour spent arguing about what a 3 means is the most valuable half hour in the whole exercise, because that argument is the specification — and if your team can’t agree, the judge was never going to.
Six rules that make the rest work
One criterion per metric. A rubric that blends accuracy, tone, and completeness produces a number whose movement you can’t attribute. Score them separately. You can always average later, and you can never un-average.
Three to five levels, not ten. Judges can’t reliably discriminate ten levels, and the extra granularity arrives as noise. Noise costs you sample size — see how much traffic you need for what a wider metric does to the run length.
Say what to ignore. Length, formatting, and confident tone leak into scores unless the rubric explicitly rules them out of scope. “Do not reward length. A shorter answer that resolves the question scores higher than a longer one that also resolves it” is the practical mitigation for verbosity bias, and it takes one line.
Give the unanswerable case an exit. Some inputs have no good answer — the question is incoherent, the required information doesn’t exist, the user is asking something your product doesn’t do. Without a defined “not applicable” outcome the judge invents a score and it silently joins your average.
Score independently unless you handle order. Pointwise scoring against a rubric is less sensitive than pairwise comparison but carries no position bias. If you use pairwise because you need the sensitivity, randomise which output comes first and say so in the write-up.
Show the input, the output, and nothing else. No variant name, no model name, no metadata that identifies the arm. A judge that can see which arm produced the output isn’t a judge.
Calibrate once, properly
A rubric is a hypothesis about what you value. Test it the same way you’d test anything else.
Take 50–100 outputs, spanning good and bad. Have a human score them against the rubric. Have the judge score the same set. Then compare — and the useful part isn’t the agreement number, it’s the disagreements.
Read every one. They fall into two kinds, and they mean different things:
- Systematic disagreement — the judge is consistently a level above you on a particular kind of output. The rubric is wrong, or an anchor is ambiguous. Fix the wording and re-run on the same set.
- Scattered disagreement — no pattern, just noise. The criterion is subjective and probably needs splitting into two things you can each define.
Re-run against the same labelled set after each edit, so you can see whether the change helped. That set is now an asset: it’s the regression test for your measurement instrument, and it’s the only way to know that next quarter’s rubric edit didn’t quietly change what you’re measuring.
Do not skip this because the scores “look reasonable”. Reasonable-looking scores are exactly what an uncalibrated judge produces — that’s the failure mode, not the absence of one.
The rubric, the judge, and the temperature are one version
Change any of the three and you’ve changed the measurement. Not degraded it — changed it, in the sense that the numbers before and after are different quantities that happen to share a name.
Which means they version together, as one unit, and every stored score records which version produced it. The consequence people resist: scores from before and after a rubric edit must not share a chart. A quality line that steps up on the day you reworded an anchor is not a quality improvement, and someone will present it as one.
This is rubric drift, and it’s the fourth of the four biases in can you trust an LLM to grade an LLM. It’s the one that’s entirely your own doing, which also makes it the one you can fully eliminate.
If you must change a rubric mid-programme — and you will — re-score a held-out sample under both versions so you know the size of the step, and mark the discontinuity on the chart.
What the number still isn’t
A calibrated, anchored, versioned judge score is a good comparison instrument. It is not a quality figure.
“Our assistant scores 4.2 on helpfulness” means nothing outside your own rubric and your own judge, and it should never leave the team as a fact about the product. What travels is the difference: variant B scored 0.31 higher than variant A, with a confidence interval, on the same rubric and the same judge version, on the same traffic. That comparison is defensible. The absolute number is an artefact of how you wrote a table.
LLMJury treats a rubric metric as a versioned object — judge model, rubric text, and parameters recorded together, so a result carries what graded it — and runs it under a hard budget cap, since an open-ended judging bill is the other way this goes wrong. Judging starts on the Pro plan; the free plan measures latency, cost, and business metrics.
Write the anchors first. Argue about the 3. Label a hundred outputs once. Everything else in your evaluation stack is downstream of whether that table says something observable.
Start free, or see what to actually measure in an LLM product for where a judge score sits among the metrics that don’t need one.