Can you trust an LLM to grade an LLM?

The objection deserves to be said out loud before anyone else says it: if you can’t trust the model to answer the question, why would you trust it to grade the answer? Anyone who has watched a model invent a citation with total confidence has earned that skepticism.

It’s a good objection. It’s also weaker than it looks — and the place where it’s genuinely right is not the place most people point to.

Grading is a smaller job than answering

Writing a good answer means retrieving the right facts, choosing a structure, and producing fluent prose that satisfies a request nobody stated precisely. Grading means reading a question, an answer, and a rubric, and saying which parts of the rubric the answer meets. The second task has all the information sitting in front of it. The first has to conjure it.

That gap is why judge models agree with human raters far more often than the general “can you trust an LLM” framing suggests. It is not perfect agreement, and this post is not going to claim it is. But the interesting question isn’t “does the judge agree with a human every time?” — it’s “is the judge good enough to compare two variants?” Those are very different bars, and almost everything written on the subject conflates them.

The four biases that actually break judges

Judges fail in specific, documented, fixable ways. Vague distrust is less useful than knowing which four things to check.

• Position bias. Show a judge two responses and ask which is better, and the one presented first wins more often than it should — the ordering itself carries signal it shouldn’t. Fix it by randomising the order across comparisons, or by scoring each output independently against a rubric instead of running pairwise comparisons at all.
• Verbosity bias. Longer answers score higher, whether or not the extra words earned it. A judge reads thoroughness into length. Fix it by naming concision in the rubric explicitly, and by tracking response length as its own metric so you can see when a “quality win” is really a padding win.
• Self-preference bias. Models rate outputs from their own family more highly. If the model under test is also the judge, you’ve built a grading system with a thumb on the scale. Fix it by never judging with the model you’re evaluating — and by keeping the judge fixed when you change the models in your arms.
• Rubric drift. You tweaked the rubric in April, changed judge model in June, and nudged temperature in July. The scores from March and the scores from August are not the same measurement, and comparing them tells you about your grading pipeline rather than your product. Fix it by versioning the judge model, the rubric text, and the sampling parameters together, so every score records exactly what produced it.

None of these are reasons to abandon LLM grading. They’re the reasons a naive judge — one prompt, no rubric, pairwise, same model family, unversioned — produces numbers that move for reasons you can’t explain.

The reframe that resolves the argument

Here is the part that matters more than everything above, and that most discussions of LLM-as-judge miss entirely.

A judge does not need to be right. It needs to be wrong in the same direction for both variants.

An A/B test does not read the level of a metric. It reads the difference between two arms measured the same way, at the same time, on the same traffic. If your judge is systematically generous — it hands out 4.2 where a careful human would say 3.6 — that generosity applies to arm A and arm B alike, and it cancels in the subtraction. What survives is the part you asked about: whether B scored higher than A, and by how much, and whether that gap is bigger than noise.

This is exactly the logic that lets a slightly miscalibrated thermometer still tell you the room got warmer. Bias in the level is survivable. What is not survivable is bias that differs between arms — a judge that likes long answers, when one of your variants produces longer answers for reasons unrelated to quality. That’s the failure to design against, and it’s why the bias list above is worth taking seriously even under this reframe.

The honest limit: this licenses “variant B graded 0.3 higher, with a confidence interval that excludes zero.” It does not license telling your board that your product scores 4.2 out of 5 on quality. The comparison is trustworthy in a way the absolute number simply isn’t.

Calibrating a judge takes one afternoon

You don’t need an annotation programme. You need one afternoon, once, per rubric:

1. Sample 50–100 real outputs from production — not from your test set, which is a photograph of last quarter’s traffic.
2. Have a human who understands the product grade them against the same rubric the judge uses.
3. Compare. You’re looking for agreement rate on the coarse decision (good / not good) and for the pattern of disagreement, which is more informative than the rate.
4. Read the disagreements. They almost always point at an underspecified rubric rather than a dumb judge — “helpful” means one thing to you and another to the model, and the fix is to write down which one.
5. Rewrite the rubric, re-run, and stop when the disagreements are cases two humans would also argue about.

The rubric is the product here, not the judge model. A specific rubric with a mediocre model beats a vague rubric with the best model available, every time.

What the judge still can’t see

A judge reads the output. That bounds what it can possibly know:

• Whether the user was satisfied. The judge sees a well-formed answer. It cannot see that the user rephrased the question twice and left.
• Whether the answer was acted on. Copied, edited, shipped, ignored — all invisible to anything reading text.
• Cost and latency. A variant that grades 4% higher and costs 40% more is a bad trade the judge has no opinion about.
• Downstream business effects. Conversion, retention, support deflection. These exist only where real users are.

Which is the real argument for keeping judge scores as one input among several rather than the scoreboard. Quality grading tells you whether the output is good. The operational and behavioural metrics tell you whether the change was worth making. Read them together or you’ll ship the version that reads beautifully and costs twice as much.

LLMJury versions every judge and rubric, so a result always records what graded it, and runs grading on a sample of live traffic rather than a frozen set. Custom metrics take a natural-language rubric plus a structured output schema, which means the rubric-writing above is the actual work — the rest is configuration. Quality grading starts on Pro; the Free plan measures latency, cost, and your own business events, which is often enough to decide.

So: yes, with conditions. Use a judge you didn’t build the product on, write a rubric specific enough to argue with, version everything, and read the difference between arms rather than the number on the page. The skepticism was well-placed. It just points at the rubric, not the idea.

Start free — no credit card — or read what else is worth measuring alongside a quality score.