Concept
LLM-as-a-jury: when a panel of judges beats one
An LLM jury is several language models scoring the same output independently, with their verdicts combined by a vote. It is a real technique with a real result behind it, and it is not what most teams need. This page is what it is, when it is worth three times the bill, and how to decide.
What an LLM jury actually is
Take one model output you want to grade. A single judge sends it to one model with a rubric and gets back one score. A jury sends the same output to several models — usually from different families, so their failure modes are not correlated — collects a score from each, and reduces them to one number.
That last step is a real decision, not a formality. A majority vote suits categorical rubrics (pass/fail, safe/unsafe) and throws away how confident anyone was. A mean keeps the gradations on a 1–5 scale and lets one outlier drag the panel. A median is the usual compromise and needs an odd number of jurors to avoid ties. Publish which one you used: two teams reporting “jury score 4.1” from the same raw scores under different aggregations are not reporting the same number.
When a panel genuinely beats one judge
The case for a jury rests on the biases a single grader has and a mixed panel does not share. Three are well documented enough to plan around:
- Self-preference. A model tends to score text it could have written above text it could not. If one of your variants is generated by the same model doing the grading, that is a thumb on the scale pointing in a known direction.
- Position bias. In pairwise “which is better” prompts, models favour one slot regardless of content. A panel does not fix this — swapping the order and averaging does — but a panel makes it visible, because jurors with different position biases disagree.
- Length bias. Longer answers score higher at equal quality. This one is stubborn: it is shared across model families, so a jury tends to inherit it rather than cancel it. A rubric that names concision is worth more than a second juror here.
The pattern underneath: a jury helps where the graders would disagree. On a subjective rubric — was this reply empathetic, was this summary faithful — disagreement is real and averaging it is doing something. On a rubric with a defensible right answer, the second and third juror mostly agree with the first, and you have tripled the cost to shrink an error that was already small.
That is measurable before you commit. Score two or three hundred outputs with two different models and look at the disagreement rate. If they agree on 95% of items, a jury buys you very little. If they disagree on a third, you have learned something more useful than the jury would have told you: your rubric is ambiguous, and tightening it beats adding jurors.
Why an experiment usually does not need one
Here is the part the jury literature mostly skips, and it is the reason this site exists.
Grading research asks: how close is this score to what a human would say? An experiment asks a different question: did arm B score higher than arm A? Those come apart. A judge that is systematically two points generous is useless for the first question and almost harmless for the second, because it is generous to both arms and the bias subtracts out of the difference. What survives is noise — and noise is exactly what a confidence interval is built to account for.
So the thing that decides whether you can call an A/B test is not grader accuracy. It is sample size against effect size. A jury costs one model call per juror per item, which for a fixed judging budget means a third as many graded outputs on a three-model panel. Under almost any assumption, a cheaper grader on three times the sample is the better experiment.
Two honest exceptions. If the judge bias is differential — if it favours one arm specifically, which self-preference does when a variant is generated by the grading model — it does not cancel, and a mixed panel is a genuine fix. And if you are reporting an absolute number to somebody rather than a difference, you are back to the grading question, where accuracy is the whole game.
What LLMJury does instead
One judge, and the effort spent on the statistics around it. Concretely, what ships today:
- A single judge model scores a sampled subset of outputs against a rubric — built-in quality, safety and relevance, or a custom metric you write as a natural-language rubric plus an output schema.
- Rubrics are versioned and locked per run. Editing a rubric mid-experiment creates a new version rather than silently re-grading, so the scores in one result were all produced by the same grader.
- Judging is sampled, cached and hard-budgeted, so grading cost stays a line item you set rather than a function of your traffic.
- The verdict is gated on statistics, not on scores. A sample ratio mismatch check halts the analysis outright, effects come with bootstrap confidence intervals, and every metric-by-variant comparison is FDR-corrected before anything is called significant.
A panel of judges is not on the roadmap, and this page is not going to hint that it is. What you can do today is grade however you like — a panel you built, a fine-tuned classifier, a human review queue — and send the aggregated score in as a business metric with track(). From there it is an ordinary numeric metric: same SRM gate, same test routing, same FDR correction as everything else.
LLM juries, answered
- Is LLMJury a jury of models?
- No. Despite the name, LLMJury scores each sampled output with a single LLM-as-judge against your rubric — one model, one score, no panel and no vote. The name is about the verdict you get at the end: a statistically gated ruling on whether one prompt beat another. The rigour is in the statistics — an SRM gate, permutation tests with bootstrap confidence intervals, and Benjamini-Hochberg correction across every metric and variant — not in the number of models doing the grading.
- What is the difference between LLM-as-a-judge and LLM-as-a-jury?
- A judge is one model scoring an output against a rubric. A jury is several models scoring the same output independently, with their scores combined by a vote, mean, or median. A jury costs one model call per juror per item, so a three-model jury is roughly three times the price and three times the latency of a single judge for the same coverage.
- Does a jury of LLMs give more accurate scores than one judge?
- Sometimes, and less often than the framing suggests. Panels help most where a single model has a bias that a different model family does not share — self-preference for its own outputs, position bias in pairwise comparisons, and length bias are the well-documented ones — and on subjective rubrics where reasonable graders disagree. They help least on rubrics with a clear right answer, where a second and third model mostly agree with the first at three times the cost. The honest test is to score a few hundred items twice and see whether the disagreement rate is high enough to matter.
- Do I need a jury if I am running an A/B test?
- Usually not, and this is the part the jury literature tends to skip. In an experiment you are not trying to establish the true quality of one output — you are comparing two arms, so a judge bias that applies equally to both arms largely cancels out, and what is left is noise that a confidence interval already accounts for. What actually decides whether you can call the result is sample size and effect size. A more expensive grader on a quarter as many samples is a worse experiment, not a better one.
- Can I use a jury with LLMJury?
- Not inside the platform today. You can score outputs however you like — with a panel, a fine-tuned classifier, or a human review queue — and send the aggregated score in as a business metric with track(). LLMJury then treats it like any other numeric metric: same SRM gate, same significance routing, same FDR correction. What is not available is configuring a multi-model panel as a built-in judge metric.