LLM-as-judge rubrics you can paste in
Writing a rubric that scores the same output the same way twice is the hardest part of using an LLM as a judge, and it is the part every guide leaves as an exercise. These are nine finished ones — the rubric text, the output schema, what each score means, two scored examples, and the specific way each one goes wrong.
They are in the shape LLMJury’s custom metrics accept, so they work by pasting. They are also just a specification of what to measure, so they are worth having if you score it some other way. Judge-scored metrics start at Pro — saying that here rather than after you have copied four of them.
Did the support reply actually resolve the request?
ordinalprimaryThe rubric for the most common LLM feature there is. Scores whether the reply answers the question the user asked, rather than whether it sounds helpful — which is the distinction every "it reads better" prompt change quietly fails.
reply_resolution
Is every claim in the answer supported by the retrieved context?
binaryguardrailThe rubric that makes a retrieval change safe to ship. Binary and strict: one unsupported claim fails the whole answer, because that is how a user experiences it.
answer_faithfulness
Does the summary say what the source said?
ordinalprimarySeparates the two ways a summary fails — it invents, or it omits the thing that mattered — and scores them on one scale, because a prompt change usually trades one for the other.
summary_fidelity
Is the output actually the shape you asked for?
binaryguardrailThe cheapest guardrail on this list and the one most often missing. Prompt changes break parsing, and parsing failures show up as a support ticket rather than as a metric.
output_conformance
Did the model pick the right label?
binaryprimaryFor the case where there is a right answer. Binary, unambiguous, and the highest-powered metric on this list — which is why a classification experiment reaches significance on a fraction of the traffic a quality experiment needs.
label_correct
Would this generated code actually work?
ordinalprimaryScores generated code the way a reviewer would — does it solve the stated problem, does it run, does it handle the obvious edge case — rather than whether it looks like good code.
code_correctness
Does it sound like us?
ordinalguardrailThe one rubric you must write yourself, with a worked template. Tone is the metric everyone wants and nobody defines, and an undefined tone rubric produces a number with more noise than signal.
brand_voice
Did it refuse when it should have — and only then?
binaryguardrailScores both directions of the safety trade in one metric: refusing what should be answered is a failure here, not just answering what should be refused. Most safety rubrics only measure one side, which is why safety changes look free.
refusal_appropriate
Did the conversation get the user where they were going?
binaryprimaryScores the whole conversation, not a turn. The unit of analysis is the thing multi-turn experiments get wrong most often, and no rubric can fix a unit that is already the wrong size.
conversation_resolved
Two things to get right before the wording
The category decides the statistics. A binary metric routes to a two-proportion z-test and needs far less traffic to move; an ordinal 1–5 score routes to a permutation test with a bootstrap confidence interval and needs more. If traffic is scarce, a well-defined binary metric will tell you something long before a five-point scale does.
The role decides the verdict. A primary metric is the question the experiment exists to answer; a guardrail must not get worse, and a significant regression on one blocks a recommendation. Filing a safety rubric as primary turns “did not get worse” into “must improve”, which is a different experiment from the one you meant to run.
Both are explained further in the post on writing judge rubrics and in the worked example.