Judge rubric

Did the support reply actually resolve the request?

The rubric for the most common LLM feature there is. Scores whether the reply answers the question the user asked, rather than whether it sounds helpful — which is the distinction every "it reads better" prompt change quietly fails.

Metric name: reply_resolution
Category: ordinal — Permutation test with a bootstrap confidence interval
Role: primary — The question the experiment exists to answer. A recommendation requires a winner here.

When to use it

Any assistant that answers an inbound question — support, help desk, in-product Q&A, the first line of a chatbot. Make it the primary metric when the experiment is about the answer itself.

What it catches: The rewrite that is warmer, longer, better organised, and answers a slightly different question than the one asked. Human reviewers reliably prefer it; users reliably do not.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

Score how completely this reply resolves the user's actual request, on a scale of 1 to 5.

Judge only resolution. Ignore tone, warmth, formatting, length, and politeness — those are
scored separately and must not influence this number.

5 — Fully resolves the request. Every part of what was asked is answered, with the specific
    information or action needed. A user who reads this needs nothing further.
4 — Resolves the main request, but leaves one minor sub-question or detail unaddressed.
3 — Partially resolves it: correct as far as it goes, but the user still has to ask a
    follow-up to get what they came for.
2 — Addresses the topic without resolving anything: general information, a restatement of the
    question, or a redirection to somewhere else.
1 — Does not address the request, or answers a different question than the one asked.

If the reply correctly declines because the request is out of scope or needs a human, score 4 —
a correct handoff is a resolution, not a failure. If the reply contains a factual claim that is
wrong, cap the score at 2 regardless of how complete it looks.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "minimum": 1, "maximum": 5 },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

5: Fully resolved — nothing further needed
4: Main request resolved; a minor detail left, or a correct handoff
3: Partial — a follow-up is still required
2: On topic, resolves nothing; or contains a factual error
1: Answers a different question, or none

Scored, on two real outputs

Scores 5

“Your plan renews on 14 March. To cancel before then, open Settings → Billing → Cancel plan; you keep access until the renewal date and are not charged again.”

Answers the date, the how, and the consequence — the three things behind the question, with no follow-up left.

Scores 2

“Great question! Billing can definitely be confusing. You can manage everything about your subscription from your account settings, and our team is always happy to help if you get stuck.”

Warm, well-formed, on topic, and the user still does not know when they renew or where to click. This is the output that wins a side-by-side eyeball test.

How this one goes wrong

Do not let length in. A judge asked for "quality" scores long answers higher almost every time; naming resolution and explicitly excluding tone is what stops that.
The factual-error cap is what makes this safe to use as a primary metric. Without it, a confidently wrong answer that covers everything scores 5.
An ordinal 1–5 metric routes to a permutation test with a bootstrap CI. That is correct, and it needs more traffic than a binary metric to move — if you have very little traffic, consider collapsing this to a binary "resolved / not resolved".

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.