Judge rubric

Did the conversation get the user where they were going?

Scores the whole conversation, not a turn. The unit of analysis is the thing multi-turn experiments get wrong most often, and no rubric can fix a unit that is already the wrong size.

Metric name: conversation_resolved
Category: binary — Two-proportion z-test, with an automatic permutation fallback below an expected count of 10
Role: primary — The question the experiment exists to answer. A recommendation requires a winner here.

When to use it

Chatbots, agents, and anything where a user takes several turns to get what they came for. Score once per conversation, not once per assistant message.

What it catches: The prompt change that makes each individual reply better and the conversation longer. Per-turn quality goes up, turns-to-resolution goes up with it, and the user has a worse time.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given the user's goal (stated or inferable from the first turn) and the full
conversation transcript. Decide whether the conversation resolved that goal.

Score 1 if the user's original goal was met by the end of the conversation, or if the
conversation correctly established that it could not be met and handed off appropriately.

Score 0 if the conversation ended without the goal being met, if the assistant lost track of
information the user had already given, if it contradicted something it said earlier, or if
the user had to repeat themselves to make progress.

Judge the conversation as a whole. A conversation of one excellent turn that resolves the goal
scores the same as one of six turns that resolves it — this metric is about the outcome. Count
the turns separately, as their own metric, and read the two together.

Losing earlier context is a score of 0 even if the final answer is correct. An assistant that
gets there after forgetting what it was told is not the same product as one that does not
forget.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "enum": [0, 1] },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

1: Goal met, or correctly established as unmeetable and handed off
0: Not met, context lost, self-contradiction, or the user had to repeat themselves

Scored, on two real outputs

Scores 1

User gives an order number in turn 1, asks about a refund in turn 3. The assistant answers using the order number without asking again.

Goal met, and earlier context carried — which is the specific thing that separates a conversation from a series of independent replies.

Scores 0

Same conversation, except turn 4 is “Could you provide your order number so I can look that up?”

The answer that eventually arrives is correct. The experience is not, and a per-turn quality metric scores this conversation well.

How this one goes wrong

The unit of analysis is the whole point. Scoring per turn and averaging gives a longer, more repetitive conversation a better score than a short one that worked — the post on multi-turn testing is about exactly this trap.
Assignment must be sticky across the conversation, which it is: bucketing is deterministic on the user identifier, so every turn lands in the same arm. If you assign per request with a random id, the experiment is measuring nothing.
The judge needs the full transcript, so this is the most expensive rubric here per evaluation. Sample conversations, not turns, and watch the judge budget.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.