Judge rubric

Did the model pick the right label?

For the case where there is a right answer. Binary, unambiguous, and the highest-powered metric on this list — which is why a classification experiment reaches significance on a fraction of the traffic a quality experiment needs.

Metric name: label_correct
Category: binary — Two-proportion z-test, with an automatic permutation fallback below an expected count of 10
Role: primary — The question the experiment exists to answer. A recommendation requires a winner here.

When to use it

Intent detection, routing, tagging, triage, sentiment, moderation category assignment — anything where a human with the guidelines in front of them would agree on one label.

What it catches: Nothing subtle. This is the rubric for when you can be precise, and its value is statistical: a clean binary metric needs far less traffic to detect a real difference than a 1–5 judged score.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given the input, the label taxonomy with its definitions, and the label the model
assigned. Decide whether the assigned label is correct.

Score 1 if the assigned label is the one the taxonomy's definitions require for this input.
Score 0 if a different label is required.

When the input genuinely satisfies more than one label's definition, score 1 if the assigned
label is among them — do not penalise a defensible choice between two correct answers.

When the input satisfies no label's definition and the taxonomy has a catch-all or "other"
label, that catch-all is the correct answer; score 1 only if it was chosen.

Judge against the definitions as written, not against your own sense of what the label ought to
mean. If the definitions are ambiguous for this input, say so in the reason field and score 1 —
an ambiguous taxonomy is a problem with the taxonomy, and marking it wrong here would put that
noise into the metric.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "enum": [0, 1] },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

1: Correct label, or a defensible one among several that fit
0: A different label was required by the definitions

Scored, on two real outputs

Scores 1

“My card was charged twice this morning” → `billing_issue`, where the taxonomy defines billing_issue as any dispute about a charge.

Matches the definition as written.

Scores 0

“My card was charged twice this morning” → `account_access`, because the user later mentions being logged out.

A secondary detail pulled the label off the primary intent. This is the classic long-input failure, and it is invisible until you measure it.

How this one goes wrong

The judge must be given the taxonomy definitions. Without them it invents its own idea of what each label means and you are measuring the judge, not the model.
The ambiguity instruction matters more than it looks. Without it, every genuinely ambiguous input becomes a coin flip that adds pure variance to a metric whose whole advantage is low variance.
If you have ground-truth labels already, you do not need a judge for this — send the correctness as a business event with track(…) and spend the judge budget on something that needs judgement.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.