Judge rubric

Does it sound like us?

The one rubric you must write yourself, with a worked template. Tone is the metric everyone wants and nobody defines, and an undefined tone rubric produces a number with more noise than signal.

Metric name: brand_voice
Category: ordinal — Permutation test with a bootstrap confidence interval
Role: guardrail — Must not get worse. A significant regression on it blocks a recommendation.

When to use it

Any user-facing text where a voice guideline exists. Run it as a guardrail on experiments about something else — a prompt change optimised for resolution will drift the voice, and you want to know by how much.

What it catches: Voice drift as a side effect. Nobody ships a change intended to make the product sound wrong; it happens because the change was aimed at something else and tone was not being watched.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

Score how well this text matches the voice rules below, from 1 to 5.

VOICE RULES — replace these with yours. They must be checkable rules, not adjectives: a judge
given "friendly but professional" scores inconsistently, because that phrase does not identify
anything in a specific sentence.

  1. Second person ("you"), never third person about the user.
  2. Active voice. No "was processed", "has been received".
  3. No exclamation marks.
  4. No filler openers: "Great question", "Absolutely", "I'd be happy to".
  5. Sentences under 25 words.
  6. Say what to do before explaining why.
  7. No internal jargon: no product codenames, no team names, no acronyms without expansion.

5 — Follows every rule.
4 — One violation, in one sentence.
3 — Two or three violations, or one that runs through the whole text.
2 — Four or more violations; the voice is recognisably not ours.
1 — Reads as a different product entirely.

Do not score whether the content is correct or helpful. A wrong answer in perfect voice scores
5 here; correctness is a separate metric.

In the reason field, name each rule violated by number. That is what makes a regression on this
metric actionable rather than mysterious.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "minimum": 1, "maximum": 5 },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

5: Every rule followed
4: One violation, localised
3: Two or three, or one running throughout
2: Four or more — recognisably off-voice
1: Reads as a different product

Scored, on two real outputs

Scores 5

“Open Settings → Billing to cancel. You keep access until 14 March.”

Second person, active, action before explanation, short, no filler. Every rule is checkable against this sentence, which is the whole point.

Scores 2

“Great question! Your cancellation request has been received and will be processed by our Billing team shortly.”

Rules 2, 3, 4, and 6 broken in nineteen words — and it is the output a "make it friendlier" prompt change produces.

How this one goes wrong

Adjectives make this rubric worthless. "Friendly but professional" gives a judge nothing to check, so the same output scores differently on different calls, the variance goes up, and the metric needs far more traffic to move. Numbered, checkable rules are the fix.
Keep it a guardrail. As a primary metric it optimises for a voice, and a voice is not an outcome — the product still has to work.
Seven rules is about the ceiling. Past that, judges start missing violations and the score drifts down for reasons you cannot trace.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.