Judge rubric

Does the summary say what the source said?

Separates the two ways a summary fails — it invents, or it omits the thing that mattered — and scores them on one scale, because a prompt change usually trades one for the other.

Metric name: summary_fidelity
Category: ordinal — Permutation test with a bootstrap confidence interval
Role: primary — The question the experiment exists to answer. A recommendation requires a winner here.

When to use it

Meeting notes, document digests, thread recaps, ticket summaries — anywhere the output is shorter than the input and someone acts on it without reading the original.

What it catches: The shorter, punchier summary that reads better and drops the one caveat that changed the decision. Compression is a lossy operation and prompt changes move where the loss lands.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given a source text and a summary of it. Score the summary's fidelity to the source
from 1 to 5.

Two kinds of failure count equally: adding what is not there, and omitting what a reader would
need. Judge both.

5 — Every claim is in the source, and every point a reader would need to act correctly is
    present. Nothing material is lost.
4 — Faithful, with one minor omission that would not change a reader's decision.
3 — Faithful to what it includes, but omits something material — a caveat, a condition, a
    dissenting view, or a number a reader would act on.
2 — Contains a claim not supported by the source, or misstates a number, date, or name.
1 — Contradicts the source, or inverts its conclusion.

Style, length, and structure are not scored. A blunt summary and an elegant one that carry the
same information get the same score.

Pay specific attention to hedges and conditions in the source ("only if", "pending approval",
"subject to"). Dropping one is a material omission, not a minor one.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "minimum": 1, "maximum": 5 },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

5: Nothing invented, nothing material lost
4: One immaterial omission
3: Material omission — a caveat, condition, or actionable number
2: Unsupported claim, or a misstated number, date, or name
1: Contradicts the source or inverts its conclusion

Scored, on two real outputs

Scores 5

Source: budget approved, conditional on Q3 headcount freeze. Summary: “Budget approved, conditional on the Q3 headcount freeze holding.”

The condition survives compression. That condition is the entire content of the decision.

Scores 3

Source: budget approved, conditional on Q3 headcount freeze. Summary: “Budget approved — we’re clear to proceed.”

Nothing invented, and it is still wrong to act on. This is the failure mode a "make it more concise" prompt change produces, and the one an eyeball test rewards.

How this one goes wrong

Naming hedges and conditions explicitly is what makes this rubric work. A generic "is the summary accurate?" judge scores the bad example above a 5, because everything in it is true.
Do not also ask the judge to score brevity here. Fidelity and brevity trade off directly, and a metric that scores both hides the trade instead of measuring it — run brevity as its own metric and read them side by side.
The judge sees the source, so the source counts against your judge token budget. Long documents make this expensive; sample harder rather than truncating, because a truncated source produces a false omission finding.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.