Judge rubric

Is every claim in the answer supported by the retrieved context?

The rubric that makes a retrieval change safe to ship. Binary and strict: one unsupported claim fails the whole answer, because that is how a user experiences it.

Metric name: answer_faithfulness
Category: binary — Two-proportion z-test, with an automatic permutation fallback below an expected count of 10
Role: guardrail — Must not get worse. A significant regression on it blocks a recommendation.

When to use it

Any retrieval-augmented answer. Attach it as a guardrail on every RAG experiment, including ones about latency or cost — those are exactly the changes that quietly trade grounding away.

What it catches: The fluent answer that fills a retrieval gap with the model’s own knowledge. It is right often enough to pass a spot check and wrong in the cases that matter, and no amount of reading ten outputs finds it.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given a user question, the retrieved context that was supplied to the model, and
the model's answer. Decide whether every factual claim in the answer is supported by the
retrieved context.

Score 1 if and only if all of the following hold:
- Every factual claim in the answer can be traced to a specific span of the retrieved context.
- No claim contradicts the retrieved context.
- Numbers, dates, names, and quantities appear in the context with the same values.

Score 0 if any claim is unsupported, contradicted, or invented — including a claim that happens
to be true in the real world but is absent from the supplied context. Being right by luck is
still ungrounded.

The following do NOT count as unsupported claims: paraphrase, summary, reasonable arithmetic
performed on numbers that are present, common-knowledge connective language, and an explicit
statement that the context does not contain the answer.

If the answer says the context does not contain enough information to answer, score 1. That is
the correct behaviour, not a failure.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "enum": [0, 1] },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

1: Every claim traceable to the retrieved context — including a correct "I don’t know"
0: One or more claims unsupported, contradicted, or invented

Scored, on two real outputs

Scores 1

Context says the refund window is 30 days. Answer: “You have 30 days from delivery to request a refund. The context does not say whether that applies to sale items.”

Every claim is in the context, and the gap is named instead of filled. Naming the gap is the behaviour you want the rubric to reward.

Scores 0

Context says the refund window is 30 days. Answer: “You have 30 days to request a refund, and refunds are usually processed within 5–7 business days.”

The processing time is not in the context. It is plausible, probably true of some company somewhere, and exactly the kind of invention a spot check waves through.

How this one goes wrong

File this as a guardrail, not a primary metric. As a primary it rewards the degenerate answer — a model that says "the context does not answer this" every time scores a perfect 1.
The judge needs the retrieved context, not just the question and the answer. If your integration does not send it, this rubric cannot work; that is a plumbing problem, not a wording one.
A binary metric with a very high pass rate has few failures, so the two-proportion z-test falls back to a permutation test on the 0/1 indicator automatically. Expect the confidence interval to be wide until the sample is large.
Faithfulness is not helpfulness. An answer can be perfectly grounded and useless — pair this with a resolution or relevance metric as the primary.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.