Judge rubric

Is the output actually the shape you asked for?

The cheapest guardrail on this list and the one most often missing. Prompt changes break parsing, and parsing failures show up as a support ticket rather than as a metric.

Metric name: output_conformance
Category: binary — Two-proportion z-test, with an automatic permutation fallback below an expected count of 10
Role: guardrail — Must not get worse. A significant regression on it blocks a recommendation.

When to use it

Anything downstream of the model parses the output — JSON responses, tool arguments, extraction pipelines, classification labels fed to another system. Attach it to every experiment on such a feature, whatever the experiment is about.

What it catches: The prompt rewrite that improves the prose and starts wrapping the JSON in a markdown fence, or adds a friendly sentence before it. Your parser throws, your error rate moves, and no quality metric notices.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given the schema the output was required to match, and the model's raw output.
Decide whether the output conforms.

Score 1 if and only if:
- The output parses as valid JSON with no surrounding prose, code fences, or commentary.
- Every required field is present.
- Every field's type matches the schema.
- Every enum-valued field holds one of the permitted values, spelled exactly.
- No fields are present that the schema does not permit.

Score 0 otherwise. Score 0 specifically for: a leading or trailing explanatory sentence, a
markdown code fence around the JSON, a trailing comma, single quotes instead of double, a
number sent as a string, an enum value with different casing, and a null in a field that is
not nullable.

Do not judge whether the values are correct or sensible. This metric is about shape only —
correctness is a different metric with a different rubric.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "enum": [0, 1] },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

1: Parses, matches the schema exactly, nothing around it
0: Any deviation in shape — including a code fence or a stray sentence

Scored, on two real outputs

Scores 1

{"intent": "cancel_subscription", "confidence": 0.91}

Parses, both fields present and correctly typed, nothing around it.

Scores 0

Here is the classification:\n```json\n{"intent": "Cancel_Subscription", "confidence": "0.91"}\n```

Three failures at once — the preamble, the fence, the enum casing, and the number as a string. Every one of them is a real parser error, and all four are things a helpful-sounding prompt rewrite introduces.

How this one goes wrong

If your provider enforces structured output natively, this metric will sit at 1 forever and is not worth a judge call. Spend the budget elsewhere — a metric that cannot move is not a guardrail.
This is shape, not correctness. A perfectly-formed JSON object with the wrong label scores 1 here; that is the design, and it is why you need a correctness metric alongside it.
Because the pass rate is usually very high, the binary test falls back to a permutation test on the indicator at small samples. A regression here is usually so large it is obvious anyway.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.