When the JSON stops parsing: structured output as a metric

A prompt edit shipped on Tuesday. On Thursday, 0.4% of requests come back with a trailing comma and fail to parse. Nothing alerts, because 0.4% is inside the noise floor of every dashboard you own, and the retry logic absorbs it silently until the day traffic doubles and the retry budget doesn’t.

Structured output is where LLM systems meet ordinary software, and it fails in an ordinary software way — quietly, at the margins, in the code path nobody instrumented.

“Valid” is five different claims#

Most teams check one of these and get hurt by another.

1. It parses. Syntactically well-formed JSON. This is what try { JSON.parse(x) } tells you and it is the weakest of the five.
2. It matches the schema. The expected fields, present, with the expected types.
3. The values are in range. The enum member is one of your enum’s members. The score is 1–5 and not 7. The date is a date.
4. It’s semantically consistent. The fields don’t contradict each other or the input — a confidence: 0.95 attached to answer: null, a total that isn’t the sum of its parts.
5. It’s usable downstream. The system you hand it to accepts it and does the right thing.

The gap that bites is between 2 and 3. A well-formed object carrying an invented enum member passes JSON.parse, passes a loose type check, and then falls through a switch with no default branch — where it becomes a silent no-op rather than an error. No exception, no log line, no alert. Just a feature that stopped working for some fraction of requests.

Validate at level 3 minimum. If you’re using a schema library, this is free — you’re just choosing to use the strict validator rather than the permissive one.

Why it degrades without anyone shipping a bug#

The failure rate is not a constant of your prompt. It moves, for reasons that have nothing to do with the format instructions:

• Prompt growth. Someone adds three paragraphs of context, and the format instructions are now further from the end of a much longer prompt. Instruction-following on long prompts is not uniform.
• A model update. Providers change served weights behind stable names. Formatting habits are exactly the kind of thing that shifts — see the silent regression.
• A new input type. A user pastes something that triggers a refusal, and the refusal arrives as a polite paragraph of prose where your parser expected an object.
• An output token cap. The response is truncated mid-object, so the last brace never arrives. This is the most common cause by some distance, it looks exactly like a model formatting failure, and it is entirely your configuration. Check this first, always.
• A longer input. Larger extractions produce larger outputs, so a cap that was generous for the median request truncates the tail. Your failure rate is now correlated with input size, which means it’s correlated with your most important customers.

That last pair is worth sitting with: two of the five most common causes are you, not the model.

The fixes, honestly ranked#

Constrained decoding, or the provider’s native structured output mode. Where available, this is not incrementally better than prompting — it changes the failure rate by an order of magnitude, because the sampler is prevented from emitting a token that would break the grammar. If your provider offers it and you’re instead iterating on the wording of “respond only with valid JSON”, you’re optimising the wrong layer.

A schema in the request rather than an example in the prose. A formal schema is unambiguous. An example is a suggestion the model interprets, and it will happily invent a sixth field that resembles your five.

A repair pass. Cheap, effective, and quietly a cost and latency line item — which means it belongs in your measurement, not just your code. A repair rate that doubles is a regression even when the end-to-end failure rate looks flat, because you’re now paying twice for a fifth of your requests.

Retry with the error message. Works well. Also doubles the latency for every request that hits it, which lands squarely in your p99 — the number your slowest and most valuable requests experience. Retries hide failures from your dashboard while showing them to your users as delay.

Validate and degrade gracefully. The alternative to a malformed object is not a well-formed one, it’s a 500. Decide in advance what the feature does when parsing ultimately fails, and make that path visible rather than swallowing the error.

Schema compliance is an experiment metric, not a health check#

This is the part that matters, and it’s the part that’s usually missed.

Compliance rate is binary, cheap, judge-free, and low-variance. That combination makes it the fastest-resolving metric you own — faster than quality, faster than cost, far faster than any business outcome. It will give you a signal in days where a judge score needs weeks, for the reasons in same prompt, different answer.

Which means it is close to an ideal guardrail on any prompt change. Consider a rewrite that improves helpfulness by a genuine two points and drops compliance from 99.8% to 99.1%. Two points of helpfulness is a real win. Seven extra failures per thousand requests is a real regression, and on a high-volume endpoint it may well be the larger effect in absolute terms. Whether you ship it is a judgement call — but it has to be a judgement call, and it only becomes one if compliance was declared as a guardrail before the run rather than discovered in a bug report afterwards.

LLMJury takes metric role as a declaration: mark compliance as a guardrail, and a regression in it vetoes the win rather than being noticed later. The mechanics are in what to actually measure in an LLM product; this is the cheapest possible instance of it.

Recording it is one line next to the parse you’re already doing:

try:
    parsed = Schema.model_validate_json(response.content)
    client.track("schema_compliant", 1)
except ValidationError:
    client.track("schema_compliant", 0)
    parsed = fallback()

Four numbers worth having on a dashboard#

• Parse failure rate — the floor, and the one everyone already has.
• Schema and range violation rate — the one that catches invented enum members.
• Repair and retry rate — the hidden cost, and the leading indicator: it moves before the end-to-end failure rate does, because repairs are absorbing the damage.
• Truncation rate — responses that hit the output cap. Nearly always the single largest contributor, and the easiest to fix once you can see it.

None of them needs a judge, a rubric, or a plan upgrade. They are the cheapest reliable signal in an LLM product, and most teams are running without them because structured output feels like plumbing rather than quality.

It’s both.

Start free — binary metrics like compliance are measured on every plan.