Judge rubric

Would this generated code actually work?

Scores generated code the way a reviewer would — does it solve the stated problem, does it run, does it handle the obvious edge case — rather than whether it looks like good code.

Metric name: code_correctness
Category: ordinal — Permutation test with a bootstrap confidence interval
Role: primary — The question the experiment exists to answer. A recommendation requires a winner here.

When to use it

Code assistants, snippet generation, migration tools, query builders. Use it as the primary metric when the experiment is about the model or the prompt that produces code.

What it catches: Code that is idiomatic, well-named, well-commented, and solves a slightly different problem. Style is the thing an eyeball test can see; correctness is not.

The rubric

Paste this into the metric builder as the natural-language rubric. Adapt the wording to your product — keep the scale definitions and the exclusions, because those are what stop the judge scoring something other than what you meant.

You are given a task description and the code the model produced. Score its correctness from
1 to 5.

Judge against the task as stated. Do not reward style, naming, comments, or idiom — a blunt
correct solution outscores an elegant wrong one every time.

5 — Solves the stated task correctly, would run as written, and handles the edge cases the task
    implies (empty input, boundary values, the error path if one is described).
4 — Solves the task and would run, but misses an edge case the task implies.
3 — The approach is correct but the code would not run as written: a syntax error, a missing
    import, an undefined name, a wrong signature.
2 — Runs, but does not do what the task asked — solves an adjacent problem, or silently drops
    part of the requirement.
1 — Neither runs nor addresses the task, or calls an API that does not exist.

Score 1 for any invented library, method, or parameter, regardless of how correct the rest is.
A confident call to a function that does not exist is the failure mode that costs the most time
downstream.

The output schema

The structured output the judge must fill. The reason field is not optional in practice: it is what lets you read why a score moved instead of guessing.

{
  "type": "object",
  "properties": {
    "score": { "type": "integer", "minimum": 1, "maximum": 5 },
    "reason": { "type": "string" }
  },
  "required": ["score", "reason"],
  "additionalProperties": false
}

What each score means

5: Correct, runnable, handles the implied edge cases
4: Correct and runnable, misses an implied edge case
3: Right approach, would not run as written
2: Runs, solves an adjacent problem or drops a requirement
1: Neither, or invents an API

Scored, on two real outputs

Scores 5

Task: parse a CSV and return rows where amount > 100. Code reads the file, skips the header, coerces amount to a number, and returns an empty list for an empty file.

Solves the task, runs, and the empty-file case is handled rather than assumed away.

Scores 1

Same task. Code uses `pandas.read_csv(path, filter_gt="amount:100")` — a well-named parameter that does not exist.

Reads as the most competent answer of the batch. The parameter is invented, so nothing about it works, and a reviewer skimming for style will not catch it.

How this one goes wrong

A judge is not an interpreter. It is good at "does this call a real API and address the stated task" and unreliable about whether a subtle algorithm is correct — if you can run the code in a sandbox, do that and send the result as a business event instead.
The task description has to reach the judge. Scoring code without the requirement it was written against measures plausibility, which is precisely the thing that is already too high.
The invented-API rule is doing most of the work here. Without it, hallucinated calls land at 4 because everything around them is exemplary.

Run it on your own traffic

Custom judge metrics start at Pro — every plan starts free, and Free measures latency, cost, and business outcomes while you set the rest up. Metrics are versioned, so a change to this rubric mints a new version and every past result still traces to the exact wording that produced it.

The mechanics — sampling, caching, and the hard judge budget — are in the documentation, and pricing has the per-plan evaluation allowances.