Using a model to score another model’s output: the biases it brings, the budget it needs, and why a consistent judge beats an accurate one for comparing two arms.
3 posts
· 5 min read
The objection is fair, and weaker than it looks. Position bias, verbosity bias, self-preference, rubric drift — and the reframe that resolves the argument: a judge doesn’t have to be right, it has to be wrong the same way for both arms.
· 6 min read
“Hallucination rate” is four different failures wearing one word. The taxonomy with a detection method for each, why groundedness has to be scored per claim rather than per answer, and why the absolute number belongs nowhere near a sales deck.
· 3 min read
Golden datasets are unit tests, not proof. Distribution drift, eval overfitting, and the metrics that only exist in production — latency, cost, and what users do next.