Deciding whether an LLM output is any good — rubrics, judges, offline datasets, and the gap between scoring well on a benchmark and working for your users.
10 posts
· 6 min read
Outcome decides, trajectory explains, cost constrains. Why the outcome assertion has to come from outside the agent, why a golden path punishes a better route, and why nine sampling steps mean more traffic rather than less.
· 5 min read
Temperature 0 makes decoding greedy, not deterministic — and the reasons are floating-point, batching, and routing. The reframe that fixes it: one output against one output is n = 1, and nobody accepts that anywhere else in engineering.
· 5 min read
The objection is fair, and weaker than it looks. Position bias, verbosity bias, self-preference, rubric drift — and the reframe that resolves the argument: a judge doesn’t have to be right, it has to be wrong the same way for both arms.
· 5 min read
No alert fires, because nothing breaks — the answers are still fluent, confident, and worse. Six causes with the detection method for each, and why a site-wide average is where regressions hide.
· 5 min read
Quality, operational, behavioural, business — four layers, what each answers, how fast it moves, and the trap in each. Plus why watching twenty metrics makes you less able to detect the effect you care about.
· 5 min read
Chunk size 512 because a blog post said so. Eight knobs, what each trades against, how to test it — and why retrieval metrics are diagnostic while end-to-end quality is the decision.
· 6 min read
“Hallucination rate” is four different failures wearing one word. The taxonomy with a detection method for each, why groundedness has to be scored per claim rather than per answer, and why the absolute number belongs nowhere near a sales deck.
· 5 min read
“Valid” is five different claims, and the one that hurts is a well-formed object with an invented enum member falling through a switch with no default. Why truncation is usually the culprit, and why compliance is the cheapest guardrail you can declare.
· 5 min read
Invented examples measure how you handle a user who read the docs; model-generated ones share your model’s blind spot by construction. The five strata with their shares, why the rubric matters more than the labels, and the questions your product can’t answer.
· 3 min read
Golden datasets are unit tests, not proof. Distribution drift, eval overfitting, and the metrics that only exist in production — latency, cost, and what users do next.