The maths that separates a real difference from noise: significance, correction for testing many things at once, confidence intervals, and the tests that suit skewed LLM metrics.
8 posts
· 5 min read
Temperature 0 makes decoding greedy, not deterministic — and the reasons are floating-point, batching, and routing. The reframe that fixes it: one output against one output is n = 1, and nobody accepts that anywhere else in engineering.
· 5 min read
The objection is fair, and weaker than it looks. Position bias, verbosity bias, self-preference, rubric drift — and the reframe that resolves the argument: a judge doesn’t have to be right, it has to be wrong the same way for both arms.
· 5 min read
“We’re too small to A/B test” is usually wrong. A sample-size table for the metrics teams actually run — task success, judge quality, cost, latency — and the four levers that make small traffic workable.
· 5 min read
A 3% imbalance on 100,000 users is not noise, it is a bug — and it invalidates every number in the experiment, not just the split. What causes SRM, why a timeout is the LLM-specific culprit, and why you must never reweight.
· 5 min read
Fourteen daily looks at a p-value turns a 5% error rate into roughly one in five — and inflates every effect you report. Stopping rules that a real team will actually follow, plus the five lines to write down before you start.
· 5 min read
Sixty thousand turns is not sixty thousand observations. The unit-of-analysis error doesn’t make an experiment noisy, it makes it overconfident — plus which metrics live at turn, conversation, and user level, and why turns-to-resolution lies on its own.
· 5 min read
A bandit optimises, an experiment explains — and you get one. The comparison table, plus the reason bandits fit LLM products badly: they need fast feedback, and a judge score sampled at 5% on a batch is not fast feedback.
· 1 min read
Why “it feels better” is not an eval strategy, and how SRM gates, FDR correction, and an LLM judge make prompt experiments trustworthy.