Splitting live traffic between versions of an LLM feature and reading which one actually won — the mechanics, the pitfalls, and when the answer is trustworthy.
8 posts
· 5 min read
“We’re too small to A/B test” is usually wrong. A sample-size table for the metrics teams actually run — task success, judge quality, cost, latency — and the four levers that make small traffic workable.
· 5 min read
The new model benchmarks better and performs worse, because your prompt is a fossil record of the old model’s flaws. A five-step migration sequence with exit criteria, and why you should leave a comparison arm running forever.
· 5 min read
Four levers ranked by effort, and the statistical point that decides it: you are not asking whether the cheap model is better, you are asking whether it is worse by less than you will trade for the money. That is a non-inferiority test.
· 5 min read
Fourteen daily looks at a p-value turns a 5% error rate into roughly one in five — and inflates every effect you report. Stopping rules that a real team will actually follow, plus the five lines to write down before you start.
· 5 min read
Chunk size 512 because a blog post said so. Eight knobs, what each trades against, how to test it — and why retrieval metrics are diagnostic while end-to-end quality is the decision.
· 5 min read
Sixty thousand turns is not sixty thousand observations. The unit-of-analysis error doesn’t make an experiment noisy, it makes it overconfident — plus which metrics live at turn, conversation, and user level, and why turns-to-resolution lies on its own.
· 5 min read
A bandit optimises, an experiment explains — and you get one. The comparison table, plus the reason bandits fit LLM products badly: they need fast feedback, and a judge score sampled at 5% on a batch is not fast feedback.
· 3 min read
Ten playground outputs can’t tell you what a prompt change does to real traffic. The case for measuring every prompt edit the way you measure every other production change.