Running experiments as a practice rather than a one-off: what to decide before you start, how long to wait, and what to record so the result still means something in three months.
8 posts
· 5 min read
“We’re too small to A/B test” is usually wrong. A sample-size table for the metrics teams actually run — task success, judge quality, cost, latency — and the four levers that make small traffic workable.
· 5 min read
A 3% imbalance on 100,000 users is not noise, it is a bug — and it invalidates every number in the experiment, not just the split. What causes SRM, why a timeout is the LLM-specific culprit, and why you must never reweight.
· 5 min read
Fourteen daily looks at a p-value turns a 5% error rate into roughly one in five — and inflates every effect you report. Stopping rules that a real team will actually follow, plus the five lines to write down before you start.
· 5 min read
Quality, operational, behavioural, business — four layers, what each answers, how fast it moves, and the trap in each. Plus why watching twenty metrics makes you less able to detect the effect you care about.
· 5 min read
Sixty thousand turns is not sixty thousand observations. The unit-of-analysis error doesn’t make an experiment noisy, it makes it overconfident — plus which metrics live at turn, conversation, and user level, and why turns-to-resolution lies on its own.
· 5 min read
A bandit optimises, an experiment explains — and you get one. The comparison table, plus the reason bandits fit LLM products badly: they need fast feedback, and a judge score sampled at 5% on a batch is not fast feedback.
· 5 min read
Invented examples measure how you handle a user who read the docs; model-generated ones share your model’s blind spot by construction. The five strata with their shares, why the rubric matters more than the labels, and the questions your product can’t answer.
· 3 min read
Ten playground outputs can’t tell you what a prompt change does to real traffic. The case for measuring every prompt edit the way you measure every other production change.