Two weeks of traffic, a prompt someone spent a fortnight on, and the verdict reads no significant difference. Somebody in the room asks what the point of that was.
It’s a fair question and it deserves a real answer, because this is the most common outcome of any honest experiment programme — and treating it as a failure is how programmes get quietly cancelled.
“No significant difference” is three different results
Here’s the thing most people are never told: that verdict is not one finding. It’s three, they mean completely different things, and the p-value can’t tell them apart.
1. The effect is smaller than you could detect. Your confidence interval is wide and contains both a win you’d ship and a loss you’d roll back. You have learned very little — only that the change probably isn’t enormous. This is an underpowered test and the honest write-up says so.
2. The effect is genuinely near zero. Your interval is narrow and sits close to zero, excluding anything you’d care about. This is a real, informative result: the change doesn’t matter. Ship whichever arm is cheaper, simpler, or easier to maintain, and stop working on this.
3. The effect exists somewhere you didn’t look. Flat overall, moving within a segment, or moving on a metric you didn’t declare primary. Suggestive, not conclusive. The average is hiding the answer is the follow-up.
Result 1 and result 2 are opposites — “we don’t know” versus “we know it doesn’t matter” — and they arrive with identical p-values.
Read the interval, not the p-value
Which is the practical instruction, and it’s worth putting on a wall: the confidence interval tells you which of the three you have, and the p-value doesn’t.
Suppose your primary metric is task success, currently around 70%, and the experiment returns no significant difference.
- Interval on the difference: −6pp to +7pp. Useless. The change could plausibly be a strong win or a serious regression. You ran an underpowered test and the only real conclusion is about your sample size.
- Interval on the difference: −0.8pp to +1.1pp. Extremely useful. Whatever this change does, it does less than a point either way — and if a point wouldn’t change your roadmap, the question is answered permanently.
Same verdict. Same p-value, near enough. Entirely different decisions. Any tool that gives you a verdict without an effect size and an interval has thrown away the part of the result you needed.
What a null actually buys you
Reframe it as value rather than consolation, because the value is real:
- It ends a recurring argument. “Should we go back to the shorter system prompt?” has an answer now, with a number, and it stays answered.
- It licenses the cheaper arm. If two variants are equivalent on quality, pick the one that costs less or is simpler to maintain — and you have evidence rather than a preference.
- It redirects effort. You pulled a lever and nothing moved, which means the lever isn’t where the quality is. That’s expensive to learn and cheap compared to learning it in month six.
- It stops you shipping something that felt better and wasn’t. This is the whole point of the apparatus. Most changes that feel better in the playground do nothing at scale, and catching one of those pays for a quarter of experimentation.
None of that is a consolation prize. It’s the thing you were buying.
Four things not to do next
This is where good teams go wrong, and every one of these has an innocent-sounding version:
Don’t go metric-shopping. Checking your other eighteen metrics until one crosses 0.05 is a multiple-comparisons problem with a bow on it. If you’re watching many metrics, a correction across them isn’t optional — it’s the difference between a discovery and a coincidence.
Don’t quietly extend the run. Adding a week because the numbers are “nearly there” means your stopping rule is now “stop when it agrees with me”. A stopping rule changed after seeing data isn’t a stopping rule. When can you call it covers why this inflates both your false positive rate and every effect size you report.
Don’t ship it anyway because it’s obviously better. You may ship it — sometimes there are good reasons, like maintainability or a strategic direction. But say that’s why. “The data was inconclusive so we went with judgement” is defensible; “the data supported it” is not, and the difference matters the next time someone cites this decision.
Don’t slice into twenty segments and report the one that moved. With twenty segments, roughly one crosses 0.05 by chance. The acceptable version of this move: a post-hoc segment is a hypothesis for the next experiment, never a conclusion from this one. Write it down as such, and if you believe it, go and test it properly.
Most nulls are a design problem, not a statistics problem
The uncomfortable diagnosis: if your experiments keep coming back flat with wide intervals, the issue usually isn’t your traffic. It’s that you’re testing changes too small to matter.
Rewording a sentence in a system prompt probably does move quality by less than you can measure at your volume. That’s not a measurement failure — it’s an accurate finding about the size of the change. The fix is to test bigger swings: a restructured prompt rather than a reworded one, a different retrieval strategy rather than a top-k of 6 instead of 5, a model change rather than a temperature change.
Decide the smallest effect worth shipping before you run, and check you have the traffic to detect it. How much traffic do you need has the arithmetic; the short version is that halving the effect you want to catch quadruples the sample you need, so the honest question is usually “is this change big enough to be worth an experiment at all?”
If the answer is no, you’ve saved two weeks. That’s also a result.
LLMJury reports effect size with a confidence interval on every metric rather than a bare significant/not-significant verdict — which is what makes the difference between result 1 and result 2 visible at a glance instead of a calculation you have to do yourself.
An experiment’s job is to make a decision cheaper. “This doesn’t matter, stop working on it” is a decision, and it’s frequently the most valuable one you’ll get all quarter.
Start free, or read writing up an experiment so the decision sticks — because a null that isn’t written down will be re-proposed within two months.