When can you call it? Peeking, p-hacking, and stopping an experiment

Day two of the experiment. Variant B is up 6% on quality, the p-value reads 0.03, and someone posts a screenshot with three rocket emojis. Shipping now would save a week and everyone would like that very much.

Don’t. Not because a week of patience is virtuous, but because that p-value doesn’t mean what the screenshot thinks it means — and the size of the win almost certainly doesn’t either.

What peeking actually does

A p-value is calibrated for a single look at a sample size you committed to in advance. That’s the contract. “There’s a 5% chance of seeing an effect this large if there were no real effect” is a statement about one draw.

Check every morning for two weeks and you’ve taken fourteen draws. Each one is another chance for noise to wander across the line, and you’ve pre-committed to acting on whichever one does. Under repeated daily testing at the nominal 5% level, the probability that at least one look crosses the threshold on a genuinely null experiment climbs to roughly one in five — four times the error rate you think you’re running. Look more often, on a dashboard that refreshes hourly, and it gets worse.

So stopping the moment a result looks significant means you ship noise regularly, and — worse, because it’s invisible — you’ll never notice, since the shipped changes don’t announce themselves as false positives. They just quietly fail to deliver the improvement you booked.

There’s a second effect that stings more once you see it. Because you stopped precisely at a favourable fluctuation, the effect size you report is inflated. You didn’t just possibly catch a phantom; when there is a real effect, you systematically overstate it. This is the winner’s curse, it’s why so many shipped wins fail to show up in the aggregate numbers later, and it’s the mechanism behind every “our A/B tests said +40% but revenue is flat” story.

What to do instead, in order of realism

1. Fix the duration or sample size in advance, and honour it. Boring, free, completely effective. Write down “we run until 8,000 samples per arm or two weeks, whichever is later” before the experiment starts, and the peeking problem evaporates because your decision doesn’t depend on when you looked.

2. Look, but don’t decide. “Never peek” is advice nobody follows, and pretending otherwise just moves the peeking somewhere undocumented. The error is not looking — it’s letting the look determine the stop. Monitoring for breakage is responsible engineering. Watching the p-value and stopping when it dips is the thing that breaks the statistics. Make the distinction explicit on your team, because “don’t look at the dashboard” will be ignored and “don’t stop on the dashboard” can actually be followed.

3. Use a sequential test if you genuinely need to stop early. Group sequential designs and always-valid confidence sequences exist precisely for this, and they’re the right tool when the cost of running long is real. The honest caveat: they aren’t free. You pay for the option to stop early with reduced power, so the same experiment needs more total traffic than the fixed-horizon version would have. Adopt them because you need early stopping, not because they sound more sophisticated.

4. Stop early on guardrails, always. A real quality collapse, an error-rate spike, a refusal rate that doubles — kill the arm immediately. That is not peeking, it’s safety, and the asymmetry is deliberate: you need strong evidence to ship a change and only weak evidence to stop hurting users. Nobody’s statistical conscience should be troubled by pulling a variant that’s on fire.

Duration is a business question, not just a statistical one

Even with a sample size that satisfies the arithmetic, there are minimums that come from how your product is used rather than from how many observations you have.

Run at least one full week. Weekday and weekend traffic are different populations asking different questions — dramatically so for B2B products, where Saturday users are practically a different market. An experiment that ran Tuesday to Thursday measured Tuesday-to-Thursday users. Two full weeks is better, because it lets you check the first week’s result held in the second.

Watch for novelty effects with returning users. A visibly different response style gets engagement for being new. If your users come back, the first days overstate what you’ll see in month two.

Remember your quality sample is smaller than your traffic. Judge grading runs on a sample of requests, not all of them, so the metric that usually matters most is also the one accumulating observations slowest. Cost and latency see every request and resolve in days; quality lags. Plan the duration around your primary metric, not your busiest one. This catches people out constantly.

The five lines to write down before you start

Before the first user is assigned:

1. The primary metric, singular, and which direction counts as better.
2. The guardrails that can veto a win regardless of the primary metric.
3. The smallest effect worth shipping — the number that makes this change worth the risk and the deploy.
4. The planned duration or sample size, and the date you’ll read the result.
5. What you’ll do for each outcome — including the null.

The fifth one is the one that gets skipped, and it’s the one that turns an experiment into a decision instead of a discussion. If you haven’t agreed in advance what “no significant difference” means for the change, you will spend the meeting relitigating whether the experiment was fair rather than deciding anything. Write it down: if it’s flat, we keep the simpler version and move on.

One more reason to be strict about the threshold

If you’re watching a primary metric plus six guardrails across two variants, that’s fourteen comparisons, and at the nominal 5% level you’d expect roughly one to look significant by luck alone. Correction for multiple comparisons — Benjamini–Hochberg controlling the false discovery rate — is what stops a broad metric set from manufacturing a winner every time. LLMJury applies it across every metric×variant comparison and reports both the raw and corrected p-values, so you can see what the correction cost you rather than just receiving its verdict.

The SRM gate runs first, before any of this, because there’s no point correcting p-values on an experiment whose arms aren’t comparable.

None of this is about being cautious for its own sake. It’s that an experiment you stop at the convenient moment is a more expensive way of doing exactly what you were doing before: shipping the version you already liked and finding a number to agree with you.

Start free — no credit card — or read how much traffic your effect actually needs.