How much traffic do you need to A/B test a prompt?

“We don’t have enough traffic to A/B test” kills more experimentation programmes than any other sentence. It is usually wrong, and it’s wrong for a specific and fixable reason: the number in people’s heads came from a different problem.

You’ve read that a web A/B test needs 100,000 users per arm. That figure is real — for detecting a half-percent lift in checkout conversion, where the effect is tiny and the noise is large. A prompt change that moves answer quality by eight percent is not that problem. It is not close to that problem.

The only formula intuition you need

Required sample size scales with two things: how noisy your metric is, and — this is the part that surprises people — the square of how small an effect you want to detect.

Halve the effect you care about, and you need four times the traffic. Cut it to a fifth, and you need twenty-five times. That single relationship explains every number below, and it explains why “how much traffic do I need?” is unanswerable until you’ve said what you’d actually ship.

Which means the question you should ask first is not how big is my traffic but how small an improvement is worth shipping. If a 2-point gain wouldn’t change your roadmap, don’t build an experiment capable of finding one.

The actual numbers

Below: samples per arm, at 80% power and α = 0.05, two-sided, for a two-arm test. The right-hand column assumes 1,000 requests a day split evenly, so each arm sees 500 a day. Scale it linearly for your own volume — ten times the traffic, a tenth of the calendar.

Two caveats on reading this table honestly. The variability figures are typical, not yours — a judge rubric with a wider spread of scores, or a product where some users paste whole documents and others type six words, will need more. And percentile metrics like p95 latency are noisier than the p50 row suggests; budget several times that sample if the tail is what you’re deciding on.

But look at the shape of it. Most of these rows are days, not quarters. The 100,000-per-arm figure appears nowhere, because nothing here is trying to detect a one-percent move in a rare event.

Four levers when traffic is genuinely tight

Measure the cheap metrics first. Cost and latency have low variance and no sampling step, so they resolve in days on almost any traffic. They’re also frequently decisive on their own: if the new prompt costs 30% more and grades the same, you have your answer, and you got it in 48 hours.

Pick one primary metric. Every additional metric you watch is another comparison, and correcting for multiple comparisons — which you must do, or your “significant” results are mostly false — costs statistical power. Twenty secondary metrics don’t give you twenty insights. They give you a higher bar on the one thing you actually cared about.

Run two arms, not five. A four-arm test is not four times harder than a two-arm test, it’s worse than that: you’re both splitting traffic four ways and making more comparisons. Test the one change you believe in. Permutations are for when you have traffic to burn.

Be honest about the minimum effect worth shipping. This is the largest lever by far, because of the squaring. Moving your target from 2pp to 5pp cuts the required sample by a factor of six. Most teams have never written the number down, so they implicitly power for “any difference at all”, which is a sample size of infinity.

Small samples aren’t zero information

The last argument is the one that should actually get you to run the thing. Suppose you’re underpowered and the experiment returns no significant difference. That result is not nothing. It says: whatever this change did, it wasn’t large. Read the confidence interval — if it spans −2pp to +3pp, you’ve learned the change is not the big win someone claimed it was, and you can stop arguing about it and move on.

Compare that to the alternative, which is one person reading ten playground outputs and forming an impression with error bars that are not merely wide but invisible. An underpowered experiment beats a confident guess, because it comes with a measurement of its own uncertainty.

What you should not do is stare at an underpowered experiment daily and ship the moment it crosses significance. That converts a modest sample problem into a systematic bias — the subject of when you’re allowed to call it.

Where the tooling helps

Sample size only becomes a real answer if the analysis is honest about it. LLMJury reports the effect size and its confidence interval rather than a bare pass/fail, so an inconclusive result looks inconclusive instead of looking like a tie. It routes each metric to a test appropriate to its category — permutation tests with bootstrap intervals for continuous and ordinal metrics, a two-proportion z-test for rates, with a permutation fallback when samples get small — and corrects across every metric×variant comparison so a wide metric set can’t manufacture a winner.

One last thing worth knowing before you plan: judge grading runs on a sample of traffic, so your effective quality sample is smaller than your event count. Cost and latency see every request. Plan the duration around whichever metric is your primary one.

So run the arithmetic before you conclude you’re too small. Pick the effect that would change your mind, look up the row, and count the days.

Start free — no credit card — or see which metrics are worth the power budget.