Every time someone proposes a fixed 50/50 split, somebody in the room asks why you’d deliberately send half your traffic to the worse variant for two weeks. It’s a fair question with a good answer, and the good answer is not “because that’s how experiments work”.
LLMJury runs fixed splits, so treat everything below as an argument made by an interested party. The concession comes first because it’s real.
What a bandit actually does
Strip the casino metaphor. A multi-armed bandit is an algorithm that adjusts traffic allocation while the test is running, moving share toward whichever arm is performing better on a reward signal, while keeping some exploration so it can notice if it was wrong.
The appeal is straightforward and it is not marketing: if variant B really is worse, a fixed split serves it to half your users for the full duration. A bandit serves it to a shrinking fraction. The difference is measured as regret — the cumulative cost of the traffic you spent on the losing arm — and minimising regret is a legitimate objective, especially when the losing arm is genuinely bad for the people who get it.
If your honest goal is “stop serving the worse thing as fast as possible”, a bandit does that better than an experiment does. Concede that fully, then look at what it costs.
Optimise or explain — you get one
A bandit optimises. An experiment explains.
At the end of a bandit run you know which arm the algorithm settled on. What you don’t get is a clean answer to “how much better was it, with what uncertainty, and for whom”. The reason is structural rather than a gap in anyone’s tooling: the sample sizes are endogenous — how much data each arm has is itself a consequence of how well it was doing — and the assignment probability changes over time, so early observations and late observations aren’t drawn from the same design. The resulting effect estimates are biased, and un-biasing them requires methods most teams won’t implement and can’t easily explain to a stakeholder.
Which matters or not depending on the question you actually have. If the decision is “route each request to the best of these five options, forever”, you don’t need an explanation, you need a good router. If the decision is “ship one of these two prompts and delete the other”, you are buying an explanation, and the number you’ll be asked for in the review is precisely the one a bandit is worst at producing.
The comparison
| Fixed-split experiment | Bandit | |
|---|---|---|
| Optimises for | Learning — a clean estimate of the difference | Cumulative reward during the run |
| Traffic to the losing arm | Fixed for the whole duration | Shrinks as evidence accumulates |
| What you have at the end | An unbiased effect size with a confidence interval | A converged allocation, and a biased estimate |
| Segment results | Available — every segment has a designed sample | Unreliable — small segments may have almost no data on the losing arm |
| Non-stationarity | The run covers a fixed window; drift after it is visible as a new result | Converged allocations go stale silently unless exploration continues |
| Delayed feedback | Fine — the analysis happens at the end | Poor — the algorithm updates on stale information |
| Operational complexity | Low: assign, log, analyse | Higher: a reward loop, a policy, and monitoring for all of it |
| Use when | The decision is permanent, feedback is slow, or you need to defend the result | Arm choice is the product, feedback is fast, or serving a bad arm is expensive |
Why bandits fit LLM products badly
This is the argument that decides it for most teams here, and it’s the one almost nothing on the open web makes.
A bandit needs fast feedback, and LLM quality metrics are slow. The algorithm works by updating its allocation on a reward signal. That requires the reward to arrive shortly after the action. Click-through on a headline is ideal: the answer comes back in seconds, it’s binary, and every impression produces one.
Now consider what a reward looks like on an LLM feature. Quality comes from a judge that runs on a sample of traffic — because grading everything is a second inference bill — and often on a batch, minutes or hours later. Conversion lands a day or two afterwards. User satisfaction, if you get it at all, arrives from a self-selected minority. So the bandit spends most of its life allocating traffic on the basis of information about a period that has already ended.
That isn’t a tooling gap you can close with a better implementation. It’s the shape of the feedback loop. A bandit whose reward is delayed and sparsely sampled is, in practice, a fixed split that occasionally lurches.
There’s a partial exception worth naming: your fast, dense metrics — latency, cost, format compliance, and any immediate behavioural signal like a copy or a retry — do arrive quickly enough to drive a bandit. If the thing you’re optimising is genuinely one of those, the objection above doesn’t apply to you.
What to actually do
Use a fixed split when the decision is permanent, when you need a defensible number, when feedback is slow or sampled, or when you need per-segment results. That covers most prompt changes, most model migrations, and every change someone will be asked to justify.
Use a bandit when arm selection is itself the product — per-request model routing, choosing among many prompt templates by query type — when you have many arms and a genuinely fast reward, or when serving a bad arm is costly enough that you’d rather converge than understand.
And consider the hybrid nobody mentions, because it’s usually the right answer: fixed split to learn, router to exploit. Run the experiment, get the estimate with its interval, understand which segments differ, and then encode what you learned as a routing rule that runs forever. You measure once and exploit indefinitely, and you keep the explanation. The routing-by-difficulty idea in cutting LLM costs without cutting quality is exactly this pattern.
The sting: a bandit still needs a control
One thing a bandit does not remove, and this is the failure mode to watch for. Once it converges, the winning arm is being trusted, not measured. Providers update models behind stable names, traffic drifts, and the conditions that made arm B better in March may not hold in July. A converged allocation reports no result, so nothing tells you it went stale.
That’s the mechanism described in the silent regression, and it’s an argument for keeping a holdout — a small permanent slice on the old configuration — regardless of which method you choose. Bandits don’t exempt you from it; if anything, their quietness makes it more necessary.
Whichever way you go, the discipline in when can you call it? is the same: know what would change your mind before you start. LLMJury runs fixed splits, reports effect size with confidence intervals, and corrects across metrics — which is a statement about what it’s for, not a claim that bandits are a mistake.