You have a prompt experiment running. Someone wants to test a retrieval change. The cautious instinct is to queue it behind the first one.
Queueing feels rigorous. It’s also how an experimentation programme dies: at two weeks per test run back to back, you get twenty-six decisions a year, and twenty-six decisions a year is not a practice, it’s a ritual.
The default answer is yes
Here’s the rule, and it’s the fragment worth remembering: overlapping experiments are safe when assignment is independent and the changes don’t interact. Independent randomisation makes each experiment’s arms balanced with respect to the other’s, so the other experiment becomes noise rather than bias.
Work through why, because the intuition is what stops the worry recurring. Experiment A is running 50/50. Experiment B starts, assigning users independently. Half of B’s control group is in A’s treatment; half of B’s treatment group is in A’s treatment. Whatever A does to quality, it does equally to both of B’s arms — so when you subtract one from the other, A cancels.
That’s the whole mechanism. It’s the same reason you don’t need to control for the weather, the day of the week, or which users happen to be sad on Tuesday. Randomisation handles anything it’s independent of.
The cost isn’t bias, it’s variance. A’s effect makes B’s measurements noisier, which costs a little power. On any metric with a decent sample that’s a rounding error next to the cost of waiting a fortnight.
When it genuinely breaks
Four cases, and only the first is common:
Shared surface. Two experiments editing the same system prompt are not two experiments. They’re four untracked variants, and neither analysis knows about the other’s text. This is the real collision in LLM products, because the system prompt is the single most-edited artefact in the codebase and everyone wants to touch it.
Genuine interaction. A change that only works in the presence of another — a larger context budget and a retrieval change that fills it. Each experiment then measures an average over the other’s arms, which may correspond to no configuration you would actually ship. If you suspect interaction, that’s an argument for a single experiment with the combination as an arm, not for queueing.
Shared capacity. Both variants pushing token spend, rate limits, or a shared cache means one experiment changes the other’s operating conditions rather than just its metrics. Latency experiments are especially prone to this.
A thin sample on both. Not bias, just arithmetic: if both experiments are already marginal on power, the extra variance can push both to inconclusive. Two nulls in three weeks is worse than one clear answer in two.
The one technical requirement: different salts
This is the part people get wrong, and it silently reintroduces exactly the confounding they thought they were avoiding.
If both experiments assign by hashing the same user id with the same function and no distinguishing input, they produce the same split. Everyone in A’s treatment is in B’s treatment. You now have two perfectly correlated experiments and no way to attribute anything to either — with a result that looks completely normal, because the split is 50/50 and every check passes.
The fix is to include something per-experiment in the hashed key: a salt, the experiment id, or
both. Then the two assignments are independent and the cancellation argument above holds. In LLMJury
the bucketing key is salt:user_id:experiment_id hashed with MurmurHash3, so two concurrent
experiments cannot correlate by construction — and the salt is pinned for the life of an experiment
version, because rotating it mid-run re-buckets everyone and destroys the comparison.
Worth checking on whatever you’re using today, including a homegrown one. It’s a two-line bug that invalidates months of results.
When you need layers, and when to serialise
For the cases that genuinely conflict, two heavier tools:
Layers — mutually exclusive groups. Each user is assigned to one layer, and experiments within a layer never share a user. Two experiments on the same system prompt go in the same layer, and each gets half the traffic. Straightforward, and honestly priced: you’re buying isolation with sample size.
Serialising — run one, then the other. Reserve this for changes that touch identical text and can’t be sensibly combined into one experiment with three arms. It’s the most expensive option and it should feel like it.
The default order is: overlap → layer → serialise, and most teams start at the wrong end.
How many is too many?
There’s no statistical ceiling — with independent assignment you could run fifty. The real limits are organisational, and they arrive in this order.
Surface contention comes first. Long before the maths complains, you run out of distinct things to change. Most LLM products have one system prompt, one retrieval config, and one model choice per feature, and three experiments on three different surfaces is already most of what exists to test.
Then attention. Every live experiment needs someone watching its guardrails, and an experiment nobody is watching is a change shipped to a fraction of users with no owner. If you can’t name the person reading each one, you have too many.
Then debuggability. With eight experiments live, an incident investigation starts with disentangling eight allocations. That’s survivable with a registry and unpleasant without one.
The practical shape for a small team: two or three concurrent experiments, on different surfaces, each with a named owner and a declared end date. That’s five to ten times the decision rate of a strict queue, and it stays comprehensible.
Keep a registry — one row per experiment with its surface, allocation, owner, start and end. It takes a minute per experiment and it’s the artefact that makes everything above possible.
What to keep track of
Two operational notes that only bite later.
Correction across a programme, not just an experiment. With five experiments live and four metrics each, you’re making a lot of comparisons. Benjamini–Hochberg correction within each experiment is the floor, not the ceiling — and a shared guardrail view across all live experiments matters much more than it does when you run one at a time, because a regression caused by an interaction shows up as “something is wrong” before it shows up as “experiment C is wrong”.
A timeline of what was live when. Six months from now, someone will ask why quality stepped up on the 14th. Without a record of which experiments were running, at what allocation, that question has no answer — and writing up an experiment so the decision sticks is where that record belongs.
Also: check sample ratio mismatch per experiment, every time. Two concurrent experiments give you two chances for an assignment bug, and a correlated split is one of the ways SRM shows up.
Default to overlapping. Use a different salt per experiment. Layer the ones that share a surface. Serialise almost nothing — and stop treating the queue as the safe option, because its cost is real, it’s just not on any dashboard.
Start free, or read when can you call it for the stopping rules that keep several concurrent experiments from turning into several concurrent opportunities to peek.