You configured a 50/50 split. Two weeks later the experiment ends: arm A has 48,210 users, arm B has 51,790. About three percent off. Nobody looks twice — randomness is lumpy, everyone knows that.
Randomness is lumpy at a hundred users. At a hundred thousand, a three-percent imbalance is not lumpy. It is a p-value with more zeros after the decimal point than you have fingers, and it means something in your pipeline is dropping users non-randomly. Every number in that experiment is now suspect — including the ones that look fine.
The definition
Sample ratio mismatch (SRM) is when the observed split of users between experiment arms differs from the configured split by more than chance can explain.
The test is a chi-squared goodness-of-fit against the expected proportions. With a 50/50 design and 100,000 total users, the standard deviation of one arm’s count is about 158 — so a ±1,790 deviation sits eleven standard deviations from expectation. That is not a coin landing oddly. That is a bug.
The counter-intuitive part is that SRM gets easier to detect as your experiment gets bigger, which is the reverse of everything else in statistics. Large experiments make small imbalances glaring. A 3% skew on 1,000 users is unremarkable; on 100,000 it is a klaxon.
Why it invalidates everything, not just the split
The instinct is to think of SRM as an accounting nit — arms are uneven, so adjust for it and carry on. That instinct is what makes SRM dangerous, because the imbalance is a symptom, not the problem.
The users who are missing did not vanish at random. Something selected them: a timeout, an error path, a filter, a client that gave up. Whatever that something was, it correlates with the behaviour you’re measuring — slow requests come from users with long inputs, errors come from particular platforms, timeouts come from exactly the traffic where quality is hardest.
So your two arms are no longer samples from the same population. They are samples from two different populations, and the difference between them mixes the effect of your change with the effect of whatever did the selecting. Randomisation is the entire basis for attributing a difference to your variant. SRM is evidence that randomisation didn’t hold.
This is why SRM is a halt signal, not a warning. You don’t interpret an experiment with SRM carefully. You don’t interpret it at all.
What actually causes it
In practice it’s nearly always one of these, and knowing the list turns a mystery into a twenty-minute debugging session:
- Assignment happens after something that can fail. If a user is bucketed downstream of a lookup, a feature check, or a parse that occasionally throws, then failures silently remove users — and not evenly across arms.
- Retries re-bucket. A retried request that generates a fresh identifier gets assigned again, double-counting some users and, if retries correlate with one arm’s behaviour, skewing the split.
- Exposure logged on the wrong side of the call. Log exposure before the model call and you count users whose request then failed; log it after and you lose everyone whose request failed. Either is defensible. Doing it inconsistently, or in a way that interacts with variant behaviour, is not.
- A variant that errors more drops its own events. The unlucky arm produces fewer logged users, precisely because it’s performing worse. The experiment then reports that it performs well, because its worst cases never made it into the data.
- Bots. Crawlers and scrapers hash into arms like anyone else, and their traffic patterns are nothing like your users’. One bad crawl can skew a split.
- Caching or CDN behaviour differing by variant. If one arm is more cacheable, its requests reach your instrumentation less often.
There is one cause that’s specific to LLM products and deserves its own paragraph, because it’s the most common and the most misleading. A slower variant hitting a client timeout. The longer prompt takes an extra 900ms at p95, some clients give up, and those requests never log an outcome. The arm loses its slowest responses — which are disproportionately its hardest questions and its worst answers. The result is that the slow, struggling variant looks better than it is, and the mechanism that made it look better is the same one that made it fail users. If you only take one thing from this post, take that one.
Checking, and what to do about it
Run the chi-squared test on the arm counts, at every analysis, automatically. Manual SRM checks are checks nobody does after week three.
Use a strict threshold: p < 0.001, not p < 0.05. At 0.05 you would flag one healthy experiment in twenty, the alert would become noise within a month, and people would learn to click through it. At 0.001 an alert means something. This is the rare case where the conventional threshold is the wrong one, and it’s wrong because of how humans respond to false alarms, not because of the mathematics.
When it fires, there is exactly one correct response: find the cause and rerun. What you must not do:
- Don’t reweight the arms. Reweighting corrects the counts and leaves the selection bias entirely intact. It makes the symptom disappear and the disease undetectable.
- Don’t trim the larger arm to match. You’d be dropping users at random to compensate for users that were dropped non-randomly. The bias survives.
- Don’t “note it as a caveat” and present the results. A caveat on a slide does not un-confound an estimate, and the number will be quoted without the caveat by Thursday.
Losing two weeks of experiment time hurts. Shipping a change based on a broken comparison hurts for longer, and you won’t know it happened.
The tooling position
LLMJury runs the chi-squared SRM check as a hard gate and halts analysis at p < 0.001 rather than rendering numbers it doesn’t believe. That’s a deliberate product decision: a result that quietly displays alongside a small yellow triangle is a result someone will screenshot without the triangle.
If you’re building experiment analysis yourself, add the check first — before the effect sizes, before confidence intervals, before anything. It’s a dozen lines of code, and it’s the only part of the pipeline that can tell you the rest of the pipeline is lying.
For the definitions of the other terms in that sentence, the glossary is short and plain. For why the correction on p-values matters once the SRM gate passes, see when you’re allowed to call a result.
Start free — the SRM gate runs on every plan, including the free one.