You’re testing a new system prompt for a support assistant. Two weeks, ten thousand conversations, sixty thousand turns. You score helpfulness per turn, run the comparison across sixty thousand observations, and get a confidence interval so narrow it’s almost a point estimate. Ship it.
Don’t. That narrow interval is the symptom, not the reward — and the mistake behind it is quiet enough that it produces confident wrong answers rather than obviously broken ones.
The unit-of-analysis error
The unit-of-analysis error is analysing at a finer grain than you randomised at.
You assigned conversations to arms. So a conversation is one observation, and you have ten thousand of them — not sixty thousand. The six turns inside a conversation are not six independent draws: same user, same problem, same context, same tone, often the same underlying misunderstanding being worked through. If the assistant handles that user’s problem badly on turn two, it is very likely to be handling it badly on turn five.
Treating correlated observations as independent inflates your effective sample size. Inflated sample size shrinks the standard error. A shrunken standard error narrows the confidence interval and drives the p-value down. The end result is significance the data does not support.
Say the consequence plainly, because this is what makes it dangerous: this error doesn’t make your experiment noisy, it makes it overconfident. Noise looks like noise — wide intervals, no verdict, everyone knows to wait. False precision looks exactly like an answer, and it gets shipped.
The fix is boringly simple
Aggregate to the unit you randomised at, then test across those units.
One number per conversation. If your metric is a per-turn judge score, take the conversation’s mean score. If it’s format compliance, take the share of compliant turns, or whether any turn failed — whichever matches the decision you’re making. Now you have ten thousand independent observations, and every standard test behaves correctly on them.
That’s it. Cluster-robust standard errors and mixed-effects models exist, they’re more efficient, and they’re the right tool when you need turn-level covariates in the model. For the overwhelming majority of teams, aggregate-first is correct, simple, and — the property that matters most — hard to get subtly wrong at 6pm on a Thursday.
The one thing you lose is turn-level diagnosis, and you don’t have to lose it: keep the turn-level data for looking at, and use the conversation-level data for deciding. Diagnostic and decision metrics are different jobs, the same split that runs through what to actually measure in an LLM product.
Which metrics belong at which level
| Level | Metrics | Why here |
|---|---|---|
| Turn | Format compliance, per-turn latency, tool-call validity, per-response judge score | Properties of a single response — diagnostic, and aggregated upward before testing |
| Conversation | Resolved without escalation, turns to resolution, abandoned mid-conversation, total cost, total latency | The randomisation unit, so this is where the decision metric lives |
| User | Return rate, repeat usage, retention, subscription | Slowest and noisiest, but the only layer that says whether the product got better |
If you randomise by user rather than by conversation — which you should, if a user can start several conversations — then the user is the unit and conversation-level metrics aggregate upward the same way. The rule generalises: whatever you hashed to pick an arm is what one row means.
Turns to resolution is a trap
One metric on that table needs its own warning, because it is the most commonly chosen and the most commonly misread.
Fewer turns is good when the problem was solved, and good news is not what it means when the user gave up. A variant that frustrates people into leaving after two turns will show a beautiful improvement in turns-to-resolution, and it will look like efficiency.
So the metric is meaningless on its own. It must be paired with a resolution signal — escalation rate, an explicit resolved flag, a follow-up contact within 24 hours, anything that distinguishes “done” from “gone”. Read as a pair: turns down and resolution flat or up is a genuine win; turns down and resolution down is a product that got worse and a metric that congratulated it.
The same caution applies to conversation length in the other direction. A longer conversation is not engagement. In a support context it’s usually failure, and in an assistant context it’s ambiguous. Neither direction is self-evidently good, which is why direction has to be declared for each metric before the run rather than interpreted afterwards.
Assignment: by conversation or user, never by request
If a user flips arms between turn three and turn four, they experienced neither variant. They got a blend, their conversation data is uninterpretable, and both arms are contaminated by it. On a multi-turn product this is the single most damaging implementation bug, and it’s easy to introduce — any assignment keyed on a per-request id does it.
Bucket on something stable: a conversation id, a session id, a user id. LLMJury computes assignment as a deterministic MurmurHash3 of that unit id, locally, with no network call — so every turn of a conversation lands in the same arm, in every process, on every machine, without any coordination.
Two related failure modes to check for while you’re there. If a conversation can outlive a config change, decide what happens to conversations in flight when a variant is edited. And if your identifier resets — a new session id per page load, say — you’ve randomised at page-load level whether you meant to or not, and your unit of analysis just changed underneath you.
If arm counts come out uneven, don’t rationalise it. Uneven splits on a multi-turn product usually mean assignment is happening at the wrong grain, and the consequences reach much further than the split — see your experiment is lying to you.
Duration has to outlast the conversation
Multi-turn products break the usual analysis-window advice. A conversation started on Friday may finish on Monday. If your analysis window closes on Sunday, Friday’s conversations are counted half-finished, and “half-finished” is not randomly distributed — long, difficult conversations are exactly the ones that straddle the boundary.
So: run long enough that the great majority of conversations begun inside the window also end inside it, and exclude conversations that started before the experiment did. The general advice in when can you call it? — at least one full week, preferably two — becomes a floor rather than a target once conversations take days to resolve.
The rest of the discipline is unchanged: one primary metric, declared direction, guardrails that can veto a win. It’s only the arithmetic underneath that has to respect what you randomised.
Aggregate to the unit you assigned. Everything else follows from that.
Start free — assignment is sticky per unit on every plan.