The average is hiding the answer: segmenting LLM experiments

Your prompt change came back flat. Then someone slices by query type, and it’s clearly better on complex questions and clearly worse on simple ones.

Nothing was happening, according to the average. Two things were happening, and they cancelled.

Heterogeneous effects, and why averages hide them#

The concept, stated so you can quote it: a treatment effect is heterogeneous when the change helps some users and hurts others — and an average across both reports their sum, which can sit at zero while nobody’s experience is unchanged.

LLM products are unusually prone to this, and the reason is structural. A prompt is a single set of instructions applied to a distribution of very different requests. Tell the model to be more thorough and you help the user with a complicated question and irritate the one who wanted a yes. Tell it to be concise and you invert exactly that. Almost every prompt change is a trade between segments; the only question is whether you measured which ones.

That makes “no significant difference” a suspicious result rather than a boring one. Before you accept it, look at whether you’re averaging a win and a loss. Your experiment came back flat covers the other two things that verdict can mean.

The paradox that reverses the sign#

Cancellation is the ordinary case. The alarming one is when the average doesn’t just hide the effect but inverts it. Simpson’s paradox, worked with round numbers:

Check the arithmetic — it holds. B beats A in the simple segment. B beats A in the complex segment. Pooled, A wins by twenty-eight points.

The mechanism is composition. Pooling weights each segment by how many of that arm’s users landed in it, and here arm A’s traffic is mostly easy questions while arm B’s is mostly hard ones. You’re not comparing prompts, you’re comparing question mixes.

Two things to take from this. First: the overall split is a perfect 1,200 against 1,200, so a top-level sample ratio check passes. The imbalance is entirely inside the segments. Run your sample ratio mismatch check per segment as well as overall, because this is exactly the pattern it exists to catch.

Second: in properly randomised assignment this shouldn’t happen, which means when it does, the finding isn’t “interesting nuance” — it’s a bug. Something is assigning by a variable correlated with query type. Find it before you interpret a single number.

The segments worth declaring in an LLM product#

Pick a handful before the run. These are the ones that plausibly respond differently to a prompt or model change:

• Query type or intent. The big one. A prompt tuned on troubleshooting behaves differently on billing questions, and most products have three or four intents that barely resemble each other.
• Input length. Short inputs give a change less to act on; long ones may hit truncation or context limits that shorter ones never reach.
• New vs returning users. Novelty effects live here, and so do different expectations — a returning user has learned to phrase things the way your old prompt liked.
• Language. Quality gaps between languages are large, systematic, and almost never separately measured. If you serve more than one, this is the segment most likely to hide a real regression.
• Customer tier, if you’re B2B. The accounts paying you most may behave nothing like the median, and an average that ignores them can green-light a change that annoys your revenue.
• Surface or device, where latency or output rendering differs materially.

Time of day and day of week belong on the list too, but as a diagnostic rather than a segment — they’re how you spot that one arm ran mostly during a weekend, not a population you’d ship differently for.

Declare them first, or they’re not results#

Here’s the discipline that keeps this honest, and it’s not optional: pre-declared segments produce results; post-hoc segments produce hypotheses.

Slicing after the fact until something is significant is p-hacking with extra steps. With twenty segments and no correction, roughly one crosses p < 0.05 by chance alone — so “it worked for enterprise users on mobile” is, on those odds, exactly what you’d expect from a change that does nothing at all.

You are still allowed to look. Looking is how you generate the next experiment. What you’re not allowed to do is present a post-hoc slice as a finding, and the fix is one line in the write-up: this segment was not pre-declared and needs its own test.

Segmentation also costs power twice over. Each segment is a smaller sample, so the detectable effect inside it is larger; and each comparison adds to the correction burden across the analysis. Three declared segments on your primary metric is usually the right budget. Twelve is a wish. How much traffic do you need has the arithmetic on what splitting your sample does to the effect you can detect.

LLMJury applies Benjamini–Hochberg correction across metric × variant comparisons for exactly this reason — a wide analysis with no correction manufactures discoveries, and the correction is what keeps a lucky slice from reading as a real one.

A split result is usually the most useful outcome#

The instinct on discovering heterogeneity is disappointment: the clean answer got complicated. It’s the opposite. A change that wins on complex queries and loses on simple ones is telling you something a uniform result never could — the answer isn’t A or B, it’s a router.

Classify the request, send the complex ones to the variant that helps them, keep the simple ones on the prompt that was already working. That’s a better product than either arm, and you only get to it by having measured the segments. The same logic underpins cost routing in cutting LLM costs without cutting quality: most traffic is easy, and treating all of it identically is what leaves value on the table.

So: declare three segments before you start, check the split inside each of them, treat any sign reversal as a bug until proven otherwise, and read a split result as a design brief rather than a disappointment.

Start free, or see what to actually measure in an LLM product for choosing the primary metric you’ll be segmenting in the first place.