Anatomy of an experiment

The assistant that answers questions during checkout runs on one prompt, and the team disagreed about it. Half thought it was too long; someone had read that bulleted answers convert better; someone else wanted it friendlier. Three opinions, no evidence, and a decision that gets made either way.

The useful reframe is that a prompt change is a release. The question is not “which of these reads better?” — it is “which of these should the next 12,048 customers get?”, and that is a question with an answer.

4 arms: the prompt in production as the control, and one rewrite per argument. Not a grid of every phrasing — every extra arm splits the traffic, so an arm nobody would ship is a week of sample spent on nothing.

• control (detailed)control25% of traffic

  You are a helpful checkout assistant. Answer thoroughly, covering shipping, returns, and payment options where relevant…

• concise25% of traffic

  You are a checkout assistant. Answer in at most three sentences. Lead with the answer, then one line of context…

• bulleted25% of traffic

  You are a checkout assistant. Answer as a short bulleted list, one fact per bullet, most important first…

• warm25% of traffic

  You are a friendly checkout assistant. Acknowledge the concern before answering, and close by offering further help…

All four run the same model (claude-haiku-4-5) at the same settings, so the only thing that differs between arms is the words. If the model varied too, a difference could not be attributed to either.

4 metrics, each with its direction and its role fixed in advance. That ordering is the whole point: a metric chosen after the numbers are in is not a measurement, it is a story. One is primary — the thing the experiment is about — and the rest are guardrails that must not get worse.

Exactly one of these needs code from you. Quality is judged, cost and response time are intercepted by the wrapper, and conversion is the one only your application knows about.

The control scores 3.41 on answer quality. A rewrite worth shipping should move that by something like a fifth of a point — smaller than that and it is not worth the churn of changing a production prompt. At four arms and that effect size, the arithmetic lands around three thousand exposures per arm.

This step is the one teams skip, and skipping it is how an experiment ends in “it looks better but it is not significant yet” three weeks running. Required sample scales with the square of the effect you are chasing, so halving the effect you want to detect quadruples the wait. Deciding that before you start is what makes the end date real — the sample-size post has the tables.

Two lines on the request path. Assignment is a local hash, so nothing here adds latency or depends on us being reachable — and the in-code default is what your users get if we are not.

from llmjury import Client

client = Client(experiments=["checkout-assistant-prompt"])
llm = client.wrap(anthropic_client, "checkout-assistant-prompt")

# The assigned arm's prompt. Your current prompt is the fallback, so the first
# deploy of this line changes nothing about what anyone sees.
p = client.get_prompt("checkout-assistant-prompt", user_id,
                      default=CHECKOUT_PROMPT)

answer = llm.messages.create(
    model="claude-haiku-4-5",
    system=p.prompt,
    messages=[{"role": "user", "content": question}],
)

# The one metric only your application can measure.
client.track("business_event", {
    "experiment_id": "checkout-assistant-prompt", "user_id": user_id,
    "variant": p.variant, "business_metric": "conversion", "value": 1,
})

The same thing in TypeScript and Java is on the quickstart.

24 days, from 2026-06-09. By day three the concise arm was ahead on quality and it would have been very easy to call it — and that is exactly the mistake. Checking repeatedly and stopping the first time a number crosses 0.05 inflates the false-positive rate well past the 5% the number claims to describe. The result you get that way is the one you went looking for.

The permissible early stop is the opposite one: a guardrail moving badly enough that you would abandon the change regardless of what the primary metric does. Nothing did. When can you call it? is the longer version.

Configured at 25/25/25/25, observed at 3,012 / 3,012 / 3,012 / 3,012. The chi-squared sample-ratio-mismatch check returns p = 0.62, comfortably clear of the 0.001 gate, so the analysis runs.

Had it failed, LLMJury would have halted and told you, rather than showing a verdict built on a broken split. That is not a formality: a mismatch usually means assignment is broken or one arm is losing users before they are counted, and no amount of correct statistics downstream repairs it. See SRM.

“concise” wins — better answers, cheaper, and it converts. 3 of 4 metrics moved significantly across 12,048 judged events. Every p-value below is corrected across all 12 comparisons in the experiment (4 metrics × 3 rewrites), because asking twelve questions and reporting the best answer is how a coincidence gets published.

The distribution says the same thing in a way an average cannot: 61% of concise’s judged answers scored 4 or 5, against 49% of the control’s. That is a real shift in the shape, not a mean dragged by a handful of outliers.

Two results here are more instructive than the win.

Response time did not survive correction. Concise is −52 ms faster with a raw p of 0.025 — significant, if you look at that metric alone. After correcting across all 12 comparisons it is 0.067, and the honest reading is “probably faster, not shown”. That is the difference between the number you would have reported and the number that is true, and it is the reason the roll-up says 3 of 4 rather than all four.

“warm” converted better without answering better. It moved conversion by +6.7% (0.005) while answer quality sat at +0.44% and did not clear significance at all. A single-metric experiment would have shipped it as a win, or as a failure, depending entirely on which metric someone happened to pick — which is the argument for declaring the whole set in step 3.

Promote concise to control. No deploy: the prompt lives in LLMJury, so the change is a version pointer moving, and moving it back is the same click. The experiment keeps the version each arm ran, the configuration, the verdict, and who approved it — so the next time someone asks “why does the assistant answer like that?”, the answer is a link rather than a memory.

The warm result stays open as its own question: it converts, and nobody knows why, which is a better experiment than the one just finished.