Most of this blog assumes production traffic. If you have forty users and a demo next week, a two-week experiment at 80% power isn’t advice — it’s a joke at your expense.
So let’s be direct: you can’t A/B test yet. That is not the same as “you can’t measure”, and the gap between those two sentences is where a lot of pre-launch products quietly get worse without anyone noticing.
What each stage can and can’t tell you
| Stage | Users | What you can measure | What you genuinely cannot | Build this now |
|---|---|---|---|---|
| Prototype | 0 | Format validity, refusal, cost per request, latency, gross failure | Preference, satisfaction, anything about real distribution | A fixed input set, versioned |
| Alpha | Tens | The above, plus rubric-scored quality on real inputs | Any statistical comparison worth the name | Runtime prompt resolution |
| Beta | Hundreds | The above, plus behavioural signals and large effects on cheap metrics | Small effects on quality; anything needing segments | Assignment and an event with a user id |
| Production | Thousands | Everything the rest of this blog is about | Effects below your minimum detectable size | The habit of declaring a metric first |
The “cannot” column is the honest half. At prototype stage you cannot measure preference — you have no users to prefer anything — and a team that convinces itself otherwise ships a feature tuned to its own imagination.
What actually works with no traffic
In rough order of value per hour spent:
A fixed input set, re-run on every prompt change. Twenty examples you keep beat two hundred you write once. The asset is the repetition, not the size: the same twenty inputs run against every version turn “I think that’s better” into “these three outputs changed and here’s how”.
Regression testing rather than scoring. With no baseline, “is this good?” is unanswerable, but “did this change break something that used to work?” is answerable today. Diff the outputs between versions and read the diffs. It’s manual, it’s ten minutes, and it’s the single highest-value habit at this stage — most pre-launch quality loss is a fix for one case silently breaking another.
Assertions before judgement. Format validity, required fields, length bounds, refusal, forbidden content — all checkable in code, no model required, and they catch most of what breaks. When the JSON stops parsing covers why “valid” is five separate claims and which of them actually hurts.
Adversarial inputs, early. Empty input, a single word, ten thousand words, the wrong language, contradictory instructions, an injection attempt. These cost an afternoon now and a Saturday later.
Cost and latency from day one. Neither needs a single user, and they’re the constraints most likely to force a redesign. Far better to discover the architecture is unaffordable before it has customers than after.
Human review with a rubric, five outputs at a time. Small, but with anchored levels it’s real data rather than an impression. How to write a judge rubric applies identically whether the scorer is a model or you on a Tuesday afternoon.
Two substitutes for traffic, and what each is worth
You will reach for both of these. They’re useful, and both have a failure mode worth naming before you lean on them.
Your colleagues. Dogfooding gets you real interaction with a real interface, which is more than any offline set gives you — and it systematically over-samples people who know how the product works. Your team phrases requests the way the prompt likes, because they wrote it. They don’t ask the confused question, the half-question, or the question about something you don’t do, and those are exactly the inputs that expose a brittle prompt. Use dogfooding to find crashes and rough edges, not to estimate quality.
Model-generated inputs. A model will happily invent two hundred plausible user questions, and they’re a reasonable starting corpus when you have nothing. But they’re generated from the same prior your product’s model reasons with, so they share its blind spots by construction — the cases it finds hard to handle are the cases it finds hard to imagine. They also come out uniformly well-formed, and real inputs are not: they have typos, missing context, three questions in one message, and text pasted from somewhere else.
The honest summary: synthetic inputs test that the machinery works. Only real ones test whether the product does. Replace them the week you have traffic — every one of them, not the ones you happen to remember.
Instrument now, analyse later
This is the part that pays for the whole post.
The expensive thing about experimentation isn’t the statistics — the statistics arrive with whatever tool you pick. The expensive thing is the plumbing: deterministic assignment, an event carrying a stable user id, and a prompt that’s addressable at runtime rather than compiled into a deploy.
Put that in at forty users and your first real experiment is a config change on the day you finally have the traffic for it. Skip it and you’ll be retrofitting assignment logic under deadline pressure, during the launch that made it necessary, on the code path you least want to touch.
The runtime-prompt half looks like this:
from llmjury import Client
client = Client(experiments=["onboarding_prompt"])
llm = client.wrap(anthropic_client, "onboarding_prompt")
# One variant today. `default` is what runs if LLMJury is unreachable — so this
# adds no hard dependency to your request path, even pre-launch.
p = client.get_prompt("onboarding_prompt", user_id, default="You are a helpful assistant.")
response = llm.messages.create(
model="claude-haiku-4-5", system=p.prompt,
messages=[{"role": "user", "content": user_input}],
)One variant is fine. The point isn’t the experiment, it’s that the prompt now has a version, a history, and a rollback that doesn’t involve a deploy — which is worth having at any scale. Get your prompts out of your codebase makes that case in full.
When does an experiment start being worth it?
Sooner than most people assume, and it depends entirely on the metric.
Cost and latency resolve first. They have low variance, they’re measured on every single request, and the differences between a terse prompt and a verbose one are large. A few thousand requests is often enough to call an operational difference — which means the first genuine experiment a small team can run is usually not about quality at all.
Quality needs much more, because judge scores and success rates are noisier and the effects are smaller. And there’s a multiplier people miss: judging is sampled, so your effective quality sample is a fraction of your request count.
The rule of thumb worth internalising: required sample scales with the square of how small an effect you want to catch, so a small product can only detect large effects — which is fine, because a small product should only be making large changes. How much traffic do you need has the table.
LLMJury’s free plan exists for exactly this stage: prompt versioning, assignment, latency, cost, and whatever business events you already emit, with no judge. The plumbing goes in before the traffic arrives, which is the entire point.
Build the fixed input set this week. Diff your outputs on every change. Put the prompt behind a runtime lookup while the codebase is small enough that it takes an hour. The statistics can wait until you have users; none of that can.
Start free, or read build your eval set out of production traffic for what to do with those twenty examples once real requests start arriving.