The experiment ran. The result was clear. The meeting happened, and the decision went to whoever spoke with the most confidence.
Then six weeks later somebody re-proposes the losing variant, because nothing that survived that meeting was written down — and the person who ran the experiment has left, or forgotten, or is now arguing the other side.
The output of an experiment is not a number
It’s a decision, plus the record that makes the decision re-examinable. A result nobody can reconstruct in six months isn’t evidence, it’s a memory, and memories lose arguments to whoever is more certain.
This matters more for LLM products than it does elsewhere, for a specific reason: your prompts
accumulate. In a year, the system prompt contains forty decisions, and nobody can say why any
particular sentence is in it. The write-up is what stops that — it’s the git blame for judgement
rather than for text.
The template
Seven sections. Steal it.
1. The decision, first. One line at the top: what you’re doing and why. Everybody reads the first line; almost nobody reads section four. “Shipping variant B to 100%: +3.4pp task success (CI 1.1 to 5.7), no guardrail regression.”
2. The question, and what you pre-registered. Primary metric with its direction, the guardrails, the smallest effect worth shipping, planned duration. Written before the run and quoted unchanged. A write-up that pre-registers after the fact is a story, not a record — and everyone can tell, because a pre-registration written after the data always happens to match the data.
3. What was actually different. The exact prompt version identifier, model, and configuration
per arm. Not a description. “We made it more concise” is not reproducible; support_prompt@v14 is.
This is the section future-you will actually need.
4. The numbers. Effect size with a confidence interval, per metric, primary first. Sample size and the observed split. The SRM check result. Significance alone is not a result — an experiment that came back flat needs the interval to be interpretable at all, and so does one that came back positive.
5. Segments and guardrails. Pre-declared segments reported as results. Any post-hoc slice labelled explicitly as a hypothesis rather than a finding — see segmenting LLM experiments for why that line matters.
6. What you said you’d do under each outcome. Written before you saw them, including the null. This is the section that turns a debate into a decision, because the argument about what counts as a win happened when nobody knew who would win.
7. Limits, honestly. Which population, which window, which judge version, which metrics were too noisy to read. A write-up that lists its own limitations is trusted more, not less — and it stops the result being over-quoted six months later by someone who wasn’t there.
Saying it to people who don’t run experiments
Three rules, and the third one is the one people break.
Give the effect in the units the business uses. Not “0.34 standardised effect” — “roughly three in every hundred support conversations resolve that wouldn’t have”.
Express uncertainty as a range. “Between 1 and 6 points better, most likely around 3.” People handle ranges fine. What they can’t handle is a point estimate presented as a fact, and then a different number next quarter.
Never say “statistically significant” to an audience that will hear “large”. Those two words mean “probably not zero”, and a stakeholder hears “big”. On a large sample you can get a statistically significant effect that is commercially meaningless, and reporting it in the language of significance sets up a promise the product will not keep.
For a null, the sentence that does the most work is: “we can rule out a difference bigger than X.” It converts an apparent non-answer into a bounded fact, and it’s usually the most useful thing you’ll say in the meeting.
Who writes it, and when
Two practical points that decide whether any of this happens.
The person who ran it writes it, and writes most of it before the result arrives. Sections two, three, and six — the pre-registration, the configuration, and the pre-agreed actions — are all knowable on day one. Draft them when you set the experiment up and the write-up costs twenty minutes at the end instead of two hours nobody has. It also removes the temptation the whole template exists to resist: you cannot rationalise a pre-registration you wrote before the data.
Write it for the null too. The strongest predictor of whether a team has a functioning experiment practice is whether flat results get documented at the same length as wins. They almost never do, and that asymmetry is exactly how the archive turns into a highlight reel — every entry a success, no record of what didn’t work, and the same failed idea proposed again next year by someone who searched and found nothing.
A null takes ten minutes to write up, because sections two through six are already drafted and section one is one line: no detectable difference, staying on A, we can rule out anything larger than 2pp.
Where it lives
Next to the experiment, not in a slide deck. A deck is written for one meeting and is unfindable afterwards; the write-up needs to be discoverable by someone who wasn’t in the room and doesn’t know the experiment’s name.
The two questions that recur, forever, are:
- Why is the prompt like this?
- Did we ever try the other thing?
Both are answered by an archive and by nothing else. And once you’re running several experiments at a time, the archive needs to record what was live when — otherwise “why did quality step up on the 14th?” is unanswerable, which is one of the operational costs discussed in running two experiments at once.
LLMJury keeps configuration, verdict, and analysis history against each experiment, with exportable results and an audit trail of who changed what — which gives the write-up something durable to point at rather than a screenshot pasted into a document.
The incentive problem
One uncomfortable note, and it isn’t cynicism.
If shipping is the only outcome that gets rewarded, then every write-up will find a reason to ship. Not through dishonesty — through a thousand small readings that all lean one way: the segment that looked good, the secondary metric that moved, the “directionally positive” framing of an interval straddling zero.
The check is simple. A team that can point at a null it acted on — a change it built, tested, and didn’t ship — has a functioning process. A team where every experiment won has a process that produces conclusions rather than tests.
Write the decision in the first line. Quote the pre-registration unchanged. Report the interval. Say what you don’t know. It takes twenty minutes, and it’s the difference between an experiment that decided something and an experiment that happened.
Start free, or read when can you call it for the pre-registration that section two is supposed to be quoting.