What to actually measure in an LLM product

There are two ways to fail at measuring an LLM product, and they look nothing alike from the inside.

The first is measuring nothing: quality is whatever the last person to read the outputs says it is. The second is measuring forty things: a dashboard with quality scores, token counts, thumbs ratings, latency percentiles, sentiment, hallucination rates, and no agreement about which one decides anything. Both end in the same place — the loudest person in the room decides — but the second one takes six weeks longer to get there and comes with a dashboard.

What follows is the shortest complete answer I know to “what should we actually measure”.

Four layers

Read it top to bottom and you have the trade: the metrics you care about most move slowest and are noisiest, and the metrics that resolve fastest are the furthest from what you’re trying to achieve. That tension is the whole design problem, and the four-layer split exists so you can use the fast layers as leading indicators for the slow ones instead of choosing between them.

Quality

The most direct measure and the hardest to get right. A judge model scores outputs against a written rubric; format compliance and grounding can often be checked programmatically, which makes them cheaper and more reliable than anything scored by a model.

The trap: judge scores are trustworthy as a comparison between arms and untrustworthy as an absolute number. The reasoning is in can you trust an LLM to grade an LLM, and the practical consequence is that “variant B scored 0.3 higher” is a finding, while “our product scores 4.2/5 on quality” is a number you should not put on a slide.

The second trap: refusal rate belongs here and is routinely forgotten. A model that declines more looks safe to a safety rubric and useless to a user.

Operational

Latency, cost, tokens, errors. Every request produces them, they need no sampling and no judge, their variance is low, and they resolve in days on modest traffic. They are the cheapest information you will ever buy.

The trap: averages. Mean latency is a number nobody experiences. Users feel the p95, and a change that improves the mean while worsening the tail is a change that makes your product feel worse to the people most likely to complain about it. Track percentiles or don’t bother.

Behavioural

The most under-used layer, and the one that most often changes people’s minds. Did the user retry? Rephrase the same question — which is a near-perfect dissatisfaction signal, and free to collect? Edit the output before using it, and by how much? Copy it? Abandon the session mid-answer?

These are proxies for satisfaction that don’t require asking anyone anything, and they capture what a judge structurally cannot see: whether the answer worked for this person, in this task.

The trap: explicit feedback — thumbs up and down — is the version of this layer everyone builds first and the weakest one. Response rates are tiny and wildly self-selected, skewed toward the delighted and the furious. Implicit signals cover all your traffic. Collect both, weight the implicit.

Business

Conversion, retention, expansion, support ticket volume, revenue. What the company actually cares about, and the honest reason they’re last rather than first.

The trap: these are so noisy, and so far downstream, that a single prompt change will almost never move them detectably. That doesn’t mean don’t measure — it means don’t expect a prompt-level experiment to resolve on retention, and don’t let “it didn’t move revenue” be the standard a quality improvement has to clear. You need the layers above precisely because this one can’t answer on a two-week timescale.

The discipline: one primary metric

Pick one primary metric per experiment. Declare it before the run, along with which direction counts as better. Then choose a small set of guardrails — metrics that can veto a win but can’t create one.

That last distinction does more work than it looks. A guardrail is asymmetric on purpose: if quality improves and cost is flat, ship. If quality improves and cost doubles, the guardrail vetoes regardless of how good the primary number looks. Writing them down before the run is what stops the post-hoc conversation where whichever metric happened to move becomes the one that mattered all along.

And there is a mathematical reason to keep the list short, not just a cultural one. Every additional metric is another comparison, and controlling the false discovery rate across all those comparisons raises the bar on every one of them. Watching more metrics makes you less able to detect the effect you actually care about. Twenty metrics is not twenty times the insight; it’s a smaller chance of confirming the one hypothesis you had. If you’re short on traffic — and most teams are more short than they think — a focused metric set is the cheapest power you can buy.

Segment, or the average will lie to you

A site-wide average hides the trade you would have refused. Improve quality 5% for short factual questions and lose 15% on long multi-step conversations, and depending on your traffic mix the average may barely move — so you ship a change that guts your most engaged use case and the dashboard congratulates you.

Segment by query type, user cohort, and conversation length at minimum. This has to be decided before you collect, because segmentation you didn’t record is segmentation you can’t recover. It’s also how a silent regression gets caught early instead of showing up in a churn report.

Making it concrete

LLMJury takes the declared direction and role for each metric — primary or guardrail, higher-is- better or lower-is-better — and routes each one to a test appropriate to its category: permutation tests with bootstrap confidence intervals for continuous, ordinal, percentile, and count metrics, a two-proportion z-test for binary rates with a permutation fallback at small samples. Quality comes from a versioned judge, operational metrics come from wrapping the model call, and behavioural and business events come from your own track calls. Benjamini–Hochberg correction runs across every metric×variant comparison, so the guardrail set can’t manufacture a winner.

The point of all four layers is that no single one of them is the answer. Quality tells you the output is better; operational tells you what it costs; behavioural tells you the user agreed; business tells you it was worth doing. Read them together and most experiment decisions make themselves.

Start free — no credit card — or click through a real verdict with no sign-up.