You shipped thumbs up and thumbs down. The dashboard says 97% positive on a 1.5% response rate, and nobody in the room believes it.
Both numbers are behaving exactly as they should — that’s what a self-selected sample looks like — and neither can tell you whether last week’s prompt change helped.
Why explicit feedback under-delivers
Three problems, and they compound.
The sample selects itself. People click when they’re delighted or furious. The vast middle — where most of your quality actually lives, and where a prompt change actually moves things — never touches the control.
The base rate makes it slow. At 1.5% response, an experiment needs an enormous amount of traffic before the difference in thumbs-up rate between two arms reaches significance. By the time it does, you’ve shipped three other changes.
The click means different things. Thumbs-up can mean “correct”, “polite”, “fast”, “this saved me”, or “I like this product”. Aggregating those into one rate produces a number that moves for reasons you can’t recover.
None of which makes it worthless. A thumbs-down with free text is the highest-information artefact you will ever collect about your product — it tells you what went wrong, in the user’s own framing, on a real request. The right use of explicit feedback is as a qualitative queue you read, not a metric you optimise. Keep the widget. Stop treating its rate as your quality signal.
The signals users give you without being asked
Behaviour is unbiased by willingness to click, available on 100% of traffic, and mostly already in your logs. Here’s the set worth instrumenting, what each plausibly means, and how each misleads.
| Signal | What it plausibly means | How it misleads |
|---|---|---|
| Copied the output | Strong positive — it was usable | Some interfaces make copying unnecessary; absence proves nothing |
| Edited it, then used it | Positive with a defect attached | The richest signal you have: the diff says what was wrong |
| Retried or regenerated | Strong negative | Confounded by curiosity when the button sits next to the answer |
| Rephrased within a minute | Negative — the answer missed | Hard to separate from a genuinely new question; use a time window |
| Abandoned mid-stream | Negative | Could be the answer or the latency — meaningless without latency beside it |
| Escalated to a human | Unambiguous failure in support | Some users escalate by policy regardless of answer quality |
| Completed the task | The strongest signal available | Needs product instrumentation, not LLM instrumentation |
| Returned within seven days | Positive, slow-moving | Driven by far more than this feature; a trend metric, not a test metric |
| Dwell time or scroll depth | Weak engagement proxy | Long dwell can mean “useful” or “confusing”; include it to discount it |
Two of these are worth more than the rest. Edit-then-use is the only signal that tells you what to fix rather than that something needs fixing — collect the diff, sample it weekly, and you have a free error taxonomy. Task completion is the one your business already cares about; if your product has a completion event, wiring it to the experiment is the highest-value afternoon in this whole post.
Validate the proxy once, before you trust it
Every one of those signals is a proxy, and a proxy is an assumption until you check it. The check is cheap and almost nobody does it.
Take a few hundred sessions. Score the outputs — by hand, or with a rubric-driven judge if you have one. Then test whether your signal actually correlates with quality in the direction you assumed.
Regenerate is the cautionary case. If your interface puts the button directly beneath the answer, some fraction of clicks is curiosity rather than dissatisfaction — and if that fraction is large, you’ve built a primary metric that partly measures button placement. You’ll only find out by looking.
An unvalidated proxy shipped as a primary metric is how a team spends a quarter optimising the wrong thing while every dashboard says it’s winning. Validate once, write down what you found, and re-check after any interface change that touches how the signal is produced.
Getting the instrumentation right
Three details decide whether these signals are usable or just present, and all three are cheap at the point you add the event and expensive afterwards.
Attach the signal to the generation, not to the session. Every behavioural event needs the id of the specific response it refers to. Without it you know a user copied something in a session that contained four answers, and you can’t say which — so the signal can’t be attributed to a variant, which was the entire purpose.
Define the window before you look at the data. “Rephrased within a minute” needs a number, and choosing it after seeing which number produces a significant result is p-hacking. Pick from behaviour — how long your users actually take to react — and write it down.
Decide what absence means. A user who did nothing is not the same as a user who explicitly abandoned, and collapsing them makes your rate move whenever traffic mix shifts. Usually you want no action treated as neutral and excluded, with abandonment defined by an observable event rather than by silence.
One more, because it catches people out: these signals live in your product, not in your LLM call.
The instrumentation that matters most is a track call somewhere in the interface layer, and the
team that owns that code is often not the team running the experiment. Sort that out early — it’s an
organisational dependency wearing a technical hat.
Using them in an experiment
Pick one behavioural signal as primary, declared before the run with its direction, and keep the rest as guardrails. The temptation is to watch all nine and call whichever moves a result — but every extra comparison costs power under correction, and what to actually measure in an LLM product has the argument for why a wide metric set makes you less able to detect the effect you care about.
There’s an LLM-specific asymmetry worth knowing: behavioural signals often reach significance before judge scores do. Judge grading is sampled — you’re not paying to grade every response — so your effective quality sample is a fraction of your event count. Copy rate and retry rate are collected on everything, cost nothing per event, and carry no rubric bias. On a small product they’re frequently the only metric that resolves inside a fortnight.
The right combination is usually a behavioural primary with a judge score as the explanatory guardrail: behaviour tells you the change landed, the judge tells you why.
In LLMJury any behavioural or business event goes through client.track(...) with a declared
direction and role — primary or guardrail — and that works on every plan including Free, which has
no judge but does have latency, cost, and whatever your product already emits.
Stop asking users what they think. Watch what they do, validate that it means what you assume, and declare one of those signals before you start. The widget stays for the free text.
Start free, or read the silent regression for what these signals look like when quality is drifting and nothing has broken.