Nothing fired. Error rates are flat, latency is fine, every dashboard is green, and the on-call rotation has been quiet for two months. Then someone notices that retention on the AI feature has been sliding since roughly June, asks when it started, and gets silence — because forty prompt edits, two library upgrades, and one provider model refresh happened in that window and nobody can say which of them did it.
That’s the shape of it. Not an outage. A slow leak with no instrument pointed at it.
Why your monitoring can’t see this
Every tool in your observability stack was designed to catch things that break. Exceptions, 5xxs, timeouts, failing assertions. An LLM quality regression breaks nothing. The response comes back on time, correctly formatted, fluent, and confident. It’s just less useful than it was.
There’s no exception to catch, no test to fail, no threshold to alert on — because “helpful” isn’t a field in the response. And your users won’t report it either. Nobody files a ticket saying “the answers are eight percent less helpful this week.” They rephrase the question, get a mediocre answer again, and quietly stop using the feature. The signal arrives as a churn number, one or two quarters late, with all the attribution gone.
This is the failure mode that makes LLM products different from ordinary software. The blast radius is the same as an outage. The detection story is nothing like it.
Six causes, and how you’d actually see each one
- Accumulated prompt edits. Each individual change was fine — reviewed, sensible, an improvement in the case it was written for. Forty of them later the prompt is a different document with rules that contradict each other. Detection: test each change against the current version, so the drift is measured one step at a time rather than discovered in aggregate.
- Provider-side model updates. Models get updated behind stable names. The version serving your traffic today is not the one that served it in spring, and no release note maps onto your product’s behaviour. Detection: a permanent control arm on a pinned configuration. When quality moves on both arms, it wasn’t you.
- Retrieval decay. In RAG systems the index grows, near-duplicates accumulate, and the same top-k that used to return the answer-bearing chunk now returns five plausible neighbours. Detection: monitor retrieval quality separately from answer quality — recall of the relevant chunk is a different metric from whether the final answer is good, and only the first one tells you where the rot is.
- Traffic drift. Your users start asking different questions: new segment, new use case, new language, a competitor’s users arriving with different habits. The prompt was tuned for the old distribution. Detection: quality segmented by query type and cohort over time, not a single average.
- Context growth. Conversations get longer than the prompt was designed for, the system prompt drifts toward the middle of a very long context, and instruction-following degrades in ways that only show up on long sessions. Detection: quality bucketed by conversation length.
- Dependency changes. A tokenizer upgrade changes where you truncate. A template library changes whitespace handling. A framework release reorders message roles. Detection: honestly, the control arm again — this class is too varied to instrument individually.
Notice how many of these have nothing to do with anything your team shipped. Roughly half of that list can degrade your product while your repository sits untouched.
Averages are where regressions hide
One structural warning before the fix. A single site-wide quality average is remarkably good at concealing exactly the failure you’re trying to catch.
If quality for 80% of traffic is unchanged and quality for one segment — the newest cohort, the long-conversation users, the second language you added — falls off a cliff, the average moves by a rounding error. You will look at the chart and conclude everything is fine, and the segment that’s leaving is the one you were counting on for growth.
Segment the time series from day one: by query type, by user cohort, by conversation length, by whatever dimension your product actually varies along. It costs nothing extra at collection time and it’s nearly impossible to add retroactively to data you didn’t keep.
What actually works
Three things, in order of effort, and each one buys a different capability.
A continuous judge sample gives you a time series. Grade a small share of production responses against a fixed, versioned rubric, every day, forever. That converts “quality seems worse” into “quality dropped on 14 July” — which is the difference between a suspicion and an investigation. The rubric has to be versioned, or you’ll be comparing measurements, not products; the reasoning is in can you trust an LLM to grade an LLM, and it applies doubly here because the whole value is in comparing across months.
A permanent holdout arm gives you a baseline. A small slice of traffic pinned to a known configuration, running indefinitely. Without it, when quality drops you cannot distinguish “we broke something” from “the ground moved”. With it, that question is answered by looking at two lines instead of one.
Per-change experiments give you attribution. This is the one that matters most, and the reason is simple: knowing quality fell is worth far less than knowing what caused it. If every prompt change, model swap, and retrieval tweak ran as an experiment, then the day quality drops you have a list of changes each with a measured effect, and the culprit is usually obvious. Without that, you have a date and forty suspects.
The cost, stated honestly
None of this is free. Judge sampling is inference you’re paying for. A holdout arm means a slice of your users are deliberately on the older configuration. Running every change as an experiment means changes take days rather than minutes, which is a real tax on velocity and the objection you’ll actually hear in the room.
Weigh it against the alternative, which is not “no cost” — it’s a quality slide discovered two quarters late, with no ability to bisect, and a rebuild of trust in a feature users have already written off. The instrumentation is cheaper. It’s just paid up front, in visible increments, which is the hardest kind of budget to defend and the easiest to regret cutting.
LLMJury runs judge grading on a sample of live traffic with versioned rubrics, keeps a control arm as an ordinary variant, and stamps the configuration version onto every event — so “what was live on 14 July” is a lookup rather than an archaeology project. Quality grading starts on Pro; latency, cost, and your own business events are measured on every plan.
Start with the time series. You can add the rest later, but you cannot go back and collect the months you didn’t measure.
Start free — no credit card — or read why offline evals won’t catch this.