“What’s our hallucination rate?” is a reasonable-sounding question that cannot be answered as asked. Not because it’s hard — because “hallucination” bundles four different failures with four different detection methods, four different fixes, and four different owners.
Teams that insist on one number get one, and then spend the next quarter unable to explain why it moved. Split it first and the measurement problem becomes tractable, one piece at a time.
Four failures wearing one word
| Type | What it is | How you detect it |
|---|---|---|
| Ungrounded claim | The answer asserts something the retrieved context doesn’t support | Ask a judge the narrow question: is every claim here supported by this context? |
| Factual error | The claim is wrong about the world, with no retrieved context to check against | Requires ground truth — a curated set, not production traffic |
| Fabricated specific | Invented citation, invented API method, invented case number, a URL that 404s | Mechanical verification: resolve the URL, look up the id, check the symbol exists |
| Unsupported confidence | The answer is stated flatly and should have been hedged | Judge against a rubric that names calibration — the fuzziest of the four |
The ordering in that table is deliberate. Fabricated specifics are the ones to automate first, because they need no model at all: a regex, a lookup, and an HTTP request will find invented citations and dead links, and those are the failures that most reliably destroy a user’s trust in one sitting. Most teams reach for an LLM judge before they’ve written the ten lines that check whether the cited document exists.
Ungrounded claims come second, because they’re cheap for a different reason: you already have the context. Asking “is this supported by that” is a far easier question than “is this correct”, and a judge does much better at it.
Factual errors need ground truth, which means a curated set — see build your eval set out of production traffic — and unsupported confidence is genuinely hard to score consistently. Leave it until the other three are running.
Groundedness is a claim-level metric
The single most common measurement mistake here is scoring at the wrong grain. An answer with nine supported sentences and one invented one gets marked 90%, which is a number no user has ever experienced. From the reader’s side, that answer is wrong. The invented sentence is the entire problem, and averaging it away is how a metric ends up moving smoothly while the product loses credibility.
Score at the claim level: decompose the answer into individual assertions, check each against the context, then report two numbers, because they answer different questions and teams routinely quote one while meaning the other.
- Share of answers containing at least one unsupported claim. This is the user-facing number. It’s the probability that a given response contains something invented.
- Share of claims unsupported. This is the diagnostic number. It tells you whether a bad answer is bad throughout or mostly right with one flaw.
The first is what you put on a guardrail. The second is what you look at when the first moves and you want to know what changed. Reporting only the second is how a team convinces itself that 3% is fine when in fact a fifth of answers are affected.
The judge is fallible in a correlated way
Now the objection, stated at full strength because it’s a good one: the judge checking groundedness is a language model, and it’s wrong sometimes. Worse, its errors correlate with the generator’s — the same ambiguity in a source document that makes a claim hard to support correctly is the ambiguity that makes it hard to check correctly. This is not an independent referee.
That’s true, and it constrains what you may do with the output. The resolution is the same one that runs through can you trust an LLM to grade an LLM?: a judge that is consistently wrong in the same direction still ranks two arms correctly, because a comparison reads the difference and the shared bias cancels. If arm A shows 4% of answers with an unsupported claim and arm B shows 7%, arm A is better, and the judge’s imperfection doesn’t change that conclusion.
What it does not license — and this deserves to be blunt — is quoting “our hallucination rate is 2.3%” as a fact about your product. That number is a measurement made by an instrument you haven’t calibrated against reality. It’s fine for tracking your own trend. It is not fine in a sales deck, a compliance document, or a board slide, and the distinction between “comparable across arms” and “true in absolute terms” is the one that gets lost the moment a number leaves an engineering channel.
If you do need an absolute figure, there’s exactly one way to get it: hand-label a sample, measure how often the judge agrees with the labels, and report the calibrated figure with its uncertainty. That’s work, and it’s honest work.
Sampling, and what to sample
Judging every response is a second inference bill, often comparable to your first. Sample instead — but sample deliberately, because uniform sampling has a specific blind spot: rare query types disappear. The category that makes up 2% of traffic and 90% of your risk will contribute a handful of graded rows a week, and its regression will be invisible for a month.
Stratify. Sample uniformly for the headline number, and sample a fixed floor from each high-stakes category so those categories have their own series. Then report them separately, because the site-wide average is exactly where a segment falling off a cliff goes to hide — the argument in the silent regression, applied to the metric people are most likely to average.
Keep the series running. The value of a groundedness number is almost entirely in its history: “4%” means nothing on the day you first compute it, and means a great deal on the day it becomes 7%. The teams who catch this early are the ones who started measuring before anything was wrong.
LLMJury’s judge is versioned along with its rubric, and it runs under a hard budget cap you set — so a groundedness metric is a custom rubric metric with a spend ceiling rather than an open-ended bill. Judging starts on the Pro plan; the free plan measures latency, cost, and business metrics.
When it’s a retrieval bug, not a generation bug
One more split, because it decides who fixes it. An ungrounded answer has two possible causes, and they live in different parts of the pipeline:
- The context contained the answer and the model ignored it or embellished past it. A generation problem. Prompt changes, output constraints, and a rubric that penalises unsupported claims will move it.
- The context never contained the answer. A retrieval problem. No amount of prompt engineering fixes it, and every hour spent rewriting the system prompt is wasted.
Distinguish them by measuring retrieval separately: was the answer-bearing chunk in the context at all? That’s the diagnostic split at the heart of A/B testing your RAG pipeline, and it’s the reason a groundedness metric on its own can send a team off in the wrong direction for weeks.
Start with the mechanical checks — dead links and invented ids, this week, no model required. Then claim-level groundedness on a stratified sample. Then the trend line, which is the thing you’ll actually use.
Start free, or see what to actually measure in an LLM product for where groundedness sits among everything else worth tracking.