Chunk size is 512 because a blog post said 512. Top-k is 5 because 5 was the default. There’s a reranker in the pipeline because adding one seemed like it should help, and nobody has removed it to check.
This is not a criticism of your team. It’s the normal state of every RAG system I’ve seen, because the pipeline has eight knobs, they interact, and the only feedback anyone gets is reading a few answers and forming an impression. The good news is that RAG is unusually rewarding to test properly — the knobs are cheap to change and the effects are often large.
Why RAG is harder to test than a prompt
A prompt change has one place to go wrong. A RAG pipeline has four, and end-to-end quality can’t tell you which one fired.
When an answer is bad, it’s because retrieval didn’t return the right chunk, or it returned it and ranking buried it below noise, or ranking was fine and the context window truncated it away, or everything upstream worked and the generation prompt didn’t use what it was given. All four produce the same symptom: a confident, fluent, wrong answer.
So measure the pipeline in two places, not one:
- Retrieval metrics are diagnostic. Was the answer-bearing chunk retrieved at all (recall@k)? Where did it rank? How much of the retrieved context was irrelevant? These don’t decide anything on their own, but they tell you which stage to go fix.
- End-to-end quality is the decision metric. It’s what you ship on. It’s also silent about causes.
Teams that only track end-to-end quality can tell that a config is worse and can’t tell why, so tuning becomes trial and error at experiment cadence — which is slow and expensive. Teams that only track retrieval metrics optimise recall@k into the ground and are then surprised when answers don’t improve, because stuffing more marginally relevant chunks into context makes generation worse. You need both, and they answer different questions.
The knobs
| Knob | Trades against | How to test it |
|---|---|---|
| Chunk size | Precision vs. complete context | 2–3 sizes as arms; watch recall@k and answer quality |
| Chunk overlap | Boundary safety vs. index size, duplicate hits | Sweep with recall@k; check for near-duplicate retrievals |
| Top-k | Recall vs. noise, cost, latency | Arms at k and k±2; quality often peaks then declines |
| Embedding model | Retrieval quality vs. a full re-index | Expensive to reverse — shadow-index and evaluate offline first |
| Reranker | Precision vs. added latency and cost | On/off arm; measure the latency it costs, not just quality |
| Hybrid vs. pure vector | Keyword precision vs. pipeline complexity | On/off arm; segment by queries containing IDs, names, codes |
| Query rewriting | Recall vs. an extra model call in the critical path | On/off arm with latency as a guardrail |
| Context formatting | Nothing obvious — which is why it goes untested | Same chunks, different presentation, as two arms |
Three of those rows deserve their own paragraph.
Top-k is not monotonic. The intuition that more context can’t hurt is wrong, and measurably so: irrelevant chunks degrade answers. They dilute attention, introduce plausible-but-wrong material the model may prefer to the right material, and push the actually-relevant passage further from the question. Quality against k usually rises, peaks, and falls. Nobody finds the peak by reasoning about it.
Hybrid search matters more than the vector-database marketing suggests. Embeddings are excellent at meaning and mediocre at exact strings. If your users search for error codes, SKUs, person names, version numbers, or API method names, pure vector search will miss things that a keyword index finds instantly. This is one to segment on: the aggregate effect can be modest while the effect on identifier-bearing queries is enormous.
Context formatting is the most under-tested knob in the entire pipeline. The same five chunks, presented in a different order, with or without source labels, with or without separators, joined into prose versus listed as documents, produce measurably different answers. It costs nothing to change and almost nobody runs an experiment on it. Put relevant chunks where the model attends best, label sources so it can attribute, and test the arrangement rather than assuming.
You cannot test the grid
Eight knobs at three settings each is 6,561 configurations. You do not have the traffic. Nobody has the traffic — this is not a “small team” problem, it’s arithmetic.
So: change one thing per arm. It gives you attribution, which is the thing you’re actually short of. The honest cost is that one-at-a-time tuning misses interactions — chunk size and top-k genuinely interact, since smaller chunks mean you need more of them — and you should know that’s a real limitation rather than pretend otherwise. Where you suspect a strong interaction, test the two or three combinations you actually believe in as named configurations, rather than sweeping.
And spend your traffic in the right order. Retrieval-side changes (chunking, k, hybrid, reranking) usually move quality more than generation-side polish, and are cheaper to reverse than an embedding model swap. Do the cheap, high-variance knobs first.
Sizing matters here too: a four-arm chunk-size sweep splits traffic four ways and makes more comparisons, which raises the bar under false-discovery correction. Work out what that costs you before you design a sweep — two arms and a decision usually beats four arms and an inconclusive result.
A RAG config decays
The property that makes RAG different from a prompt: it degrades on its own, without anyone touching it.
The index grows. Near-duplicates accumulate. The top-k that used to surface the one relevant document now surfaces five plausible neighbours from documents added last quarter. Nothing in your repository changed and retrieval quality fell anyway — which is the textbook shape of a silent regression, and why retrieval metrics belong in a permanent time series rather than in a one-off tuning exercise.
Re-run the k and reranker experiments after significant index growth. The optimum moves.
It’s just a variant
Mechanically, none of this is special. A retrieval configuration is an arm: assign users deterministically, serve them the config, record retrieval metrics and answer quality and latency and cost, read the verdict. The same shape as testing a prompt or a model, with more instrumentation upstream of the model call.
LLMJury doesn’t care what your variant is — it assigns users, grades a sample of outputs against
your rubric, accepts your own retrieval metrics through track, and returns an SRM-gated,
FDR-corrected verdict with cost and latency beside quality. The pipeline stays yours; the
comparison stops being guesswork.
Pick the knob you’ve never justified — for most teams that’s top-k or context formatting — and make it prove itself.
Start free — no credit card — or read the five-minute quickstart.