Skip to content

Frameworks & runtimes

LLMJury + LlamaIndex

The interesting RAG experiment is rarely the prompt — it is top_k, the chunk size, the reranker. Put the whole retrieval configuration behind one variant key and test the permutation.

⌘ Key-only — you send the metrics

The short answer: Works on the key-only path, and it is the natural fit — a RAG experiment varies retrieval settings, which are configuration values, not a client to intercept.

How it works

RAG changes are the ones teams argue about longest and measure least, because the thing being changed is not a prompt you can eyeball. Whether top_k=5 beats top_k=3, whether the reranker earns its latency, whether 512-token chunks beat 1024 — none of that is visible in ten sampled outputs, and all of it moves answer quality.

get_variables(…) returns the variant’s whole configuration merged over your in-code defaults, so one call gives you every retrieval knob at once. You build the query engine from those values. The permutation is what is under test, not any single setting — which is the right unit, because these interact.

Assignment is by user and it is sticky, so someone who asks three questions in a session gets one configuration for all three. That matters more in RAG than almost anywhere else: a user bounced between retrieval configurations mid-session produces an experience neither arm is responsible for.

Send the outcome the way you already judge a RAG answer — the user accepted it, they did not re-ask, they clicked the citation. If you want faithfulness graded automatically, send the retrieved context and the answer as a model_call event and put a groundedness rubric on the metric.

What it looks like in your code

from llama_index.core import VectorStoreIndex
from llmjury import Client

client = Client(experiments=["rag-retrieval-config"])

# the whole retrieval configuration for this user's arm, in one call
v = client.get_variables("rag-retrieval-config", user_id,
                         defaults={"top_k": 3, "chunk_size": 512, "rerank": False})

engine = index.as_query_engine(
    similarity_top_k=v.values["top_k"],
    node_postprocessors=[reranker] if v.values["rerank"] else [],
)
answer = engine.query(user_input)

client.track("business_event", {"experiment_id": "rag-retrieval-config", "user_id": user_id,
                               "variant": v.variant, "business_metric": "answer_accepted", "value": 1})

The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.

What gets measured

  • Whatever outcome tells you the answer was good — acceptance, no re-ask, a clicked citation
  • End-to-end latency, if you send it: reranking costs milliseconds and they belong in the comparison
  • Judge-scored faithfulness and relevance, when you send the retrieved context alongside the answer
  • Exposures per arm and the sample-ratio-mismatch gate

What this does not do

  • We do not index, embed, retrieve, or store your documents. LLMJury holds the configuration values and the events you send, not your corpus.
  • No automatic instrumentation of a LlamaIndex pipeline — no retrieval traces, no per-node timings. Those are a tracing tool’s job.
  • Retrieval quality metrics that need ground-truth labels (recall against a known answer set) are an offline eval, and an offline eval is a different tool from this one.

Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.

Point it at LlamaIndex and see

Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.

Free plan · no credit card required

Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.

Other frameworks & runtimes