Solutions

Tuning parameters and configuration

Temperature, max tokens, retrieval settings, chunk size — test the whole permutation behind one variant key instead of arguing about defaults.

Who this is for: Teams whose retrieval and generation settings were chosen once, early, by whoever was there — and have never been measured since.

What most teams do today — and where it stops working

Chunk size is 512 because a blog post said 512. Top-k is 5 because 5 was the default. Temperature is 0.7 because it is always 0.7. Each of those is a decision that was never made, and together they are most of what determines the output.

Tuning them by hand runs into two walls. The settings interact — raising top-k with a small context window makes things worse, not better, and top-k is not even monotonic — so one-at-a-time tuning finds a local answer. And the effect of any single knob is usually small enough that eyeballing cannot see it at all.

How LLMJury does it

A variant is the whole configuration

Prompt, model, temperature, max tokens, and whatever else you route behind the variant key — the arm is the permutation, so interactions are inside the thing you are measuring rather than outside it.

Compare more than two arms at once

Run the control against three or four configurations in the same experiment. Every arm is scored against the control on every metric, and the correction accounts for the fact that you asked several questions.

The knobs that only you can route

Retrieval settings, rerankers, and chunk sizes live in your code, not ours. The SDK gives you the assigned variant key deterministically; branching on it is one `if`, and the measurement is identical either way.

Small effects, correctly sized

Parameter changes usually move things a little. The results view reports the effect size with a confidence interval and tells you when the sample is too small to distinguish it from nothing — instead of calling it either way.

What you would set up

1Write down the permutations worth testingNot every combination — the three or four you would actually ship. Every extra arm costs sample.
2Route on the variant keyFor settings LLMJury holds, read them from the variant. For settings in your own pipeline, branch on `client.assign(...)` and configure your retriever accordingly.
3Pick one decision metric and keep the diagnostics separateEnd-to-end quality decides it. Retrieval-level numbers are diagnostic — useful for understanding why, wrong for choosing.

try (LlmjuryClient client = LlmjuryClient.builder()
        .experiments("retrieval-config").build()) {

  // Deterministic assignment — no network call. Branch your own pipeline on it.
  String variant = client.assign("retrieval-config", userId);
  RetrieverConfig cfg = RETRIEVER_CONFIGS.get(variant);

  // ... run retrieval with cfg, then call your model as usual

  client.track("business_event", Map.of(
      "experiment_id", "retrieval-config",
      "user_id", userId, "variant", variant,
      "business_metric", "resolved", "value", 1));
}

The whole path, in all three SDKs, is on the five-minute quickstart.

What this does not do

Every arm splits the traffic, so four configurations need roughly twice the sample of two to detect the same effect. Test the permutations you would ship, not the grid.
LLMJury does not run your retriever or store your embeddings. It assigns and measures; the pipeline stays yours.
Plan limits cap how much statistical compute one experiment can use (permutations and samples per arm). The defaults suit almost everyone; very large experiments are what the higher tiers are for.

Try it on the thing you are arguing about this week

Free plan, no credit card, and the sample experiment is already in your account so there is something to read before you have any data of your own.

Prefer to look first? The interactive demo is a real verdict with no sign-up.