Solutions

A/B testing prompts on live traffic

Run two or more rewrites against each other on real users and get a graded verdict — which one wins, by how much, and whether the number means anything.

Who this is for: Anyone who has picked between two prompts by reading ten outputs and going with the one that felt better.

What most teams do today — and where it stops working

The usual method is a handful of examples in a notebook. Ten outputs can look clearly better while the change quietly makes half your real traffic worse — because the ten you looked at are the ten you thought to try, and your users did not read that list.

The alternative is an offline eval over a curated dataset. That is a real improvement and still answers a different question: curated sets drift away from production traffic, and a prompt can be tuned until it scores well on them without getting better for anyone.

So the change ships on judgement, and at the launch review "I looked at some outputs" is the entire case. There is no number to point at, and no way to know afterwards whether it helped.

How LLMJury does it

Deterministic assignment, no network call

Each user is bucketed by a frozen MurmurHash3 hash computed locally, identical in the Python, TypeScript, and Java SDKs. The same user always sees the same variant, and nothing is added to your request path.

An LLM-as-judge scores a sample

Built-in quality, safety, and relevance grading, plus your own rubric in plain English. Sampled and hard budget-capped, so grading never runs away with your bill, and run after the fact rather than on your users’ path.

Speed, cost, and conversions in the same experiment

Latency and token cost are captured by the SDK wrapper. Your business outcome is the one metric you send by hand. All of it is read next to the quality scores, as one verdict about one change.

A broken test is stopped, not reported

A chi-squared sample-ratio-mismatch check runs continuously. If the observed split diverges from what you configured, analysis halts and says so rather than showing you a result built on broken assignment.

A lucky result does not read as a real one

Every metric × variant comparison goes through Benjamini–Hochberg FDR correction, and both the raw and corrected p-values are reported. The effect size and its 95% confidence interval are on the same screen.

What you would set up

1Name the experiment and add your variantsThe control is what you ship today. Each rewrite is another variant with its own prompt, model, and temperature.
2Declare the metric that decides itOne primary metric, plus guardrails you do not want to regress. Declaring direction and role up front is what stops the result being chosen after the fact.
3Wrap the model callTwo lines: fetch the assigned variant’s prompt, and let the wrapper capture latency, tokens, and errors.
4Send your business outcome, then waitOne `track` call when the thing you care about happens. Then leave it alone until the sample is there — the verdict tells you when it is.

import { Client } from 'llmjury-sdk';

const client = new Client({ experiments: ['checkout-prompt'] });
const llm = client.wrap(anthropicClient, 'checkout-prompt');

// Assignment is a local hash — the same user always lands in the same arm.
const p = client.getPrompt('checkout-prompt', userId,
                           'You are a helpful assistant.');

const res = await llm.messages.create({
  model: 'claude-haiku-4-5',
  system: p.prompt,
  messages: [{ role: 'user', content: userInput }],
});

// The only metric you send by hand: your business outcome.
client.track('business_event', {
  experiment_id: 'checkout-prompt', user_id: userId,
  variant: p.variant, business_metric: 'conversion', value: 1,
});

The whole path, in all three SDKs, is on the five-minute quickstart.

What this does not do

Automatic quality grading with an LLM-as-judge starts on the Pro plan, because judging runs real model inference. On Free you still measure latency, cost, and business outcomes on live traffic.
An experiment needs traffic. How much depends on the effect you are trying to detect — a small effect needs a lot, because the required sample scales with the square of it.
A judge is not a ground truth. It is a consistent scorer applied identically to every arm, which is what makes the comparison valid even where the absolute score is arguable.

Try it on the thing you are arguing about this week

Free plan, no credit card, and the sample experiment is already in your account so there is something to read before you have any data of your own.

Prefer to look first? The interactive demo is a real verdict with no sign-up.