Your first LLM A/B test, end to end

All three SDKs are open source under Apache-2.0 and behave identically — the same bucketing hash, the same non-blocking event buffer, the same 24-hour offline replay.

• Python

  pip install llmjury-sdk

  Copy

  Python 3.8+ · imports as llmjury

• TypeScript

  npm install llmjury-sdk

  Copy

  Node 18+ or the browser · ESM + CJS

• Java

  implementation "com.llmjury:llmjury-sdk:0.1.0"

  Copy

  Java 11+ · Maven Central

Create a free account, then open Settings → API keys and issue a key. Use the publishable key (llmj_pk_…) in your app — it can only assign, track, and read config, so it is safe in client-side code. Secret keys (llmj_sk_…) are server-side only.

Every SDK reads the key from an environment variable, so it never lands in your repository:

export LLMJURY_API_KEY=llmj_pk_...

If your platform injects secrets another way, pass it to the constructor instead: Client(api_key=…), new Client({ apiKey: … }), LlmjuryClient.builder(…). With no key found, the client fails at construction and tells you so — it never fails silently at request time.

In the dashboard, create an experiment with two variants (control and treatment is enough to start) and pick your metrics. Give it a memorable name like checkout-prompt — that name is how your code addresses it, so there are no UUIDs to paste around.

Each variant’s prompt, model, and temperature live in the dashboard, not your repo. That is the point of the whole loop: changing a prompt after this step never needs a deploy.

Two calls on the request path. get_prompt resolves this user’s variant from client memory — a local hash, no network call, no added latency — and wrap() intercepts your provider client so every call records latency, tokens, and errors without a single line at the call site.

from llmjury import Client

# ---- once, at startup -------------------------------------------------------
client = Client(experiments=["checkout-prompt"])        # prefetch by experiment NAME
llm = client.wrap(anthropic_client, "checkout-prompt")  # every model call is now traced

# ---- per request ------------------------------------------------------------
# The variant prompt comes from client memory; your in-code default survives an outage.
p = client.get_prompt("checkout-prompt", user_id,
                      default="You are a helpful assistant.")

response = llm.messages.create(
    model="claude-haiku-4-5", max_tokens=1024,
    system=p.prompt,
    messages=[{"role": "user", "content": user_input}],
)

# The only metric you send by hand: your business outcome.
client.track("business_event", {
    "experiment_id": "checkout-prompt", "user_id": user_id,
    "variant": p.variant, "business_metric": "conversion", "value": 1,
})

The default you pass is the prompt your app uses if LLMJury is unreachable or you have run out of free-tier events. Nothing here blocks, and nothing here throws into your application.

Latency, cost, and token counts come from the wrapper. Quality comes from the judge on paid plans. The one thing only you can measure is whether the user actually did the thing — the conversion, the purchase, the accepted suggestion. That is the track call in the sample above, and it is the reason a verdict can say “better answers and more conversions” rather than just “higher score”.

Events are buffered and flushed in the background, and replayed for up to 24 hours if the network is down — so a failed send is never a lost measurement and never a slow request.

Once enough events have landed, the experiment page leads with the decision in words: which variant won, on how many metrics, over how many judged events. Behind it sits the full detail — effect size, 95% confidence interval, raw and FDR-corrected p-values, the test that was used, and the sample size per arm. Significance is judged on the corrected p-value, and if the traffic split breaks the SRM check, analysis halts rather than showing you a number that is not true.

Want to see the shape of that before you send an event? The interactive demo is the real results view on a sample experiment, with no sign-up.