Skip to content

Model providers

LLMJury + Anthropic

The client every example on this site wraps. Two lines put your Claude calls behind an experiment key, with prompt versioning and a graded verdict on top.

⚡ Automatic capture with wrap()

The short answer: Fully supported on the interception path — the anthropic client is the other shape wrap() recognises, and the one every sample on this site uses.

How it works

The anthropic client is the shape the quickstart, the home page, and the dashboard snippets all use, so this is the path with the least to explain: wrap the client at startup, bind the user for the request, call messages.create exactly as you do now.

Comparing Claude models against each other is the common case — Haiku against Sonnet on your own traffic, to find out whether the cheaper one is actually worse at your job rather than at a benchmark. Put the model in the variant’s variables and read it back with get_variables(…); the arms differ only in the thing you are testing.

One nuance worth knowing: the judge that scores your outputs also runs on Claude. That is a sub-processor relationship we publish rather than bury — it is on the security page and in the privacy policy, and it is the same list in both places.

What it looks like in your code

from llmjury import Client
import anthropic

client = Client(experiments=["checkout-prompt"])
llm = client.wrap(anthropic.Anthropic(), "checkout-prompt")   # every call through llm is now measured

with client.as_user(user_id):
    # the variant's prompt, from client memory — your literal is the fallback
    p = client.get_prompt("checkout-prompt", user_id,
                          default="You are a helpful assistant.")
    response = llm.messages.create(
        model="claude-haiku-4-5",
        max_tokens=1024,
        system=p.prompt,
        messages=[{"role": "user", "content": user_input}],
    )

# the only metric you send yourself
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.

What gets measured

  • Latency and time-to-first-token per variant, as percentiles
  • Input and output tokens, and computed cost — including the effect of prompt caching
  • Errors, separated from successful calls
  • Judge-scored quality, safety, and relevance on a sampled subset (Pro and up)
  • Your business outcome, sent with track(…)

What this does not do

  • The interception path sends us the prompt and the response text and we store both raw for your plan’s retention window. The key-only path sends neither.
  • We do not proxy your calls to Anthropic. Your client talks to Anthropic directly with your key.
  • Claude models served through Amazon Bedrock or Google Vertex are a different client object — see those two pages, which are not on this path.

Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.

Point it at Anthropic and see

Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.

Free plan · no credit card required

Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.

Other model providers