Skip to content

Model providers

LLMJury + Amazon Bedrock

Bedrock’s boto3 client is not one of the two shapes wrap() recognises, so this is the key-only path: resolve the variant, call Bedrock the way you already do, send the outcome.

⌘ Key-only — you send the metrics

The short answer: Works on the key-only path. There is no Bedrock-specific interception, and the trade-off is explicit: you send the metrics yourself, and no prompt or response text reaches us.

How it works

Bedrock is reached through boto3’s bedrock-runtime client, whose surface looks nothing like the OpenAI or Anthropic clients. wrap() does not recognise it, and pretending otherwise would be the kind of claim that falls apart on someone’s first afternoon. So the integration is the key-only one, which is provider-agnostic by construction.

In practice that means three calls. get_prompt(…) or get_variables(…) resolves this user to their variant’s prompt and settings, entirely from client memory. You invoke Bedrock exactly as you do today. Then track(…) sends the outcome you care about — and, if you want latency and tokens per variant, you send those too, from the response metadata Bedrock already hands you.

For teams who chose Bedrock precisely so that model text stays inside their own AWS account, this is the better path rather than the lesser one: assignment is local, the events you send are the events you chose, and no prompt or completion ever reaches us. What you give up is judge-scored quality, which needs the text to score.

Claude models served through Bedrock are still Claude, but they arrive through the Bedrock client, so they are on this page rather than the Anthropic one. The exception is AnthropicBedrock from the anthropic package, which is Anthropic-shaped and therefore back on the interception path.

What it looks like in your code

import boto3, json, time
from llmjury import Client

client = Client(experiments=["checkout-prompt"])
bedrock = boto3.client("bedrock-runtime")

# 1. resolve the variant — local computation, no network call
p = client.get_prompt("checkout-prompt", user_id,
                      default="You are a helpful assistant.")

# 2. call Bedrock exactly as you already do
started = time.perf_counter()
result = bedrock.invoke_model(
    modelId="anthropic.claude-3-5-haiku-20241022-v1:0",
    body=json.dumps({"anthropic_version": "bedrock-2023-05-31", "max_tokens": 1024,
                     "system": p.prompt,
                     "messages": [{"role": "user", "content": user_input}]}),
)
usage = json.loads(result["body"].read())["usage"]

# 3. send the measurements you want compared — no prompt or response text
client.track("model_call", {"experiment_id": "checkout-prompt", "user_id": user_id,
                            "variant": p.variant, "model": "claude-3-5-haiku",
                            "latency_ms": int((time.perf_counter() - started) * 1000),
                            "tokens_input": usage["input_tokens"],
                            "tokens_output": usage["output_tokens"]})
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.

What gets measured

  • Anything you put on the model_call event yourself — latency, tokens, cost, model id
  • Your business outcome, sent with track(…)
  • Traffic split health: exposures per arm, checked for sample ratio mismatch like any other experiment
  • The full statistical treatment is identical — the path changes what you send, never how it is analysed

What this does not do

  • No automatic latency, token, or error capture. Bedrock hands you those numbers in the response; putting them on the event is code you write.
  • No judge-scored quality metrics on this path, because the judge needs the prompt and the response text and this path sends neither. If you want them, you have to send the text — which is the trade-off, stated rather than hidden.
  • We do not build a Bedrock-specific client wrapper today. If enough teams ask, that is exactly the kind of thing the roadmap is for.

Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.

Point it at Amazon Bedrock and see

Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.

Free plan · no credit card required

Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.

Other model providers