Skip to content

Model providers

LLMJury + OpenAI

Wrap the openai client once at startup and every GPT call becomes an experiment — prompt, response, latency, tokens, model, and errors recorded with no call-site code.

⚡ Automatic capture with wrap()

The short answer: Fully supported on the interception path — the openai client is one of the two client shapes wrap() recognises.

How it works

The openai Python and Node clients are one of the two shapes wrap() was built for. You pass the client you already constructed, it hands back a wrapper with the same interface, and every call made through it records an exposure plus a model_call event carrying latency, token counts, the model id, and any error. Your call sites do not change.

What varies between arms is up to you. A prompt-only experiment reads get_prompt(…) and passes the result as your system message. An experiment that also swaps the model or the temperature reads get_variables(…) instead and passes those through — which is how you compare gpt-4o against gpt-4o-mini on the same live traffic rather than on a saved eval set.

Assignment stays local. The variant for a user is a MurmurHash3 computation inside the SDK against the polled config, so nothing on your request path waits on us, and the same user sees the same arm on every request and in every service that shares the experiment key.

What it looks like in your code

from llmjury import Client
from openai import OpenAI

client = Client(experiments=["checkout-prompt"])
llm = client.wrap(OpenAI(), "checkout-prompt")   # every call through llm is now measured

with client.as_user(user_id):
    # the variant's prompt, from client memory — your literal is the fallback
    p = client.get_prompt("checkout-prompt", user_id,
                          default="You are a helpful assistant.")
    response = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": p.prompt},
                  {"role": "user", "content": user_input}],
    )

# the only metric you send yourself
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.

What gets measured

  • Latency and time-to-first-token, per variant, as percentiles rather than a mean
  • Prompt and completion tokens, and the computed cost that follows from them
  • Errors and refusals, separated from successful calls rather than averaged in with them
  • Judge-scored quality, safety, and relevance on a sampled subset (Pro and up)
  • Whatever business outcome you send with track(…) — conversion, revenue, retention

What this does not do

  • The interception path sends us the prompt and the response text and we store both raw for your plan’s retention window. If that is not acceptable for your data, use the key-only path — see the security page for the full breakdown.
  • We do not proxy your OpenAI calls. Your client talks to OpenAI directly with your key; the wrapper observes it. That also means an OpenAI outage is an OpenAI outage — we are not in a position to retry it for you.
  • Streaming responses are measured on the call, not token by token as they arrive.

Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.

Point it at OpenAI and see

Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.

Free plan · no credit card required

Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.

Other model providers