Skip to content

Model providers

LLMJury + Self-hosted and OpenAI-compatible endpoints

vLLM, Ollama, LiteLLM, Together, Groq, and anything else that speaks the OpenAI API: point the openai client at your base URL and wrap() intercepts it like any other.

⚡ Automatic capture with wrap()

The short answer: Supported on the interception path whenever you reach the endpoint through an OpenAI-compatible client — which is how most of these are reached.

How it works

The reason this works is unglamorous: wrap() recognises a client shape, and the OpenAI client pointed at http://localhost:11434/v1 is the same object as the OpenAI client pointed at OpenAI. vLLM, Ollama, LiteLLM, Together, Groq, Fireworks, and most self-hosted gateways all publish an OpenAI-compatible surface for exactly this reason, and they inherit the integration for free.

This is the setup where the interception path’s privacy trade-off is sharpest and worth stating twice. Teams self-host precisely so text does not leave their infrastructure — and wrap() sends the prompt and the response to us. If that is the whole point of your deployment, use the key-only path: assignment is local, and the events you send are the events you chose.

The experiment people usually want here is the one the hosted providers make awkward: is the open-weights model we run ourselves actually worse than the API we are paying for? Two arms, one pointing at each, same traffic, same judge, same statistics. That comparison is hard to fake and hard to argue with.

Quantisation, batching, and context length are all just variables. Put them in the variant config and read them back with get_variables(…) — you are testing a serving configuration, which is the same shape of experiment as testing a prompt.

What it looks like in your code

from llmjury import Client
from openai import OpenAI

client = Client(experiments=["checkout-prompt"])
llm = client.wrap(OpenAI(base_url="http://localhost:11434/v1", api_key="ollama"), "checkout-prompt")   # any OpenAI-compatible endpoint, same interception

with client.as_user(user_id):
    # the variant's prompt, from client memory — your literal is the fallback
    p = client.get_prompt("checkout-prompt", user_id,
                          default="You are a helpful assistant.")
    response = llm.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "system", "content": p.prompt},
                  {"role": "user", "content": user_input}],
    )

# the only metric you send yourself
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
                                "variant": p.variant, "business_metric": "conversion", "value": 1})

The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.

What gets measured

  • Latency and time-to-first-token — the numbers that decide whether self-hosting is actually faster
  • Token counts and cost, where your endpoint reports them
  • Errors, timeouts, and refusals per variant
  • Judge-scored quality on a sampled subset (Pro and up) — the comparison that decides an open-weights migration
  • Your business outcome, sent with track(…)

What this does not do

  • The interception path sends us prompt and response text. On a self-hosted deployment that is very often the thing you were avoiding — use the key-only path if so, and read the security page before deciding.
  • If your gateway is not OpenAI-compatible, this page does not apply to you and the key-only path does.
  • We do not host, serve, or scale models. LLMJury measures the calls; the serving is yours.
  • LLMJury itself is not self-hostable today. That is a real limitation for teams whose whole architecture is on-premises, and it is a known gap rather than a decision we are defending.

Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.

Point it at Self-hosted and OpenAI-compatible endpoints and see

Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.

Free plan · no credit card required

Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.

Other model providers