Model providers
LLMJury + Google Vertex AI and Gemini
The Google GenAI client is its own shape, so Gemini runs on the key-only path — with one exception: Claude on Vertex through AnthropicVertex is back on the interception path.
The short answer: Works on the key-only path for Gemini. Claude served through Vertex via AnthropicVertex is Anthropic-shaped and works on the interception path.
How it works
google-genai and the Vertex SDK expose their own request shape — generate_content, with its own config object — which is neither of the two shapes wrap() intercepts. So Gemini is a key-only integration, and the honest version of that sentence is on this page rather than buried in a footnote.
The mechanics are the same as any other key-only provider. Resolve the variant from client memory, call Vertex however your code already calls it, and send the outcome. If you want latency and token counts compared per arm, read them off the usage_metadata Vertex returns and put them on the event.
The exception is worth knowing about, because it changes which page you are on. Anthropic publishes an AnthropicVertex client for Claude models hosted on Vertex; that object has the Anthropic shape, so wrap() intercepts it and you get automatic capture and judge scoring. Same cloud, same billing relationship, different client, different path.
Gemini experiments tend to be model-versus-model — Flash against Pro on your own traffic, where the question is whether the cheaper, faster model is worse at your job specifically. That is a variables experiment: put the model id in the variant and read it back with get_variables(…).
What it looks like in your code
from google import genai
from llmjury import Client
client = Client(experiments=["checkout-prompt"])
gemini = genai.Client()
# the variant's model and prompt, merged over your in-code defaults
v = client.get_variables("checkout-prompt", user_id,
defaults={"model": "gemini-2.0-flash",
"prompt": "You are a helpful assistant."})
response = gemini.models.generate_content(
model=v.values["model"],
contents=user_input,
config={"system_instruction": v.values["prompt"]},
)
# send what you want compared across arms
client.track("model_call", {"experiment_id": "checkout-prompt", "user_id": user_id,
"variant": v.variant, "model": v.values["model"],
"tokens_output": response.usage_metadata.candidates_token_count})
client.track("business_event", {"experiment_id": "checkout-prompt", "user_id": user_id,
"variant": v.variant, "business_metric": "conversion", "value": 1})The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.
What gets measured
- Whatever you put on the model_call event — latency, tokens, cost, the model id you tested
- Your business outcome, sent with track(…)
- Exposures per arm and the sample-ratio-mismatch gate, exactly as on any other path
- Judge-scored quality only if you send the text — which on this path is your decision, not a default
What this does not do
- No automatic capture for Gemini. Latency and tokens are in the response; putting them on the event is your code.
- No judge scoring unless you choose to send prompt and response text.
- Vertex’s own evaluation tooling and ours are different things solving different halves — theirs scores a dataset, ours runs the online experiment. Read the comparison pages if you are deciding between them rather than using both.
Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.
Point it at Google Vertex AI and Gemini and see
Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.
Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.