Frameworks & runtimes
LLMJury + LangChain and LangGraph
Resolve the variant before the chain runs and pass it in as a parameter. There is no LangChain callback handler — the prompt is an input to your chain, which is all the integration needs to be.
The short answer: Works on the key-only path. We do not ship a LangChain callback handler, and for an online experiment you do not need one.
How it works
A LangChain integration in the tracing sense means a callback handler that records every step of a chain. We do not have one, and this page is not going to imply that we do. What we do is a different job: the unit of an online experiment is not a span, it is a user assigned to an arm and an outcome that follows.
So the integration point is one line earlier than you might expect. Before the chain runs, get_prompt(…) or get_variables(…) resolves this user to their variant — from client memory, no network call — and you pass the result into the chain as the parameter it already takes. The chain does not know it is in an experiment, which is exactly why nothing about it has to change.
For LangGraph, the same logic applies at the graph’s entry point: resolve once, put the variant on the state, and every node downstream reads it. Assignment is deterministic, so a retry, a resumed thread, or a second call for the same user lands on the same arm.
What you send back is the outcome, once, when the chain finishes — task success, a conversion, whether the user accepted the answer. That is one track(…) call, and it is the metric the verdict is actually about. If you want the per-step detail as well, a tracing tool is the right shape for that job and the comparison pages say so plainly.
What it looks like in your code
from langchain_core.prompts import ChatPromptTemplate
from llmjury import Client
client = Client(experiments=["support-agent-prompt"])
# resolve the arm before the chain runs — local, no network call
p = client.get_prompt("support-agent-prompt", user_id,
default="You are a helpful support agent.")
# the chain is unchanged; the system prompt is just an input to it
prompt = ChatPromptTemplate.from_messages([("system", p.prompt), ("human", "{question}")])
answer = (prompt | llm | parser).invoke({"question": user_input})
# one outcome per finished chain — the metric the verdict is about
client.track("business_event", {"experiment_id": "support-agent-prompt", "user_id": user_id,
"variant": p.variant, "business_metric": "resolved", "value": 1})The same path in all three SDKs is on the five-minute quickstart, and the per-method reference is in the documentation.
What gets measured
- The outcome of the whole chain, which is the thing a chain-level experiment is about
- Any per-arm measurement you choose to send — end-to-end latency, total tokens across steps, retry counts
- Exposures per arm and the sample-ratio-mismatch gate
- Judge-scored quality if you send the final input and output as a model_call event
What this does not do
- No callback handler, no per-step traces, no span tree. If you need to see inside a chain run, that is a tracing tool’s job and we are not one — the Langfuse and LangSmith comparisons say where the line is.
- Automatic latency and token capture is per model call, not per chain. A chain’s total is something you compute and send.
- An agent that makes a variable number of model calls needs its unit of analysis chosen deliberately — the post on evaluating agents is about exactly that trap.
Which of the two paths you take decides whether your prompt and response text reaches us. The full breakdown is on the security page.
Point it at LangChain and LangGraph and see
Free plan, no credit card. The sample experiment is already in your account, so there is a real verdict to read before you have any traffic of your own.
Using something we have not listed? Tell us — the key-only path already works with it, and which providers people ask about is how the interception list grows.