Evaluating a prompt is straightforward in shape: one input, one output, one judgement. Evaluating an agent is not, because between the input and the output sit a plan, six tool calls, two retries, and a self-correction — and any of them can be the reason it worked.
Score only the final answer and you can’t tell a good agent from a lucky one. Score only the path and you’ll punish an agent that found a shorter route than the one you had in mind. Most teams shipping agents right now cannot say whether last week’s change made theirs better, and it isn’t because they’re careless — it’s because the single-turn playbook doesn’t transfer.
The three axes
Everything worth knowing about an agent run sits on one of three axes, and confusing them is the root of most agent evaluation confusion.
Outcome — did the task get done? Defined by an assertion about the world: the file exists and contains the right rows, the ticket moved to closed, the returned figure matches the source system. Outcome is the only axis your user cares about, and the only one that tells you nothing about why.
Trajectory — how it got there. Step count, tool-call validity, whether it recovered from an error, whether it went round in circles. Diagnostic, never the verdict.
Cost — what it consumed. Tokens, wall-clock, number of model calls. An agent that succeeds in forty steps and nine dollars has not succeeded.
The critical rule: outcome decides, trajectory explains, cost constrains. A change that improves outcome and doubles trajectory length is a change you probably still ship, and a change that improves trajectory while leaving outcome flat is a change you have no evidence for.
Why the outcome check must be external
There is one hard requirement on outcome measurement, and skipping it invalidates everything above it: the outcome assertion must be made by something other than the agent.
An agent that declares victory is reporting on its own work. Ask a model whether it completed the task and you get a number that measures the model’s optimism, and agent optimism is not evenly distributed — it rises exactly when the task was hard and the trajectory went badly, which is precisely when you need the measurement to be honest.
So: query the database, read the file, call the API, diff against a known-good result. Where an external assertion genuinely isn’t available — open-ended research, drafting, summarising — use a judge, but judge the artefact against a rubric, not the agent’s claim about the artefact.
Grading the path without punishing invention
The obvious way to score a trajectory is to write down the correct sequence of steps and measure how far the agent deviated. Don’t. A golden path scores agreement with your plan, and an agent that solved the problem in three steps when you expected five gets marked down for being better than you.
What survives that objection is a small set of trajectory properties that are bad regardless of route:
- Looping — the same tool called with the same arguments twice. Detect with a cycle check over the trace, cap it, and log the cap as a failure rather than swallowing it.
- Premature completion — the agent stops and declares done without doing the work. Only an external outcome check catches this, which is the second reason that check is non-negotiable.
- Tool misuse — a well-formed call to the wrong tool. Individually invisible; obvious in aggregate if you log tool choice by task type and look at the distribution.
- Context exhaustion — the run gets long, the early instructions fall out of effective attention, and behaviour changes mid-task. Detect by segmenting outcome against step count: a success rate that falls off a cliff after step twelve is this.
- Error cascade — one bad retrieval at step two poisons everything after it. The trace reads as confident throughout, which is what makes it hard to spot without checking intermediate results.
Each of those is a bug with a fix. None of them requires you to have known the right answer in advance.
Binary success throws away most of your information
Agent tasks are rarely pass/fail, and a binary success rate treats “failed at step one” and “did everything except the final write” as the same event. That’s a real loss: those two runs need completely different fixes, and a metric that can’t distinguish them can’t tell you the change helped.
The fix is a small ordinal rubric, typically 0–3, with each level pinned to something observable:
| Level | Meaning | Example anchor |
|---|---|---|
| 0 | No progress | No valid tool call, or gave up immediately |
| 1 | Partial | Gathered the right information, took no action |
| 2 | Complete with intervention | Right result, needed a retry or a correction |
| 3 | Clean completion | Right result, no wasted or failed steps |
Two things to be honest about. First, the levels are only as good as their anchors — “partial” without an example means two people score the same run differently, and the metric drifts. Second, an ordinal score is not a quantity. The gap between 1 and 2 isn’t the same size as the gap between 2 and 3, so averaging it and running a t-test assumes something false. LLMJury routes ordinal metrics to a permutation test with bootstrap confidence intervals for exactly this reason, rather than treating a 0–3 rubric as if it were normally distributed.
Agents are noisy, and that means more traffic, not less
Here is the part that surprises people. A single-turn prompt has one sampling step. A nine-step agent has nine, and each one can branch the run somewhere new. Variance compounds along the trajectory, which means the same true improvement is harder to detect in an agent than in a prompt.
Required sample size scales with variance. So an agent experiment needs more data than a prompt experiment to reach the same confidence about the same effect — see how much traffic do you need to A/B test a prompt? for the arithmetic, and note that the agent case sits at the demanding end of every row in that table.
The instinct when a system is this noisy is to fall back on judgement: read ten traces, form an impression, ship. That instinct is exactly backwards. Ten traces from a high-variance system is a sample so small that you would reject it from anyone else, and the fact that each trace is long and detailed makes it feel like more evidence than it is. Noise is an argument for measurement, not against it.
Cost is where the noise bites hardest. Token spend per run is long-tailed — most runs are ordinary and a few explore for thirty steps — so the mean is dragged around by the tail and tells you very little. Quote p95. A change that leaves the median untouched and halves the p95 is one of the more valuable things you can do to an agent, and a mean will hide it.
What to measure first
If you’re starting from nothing, the order that gets you a usable signal fastest:
- One external outcome assertion for your highest-volume task type. Binary is fine to begin with.
- Cost and step count per run, at p50 and p95. Cheap, no judge required, and frequently decisive on their own.
- Loop and cap-hit rate. A single number that catches a whole class of embarrassing failures.
- Then the ordinal rubric, once you’ve seen enough failures to write anchors that mean something.
That ordering is deliberate: it front-loads the metrics that need no model to compute, which is the same principle as the layered approach in what to actually measure in an LLM product. Quality scores are the hardest thing to get right and the last thing you should build.
The agent is a system with a verdict at the end of it. Once outcome is external and cost is recorded, an agent change is an ordinary experiment: two arms, one primary metric, one decision.
Start free — cost, latency, and outcome tracking are on every plan.