Your dashboard says average response time is 2.1 seconds. Your users say the product feels slow. Both are correct, and the dashboard is reporting a number that nobody in your product has ever experienced.
Latency in an LLM product isn’t one measurement. It’s four, they diverge sharply, and which one you should be optimising is decided by your interface rather than your infrastructure.
The four numbers
Time to first token (TTFT). How long the user looks at nothing. In a streaming interface this is perceived latency — once text starts arriving, the user is reading, and the rest of the generation happens behind their attention.
Inter-token latency, or tokens per second. Whether the text arrives faster than the user reads. Above reading speed, further improvement is close to worthless. Below it, the product feels like it’s labouring, and every additional token per second is felt directly.
Total completion time. The only number that matters when the output isn’t streamed: a function call, a classification, an extraction, a background job. Nobody is reading along, so the whole duration is dead time.
End-to-end request time. TTFT plus generation plus everything you added around it — retrieval, a reranker, a guardrail classifier, a repair pass, a judge you unwisely put in the critical path. This is what your users’ stopwatch measures and the one most likely to have grown without anyone noticing.
From which follows the rule the rest of the post depends on: which number to optimise is a product decision, not an engineering one. A team optimising total completion time on a streaming interface is spending real money improving a figure no user has perceived. A team optimising TTFT on a non-streaming extraction endpoint is doing the same thing in the other direction.
Percentiles, and why the tail is fat here
Mean latency is dominated by your short requests, and the tail is where abandonment lives. Track p95 and p99 — standard advice, and it applies with more force here than in ordinary services.
The reason is specific to LLMs: latency inherits the distribution of output length, and output length varies enormously per request. A one-line confirmation and a nine-paragraph explanation come from the same endpoint with the same prompt. No amount of infrastructure work will make that distribution normal, because the variation isn’t in your serving stack, it’s in what the model decided to say.
That has a direct consequence worth stating plainly: capping or shortening output is usually the largest latency lever available to you, and it’s the same lever as the cost one, since you pay per output token. When those two objectives point at the same change, it’s usually the change to make first — the argument in cutting LLM costs without cutting quality applies unmodified, with latency as a second dividend.
The levers, and what each costs
- Stream. Perceived improvement, not real, and enormous. Moves TTFT to the front of the user’s experience and hides everything after it.
- Shorten the output. The biggest real lever. Cap max tokens, ask for brevity in the rubric, cut the preamble the model writes before answering.
- Shorten the prompt. Prefill scales with input length, so a 2,000-token system prompt is TTFT you pay on every request.
- Cache the prompt prefix. A cache hit skips prefill entirely — see prompt caching, which is a latency lever at least as much as a cost one.
- Use a smaller model for the easy share. Small models are usually faster as well as cheaper, which is the free half of a routing change.
- Parallelise. Retrieval, safety checks, and enrichment that don’t depend on each other should not be sequential. Easy to get wrong, easy to verify.
- Take optional work off the critical path. Logging, analytics, judging, enrichment — none of it needs to happen before the user sees a response. This one is free, and it’s the one most frequently left undone.
That last point deserves emphasis in one direction: never put a judge in the request path. Grading is an offline operation on sampled traffic. A judge in the critical path doubles your latency to compute a number the user will never see.
Latency is the cheapest experiment metric you own
Here’s why this belongs on a site about experimentation rather than in a performance handbook.
Latency is low-variance relative to quality, needs no judge, costs nothing to collect, and resolves in days rather than weeks. In practice that makes it one of the first metrics to produce a verdict in any experiment — often before the quality metric has moved at all. And it’s frequently decisive on its own: a variant that’s 40% slower rarely survives, however good its answers are, because the users who would have benefited abandoned before seeing them.
So measure it on every experiment, including the ones that have nothing to do with speed. A prompt rewrite you expected to be quality-neutral-and-slower may turn out to be quality-neutral-and-faster, which is a shipping decision made in three days instead of three weeks.
There’s a second, sharper reason. A latency regression is the most common cause of a corrupted experiment. The slow variant hits client timeouts, those requests never log an outcome, and the arm quietly loses its slowest responses — which are disproportionately its hardest questions and its worst answers. The struggling variant then reads as the better one, and the mechanism that made it look good is the same one that was failing users.
That’s sample ratio mismatch, and it’s why the arm counts and the latency distribution should be read together, every time. Your experiment is lying to you covers the detection; the practical version is that an unexplained latency gap between arms and an unexplained count gap between arms are usually the same bug.
What to put on the dashboard
- TTFT p50 and p95, if you stream. This is the perceived-speed number.
- Total completion p50 and p95, if you don’t.
- End-to-end p95, always — it’s the one that grows silently as you add pipeline stages.
- Output token count, alongside latency, because it explains most of the movement in it.
- Timeout and abandonment rate, which is where the tail turns into lost users.
LLMJury records latency per arm on every plan, including the free one — it needs no judge and no content, only the timing of the call it wraps.
Optimise the number your interface makes the user feel. Measure all four anyway, because the one you aren’t watching is the one that will move.
Start free, or read what to actually measure in an LLM product for where latency sits among the rest.