Look at where your token spend actually goes and there’s a good chance the answer is input, not output — a system prompt, a set of tool definitions, a few-shot block, and a stack of retrieved documents, resent in full on every single request.
Providers will let you stop paying full price for text they’ve already processed. Most teams have the feature switched on and are getting a fraction of the discount available, because of one rule about where in the prompt things sit.
What a cache hit actually is
A prompt cache is a prefix cache. The provider keeps the computed attention state for the beginning of your prompt, and when the next request starts with exactly the same tokens, it reuses that state instead of recomputing it.
Which gives the rule the whole post rests on:
The cache breaks at the first token that differs, and everything from that point onward is recomputed at full price.
Read that twice, because the consequence is not obvious and it’s expensive. A single volatile value
near the top of your prompt — a timestamp, a user id, a session variable, today’s date, an A/B
variant label — destroys the cacheability of everything below it. Not the line it’s on. Everything
below it. A team with a 3,000-token system prompt and a Current time: 14:32:07 on line two is
paying full price for all 3,000 tokens on every request, and their dashboard says caching is
enabled.
Go and look at the first two hundred tokens of your longest prompt right now. This is the highest ratio of saving to effort on the entire site.
Order the prompt by how often things change
The fix is a reordering, and it’s usually a refactor with no behavioural change at all:
- Stable, first. System instructions, tool and function definitions, few-shot examples, long reference documents, style guides. Anything identical across every request.
- Semi-stable, next. Conversation history — which is well-behaved, because it grows by appending, so each new turn extends a prefix that’s still intact.
- Volatile, last. The user’s message, timestamps, request ids, per-request context, anything that differs every time.
Two things to check while you’re in there. Serialisation order matters: if you build the prompt from a dictionary or a map with non-deterministic iteration order, your “identical” prefix isn’t identical and you will never hit. And whitespace matters — a template that trims differently depending on input length breaks the prefix just as effectively as a timestamp.
The caveats, and which are structural
Be clear about which parts are permanent facts and which are current pricing:
Structural, and true everywhere. Cache entries have a lifetime and expire; a cold entry costs what it always did. There’s a minimum prefix length, so short prompts don’t qualify. And a prefix used only once may be a net loss where writing to the cache carries a premium.
Provider-specific, and moving. The discount rate on a hit, the entry lifetime, whether writes cost extra, the minimum token count, and whether caching is automatic or has to be marked up in the request. These differ between providers and change over time, so check the current pricing page rather than a blog post — including this one.
The structural facts are what you should design around. The rates are what you should recompute every quarter.
What the arithmetic looks like
Illustrative — round numbers, not a claim about anyone’s bill. Take a request with a 4,000-token stable prefix and a 200-token user message, and suppose a cache hit costs a tenth of the normal input rate.
| Scenario | Full-price input tokens | Relative input cost |
|---|---|---|
| No caching | 4,200 | 100% |
| Caching, prefix broken by a timestamp on line 2 | 4,200 | 100% |
| Caching, correctly ordered, 90% hit rate | 200 + 400 (hits at 1/10) + 420 (misses) | ~24% |
The middle row is the point of the table. It is not a hypothetical — it’s the default outcome of enabling caching without auditing prompt order, and it is indistinguishable from the top row on every dashboard except the invoice.
It’s a latency lever too
Caching posts usually stop at cost, which misses half the benefit. A cache hit skips the prefill computation, and prefill is a large share of time to first token on any request with a long prompt.
So a correctly ordered prompt improves the number your users actually perceive in a streaming interface — often more noticeably than the bill improves, because a bill is felt monthly and TTFT is felt every time. The four latency numbers and which one your interface makes users feel are covered in LLM latency: the number your users actually feel; caching moves the first of them.
Cache hit rate belongs in your experiments
Here’s the operational trap, and it’s why this post sits on a site about experimentation.
Someone rewrites a prompt to improve quality. In the process they move a section, or add a per-request line near the top, or reorder a few-shot block. Quality goes up, the experiment says ship, and the cache hit rate silently collapses. The cost increase shows up on next month’s invoice, weeks after the change, at which point it gets attributed to traffic growth — because nobody connects an invoice to a prompt edit from three weeks ago.
An experiment that measures cost per request per arm catches this during the test, before the change ships. The cost of a variant is not a property of the model you chose; it’s a property of the prompt you wrote, and prompt structure affects it as much as prompt length does.
Two things to track per arm, then: cost per request, and cache hit rate if your provider reports it. LLMJury records cost alongside quality and latency per arm, so a quality win that’s also a cost regression shows up as both — the trade-off in cutting LLM costs without cutting quality applies to prompt rewrites just as much as to model choices.
Reorder the prompt first. It costs an afternoon, changes no behaviour, and is the rare optimisation with no trade-off attached.
Start free — cost per arm is measured on every plan.