Someone has looked at the inference bill and asked what can be done about it. The obvious answer is a smaller model — the savings are enormous and immediate. The obvious objection is that quality might drop, and nobody can say by how much.
So nothing happens. The decision doesn’t get made; it gets deferred, repeatedly, because one side has a number and the other has an anxiety. The way out isn’t to argue harder about the anxiety. It’s to measure it, and to ask a different statistical question than the one you’re used to asking.
Four levers, ranked by what they cost you
Cap the output length. Frequently the single largest line item, frequently free in quality terms. If your median useful answer is 200 tokens and your limit is 4,000, you’re paying for rambling that nobody reads. Cheapest thing on this list to try.
Shorten the system prompt. A 2,000-token system prompt is billed on every single request forever. Most long prompts have accumulated rules that no longer earn their place — instructions added for a model you no longer use, examples that duplicate each other. Worth an honest audit, with one caveat that changes the arithmetic: if you’re using prompt caching, a long stable prefix is far cheaper than the raw token count suggests, and trimming it may save less than you expect while costing you the cache hit. Measure before you cut.
Route by difficulty. The biggest win and the most work. Most traffic is easy — short questions, routine lookups, well-covered ground — and a small fast model handles it indistinguishably. Send the easy share there and escalate the rest, either with a cheap classifier or by having the small model signal when it’s out of its depth. Getting the routing threshold right is the whole game, and it is itself something to experiment on.
Downgrade wholesale. Simplest to implement, riskiest, and the one that most needs evidence before it ships. It’s also the one people reach for first, which is backwards.
You’re asking the wrong statistical question
Here’s where most cost-reduction efforts go wrong, and it’s not a matter of effort or care. It’s that the default analysis answers a question you didn’t ask.
The standard test asks: is the cheap model better than the expensive one? Almost certainly not. You don’t care. That was never the proposal.
What you’re actually asking is: is it worse by less than I’m willing to trade for the money? That’s a non-inferiority test, and the distinction is not pedantry — it changes what counts as success and what counts as evidence.
The trap in between is the most common analysis error in the wild. You run a standard test, get “no statistically significant difference”, and read that as “quality is the same”. It is not. “No significant difference” and “no difference” are different claims, and the gap between them is usually just an underpowered experiment. An experiment with 200 samples per arm will fail to detect a 5-point quality drop with high probability. Reading its silence as reassurance is how you ship a regression with a green tick next to it.
The fix is to declare a margin, in advance:
I will accept up to 2 percentage points lower task success in exchange for 60% lower inference cost.
Then the test is not “is the difference zero?” but “does the confidence interval on the difference stay entirely above −2pp?” If the interval runs from −0.4pp to +1.1pp, you’ve demonstrated non-inferiority and you ship. If it runs from −3.5pp to +0.9pp, you have not — the data is compatible with a loss larger than you agreed to accept, and the honest answer is not yet, not probably fine.
Declare the margin before you see the result. Deciding what loss is acceptable after looking at the number is not analysis, it’s negotiation with yourself, and you will win.
One counter-intuitive consequence worth internalising: proving non-inferiority to a tight margin takes more traffic than proving a difference, not less. A small acceptable loss is a small effect, and small effects need large samples. Teams routinely underestimate this because “we’re just checking nothing broke” sounds like the easy version. It’s the expensive version.
Worked example
Round numbers, purely illustrative — substitute your own.
Say you serve 3 million requests a month, averaging 800 input and 300 output tokens, on a model costing $3 per million input and $15 per million output tokens. That’s about $7,200 input plus $13,500 output — call it $20,700 a month.
Suppose classification says 70% of that traffic is routine, and a small model priced at $0.25 / $1.25 per million tokens handles it. The routine share drops from roughly $14,500 to about $1,200; the remaining 30% stays at roughly $6,200. New total: about $7,400. You’ve removed nearly two thirds of the bill.
Now the question the whole post is about: what does that cost in quality? If a 2pp margin is acceptable and the experiment shows the confidence interval clearing it, that’s $13,000 a month for a quality difference you’ve bounded and agreed to. If it doesn’t clear, you’ve learned the routing threshold is too aggressive — which is a tuning problem, not a dead end.
Guardrails to watch alongside cost
- Quality, obviously, against your declared margin.
- Latency — this is the free win, and it’s easy to forget to claim it. Small models are usually meaningfully faster, and users feel that.
- Refusal rate. Smaller models sometimes decline more, and a refusal is a failure your quality rubric may score as “safe” rather than “useless”.
- Format compliance. The most likely thing to break outright, and the most likely to be caught by your parser instead of your metrics.
- Escalation rate, if you’re routing. It determines your actual savings, and it will drift as traffic changes.
Making the trade-off one screen
The reason this is hard organisationally is that cost lives in a finance dashboard and quality lives in an eval report, and nobody sees both at the same moment. LLMJury reports cost per request, latency, and judge quality per arm from the same experiment, so “3% worse, 61% cheaper” is a single sentence someone can decide about — with confidence intervals attached, so the honest version of “3% worse” includes how sure you are.
Model swaps run as ordinary variants, so testing a cheaper model works exactly like testing a newer one: control arm, treatment arm, sticky assignment, one verdict.
Pick your margin, run the test, and either bank the savings with a number attached or find out what the routing needs to change. Both beat deferring the decision another quarter.
Start free — cost and latency metrics are on every plan, including the free one — or see what the plans include.