Fine-tuning gets proposed most often at the point where a prompt has stopped improving. Someone says “maybe we need to fine-tune”, nobody disagrees, and a quarter disappears into building a training set.
The trouble is that “the prompt isn’t working” is four different problems wearing one sentence, and fine-tuning fixes one of them.
Which problem do you actually have?
Before anything else, work out what you’re looking at. The symptom tells you the technique, and it’s almost never ambiguous once you name it out loud.
| Symptom | What it usually means | What actually fixes it |
|---|---|---|
| It doesn’t know our facts, or invents them | Missing knowledge | Retrieval. Fine-tuning stores facts badly and re-trains on change |
| Right content, wrong shape, format, or voice | Unlearned convention | A prompt with examples first — then fine-tuning if that plateaus |
| Easy cases right, hard cases wrong | Capability gap | A more capable model, or decomposing the task into smaller steps |
| Correct, but too slow or too expensive | Cost and latency | Distillation into a smaller model — the strongest case for tuning |
| Good most of the time, inconsistently | Variance, not deficit | Measurement and aggregation, not training |
Two of those rows deserve a sentence more.
Missing knowledge is a retrieval problem, always. Fine-tuning teaches behaviour; it is a poor and expensive way to store facts, and it’s worse the more those facts change. A model tuned on last quarter’s policy document confidently states last quarter’s policy, with no mechanism to notice. Retrieval has a version and an update path. Training weights do not.
Inconsistency is not a deficit. If the same input produces a good answer four times in five, you don’t have a quality problem you can train away — you have a variance problem, and testing a model that won’t sit still is the thing to read before you spend money. Fine-tuning on top of an unmeasured variance problem doesn’t fix it; it hides it behind a new artefact you now have to maintain.
What fine-tuning genuinely wins
The case for it is real, and the honest version is narrower and more useful than the marketing one.
Behaviour you could never reliably instruct. House style, a specific tone, an idiosyncratic output format, domain conventions that take four paragraphs to describe and still don’t stick. Two hundred good examples can encode what two thousand words of prompt can’t.
A smaller model doing a bigger model’s job. This is the strongest case. Take the outputs of a capable expensive model on your real traffic, train a small one on them, and — for a narrow enough task — get most of the quality at a fraction of the cost and latency. That’s distillation, it’s well-evidenced, and it’s the version of fine-tuning most likely to pay for itself. Cutting LLM costs without cutting quality covers how to prove the trade is acceptable rather than assuming it.
A shorter prompt. Behaviour in the weights is behaviour you stop paying for on every request. If you’re sending 2,000 tokens of instructions and examples to every call, moving that into a tuned model changes your cost per request structurally, not marginally.
What it costs after it ships
The comparison people run is “training run cost vs inference savings”. That’s the small half of the bill. Four costs land later:
Data you don’t have on day one. Not just volume — consistency. A thousand examples labelled by four people who disagreed is a thousand examples of your disagreement, and the model learns it faithfully. Building this set well is the actual project; the training run is an afternoon.
A model artefact with a lifecycle. You now own something. Every base model upgrade is a re-run, a re-evaluation, and a re-deploy. Providers deprecate base models on their schedule, not yours.
Iteration slows from minutes to hours. This is the cost people feel most and predict least. A prompt change is a config edit you can test immediately; a behaviour change on a tuned model is a data change, a training run, and an evaluation cycle. Teams routinely under-price how much that slows down the part of the product that was moving fastest.
Portability. Casual provider switching ends. Whatever leverage you had from being able to move between models — on price, on capability, on availability — you’ve traded for the tune.
None of that is disqualifying. All of it belongs in the decision, and usually isn’t.
You cannot compare a fine-tune to a first-draft prompt
Here’s the methodological error that makes most published fine-tuning wins meaningless, including some you’ve read.
The comparison is run against the prompt as it existed when someone got frustrated with it. That prompt has not been optimised. The tuned model, meanwhile, is the output of a deliberate effort with a curated dataset and someone’s full attention. Of course it wins. You’ve compared a serious attempt to a casual one and attributed the difference to the technique.
The honest sequence is:
- Optimise the prompt until it stops improving. Examples, structure, decomposition, output constraints — measured, not eyeballed. Note where the curve flattens.
- Then fine-tune from that ceiling, using what you learned about the failure modes to choose training examples that target them.
- Then run both as arms on live traffic, with cost and latency reported alongside quality, because a fine-tune that wins on quality and loses on iteration speed may still be the wrong call.
If you haven’t done step one, you don’t have a fine-tuning question yet. You have a prompting question, and it’s much cheaper to answer.
The failure mode when you evaluate it
Fine-tunes fail in a specific direction: narrowing. The model gets better on the distribution you trained on and quietly worse just outside it. It becomes excellent at the ninety per cent of traffic your examples covered and starts mishandling the tail — the unusual request, the other language, the question your training set had three of.
The evaluation trap follows directly. If your eval set is drawn from the same pool as your training data, it cannot see narrowing by construction. Both sets share the blind spot, and the numbers look wonderful.
So: hold out a slice from a different time period, keep a guardrail on traffic you deliberately didn’t train for, and watch refusal and format compliance as well as quality — narrowing shows up in those first. Build your eval set out of production traffic has the stratification that makes this workable.
Fine-tuning is a compression technique, not an intelligence technique. It takes behaviour you can already produce and makes it cheaper, faster, or more consistent. It very rarely makes a model able to do something it couldn’t do at all — and when a team believes it will, that belief is usually standing in for an unmeasured prompt.
In LLMJury a fine-tuned model is an arm like any other, which means the comparison you run is against your best prompt rather than your first one — and cost, latency, and quality arrive on the same screen so the trade is visible rather than assumed.
Start free, or read how to choose an LLM for your product if the real question underneath this one is which base model you should be tuning in the first place.