A new model lands. It beats the one you’re using on every public benchmark, the announcement post is convincing, and someone in the standup asks why you haven’t switched yet. You switch. Quality gets worse.
This happens often enough to be a genre. The reason is not that the benchmarks lied. It’s that they measured the model, and your product is the model plus your prompt plus your traffic — and only the first of those three is public.
Your prompt is a fossil record of the old model’s flaws
Read your system prompt honestly and you’ll find instructions that exist for no reason other than that the old model did something annoying. “Do not repeat the question back.” “Answer in at most three sentences.” “Always include the source.” Each one was added on a specific afternoon to fix a specific failure.
A new model has different failures. It doesn’t repeat the question, so that line is dead weight. It was already concise, so the three-sentence cap now makes it clipped and unhelpful. It cites by default, so the source instruction pushes it into over-citing. Your prompt was a set of corrections; applied to a model that doesn’t need correcting, corrections become distortions.
The team then concludes the new model is worse. Sometimes it is. More often, the old prompt is.
What actually moves when you change models
Beyond quality, which is what everyone watches, these move — and several of them break things loudly:
- Format adherence. The JSON your frontend parses. New models are usually better at this, but “better” can still mean differently — an extra field, a changed null convention, markdown fences around the object.
- Verbosity. And therefore cost per request and perceived latency, neither of which appears in a quality score.
- Refusal behaviour. The safety boundary sits somewhere slightly different. Questions your product answered last week may now be declined, and refusals rarely show up as errors.
- Instruction-following on long prompts. A 2,000-token system prompt with fifteen rules is followed differently by different models. The rule that mattered most may now be the one that gets dropped.
- Tool-calling shape. When to call, how often, how it handles a failed call.
- Tokenisation. Your cost per request changes even at identical output length, because the same text is a different number of tokens.
Note how many of these are invisible to an offline quality harness. That harness is a photograph of the examples you thought to write down, and none of them measure cost at your real token distribution.
The migration sequence
Five steps, each with a condition for moving on. The discipline is in the exit criteria — skip those and this becomes a to-do list you can complete while learning nothing.
1. Run your offline harness. Cheap, fast, catches outright breakage: format violations, safety failures, obvious collapses. Exit when nothing fails outright. This step cannot tell you the new model is better. It can tell you it’s broken, which is worth knowing before you spend traffic.
2. Shadow it. Send a copy of real production traffic to the new model without serving the result. Now you have cost, latency, output length, and format compliance measured on your actual distribution rather than on sampled test questions. Exit when the operational numbers are acceptable. If the new model is 40% more expensive at your traffic mix, you want to know that before you argue about quality.
3. Run the real experiment. Small share of live traffic, old model as control, quality as the primary metric, cost and latency and refusal rate as guardrails. Exit on a verdict, not on a date. This is the only step that measures the thing you care about — whether users get better answers — and it needs enough samples to say so. Work out how many before you start.
4. Re-tune the prompt for the new model, and test that as its own arm. This is the step everybody skips, and it’s why “the new model is worse” is so often really “our prompt is old”. Strip the workarounds that no longer apply, then run the re-tuned prompt against both the old configuration and the naive port. Exit when you know which of the three is best.
5. Ramp. Increase the share while the guardrails hold, with a rollback that’s a configuration change rather than a deploy. Exit at 100%, with the old configuration still one click away.
The mistake that wastes the whole exercise
Changing the model and the prompt in the same arm.
You end up with “the new configuration scored 0.3 higher” and no way to attribute it. Was it the model? The rewrite? Did a great prompt rescue a worse model, so you’re now paying more for a result you could have had by just rewriting the prompt on the old one? You cannot tell, and next quarter, when the next model arrives, you’ll be starting from a baseline you don’t understand.
If you want to test both, that’s two arms plus a control, not one arm with two changes. It costs more traffic. It’s the difference between a result and an anecdote.
Then leave it running
The framing that saves you the most pain long-term: model migration isn’t a project that ends.
Providers update models behind stable names. The model serving your traffic under a given identifier in November is not bit-identical to the one that served it in August, and you will not receive a changelog entry that maps onto your product’s behaviour. A permanent comparison arm — a small slice of traffic on a pinned configuration — converts that from an unknowable into a measurable. When quality shifts, you can see whether it shifted for both arms (something changed underneath you) or only one (something you did). Without it you get a silent regression nobody can date.
LLMJury treats a model swap as an ordinary variant: same experiment shape, same assignment, same verdict, with cost and latency reported per arm alongside quality so the trade-off arrives on one screen. Assignment is deterministic and sticky, so a user doesn’t switch models between two turns of the same conversation.
The new model probably is better. Almost all of them are. Make it prove it on your traffic, with your prompt, against your metrics — and then re-tune the prompt and let it prove it again.
Start free — no credit card — or read how to test a cheaper model the same way.