Deciding which model your product should run on in the first place — what a public benchmark can and cannot tell you, and which criteria only your own traffic can answer.
2 posts
· 6 min read
A leaderboard measures the model; your product is the model plus your prompt plus your traffic. Nine criteria with how to measure each on your own inputs — and why the prompt you tuned for the incumbent is a rigged instrument for testing a challenger.
· 5 min read
“The prompt isn’t working” is four problems wearing one sentence, and fine-tuning fixes one of them. The symptom-to-technique table, the four costs that land after the training run, and why most published fine-tuning wins are measured against a first-draft prompt.