What changes once an LLM feature has real users — drift, silent regressions, cost at volume, and the failure modes that only exist outside a notebook.
10 posts
· 5 min read
A hardcoded prompt means a deploy to change a comma, no real rollback, no history — and, invisibly, no way to run two versions at once. The five properties a prompt version actually needs.
· 5 min read
The new model benchmarks better and performs worse, because your prompt is a fossil record of the old model’s flaws. A five-step migration sequence with exit criteria, and why you should leave a comparison arm running forever.
· 5 min read
Four levers ranked by effort, and the statistical point that decides it: you are not asking whether the cheap model is better, you are asking whether it is worse by less than you will trade for the money. That is a non-inferiority test.
· 5 min read
No alert fires, because nothing breaks — the answers are still fluent, confident, and worse. Six causes with the detection method for each, and why a site-wide average is where regressions hide.
· 5 min read
“Valid” is five different claims, and the one that hurts is a well-formed object with an invented enum member falling through a switch with no default. Why truncation is usually the culprit, and why compliance is the cheapest guardrail you can declare.
· 5 min read
TTFT, tokens per second, total completion, end-to-end — four numbers that diverge, and your interface decides which one matters. Why the tail is fat, why output length is the real lever, and why a latency gap between arms is usually an SRM.
· 5 min read
The cache breaks at the first token that differs, and everything below it is recomputed at full price — so one timestamp near the top of a 3,000-token system prompt voids the whole discount. Order by volatility, and track hit rate per variant.
· 5 min read
Invented examples measure how you handle a user who read the docs; model-generated ones share your model’s blind spot by construction. The five strata with their shares, why the rubric matters more than the labels, and the questions your product can’t answer.
· 6 min read
The numbers an experiment needs are not the text. The minimum viable event, field by field, with what is personal data and what isn’t — plus why a pseudonymous id is still in scope, and why text and metrics need completely different retention.
· 3 min read
Golden datasets are unit tests, not proof. Distribution drift, eval overfitting, and the metrics that only exist in production — latency, cost, and what users do next.