Topics
The blog is about four or five things, depending on how you count. These are them.
evaluation · 10 posts
Deciding whether an LLM output is any good — rubrics, judges, offline datasets, and the gap between scoring well on a benchmark and working for your users.
production · 10 posts
What changes once an LLM feature has real users — drift, silent regressions, cost at volume, and the failure modes that only exist outside a notebook.
A/B testing · 8 posts
Splitting live traffic between versions of an LLM feature and reading which one actually won — the mechanics, the pitfalls, and when the answer is trustworthy.
statistics · 8 posts
The maths that separates a real difference from noise: significance, correction for testing many things at once, confidence intervals, and the tests that suit skewed LLM metrics.
experimentation · 8 posts
Running experiments as a practice rather than a one-off: what to decide before you start, how long to wait, and what to record so the result still means something in three months.
metrics · 4 posts
Choosing what to measure: primary metrics, guardrails, and why a wide metric set costs you statistical power.
LLM-as-judge · 3 posts
Using a model to score another model’s output: the biases it brings, the budget it needs, and why a consistent judge beats an accurate one for comparing two arms.
prompt engineering · 3 posts
Writing and changing prompts on purpose — how to tell a rewrite helped, and why the ten outputs you looked at were the ten you thought to try.
SRM · 2 posts
Sample ratio mismatch: when your traffic split does not match what you configured, what causes it, and why it invalidates the whole analysis rather than just the split.
cost · 2 posts
What LLM features cost at volume, and how to cut the bill without quietly paying for it in quality.
RAG · 2 posts
Retrieval-augmented generation as a system with knobs — chunk size, top-k, rerankers — and how to tell which of them is actually carrying the result.
latency · 2 posts
How long an LLM feature takes, and which of the four numbers — time to first token, tokens per second, total completion, end-to-end — your interface makes a user actually feel.
Written about once
These have a single post each, so they do not get a page of their own until there is a second one — a topic page listing one article is just that article, twice.