Choosing what to measure: primary metrics, guardrails, and why a wide metric set costs you statistical power.
4 posts
· 6 min read
Outcome decides, trajectory explains, cost constrains. Why the outcome assertion has to come from outside the agent, why a golden path punishes a better route, and why nine sampling steps mean more traffic rather than less.
· 5 min read
Quality, operational, behavioural, business — four layers, what each answers, how fast it moves, and the trap in each. Plus why watching twenty metrics makes you less able to detect the effect you care about.
· 5 min read
“Valid” is five different claims, and the one that hurts is a well-formed object with an invented enum member falling through a switch with no default. Why truncation is usually the culprit, and why compliance is the cheapest guardrail you can declare.
· 5 min read
TTFT, tokens per second, total completion, end-to-end — four numbers that diverge, and your interface decides which one matters. Why the tail is fat, why output length is the real lever, and why a latency gap between arms is usually an SRM.