latency

How long an LLM feature takes, and which of the four numbers — time to first token, tokens per second, total completion, end-to-end — your interface makes a user actually feel.

2 posts

• LLM latency: the number your users actually feel

  27 July 2026 · 5 min read

  TTFT, tokens per second, total completion, end-to-end — four numbers that diverge, and your interface decides which one matters. Why the tail is fat, why output length is the real lever, and why a latency gap between arms is usually an SRM.

  • latency
  • production
  • metrics

• Prompt caching: the discount most teams leave on the table

  22 July 2026 · 5 min read

  The cache breaks at the first token that differs, and everything below it is recomputed at full price — so one timestamp near the top of a 3,000-token system prompt voids the whole discount. Order by volatility, and track hit rate per variant.

  • cost
  • latency
  • production