SRM

Sample ratio mismatch: when your traffic split does not match what you configured, what causes it, and why it invalidates the whole analysis rather than just the split.

2 posts

• Your experiment is lying to you: sample ratio mismatch

  4 August 2026 · 5 min read

  A 3% imbalance on 100,000 users is not noise, it is a bug — and it invalidates every number in the experiment, not just the split. What causes SRM, why a timeout is the LLM-specific culprit, and why you must never reweight.

  • SRM
  • statistics
  • experimentation

• Announcing LLMJury: statistically defensible A/B testing for LLM products

  1 July 2026 · 1 min read

  Why “it feels better” is not an eval strategy, and how SRM gates, FDR correction, and an LLM judge make prompt experiments trustworthy.

  • announcements
  • statistics
  • SRM
  • FDR