Glossary: LLM experimentation terms
The statistics behind trustworthy LLM A/B testing, in plain English.
- SRM (sample ratio mismatch)
- A statistically significant divergence between the observed traffic split and the configured allocation (e.g. 55/45 when you configured 50/50). SRM usually indicates broken assignment, bot traffic, or redirect loss, and it invalidates experiment results. LLMJury runs a chi-squared SRM check and halts analysis when p < 0.001.
- FDR (false discovery rate)
- The expected fraction of significant results that are false positives. When an experiment tests many metrics at once, uncorrected p-values inflate false discoveries; the Benjamini–Hochberg procedure controls FDR by adjusting p-values across all comparisons. LLMJury reports both raw and FDR-corrected p-values and judges significance on the corrected value.
- Permutation test
- A nonparametric significance test that repeatedly shuffles group labels and recomputes the statistic to build the null distribution empirically. It makes no normality assumptions, which suits skewed LLM metrics such as p95 latency; the p-value granularity is 1/permutations.
- Bootstrap confidence interval
- A confidence interval built by resampling the observed data with replacement many times. LLMJury pairs bootstrap CIs with permutation p-values for percentile and count metrics.
- Welch's t-test
- A two-sample t-test that does not assume equal variances — the safe default for comparing means of continuous metrics like latency or cost between variants.
- Mann–Whitney U test
- A rank-based nonparametric test comparing whether one distribution stochastically dominates another. LLMJury uses it for ordinal metrics such as 1–5 judge scores.
- LLM-as-judge
- Using a language model to score another model’s outputs against a rubric (quality, safety, relevance, or a custom definition). LLMJury samples outputs, caches judge verdicts, versions rubrics, and enforces a hard judge budget.
- Bucketing hash
- The deterministic function mapping (salt, user id, experiment id) to a variant so assignment is sticky without storing state. LLMJury uses MurmurHash3 x86 32-bit onto integer allocation boundaries, identical across its Python, TypeScript, and Java SDKs.
- Exposure event
- The event recording that a user actually experienced a variant. Results are computed over exposed users only, and the exposure counts feed the SRM check.
- Analysis window
- The date range an experiment is analyzed over. In LLMJury it is the experiment’s own duration — never artificially capped by plan tier; plan retention only bounds how long raw events stay queryable.