# LLMJury

> LLMJury is a statistical experimentation (A/B testing) platform built specifically for LLM
> products: it deterministically assigns users to prompt/model variants, scores a sample of
> outputs with an LLM-as-judge, and computes SRM-gated, Benjamini–Hochberg FDR-corrected results
> over the live experiment window. Every plan starts free with no credit card.

Key facts:

- Assignment: frozen MurmurHash3 x86 32-bit bucketing hash, identical across the Python,
  TypeScript, and Java SDKs; sticky (editing an experiment never reshuffles exposed users).
- Statistics: chi-squared SRM hard gate (analysis halts at p < 0.001); per-category test routing
  (Welch's t for continuous, two-proportion z for binary, Mann–Whitney U for ordinal, permutation
  + bootstrap for percentile/count); FDR correction across all metric×variant comparisons; raw and
  corrected p-values both reported.
- Judge: built-in quality/safety/relevance metrics plus custom metrics (natural-language rubric +
  structured output schema), versioned, sampled, cached, hard-budgeted.
- Pricing (USD/month): Free $0 (50k events/mo, 7-day retention, 1 seat, no LLM-as-judge); Pro $149
  (1M events/mo, 30-day retention, 10 seats, LLM-as-judge with 50k evaluations/mo, +$49 per extra
  500k events); Business $799 (10M events/mo, 90-day retention, 50 seats, 500k judge
  evaluations/mo, +$99 per extra 2M events); Enterprise custom (coming soon — register interest).
  LLM-as-judge quality grading starts at Pro; Free measures latency, cost, and business metrics.
  The analysis window is never capped by tier.
- SDKs: Python (pip `llmjury`, 3.8+), TypeScript (`@llmjury/sdk`, Node 18+/browser, ESM+CJS),
  Java (`com.llmjury:sdk`, Java 11+). All SDKs: local assign, non-blocking buffered track,
  24h offline replay.

## Pages

- [Home](https://llmjury.com/): product overview and highlights
- [Pricing](https://llmjury.com/pricing): plans and plan limits
- [FAQ](https://llmjury.com/faq): direct answers on SRM, FDR, judging, and pricing
- [Glossary](https://llmjury.com/glossary): definitions of SRM, FDR, permutation test, and more
- [Statsig vs LLMJury](https://llmjury.com/compare/statsig-vs-llmjury): factual comparison
- [Blog](https://llmjury.com/blog): engineering notes
- [Docs](https://docs.llmjury.com): integration guides and API reference
- [Schedule a demo](https://llmjury.com/demo): book a 30-minute walkthrough
- [Contact us](https://llmjury.com/contact): hello@llmjury.com — sales, support, and questions
- [Design partners](https://llmjury.com/design-partners): apply to the design partner program
- [App sign-up](https://app.llmjury.com/sign-up): start free, no credit card

LLMJury is a product of PermuteLab LLC. © PermuteLab LLC. All rights reserved.