About LLMJury
LLMJury is built and operated by PermuteLab LLC. It is a small, self-funded team, and the product is in open beta.
Why this exists
Every team shipping an LLM feature makes the same call every week: is this prompt better than that one, is the new model worth the money, did that change help. And almost all of them decide it the same way — by reading ten outputs and forming an impression.
That method is not lazy. It is what is available. The alternative used to mean building an assignment layer, an event pipeline, a grading loop, and a statistics engine that does not lie to you when the traffic split breaks — which is months of work that ships no features, and which most teams correctly decline to do.
It comes out of years spent building observability platforms at large scale, where the lesson is uncomfortable and consistent: a system nobody can measure is a system where the confident answer and the correct answer come apart quietly, and stay apart. LLM products are that, with an extra layer — the output is text, so the regression does not throw an error, it just gets worse.
So LLMJury is the narrow bet that the whole loop — version a prompt, split real traffic, grade the outputs, apply the statistics honestly, and roll out or roll back — should be two lines of SDK rather than a quarter of engineering time.
What beta means, in commitments
“Open beta” is a word that can mean anything, so here is what it obliges us to. Each of these is checkable, and the link is where you check it.
- We ship most weeks, and you can check
- The changelog is the receipt for that claim, not a marketing artefact. It lists what shipped and when, newest first, and it has an RSS feed if you would rather watch than visit. Read the changelog →
- A person answers, within one business day
- Support, sales, security questionnaires, and the awkward architecture questions all reach the same small group. There is no tier of support you have to buy to get a reply. Ask us something →
- Your price does not change under you
- Pricing is locked for existing subscribers during beta. If the plans change, they change for new subscriptions — not for a bill you already agreed to. See the plans →
- We say what we have not done
- We are not SOC 2 or ISO certified, there is no uptime SLA, the product cannot be self-hosted, and the interception path stores your prompt and response text raw. All four are on this site in plain words, above the fold of the pages where they matter. Read the security page →
How the product is built
The statistics are published, not asserted
The SRM gate, the FDR correction, the permutation tests, and the rule that routes each metric category to its test are all written down in full — including the thresholds. A verdict you cannot audit is a verdict you should not act on, and that applies to ours.
The SDKs are open source
Python, TypeScript, and Java, Apache-2.0, on PyPI, npm, and Maven Central. The code that sits closest to your application — the bucketing hash, the fallback behaviour, the flush loop — is the code you can read before you install it.
Your application degrades to your own default
Every SDK takes an in-code fallback prompt. If we are down, over your plan limit, or unreachable, your app serves the prompt you shipped. An outage here is not allowed to become an outage there.
Nothing is trained on your data
Not our models, not a vendor’s. Sampled outputs go to the judge model under an agreement that excludes training on submitted data, and one customer’s data never improves another’s results.
The methods themselves are in the glossary and the documentation, and one worked example runs a whole experiment end to end with the real numbers.
Where the product is today
The statistical engine and the SDKs are production-grade and tested. The product around them is young: 18 releases are on the changelog since 2026, the interface changes often, and there are rough edges you will find before we do. That is the trade — you get a direct line to the people building it and outsized influence on what gets built next.
The teams who get the most out of that are the ones we work closest with. Six months of Business free, founder access, and real say over the roadmap — the design-partner programme is open, and 5 slots are left.
Talking to us
[email protected] for anything commercial or general, [email protected] for the product, security questionnaires, and vulnerability reports, and [email protected] for the design-partner programme. All three reach the same people. PermuteLab LLC is the legal entity on every invoice and every agreement — the commercial terms are in the terms of service.