Comparison
promptfoo vs LLMJury
promptfoo is where you test a prompt before it ships. LLMJury is where you find out what it did after.
promptfoo is open-source prompt testing, evaluation, and LLM red-teaming, run from the CLI. Below is what it does well, where LLMJury does something different, and why the two are usually complements rather than a choice.
What promptfoo is good at
- It is genuinely the fastest way to start. A YAML file, a list of test cases, `npx promptfoo eval`, and you have a matrix of prompts against models with no account and no service.
- Open source, local-first, and free. Your prompts and test cases never leave your machine unless you send them, which settles a lot of arguments before they start.
- It fits CI. Assertions on outputs mean a prompt change can fail a build the same way a broken test does — the single best habit a team shipping prompts can pick up.
- Red-teaming and adversarial scanning are a first-class part of the tool, not an afterthought.
Where LLMJury does something else
A promptfoo run answers "how does this prompt do on the cases I wrote?". That is a real question, and it is bounded by the cases you thought of. The cases you did not think of are the ones your users bring, and they are also where prompt changes usually go wrong.
LLMJury asks the other question: on live traffic, over the whole distribution of what people actually send, does this version do better? Users are split deterministically, a sample of real outputs is graded, and latency, cost, and business outcomes are measured in the same experiment. The verdict clears an SRM check and FDR correction before it is shown at all.
The two are not rivals so much as consecutive. Offline assertions are unit tests for a prompt; the online experiment is the production rollout that tells you whether the unit tests were testing the right thing.
Side by side
| Dimension | LLMJury | promptfoo |
|---|---|---|
| Built for | LLM products: prompt and model variants, judged on live traffic | Testing and evaluating prompts against declared test cases, locally and in CI |
| Core unit | The online experiment — a variant split running on real users | The eval run — prompts × models × test cases |
| Quality measurement | Built-in LLM-as-judge (quality, safety, relevance) plus your own rubric metrics | Assertions and model-graded rubrics over your test cases |
| Live-traffic comparison | The product — real users split, exposed, and measured | Not the focus; the inputs are the ones you wrote |
| Statistical treatment | Chi-squared SRM hard gate (analysis halts, no result shown), per-metric test routing, Benjamini–Hochberg FDR correction across every comparison, raw and corrected p-values both reported | Pass/fail assertions and scores per test case |
| Prompt management | Prompts live outside your repo with version history, audit trail, diff, and one-click rollback — and the version you edit is the variant that gets tested | Prompts as files in your repo, versioned by git |
| Where it runs | Hosted; SDKs are open source | Your machine or your CI runner |
Written from each product’s own published positioning, and last checked against it on 3 August 2026. Both products move — if something here is out of date or unfair, tell us at [email protected] and we will correct it.
Should you run both?
Keep promptfoo in CI so an obviously worse prompt never reaches a user, and use LLMJury for the change that passes CI and still might be wrong. The offline suite is cheap and catches the blatant regressions; the online experiment costs traffic and time, and is the only thing that can tell you a plausible-looking rewrite made your users worse.