What actually shipped, written by the people who shipped it. We say elsewhere on this site that LLMJury is in open beta and moving fast — this is the page where you get to check.
Customer-facing changes only. Internal plumbing, CI, and refactors are real work and are not listed here.
The whole path from installing an SDK to reading a verdict now lives on one public page at /quickstart — six steps, copy-paste code in all three SDKs, no account needed to read it. Documentation is reachable from the site navigation rather than only the footer.
The interactive demo at /interactive-demo is the real results view running on the sample experiment: switch metrics, open a comparison, toggle an arm in the histogram. No account, no API call. Alongside it: comparison pages for Langfuse, LangSmith, Braintrust and Statsig, a blog feed, copy buttons on every code sample, and published privacy and terms pages.
Distribution charts mark where each arm’s centre actually sits instead of leaving you to eyeball it, the traffic tab shows exposures over time where you would expect them, and every metric page states its own context and what the result means.
Results can now be exported, an experiment can be marked complete, and any experiment can be duplicated into a pre-filled wizard. Each result says how old the analysis is and who made each configuration change, so the numbers on screen carry their own provenance.
When a result says to stop, you can now act on it from the same screen. Every failed read shows one classified error state with a retry instead of a dead panel, the configuration surface shows the fields that actually decide the verdict, and metric capture is classified by how a metric is defined rather than an exact-name allowlist.
A metric now declares whether higher is better and whether it is a guardrail, instead of being guessed from its name. A result that has not reached significance tells you the sample size that would settle it. The experiment list shows each experiment’s verdict and whether it is receiving data at all, and the results read without dragging them sideways on a narrow screen.
The effect chart is a horizontal forest plot, proportion effects are scaled to percentage points rather than raw fractions, disabled and in-flight buttons look the part, and the explanation wall above the data is collapsed by default so the numbers come first.
Event ingest is now at-least-once — the queue is acknowledged only after the store write succeeds. Judging moved off the ingest thread so grading can never slow intake, the nightly pipeline catches up experiment-days it missed, and over-cap arms are subsampled at random rather than truncated, which was quietly biasing long-running experiments.
An experiment opens on the decision — which variant won, on what, over how much data — with the statistics one click behind it. Effect sizes show relative change against a baseline with units, the create wizard opens with defaults worth launching, and the whole app works below a desktop viewport. Team seats went up: 5 on Free, 100 on Pro, unlimited on Business.
Every new account opens on a shared read-only experiment — a checkout assistant tested as four variants with about 12,000 judged events — so you can read a real verdict before sending a single event of your own.
Usage is visible in the dashboard and an org’s admins get an email before the allowance runs out, backed by a durable ledger so overage is actually accountable. API keys and webhook secrets are encrypted at rest, seat limits are enforced at the identity provider rather than only in the UI, and the raw-event archive is a restorable backup rather than a hypothesis.
llmjury-sdk v0.1.0 is live on PyPI, npm, and Maven Central under Apache-2.0, in public repositories. Same bucketing hash, same non-blocking buffer, same 24-hour offline replay in Python, TypeScript, and Java — and you can read all of it.
Events, judge evaluations, retention, and statistical ceilings are enforced per plan rather than documented and ignored. API keys are revealable any time behind a masked field, so losing one no longer means rotating it. Contact and design-partner forms post to a real backend that reaches a founder.
The platform runs on real production infrastructure with authentication, TLS to the origin, and a deployment pipeline that gates every promotion on a green build. Before this, everything above was running somewhere that was not here.
Self-serve checkout, a billing page, and a customer portal, with plan changes syncing straight through. The results view gained a Distributions tab and now leads with the key findings, and a manual analysis run is stored permanently so a preliminary look you took is still there tomorrow.
Exposures and histograms appear within seconds of an event arriving instead of waiting for the nightly pipeline. The LLM judge is wired into ingest so scores become derived events as they land, event metering is enforced server-side, and key rotation and method overrides are restricted to admins.
Experiments, daily results, API keys, the audit log, and identity are all persisted to Postgres. A restart no longer costs you an experiment.
A one-time wrap of your provider client intercepts every model call, so latency, tokens, and errors are captured with no code at the call site — the only event you send by hand is your business outcome. Experiments are addressed by their name rather than a UUID, and per-variant custom variables resolve from client memory with no network call on the request path.
All of it is on the Free plan except automatic quality grading, which starts at Pro. The quickstart is five minutes; if you would rather see it first, the interactive demo needs no account at all.
Start free — no credit card