Privacy Policy

LLMJury is operated by PermuteLab LLC. For anything in this policy — including a deletion or export request — write to [email protected]. A founder reads it, and you will have a reply within one business day.

Where the GDPR applies: for the account data we hold about you as a LLMJury user, we are the controller. For the experiment data your application sends us, you are the controller and we are your processor — we act on your instructions and nothing else.

LLMJury stores what your SDK sends and nothing more. There is no hidden collection, no fingerprinting, and no third-party tracker embedded in the SDKs — they are open source on PyPI, npm, and Maven Central, so you can verify that claim rather than take it.

Experiment configuration

The prompts, models, parameters, traffic weights, and metric definitions you create — plus the full version history of each, with the email of the account that made the change.

Exposure events

That a user identifier you supplied was assigned to a variant, and when. The identifier is whatever string your code passes; we never ask for and cannot derive a real-world identity from it.

Sampled model outputs

When LLM-as-judge grading is enabled, a sampled subset of the inputs and outputs of your model calls, so the judge can score them against your rubric.

Operational measurements

Response latency, token counts, and computed cost for the calls the SDK wraps.

Business events

Whatever you explicitly send via client.track(…) — typically a conversion flag or a revenue amount, plus the experiment and user identifier it belongs to.

Account data

Your name, email address, and organization, held by our identity provider; and billing contact and card metadata (never the card number) held by our payment processor.

You decide how much personal data is in there. We cannot see inside your application, so we cannot strip personal data out of a prompt you send us. Pass a pseudonymous user identifier rather than an email address, and redact what you would not want graded. Variant assignment happens locally inside the SDK — the identifier is hashed on your machine and no network call is made on your request path — so an identifier only ever reaches us attached to an event you chose to send.

We use your data to run your experiments: assign variants, grade sampled outputs against your rubrics, compute statistics, show you results, enforce your plan limits, bill you, and support you when you ask. That is the complete list.

We never train models on customer data. Not our own models, not a vendor’s. We do not sell your data, we do not share it with advertisers, and we do not use one customer’s data to improve another customer’s results.

Raw event retention is tied to your plan: 7 days on Free, 30 days on Pro, 90 days on Business. After that window, raw events are deleted.

Finalized results are kept, because deleting them would destroy the record of a decision you already made. They are aggregate statistics — effect sizes, confidence intervals, p-values, and sample counts — not the underlying events. An experiment is always analyzed over its own full duration; retention limits re-analysis of raw events, never the experiment itself. See pricing for the per-plan detail.

Prompts, versions, and experiment configuration are kept for as long as your account is open, because their whole purpose is a durable audit trail. Close your account and we delete everything within 30 days, backups included.

These are the processors involved in running LLMJury today. We will update this list before adding another, and you can ask us to notify you when it changes.

• Amazon Web ServicesApplication hosting, databases, backups, and outbound email (SES), in the United States.
• CloudflareDNS, CDN, and static hosting for this website and the documentation site.
• ClerkAuthentication and organization membership — holds your name, email, and org role.
• StripeSubscription billing. Card details go to Stripe directly; LLMJury never receives or stores them.
• AnthropicThe LLM-as-judge model. Sampled outputs are sent for grading under an API agreement that excludes training on submitted data.
• CalendlyThe demo scheduler embedded on the /demo page only. Loading that page loads Calendly’s script; no other page does.

Everything the SDK and the dashboard send travels over TLS. Data is encrypted at rest by the underlying AWS storage. API keys are scoped to one organization and can be rotated or revoked from the dashboard at any time. Access to production is limited to the people who operate the service.

We have not completed a SOC 2 or ISO certification — LLMJury is a young product in open beta and we would rather say so than imply otherwise. If you need to understand our setup in detail before trusting it with something sensitive, ask at [email protected] and we will answer specifically.

This marketing site sets no cookies. It stores one item in your browser’s local storage — llmjury-theme, which remembers whether you chose light or dark — and it never leaves your device. That is why you have not seen a consent banner here: there is nothing to consent to.

The demo page embeds Calendly’s scheduler, which does set its own cookies when you load that page. The application at app.llmjury.com sets a session cookie so you can stay logged in.

If the GDPR, the UK GDPR, or the CCPA/CPRA applies to you, you have the right to access, correct, export, delete, and restrict processing of your personal data, and to object to it. We do not sell or share personal information as those laws define it, so there is nothing for you to opt out of — but you may still exercise every other right.

You do not need a form or a legal basis. Email [email protected] and say what you want. We will do it within 30 days and confirm when it is done. If you are in the EEA or the UK and you are unhappy with our answer, you may complain to your local supervisory authority.

LLMJury runs in the United States. If you are outside the US, using the product means your data is transferred there. Our processors that operate internationally do so under Standard Contractual Clauses. Data residency in another region is on the Enterprise roadmap — tell us if you need it and it moves up.

LLMJury is a developer tool sold to businesses. It is not directed at anyone under 16, and we do not knowingly collect their personal data. If you believe we have, tell us and we will delete it.

When this policy changes we update the date at the top. For a change that materially affects how we handle your data — a new category of collection, a shorter retention window, a new sub-processor — we will email account owners before it takes effect rather than relying on you to re-read the page.

See also the Terms of Service.