Security
You are considering sending us your prompts and samples of what your model said to your users. This page says what happens to them, what we do to protect them, and — first, because you will want to know — what we have not done yet.
We are not SOC 2 or ISO certified
LLMJury is a young product in open beta and has not been through a formal certification. We would rather write that here than let you discover it in a questionnaire. What we can do instead is answer specifically: describe our architecture, walk through a data flow, or fill in your review document. Write to [email protected] and a founder answers, within one business day.
If certification is a hard requirement for you today, tell us — it is on the list, and the people asking for it are how it gets prioritized.
What the SDK sends us
LLMJury stores what your SDK sends and nothing more. There is no hidden collection.
- Exposure events
- That a user identifier you supplied was assigned to a variant, and when. The identifier is whatever string your code passes — we never ask for and cannot derive a real-world identity from it.
- Operational measurements
- Response latency, token counts, and computed cost for the model calls the SDK wraps.
- Sampled model outputs
- Only when LLM-as-judge grading is enabled: a sampled subset of the inputs and outputs of your model calls, so the judge can score them against your rubric.
- Business events
- Whatever you explicitly send via client.track(…) — typically a conversion flag or a revenue amount.
And what it does not
- Nothing on assignment. Variant assignment is a local MurmurHash3 computation inside the SDK — no network call is made on your request path, so an identifier only reaches us attached to an event you chose to send.
- No fingerprinting, no device or browser telemetry, and no third-party tracker inside the SDKs. They are open source on PyPI, npm, and Maven Central, so that is a claim you can verify rather than accept.
- No model calls of your own are proxied through us. Your provider client talks to your provider directly; the wrapper observes it.
How much personal data reaches us is your decision. We cannot see inside your application, so we cannot strip personal data out of a prompt you send. Pass a pseudonymous user identifier rather than an email address, and redact what you would not want graded.
How it is protected
Encrypted in transit and at rest
Everything the SDK and the dashboard send travels over TLS. Data is encrypted at rest by the underlying AWS storage.
Keys are scoped, revealable, and revocable
An API key is scoped to one organization. Publishable keys (llmj_pk_…) can only assign, track, and read config, which is why they are safe in client-side code; secret keys (llmj_sk_…) are server-side only. Both can be revealed, rotated, or revoked from the dashboard at any time, and are encrypted at rest.
Role-enforced privileged operations
Key rotation and statistical method overrides require an organization admin. Roles are enforced server-side, not just hidden in the interface.
Limited production access
Access to production is limited to the people who operate the service. Every configuration change carries an audit record of who made it and when, which you can read yourself in the dashboard.
We never train on your data
Not our own models, not a vendor’s. Sampled outputs go to the judge model under an API agreement that excludes training on submitted data. We do not sell your data and we never use one customer’s data to improve another’s results.
Retention is bounded by your plan
Raw events last 7 days on Free, 30 on Pro, 90 on Business, and are deleted after that. Finalized results are snapshotted and kept, because deleting them would destroy the record of a decision you made.
Sub-processors
The vendors that touch customer data today. This is the same list the privacy policy publishes — one source, so the two cannot disagree.
- Amazon Web Services
- Application hosting, databases, backups, and outbound email (SES), in the United States.
- Cloudflare
- DNS, CDN, and static hosting for this website and the documentation site.
- Clerk
- Authentication and organization membership — holds your name, email, and org role.
- Stripe
- Subscription billing. Card details go to Stripe directly; LLMJury never receives or stores them.
- Anthropic
- The LLM-as-judge model. Sampled outputs are sent for grading under an API agreement that excludes training on submitted data.
- Calendly
- The demo scheduler embedded on the /demo page only. Loading that page loads Calendly’s script; no other page does.
Getting your data out, or gone
Email [email protected] and say what you want — export, correction, or deletion. No form and no legal basis required. We do it within 30 days and confirm when it is done. The full set of rights under the GDPR, UK GDPR, and CCPA/CPRA is in the privacy policy.
Found something, or need more?
If you believe you have found a vulnerability, write to [email protected] with enough detail to reproduce it, and please give us a chance to fix it before publishing. We will acknowledge within one business day and keep you updated until it is closed. We do not currently run a paid bug bounty; we do credit anyone who wants it.
For the legal documents themselves, see the privacy policy and terms of service.