Someone wants to start measuring their LLM feature. They ask legal whether they can store prompts and completions. The answer is no, or a heavily qualified maybe, and the project stops there — because everybody assumes measurement means keeping the text.
It doesn’t, and the reason is more interesting than a compliance checklist. This isn’t legal advice; what your obligations are depends on your jurisdiction, your sector, and what you told your users you would do. What follows is the engineering shape of the answer.
Both sides of this are right
Engineering wants the text, and has good reasons. A quality score you can’t inspect is unauditable — when the judge marks something a 2, someone will want to see the 2. A debugging session without the prompt is guesswork. A regression you can’t reproduce is a regression you argue about rather than fix.
Legal is also right, and often more right than engineers credit. A prompt is the highest-risk payload in your system, because users paste whatever they happen to have: a customer record, a medical history, a full email thread with four other people’s names in it, occasionally a password. None of it was collected under a purpose you declared, none of it went through your intake validation, and a free-text box is the one place your data model has no opinion about what arrives.
The way through is not for one side to win. It’s to notice that they’re arguing about a bundle that can be taken apart.
The numbers are not the text
Here is the observation the rest of this post rests on.
A statistically valid experiment needs an assignment, a metric value, a timestamp, and a configuration. None of those is the user’s content.
A judge score of 4. A latency of 812 milliseconds. A cost of $0.0021. A conversion flag. A schema compliance boolean. Every one of those is a number that says nothing about what the user asked or what the model replied — and those numbers are the entire input to the comparison that decides which variant wins.
Most teams have never separated these two questions, because historically the same log line answered both: one row with the prompt, the completion, and the timings. Split that row in two and the privacy conversation changes completely, because now you're deciding about content retention and measurement independently rather than trading one against the other.
The minimum viable event
| Field | Why the analysis needs it | Personal data? |
|---|---|---|
| Unit id (user, session, or conversation) | Sticky assignment, and one row per randomisation unit | Yes — a pseudonymous id is still personal data under GDPR |
| Variant assignment | Which arm this observation belongs to | No |
| Timestamp | Analysis windows, time series, novelty effects | No |
| Metric name and value | The measurement itself | No, if the value is a number |
| Model and prompt version | What configuration produced the result | No |
| Token counts in and out | Cost, and explains latency | No |
| Latency | Operational metrics | No |
| Cost | The economics of the arm | No |
| Outcome flag | The business metric that decides it | Depends on what the outcome is |
That table is the whole event. Note the first row especially, because it is the most common misunderstanding in this area: a pseudonymous identifier is still personal data. Hashing a user id does not take it out of scope — it reduces risk, it doesn’t remove the obligation, and a vendor who tells you otherwise is wrong.
And here’s what’s high-risk and optional: the prompt text, the completion text, the retrieved documents, and — the one people forget — the judge’s written rationale, which frequently quotes the output verbatim and lands in your analytics store wearing an innocent field name.
Techniques, cheapest first
Don’t collect it. Available far more often than people assume, and it’s the only technique with no residual risk. If you don’t need the text to make this decision, don’t store the text.
Redact before storage. An entity recogniser over prompts and completions before anything is written. Be honest about the limitation: recall is not 100%, especially for identifiers that don’t look like names — internal case numbers, unusual formats, anything domain-specific. Treat redacted text as lower-risk, not risk-free, and don’t let “we redact” become a reason to relax everything downstream.
Hash or tokenise identifiers. Joins still work, values don’t leak. Salt it, keep the salt out of the analytical store, and remember the caveat above about what hashing does and doesn’t achieve.
Split the retention. This is the technique with the best ratio of benefit to effort, and it works because text and metrics have completely different useful lifetimes. Prompt text is for debugging this week: after thirty days almost nobody opens it. Metrics are for comparing this quarter to last: they need to live for years. Storing both for the same duration means either you throw away your history or you keep content long past its usefulness. Short retention on content, long retention on numbers.
Sample the content. A 2% sample of full request/response pairs debugs nearly as well as 100% and carries a fiftieth of the exposure. Most debugging is pattern-finding, and patterns show up in samples.
Separate the stores. The analytical database that everyone queries, exports, and builds dashboards on holds no content at all. Content lives somewhere with tighter access, shorter retention, and an audit log. This is the control that survives an employee doing something careless with a CSV.
The boring parts that block deals
If you’re evaluating a vendor rather than building this, the questions that actually matter:
- What does the SDK transmit by default? Not what it can be configured to transmit — what it sends if you follow the quickstart.
- A data processing agreement, and a subprocessor list you can read without asking.
- Where data is stored, and whether that’s configurable by region.
- Deletion on request as a real code path with a defined completion time, not a policy sentence.
- What your model provider does with API traffic — retention, and whether it’s used for training. This one is upstream of everything above and is regularly missed because it isn’t your code.
LLMJury’s own answers are on the security page and the privacy policy, including the subprocessor list and an honest statement about what it is not yet certified for.
Where LLMJury sits
Applying our own table to ourselves, since it would be a poor post that didn’t.
Assignment is a local MurmurHash3 of the unit id — computed inside the SDK, with no network call on
your request path. Exposure events, operational measurements, and the business events you send via
client.track(...) carry no model input or output at all: they are the rows in the table above.
Model inputs and outputs are transmitted in one case, and it’s the unavoidable one — LLM-as-judge grading, which cannot score an output it can’t see. That’s a sampled subset rather than every request, it runs under a spend cap you set, and it starts on the Pro plan. The free plan measures latency, cost, and business metrics, which is exactly the no-content configuration this post describes: a complete, statistically valid experiment on a stream of events containing none of your users’ text.
Raw events are deleted on a per-plan retention clock — 7 days on Free, 30 on Pro, 90 on Business — while finalised results are snapshotted and kept, which is the split-retention technique above applied as a default rather than offered as an option. The full list of what the SDK sends and what it doesn’t is on the security page.
So the constraint was narrower than it looked. “We can’t store prompts” rules out one class of metric and leaves the rest of the discipline intact — and the layers in what to actually measure in an LLM product that resolve fastest are, as it happens, the ones that need no content at all.
If you do go on to keep examples, the obligation follows them: an eval set curated from production is user data in a file that gets copied around, which is the caveat at the end of build your eval set out of production traffic.
Start free — the free plan measures latency, cost, and business metrics, with no judge and no model text.