Security asks whether your LLM feature is safe. You say you’ve hardened the system prompt. They ask how you know, and there the conversation stops — because the honest answer is that you changed some wording and it felt more robust.
Prompt injection has no clean fix. It’s the direct consequence of instructions and data sharing one channel, and every mitigation is probabilistic. So the answerable question isn’t “are we safe”. It’s what fraction of adversarial inputs get through today, and did that number move when we changed the prompt?
Injection and jailbreaking are not the same threat
They get used interchangeably and they have different blast radii.
Jailbreaking is your user talking your model out of its own rules. They want the output; they’re attacking on their own behalf. The cost is embarrassment, brand damage, and occasionally a screenshot that travels.
Prompt injection is a third party’s text being read as instructions — a retrieved document, a web page, a support ticket, an email, a tool result. Your user may be entirely innocent. The attacker is whoever wrote the content your pipeline ingested, and their instructions arrive with your system’s privileges attached.
If you run RAG or agents, the second one is the one to be afraid of, because it reaches your tools and your context rather than just your reputation. An injected instruction inside a document your agent retrieves is executing in a session that already has credentials.
Four outcomes, and the one nobody counts
Any adversarial input lands in one of four places. Instrument all four.
| Outcome | What happened | What it costs you |
|---|---|---|
| Blocked | Refused and logged | Nothing, if the refusal was correct |
| Deflected | Answered safely without complying with the injected instruction | Nothing — usually the best outcome for a real user |
| Complied | Followed the attacker’s instruction | The failure you’re counting: data, actions, reputation |
| Over-refused | A legitimate input caught by the defence | Real users turned away, silently, with no ticket filed |
That last row is the point of the table. A defence with no over-refusal number attached is a defence with an unknown price. Tightening a system prompt until compliance approaches zero is trivially easy — refuse everything — and every increment of that tightening costs you legitimate requests you never hear about, because a user who gets refused doesn’t complain, they leave.
Compliance and over-refusal move in opposite directions. That’s a trade-off, which means it belongs in an experiment rather than in someone’s judgement.
Building the measurement
Four pieces, none of them exotic:
A red-team set. A few hundred adversarial inputs spanning the categories you face: direct instruction override, role-play and hypothetical framing, encoded or obfuscated instructions, instructions embedded inside retrieved documents, and — if you’re agentic — poisoned tool results. Write them for your product; generic jailbreak lists test a generic product.
A matched benign set. Legitimate inputs that superficially resemble the attacks: the security researcher asking a real question, the user quoting an error message that contains an instruction, the document that legitimately says “ignore the previous section”. Without this set, over-refusal is unmeasurable, which is why most teams don’t measure it.
A gate on every prompt version. Both sets run before a prompt ships. This is the part that converts security from a review into a metric: every system prompt edit is a change to your attack surface, and right now most teams make several a week with no measurement at all.
A production sample. Classify a slice of real traffic the same way. Your actual attackers are not in your test set, and the gap between what you imagined and what arrives is the interesting part.
Be aware the red-team set decays. An attack corpus written six months ago measures six-month-old attacks against a model that has since been updated underneath you. It needs an owner and a refresh cadence, not a one-off project.
These are guardrails, not primary metrics
Here’s how this fits into experimentation, and it’s the part that makes it sustainable.
You are not trying to improve injection resistance in every experiment. You’re trying to prove each change didn’t degrade it. That makes compliance rate and over-refusal rate guardrail metrics — declared before the run, watched throughout, and able to veto a variant that wins on quality.
Because that’s the realistic failure: a prompt rewrite that improves task success by four points and also makes the model more agreeable to injected instructions. Every quality metric says ship it. The only thing standing between that change and production is a guardrail somebody declared in advance. What to actually measure in an LLM product covers the primary versus guardrail split; this is the case where it earns its keep.
LLMJury lets you declare a guardrail that can veto a winning variant, which is the mechanism this needs — a metric nobody is optimising but everybody has to pass.
What this doesn’t give you
Say this out loud, because a security reader who catches you overclaiming will discount everything else you’ve written.
This measures your prompt-layer posture against the attacks you thought of. It is not a substitute for the controls that actually contain the damage:
- Treat all model output as untrusted input. If a model’s response reaches a shell, a query, a browser, or another model’s instruction slot, it needs the same sanitisation as a form field.
- Least privilege on tools. An agent that can only read the three things it needs is a much smaller incident than one holding a general-purpose credential.
- A human in the loop for consequential actions — payments, deletions, external messages.
- Separation between retrieved content and instructions wherever your framework supports it.
Measurement tells you whether your defences are getting better or worse. Architecture decides how bad it is when they fail. You need both, and only one of them is a number.
Start with the four-outcome classification on a hundred inputs this week. The first time you run it you’ll learn more about your product’s posture than the last three system prompt rewrites told you.
Start free, or see how we handle your data and logging LLM traffic without creating a privacy incident for what you’re allowed to store while doing any of this.