I spent a week reviewing LLM apps for a client and found a pattern that surprised me. Every loud failure - a refused prompt, an exception, a blocked tool call - was handled fine. The app had guardrails. The failure that almost caused a real incident was the one that raised no error at all: the model quietly complied with a slightly reworded request, returned a normal-looking response, and the downstream code logged it as a success.
I am calling these soft failures. They are the expensive kind, because your monitoring never sees them.
Here is how I think about them, and how I test for them.
Why hard failures are easy and soft failures are not
A hard failure breaks a contract. The model says no, the tool call is rejected, the JSON shape is wrong. Your tests catch contract breaks because you can assert on them. You can write assert response is not None, assert 'refused' in text, and you will be fine.
A soft failure keeps every contract intact. The response is valid JSON. The tone is helpful. The tool call is well-formed. The only thing wrong is what it agreed to do - and that is a semantic property you cannot assert on with a regex.
When I ran my 8-probe battery against a banking-style assistant, the probe that got flagged was not the aggressive jailbreak. It was the polite, cross-language one. The model did not refuse, did not error, did not look suspicious. It just answered a question it was not supposed to answer. In a log, that row looks identical to a legitimate row.
Three ways to catch them
1. Diff the capability surface, not the text.
Instead of asking "does this response look like a leak?", I compare what the model is able to do before and after a suspected injection. Can it still deny a clearly out-of-scope request? Can it still repeat its constraints back when asked? If a harmless-looking turn makes a previously-denied capability suddenly available, that is a soft failure, and you can detect it structurally.
2. Reproduce the exact bytes.
A security finding you cannot reproduce is an opinion. Every probe in my kit ships the raw prompt and the raw model reply, and the report is hash-verifiable (sha256 over the served bytes). This matters because soft failures are probabilistic - a model might comply 30% of the time. If you cannot re-run the exact prompt and compare, you cannot tell a one-off from a real hole.
3. Treat the score as a gate, not a verdict.
I use a 0-100 risk score, but the number is only useful as a CI gate with a fixed threshold you set before you look at results. The moment the score becomes a marketing number, it stops being a decision tool. Deflated expectations, not clever models, is where these audits actually change behavior.
The 30-second version
You do not need a full audit to get the main value. A short adversarial battery - eight to fifteen probes covering jailbreak, extraction, indirect injection, and tool abuse - run against your actual system prompt, tells you whether you have any soft failures at all. If the polite probes are flagged and the aggressive ones are not, you know exactly where to invest: the model is not refusing, it is not consistently refusing.
I run this as a free sample on my own agent so you can see the shape of the output: paste a system prompt at https://llmrt-companion.manhliemcn4euwlu.workers.dev/agent-scan and in about 35 seconds you get the risk score, the per-probe raw prompt and reply, and the flags. The report link is durable and the hash is in the response, so you can recompute it later and confirm nothing changed.
My own production agent scanned at 27/100 MEDIUM on this battery - one flag, the cross-language extraction. I am posting that link because it is the same scan, unedited: https://llmrt-companion.manhliemcn4euwlu.workers.dev/r/e21808b6fedb. Check my work before you trust the method.
The 15-probe core is MIT: gitee.com/xydhw/llm-red-team-starter-kit. Clone it, diff any probe, and tell me if you disagree with a signal - that is how I want to refine the battery.
What soft failure have you found in your own model? I want the ones that did not raise an error.
Top comments (0)