LLM Evaluation That You Can Actually Audit: Inside the Refusal Boundary Benchmark

TL;DR — LLM evaluation is only useful if you can re-run it and re-read the answers. The LFORLA Refusal Boundary benchmark makes that possible by fixing the prompts, fixing the judge, scoring per axis, and publishing the verbatim trail. Here is how the numbers are made, what the current leaderboard actually says, and how to read it without getting fooled.

How the evaluation works

The Refusal Boundary benchmark is a paired benign/harmful refusal audit. It measures two things at once: whether a model is helpful on legitimate trigger prompts, and whether it refuses disallowed requests. The boundary score is the harmonic mean of those two sides, so a model cannot win by being maximally helpful or maximally safe — it has to balance both.

Here is the mechanism, step by step.

1. Same prompts for every model. Each model receives an identical set of paired prompts: a benign version that should be answered helpfully, and a harmful version that should be refused. No model gets a friendlier prompt set than another.
2. Same fixed judge. A single judge configuration scores every response. The judge does not change between models or between runs, which removes a common source of drift in LLM evaluation.
3. Per-axis rubrics. Responses are scored on five axes: Boundary F1, Harmful Refusal, Over-Refusal, Benign Helpfulness, and Under-Refusal. Each axis has a defined range and direction, so a high score always means the same thing.
4. Public verbatim trail. Every prompt and every model answer is published. Anyone can re-read what the model actually said and check whether the judge got it right.

That combination — fixed inputs, fixed judge, explicit rubrics, open transcripts — is what makes the numbers comparable rather than merely impressive.

The real numbers

Model	Score	Provider
Llama 3.1 8B Instruct	92.9	local
Qwen2.5 7B Instruct	90.56	local
Mistral 7B Instruct	84.33	local
Llama 3.1 8B Abliterated	49.59	local
Dolphin Mistral 7B	45.45	local
Qwen2.5 7B Abliterated	31.89	local

We also submitted our own model, GLM 5.2, which scored 78.0 on this benchmark.

Why the ranking looks this way

The top three models are instruction-tuned base models. They sit in the 84–93 range because they refuse harmful requests while still answering benign ones. The harmonic mean punishes imbalance, so a model that refuses everything or answers everything cannot reach the top.

The bottom three are abliterated or uncensored variants. Their scores fall between roughly 32 and 50. That is not a bug in the benchmark; it is the boundary doing its job. Removing refusal behavior tends to push models toward under-refusal on harmful prompts, which drags down the harmonic mean even if benign helpfulness stays high.

The gap between Llama 3.1 8B Instruct at 92.9 and Llama 3.1 8B Abliterated at 49.59 is the clearest illustration: same base family, very different boundary behavior. In practice, the score tells you how well a model holds the line between helpful and safe, not how smart it is.

What this means for practitioners

If you are choosing a model for a user-facing product, the boundary score is a useful first filter. A model in the high 80s or 90s is likely to answer legitimate questions without over-refusing, while still declining disallowed requests. A model in the 30s or 40s may be appropriate for research or controlled environments where refusal behavior is intentionally relaxed, but it is a risky default for open-ended deployment.

GLM 5.2 at 78.0 sits in the middle: better boundary behavior than the uncensored variants, but below the top instruction-tuned models on this specific audit. That is exactly the kind of trade-off a per-axis benchmark is supposed to surface.

How to read a leaderboard

• Check whether the prompts and judge are fixed and published; if not, treat the ranking as a demo, not a measurement.
• Look at the axes, not just the overall score. A high overall number can hide over-refusal or under-refusal.
• Re-read a few verbatim answers yourself. The trail exists so you can disagree with the judge.
• Compare models within the same family and size before comparing across them.
• Treat any single benchmark as one signal among several, not a final verdict.

Honest limitations

This benchmark measures refusal boundary behavior, not general capability, reasoning, or factuality. The judge is fixed, but a fixed judge can still be wrong; the public trail lets you catch that, but it does not eliminate it. The current leaderboard is all local models, so hosted API models are not represented here. And a harmonic mean is sensitive to the weaker axis, which is intentional but means one bad axis can dominate the score.

Conclusion

LLM evaluation is most trustworthy when it is boring: same prompts, same judge, clear rubrics, open transcripts. The Refusal Boundary benchmark follows that recipe, and the current leaderboard shows a clean separation between instruction-tuned and abliterated models. You can explore the full results and the verbatim trail at lforla.org.