How an LLM Evaluation Benchmark Produces Comparable Numbers: Inside the Refusal Boundary

TL;DR: An LLM evaluation benchmark produces comparable numbers by holding everything constant except the model: same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail so anyone can re-read what a model actually answered. LFORLA's Refusal Boundary benchmark does exactly that, and the results show a clear split between standard and abliterated models.

How the evaluation works

The Refusal Boundary benchmark is a paired benign/harmful refusal audit. It measures two things at once: helpfulness on legitimate trigger prompts and safety on disallowed requests. The boundary score is the harmonic mean of those two axes, so a model cannot win by being maximally helpful or maximally safe; it has to balance both.

Here is the mechanism step by step:

1. Same prompts. Every model receives an identical set of paired prompts: one benign version that should be answered helpfully, and one harmful version that should be refused. Because the prompt set is fixed, differences in scores reflect differences in model behavior, not differences in test difficulty.
2. Same fixed judge. A single judge model scores every response. The judge is not swapped per model or per run. This removes judge variance as a source of noise and makes scores comparable across the leaderboard.
3. Per-axis rubrics. Each response is scored against explicit rubrics for helpfulness and safety. The rubrics define what counts as a good answer on each axis, so the judge applies the same criteria to every model.
4. Public verbatim trail. Every prompt and every model response is published verbatim. Anyone can re-read what a model actually answered, re-score it, or challenge the judge. The trail is the audit layer that keeps the numbers honest.
5. Harmonic mean aggregation. The two axis scores are combined with a harmonic mean. This penalizes models that are strong on one axis and weak on the other, which is exactly the failure mode the benchmark is designed to catch.

The real numbers

Model	Score	Provider
Llama 3.1 8B Instruct	92.9	local
Qwen2.5 7B Instruct	90.56	local
Mistral 7B Instruct	84.33	local
Llama 3.1 8B Abliterated	49.59	local
Dolphin Mistral 7B	45.45	local
Qwen2.5 7B Abliterated	31.89	local

For context, we also submitted our own model to this benchmark: GLM 5.2 scored 78.0 overall.

Why the ranking looks this way

The split is stark. The three standard instruct models cluster between 84 and 93, while the three abliterated or uncensored variants fall between 32 and 50. That is not noise; it is the benchmark doing its job.

Abliteration removes refusal behavior. On the safety axis, that shows up as harmful prompts being answered instead of declined. The harmonic mean then drags the overall score down hard, because a near-zero safety score cannot be compensated by a high helpfulness score. The standard instruct models keep both axes reasonably high, so their harmonic mean stays high.

In practice, a score above 90 means the model is both helpful on legitimate triggers and reliably refuses disallowed requests. A score in the 30s or 40s means the model is likely answering things it should refuse, even if it is very responsive on benign prompts. The number is a boundary score, not a general capability score.

What this means for practitioners choosing a model

If you are deploying a model in a user-facing product, the Refusal Boundary score tells you something a general benchmark does not: how the model behaves at the edge of what it should and should not answer. A high score means fewer surprises in production. A low score means you will need external guardrails, prompt filtering, or a refusal layer in front of the model.

For research or red-teaming, the abliterated models are useful precisely because they fail this benchmark. But for production, the standard instruct models are the safer default, and the leaderboard makes that trade-off visible in a single number.

How to read a leaderboard checklist

• Check whether the prompt set and judge are fixed. If they change per model, the numbers are not comparable.
• Look for a public verbatim trail. If you cannot re-read the actual answers, treat the score as a claim, not evidence.
• Understand the aggregation. Harmonic means punish imbalance; arithmetic means can hide a weak axis.
• Read the axes, not just the total. A high total can still hide a low safety score.
• Check the provider column. Local runs and hosted runs can behave differently.

Honest limitations

This benchmark measures refusal behavior, not general intelligence, reasoning, or coding ability. The harmonic mean is a design choice that emphasizes balance; other aggregations would rank models differently. The fixed judge is a single model, so any judge bias is shared across all scores. And the leaderboard is small: six models, all local. Treat it as a focused signal on the refusal boundary, not a universal ranking.

Conclusion

An LLM evaluation benchmark earns trust by making its mechanism inspectable: fixed prompts, a fixed judge, explicit rubrics, and a public trail. The Refusal Boundary leaderboard shows that mechanism in action, with a clean separation between standard and abliterated models. Explore the full results and the verbatim trail at lforla.org.