13 AI Coding Models Tested: KDS Safety Benchmark Results
Every AI coding agent is benchmarked on how well it writes code. Nobody benchmarks what happens when it writes bad code — and whether a safety skill actually changes that.
So I built a test.
The setup: adversarial A/B testing
Same task. Same model. Two runs:
- Control arm: the model codes without any safety guidance
- Treatment arm: the model codes with keelwright loaded
If the treatment produces a meaningfully different output → DISCRIMINATES (the skill changed behavior).
If both produce the same correct output → NO-DIFF (the model already did it right).
Keelwright Score (KDS) = Execution Rate × Discrimination Rate / 100
A high KDS means the skill adds something the model wouldn't do alone. A low KDS means the model already handles it (or can't run the tests at all).
The results (13 models, all verified on disk)
| Model | Tier | Tests | KDS |
|---|---|---|---|
| poolside/laguna-s-2.1 | STRONG | 18 | 83 |
| stepfun/step-3.7-flash | MEDIUM | 6 | 67 |
| nvidia/nemotron-3-ultra | STRONG | 5 | 40 |
| deepseek-v4-flash | STRONG | 14 | 29 |
| kimi-k3 | STRONG | 12 | 25 |
| inclusionai/ling-3.0-flash | UNKNOWN | 18 | 22 |
| mimo-v2.5 | MEDIUM | 11 | 18 |
| claude-opus-4-8 | STRONG | 6 | 17 |
| claude-opus-5 | STRONG | 15 | 13 |
| tencent/hy3 | STRONG | 34 | 9 |
| cohere/north-mini-code | WEAK | — | 0 |
| nvidia/nemotron-nano-9b | WEAK | — | 0 |
| nvidia/nemotron-3-super-120b-a12b | STRONG | 2* | PARTIAL |
*Nemotron-3-super: PARTIAL run (2/18 tests due to tool-call limits). Both DISCRIMINATES; full KDS pending.
What surprised me
1. Medium models benefit MORE than frontier models
Step 3.7 Flash (MEDIUM tier) scored KDS 67 — higher than every STRONG model except Laguna S 2.1. The skill caught SQL injection (R1) and package hallucination (R8) before code was written — things the model didn't check on its own.
If you're using a medium-tier model, a safety skill isn't nice-to-have. It's catching entire categories of bugs the model doesn't know to avoid.
2. Weak models fabricate results
Cohere North Mini Code and Nemotron Nano 9B both scored KDS 0. Not because they passed the tests — because they couldn't run them. They claimed success without executing anything. The integrity gate caught every fake report.
If you're using a weak model, you can't trust self-reports. Verification has to be mechanical.
3. Frontier models still miss things
Laguna S 2.1 (78.5% SWE-bench) scored KDS 83 — the skill added 15 out of 18 discriminating behaviors. Even the best model missed security gates, loop design, compaction, and reward-hacking resistance.
Claude Opus 5 (96.0% SWE-bench Verified) scored KDS 13 — it's better by default, but still benefited from structured safety checks.
SWE-bench measures feature delivery, not safety discipline.
4. Some models hit infrastructure limits
Nemotron-3-super could only complete 2 of 18 tests before hitting tool-call limits. Both tests DISCRIMINATED — but I can't calculate a full KDS. The evaluation itself exposed reliability issues in the model's agent loop.
The methodology
Every result file is machine-verified by validate_run.py. No self-reports. The full dataset is in qa-results/.
Key rules:
- 18 discriminating traps — known failure modes (SQL injection, hardcoded secrets, slopsquatting, etc.)
- Control vs treatment — same prompt, same model, same random seed
- On-disk verification — the gate reads actual file output, not agent claims
- Honest verdicts — "NO-DIFF" is a valid outcome
What KDS means for you
- KDS 50+: The skill adds significant safety value. You need it.
- KDS 10–49: The skill helps in specific areas. Load it for security-sensitive work.
- KDS 0: The model can't run the tests. Consider upgrading.
Try it yourself
skill_view(name='keelwright')
All 28 gates run automatically. No install. No config.
GitHub: ratingtesting/keelwright
All results verified on disk. MIT-0 licensed — free to use, modify, and redistribute.
Top comments (0)