DEV Community

John Daniel Kearney
John Daniel Kearney

Posted on

I shut down my agent-safety toolkit to audit AI graders

#security #ai #testing #llm

Earlier this year I posted here about SafeClaw and Authensor's policy engine for AI agents. That product line is retired, and those posts now say so at the top. This is what Authensor works on instead.

Authensor is an AI evaluation integrity firm in San Francisco. It audits the tools the AI industry measures itself with: evaluation harnesses, benchmarks, LLM judges and scoring pipelines. The question for each one is whether the system being evaluated can move its own score.

Why the switch

Two of the early bugs I fixed upstream were in agent-control tooling. In April 2026, UK AISI's ControlArena passed tool-call arguments and chain-of-thought text into its monitor prompts unsanitized, so the agent being monitored could write straight into the prompt of the model watching it. The same pattern kept turning up in graders, judges and benchmarks. If the measurement can be steered by the thing being measured, the safety numbers built on it don't mean much, so that became the work.

The Evaluator Trust Boundary

The research is organized around one definition: an evaluator has an Evaluator Trust Boundary (ETB) failure when its score depends on an artifact the evaluated system can influence. The taxonomy names ten mechanisms, ETB-01 to ETB-10. The preprint is on Zenodo: 10.5281/zenodo.21633620.

Some of the cases now fixed upstream:

  • SWE-bench (August 2026, ETB-09): the grader counted PASSED lines that the patch under test could print itself. Fixed in PR #620.
  • Promptfoo (October 2026, ETB-03): an llm-rubric judge response with no verdict defaulted to a pass. Fixed in PR #10611.
  • UK AISI Inspect AI (June 2026, ETB-04): scoring metrics silently dropped errored and inconclusive samples from the denominator. Fixed in #4288.
  • NVIDIA garak (July 2026, ETB-06): string detectors did no Unicode normalization, so homoglyph and zero-width variants got through. Fixed in #1884.
  • Instructor (July 2026, ETB-01): JSON extraction took the first object in a code block, so planted JSON outranked the real output. Fixed in #2434.
  • UK AISI ControlArena (April 2026, ETB-02): tool-call arguments entered monitor prompts unsanitized. Fixed in #798.

The full list, with every report and fix, is at authensor.com/etb. Separately, the MLflow advisory for CVE-2026-64849 credits Authensor as an independent finder.

Check your own judge

etb-scan is a free, MIT-licensed scanner that measures ETB-01 (injected verdicts) against your own judge. It runs offline on the standard library, with no API keys.

pip install etb-scan
etbscan --judge mypkg:judge --trials 10
Enter fullscreen mode Exit fullscreen mode

If you've published an eval number and want it checked against the instrument that produced it, Authensor does that free within 48 hours: john@authensor.com.

The hardware side

15 Research Lab is my separate lab in San Francisco for embodied AI safety and experimental hardware. The same kind of bug shows up there. Yosys's formal exporters were silently dropping every assertion, so solvers reported PASS on designs that should fail. I reported it as #6108 and Zaneham fixed it in #6117. Write-up.

Links

Top comments (0)