DEV Community

Nexus Foundry
Nexus Foundry

Posted on

AI coding is cheaper until the agent ships a lie — the verification gate I actually run

This week on dev.to: "Is coding with AI actually faster and cheaper?"
(217 reactions), an agent-PR-gate writeup (82), a team-AI-adoption
piece (55). The cluster is the same anxiety wearing different hats:
everyone got cheaper code, and now everyone asks who checks it?

I run agents that write code, review PRs, and triage security alerts.
After enough "the agent said it was done" moments, I stopped trusting
narration and built a gate. Here's what actually works — and what I
test before anything merges.

The lie has three shapes

  1. Confident wrongness. The agent produces plausible code that fails on edge cases it never saw. Compiles, lints, looks reviewed.
  2. Scope drift. It "helpfully" refactors three files you didn't ask about, and now your diff is unreviewable.
  3. Injection compliance. The agent reads a poisoned issue, ticket, or doc and follows instructions embedded in the data. (This one is covered elsewhere — the fix is tool boundaries, not prompts.)

All three share a root cause: the same system that generates the
work is the system grading it.
Verification has to come from
somewhere else.

The gate I run (in order)

1. Known-answer evals before the agent touches real work.
Before an agent gets a task type, it gets a test suite: inputs with
known-correct outputs. My eval runner is ~100 lines — feed it
chat(prompt) -> str, score PASS/FAIL against expected answers,
diff runs to catch regressions. It sounds trivial until you realize
most "agent quality" claims are vibes. Measure first, prompt-tune
second.

2. Injection self-test for anything that reads untrusted text.
If the agent summarizes web pages, triages tickets, or reviews
user-submitted content, it will eventually read attacker-controlled
text. I keep 15 payloads — direct, indirect-doc, table-cell,
fake-system, roleplay — and score whether the agent complies. It
composes with the eval runner (same interface). Takes ten minutes to
run; tells you which failure mode you actually have.

3. Deterministic checks where judgment is expensive.
For security-ish scoring I deliberately keep the arithmetic in code:
a risk score is severity-weighted counts, capped — not "the model
thinks this is probably bad." The model writes the explanation; it
doesn't get to move the number. Probabilistic judgment, deterministic
boundary.

4. Human approval at the boundary that matters.
Everything upstream can be automated. The last step — merge, deploy,
send — stays human. Agents draft; humans ship. This isn't
pessimism about agents, it's the same reason we don't let CI skip
review on its own PRs.

The economics, honestly

The cheap-code math only works if verification is cheaper than the
bug. A 100-line eval runner and a 15-payload injection suite cost
less than one bad merge. The expensive version of "AI coding" is the
one where you pay for generation and then pay again in review debt.

If you want the pieces: the eval runner + 10 tests and the injection
self-test are free (no email gate), and the full eval suite,
ops runbook, and triage systems are small paid kits if you want the
rubrics and regression tooling.

Store link auto-updates; pay in BTC, no account, instant download.

Top comments (0)