DEV Community

Raymond Chang
Raymond Chang

Posted on

I Fed My AI-Built Proof Verifier Garbage. It Said Valid Every Time.

I build tools with Claude Code and then go looking for what the AI got wrong. This round, the tool was a zero-knowledge proof library: you prove a number about your data, say an average, and someone else checks it without seeing the data.

The whole thing lives or dies on one function, the verifier. Its one job is to say no when the proof is bad.

The AI delivered a Rust part, a Python part, a test file and docs. The tests came back green.

The tests that never ran

The test file had two tamper cases in it. Both were marked should fail. Neither one ever ran.

So the green run said nothing about tampering. It's a smoke detector test where nobody lights a match.

Four bad proofs, four VALIDs

My coding assistant copied the old verify code into a quarantine folder and wrote a small harness that feeds it four cases:

  • Cases 1 and 2: the same 128 random bytes as the proof, once with the real public mean and once with the mean changed by 1.0. Verdict: valid, and valid.
  • Case 3: what the library's simulation mode calls a proof. It is a SHA-256 hash of the public numbers, checked against a verification key that is the literal bytes mock_verification_key. Anyone can compute that hash with no data at all. Verdict: valid.
  • Case 4: the same random bytes again, with the changed mean, through the third source's verify path. Verdict: valid.

The first line the library printed, before any of this:

Warning: Rust backend not available, using simulation mode
Enter fullscreen mode Exit fullscreen mode

One warning, then valid to everything.

On the desk and on the phone

A terminal is easy to ignore, so I put it on my desk. A Stream Deck key called GARBAGE sends random bytes to that old verify code. The answer comes back in a window: LEGACY: VALID.

A second key shows a QR code. My phone scans it and opens a verify page from the same server. Big green check, VERIFIED, for any input you send it.

Nothing here is a real proof. The numbers are 20 synthetic trades from a fixed seed. I took no measurements this round.

Steal this

When you test a checker, write the "no" tests first, and make sure they run:

import os

def test_rejects_random_bytes(public):
    assert not verify(os.urandom(128), public)

def test_rejects_changed_number(proof, public):
    public = dict(public, mean=public["mean"] + 1)
    assert not verify(proof, public)
Enter fullscreen mode Exit fullscreen mode

Then check your test report for the word "skipped" or a test count lower than you expect. A tamper test that never runs looks exactly like a tamper test that passes.

If the checker says yes to garbage or to a number you changed by one digit, it's broken.

Follow the lab

Security Lab episodes go up on my YouTube channel first, with the terminal, the desk and the phone on screen:

Subscribe to Dr_Delta on YouTube

Have you ever had a test suite go all green when it never checked the failure case? How did you find out? Tell me in the comments.

Top comments (1)

Collapse
 
dr_delta234 profile image
Raymond Chang •

Author here. Some context: this is a library my AI coding assistant wrote, not something in production. The legacy verify path was moved into a quarantine folder and run through a small harness. All numbers are synthetic, and the library was in simulation mode the whole time, which is part of the point: it printed one warning and then accepted everything.

The part I think generalizes: the test file had two should_fail tamper cases that never executed, so the green run told me nothing about rejection. Happy to answer questions about the harness or the Stream Deck setup.