DEV Community

Raymond Chang
Raymond Chang

Posted on

The AI Coding Workflow Everyone Teaches Fooled Me 3 Times

A handful of talks from the AI Engineer conference on how to code with AI have been watched about six million times. I watched them, took notes, and I do most of what they say.

Put the talks side by side and you get four boxes. A human decides. The AI builds. Something checks it. Whatever breaks goes back in for the next lap.

I run that loop on two projects. One is a trading desk that only sends orders to a sandbox. The other is a security lab where five AI agents are building a zero-knowledge proof toolchain. Together they have over 8,000 tests.

My AI still fooled me three times, with every test green. All three happened in the third box.

1. A review of the summary

One Friday my trading desk crashed on launch. The bug had been in the code for eighteen days, and a test aimed right at that code had been green the whole time.

I did have a checker. A second AI reviewed the first one's work. It never opened the code. It was grading a report about the code.

2. Every rule tested alone

I added five new kinds of trades to the desk. The AI wrote the checks for them, and thousands of tests passed. Then I tried to place one in the sandbox.

So I made the AI take real data, try every way in, and count how many candidates got past all the rules at once. 34,622 tries, zero winners.

Every rule had its own passing tests. No test asked whether a single real trade could get through all of them together.

3. Nobody tested the no

In the lab, the AI wrote a proof verifier. Its one job is to say no to a bad proof. The tests came back green.

I fed it random bytes and it said valid. The two tests that would have caught that were sitting in the test file. They never ran. I wrote that one up separately: I Fed My AI-Built Proof Verifier Garbage. It Said Valid Every Time.

The four checks I run after green

  1. Test the no. Hand your checker exactly what it should reject: a garbage input, a number changed by one digit. Then open the test report and confirm those tests ran.
  2. Prove one real case gets through. If you have a stack of rules, take real data and make sure at least one case passes all of them at once.
  3. The reviewer opens the real thing. Whoever signs off reads the code itself. Matt Pocock's reviewer reads the actual commits. Mine was reading a summary.
  4. Run it for real. Real parts, the way you would use it. My desk has a practice mode in the sandbox, and it found three problems that thousands of tests missed.

Two of these come straight from the talks. I wasn't doing them right.

What I still haven't fixed

On the trading desk, the reviewer only sees the real file when I paste it in, and that doesn't happen every round.

Review reports now carry a file name and a line number, and anything nobody checked gets labeled unchecked. The script that would verify those automatically isn't built yet.

The talks

All from the AI Engineer conference, and worth your time:

Follow along

New episodes go up on my YouTube channel first, with the desk, the lab and the terminal on screen:

Subscribe to Dr_Delta on YouTube

What's the one check you run after the tests go green? Tell me in the comments.

Top comments (0)