Google mapped the complete male fruit fly brain, and my first question to an agent was “Can this run on a laptop?” The simulation has 176,422 neurons and runs on a laptop CPU.
My second question was “Is there something ridiculous that we can do with it, though? That’s engineering-focused: make it do a code review or clear context or something.”
The fly can’t read code, so Jev, a TypeSafe model, answers thirty yes/no questions about each change and the answers become a smell. I tried to train the fly on the codebase by using mutation testing to identify good and bad parts of the code and then train the fly to have memories based on that. On 301 changes it had never seen, it agreed with the tests 290 times. A logistic regression on the same thirty answers got 298.
If a fruit fly can do code review moderately well, then why are humans still doing it?
Reviewing code is the bottleneck. Humans aren’t involved in writing it anymore, and the functionality is what matters. Models are getting good, which means the code quality is getting good and we’re at the precipice of starting to let go of that control.
It’s a big trust issue for humans. Feels like giving up a lot of control but I think that humans are just as capable of slop as agents are. Agents are able to be persistent and review, given the right set of tools and the right context on the outcome.
I want evidence and a receipt that it tested the results and that what it’s proposing to ship actually solves the problem or adds the feature or addresses the issue in the way that’s intended. I want:
- Videos
- Screenshots
- Test results
- Command output
- Smoke tests
- An easy way for me to replicate it myself
On the last agent PR I approved, I looked at the most important bits. The part that made it most important was just where the core business logic was. Agents can verify that it’s logically correct, but we need to make sure that it makes sense for the use case or the application.
Humans should understand why code is changed and why the way that it’s changed matters, but at a higher level, not necessarily line for line. They should just have an understanding of what the code is doing and why, and specifically why not other methods.
Top comments (1)
The logistic-regression baseline is exactly the right comparison. I would split the evaluation by change type as well—tests, refactors, and behavioral changes—because 290/301 agreement may hide a weak slice that matters in review. A simple baseline that wins overall can still reveal where the representation needs work.