I Built an AI Agent That Optimizes Code — But Benchmarks Decide What Survives

Gal B — Wed, 16 Sep 2026 10:03:01 +0000

I Built an AI Agent That Optimizes Code — But Benchmarks Decide What Survives

AI coding agents are getting very good at proposing changes.

The problem is that they are also very good at making changes that look like optimizations.

A refactor can appear cleaner.
A loop can look more efficient.
A data structure can seem like a better choice.

But there is only one question that really matters:

Did it actually make the code faster?

That is the idea behind autor3search.

GitHub:
https://github.com/autor3search

The core idea

autor3search lets an AI coding agent repeatedly experiment with performance optimizations.

But there is an important constraint:

The agent does not decide whether its own change is good.

Instead, every experiment is evaluated by a frozen benchmark and test harness.

The loop looks like this:

Agent proposes an optimization
        ↓
Agent changes the implementation
        ↓
Tests run
        ↓
Benchmarks run
        ↓
Measure the result
        ↓
KEEP or DISCARD
        ↓
Try another experiment

If the change improves the measured result and still passes the required checks, it can survive.

If it does not, it gets discarded.

Why freeze the benchmark?

This is probably the most important part of the design.

If an autonomous agent is allowed to modify both the implementation and the evaluation criteria, you can easily end up with something that appears faster without actually solving the original problem better.

For example, an agent could unintentionally:

Reduce the amount of work performed
Change the benchmark input
Remove validation
Optimize specifically for the benchmark
Modify test behavior
Exploit measurement noise

So the benchmark harness has to act as an external judge.

The agent can modify the implementation.

It cannot modify the rules of the game.

Inspired by autoresearch

The project is inspired by the idea behind Karpathy's autoresearch.

Instead of a human manually trying one experiment after another, an agent can continuously:

Form a hypothesis
Make a change
Run an experiment
Measure the result
Keep or reject the change
Continue searching

I wanted to explore what happens when you apply that same idea to ordinary software optimization.

Instead of optimizing a model-training experiment, autor3search searches for faster software implementations.

Example

Imagine the baseline benchmark is:

Baseline: 125 ms

The agent tries an optimization.

Experiment 1: 117 ms
Tests: PASS
Result: KEEP

Then another:

Experiment 2: 131 ms
Tests: PASS
Result: DISCARD

Then another:

Experiment 3: 108 ms
Tests: PASS
Result: KEEP

The important part is that the agent can keep searching without needing a human to manually evaluate every attempt.

The measurement system becomes the feedback loop.

The difficult part isn't generating optimizations

Modern coding models can generate plenty of optimization ideas.

The harder problem is creating a trustworthy evaluation loop.

Some of the questions I am currently exploring are:

Benchmark variance

A benchmark rarely produces exactly the same result twice.

If one run is:

100 ms

and the next is:

98 ms

is that a real improvement?

Or just noise?

A production-quality system probably needs multiple benchmark runs, statistical comparison, warm-up handling, outlier detection, and configurable acceptance thresholds.

Correctness

Performance cannot come at the expense of correctness.

An optimization that makes the program 30% faster but changes the output is obviously not an optimization we want to keep.

So tests need to remain part of the acceptance criteria.

Benchmark gaming

Autonomous agents are extremely good at optimizing toward whatever signal you provide.

That is useful, but it also means the evaluation system has to be carefully designed.

If there is a shortcut in the benchmark, eventually the agent may find it.

Stopping criteria

When should the search stop?

After:

A fixed number of experiments?
A certain amount of time?
No improvement for N experiments?
A performance target is reached?
The expected improvement becomes too small?

This becomes an interesting search problem by itself.

Multi-language support

I currently have implementations targeting:

Go
Rust
Python
TypeScript / JavaScript
Java
C#

One of the things I want to explore is how differently agents optimize across languages.

For example, optimization strategies in Python may be completely different from what works in Rust or Go.

Why I think this is interesting

Most AI coding tools currently operate like this:

Human asks for optimization
        ↓
AI writes code
        ↓
Human reviews it

I am interested in a slightly different model:

Human defines the goal and evaluation
        ↓
Agent performs many experiments
        ↓
Benchmarks provide feedback
        ↓
Agent searches autonomously
        ↓
Human reviews the best result

The human moves from manually directing every optimization to defining the objective and the constraints.

That feels like a much more interesting use of coding agents.

Open source

autor3search is open source and still early.

You can find it here:

https://github.com/autor3search

I would especially appreciate feedback around:

Benchmark methodology
Preventing agents from gaming evaluations
Statistical confidence
Experiment isolation
Search strategies
Real-world repositories that would make good test cases

If you've worked on performance engineering, coding agents, compilers, benchmarking, or autonomous development workflows, I'd be very interested to hear how you would design this system.

DEV Community: Gal B

I Built an AI Agent That Optimizes Code — But Benchmarks Decide What Survives

I Built an AI Agent That Optimizes Code — But Benchmarks Decide What Survives

The core idea

Why freeze the benchmark?

Inspired by autoresearch

Example

The difficult part isn't generating optimizations

Benchmark variance

Correctness

Benchmark gaming

Stopping criteria

Multi-language support

Why I think this is interesting

Open source