Over the last few months, I've been building an engineering runtime for AI coding agents called FROST.
Ironically, while building it, I realized I was solving the wrong problem.
Like everyone else in the AI tooling space, I initially obsessed over things like:
- Context windows
- Token reduction
- Multi-agent systems
- Branching strategies
- Benchmarks
- More tools
The assumption was simple:
AI coding agents fail because they aren't intelligent enough.
I don't think that's true anymore.
The Problem
Let's take a real engineering task.
Upgrade a production FastAPI repository from 2023 standards to 2026 standards.
That sounds straightforward until you realize what that actually means.
The repository needs to evolve from:
Python 3.10
↓
Python 3.14
Pydantic V1
↓
Pydantic V2
SQLAlchemy 1.4
↓
SQLAlchemy 2.0
Legacy cryptography libraries
↓
Modern cryptography stack
Old Docker configuration
↓
Modern Docker configuration
Old dependencies
↓
Latest ecosystem versions
And that's before you even run the tests.
Suddenly you're dealing with:
- Breaking API changes
- Dependency conflicts
- Migration strategies
- Compatibility layers
- Database issues
- Threading issues
- Documentation updates
- CI failures
- Multiple valid engineering approaches
None of this is a prompting problem.
It's an engineering execution problem.
AI Coding Agents Are Surprisingly Bad at This
Most AI coding agents are excellent at writing code.
They're much worse at evolving codebases safely.
A typical workflow looks something like this:
Run tests.
↓
47 failures.
↓
Try random fix.
↓
31 failures.
↓
Try another fix.
↓
Repository is now partially broken.
↓
Context window is polluted.
↓
The agent forgets why it made the previous decision.
↓
Start over.
After enough iterations, the repository becomes harder to reason about than when we started.
The problem isn't intelligence.
The problem is that repository evolution is inherently uncertain.
The Biggest Thing I Learned Building FROST
The biggest realization I had while building FROST was this:
Difficult engineering tasks are execution problems, not reasoning problems.
We don't need a smarter model to know that Pydantic V2 exists.
We need infrastructure that can safely answer questions like:
- Should we add a compatibility shim?
- Should we refactor the public API?
- Should we pin a dependency temporarily?
- Should we rollback this migration?
- Should we branch into multiple approaches?
- Have we already tried this fix before?
These are engineering questions.
Repository Evolution Isn't Linear
A repository migration rarely looks like this:
Task
↓
Fix problem.
↓
Done.
It looks more like this:
Upgrade repository.
↓
Tests fail.
↓
Engineering uncertainty detected.
↓
Explore alternatives.
↓
Kill losing approaches.
↓
Merge the winning solution.
↓
Continue execution.
↓
Next uncertainty point.
↓
Repository is green.
Repository modernization is essentially a series of uncertainty points.
That's where AI coding agents tend to struggle the most.
The Real Dogfooding Test
Instead of building synthetic benchmarks, I cloned an older release of the FastAPI Full Stack Template from early 2023 and attempted to modernize it.
The initial state looked something like this:
Python 3.10
Pydantic V1
SQLAlchemy 1.4
Legacy cryptography stack
54 tests failing under modern Python environments.
During modernization we encountered:
- Pydantic V2 schema issues
- SQLAlchemy 2.0 relationship problems
- UUID deserialization failures
- SQLite threading issues
- Cryptography incompatibilities
- Test failures caused by newer Python versions
The final result:
Python 3.14
Pydantic V2
SQLAlchemy 2.0
Modern cryptography stack
54 / 54 tests passing.
The interesting part wasn't that the tests passed.
The interesting part was that none of the problems required a more intelligent model.
They required safe repository evolution.
Token Reduction Isn't The Product
For a long time, I marketed FROST internally as:
- Compression
- Checkpointing
- Branching
- Loop detection
- Token reduction
Eventually I realized nobody cares.
Nobody is paying for 95% token reduction.
People pay because:
Claude Code spent 45 minutes refactoring my production codebase and didn't lose its mind halfway through.
Everything else is an implementation detail.
My Current Thesis
Today, my thesis is much simpler.
AI coding agents don't have a reasoning problem. They have an execution problem.
The next generation of AI developer tools probably won't win by being smarter.
They'll win by becoming better at:
- Repository evolution
- Failure recovery
- Engineering uncertainty detection
- Progress preservation
- Safe migrations
- Long-running engineering execution
I suspect we'll eventually stop thinking about AI coding agents as code generators and start thinking about them as software engineers that need good engineering infrastructure underneath them.
That's the problem I'm interested in solving.
I'm curious whether others building or using AI coding agents have noticed the same thing.
Are your biggest failures actually reasoning failures, or are they engineering execution failures?
Top comments (0)