A few months ago, I wrote about a feeling I still have today: AI models, and especially coding agents, no longer give me the same sense of huge leaps that they used to.
I am not saying they are not improving. Newer models usually make fewer mistakes, follow instructions better, and sometimes solve problems that older versions could not. But it is becoming harder for me to feel those improvements as a real jump in capability.
Recently, I saw a post arguing, more or less, that at this point models are no longer better or worse than each other, but simply have different behaviors, and that what is actually good or bad is the harness controlling them. It even suggested that if someone is still complaining about current models, they probably do not know how to use or control them properly.
I agree with part of that idea.
But I think taking it that far is a mistake.
The harness matters a lot
First, we need to separate things that are often thrown into the same bucket.
A model is not an agent.
GPT, Claude, Gemini, GLM, or any other LLM is only one part of the system. Around the model there is an entire infrastructure: tools, context management, system prompts, planning, retries, and many other things.
That is, simplifying a lot, the harness.
And after spending time building my own coding agent, I am even more convinced that this layer matters enormously.
You can put the exact same model inside two different products and get completely different experiences.
One agent may manage context better than another. It may have better tools, recover from errors more effectively, or provide the model with more relevant information for completing a task.
All of that changes the result.
So yes: simply saying “this model is bad because it performed badly inside X agent” can be unfair.
But there is a huge distance between that and saying that models are no longer better or worse.
The result does not depend only on the harness
To me, the result of an agent looks more like this:
Result = model + harness + context + tools + instructions + task
Obviously, this is not a real mathematical formula, but it is useful for understanding the problem.
Change the harness, and the result changes.
Change the model, and the result changes too.
I have seen this many times while using different models in similar environments.
I have tried models that, on paper, were supposedly comparable to frontier models, but in the same kinds of projects they simply performed worse. They needed more corrections, forgot instructions more often, or made worse decisions.
The harness was still practically the same.
What changed was the model.
That is why I do not think we can reduce every current difference between models to simple “behavior.”
If a certain behavior causes one model to complete a task correctly much more often than another, that difference eventually becomes practical capability.
The question is no longer whether they can do it
I think this is the most interesting part of the whole discussion.
A few years ago, the question was:
Can an AI do this?
Can it build an application? Can it modify multiple files? Can it understand an error and fix it?
Today, for many of those questions, the answer is simply yes.
The problem is now different:
Can it do it correctly and consistently?
Because the fact that a model can do something does not mean you can trust it to do it correctly every time.
And that is where I still see major problems.
A very simple example: do not use alert()
Recently, I was working with Claude Code on an existing project.
I explicitly told it that I did not want to use alert() for confirmations. The application was supposed to use a modal.
Claude Code understood the instruction.
In the section we were working on, it correctly replaced the alert() with a modal.
But later, in another related part of the system, it left another alert() in place.
And I think this is a pretty good representation of the current state of many agents.
The system knew how to do the job.
It had understood my instruction and had even demonstrated that it knew how to implement the correct alternative.
And yet it failed to maintain that decision consistently across the project.
What makes this example interesting is that I was not using some experimental harness I had just built. I was working with Claude Code, a tool specifically designed to work agentically on codebases.
So an important question appears:
Whose fault was it?
It could have been my fault if I had never given the instruction.
It could have been the harness if that instruction had disappeared from the context.
It could have been an agent problem if it never inspected the other relevant parts of the project.
But if the model has the instruction, has access to the necessary code, and still ignores an explicit constraint, there is a point where we have to accept something pretty simple:
the model made a mistake.
Not every failure can automatically become a harness failure.
A good harness can hide a model's limitations
I think there is an important source of confusion here.
If we know that our models often forget certain constraints, we can build systems that review their work, run tests, retry failed attempts, or repeatedly reinsert important project rules into the context.
All of that can improve the final result dramatically.
But there is something we should not confuse:
we did not necessarily make the model smarter.
We built mechanisms to detect, prevent, or correct its mistakes.
And that is progress too. In fact, it is an extremely important part of building agents.
But a good harness often does not remove the model's limitations.
It compensates for them.
And if it works well enough, it can even make those limitations almost invisible to the user.
This also makes measuring progress more complicated.
If a new version of an agent feels much better, did the model itself really improve? Did context management improve? Did they add a review phase? Does it now retry more often before giving up?
From the outside, all of that often gets summarized as “the AI got better,” even though technically those are very different things.
From capability improvements to reliability improvements
I think this is one of the reasons why I feel that models are advancing more slowly.
During some stages of LLM development, the jumps were very easy to notice.
Before, a model simply could not do a certain task.
Then it could.
That is a qualitative jump.
The current feeling, at least for me, is different.
A lot of the progress now looks more like the model making slightly fewer mistakes, needing fewer attempts, or staying on track for longer tasks.
That is still progress.
And in production, it can be extremely valuable.
But it does not feel like having a model that is ten times more intelligent.
It feels like having one that is less fragile.
Maybe we are entering a stage where many of the visible improvements are improvements in reliability rather than entirely new capabilities.
That also explains why trying a new model sometimes leaves me relatively unimpressed.
It may be better.
It may make fewer mistakes.
But I still have to review its work, correct strange decisions, and verify that it actually did everything I asked.
The jump is still not large enough to completely change the way I work.
Building software from scratch is still easier than maintaining it
There is another limitation I keep seeing: models are much better at creating software from scratch than working on existing systems.
When a model starts a new project, it gets a huge advantage.
It can choose the architecture, establish its own conventions, and create the abstractions it needs.
In a sense, it is solving a problem inside a world that it just created itself.
Working on an existing project is completely different.
Now it has to understand decisions that were already made, find dependencies, respect conventions, and avoid breaking existing behavior.
It has to reconstruct something close to the mental model of a developer who already knows the system.
That is where I still see a lot of problems.
Sometimes the generated code is individually correct, but the complete change is still wrong.
One part uses the new implementation while another continues using the old one. A component gets changed but not every place that depends on it. Or an existing design decision gets broken somewhere else in the project.
The AI may understand every individual piece and still lose global coherence.
That is why I think one way of describing the current problem is:
LLMs are much better at generating code than maintaining invariants across complex systems.
And that matters a lot in real software engineering.
A new landing page can come out reasonably well from a single prompt.
Modifying an application that has spent months or years accumulating decisions, dependencies, and conventions is a different story.
And that is where I still do not feel we have seen the huge jump that is often advertised.
The problem with saying it is always the harness's fault
This is the part that bothers me most about the idea that if someone complains about current models, they simply “do not know how to control them.”
Because that can easily become an unfalsifiable position.
If the model works well, that proves the model is good.
If it performs badly, the harness was bad.
If someone gets worse results, they do not know how to use it properly.
At what point are we allowed to accept that the model itself simply has a limitation?
If every failure can be explained after the fact as a problem external to the model, then we have created a hypothesis that can never lose.
And that is not a particularly useful way to evaluate technology.
Of course we should improve our agents and learn to use models better.
But we should also be able to say:
The model failed here.
Without automatically turning the error into the user's fault.
And there is another point: when we need to build more and more layers around a model to make sure it completes a task correctly, those layers are also telling us something about the model's limitations.
The harness exists precisely because the model by itself is not enough.
That is not a criticism of the harness.
It is its reason for existing.
Prompt engineering did not disappear either
Something similar happened with the idea that models would become so intelligent that prompt engineering would no longer matter.
I do think it has changed.
I no longer think it is very useful to fill prompts with strange phrases or search for magical word combinations. Current models understand normal instructions much better.
But that does not mean structuring instructions stopped mattering.
If I want an agent to inspect an existing implementation first, respect certain constraints, and avoid modifying specific parts of the project, telling it that is still useful.
Providing relevant context and clearly defining the expected result still matters.
Maybe prompt engineering simply evolved into something larger: context engineering.
We are no longer thinking only about the sentence we type.
We are thinking about what information the model needs to know when it makes each decision.
And that is directly related to harness design.
Models do not need to feel radically different anymore
I also think there is another phenomenon at play.
Current models have already crossed a fairly high capability threshold.
Good models can code, understand existing code, use tools, and work through multi-step tasks.
Once all of them pass that minimum level, the differences stop feeling as dramatic.
A model that successfully completes 85% of my tasks and another that completes 90% can feel relatively similar during a conversation.
And both of them are still capable of doing something incredibly stupid five minutes later.
That remaining percentage of errors may also appear precisely in the hardest parts of the work.
So two statements that seem contradictory can both be true:
Current models are extraordinarily capable.
And:
Current models are still frustratingly unreliable.
I think both are true.
And maybe that is why many comparisons today focus more on small percentage improvements than on completely new capabilities.
That does not mean those improvements are irrelevant.
It simply means they do not all represent a major leap in intelligence.
So, are agents actually improving?
Yes.
But I think we should be more precise when we talk about those improvements.
Sometimes the model improves. Other times the harness improves, the tools improve, or simply our way of using them improves.
And usually, all of those things evolve at the same time.
Putting everything under a sentence like “AI is much smarter now” makes it difficult to understand what kind of progress we are actually seeing.
My current impression is that a significant part of the recent improvement in coding agents comes from building better systems around models that were already extremely capable.
That does not mean models stopped improving.
And it definitely does not mean that all models are equivalent.
It means that maybe the frontier we are trying to push has changed.
For years, we were trying to make models capable of doing new things.
Now the challenge may be getting them to reliably do the things we already know they can do.
Because an AI being able to write an application does not surprise me that much anymore.
What would surprise me is being able to leave it working for hours on a real codebase, come back, and find that it understood the existing decisions, preserved the important constraints, and did not leave behind a collection of small mistakes that I now have to chase down.
That, to me, would feel like another real leap.
Maybe the next major breakthrough in agents will not be getting them to do new things.
Maybe it will be getting them to do the things they already know how to do well, consistently, and without constant supervision.
Top comments (0)