Every week there's a new benchmark.
Claude beats GPT.
GPT beats Gemini.
Gemini beats Claude.
A new open model arrives and suddenly "everything changes."
After a while, I realised I was asking the wrong question.
Instead of asking:
Which model is better?
I started asking:
Which model actually gets the engineering task done?
The answers were surprisingly different.
Benchmarks Don't Ship Software
Most model evaluations focus on intelligence.
- Coding benchmarks
- SWE Bench
- HumanEval
- MMLU
- LiveCodeBench
These are useful, but they only measure one part of the problem.
Real engineering looks more like this:
Receive task
↓
Understand repository
↓
Install dependencies
↓
Navigate existing architecture
↓
Edit multiple files
↓
Run tests
↓
Fix failures
↓
Repeat until green
↓
Create clean commit
The model only owns one of those steps.
The Real Bottleneck Isn't Intelligence
While building an engineering execution runtime, I kept seeing the same pattern.
The models were usually good enough.
The execution wasn't.
Agents would fail because they:
- couldn't resolve dependencies
- edited the wrong files
- lost context after several iterations
- entered infinite edit loops
- couldn't recover after a failed test
- produced changes they never validated
None of these failures happened because the model couldn't write code.
They happened because software engineering is much bigger than code generation.
Cost Per Token Is the Wrong Metric
People love comparing pricing.
"$X per million tokens."
But that's not what companies actually pay for.
Imagine two models.
Model A
- Costs £2
- Finishes the task first try
Model B
- Costs £1
- Needs four retries
- Requires manual intervention
- Fails validation twice
Which one was cheaper?
The invoice says Model B.
Your engineering team says Model A.
The metric that matters is:
Cost per successfully completed engineering task.
Everything else is secondary.
Orchestration Changes Everything
Something interesting happened once I stopped evaluating models in isolation.
The differences between models became much smaller.
The differences between execution systems became much larger.
Things that mattered more than swapping models included:
- deterministic execution
- validation pipelines
- rollback strategies
- isolated environments
- repository awareness
- tool selection
- context management
- recovery after failures
A great model inside a poor execution system still produces unreliable engineering.
A good model inside a great execution system often produces excellent engineering.
Benchmarks vs Production
A benchmark rewards intelligence.
Production rewards reliability.
Those aren't the same thing.
An engineering team doesn't care if a model scores 96%.
They care whether today's pull request merged without someone spending an hour fixing the output.
Where I Think AI Engineering Is Heading
I don't think the next wave of innovation comes from finding "the smartest model."
It comes from building better execution systems around them.
Models are becoming commodities.
Reliable engineering workflows are not.
The companies that win won't necessarily own the smartest model.
They'll own the systems that can consistently turn model output into production-ready software.
That's a much harder problem.
And, in my opinion, it's also the more interesting one.
What do you optimise for today?
- Benchmark scores?
- Token cost?
- Or successful task completion?
I'm curious whether others are seeing the same shift in their own engineering workflows.
Top comments (0)