When Claude Opus 5 and GPT-5.6 arrived, I expected my coding agents to become noticeably better.
Instead, some of my workflows felt worse.
The agents seemed more eager, less predictable, and occasionally “dumber” than before. Naturally, I blamed the new models.
Very scientific of me.
Maybe it was a skill issue
Then I watched Andrej Karpathy’s interview on the No Priors podcast. One idea stuck with me: when an agent fails, the capability may already exist. The problem could be how we instruct it, what memory we provide, or how we arrange the workflow.
That made me question something I had mostly ignored:
Were my custom skills still compatible with the newer models?
I gave my agent this prompt:
Can you audit our custom skills against the current models? Flag stale prompts, conflicting instructions, outdated assumptions, and anything that should be simplified or removed. Then test each skill on a representative task and propose the smallest updates needed.
The audit found instructions written around the behavior of older models. Some were redundant. Some were no longer necessary. Others pushed the newer models too hard and caused them to overdo tasks.
After cleaning those up and testing the skills again, the results felt noticeably better.
The official guidance supports this
This was not proof that every disappointing result is caused by an outdated prompt. Models can still regress, behave inconsistently, or introduce genuine breaking changes.
But both Anthropic and OpenAI recommend recalibrating instructions during model migrations.
Anthropic’s Claude Opus 5 documentation says the model now verifies its work without being told. It specifically recommends removing verification instructions carried over from earlier models because they can cause over-verification.
OpenAI’s GPT-5.6 guidance recommends removing repeated instructions, simplifying tool descriptions, and running the same evaluations after each change. In OpenAI’s internal coding-agent evaluations, leaner system prompts improved scores by roughly 10–15% while reducing token usage. OpenAI describes those numbers as directional and recommends validating them against your own workload.
Research also shows that prompt performance does not transfer perfectly between models. An ICLR 2024 study found that performance across prompt formats correlated only weakly between the models it evaluated.
In other words, a prompt that helped yesterday’s model may confuse tomorrow’s.
Model updates need skill maintenance
My new model-update checklist is simple:
- Run existing skills against representative tasks.
- Look for instructions created to work around old model behavior.
- Remove duplicated or conflicting rules.
- Delete instructions the new model follows naturally.
- Change one group of instructions at a time.
- Run the same tests again.
- Keep only changes that produce a measurable improvement.
This is basically dependency maintenance, except the dependency is natural-language behavior.
The lesson
Custom skills are not permanent documentation.
They are part of the agent system, and that system changes whenever the underlying model changes.
Before concluding that a new model has become worse, audit the instructions surrounding it. You may have upgraded the engine while keeping the old owner’s manual.
Or, in developer terms:
The model might be fine. Your prompts may just have technical debt.
Sources
- Anthropic: What’s new in Claude Opus 5
- OpenAI: GPT-5.6 model guidance
- ICLR 2024: Quantifying Language Models’ Sensitivity to Prompt Design
- No Priors: Andrej Karpathy on Code Agents
Disclosure: I used OpenAI Codex to help organize and edit this article. The experience, observations, conclusions, and final review are mine.
Top comments (1)
I like the dependency-maintenance framing. The failure mode I see is teams treating prompts as docs rather than executable config. A small eval set across model changes catches more than another paragraph of instruction, especially around tool choice and refusal boundaries.