The model leaderboard reshuffles every six weeks and everything you learn about a specific model has a half-life of months. Five things do not decay, and they are the same five that the people running the swarms say they still need humans for. This is a list for the engineer who is tired.
The scoreboard that changes every six weeks
Here is what the last ten days sounded like if you are an engineer with a job and a family and a
finite amount of evening.
Theo Browne released a video saying that if you cannot feel the difference between this
generation's frontier models and the last one, you suck at prompting, and he meant it kindly but he
meant it. ThePrimeagen spent an episode on how a company that invented the transformer and the
tensor processing unit is, on the coding leaderboards he showed, being beaten by a lab with a few
hundred employees, and then left one of its models on a trivial bug for forty minutes and watched
it spend 330 million tokens reading the same file. AI Explained walked through six axes on which
the labs say capability will keep improving, none of them near saturation, and quoted a researcher
saying there is a large gap between how fast progress looks from inside and from outside. A model
called Astra that could do things nothing before it could. Another called Fable that was the safe
choice three weeks ago. A Chinese open-weight model that is months old and beating both on some
table.
If you tried to keep up with that, you did not sleep and you learned nothing durable, because
almost every specific fact in the paragraph above will be wrong by November. That is not a
prediction about any particular model. It is a description of the scoreboard. It reshuffles every
six to eight weeks and it has done so for two years and every lab in the AI Explained video says it
will keep doing so.
So the anxiety is real and the scoreboard is real and the two have almost nothing to do with each
other. The question worth an evening is which things you can learn now that will still be true when
the scoreboard has reshuffled four more times.
The half-life of what you know
Sort what an engineer learns about these tools by how long it stays true.
At the bottom, decaying in weeks: which model is best at which task, which one reads the same file
twenty-six times, which one is over-eager and which one is lazy, the prompt phrasing that makes a
particular model stop apologising. This is what most of the content is about, because it changes
constantly and change is what content is made of. Learning it is not worthless. Learning it is
maintenance, like knowing this quarter's prices.
In the middle, decaying in months: the harness. Which tool, which agent runtime, which instruction
file format, which permission model. The study we wrote about earlier this week found that the
harness is a first-class variable, more important than the leaderboards admit, and that is true. It
is also true that the harnesses are being rewritten as fast as the models, and that the changelog
entry that got seven hundred points this week was about one tool starting to read another tool's
config file.
At the top, not decaying at all as far as anyone can tell: five things. We will name them and then
argue for each one, because the argument is where the reassurance actually lives.
What an engineer learns about AI tools, sorted by how long it stays true. Almost all the content is about the bottom row. Durations illustrative.
The five things
Specification. The single skill that has appreciated most in two years is the ability to say,
precisely and in writing, what a system should do, including what it should refuse to do and every
exception. This used to be a skill people apologised for having, because the code was the
specification and writing it down twice was waste. Now the specification is the input to the
machine that writes the code, and the quality of the output is bounded by the quality of that
input in a way that no model upgrade changes. Every team that got disappointing results from agents
this year got them from underspecified tasks. The detail.dev post says the most valuable engineering
work is going to be having good ideas, and an idea that cannot be specified is not yet an idea.
Verification. Knowing whether a thing works, and being able to prove it to someone else, is the
skill the machines are worst at and the one they generate the most demand for. Every agent-written
change needs someone who can say what test would catch the failure, whether the test that was
written is that test, and whether the green result means what it appears to mean. Dan Luu's essay
this week is on the front page for a reason: there is no point at which turning your brain off
works, and verification is the name for the brain being on. This is also, not coincidentally, the
skill Theo was actually describing when he said people prompt badly. The people who get good
results from strong models are the people who can tell when the result is bad.
Judgement. Which of the forty things the model could do is the one worth doing. Which of the
three approaches it offered is the one that will still be fine in a year. When to stop. When the
obvious solution is obvious because it is wrong. Every essay on this site comes back to this because
every failure we have watched comes back to it: the model did something competent that nobody
should have asked for. Judgement is slow to build, does not transfer from a video, and is the entire
reason a senior engineer is paid more than a junior one with the same tools.
Domain intimacy. Knowing the business, the customers, the forty exceptions and why the freight
claim in 2023 changed how one account is handled. We have argued at length that companies do not
have processes, they have habits, and that the habits live in heads. The heads are the moat. A
model with the rumour builds the obvious version of your product; the person who knows why the
obvious version fails for the second-largest customer is the person the swarm cannot replace, and
that person is more valuable this year than last, not less.
The seat. The ability to occupy the approval seat well: to look at a model's output, know where to
spend eight seconds, sign for it, and stand behind the signature. This is the one that combines the
other four and it is the one the organisation will pay for when the volume of model output has
outrun everyone's ability to read it. It is also the one nobody teaches, because until eighteen
months ago the person in the seat had also done the work.
What this means for the evening
Spend it differently.
Stop trying to keep up with the scoreboard and start reading it the way you read exchange rates:
glance, note the direction, move on. Pick one strong model and one harness and learn them properly
for a quarter rather than four of each badly. Theo's floor-and-ceiling point is correct and it is
also an argument for depth: you learn where a model's floor is by using it on dull tasks for weeks,
not by watching someone else's demo of its ceiling.
Then spend the real time on the five. Write the specification for the next thing your team builds
before anyone, human or model, writes a line, and notice how much you did not know. Write the test
that would catch the failure before you look at the agent's tests. When an agent offers three
approaches, write down why you picked one, and read your reasons back in a month. Learn the part of
the business your team pretends is simple. Volunteer for the review seat that everyone else is
avoiding because the queue is long, and get fast at it, because that queue is the most important
place in the company and almost nobody wants to sit there.
The trade an engineer can make this quarter. The left column is what the content wants you to do; the right is what compounds.
The honest caveat
It is possible to hide from the tools behind this list, and some people will. Specification and
verification and judgement are worth nothing if you refuse to use the machines that make them
valuable, and the engineer who has not sat in front of a frontier model for a hundred hours does not
actually know where its floor is and cannot occupy the seat. The scoreboard is not the point, but
the tools are, and the argument here is for depth with them rather than distance from them.
The researchers in the AI Explained video may be right that the gap between inside and outside is
large and that things will move faster than the outside expects. If so, the scoreboard will
reshuffle faster, not slower, and the half-life of everything in the bottom two rows gets shorter.
The five things at the top do not get shorter. They get more expensive, because the volume of
machine output that needs specifying, verifying, judging, situating and signing for goes up with
every release, and the number of people who can do those things well does not.
You are not behind. You have been watching the part of the screen that changes. Look at the part
that does not, and get good at it while everyone else is refreshing the leaderboard.
Sources
- Please stop using stupid models (t3.gg, video) (18 Sep 2026)
- How are they losing so bad (ThePrimeagen, video) (15 Sep 2026; the token count and the model timeline are as reported on stream)
- What AI Researchers Saw, Before Their Demand to 'Pace' AI (AI Explained, video) (16 Sep 2026)
- Towards Self-Driving Codebases (detail.dev) (HN, 119 points, 17 Sep 2026)
- There's no point at which turning your brain off will work (Dan Luu) (HN, 196 points, 18 Sep 2026)
Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.
Top comments (0)