DEV Community

Cover image for If a rumour can summon ten thousand agents, your roadmap is a starting gun
Levelbrook Consulting
Levelbrook Consulting

Posted on Originally published at ai.levelbrook.com

If a rumour can summon ten thousand agents, your roadmap is a starting gun

Two mathematicians spent a year on a problem, a rumour of their approach reached a lab, and a swarm of agents reportedly reproduced it in days. Terence Tao's warning about open science applies just as well to your product. What was defensible last year is now a prompt.

The story, as told, with the caveats attached

Here is the account that circulated last week, and it is important to say up front that it is a
summary of public statements by parties who disagree with each other, so treat every clause as
"reportedly".

A mathematics professor and a collaborator who works at one of the labs spent about a year on the
Navier-Stokes existence problem, one of the Millennium Prize problems. In the last month of that
year they leaned heavily on coding agents and their progress accelerated; in mid-August they got a
simpler related system to break, the closest anyone had come. Then, according to their statement, a
different lab, having heard rumours of the approach, pointed a very large number of agents and a
very large amount of compute at the problem and, within days, claimed a result on the full problem
using what the mathematicians say was their novel approach. There was a phone call. The accounts of
the phone call diverge sharply. Both sides published, one after the other, on the same Tuesday.

You can read Fireship's telling, which is where most engineers heard it, and you can read the
statements. We are not adjudicating it. What we want to point at is the line Terence Tao is
reported to have written in response, because it is the most important sentence of the month for
anyone who runs a product team, and almost nobody outside mathematics noticed it.

If the mere rumour of your research can trigger a swarm of agents racing to front-run it,
mathematicians will simply stop sharing ideas, and that will undo centuries of open science.

Replace "research" with "roadmap"

Now do the substitution. If the mere rumour of your feature can trigger a swarm of agents racing to
front-run it, what happens to your roadmap?

For the entire history of the software business, the gap between having an idea and shipping it was
the moat. Not the idea itself; ideas were always cheap and always leaked. The moat was that turning
the idea into a working, deployed, maintained thing took a team a quarter, and a competitor who
heard about it on a Tuesday could not have it by Friday. Product strategy, fundraising, hiring
plans, launch timing, all of it was built on that gap being months wide.

The maths story is what it looks like when the gap closes to days at the top of the market. The
Dream RSI paper that Fireship covered the same week is what it looks like a level down: a search
loop that, per the video's summary, wrote a solver that beat a standard library in about 300
attempts where the fixed-policy version needed 550 and the previous record needed roughly 51,000.
That is a search process getting two orders of magnitude cheaper at finding a solution somebody
else already knew the shape of. Knowing the shape of the solution is what a roadmap leak gives a
competitor.

The idea-to-shipped gap was the moat. Durations illustrative; the collapse is the point.
The idea-to-shipped gap was the moat. Durations illustrative; the collapse is the point.

This is not an argument for secrecy, for the same reason Tao's line is not an argument for
mathematicians to stop publishing. Secrecy does not work either: your customers know what they
asked for, your job postings say what you are building, and your own agents' instruction files
describe the product in more detail than any leaked slide. It is an argument for being honest about
what is still defensible when the build is no longer the hard part.

What a swarm cannot front-run

Go back to the maths. A swarm with a rumour and twenty million dollars of compute could,
reportedly, reproduce a result. What it could not do is the year before the rumour: choosing that
problem, choosing that approach out of the dozens that do not work, building the intuition that
made the approach look promising when it looked like nothing to everyone else. The swarm needed the
shape. The year produced the shape.

The commercial version of that year is the thing we wrote about earlier this month under the
heading that companies do not have processes, they have habits. The forty exceptions that live in
one person's head. The knowledge of which customer needs it done differently because of the freight
claim in 2023. The understanding of why the obvious version of the feature fails for the second
largest account. A swarm given the rumour builds the obvious version, perfectly, in days. The
obvious version is the one your customers already rejected.

What a swarm reproduces from a rumour, and what it cannot. The right-hand side is the whole of the defensible business.
What a swarm reproduces from a rumour, and what it cannot. The right-hand side is the whole of the defensible business.

So the defensible assets in a world of swarms are the ones on the right of that figure, and they
are exactly the assets most product organisations have neglected because they were not the
bottleneck. The written specification, including the exceptions, which almost nobody has because
the code was the specification and the code was expensive. The proprietary data about how the
product fails and for whom. And the accountability, the fact that when the thing goes wrong there
is a named organisation that picks up the phone and fixes it, which no swarm has ever offered
anyone.

What changes on the roadmap

The practical consequences are uncomfortable for the way product teams have worked.

Announcements shift from features to outcomes. Announcing "we are building X" now hands the shape
to anyone with a swarm. Announcing "our customers in this segment no longer have this problem" hands
them nothing they can prompt with, because the interesting part is which segment and which problem
and why the obvious solution did not work, and that lives in your specification.

The specification becomes the product. Not a slide, the actual document: what the system does,
what it refuses to do, every exception and why. If your team's competitive advantage is knowledge
that lives in heads, the swarm era is the strongest incentive you will ever get to write it down,
because the written version is the only form in which it compounds and the only form in which it
can be defended.

Speed stops being a strategy and becomes table stakes. Being first to ship the obvious version was
worth a great deal when it took a quarter. It is now worth roughly a week of attention. The teams
that win will be the ones that were already on the third version, informed by the failures of the
first two, when the swarm shipped the first one.

And the Gowers point, from his essay on why he declined to sign the Fields medallists' letter,
applies in full. He has argued for twenty-five years that mathematics contains two cultures,
problem-solvers and theory-builders, and that the field needs both. The swarms are extraordinary
problem-solvers. Theory-building, meaning the slow accumulation of a coherent understanding of a
domain, is what makes the problems worth solving and tells you which one to solve next. Every
product organisation is about to discover which of the two it was actually good at.

What to write down this quarter

The uncomfortable implication of the figure above is that the defensible assets are documents, and
most organisations do not have them. So here is the writing programme, in priority order.

The exceptions register. For the product's core workflow, every case where the obvious behaviour is
wrong and why. Not the happy path, which anyone can reproduce, and not the code, which a swarm can
regenerate. The list of forty things that live in the head of the person who has been there
longest, with the reason attached to each. Ask that person to talk for two hours, record it, have a
model draft the register, and have the person correct it. This is the single most valuable document
the company can own and it usually does not exist.

The refusal list. What the product deliberately does not do, and what happened when someone tried.
A competitor building from a rumour will build the features you rejected, because they look like
features. Knowing why they were rejected is a year of learning that cannot be prompted for.

The failure data. Which customers hit which failures, how often, and what it cost. Nobody outside
the company has this, and it is the input that decides which of the next ten things to build. Keep
it in a form a model can read, because your own agents should be making the obvious version of the
next feature for you, informed by it, before anyone else makes the obvious version uninformed.

The accountability map. Who picks up the phone when the thing is wrong, in what timeframe, with
what authority to fix it. Write it down, publish the shape of it to customers, and mean it. A swarm
can ship a product. It cannot answer for one, and in the year when every product has a swarm-built
twin, the answering is the product.

Where this is wrong

The counter-argument is that most companies do not compete with labs holding twenty million dollars
of spare compute, and that is true today. It will be less true every quarter, because the cost of
the swarm is falling on a schedule nobody in your market controls, and because the swarm does not
need to be a lab. It needs to be a competitor with a credit card and a rumour. The Navier-Stokes
story is not the shape of your next year. It is the shape of your next three, arriving at the top
of the market first, as these things do.

Tao's warning was that scientists would stop sharing. The commercial equivalent is worse: companies
will keep sharing, because they have to, and will discover that the only part they could ever have
kept was the part they never wrote down.

Sources


Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

Top comments (0)