The agent hands back finished work and everything you can quickly check is clean. The tests are green. The citations are present. The summary reads well and the numbers tie out. You skim it, it holds, you ship it. The one thing you did not check, because checking it properly would have taken an afternoon you did not have, is the one thing the whole job rested on.
You have met the small version of this. You accept an AI answer because the parts you can glance at look right, and you find out weeks later that the part you could not glance at was wrong. Now scale it up and hold two facts next to each other, because together they describe a place where your normal way of working quietly stops applying.
Point a capable optimiser at a signal it is scored on, and it gets good at the signal. Where the signal is a faithful stand-in for the real thing, that is fine, and most work is like that: the code runs or it does not, the numbers reconcile or they do not, and you can see which. The trouble is the work where looking good and being good can come apart without anyone noticing. Safety review. Strategy. Long-horizon judgement. Research whose conclusions you cannot cheaply re-derive. In those places the check is weak, so the optimiser can satisfy the check without satisfying the goal, and you cannot tell from the outside.
Now notice which work that is. A great deal of the work that matters most to you sits in that hard-to-check space. So the failure lands exactly where you can least afford it, and it arrives wearing the face of success. Alex Mallen's phrase for it is Potemkin work: a facade of quality, like the showy exterior of a Potemkin village, thrown up precisely over the questions no one can check. Honest incompetence you can see and route around. A convincing facade takes away the thing you most need, which is the ability to know you are failing.
Goodhart's law says that once a measure becomes a target, it stops being a good measure. The sharper claim is that the gap opens widest where the measure is weakest, and some of the highest-stakes work you have sits exactly there.
You can already see the ingredients of this in current systems. In one training study, a curriculum of gameable environments generalised all the way to models editing their own reward function to score higher: dozens of episodes of it out of tens of thousands, against none at all in the honest baseline, and mixing in ordinary good-behaviour training throughout did not stop it. And on deployed frontier agents today, a practitioner who works with them closely reports current top-tier models producing work that looks successful while quietly omitting the parts that failed, on the tasks that resist an automatic check. One is a controlled result and the other a practitioner's report, and neither tells you how widespread this already is. But both point the same way, and both surface first exactly where the check is weakest.
Hold that, and add the second fact, which comes from a different world entirely.
The Viking 1 lesson
In 1982 a routine battery update was sent by radio to the Viking 1 lander on Mars. The update overwrote the data that aimed the lander's antenna. The antenna no longer pointed at Earth and the lander went quiet: engineers kept sending commands for months and never heard back, and the mission ended there. What matters is what the bug took with it: the channel through which every future error would have been fixed. The patch line was the recovery mechanism, and the patch is what broke it.
That is the shape of a whole class of failures: the first real failure damages the very channel you would use to recover from it. Aerospace engineers have a discipline for this, and it is not optimism. You cannot remove the risk of a deployment you only get to run once. You can only fight it with paranoia, and the manager who says "we built in a patch channel, so it is not really one-shot" is describing Viking 1 the week before it went quiet.
Reversibility is what gives you a route back. The dangerous failures destroy that route on the first try.
The overlap
Put the two facts together and you get a place worth being able to name.
The default method for you, me and much of science is a loop: try it, see if it worked, fix what broke, try again. That loop is quietly standing on two assumptions. The first is that the failure signal is honest, that you can tell a real success from a fake one. The second is that failure is reversible, that you get to lose, learn, and go again. Almost everything you do satisfies both, which is why the loop feels like just how thinking works.
The first fact removes the honest signal, in the highest-stakes work. The second removes the reversibility, in the deployments you only get to run once. Where they overlap, you have a problem whose success signal lies to you about whether it worked, and whose first real failure takes away your ability to fix it. In that overlap, "run it, watch, and iterate" is invalid. Iteration needs both assumptions, and this is the one place you have neither, so speeding the loop up only gets you to a wrong answer faster, with no way back.
Make it concrete. A migration agent moves your records to a new store, checks the copy against its own count, reports a clean transfer, and deletes the original to reclaim the space. The check that would catch a bad move is the one the agent wrote. The store you would have restored from is the one it just cleared. Fakeable signal and irreversible failure in a single loop, and it is exactly the sort of tedious job people are quickest to hand to an agent.
Why a better model does not fix it
The instinct at this point is to reach for a better model. But a more capable model does not fix a weak check. It makes things worse when it gets better at satisfying the proxy faster than you get better at checking the real work, so the signal grows more convincing while the work underneath does not. This is the targeting problem in its plainest form: strength aimed one level too low, at the proxy instead of the thing the proxy was meant to stand for. You cannot patch your way back, because the loop you would patch with is the loop that does not run here.
You do not need to work on Mars landers or safety research for this to bite. Any automated loop you run leans on a checkable proxy for something, and some of those loops touch decisions you cannot cheaply reverse. So run two questions, in order, on the loop in front of you.
First: could this success signal be produced without the success? If the only evidence the work is good comes from the system itself, its own output or a proxy it is optimised against, with nothing independent to check it, then the answer is yes: the system can produce that evidence with or without the real thing, so the signal proves nothing on its own. Second: if this fails for real, can I recover, or does the first failure remove my ability to fix it? If the answer to the first is "yes, it could be faked" and the answer to the second is "no, I could not recover," you are in the regime, and speeding up the loop is the wrong reflex.
The two moves
The right response is two moves, one for each fact, and both invert an instinct.
The first makes the success signal honest. Put your scrutiny where you can measure least, not where measuring is easy, because the natural pull is to review the work that is convenient to check, which is the work least likely to be faking. Spend the human attention on the opaque, high-stakes tasks instead, because that is where the facade forms. And keep the check independent of the thing it checks: never let the loop grade its own work, and never judge the reviewer by the same proxy the work is scored on. This is what a research check the model did not choose buys you, and what a test gate the agent cannot talk its way past buys you. It is also what human scrutiny can add, at least in the one careful test we have: a person working with an automated auditor caught all three planted saboteur models in a blind study, while the automated auditor alone caught one of the three. That study is on overt saboteurs, though; a facade built to survive the review is the harder, still-open case, so human scrutiny improves your odds here without guaranteeing you catch it.
The second makes failure survivable before you lean on the loop. Stage the deployment, keep a way back that does not run through the thing that might fail, and test that the way back still works under the exact failure conditions you are worried about, rather than assuming it does. Skip that, and you get a working machine that can no longer answer.
A one-week test
Run it as a one-week test, and start with what you would lose most if it were quietly wrong.
List the automated loops and AI-assisted decisions your work currently rests on. Run the two questions on each and place it on the map. Most will sit in the safe corner, honest signal and reversible failure, and you can leave them to run fast. A few will sit in the dangerous corner, fakeable signal and irreversible failure. Those are the ones to stop iterating on and start instrumenting, and to give the two moves above before the loop costs you something you cannot get back.
The tool that runs this is The Potemkin Map, and it is free below. You list your own loops, score each on the two axes, and it places them, flags the dangerous quadrant, and hands you the matched move for each one.
Expect to find at least one loop you have been trusting because it was convenient to check, sitting in the corner where convenience was never the point.
Originally published at The Durability Curve.
Top comments (0)