A lawyer told me his job was safe, and gave a better reason than most
people give.
He catches AI mistakes constantly, he said, but that is not the point. The
point is where his work lives. Roughly 20% of it happens in courtrooms and
in private negotiation — rooms that produce no transcript anyone will ever
train on. The rest, the drafting and the research, is already in the
corpus. His conclusion: "AI at best can rise to the level of what a law
student knows after graduating."
I answered honestly, and the answer was worse for me than for him:
Your 20% is protected because it never gets written down. My work is
100% digital — all the code, all the solutions, everything is basically
the training data. Maybe my whole job is that 80%.
That is the uncomfortable version of the question. Not "will AI replace
developers", which is exhausted, but: if none of my work is protected by
scarcity, what is left that is actually mine?
The obvious answer — orchestration, verification, taste, being the
interface between the human and the machine — is also the answer everyone
gives, which is a good reason to distrust it. So I spent a stretch of work
watching for it specifically. Not asking whether steering matters. Asking
what it catches that nothing else does.
The result splits cleanly in two, and the split is more useful than the
slogan.
What a second model catches perfectly well
A large share of the "only a human can" list is not human work at all. It
is verification work, and a second model does it well — often better than
the first one did, because it has no stake in the first answer being
right.
I don't have to be in the room for this part, and mostly I'm not. Over one
two-day stretch my agents caught each other repeatedly: a reviewer refuted
a diagnosis another agent had already written up, an executor stopped
dead on a brief whose premise was wrong and explained why, a fresh pass
turned up ten defects in code a previous pass had promised to check and
quietly skipped — including one that charges a user a fee and then denies
the request.
Not one of those needed me. My contribution was structural, not
intellectual: a setup where a second opinion is cheap, and rules that make
refusal legal — an agent that hits a false premise is allowed to stop
rather than build on it.
That is real, and it is not nothing. But it is also automatable, and
anyone reading this can copy it in an afternoon. If this were the whole
answer, the lawyer would be right about me.
What it structurally cannot
Then there is the other category, and it behaves differently.
Case one. A feature had been built, verified, and shipped. Six
independent verification passes came back green. Every one of them
confirmed the code did what the spec said, and none of them was sloppy —
the spec wasn't wrong either.
Then I used the product, and the feature was silent. A single line on the
client discarded the value the server had just computed, so the whole
thing ran end to end and produced nothing a person could hear.
The lesson isn't "human testing matters", which everyone already agrees
with and nobody schedules. It's narrower and less comfortable: every one
of those passes was asking whether the code matched the description. Not
one was asking whether the result reached a human ear, because that isn't
a test — it's use. A system that verifies itself perfectly will still
converge on the description, and the description is exactly the artifact
that can be complete, correct, and beside the point.
Case two. My agents handed me a list of nine defects queued for
fixing. I removed three of them on sight — not because the analysis was
wrong, but because the behaviour was intended. A toast notification
covering the navigation bar is a defect if you assume it shouldn't; it's a
design decision if I decided it should. No verification pass can separate
those two, because the difference isn't in the code. It's in what somebody
wanted, and I'm the one who wanted it.
Case three. The plan on my desk was to build a new empty-state
component for a slot that was stuck showing an infinite loading animation.
I stopped it: the component already had a paused state, because I had
asked for one when the design system was built. The fix went from a new
component to a single boolean.
There is a pattern, and it is not "humans are wiser". In all three cases
the machine had every fact and still could not get there, because the
missing input was not a fact. It was intent — and intent lives in the
person who formed it, not in the artifact.
A verifier can prove a system does what it says. It cannot tell you
whether what it says is what you meant. That gap does not close with a
better model, because it isn't a knowledge gap.
Where I handed the seat over
Any version of this argument that only lists wins is an advertisement, so
here is the stretch where I lost — and the interesting part is exactly
where I lost it.
I'm a frontend developer. The work in question was backend: a caching
layer I can read but could not have designed, in a language I don't write
daily. And that is precisely where I loosened up. Not on how much work I
handed over — on how much judgement. In my own domain I argue with the
machine constantly, because I can feel when an answer is shaped wrong.
Outside it I have no such feel, so I let conclusions through.
The conclusion that got through: a cache would serve stale data. It was a
good-looking claim. The cache was real, its key was genuinely missing the
field it was said to be missing — and the whole thing sat on a code path
the application never takes. Ten call sites, not one of them reaching it.
A single search settles it. For two days nobody ran the search, and I
wasn't equipped to know one was needed.
The cost wasn't a bug. It was a plan: work split across two repositories,
one of them another team's, and a dependency communicated that did not
exist.
It did get caught, and how it got caught is the part worth keeping. Not by
me spotting the flaw — I could not have. By me refusing to sign off. I
kept pushing the plan back and forth, asking for it to be exact, making
the pieces line up, and the contradiction eventually surfaced under that
pressure. All of the errors came out in the end, but not one of them came
out because I knew better.
So the thing the seat supplies isn't expertise — I had none here, and it
still worked. It's the refusal to accept "close enough" from a system that
is fluent, fast, and confident. That is a much smaller qualification than
domain knowledge, and much easier to give up, because giving it up costs
nothing today and feels like trust.
The honest size of it
Back to the lawyer. His 20% is protected by scarcity: those rooms produce
no data. That protection is real but passive, and it only holds while the
rooms stay closed.
The seat I'm describing has a different property. It isn't protected by
missing data — it's the seat that supplies something data cannot
contain, which is what "correct" was supposed to mean in the first place.
No corpus confers that, because it was never a fact about the world; it
was a decision somebody made.
But I want to be precise about the size, because this is where these
arguments usually inflate. That seat is not 80% of my job. Most of my week
is still specification, review, and repair — all of it in the corpus, all
of it increasingly done better and faster by the machine. The part that is
structurally mine is small, intermittent, and mostly consists of knowing
what I actually wanted.
Which leaves the real risk, and it isn't replacement.
My wife doesn't work in tech. She read an earlier version of this and gave
me the sharper version of it:
I changed my mind a bit about AI and robots. It's not going to be some
uprising because they don't need us. Humanity will be at fault, because
it'll stop controlling and checking, and it'll just flow that way on its
own.
And she named the cause without softening it:
Because of human stupidity and degradation, in a way. Like — whatever,
why check, it does it itself.
What happened next is the reason this section exists at all. I put that to
one of my agents, and it returned a clean economic account: checking is
boring and expensive, the machine is right most of the time, the cost of
verifying exceeds the expected value, so people rationally stop. Tidy,
plausible, and wrong in a way that is hard to see, because it is the kind
of wrong that reads as insight.
There is no economic argument. "I don't care" is not a weighed cost — the
weighing never happens. That is exactly what makes it durable: you cannot
refute a decision that was never made. Give it a model and you have
already mistranslated it into something answerable.
The operational version isn't "checking is too expensive". I can write it
out myself, because it is already how I think on a good week, when
everything is landing:
I just won't check, because it all works anyway. Why bother doing extra
work if everything's fine. When the mechanism visibly starts failing,
then sure, you can check. But by then it's already too late.
So the seat doesn't disappear because something takes it. It disappears
because sitting in it stops feeling necessary — and the signal that it was
necessary arrives after the fact, if it arrives at all.
Her conclusion was one sentence, and I have not improved on it: you can't
let it run by itself, you have to keep some control and know what's going
on.
The lawyer exchange is from a Reddit thread. The framing about training
data, and the closing line — that "mashing keys" disappears while "being
the interface between the human and the machine" remains — are his.
Top comments (5)
The self-report anchor is the part I keep seeing bite agent setups. A second model can compare code to a spec all day, but if the spec was written by the same run, it is mostly checking consistency. I try to keep one artifact outside the loop, even if it is boring, like a failing test, a user-facing invariant, or a tiny acceptance note written before the agent starts.
I agree, just one thing i would add: a failing test is only outside the loop if the test itself
is. When agent hands me a fix plus a test, i break the fix on purpose
and require the test to go red, and i run that myself instead of taking
the report.
It caught a real one recently: a security test pinned exactly
one attack string, so replacing the guard with a naive string match would
have stayed green while three other cases walked straight through. The
test was passing. It just wasnt pinning anything.
Good write-up — and I'd add a category to "what it structurally cannot" that sits between your verification cases and the intent cases: verification against an anchor that is itself a self-report.
Your case one sets it up perfectly. Six passes asked "does the code match the description" and all came back green — that works as long as the description is a stable external artifact. But when the description is produced by the same thing being checked, every pass converges on a shared self-report and the check goes circular.
We hit that exact shape with a context-size gate: it compared a locally-estimated context size against a threshold, and every check passed because the estimate was internally consistent — it had the same context the model had. The number was wrong in one direction only (148K estimated vs 222K the provider actually billed), so the gate never fired once. A second model wouldn't have caught it either: it reads the same self-report and has no better access to the provider's counter. The anchor that worked was outside the system entirely — the provider-returned token count. Same shape as your case one, one level up: the passes were asking "does the estimate match what we believe", not "does it match what the provider measured". The description was the leak, not the deduction.
And that's also where I'd push back gently on the closing section. The agent's tidy economic account and your wife's "you can't let it run by itself" both assume a human seat that has to stay occupied. The more robust answer for autonomous systems is to make the anchor external and the check fail-closed — so "not checking" stops being a decision a human can make, because the meter isn't theirs to skip. Your seat ("refusing to accept close enough") becomes a property of the mechanism, not of whose attention happens to be on that week. The intent seats stay human; the verification seats don't have to be.
The third category is a real gap in the piece, and it's the one that
actually matters. I split things into "verification catches it" and "only
intent catches it", and you're pointing at a case that looks like the
first and behaves like neither: a check whose anchor is produced by the
thing being checked. Six passes against a description, when the
description came out of the same system, is not six checks — it's one
belief, restated six times. That belongs in the article and isn't there.
On the pushback: you're right, and my own argument pointed there before I
did. I wrote that the verification setup is automatable and anyone can
copy it in an afternoon — then closed as if the seat had to stay warm.
Making the anchor external and the gate fail-closed is strictly better
than relying on someone remembering to look.
Where I'd push back is on "stops being a decision a human can make."
Your own case is the evidence. The provider's token count existed the
whole time. Nothing technical prevented anchoring on it — the gate ran on
a local estimate because nobody had routed it yet. That's the same
non-decision, one level up: not "I chose not to check" but "nobody
decided to build the thing that checks." And fail-closed gates have a
known second act — they get switched off the week they block a release,
by someone with a deadline and a good reason.
So I don't think the seat disappears; I think it changes frequency. It
moves from inspection, which is daily, boring, and skippable without
noticing, to instrumentation — which is rare, deliberate, and easier to
hold precisely because it's rare. That's a real improvement and I'll take
it. But the rare decision is also the one nobody revisits, and instruments
decay quietly.
Your rule is the best line in this exchange, and it survives the move:
an estimate that cannot be checked is a belief. So is a gate nobody has
looked at since it was written.
You're right, and it's the strongest version of the objection: routing the provider count was itself a decision that could have gone unmade, and nothing in the system forced it. Same class of non-decision, one level up. I don't think you can exit that regress by choosing a better anchor — any anchor is something a human had to pick.
What I'd defend is that the fix changes the failure signature, not the decision class. The first non-decision was silent and self-consistent: the gate's only input was the system's own estimate, so the error compounded invisibly — 148K estimated, 222K actually sent, gate never fired, and nothing in the loop could have noticed. The system reported a healthy number using a number only it produced. That's what makes it a blind spot rather than a missed improvement: there was no observable signal at all.
Anchored on the provider count, two properties hold that didn't before. First, the projection error is bounded to one round's delta — the anchor is re-measured on every response, so the routing isn't a one-time act that can be forgotten; it re-fires mechanically each round. The non-decision can't persist the way it did, because persisting required remembering, and the mechanism doesn't need a memory. Second, the residual blind spot moves outside the loop: for the projection to go wrong now, either the provider has to mis-account or the plumbing has to break — both external to the estimate, both observable. The original failure had no external component.
The honest caveat, having just read the gate: when the anchor is missing (first round, post-compact, or the provider stops returning usage), the implementation falls back to the plain estimate — so it's not strictly fail-closed, it degrades to the old behavior. But the drift that mattered before was cumulative and invisible; the fallback is now the exception rather than the steady state, and the steady state is re-anchored from a number the system doesn't produce. The regress is real; the only honest move is to shift where it lands — from invisible and cumulative to bounded and externally visible.