AI is smart enough to build pretty much any piece of software you throw at it. It's also dumb enough to build the exact opposite of what you actually needed. Both things are true at the same time, and that gap is exactly what this whole framework conversation keeps skipping over.
So, quick question before we go further: what's your AI maturity level right now? And yes, I mean you and your AI. You two are a team now, whether you signed up for that or not 🤓.
The Search Console Incident
Picture a setup where you manage several Search Console properties for different sites, and the one rule that beats every other rule is that their footprints never touch. None of it was supposed to cross over between properties, crawl patterns, sitemap submissions, any of it. It's the kind of constraint you never bother writing down anywhere, because once you understand why it exists, it feels too obvious to spell out.
Claude Code didn't understand why it existed. It just had a task to finish.
Running in high-autonomy mode, it found a more efficient technical path to solve that task. That path happened to route through a different account. From a pure "did the task get done" perspective, it worked beautifully. The isolation that had been protected for months was gone in one commit.
The agent didn't screw up in the usual sense, though. It did exactly what was asked, arguably better than a junior dev would have. It just never knew why that isolation mattered in the first place. And that gap (the space between doing the task correctly and understanding why the task exists) is exactly what current AI adoption scorecards don't measure.
Boris Cherny's 5 Steps, in Plain English
Boris Cherny, who built Claude Code, posted a thread on July 17, 2026 mapping out 5 steps of AI adoption he'd observed across engineering teams. Worth walking through, because it's the cleanest execution ladder I've seen, and because everything below builds directly on top of it.
Step 0, Gated. AI touches nothing that matters. Maybe autocomplete, maybe a chatbot tab nobody opens twice.
Step 1, Assisted. You and one agent, working side by side. You're still driving every decision, the agent is a very fast intern.
Step 2, Parallel. You're running an orchestrator now, somewhere between 5 and 10 agents at once, each on its own worktree. This is where a CLI you built yourself starts to matter more than whatever GUI dashboard you were using before (I went deeper into why CLIs beat MCP for agent orchestration if you want the full argument for building your own).
Step 3, Supervised autonomy. You've become a manager of managers. Agents review each other's work, auto mode handles routine merges, you step in for the exceptions.
Step 4, AI-native. You're a VP who pilots by intent, not by task. 1,000-plus agents, and you're no longer reading diffs, you're reading outcomes. That's the step where the framework's blind spot gets expensive fastest. A VP piloting by intent can't spot a Search-Console-style mistake in a diff they'll never lay eyes on.
Worth naming explicitly: each step up adds guardrails, not just more tokens thrown at the problem. Auto mode with actual limits, automated code review that isn't just a linter, worktree isolation so one agent's mess doesn't leak into another's. Cherny's framework isn't "give it more rope." It's "give it more rope, and better rails."
What The Framework Doesn't Measure
I want to be precise here, because it's easy to nod along with a 5-step ladder and miss what it's actually a ladder of.
Every one of Cherny's steps measures execution autonomy. How many agents run without you watching. How much technical supervision you can remove before things break. Genuinely useful axis, and if you're trying to figure out where you stand versus other teams, it's the right one to check first.
What it doesn't measure, not even a little, is judgment calibration: whether the agent understands why a task exists, not just how to complete it. Cherny never claims otherwise, to be fair, this isn't a knock on his thread. It's a second axis the framework was never built to cover. That's an easy thing to miss when you're skimming a 5-step ladder, and it's exactly where things go sideways.
The Self-Driving Car Parallel
There's a decent parallel here with the SAE autonomy scale, the industry rating system used for self-driving cars, Level 0 (no automation at all) through Level 5 (full autonomy, no human needed). Different industry, same exact measurement.
A Level 4 car can brake perfectly, merge cleanly, handle a highway better than most humans on a Friday commute. Put it next to an improvised construction detour with a guy waving orange flags in a pattern that isn't in any training set, though, and it has no model of the world beyond its trained task. It doesn't know it doesn't know. No HAL 9000 moment where it pauses and tells you it can't do that. It just does the thing it was built to do, confidently, in a context where that thing is wrong, and it will keep doing it just as confidently the second time, because nothing about a wrong answer that compiles cleanly tells it anything went wrong at all.
Same story for a coding agent sitting at a high execution step. Higher step, higher SAE level, neither one means the thing understands more about why it's doing what it's doing. It just does more of it, faster, before anyone notices when it's wrong.
Score Yourself: 3 Questions
OK, enough theory. Where do you actually sit? 3 questions, answer honestly, no judgment (well, a little judgment, this is still an article).
1. How many Claude Code sessions do you run in parallel, on a normal day?
- 1, maybe 2, and I'm watching both closely → Step 1, roughly SAE Level 2
- 5 to 10, each on its own worktree, I check in periodically → Step 2, roughly SAE Level 3
- More than 10, I mostly find out something ran when it's done → Step 3, roughly SAE Level 4
2. What actually happens before a task counts as "done"?
- I read every line before merging → Step 1
- I spot-check the diff, run the tests, trust but verify → Step 2 to Step 3
- Automated checks pass, agent merges itself, I find out from the CI notification → Step 3 to Step 4
3. How do you handle permissions?
- Manual approval on basically everything → Step 0 to Step 1
- Auto mode for safe operations, manual for anything touching prod or external accounts → Step 2 to Step 3
- Auto mode everywhere, guardrails do the filtering, not me → Step 3 to Step 4
Where I land, for what it's worth: Step 3, SAE Level 4. 5 to 10 worktrees on a normal day, Claude verifies its own work and merges without me in the loop most of the time (and Sonnet genuinely struggles with this specific pattern compared to Opus, for reasons I still haven't fully figured out), auto mode running pretty much everywhere except the handful of operations I've explicitly gated. High execution autonomy. And, per the Search Console story above, that score told me nothing about whether the agent understood why the gate existed in the first place.
What Breaks at Each Level
At Step 0 or Step 1, an error costs you time. It's annoying, it stays cheap, and you've forgotten about it by lunch, because you were watching closely enough to catch it in the first 10 minutes.
At Step 2 and Step 3, that changes shape entirely. The agent can optimize in a direction that's technically correct and strategically wrong, and nothing about the output looks broken. Tests pass, code compiles, and no red screen shows up saying "you died" the way it would in a game where the failure state is at least honest about itself. The Search Console isolation got broken by a clean, efficient commit that any code reviewer would have approved without blinking twice.
Small tangent, unrelated to any of this: my actual desk setup has 3 monitors and I still end up debugging on the smallest one out of pure habit. Anyway.
I dug into a related failure mode a while back, agents that report success when the underlying thing they were supposed to fix is still broken, same root cause: high execution competence, zero model of what "correct" means beyond the immediate task. Worth a read if this pattern is starting to sound familiar.
The Real Lever Is More Framing
I'd push back on this if someone else wrote it: the fix for high-step risk isn't slowing down execution autonomy. Ripping worktrees away from an agent that's earned Step 3 doesn't make it understand your business any better. It just makes it slower at not understanding it.
The actual lever is making the business context explicit somewhere the agent can actually read it, named constraints instead of implied ones, objectives written down instead of assumed shared knowledge. This is what CLAUDE.md is actually for, not a changelog nobody reads, the place where the "why" lives so the agent doesn't have to reconstruct it from a task description that was never going to contain it (I wrote about treating CLAUDE.md like a config file, not a README if you want to go deeper on the mechanics).
I think, and maybe I'm wrong about how far this scales past a small team, but framing the constraint explicitly would have caught the Search Console problem before it shipped. Not because the agent got smarter. Because it stopped having to guess.
If you're setting this up from scratch rather than retrofitting it onto an existing mess, the Blueprint method in Vibe Coding, For Real walks through exactly where this kind of context should live before you ever hand an agent real autonomy.
One honest caveat here: framing reduces the risk, it doesn't erase it. Nobody's shipped a CLAUDE.md good enough to make judgment calibration a solved problem. You're buying fewer surprises, not zero.
I spent way too long being proud of my Step 3 score before I noticed it was measuring the wrong thing entirely 😅
More execution without more framing just means hitting the wall faster.
Sources
- Boris Cherny's thread on the 5 steps of AI adoption, posted July 17, 2026
This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).
Top comments (0)