DEV Community

Cover image for The bottleneck isn't the AI
Hideki Mori
Hideki Mori

Posted on

The bottleneck isn't the AI

On a Friday in July I started the morning convinced a production model had lost its mind — handed an image it couldn't read, it didn't error, it fabricated a complete invoice. By evening the investigation had gone through a misdiagnosis, a correction, and a statistical confirmation — seventy runs out of seventy — and ended with the product description fixed in production and a memo for the article you may have already read. Discovery to ship, one day, one loop.

I assumed this was an ordinary way to work now. Everyone has AI; surely everyone does this. Then I looked at the numbers.

The far end of the distribution

The largest usage study to date — 1.5 million ChatGPT conversations, analyzed by OpenAI with NBER — reads like an inventory of small tasks. Writing alone is 40% of work-related messages, and most of those are edits to text the user already had. Around half of all messages are "Asking" — people really do use models as advisors — but as a transaction: one question in, one answer out.

Where I live, access has just stopped being the story. Japan's FY2025 government survey puts individual experience with generative AI at 58.8% — a majority at last, up from 26.7% a year earlier (52.2% on last year's age basis; the survey newly added teenagers). Intensity hasn't caught up. Among people who have used it, about 30% report near-daily use; 11% report near-daily use of an hour or more. Across the whole population, "an hour a day with a model" describes roughly one person in fifteen.

So the median AI interaction is a discrete task, and the median AI day contains a few of them. A day of accumulated context that closes with execution before the date changes isn't the median, and it isn't what people mean by "power user" either. What's rare was never asking an AI for advice. It's the product of three factors — depth × duration × coupling to execution — and the last factor is the one no model can supply. Hold that thought; it's where this ends.

What follows isn't a prescription. It's the checklist I run so a future me can rebuild the session in thirty seconds: four parts, in the order they earn their keep. When one is missing, the day produces conversation instead of decisions.

Part 1 — Frame with questions, not a request for answers

The session that produced No sentence, no component did not start with "design me a job-distribution system." It started by fixing five questions before any answer was allowed: Can the job be split? Does it survive failure? Am I adopting anything I can't operate? Can twenty-year-old tools cover it? How simple can it get?

Fixing the questions first changes what the model's output is. Ask for a design and you get proposals — fluent, plausible, interchangeable. Ask it to testify against five fixed questions and every answer acquires an axis it can fail on. The question set is also where you return when the day drifts. It drifts. By mid-afternoon the anchor is worth more than the intelligence.

Rule: a fixed question set turns proposals into testimony.

Part 2 — Cross-examine; never let it agree with you

Agreement is the failure mode. A model's default is to be helpful, and the cheapest way to be helpful is to extend whatever you just said.

Three mechanics: I withhold my hypothesis and make the model commit first. I make it defend the conclusion I'm leaning away from. And I run a second model against the first's output — different vendor, different failure habits.

Two artifacts from real days. During the fabrication investigation, an early test convicted one model of a brand-new decimal bug — a tidy conclusion with numbers attached. The conviction was wrong. My example output used decimals like 10.0, which the JSON path silently flattens to integers, so the schema had been demanding integers from the start; the "bug" was my own test design echoing back at me. What broke the story wasn't another run — it was the question I'd skipped: had the example itself even survived the JSON round-trip? Re-tested with 12.34-style decimals, every model walked free. Agreement hadn't meant the diagnosis was right; it meant the test and I shared a blind spot.

The second: another AI once handed me a confident, well-structured plan — an SEO play for my own name, complete with a profile template. A separate session audited it against primary sources: the actual search results page, the actual profiles. Every link in the template pointed at a platform homepage instead of my page — checkably wrong, replaced. Then the auditor overreached: it flagged a board-level title in the template as a fabrication. The title is real. The auditing model had no registry to check. Somewhere in its reasoning, unverifiable hardened into invented. Cross-examination doesn't produce verdicts; it produces two piles — falsified and unverified — and the second pile goes to the only party who can settle it.

This is the part that scales with speed — in the wrong direction. Compression amplifies error: a one-day loop without cross-examination is a day spent accelerating into a mistake with rising confidence.

Rule: suspect your test design before you suspect the model. Agreement is data about your blind spots, not about the world.

Part 3 — Log the discards

Dialogue transcripts are write-only. Nobody — including me — re-reads three thousand lines of chat. What survives the day is a decision log, and its unit is not "what I picked" but traded / got / rule: what each option would have poisoned me with, what I received instead, and the one sentence that generalizes.

The discipline that matters is keeping the reversals in. The fan-out log contains, on the record, a design I wrote first because it was beautiful — and discarded when the operational bill arrived. A log of only final answers reads like genius and teaches nothing. The discarded branch is where the information lives; it is also the only honest reply to a future me asking "did you consider X?" Yes. Here is exactly what it would have cost.

Rule: the discarded branch is the payload.

Part 4 — Exit with execution already started

A session does not end with next steps. It ends with something already shipped, changed, or committed — same day.

The fabrication Friday closed with the catalog description corrected in production and the article memo written. The SEO session closed with every profile already updated; the exit note held exactly one line — measure in two to four weeks — with a date on it. That's the whole test of an exit: the only things allowed to cross the date line are measurements and external waits. Nothing else.

If execution hasn't started when you close the laptop, the session didn't fail — the exit condition was mis-designed. Shrink the decision until a piece of it can ship today, and ship that piece.

Rule: only measurements and external waits may cross the date line.

Why a day is enough

A committee's day is spent identifying stakeholders and finding a slot on five calendars. That isn't slowness; that is what a committee's day is for.

My day has a different rate limiter. The slowest part isn't asking the model — it's waiting for my own scripts to finish. That's by design: verification happens outside the dialogue. Run the test locally, paste the results back — fixed criteria, reproducible, no burning tokens on theater. And the counterpart doesn't tire and doesn't take offense. You can ask the same question from ten angles, make it argue against its own morning position, twice, before lunch. No human sparring partner offers that — at any price you'd be willing to name to their face.

The honest limit

Everything above, you can do with a laptop. Except the ignition.

Framing, cross-examination, logging — a model supplies or supports all three. The fourth part, exiting into execution, is the one input that comes from somewhere else entirely: the license to act. I hold it by construction; I work solo. Most people running the same session inside an organization would do everything right and then hit the real exit condition of their environment: schedule a meeting.

That is not a character defect. It's incentive design doing exactly what it was built to do — nobody in the room owns both the risk of acting today and the reward of having acted. So their day ends where mine begins.

Which is why buying AI does not automatically make an organization faster. If judgment closes in a day but approval crosses weeks, the model has added thinking quality to the waiting time — a better-reasoned queue. The speedup lands only where judgment and execution live in the same person: solo operators, and small teams with genuinely delegated authority.

So when someone tells me AI hasn't changed their company's speed, I believe them, and I don't blame the model. A good session gets you to the decision by lunch. Whether anything has moved by dinner was decided long before anyone opened a chat window.

The bottleneck isn't the AI.

Top comments (0)