Hayati

Posted on Nov 8 Originally published at hayatibis.dev

Nobody asked for this

#softwareengineering #ai

Originally published on hayatibis.dev, where the figures move and the toys are interactive.

I’ve run into the gap between an idea and what gets built many times. An illustration:

A dental clinic finds a freelance developer through a friend and asks for “an online booking system”. The developer builds a calendar, patient records and an admin panel. At the handover the clinic says “we didn’t ask for this”. Their real problem was patients who don’t show up, chairs sitting empty in the middle of the day. A reminder sent the day before would have solved most of it.

In a product company, again as an illustration, the same gap comes in other clothes. Sales asks for an “export to Excel” button, and the team builds a filtered export on every table. What sales wanted was a list, every Monday, of the customers whose usage dropped, so they’d know whom to call.

Nobody said anything misleading, but the meaning got lost on the way.

Illustrative. After the tree swing cartoon engineers have passed around for decades, redrawn for the clinic. (interactive version)

Why it shows at the end

The misunderstanding happens at the start, but it only shows at the end, so the last person in the chain pays for it. Even a contract usually writes down the sentence that was asked for, not the need behind it.

Illustrative. The same request passes through four hands. Without the questions, the sentence survives and the need doesn’t: every hand builds a little further from it, and the agent’s green checks only prove it built what it was told. Ask first, at the clinic, and the need travels with the work as an example the clinic approved and a test can check. (interactive version)

Fred Brooks saw this in 1986. In No Silver Bullet, his essay on why no single tool would make software ten times easier to build, he writes: “The hardest single part of building a software system is deciding precisely what to build.” A paragraph later he adds that clients “usually do not know what questions must be answered”.

So someone has to ask them. Most of us do ask, just too late, in the design review or at the handover.

The right time, the right place, the right questions

The right time is before anyone talks about a solution. Once “a booking system” is on the table, every question starts circling it.

The right place is the desk of the person who does the job. Watch how the clinic’s receptionist keeps appointments today, and how they notice a patient who didn’t come, with whatever they use now.

The questions are plain:

What decision will you make when you open this screen?
How do you make that decision today?
Can you show me a real case from last week?

At the decision points, the questions that come up while building aside, I try to stay with one sentence:

you can outsource your thinking
but you cannot outsource your understanding

— kache (@yacineMTB), 4 February 2026

In April Karpathy said in a talk at Sequoia that he keeps thinking about it “like every other day”, and a day after the video went up he quoted it as the quote he’d been citing a lot recently. In the talk he adds: “the LLMs certainly don’t excel at understanding. You still are uniquely in charge of that.” You can hand the thinking to an agent; you can’t hand over the understanding. An agent writes the calendar in a few hours. Understanding that the clinic’s problem is empty chairs is something else.

Then turn the answers into a few concrete examples the people who asked have approved, like “every patient with an appointment tomorrow gets a message today at 18:00” (illustrative), and turn the examples into tests. The argument stops being about who said what and becomes about whether the example passes.

We used to work hard to keep the developer’s part limited to technical questions. But times have changed: the person who designs a feature and the person who builds it are slowly becoming the same person. I don’t know if that separation is still needed; the developer doesn’t seem to be a separate role any more. So asking the question and understanding the answer land on the same person.

When the last hand is a loop

More and more often the hand that builds is an agent. First came vibe coding, where you describe what you want and the model writes it, then agents writing and reviewing while a person steers. Now we talk about loops that run for hours on a goal without anyone in between.

Karpathy’s autoresearch is the cleanest example I know. The idea, in its own words: “give an AI agent a small but real LLM training setup and let it experiment autonomously overnight.” The agent changes the training code, trains for five minutes, checks if a single metric got better, keeps or discards the change, and repeats.[1] “You wake up in the morning to a log of experiments and (hopefully) a better model.”

It works because the task is verifiable. Karpathy wrote about this in a short post, Verifiability: for a task to be automated this way, the environment has to be resettable, efficient and “rewardable (there is some automated process to reward any specific attempt that was made)”. He sums it up in two lines: “Software 1.0 easily automates what you can specify. Software 2.0 easily automates what you can verify.”

Now put the clinic’s booking system in that loop. The tests pass, the calendar shows every appointment, the lint is green. But whether the chairs stay empty can’t be measured overnight; the answer comes weeks later, at the clinic.

Illustrative numbers. autoresearch trains for five minutes per experiment, about 12 an hour and about 100 while you sleep (its README), and keeps a change only if val_bpb goes down. Switch to the clinic and the same tireless loop has only the tests to go by: they stay green all night while it adds screens, and the empty chairs never reach its log. (interactive version)

A loop that never gets tired keeps chasing the question it was given. If the question is wrong, it grows the wrong answer with the same patience.

When I vibe code a part I don’t understand, building something many people will use and get value from is, I think, very hard.

Can the agent ask instead?

Partly. This essay is a small example. While I was preparing it, an agent asked me one question at a time, because a skill file someone wrote tells it to ask first and build later. Nothing it asked changed my mind, but it let the idea cook and mature. The answers, what I’d seen and what bothered me about it, it couldn’t find anywhere; they had to come from me.

So an agent can ask the right questions if we show it how. Unless someone brings the problem into its context, it can’t go to the clinic and ask the receptionist the right question, for now :)

Where the work is going

Karpathy’s MenuGen story is a good example here. He often got stuck on restaurant menus, not knowing what the dishes were, so he built an app that shows a picture of each dish when you photograph the menu, without writing a line of it himself: MenuGen. In the same talk he told the rest. Later it was enough to give the photo to Gemini and ask Nano Banana to draw the dishes onto the menu; the model gave back the same photo of the menu with the dishes rendered into it. In his words, “all of my MenuGen is spurious”, and “that app shouldn’t exist”.

Karpathy’s story from his Sequoia talk, redrawn; the menu and the drawings are illustrative. The app went away, and the need stayed: seeing what a dish looks like. (interactive version)

In the same talk he describes the LLM as a new kind of computer. Today neural nets run “virtualized on existing computers”; one day that may flip, and “the neural net becomes kind of like the host process” with the CPUs as the co-processor. Going on from MenuGen, he says “a lot of this code shouldn’t exist and it’s just neural networks doing most of the work”.

The app went away, but the need stayed: seeing what a dish looks like. The clinic’s problem wasn’t a booking system either; it was empty chairs.

What we build can change, and like MenuGen it may one day not be needed at all; what doesn’t change is understanding what’s needed.

As writing code gets cheaper, the part before it gets more valuable: asking at the right time and in the right place, and turning the answer into something a loop can check.

The card

Five lines to fill before work starts, by a person or by an agent, a small checklist. I filled this one in with the clinic’s request; tap a line to call it out.

Fig. 5 · Illustrative. A card filled in with the clinic's request. The fourth line is written so a test can check it. (interactive version)

The fourth line is also a very good instruction you can give an agent: it turns “build a booking system” into something it can verify.

Back at the clinic, the calendar worked, and nobody had asked for it. When did your agent last ask you, without you telling it to, who is going to use this, and for what?

Stands on

Notes

The metric is val_bpb, validation bits per byte, and lower is better. The human edits one file, program.md, which tells the agent how to run the research; the agent edits train.py.

A new essay every Sunday: get it by email. If you'd like more of these, you can support the writing on Patreon, or buy me a coffee once.

DEV Community