DEV Community

Sho Naka
Sho Naka

Posted on AI-assisted

I'm dropping spec-first AI development for smaller experiments

The decisions and experiments here are the author's own. An AI assistant drafted the English text from the author's notes, and an independent AI review checked the links and the one third-party claim before publication. #ABotWroteThis

Quick answer

For my own reversible development work I am replacing specification-first as the default with small working experiments: state the outcome, build the smallest change that lets me try it, and judge the running result. Prohibited operations are rejected by a pre-execution hook rather than described in a warning, and the main measure is elapsed time from request to a usable result. This is a decision under test, not a measured productivity gain.

I gave my coding agent a link to Ponytail, a set of rules intended to stop agents from overbuilding. It returned a cautious assessment: pushing for fewer files and one-line implementations might hurt readability or fit poorly with the existing architecture.

My response was a question: readable for whom?

If an agent does most of the reading, changing, and checking, I want the structure that helps it make the next correct change with the least context. An architecture should earn its maintenance cost, even if I designed it myself.

That conversation led me to a decision: for my personal development work, I want to replace the default specification-first process with small working experiments. I want short instructions for judgment, hard enforcement for prohibited operations, and elapsed time to a usable result as the main measure.

This is a decision I am testing, not a report of proven productivity gains.

Keep the outcome stable. Let the implementation change.

My reason for changing this default is the human cost of evaluating the plan. Even if an agent writes a careful plan instantly, I still have to imagine the resulting behavior and find what is missing.

I have asked an AI to check implementation concerns in plan mode and proceeded carefully, yet the resulting system did not meet the customer's needs. That experience does not establish that plan mode caused the failure. It does show that checking whether something can be implemented is insufficient evidence that it will serve the customer's work. Plan mode and specification-driven development are not identical; I am questioning the shared habit of expanding a document before testing the behavior it describes.

Some behavior only becomes clear to an industry expert or someone doing the work. For example, correcting an already processed request might require preserving its history and returning it for review, rather than overwriting it. Both behaviors are technically implementable. This is an illustrative example, not a claim about the customer system. A more internally consistent plan cannot supply a missing business premise by itself.

I would rather spend ten seconds writing a prompt, let the agent implement a small change in a minute, and judge the result. Those numbers describe the short loop I want, not a measured guarantee. A working interaction can make a mismatch easier to identify than a description that I must simulate in my head.

By spec-first, I mean making a written specification, plan, and detailed task breakdown the default prerequisites to implementation. Tools such as GitHub Spec Kit put specifications at the center of development. I am changing that default for my own reversible work, not claiming that every specification-driven process is slow.

Suppose I want to find an item in an existing list. I can ask an agent to design a search subsystem, or I can ask it to make the smallest working filter and let me try it.

The first experiment needs a concrete outcome: a user can find a known item by its title. It does not need a commitment to a particular component library, search service, or abstraction.

I would start with a brief like this:

Outcome: find a known item by title in the existing list.
Use existing code or platform features where they meet that outcome.
Build the smallest reversible change that lets us try it.
Check a matching query, a query with no matches, and clearing the query.
Follow the existing authorization boundaries.
Report what works, the observed limitation, and the next change if needed.
Enter fullscreen mode Exit fullscreen mode

Those checks define this experiment's scope. They do not prove that a production search feature handles pagination, large datasets, permissions, accessibility, or every language correctly. If the list only contains one loaded page, that limit must be visible when judging the result.

The useful loop is short: build, try, change or discard. A prototype that cannot answer its intended question has not completed the experiment.

Readability is the cost of the next correct change

Before considering code structure, I need to address a trap in the shorter loop: the agent and I can keep saying “it works” while neither of us has the missing domain knowledge. Replacing plan iterations with implementation iterations does not automatically introduce customer evidence.

Marty Cagan distinguishes value, usability, feasibility, and business viability risks. Checking that we can build something addresses only one of them. The Four Big Risks

For the next experiment, I want a concrete example from the actual work, appropriately anonymized, rather than only tidy dummy data. Existing understood behavior can be checked automatically. An unresolved business decision should be shown to a practitioner through that specific example. I do not need to ask the customer about every edit; I need their knowledge where it can change the next decision.

Martin Fowler describes discovering requirements by observing what users try to do and where they abandon a task. That is a useful counterweight to checking only our own written assumptions. Observed Requirement

Another trap is accumulating every quick prototype. If an experiment disproves the approach, discard the implementation. Fast generation stops helping when each addition makes the next change harder to understand and verify.

Keep the next change affordable

Ponytail's rules ask the agent to look for existing code, the standard library, native platform features, and installed dependencies before writing something new.

That order appeals to me. A native date input can remove a custom component from the code I must maintain, provided it meets the actual requirements.

But I am not optimizing for the smallest line count. A dense expression that needs repeated explanation can cost more than a few straightforward statements. Equally, a small change spread across wrappers and duplicate definitions can require more context than keeping the behavior together.

My question is: how much reading, reasoning, editing, and verification does the next change require? If a human will maintain the result, their understanding belongs in that cost too. “It matches the existing architecture” is evidence of consistency, not proof that the architecture is worth preserving.

Make prohibited operations impossible through the controlled path

I also told the agent that I want enforcement, not warnings.

A long instruction saying “be careful” still asks the model to interpret the rule on every relevant turn. A warning that allows an operation to continue leaves the final decision elsewhere.

For a mechanically decidable prohibition, I want a pre-execution hook or equivalent control to reject the operation. The response should identify the failed condition and a permitted next action. If the agent can repair the request within its authority, it should continue.

This only works when the operation actually passes through the control. Blocking one shell command does not enforce a boundary if another tool can perform the same write. Enforcement belongs at the relevant execution boundary, with the paths it covers verified.

Product quality is a different job. A hook cannot generally decide whether the filter is useful or whether the screen makes sense. I want those questions answered by the working result and a focused check, rather than an expanding collection of vague warning rules.

Optimize the whole request, not the last response

The agent also noted that adding a plugin can add hook processes. That is a cost to measure, but it is not a reason to reject the approach by itself.

A small deterministic check might replace repeated instruction processing and warning conversations. Or a verbose ruleset might add more input and reasoning than the shorter implementation saves. The balance has to be measured.

Ponytail's author reports exactly this kind of variation. A separate single-shot experiment found higher cost and worse latency with GPT-5.5. That is not a result for my workflow, but it is enough to reject “less code always means cheaper and faster” as an assumption. Cost verification

I therefore want to replace redundant instructions, not keep adding new ones. A short reply that stops halfway through the task and requires another context reload is not automatically efficient.

Start the clock at the request

When I asked the agent to turn this decision into an article, I added: elapsed time is itself part of the result.

I want to measure from the request to the artifact I can actually use or judge. Research, setup, waiting, and repairs stay on that clock. If I ask for a working feature, the clock does not stop at code generation. If I ask for an experiment, it stops when the intended question has been tested.

For a first comparison, I would record:

Measure What it tells me
Request-to-result elapsed time How long I actually waited
Total recorded tokens, where available The model work spent across the request
Repair rounds Whether the initial shortcut moved work later
The same outcome checks Whether the two results are comparable

A fair comparison keeps the starting state, model settings, requested outcome, and necessary checks the same. Without that comparison, I can report a duration; I cannot attribute an improvement to dropping specifications.

I am starting with one reversible task. Keep the outcome clear, build only enough to test it, enforce the known prohibitions, and count the full time to the result. If that loop does not improve delivery, the next thing to question is the process I just introduced.

FAQ

Is this an argument against specification-driven development in general?

No. It is a change of default for my own reversible, personal work. Tools such as Spec Kit exist for cases where a shared written specification is the cheapest way to align people; my problem is the cost of evaluating a plan in my head when a running change would answer the same question faster.

Why enforce prohibitions with a hook instead of instructions?

An instruction asks the model to re-interpret the rule on every turn, and a warning that lets the operation continue leaves the decision elsewhere. A mechanically decidable prohibition belongs at the execution boundary, where it can reject the operation and name a permitted next action. It only works if every path that can perform the operation goes through that boundary.

How will I know whether the shorter loop is actually faster?

By measuring request-to-result elapsed time, recorded tokens where available, repair rounds, and the same outcome checks, with the starting state and model settings held constant. Until that comparison exists I can report a duration, not an improvement.

This article was drafted with AI from my actual discussion and decisions. The proposed workflow's productivity effect has not yet been established.

Top comments (0)