DEV Community: Shinsuke KAGAWA

Was the Monorepo Really Best for Coding Agents?

Shinsuke KAGAWA — Thu, 20 Aug 2026 10:55:58 +0000

Earlier this year, I thought monorepos were the obvious choice for AI-native development.

The reason was context closure.

A coding agent works better when the code, tests, design decisions, infrastructure, and development rules it needs are available inside the same execution environment. It should not have to wait for a person to paste a decision from Slack, explain a convention that lives in someone's head, or retrieve a requirement from a system it cannot update.

A monorepo made that easier. Put the product and its development context in one place, give the agent tools to search it, and the repository becomes a closed environment in which the agent can work.

I still believe context closure matters. What changed was my view of what the repository was closing.

I was involved in a discussion about introducing a shared agent harness across a large repository. The technical proposal was not the hardest part. The harder question was what would happen to the workflows engineers already used.

If the shared harness required people to stop using their personal skills, instructions, and workflows, would they actually do it?

There was no confident yes.

By then, the repository already contained more than code. It contained several partially overlapping ways of telling agents how to work, some checked in by teams and others maintained by individuals. Introducing a common harness was no longer an installation. It was a migration away from systems people depended on every day.

I had seen the opposite failure too. Centralize before the organization knows how to design, evaluate, and maintain a reliable harness, and the shared system becomes a ceiling. Poor instructions affect everyone. Reviewers absorb the correction cost. AI usage remains shallow because the approved path is less effective than what capable individuals could build for themselves.

The two approaches appeared to fail in opposite directions. Centralize too early, and the organization standardizes what it has not yet learned to do well. Standardize too late, and local workflows become too embedded to replace.

Where the code lived mattered less than who could define how an agent should work across it, resolve conflicting context, and keep the system correct as both the code and the models changed.

The repository stopped being the context boundary

The monorepo argument was stronger when coding agents were less capable.

An agent that could only sustain a short task benefited enormously from having everything nearby. Repository boundaries were practical context boundaries. Work that crossed them required more prompting, more retrieval infrastructure, and more human coordination. Keeping related systems together reduced the chance that the agent would miss something important.

The current long-running models I use behave differently. Given access to the relevant repositories and tools, they can search across projects, trace dependencies, compare implementations, and make coordinated changes without treating each repository as a separate task. Access still has to be designed, but the repository boundary is no longer the same kind of capability boundary.

Cross-repository reach does not make the surrounding structure optional. In a monorepo, broad search still needs package, ownership, and domain boundaries to separate relevant code from similar but unrelated code. Across repositories, search depends on a trustworthy inventory and dependency map, or an incomplete result can appear complete. Better models need less assistance to follow that map. They do not remove the need to maintain it.

This claim is about context acquisition and edit coordination. Repository boundaries still matter for atomic commits, CI, compatibility, and deployment, and better models do not remove those constraints. They do change the monorepo tradeoff for AI development.

Code visibility is still useful, but visibility is no longer exclusive to a monorepo. An agent can discover an API or inspect a dependent service across repositories. It can update several repositories in one run. Producing the edits is usually easier than determining whether they may be deployed together, which compatibility window must be preserved, and how the rollout should be sequenced.

As repository boundaries stopped being context boundaries for coding agents, the monorepo's main AI advantage moved from retrieval to governance.

A monorepo gives product-wide instructions, shared skills, architectural context, and guardrails a common home. The same repository can carry the evaluations and maintenance work that keep them current. That is a real advantage. It is also the part most organizations are least prepared to operate.

Context closure is not context governance

In What Makes Development AI-Native?, I described context closure as the condition in which product context is searchable, readable, and updatable within the LLM's workflow.

That remains necessary. But availability answers only one question: can the agent reach the context?

Governance answers the questions that become important after it can:

Which context is authoritative?
Where does it apply?
Who owns it?
What happens when two instructions disagree?
How is an obsolete rule removed?
How was it evaluated, and who is responsible when local context changes product-wide behavior?

A repository can close context without answering any of these.

This becomes harder as the organization grows. A product large enough to justify a monorepo is often developed by multiple teams. Those teams are deliberately kept loose enough to work without constant coordination. Their code, architecture, release processes, domain knowledge, and agent workflows diverge because some divergence is necessary for autonomy.

A team creates a skill to fix a recurring failure. An engineer adds a personal instruction because changing the team's workflow would affect everyone. Another team solves a similar problem differently. Each addition is locally reasonable. Over time, the repository, team, and individual layers accumulate overlapping assumptions about how work should be planned, reviewed, tested, and completed.

The engineer becomes isolated inside progressively narrower context:

product context
  -> system context
    -> team context
      -> personal context

The deeper the local system becomes, the harder it is for that engineer to change the shared one. A personal workaround is cheap. Correcting a repository-wide workflow requires agreement, migration, and responsibility for everyone it affects.

Local convenience gradually removes the force that would otherwise improve the common system.

Better models make conflicting context more expensive

Missing context is still dangerous. A model cannot reliably honor a constraint it cannot discover.

But missing context is no longer the only important failure mode. Incorrect context, conflicting context, and excessive control can be worse because a capable model can execute them more thoroughly.

I described this shift in When Better Models Make Old Agent Workflows Worse. Rules written to compensate for earlier model failures can become work-generating constraints, leading the model to create the documents, tests, reviews, retries, approvals, and abstractions they imply. The same problem appears at organizational scale in a monorepo.

One team skill may require every change to pass several review phases. Another may instruct the agent to choose only the narrowest proof boundary. A product-level rule may preserve a public contract while a local workflow optimizes for an internal replacement. A personal instruction may require a planning artifact that the shared harness no longer uses.

Each instruction can be defensible in isolation. Their combination can reduce execution quality while appearing more mature because more context and more process are present.

AGENTS.md at least provides a directory-based preference. Codex loads instructions from the repository root toward the working directory, and instructions closer to the working directory take precedence. That does not prove that the local rule is correct; the product-wide rule may be the one that should win. But it gives the model a deterministic guide when the two disagree.

Skills are harder to reason about because they use progressive disclosure. The agent initially sees names and descriptions, then decides which full instructions to load for the task. What enters the context depends on how the task is interpreted. When several skills apply, there is no equivalent semantic hierarchy that says a product-level decision must override a team workflow or a personal preference.

Which instructions enter a task can therefore be hard to predict. Changes in task wording, working directory, or model version can activate different combinations, allowing an organization to improve its harness while quietly reducing performance.

Centralizing before the capability exists

A natural response is to centralize everything, but a shared harness for a large product requires much more than prompt files. Its maintainers need enough product knowledge to separate durable boundaries from old compensation rules and keep context aligned with the codebase. They also need representative evaluations that expose model regressions and workflows that no longer earn their cost.

Prompt and context engineering skill is necessary, but not sufficient. A person who understands model behavior but has not developed the product cannot create accurate domain context. A person who knows the product but cannot evaluate agent behavior may encode every historical incident as another permanent rule.

No small central group can deeply understand every domain in a large monorepo while also keeping pace with model changes. If that group becomes the sole author and gatekeeper, humans become the bottleneck. The common harness drifts away from actual development, teams work around it, and standardization suppresses the very experimentation needed to improve it.

Usage and spending may still increase even while outcomes stagnate. When a weak workflow is centralized, its limitations become organization-wide.

Standardizing after local systems are embedded

Letting teams and individuals find what works produces faster initial adoption. Engineers can experiment without changing anyone else's process, and useful practices emerge from real work rather than from a central design exercise. Autonomy without a path to convergence, however, creates its own lock-in.

A personal skill is not merely a file once someone has shaped daily work around it. It contains learned behavior, trusted shortcuts, and assumptions about how the agent will respond. A team workflow becomes part of planning, review, and delivery. Replacing it has a switching cost even when the replacement is better.

By the time the organization needs a repository-wide harness, it has postponed the difficult decisions: which local behavior should become shared, who has authority to retire the old systems, and how to prove that migration will not reduce performance. Without those answers, the shared harness joins the existing workflows instead of replacing them.

This brought me back to the question from the original discussion: would engineers give up the personal systems they already relied on? It had sounded like an adoption question. It was really a question about migration authority. The organization wanted one harness but had never established who could retire the systems it would replace.

Centralize governance, distribute its maintenance

If a monorepo is going to provide a common AI development environment, its context and harness need a common authority. Product-wide constraints cannot become optional because a team prefers another workflow. The organization needs one way to decide what is canonical, how conflicts are resolved, and when a rule is obsolete. Common authority, however, does not mean that a central team can do all the work.

Shared authority needs distributed capability

Engineers need more than the ability to ask a model for code. They need to understand how task framing, context selection, instruction scope, progressive disclosure, verification, and model behavior affect the result.

Not everyone needs to become a specialist in agent evaluation. Everyone contributing context to the shared system needs enough skill to avoid turning a local preference or historical failure into a product-wide obligation.

The capability cannot be hidden behind a platform team. Product and system context has owners, consumers, compatibility requirements, and a lifecycle. Personal preferences can remain personal when they affect presentation or reversible working style, but they cannot silently redefine correctness, product behavior, approval boundaries, or completion criteria.

Teams that own a domain must also own the accuracy of the context that describes it. Some members need explicit capacity beyond immediate delivery: maintaining shared instructions, reviewing cross-domain assumptions, contributing representative tasks, and resolving conflicts in the repository-wide harness. Governance is centralized; its evidence and maintenance are distributed across the organization.

The harness should need less context over time

Context quality cannot depend on occasional manual cleanup. References, ownership, duplication, schemas, and retired files can be checked mechanically. Representative tasks can detect whether a model update changes behavior. Conflict cases can test which rule wins when product, system, and team instructions disagree. The harness can measure whether a rule improves accepted outcomes or merely creates more work.

Automation should concentrate human judgment where it changes a real decision rather than spend it on routine consistency checks. The evaluation set must evolve with both the codebase and the models. A skill that passed six months ago has not earned permanent authority.

Continuous evaluation is only half of the work. Context and harnesses cannot compensate indefinitely for an inconsistent codebase.

When two parts of the system solve the same problem differently, an instruction can explain the difference. When five parts do, more instructions become a map of unresolved design debt. The model must spend more tokens discovering which example is current, which exception is intentional, and which pattern is safe to extend.

The durable response is to reduce the inconsistency itself. Public contracts should be explicit. Invalid dependencies should be mechanically rejected. Types, tests, build boundaries, schemas, and permissions should enforce what prompts currently have to explain. Temporary compatibility paths should be removed instead of accumulating permanent context around them.

The strongest agent instruction is often a codebase in which the wrong change does not fit.

Judge governance by the cost of outcomes

How should an organization know whether this governance is working? Usage, generated lines, completed tasks, and total spend reward activity. The relevant economic unit is the total inference cost required to produce one accepted outcome at the required quality without human correction.

That includes model choice, retries, abandoned runs, review agents, unnecessary artifacts, excessive exploration, and work generated by obsolete instructions. Some of that cost is legitimate. Exploration can establish that no change is needed, but it still belongs in the cost of reaching that outcome. Repeated exploration that better context could prevent should fall over time. A poor harness may still complete the task by consuming more reasoning, but the organization is purchasing inference that its development environment should have made unnecessary.

The number is a health indicator, not an optimization target. A lower cost is an improvement only while the acceptance bar holds and accepted outcomes stay correct after review.

If a reviewer has to reconstruct the result, the autonomous workflow did not produce the accepted outcome. This measure also gives the organization credit for restraint. The right outcome may be a small change, reuse, removal of a conflicting workflow, or a decision that nothing should be built. As models become capable of sustaining more work, preventing unnecessary work becomes part of operating them well.

The monorepo is not the answer by itself

I still see a strong case for monorepos in AI-native development. One repository can provide a canonical harness, keep product-wide context maintainable through the same workflow as the code, and make shared evaluation and mechanical enforcement easier to operate.

Realizing those benefits requires common context ownership and distributed product knowledge. Without them, a monorepo can contain more contradictions than an agent can reliably reconcile, while a shared harness can enforce the wrong workflow everywhere. A multi-repository system with deliberate cross-repository access, explicit contracts, and well-governed context may no longer impose the disadvantage it once did for capable agents.

Repository topology alone cannot decide the outcome. What matters is whether the organization can operate the context system that topology makes possible.

Conway's law did not stop applying when the executor became a model. The communication structure of the organization shapes the codebase; it now also shapes the instructions, skills, workflows, and evaluation systems through which agents change that codebase.

If teams are responsible only for local delivery, their agent context will optimize for local delivery. If individuals are rewarded for personal productivity, their best workflows will remain personal. If a central group owns the harness without the participation of domain teams, the harness will reflect the central group's limited view of the product.

AI-native development therefore requires more than distributing a coding tool or checking a set of skills into a repository. The development process itself has to become an engineering responsibility. Teams must treat product-wide quality as part of their work, maintain the system that turns intent into autonomous execution, and make the codebase consistent enough to require less explanation over time.

The monorepo once looked like the answer because it put everything the agent needed in one place.

Now the agent can reach much farther.

The harder question is whether the organization can agree on what the agent should find when it gets there.

The repository is no longer the context boundary.

The organization is.

Reasoning Effort Is Not a Quality Setting

Shinsuke KAGAWA — Wed, 05 Aug 2026 13:25:25 +0000

I expected Claude Opus 5 high to produce a better design than Opus 5 medium.

It did not.

I had given both configurations the same repository, feature request, and design template. The task was to add dependency-aware execution to Galley, an open-source local runtime for supervised, multi-model AI coding. A task could declare other tasks as dependencies, and the daemon would claim it only after all of them had succeeded.

In the blind evaluation, Opus medium's design received 84 points from Claude and 82 from Codex. Opus high received 68 and 50. They took almost exactly the same amount of time (21 minutes and 12 seconds for medium, 21 minutes and 25 seconds for high), while high processed 2.29 million tokens to medium's 1.56 million.

My first reaction was that something had gone wrong in the evaluation. I read both critiques again, then the designs and session logs. The scores were not the most interesting part. The designs were.

High explored waiting deadlines, persisted archive history, additional configuration, more state transitions, recovery behavior for those transitions, and tests for the machinery it had introduced. Much of it was technically plausible. Some of it addressed real failure modes. Taken together, those additions made the requested feature larger and harder to operate.

Medium explored less. It stayed closer to Galley's existing lifecycle and reused more of the current claim and failure machinery.

High found more things to think about.

That was the problem.

I had treated effort as a quality setting. In these runs, it behaved more like an exploration multiplier.

What I actually measured

The feature was not a self-contained coding exercise. Galley stores tasks as YAML files and moves them through lifecycle directories. Multiple daemon processes may compete to claim work. Adding dependencies affected parsing, scheduling, failure recovery, CLI output, backward compatibility, filesystem error handling, and tests that had to prove execution order without calling an external model.

I ran seven model-and-effort configurations through four separate phases:

technical design;
design review;
implementation from a common approved design;
code review of a common implementation.

The configurations were GPT-5.6 Sol medium and high, GPT-5.6 Terra xhigh, GPT-5.6 Luna max, Claude Sonnet 5 high, and Claude Opus 5 medium and high. Within each phase, every configuration received the same repository state and the same inputs. These were four controlled phase comparisons around one feature, not seven end-to-end projects in which an early design changed the later implementation.

For design review, all seven configurations reviewed the unmodified Opus medium design. Between phases, I worked with the orchestrating LLM to prepare controlled inputs from the original requirements and the recorded artifacts. For implementation, all configurations received the same deliberately revised design. For code review, I fixed the target to the Sonnet high implementation from the implementation phase so every reviewer inspected the same production code and tests.

I deliberately did not tell the models to produce the smallest possible change or prescribe a convergence method. The request fixed the outcome, constraints, and exclusions; deciding how much machinery they justified was part of the engineering task I wanted to observe. An explicit subtraction rule would test compliance with a supplied convergence strategy, which is a different question.

The outputs were anonymized and evaluated independently by Claude and Codex against a common rubric. I treat the scores as measurements attached to the artifacts, not as verdicts on which model “won.” Concrete mechanisms, defects, preserved boundaries, and demonstrated failures carry more weight than small differences in averages, so I preserved both critiques when the evaluators disagreed.

There is one important defect in the design scoring that needs to be stated here rather than hidden in the notes. The original design request said that only runnable queued tasks could be claimed. The prompt used for the later blind scoring weakened this to saying that tasks with unfinished dependencies must not be executed; it omitted the prohibition on claiming them. This affected the later blind scoring of the design artifacts, not the design-review runs discussed below, which received the original requirements. Both Opus designs proposed claiming some unresolvable tasks and failing them afterward, so the omission did not obviously favor only one of them, and the evaluators separated the designs on broader scope and compatibility grounds as well. The design scores are still not absolute measures of requirement compliance.

In the design phase, both evaluators independently placed Sol high first and Opus high last, so the main direction was not explained by simple provider loyalty. Family and style preference did appear elsewhere, especially in design-review scoring. What I have is a structured case study from one repository, not a population estimate or a model leaderboard.

What high effort changed in Opus

Opus high did not misunderstand the repository. It inspected the implementation extensively and found real questions around concurrency, archive behavior, timing, recovery, and tests. The difference appeared after discovery: once high found a plausible concern, it was more likely to preserve that concern in the final design.

A rare race suggested another check. The check introduced a state transition. The transition needed persistence and recovery. The recovery path then needed tests and an operational explanation. Each step followed reasonably from the previous one, but the chain had little pressure to return to the current requirement and ask which mechanisms could be removed.

On a quick read, parts of the high-effort design looked reassuring. It had considered more failure modes. The overreach became clearer only when I traced each mechanism back to the requested behavior and asked what would stop working without it.

Opus medium showed many of the same instincts at a smaller scale. It still investigated failure handling and concurrent claims, but it generated fewer branches than it had to rank, combine, or reject. In this run, that amount of exploration fit its ability to turn the result into a bounded design.

I think of the observed pattern as an exploration-convergence mismatch. Exploration discovers files, contracts, alternatives, side effects, and failure conditions. Convergence decides what matters now, merges issues that share one correction, rejects optional hardening, and produces a result that fits the approved scope. Increasing effort can expand the first activity faster than the second.

The failure does not look careless. It looks like serious engineering applied to a larger problem than the one currently being solved. In a real codebase, someone still has to own every new state, setting, branch, test, and recovery path after the impressive design document is gone.

Twenty-two million tokens missed one sentence

The design-review phase showed the same imbalance from another direction.

Every reviewer received the original requirements, the same design, and the same repository state. The requirement explicitly said that a queued task with unresolved dependencies must not be claimed or executed. The design under review proposed claiming some such tasks first, then recording a failure afterward.

All seven configurations understood and summarized that mechanism. All seven found real problems elsewhere. Between them they processed 22.64 million tokens and produced 115 findings.

None identified the direct contradiction with the requirement.

The reviewed artifact was the Opus medium design from the opening. It was more bounded than the high-effort design and scored better in the blind design evaluation, but it still violated this governing condition. Better scope control and a stronger score did not make it implementation-ready.

They found partial-write hazards, race conditions, duplicate IDs, stale state, archive behavior, unnecessary scans, failure propagation, notification problems, and recovery gaps. Luna max uniquely followed a dependency cycle outward to a task that was not itself in the cycle. Opus medium found a subtle conflict between stale-claim recovery and a task that had waited for a long time.

The missing check was simpler than most of what they found. More independent exploration created diversity around the technical consequences of the design; it did not guarantee that anyone returned to the governing request and compared the selected mechanism with it directly.

The contradiction did not propagate into implementation. Between phases, I worked with the orchestrating LLM to prepare a revised common design from the original requirements and the review evidence. That controlled input kept both waiting and blocked tasks queued. The correction came from dataset preparation at the orchestration boundary, not from any of the seven review artifacts. This separation was deliberate: each phase compared model behavior against one controlled input instead of letting an early artifact determine every downstream result.

This is why I do not treat a high-scoring review as an approval decision. A review can be relatively strong, contain valuable findings, and still miss the condition that determines whether implementation may proceed.

The same behavior helped in code review

The code-review phase changed the value of wide exploration.

The common implementation contained defects outside its normal path. A lightweight task reader interpreted some valid YAML differently from the standard loader. Duplicate task IDs could cause one dependency result to be reused for another task. A partial failure while reading the task store could allow a dependent task to run before success had been established. Some CLI output described a different state from the one used by the daemon.

Opus high inspected production code, user-visible behavior, performance, and tests. More importantly, it changed a central dependency guard in a safe copy and showed that the main integration test still passed. The test looked like evidence for execution ordering. The mutation demonstrated that it was not sensitive to the behavior it claimed to prove.

No other configuration established that point as clearly. The blind evaluations reflected this: Opus high received 91 and 93, while medium received 88 and 91.

The high-effort review still carried the same weakness seen in design. It produced nine findings, six of them minor, split related problems, and did not always rank incorrect execution above a presentation issue. But a review has a downstream consumer. Candidate findings can be merged, reranked, or declined before they become production structure. Missing a defect is often more expensive than discarding an extra review item.

Design has no equivalent cleanup stage once unnecessary machinery becomes the architecture. The same search behavior that expanded a design helped expose a test that was green for the wrong reason.

Sol showed why this is not a medium-versus-high story

The Sol results moved in a different direction. Sol high received higher blind averages than Sol medium in all four phases, although the size and cost of the difference varied.

In design, Sol high explored the existing code deeply, initially considered a path that would add work to tasks without dependencies, and then corrected itself before completing the document. It avoided introducing a new persistent lifecycle state and allowed blocked tasks to recover when dependency state changed. Both evaluators placed it first, with scores of 89 and 93.

The added exploration was useful there. It was also useful in implementation, where the average moved from 66 for medium to 80.5 for high and the resulting implementation covered more meaningful behavior.

In design review and code review, substantially higher recorded token use accompanied much smaller differences in the measured result. Sol high used 3.24 million tokens in design review compared with medium's 1.13 million, while their averages differed by three points. In code review, high used 4.90 million tokens compared with 3.18 million, with a 2.5-point difference.

The important observation is not that one effort level beat another. Effort expanded the search frontier in both families. Opus high carried more of that frontier into the final artifact; Sol high more often produced a smaller final result from the broader search. The usefulness of the remaining exploration then depended on whether the phase needed selection, implementation coverage, or review recall.

The implementations separated at the edges

All seven implementations handled much of the ordinary dependency flow. They could add the YAML field, wait for unfinished tasks, recognize successful dependencies, display reasons, and pass broad test suites. The models separated where the new feature met existing behavior:

What happens when only part of the task store can be read?
Does an alternate or relative path use the same interpretation as the daemon?
Can malformed input change behavior in an existing command?
Does a feature that is not being used add a full-store scan to an old path?
Is a cycle result stable, or does it depend on evaluation order?
Does a green integration test observe the ordering guarantee, or merely execute two tasks that would have succeeded anyway?

Opus medium produced the implementation with the fewest observed gaps. It connected more of these boundaries without changing unrelated behavior. Luna max implemented the ordinary feature at a much lower estimated API cost, but replaced broader search and display paths, changed the treatment of damaged tasks, and added work even when dependencies were unused. Sol high kept the implementation comparatively small but missed alternate-path behavior and direct daemon-level proof.

These differences are hard to compress into a pass rate. The feature works until the repository is incomplete, an older path reads the same YAML differently, or the test suite gives confidence without observing the protected behavior. That is not benchmark trivia. It is where software maintenance happens.

Resources describe activity, not quality

Across all 28 runs, the logs recorded 167.52 million total tokens. Most were cached input repeatedly read across tool turns, not fresh input or hidden reasoning tokens. The summed active execution time was just under eight hours, excluding human approval waits but including tests and builds; the configurations ran in parallel, so this is not the experiment's wall-clock duration. Applying the standard API prices published on August 3, 2026 produced an estimated cost of $104.58, not an actual CLI subscription charge.

Those resource figures describe activity, not quality. Luna max processed the most tokens in design review and remained inexpensive because cached tokens were cheap, but it was also the slowest configuration in that phase. Token volume, price, exploration breadth, latency, and artifact quality were separate variables.

Effort belongs to the phase

I no longer choose one effort level for a model and carry it through an entire workflow. The useful exploration budget depends on what the phase must produce and whether another stage can remove excess work.

Phase	What additional exploration must earn
Design	A smaller justified decision after inspecting the real constraints
Implementation	Coverage of another real contract or execution path without changing unrelated behavior
Review	A supported candidate finding that can still be merged, ranked, or declined downstream

This is related to a conclusion I reached while simplifying an older agent workflow: be strict about boundaries and evidence, but flexible about the route between them. More capable models made detailed route instructions age badly. These measurements add a second problem. Giving the model more room to search is not sufficient when the phase also requires aggressive subtraction.

The operational question is therefore not “How important is this task?” Important work does not automatically deserve high effort. The better question is “What will happen to the additional possibilities this configuration finds?”

If the answer is that they become architecture immediately, uncontrolled breadth is expensive. If they remain review candidates that another step can verify and consolidate, higher recall may be worth buying.

What I can and cannot conclude

This comparison used one run per configuration per phase. Another task or another run could change close measurements. Claude and Codex sometimes disagreed by more than twenty points, and the overlap between evaluator families and candidate families leaves room for style preference. The design scoring prompt omitted one original claim constraint. The tools and caching behavior also differed between providers, so raw token totals are useful descriptions of these sessions but imperfect provider-level comparisons.

The experiment did not isolate an internal mechanism inside Opus or Sol, and I cannot show that a particular latent “convergence ability” caused the outputs. What I can show is the external pattern in this setup: higher effort expanded the work explored; the configurations differed in what they retained; and the value of that retained work changed across design, implementation, and review.

I also cannot conclude that medium effort is generally better, or that high effort causes overengineering. Sol contradicts both claims. An explicit convergence rule might change the outputs, but testing that would measure the models under a new intervention. Here, I intentionally left engineering selection to the models because that judgment is part of how I use them in real development.

The result is strong enough to change how I assign effort without pretending to be a universal model ranking.

Exploration is not the finished product

Recent coding models are much better at searching a repository, following effects across components, and generating technically plausible responses to what they find. That creates a new failure mode: a model can discover more work than it can economically judge.

Reasoning effort pushes on that boundary. Sometimes it exposes the missing execution path that makes an implementation safe. Sometimes it produces the review finding nobody else could prove. Sometimes it turns a small design into a system for managing the risks introduced by the design itself. The setting does not know which one is happening.

Calling it “high” invites the wrong intuition. High effort is not a higher position on a quality scale. It is a larger search budget handed to a particular model inside a particular phase.

The practical limit of a coding agent may no longer be how much it can explore. It may be how much of that exploration it can turn back into one coherent change.

Complete phase scores

These are the simple averages of the two blind evaluations. They summarize the recorded critiques; they do not override evaluator disagreement or the artifact-level evidence discussed above.

Configuration	Design	Design review	Implementation	Code review
GPT-5.6 Sol medium	82.0	80.5	66.0	86.5
GPT-5.6 Sol high	91.0	83.5	80.5	89.0
GPT-5.6 Terra xhigh	71.5	68.0	66.0	74.0
GPT-5.6 Luna max	72.5	84.5	79.0	61.5
Claude Sonnet 5 high	70.5	71.0	78.5	68.0
Claude Opus 5 medium	83.0	76.5	94.5	89.5
Claude Opus 5 high	59.0	72.5	91.5	92.0

Methodology notes

Human approval wait time was excluded from active elapsed time. Test and build execution time was included.
Token totals include normal input, cached input, cache writes where applicable, and output. Cached input accounted for most of the total.
Cost estimates apply the standard API rates available on August 3, 2026 to recorded usage. They are not actual CLI subscription charges.
Artifacts were anonymized before evaluation. Scores are averages of one Claude evaluation and one Codex evaluation; original disagreements were preserved rather than adjudicated into new scores.
The candidate and evaluator families overlap, so the blind process removes model-name knowledge but not possible preference for a family-specific style.
The comparison used one run per configuration per phase and should be read as an instrumented case study rather than a population estimate.

When Better Models Make Old Agent Workflows Worse

Shinsuke KAGAWA — Sun, 02 Aug 2026 14:11:20 +0000

I recently watched a coding agent refuse to start an approved implementation.

The work plan contained this:

## WorkPlan Review

- **Status**: approved (2026-08-02)

The agent found the line and understood the word approved. Then it asked for approval again because the workflow expected a field named Implementation Approval.Status.

Nothing was missing in substance. The label was wrong.

The agent was following a repository-resident workflow I had designed to carry a change through requirements, design, review, planning, and implementation. I had spent months adding status fields, handoff contracts, review gates, retry rules, and escalation paths. Each addition addressed a failure I had actually seen. Together, they had made the workflow brittle enough to stop on a heading.

I had built the system around an old question: how do I stop the model from losing the path?

The model could now follow the path. My path had become the problem.

Two kinds of constraint

The approval incident exposed a distinction I had missed.

A boundary constraint reduces the solution space:

Preserve the public API contract.
Do not perform an irreversible external action without authority.
Implement the confirmed requirements and keep recorded non-goals out of scope.
Treat the task as complete only when the required behavior is observable.

A work-generating constraint creates obligations:

Always produce three alternatives.
Require unit, integration, and end-to-end tests for every feature.
Require a mitigation task, owner, rollback plan, and approval record for every risk.
Resolve every review finding before continuing.
Stop when an optional field is absent or uses an unexpected label.

Both can look like safeguards. They behave very differently once the model is capable of carrying them out.

Boundary constraints tell the agent where it may operate. Work-generating constraints are read as requested work. A reliable agent will reliably create the extra artifacts, tests, abstractions, and stop conditions they imply.

That gave me the principle I now use when changing an agent workflow:

Be strict about boundaries and evidence. Be flexible about the path between them.

Why the problem surfaced now

The capability shift behind this is real, although public measurement often trails the models people are using.

METR's task-completion time horizon shows a strong historical increase in the difficulty of software tasks that agents can complete. METR also warns that the benchmark covers well-specified tasks, that capability remains jagged, and that new releases may go unmeasured for weeks or be skipped entirely. As of August 2, its public page was still dated May 8, and even its list of recent unmeasured models had not caught up with later releases.

Actual usage has moved toward longer execution as well. An analysis of roughly 400,000 coding-agent sessions found a recognizable division of labor: people made most planning decisions, while agents made most execution decisions. The authors describe it as people deciding what to build and the agent deciding how to build it.

Neither source proves how today's models behave in my workflow. The model families involved in the failures described here were released after the measurements and benchmarks I found. I did not find a public rerun on those releases.

That timing matters most for FixedBench. In May 2026, the researchers tested five then-current coding models on 200 issues where the code had already been fixed. The agents still made undesirable changes in 35% to 65% of cases. It is useful evidence that action bias existed in that generation. It is not a current failure rate.

The more interesting result is what happened when the researchers changed the instruction. Telling the agents to verify first and treat abstention as success reduced edits to already-correct code. On partially fixed code, the same instruction made them abstain when more work was still required. Prompt engineering traded action bias for passivity.

That is exactly the trap I had created in my own workflow. A model failed once, so I encoded the opposite behavior as a universal rule. When the model or task changed, the compensation remained.

The strongest evidence here comes from the failure history of this workflow on the current models I use. The studies explain why those failures are plausible and why simple prompt fixes decay. They do not substitute for observing the system itself.

How reliability started manufacturing work

The strict approval field was only the most visible failure.

In another run, the workflow stopped before implementation because it could not find external accounting evidence, document approval records, an exact release command, and detailed E2E environment assumptions. The repository, local services, and test tools were available. Most of the implementation could have proceeded. The workflow had converted useful planning context into mandatory readiness gates, then treated every missing gate as a user-owned decision.

The documents had expanded too. Product requirements and design docs started collecting stakeholder approval, live-service access, release procedures, dashboards, and operational evidence. A work plan for a data-export change included account setup and production concerns even though the requested outcome ended at implementation.

Review made the expansion worse. During that same planning flow, a reviewer requested a deterministic 20,000-row proof. The planner accepted the finding without asking what decision the proof would change or whether a cheaper boundary check would be enough. The reviewer then found that the proposed proof data contradicted an existing aggregation contract, which triggered another revision. Every step was technically defensible. The loop itself had no economic judgment.

I had made the reviewer authoritative and the author obedient. “Review until approved” had become “add work until the reviewer runs out of ideas.”

That is not convergence. It is a ratchet.

What I changed

I did not solve this by deleting every rule. I changed where the workflow is strict.

Decide the outcome before designing the solution

A user request is no longer accepted as an automatically valid implementation scope. Before design, the workflow records:

the observable outcome;
what is required now;
what describes the current state, a desired future, or speculation;
explicit non-goals;
a rough view of implementation cost and structural impact.

The cost assessment stays rough because this is requirements work, not detailed estimation. Its job is to make a bad tradeoff visible while scope is still cheap to remove.

“No change” and “reuse what already exists” are valid conclusions, but neither is a default. The agent must inspect enough evidence to decide whether the outcome is already satisfied, partially satisfied, or still requires implementation. That avoids turning FixedBench's action bias into its mirror image.

This also changes what users need to communicate. A useful request contains the boundary of the current outcome, not just a list of desired features. “Extend the existing authentication path without changing the public response contract” is more valuable than a long list of implementation rules. It states both the result and the point at which design needs to be reconsidered.

Give the model decisions, not a predicted route

The old workflow described complete routes: run one agent, populate every field, run another agent, stop on any missing input, repeat until a serialized state appears.

I now try to give each phase four things:

the purpose it owns;
the evidence that can change its decision;
the criteria for choosing the next action;
the smallest result the next consumer needs.

“Run every test lane” predicts a route. “Use the narrowest test that observes the required boundary” provides a decision criterion. Exact schemas still matter where software parses the response. A human-readable work plan does not need to fail because two headings express the same approved state. I changed the approval check to accept semantically equivalent evidence instead of requiring one heading.

The main agent also owns lightweight resolution. It can interpret semantically equivalent states, resolve repository-local ambiguity, retry with new evidence, and continue unaffected work. User escalation is reserved for decisions that belong to the user: a changed product outcome, a new requirement, a major approved design change, unavailable authority, or an irreversible external action.

Let review findings be declined

The receiver of a review now has three choices:

Apply a finding that contradicts an approved requirement, accepted design, repository rule, or observable correctness.
Decline a finding that adds scope, reverses an exclusion, duplicates proof, requests optional hardening, or costs more than its observable effect justifies.
Return for user decision when resolution changes the product outcome or a major approved decision.

A decline includes evidence and goes back to the reviewer. The reviewer may maintain the finding when that evidence still leaves the result incorrect or unverifiable. Repeating the same preference without new evidence does not block the workflow.

A useful review process preserves a basic engineering skill: deciding not to implement a technically reasonable suggestion.

The constraint I had to put back

Removing rules can be just as careless as adding them. I proved that during this rewrite.

The planning flow generated integration and E2E test skeletons so the first vertical slice could prove an acceptance criterion across a real boundary as early as possible.

After I simplified the handoff, the planner treated the generated test skeleton as an unrelated file and planned a new E2E test later in the project. The artifact still existed, so the workflow looked complete. Its reason for existing had disappeared.

I restored the skeleton's purpose in the planner: consume the exact generated file in the earliest task that can make its boundary executable. Shared infrastructure may come first only when no acceptance criterion can work without it, and only the infrastructure needed by that first slice belongs there.

This is a boundary constraint worth keeping. It connects a planned implementation to observable proof and prevents integration risk from being postponed until every component has been built. Removing it made the workflow smaller and less reliable.

What remains strict

Several controls still earn their cost:

User approval for product requirements and major design decisions.
Explicit authority and mechanical containment for irreversible actions.
Exact schemas at genuinely machine-consumed boundaries.
Plans for work with real dependencies.
Early vertical proof of an accepted outcome.
Observable verification before completion.
Independent review where a separate perspective can catch false confidence.

For dangerous operations, sandboxes, restricted credentials, isolated filesystems, and network controls are stronger than long prompt instructions. They limit the blast radius even when judgment is imperfect. Repeated approval prompts are a weaker substitute: one implementation report describes high approval rates and declining attention, and responds by using automated judgment to reduce routine prompts.

The same division applies to the workflow as a whole. Enforce the durable boundary mechanically where possible. Let the model choose among reversible actions inside it. An account of an agent-first engineering environment uses a similar principle: treat the top-level instructions as a table of contents rather than an encyclopedia.

The four questions I now ask

I now use four questions when reviewing an existing rule:

What changes if I remove it? Name the decision, irreversible boundary, downstream consumer, or observable failure it affects.
Does it limit scope or create work? Keep durable boundaries. Require current evidence before generating artifacts, test lanes, approval records, retry loops, or operational tasks.
Can the model choose from evidence? Replace exhaustive branches with a purpose and selection criteria when the choice is reversible.
Does it work in a fresh session? Rerun a representative task from the beginning. Continuing the old conversation only proves that a human can steer around the failure.

The fresh-session test has to cover both sides. A small change should remain small, ordinary ambiguity should not return to the user, and optional review findings should not expand scope. Required tests, approvals, and quality checks must still run. Fewer rules are not evidence of a better workflow.

A workflow can become technical debt

The hardest lesson for me was that reliability work can itself become overengineering.

Every workflow rule depends on an assumption about model behavior. A rule added after yesterday's failure can become redundant when the model improves, or harmful when a different task needs the opposite judgment. Status fields and review gates add coupling just as surely as public interfaces and shared abstractions do.

Boundary constraints age better because they describe what must remain true: the approved outcome, the public contract, the authority available, and the evidence required for completion. The route between those boundaries can change with the model.

I no longer ask only whether a new rule would have prevented the last failure. I also ask what work it will create, what decision it will remove from the model, and whether it will still make sense after the failure mode disappears.

The job of an agent workflow is no longer to make a model capable of finishing the process. It is to help a capable model finish the right amount of work.

A second pair of eyes for Claude Code: building Galley, a local runner that checks the work before the PR opens

Shinsuke KAGAWA — Mon, 11 May 2026 12:16:11 +0000

Update: claude -p pricing changed after this was published, which made running the cross-model loop continuously more expensive than I wanted to defend. Galley's executor is now selectable between Claude Code and Codex (it was Claude-only before), so a same-model Codex → Codex setup is a viable cost-optimized option alongside the original Claude → Codex pairing. See "Where it is" for the current pairing recommendations.

TL;DR

I run three client projects plus an OSS repo. Agentic coding got good enough that for a lot of tasks I just hand over a goal and a list of acceptance criteria and let it run. The catch: Opus 4.7's reliability made me nervous enough that I started ending almost every task with a manual round of "now have Codex look at this."
That ritual happened often enough that I automated it. Galley is the result: a local runtime where Claude Code executes a task inside a git worktree, then a supervisor (Claude, or Codex when I want a different reviewer) checks the run evidence against the acceptance criteria and either bounces it back for another attempt or approves it and opens a PR.
The parts I'm happiest with aren't the loop itself. They're the boring scaffolding around it: the tool is installed and configured by a Skill, acceptance-criteria test skeletons get written into the worktree before the first attempt, there's a per-repo quality.yaml I keep growing, and a second model on review duty catches things the model that wrote the code does not.
Galley now does a noticeable chunk of its own development. It's MIT-licensed and available on GitHub.

The setup that made me build this

For the last while my day job has been three concurrent product codebases plus maintaining an open-source workflow framework. None of them are huge, but the context-switching tax is real, and I'd been leaning harder and harder on Claude Code to carry whole tasks rather than babysitting them line by line.

At some point a threshold got crossed. Not "AI writes all my code now," more like: for a task of a certain size and shape, I no longer needed to plan the implementation. I needed to write down what done looks like (the goal, the acceptance criteria, the paths it's allowed to touch), and that was genuinely enough. The model would go figure out the how. That's a nice feeling the first few times it works.

Then Opus 4.7 happened, and the feeling got complicated. I won't relitigate it; plenty of people have. The short version for me was: the ambition was still there, the output still looked plausible, but I stopped trusting "looks plausible." So I started doing something I'd done occasionally before, but now every single time: after Claude Code finished, I'd open Codex, hand it the diff and the requirements, and ask it to find what was wrong. It usually found something. Different model, different blind spots: Codex would flag an edge case Claude glossed over, and Claude would have written cleaner structure than Codex would have. The combination of the two was reliably better than either alone.

So now every task had a manual final step that I did by hand, with copy-paste, in a separate terminal, every time.

"Third time, automate it"

I have a rule I mostly stick to: the first time you do a thing manually, fine. The second time, grumble. The third time, you build the tool. The cross-model review had blown well past three.

But the more I sketched it, the more it stopped being "a script that pipes a diff into Codex" and turned into something with a shape: if a supervisor model is going to approve work, it needs the work in a reviewable form, not just a diff but the command plan, the executor's own report, git status, the structured result. If it's going to reject work, the rejection needs to come back to the executor as a new attempt with the feedback attached, not as a Slack message to me. And if it can do all that, it can open the PR itself, and I can do final tweaks from PR comments instead of from my editor.

What Galley actually does

It's a local Go binary plus a daemon. You point it at a repo, hand it a task, and it runs:

your repo  ──task YAML──▶  galley daemon (local)
                               │
                               ├─▶ executor: Claude Code or Codex, inside a git worktree
                               │      writes the code, returns a structured result
                               │
                               └─▶ supervisor: Claude or Codex
                                      reads the run evidence, issues a verdict,
                                      opens the PR on accept

The task itself moves through a file-backed queue, and the supervisor's verdict decides where it lands:

draft task YAML
        |  galley task queue
        v
   tasks/queued/  →  daemon claims it  →  tasks/running/
        |
        |  Claude Code executes in a git worktree
        v
   supervisor review (Claude or Codex)
        |
        +-- accepted  -------------------→ tasks/done/  (+ open PR if enabled)
        +-- needs_revision  -------------→ retry, while loop budget remains
        +-- needs_supervisor_review  ----→ tasks/failed/  (escalate to me)
        +-- hard_stop  ------------------→ tasks/failed/  (no retry)

Everything runs locally, every change stays as git-visible diffs, and every attempt writes its evidence to disk: command_plan.json, run_result.json, the supervisor's verdict, git_status.json, diff.patch. When the loop escalates to me, I'm not guessing; I'm reading the file the supervisor read.

The task file is the trusted input. It's where the goal, the acceptance criteria (each with an ID the executor has to report back against), the allowed and forbidden paths, the loop budget, and the PR behavior live. The model never gets to redefine its own success criteria mid-run. It gets to satisfy them or fail them.

The decisions I'd actually defend

The loop is the obvious part. Here's the stuff that took longer to get right and that I think matters more.

A Skill installs the tool

This is the bit that still feels a little science-fictional to me. Galley ships an Agent Skill: a Claude Code plugin, and a Codex marketplace entry. You install the skill first. Then you ask it, in plain language, to set up the repo. It installs the galley binary, inspects the repository, drafts a quality.yaml and an environment.yaml, explains the execution settings to you, writes a valid task YAML, validates it, and queues it. It only queues after you say yes.

I've shipped CLIs before. The onboarding was always a README and a prayer. Here the onboarding is an agent, and "explain what this config field means and pick a sensible default for my repo" is just a thing it does. That splits your docs in two: the README is for the human who wants to understand the system, and the skill's reference files are for the agent that has to operate it correctly without you in the loop. They overlap less than you'd think, and I'm still figuring out where each line belongs.

Acceptance-criteria test skeletons go in first

This one's a direct response to the 4.7 trust problem. There's an optional preflight step. Before the first executor attempt, Galley runs a built-in test-creator pass that writes test skeletons into the worktree, one per acceptance criterion, and records the mapping back onto the running task: for each AC, the skeleton's path, the behavior it's meant to pin down, and where it plugs into the codebase. In a Go repo a skeleton is about what you'd expect:

func TestAC1_RunAgentOverridesTimeoutPerCall(t *testing.T) {
    t.Skip("AC1: run_agent callers can override execution timeout per call")
    // executor fills this in
}

Those skeleton paths are validated against the task's allowed paths, so the test-creator can't scatter files wherever it likes. And the executor can't get an "accepted" verdict while those tests are still skipped and the required checks haven't run green; the supervisor downgrades that to needs_supervisor_review. The effect is small but real: the implementation has to converge on something the AC-shaped tests accept. A model that's drifting toward a clever-but-wrong solution runs into the skeleton and has to reckon with it. It's harder to wander off when there's already a fence where the spec said the fence should be. (It's off by default, since some tasks genuinely shouldn't have it, but for "implement feature X with these three behaviors," I turn it on.)

`quality.yaml` is a thing I grow

Each repo gets a quality profile: which checks are required, which review dimensions must pass, what evidence the supervisor should expect, what severity of finding blocks acceptance. It starts small. Then every time a run produces something technically-passing-but-wrong-for-this-codebase, I add a line. Over time the profile becomes the codebase's accumulated opinion about what "good" means here, and both the executor and the supervisor get handed that opinion at the start of every task. Implementations stop drifting because the definition of "done well" stopped being implicit.

Claude writes, a second model signs off

Supervisor review defaults to Claude. But I can flip it to Codex per task, and for anything I'd have manually double-checked before, I do. Same-model review (Claude checking Claude) is fine and catches plenty. A different model catches a different category of mistake, and because a rejection comes back as another attempt with the feedback attached, the executor gets to fix what the reviewer flagged instead of just failing the task. So a long unattended run doesn't drift the way an unreviewed one does: every accepted step has had a second model poke at it, and the diff you end up with has those corrections baked in.

The deterministic / non-deterministic seam

Building this kind of tool, the genuinely fiddly part isn't the AI calls — it's the boundary between the parts that must be exact and the parts that get to improvise. Two places I had to draw a hard line:

The executor has to return a structured JSON result, every time, or the retry-and-review loop has nothing to stand on; the supervisor can't evaluate a free-form essay. So Galley installs a small guard plugin into the executor's Claude Code that enforces the output format. The creative work is non-deterministic; the envelope it arrives in is not.
Opus 4.7's defaults made me uneasy enough that I replaced the executor's system prompt outright with one derived from Codex-style prompting and from my own claude-code-workflows OSS, and did the same kind of swap for the supervisor prompts. The task YAML literally has prompt_mode: replace for this. I'd rather pin the behavior than hope for it.

Neither of these is glamorous. Both are the difference between a demo and something I leave running while I'm in a meeting.

Galley building Galley

The thing I didn't plan for: once it worked, the obvious next move was to have Galley develop Galley.

A few recent fixes, all queued through the skill and executed by Galley itself. The PR body was rendering every acceptance criterion as not_satisfied even when the supervisor had accepted them with evidence, which is confusing for anyone reading the PR. galley task show, the command I run constantly, was loudly reporting latest_claude_status: failed on tasks that had actually been accepted with a PR open; true as raw history, wrong as the headline. The PR-comment trigger only recognized /galley rerun ... and /galley requeue ..., when what I actually wanted was to type /galley fix the failing test and have it pick that up as the request. And new worktrees were being branched off whatever the source repo's HEAD happened to be instead of the configured base branch, so one PR's commits could leak into the next.

Some of those I caught by using the thing. Some — like the test-skeleton preflight — came from sitting down with a pile of runs/ evidence and asking what would have stopped the bad run earlier. Either way, the loop closes: I notice a gap, I write it up as a task with acceptance criteria, Galley implements it, a supervisor checks it, a PR shows up, I tweak it from a comment.

It's not fully autonomous and I'm not pretending it is. I review the task drafts. I read the escalations. I still tweak PRs. But the ratio of "me describing what I want" to "me typing the code" has tilted further than I expected, and the safety rails (evidence on disk, AC-shaped tests, a growing quality profile, a second model's sign-off) are what let me actually trust the tilt.

Where it is

Galley is MIT-licensed and available on GitHub at shinpr/galley. Both the executor and the supervisor are selectable between Claude Code and Codex. If you want a different model checking the work — the original premise of this post — Claude → Codex is still what I'd reach for; a different model's blind spots are the whole point. Codex → Codex is the cost-optimized alternative: same model on both sides, but the supervisor only ever sees ACs and evidence files, not the executor's conversation, which mitigates self-preference bias even without going cross-model. It's built for trusted local repositories: task YAML is trusted input, quality checks run locally, PR comments can request a requeue but can't rewrite your gates, and only the PR author (who is also a repo owner or collaborator) can drive it from comments. It also leans on git, a git worktree per task, and gh for the PR path.

To try it, add the plugin and let the Galley skill do the setup: it installs the galley CLI if it isn't already on your PATH, inspects the repo, drafts the quality.yaml and environment.yaml profiles and the task YAML, and queues only after you approve. That "install a skill, have it install the tool" loop is the part that still feels new to me.

Claude Code

/plugin marketplace add shinpr/galley
/plugin install galley@galley-tools
/reload-plugins
/galley:galley Set up Galley for this repository.

Codex

codex plugin marketplace add shinpr/galley

That only adds the Galley marketplace. Next, start a Codex session, run /plugins, select Galley, and install it from there.

Then invoke the skill with $galley:

$galley Set up Galley for this repository.

If you'd rather install the CLI yourself first, it's a one-liner: curl -fsSL https://raw.githubusercontent.com/shinpr/galley/main/scripts/install.sh | sh. Either way you end up describing tasks to the skill in plain language from there.

I'm curious how other people have dealt with the same trust gap. Have you put a second model in the review loop? Did a cross-model pairing actually buy you something, or was Claude-reviewing-Claude enough? And if you've found a better answer than "evidence on disk plus tests that go in before the code," I'd genuinely like to hear it. If you build something around this, or break it in an interesting way, even better.

I Built a Skill Reviewer. Then I Ran It on Itself.

Shinsuke KAGAWA — Thu, 02 Apr 2026 11:44:56 +0000

I built a tool that reviews Claude Code skills for quality issues.

Then I pointed it at its own source files. It found real problems.

The irony wasn't lost on me. But the more interesting question is: why did this happen, and what does it tell us about how LLM-based quality tools actually work?

The Setup

I maintain rashomon, a Claude Code plugin for prompt and skill optimization. It includes a skill reviewer agent that evaluates skill files against 8 research-backed patterns (BP-001 through BP-008) and 9 editing principles.

One of those patterns—BP-001—says: don't write instructions in negative form. Research shows LLMs often fail to follow "don't do X" instructions—negated prompts actually cause inverse scaling, where larger models perform worse. The fix is to rewrite them positively: instead of "don't skip P1 issues," write "evaluate all P1 issues in every review mode."

Simple enough.

Except both my agent definition files had a section called ## Prohibited Actions full of "don't" instructions.

The Discovery

I noticed this by reading my own code. But I wanted to see what happens when the tools catch it—or don't.

First, I ran the prompt-analyzer agent against both files. It analyzed them, found some issues, but gave the Prohibited Actions sections a pass. Its reasoning: these qualify as "safety-critical" exceptions to BP-001, since they constrain "destructive" behaviors.

That felt off. "Don't invent issues not supported by BP patterns" isn't a safety-critical instruction. It's a quality policy. The caller can override or discard the output.

So I ran the skill-reviewer agent against the same two files. The results were more interesting.

For skill-reviewer.md (reviewing itself), it flagged all four items in Prohibited Actions as BP-001 violations—P2 severity. Correct call.

For skill-creator.md (reviewing the other agent), it gave Prohibited Actions a pass. Same structure, same pattern, opposite judgment.

The same reviewer, applying the same criteria, reached opposite conclusions on the same construct.

Digging Into Logs

I could have speculated about why. Instead, I checked the subagent conversation logs.

The skill-creator review log showed this in the Step 1 pattern scan:

BP-001 (Negative Instructions): Lines 197-202 "Prohibited Actions" section uses negative form. However, per the BP-001 exception in skills.md, these are procedural/irreversible consequences (inventing knowledge, removing examples, overwriting files). The exception applies.

It did scan for BP-001. It found the section. But it classified the items as "irreversible consequences" and applied the exception.

The problem was clear: the exception rule said negative form is okay for "safety-critical operations, destructive actions, or order-dependent procedures." That's vague enough to stretch. "Inventing domain knowledge" sounds serious. "Removing user-provided examples" sounds destructive. If you squint, anything can be "destructive."

Nothing was wrong with the reviewer. It was doing exactly what I told it to do. That was the problem.

Fixing the Criteria, Not the Reviewer

The instinct is to blame the LLM: "it self-justified," "it was biased toward leniency." But the actual cause was simpler: the exception rule was written in a way that allowed two reasonable readings.

The fix wasn't to make the reviewer "smarter." It was to make the criteria harder to misread.

I replaced the broad exception language:

Exception: safety-critical operations, exact command sequences,
destructive actions, or order-dependent procedures

With a 4-condition checklist:

Exception: Negative form is permitted only when ALL are true:
(1) Violation destroys state in a single step
(2) Caller or subsequent steps cannot normally recover
(3) The constraint is operational/procedural, not a quality policy
(4) Positive rewording would expand or blur the target scope

And added concrete boundary examples—what qualifies, what doesn't:

Permitted (exception applies)	Not permitted (rewrite positively)
"Do not modify the command"	"Do not invent issues" -> "Base every issue on BP patterns"
"Do not add flags"	"Do not skip P1 issues" -> "Evaluate all P1 in every mode"
"Do not execute destructive operations"	"Do not create overlapping skills" -> "Verify no overlap before generating"

The key addition: "Outputs that the caller validates, overwrites, or discards are never irreversible." This one sentence eliminates most of the ambiguity. A subagent's output goes to a caller. The caller decides what to do with it. That's not irreversible.

The Retest

After updating the criteria, I ran the skill-reviewer again on both files.

skill-reviewer.md: Prohibited Actions flagged as BP-001 P2. All four items caught.

skill-creator.md: Two items flagged as quality policies that should be positive form. The remaining items—which are genuinely about operational constraints—were accepted.

Consistent. Explainable. And the reviewer could now articulate why each item was or wasn't an exception, because the criteria forced it to check specific conditions rather than make a gestalt judgment.

But I wasn't fully satisfied. In a further round of testing, the reviewer still occasionally applied exceptions loosely—recording "irreversible" in the justification field without explaining how it's irreversible.

So I added structured evidence to the output schema:

"patternExceptions": [{
  "pattern": "BP-001",
  "location": "section heading",
  "original": "quoted text",
  "conditions": {
    "singleStepDestruction": "true|false + evidence",
    "callerCannotRecover": "true|false + evidence",
    "operationalNotPolicy": "true|false + evidence",
    "positiveFormBlursScope": "true|false + evidence"
  }
}]

You can't just write "irreversible" anymore. You have to answer four yes/no questions with evidence. If any answer is no, it's not an exception.

What This Comes Down To

The criteria had a loophole wide enough to drive a truck through. Better criteria produced better reviews without changing the reviewer at all. The LLM wasn't "inconsistent"—the instructions were ambiguous. Two reasonable people could have read the old exception rule and reached different conclusions too.

Structured output helped more than I expected. The 4-condition checklist wasn't just about auditability—it changed how the reviewer thinks. When you have to fill in four fields with evidence, you can't hand-wave. The output structure becomes a thinking scaffold.

And running the tool on its own source files was uncomfortable in a useful way. The temptation is to say "well, I know what I meant." But the tool doesn't know what I meant. It reads what I wrote.

The Broader Problem: Skill Quality Is Hard

If you're building Claude Code skills, custom agents, or any kind of structured LLM instruction set—you've probably experienced this: the instructions work fine in your head, but the LLM does something unexpected. You add more instructions. It gets worse. You simplify. Something else breaks.

The issue is that you can't see your own blind spots. You know what you meant. The LLM reads what you wrote. The gap between intent and text is where bugs live.

This is why I built rashomon. It includes:

Skill review: Evaluate skill files against BP-001~008 patterns and 9 editing principles, with structured quality grades
Golden scenario evaluation: Test whether a skill actually works by comparing execution results with and without the skill, or before and after changes—not just whether it was loaded, but whether it made a measurable difference

The golden scenario part matters. "The skill was loaded" doesn't mean "the skill helped." You need to see the actual output difference to know if your skill is doing anything useful.

Try It

Rashomon is a Claude Code plugin. Install it and point the skill reviewer at your own skills.

# In Claude Code
/plugin marketplace add shinpr/rashomon
/plugin install rashomon@rashomon
# Restart session to activate

It will find problems. I know because it found problems in itself—and it's better for it now.

What's your experience with skill quality? Have you found ways to validate that your instructions actually do what you think they do?

Same Framework, Different Engine: Porting AI Coding Workflows from Claude Code to Codex CLI

Shinsuke KAGAWA — Wed, 18 Mar 2026 11:50:48 +0000

TL;DR

I built a sub-agent workflow framework for Claude Code that solved context exhaustion through specialized agents and structured workflows
For 8 months, Codex CLI had no sub-agents — the framework was Claude Code-only
Codex finally shipped sub-agent support — I expected days of migration, it took an afternoon
What surprised me most: if you design workflows around agent roles and context separation rather than tool-specific features, your investment survives platform shifts

The 8-Month Wait

Back in July 2025, I released the first version of this workflow as a Claude Code boilerplate. By October 2025, it had evolved into a full sub-agent framework — specialized agents for every phase of development, from requirements analysis through TDD implementation through quality gates. The idea was pretty simple: break complex coding tasks into specialized roles (requirement analyzer, technical designer, task executor, quality fixer...), give each agent a fresh context, and orchestrate them through structured handoffs. No single agent ever hits the context ceiling because no single agent tries to do everything.

The problem? Codex CLI had no sub-agent capability. Codex had been around since mid-2025, and I wanted the same workflow there too. So I kept trying to bridge the gap.

First, I built an MCP server in August 2025 that let any MCP-compatible tool — Codex, Cursor, whatever — define and spawn sub-agents through a standard protocol. It worked, but MCP added a layer of indirection that wasn't there in Claude Code's native sub-agents.

Then in December 2025, Codex shipped experimental Agent Skills support. I saw an opening and built sub-agents-skills — cross-LLM sub-agent orchestration packaged as Agent Skills, routing tasks to Codex, Claude Code, Cursor, or Gemini. Closer, but still not native sub-agents.

Through all of this, my main development stayed on Claude Code. The context separation and the small context windows of the time made it the clear choice for serious work. Codex filled a supporting role — I used it for skills refinement and as an objective reviewer on complex implementations, a fresh set of eyes from a different LLM.

I don't use hooks extensively — I prefer keeping tasks small and baking quality gates into the completion criteria themselves. So what I was really waiting for was native sub-agent support in Codex, which would let the full orchestration workflow run without workarounds.

On March 16, 2026, Codex CLI shipped sub-agent support. During pre-release validation, I noticed something encouraging: Codex followed the workflow stopping points more strictly than expected. If the behavior stabilizes, it could be a viable primary development tool, not just a supporting one.

The port took almost no effort.

What "Near-Zero Migration" Actually Looks Like

When I say "the same framework," I mean it. The core architecture didn't change:

User Request
    ↓
requirement-analyzer → scale determination [STOP for confirmation]
    ↓
technical-designer → Design Doc
    ↓
document-reviewer [STOP for approval]
    ↓
work-planner → phased task breakdown [STOP]
    ↓
task-decomposer → atomic task files
    ↓
Per-task 4-step cycle:
  task-executor → escalation check → quality-fixer → git commit

22 sub-agents. 26 skills. The same stopping points, the same quality gates, the same TDD enforcement.

What changed was the container format, not the content:

Aspect	Claude Code	Codex CLI
Agent definitions	Markdown with YAML frontmatter (`agents/*.md`)	TOML files (`.codex/agents/*.toml`)
Skills location	`skills/`	`.agents/skills/`
Tool declarations	Explicit in frontmatter (`tools: Read, Grep, Glob...`)	Not needed (inferred from sandbox mode)
Skill references	Comma-separated names	`[[skills.config]]` arrays
Config directory	`.claude/`	`.codex/`

That's it. The agent instructions — the actual substance of what each agent knows and does — are the same. The workflow logic is the same. The quality criteria are the same.

Why This Worked: Design Decisions That Paid Off

It worked for a surprisingly simple reason — three choices I made early on:

1. Natural Language as the Interface Layer

Every sub-agent's behavior is defined in natural language instructions, not in platform-specific tool calls. The requirement-analyzer isn't wired to Claude Code's Agent tool or Codex's spawn_agent — it follows a written protocol: "Extract task type, determine scale (1-2 files = Small, 3-5 = Medium, 6+ = Large), identify ADR necessity, output structured JSON."

This means the instructions work on any LLM-powered agent system that can read text and follow procedures. In practice, that turned out to be enough. The framework is fundamentally a set of well-written job descriptions, not a set of API integrations.

2. Context Separation as Architecture

The core insight from the original article still applies: each agent runs in a fresh context without inheriting bias from previous steps. The document-reviewer doesn't know what the technical-designer was "thinking" — it just reviews the output. The investigator explores without confirmation bias from whoever reported the bug.

This isn't a Claude Code feature or a Codex feature. It's an architectural pattern that happens to be implementable on both platforms once they support sub-agents.

3. Structured Handoffs Over Shared State

Agents communicate through artifacts (documents, JSON outputs, task files), not through shared memory or conversation threading. The technical-designer writes a Design Doc. The work-planner reads that Design Doc. Neither needs to know which platform spawned the other.

docs/
├── prd/          # PRD artifacts
├── adr/          # Architecture decision records
├── design/       # Design documents
├── plans/        # Work plans
│   └── tasks/    # Atomic task files (1 commit each)

This file-based protocol turned out to be surprisingly platform-agnostic.

The Framework in Action

codex-workflows ships as an npm package:

npx codex-workflows install

This drops the full agent and skill infrastructure into your project. From there, you invoke recipe skills that orchestrate the sub-agents:

Recipe	What It Does
`$recipe-implement`	Full lifecycle: requirements → design → implementation → quality
`$recipe-design`	Requirements → design documents only
`$recipe-build`	Execute pre-planned tasks autonomously
`$recipe-diagnose`	Problem investigation → root cause → solutions
`$recipe-front-build`	React/TypeScript implementation with frontend-specific agents
`$recipe-fullstack-implement`	Cross-layer features with layer-aware task routing

Each recipe automatically loads the right foundational skills (coding standards, testing principles, documentation criteria) and spawns the right sequence of agents with the right stopping points.

Here's what a real session looks like. I ran $recipe-implement Add a login screen with email/password and a simple auth API in Codex CLI against a fresh project with placeholder files:

1. Requirement analysis — the first mandatory stop:

• Spawned Plato [requirement-analyzer] (gpt-5.4-mini medium)

• Updated requirement analysis is now confirmed:
  - Scope: email/password login form, local mock auth abstraction,
    hardcoded success credentials demo@example.com / password123,
    inline success message only
  - Route: fullstack
  - Scale: medium
  - ADR: not required

  Please confirm this analysis and answer these three questions:
  1. Should the auth API be a real HTTP endpoint, or is a local/mock
     API abstraction enough?
  2. What credentials should succeed initially?
  3. On successful login, should it only show success, navigate
     somewhere, or persist auth state too?

The workflow stops and waits. No implementation starts until I confirm scope.

2. Parallel design — backend and frontend at the same time:

• Spawned Volta [technical-designer] (gpt-5.4-mini medium)
  └ Create a backend Design Doc...

• Spawned Pasteur [technical-designer-frontend] (gpt-5.4-mini medium)
  └ Create a frontend Design Doc...

• Spawned Epicurus [document-reviewer] (gpt-5.4-mini medium)
  └ Review login-backend-design.md...

• Spawned Bohr [document-reviewer] (gpt-5.4-mini medium)
  └ Review login-frontend-design.md...

• Spawned Euclid [design-sync] (gpt-5.4-mini medium)
  └ Verify consistency between Design Docs...

Five agents in parallel: two designers, two reviewers, one cross-layer sync checker. Each running in its own fresh context.

3. Design gate — another mandatory stop:

• The design gate is now clean enough to stop for approval.

  - Backend design: approved
  - Frontend design: approved
  - Cross-layer sync: NO_CONFLICTS

  Aligned contract:
  - Success: { ok: true }
  - Failure: { ok: false, error }
  - Inline success copy is UI-owned, not backend-owned

  Please approve the design docs so I can move to
  acceptance-test generation and the work plan.

4. Autonomous execution after batch approval:

› batch approval

• Spawned Ohm [task-decomposer] (gpt-5.4-mini medium)

• Verification passed:
  - npm test
  - npm run build

After batch approval, the framework decomposed the work plan into tasks and executed them autonomously — no more stopping points until the quality gates pass.

The whole flow from $recipe-implement to green tests took one session. The same flow, the same stopping points, the same agent roles that I've been running on Claude Code for months.

Getting Started

The framework is open source:

Codex CLI version: codex-workflows
Claude Code version: claude-code-workflows

If you're already using the Claude Code version, the Codex version follows the same patterns. If you're new to both, pick whichever CLI you're already using — the workflow knowledge transfers either way.

npx codex-workflows install

The whole port changed config file formats and directory conventions. The agent instructions — the part that actually matters — didn't need a single edit. That's the thing I'd want to know if I were deciding whether to invest time in workflow design for AI coding tools.

If you've been running sub-agent workflows with either Claude Code or Codex CLI, I'd be curious how your setup compares. What worked? What broke?

Letting LLMs Jump — and Then Verifying Ruthlessly

Shinsuke KAGAWA — Thu, 12 Feb 2026 13:35:03 +0000

The "First Plausible Answer" Problem

You've probably seen this: you ask an LLM to investigate a bug, and it latches onto the first plausible explanation. It confidently proposes a fix before thoroughly exploring alternatives. Sometimes it works. Often it doesn't—and you're left debugging the debugger.

I ran into this repeatedly in my personal projects. The LLM would find something that looked like the cause, stop investigating, and immediately suggest a solution. When the codebase was small, this worked fine. As it grew, I started getting fixes that didn't actually fix anything.

This is not for small scripts or simple bugs.

I only started needing this once my codebase grew large enough
that "just try a fix" stopped working.

The root issue? How I was defining the task's purpose.

Planning works well when the problem is understood.

But when the problem itself is unclear,
planning alone is not enough.

This article focuses on those cases.

The Factor That Made the Difference: Purpose

When delegating tasks to LLMs, two factors affect execution accuracy: Context (staying within ~70% of the context window) and Purpose (how you define the task's goal).

Context management matters, but this article focuses on the second factor—because that's where I was getting it wrong.

Where you set the task's goal matters more than you might think. The purpose you define determines the task granularity, and the right granularity depends on your codebase complexity.

A Real Example: Bug Investigation

The Old Approach

A single session handling "Investigation → Solution Proposal → Verification," followed by a separate review session.

What I Changed

My original goal was simple: "propose a solution" and "review it objectively."

Originally, I'd just have the LLM investigate, propose a fix, and implement it directly. But as the codebase grew, I started getting solutions that didn't actually work. So I added a review step—opening a fresh session to check the proposal with clean context.

This worked for about 60-70% of problems, but occasionally even this approach couldn't reach the root cause no matter how many iterations.

Here's what I changed:

Problem Structuring: Structure my instructions upfront to make them easier for LLMs to parse in later steps
Investigation: Conduct comprehensive investigation and report results
Verification: If there's uncertainty in the report, perform additional verification
Solution Derivation: Receive investigation and verification results, then derive solutions

By setting "investigation" as the purpose, the model stopped jumping to the first candidate and instead collected information from multiple angles.

Implementation Example

This setup is probably overkill for small scripts. I only started doing this after my codebase crossed a certain complexity threshold.

Here's how I structured the diagnosis workflow using Claude Code's slash commands and sub-agents. Full implementation is available at github.com/shinpr/claude-code-workflows.

Main Command (diagnose.md)

---
description: "Investigate problem, verify findings, and derive solutions"
---

**Command Context**: Diagnosis flow to identify root cause and present solutions

Target problem: $ARGUMENTS

## Step 0: Problem Structuring (Before investigator invocation)

### 0.1 Problem Type Determination

| Type | Criteria |
|------|----------|
| Change Failure | Indicates some change occurred before the problem appeared |
| New Discovery | No relation to changes is indicated |

### 0.2 Information Supplementation for Change Failures

If the following are unclear, **ask with AskUserQuestion** before proceeding:
- What was changed (cause change)
- What broke (affected area)
- Relationship between both (shared components, etc.)

## Diagnosis Flow Overview

The goal of investigation is not to propose solutions.
It is to eliminate wrong explanations.

**Context Separation**: Pass only structured JSON output to each step.
Each step starts fresh with the JSON data only.

Sub-agent: Investigator

Think of the Investigator as a junior engineer whose only job is to gather facts—not to be clever. Its purpose is explicitly limited to evidence collection only—no solutions:

This is one concrete implementation.
The important part is the separation of purpose—not the specific tooling.

## Output Scope

This agent outputs **evidence matrix and factual observations only**.
Solution derivation is out of scope for this agent.

## Core Responsibilities

1. **Cross-check multiple sources** - Don't rely on a single source
2. **Search external info (WebSearch)** - Official docs, Stack Overflow, GitHub Issues
3. **List hypotheses and trace causes** - Multiple candidates, not just the first one
4. **Identify impact scope** - Where else might this pattern exist?
5. **Disclose blind spots** - Honestly report areas that could not be investigated

Key output structure:

{
  "hypotheses": [
    {
      "id": "H1",
      "description": "Hypothesis description",
      "causeCategory": "typo|logic_error|missing_constraint|design_gap|external_factor",
      "causalChain": ["Phenomenon", "→ Direct cause", "→ Root cause"],
      "supportingEvidence": [...],
      "contradictingEvidence": [...],
      "unexploredAspects": ["Unverified aspects"]
    }
  ],
  "comparisonAnalysis": {
    "normalImplementation": "Path to working implementation",
    "failingImplementation": "Path to problematic implementation",
    "keyDifferences": ["Differences"]
  }
}

Sub-agent: Verifier

The Verifier plays the annoying senior reviewer who assumes everything is wrong. It actively seeks refutation:

## Core Responsibilities

1. **Cross-check multiple sources** - Explore information sources not covered
2. **Generate alternative hypotheses** - What else could explain this?
3. **Play devil's advocate** - Assume "the investigation results are wrong"
4. **Pick the hypothesis with fewest holes** - Not "most evidence," but "least refuted"

Sub-agent: Solver

The Solver is the engineer who actually has to ship something. Only after verification does it derive solutions:

## Output Scope

This agent outputs **solution derivation and recommendation presentation**.
Trust the given conclusion and proceed directly to solution derivation.

## Core Responsibilities

1. **Multiple solution generation** - At least 3 different approaches
2. **Tradeoff analysis** - Cost, risk, impact scope, maintainability
3. **Recommendation selection** - Optimal solution with selection rationale
4. **Implementation steps presentation** - Concrete, actionable steps

Practical Guidelines

When designing LLM tasks, I now check two things:

Purpose Clarity - "Don't create tasks with unclear purposes"
Context Efficiency - Can it be completed in one session with sufficient information? (Ideally using 60-70% of working space)

I don't blindly split tasks into smaller pieces. Instead, I consider ROI and break down from larger tasks only when necessary.

By explicitly separating "investigation" from "solution," you prevent the model from rushing to conclusions before it has gathered sufficient evidence.

A Lesson I Learned the Hard Way

Early on, I made the Verifier run every single time. The problem? Even when the investigation was clearly off track, the Verifier would dutifully try to verify nonsense.

That's when I realized: you need a quality gate between steps, not just separation.

Now I have a checkpoint between Investigation and Verification. If the investigation output doesn't meet basic quality criteria (missing comparison analysis, shallow causal chains, etc.), it loops back instead of wasting cycles on verification.

I also added Step 0 (Problem Structuring) to help the LLM understand my intent better before diving in. These two changes—quality gates and upfront structuring—made the whole pipeline actually usable.

Design Integration Checkpoints Before Letting LLMs Code

Shinsuke KAGAWA — Wed, 04 Feb 2026 13:23:55 +0000

Once you stop trying to control AI generation and start designing verification, you immediately hit the next problem: integration.
And this is where most AI-generated systems actually break.

Everything works.
Until it doesn't.

Each layer looks correct in isolation.
Tests pass.
Types line up.

And then the system breaks where those layers meet.

Why "Everything Works" Until It Doesn't

This is a verification problem, not an implementation problem.
When you build systems layer by layer, integration happens very late.

Layer-by-layer development
Phase 1: Data layer ────────────────✓
Phase 2: Service layer ─────────────✓
Phase 3: API layer ────────────────✓
Phase 4: UI layer ─────────────────✓
Phase 5: Integration ── 💥 breaks here

Each layer is implemented in isolation.
So you don't actually know if everything connects correctly until the end.

This problem becomes much worse with AI-generated code.

LLMs don't hold the entire system in mind at once.
They optimize locally, based on the current context — and they often miss hidden contracts between layers.

A Painful Integration Bug That "Worked"

One of the most painful bugs I faced didn't involve crashes or errors.

The AI chatbot worked.

It returned responses.
Logs looked normal.
Nothing failed.

But when we tested it in the real environment, the answers were subtly — but consistently — wrong.

What actually went wrong

The root cause wasn't a single mistake, but a combination of issues across layers:

Mock implementations silently left in place
LLM fallbacks that prioritized "returning something" instead of failing fast
Duplicate logic across layers, created while implementing each layer separately

The common thread? I wasn't tracking what else might break.

Each layer looked correct in isolation.
Tests passed.
No alerts fired.

Because the system always returned some response, it created a false sense of confidence.
We didn't notice the problem immediately — and by the time we did, identifying the real cause across layers was extremely difficult.

Bugs that silently "work" are far more dangerous than bugs that crash.

Make Integration Explicit

I now spend about five minutes defining integration checkpoints.
Not documentation. Just verification.

The goal is simple: define where things must connect, and how I'll know they actually do.

Now, before implementation, I write a very small design note.

Not a formal design document.
Nothing formal.

Just a checklist that answers two questions:

What parts of the system are affected?
Where do things need to integrate — and how do I verify it?

Step 1: List What's Affected

First, I write down what is directly or indirectly impacted.

Change: Add image generation feature

Direct impact:
  - infrastructure/image/functions.ts
  - application/services/queryClassificationService.ts
  - application/services/imageGenerationService.ts

Indirect impact:
  - conversationService.ts (function calling flow)

No impact:
  - existing text generation services
  - other function handlers

This immediately clarifies the blast radius.

I don't aim for perfection —
I just want to avoid being surprised later.

Step 2: Define Integration Checkpoints

Next, I decide where integration must be verified and how.

Integration point 1: Function selection
Location: ConversationService.generateContentWithFunctionCalling

How to verify:
  1. Send a request asking for an image
  2. Confirm query classification returns `image_generation`
  3. Confirm the correct function is selected in logs

Expected result:
  - Log shows: Executing function: generateImage

And another one:

Integration point 2: Image generation and posting
Location: ImageGenerationService → MessagingClient.uploadFile

How to verify:
  1. Image data is returned from the image client
  2. The file is posted to the chat thread

Expected result:
  - Image appears in the chat

Now I know exactly what "working" means.

That's it.

Why This Works (Especially with AI)

When I give this to an LLM, it changes how implementation happens.

Instead of "build this feature," it's more like:
"Connect A to B. Here's how we'll know it works."

This also pairs well with building features end-to-end:

Feature-based development
Feature A: Data → Service → API → UI → Verify
Feature B: Data → Service → API → UI → Verify
Feature C: Data → Service → API → UI → Verify

Each feature is fully integrated before moving on.

The Result

Before this habit, integration bugs often cost me hours.

After introducing these small design notes:

AI-generated code still has small issues
But features no longer completely break at integration
Unexpected behavior is caught much earlier

Five minutes of thinking up front easily saves hours of debugging later.

Who This Is For

This approach works well if you:

Use AI coding tools
Build layered architectures
Want fast feedback instead of perfect design docs

This is not about writing more documentation.
It's just about making integration explicit before code is written.

Final Thoughts

AI tools are incredibly powerful — but they optimize locally.

If we don't define integration points explicitly, we end up debugging systems that look correct but behave incorrectly.

A small design checklist has made a huge difference for me.

Hope this saves you some painful debugging.

Planning Is the Real Superpower of Agentic Coding

Shinsuke KAGAWA — Mon, 26 Jan 2026 12:24:31 +0000

I see this pattern constantly: someone gives an LLM a task, it starts executing immediately, and halfway through you realize it's building the wrong thing. Or it gets stuck in a loop. Or it produces something that technically works but doesn't fit the existing codebase at all.

The instinct is to write better prompts. More detail. More constraints. More examples.

The actual fix is simpler: make it plan before it executes.

Research shows that separating planning from execution dramatically improves task success rates—by as much as 33% in complex scenarios.

In earlier articles, I wrote about why LLMs struggle with first attempts and why overloading AGENTS.md is often a symptom of that misunderstanding. This article focuses on what actually fixes that.

Why "Just Execute" Fails

This took me longer to figure out than I'd like to admit. When you ask an LLM to directly implement something, you're asking it to:

Understand the requirements
Analyze the existing codebase
Design an approach
Evaluate trade-offs
Decompose into steps
Execute each step
Verify results

All in one shot. With one context. Using the same cognitive load throughout.

Even powerful LLMs struggle with this. Not because they lack capability, but because long-horizon planning is fundamentally hard in a step-by-step mode.

The Plan-Execute Architecture

Research on LLM agents has consistently shown that separating planning and execution yields better results.

The reasons:

Benefit	Explanation
Explicit long-term planning	Even strong LLMs struggle with multi-step reasoning when taking actions one at a time. Explicit planning forces consideration of the full path.
Model flexibility	You can use a powerful model for planning and a lighter model for execution—or even different specialized models per phase.
Efficiency	Each execution step doesn't need to reason through the entire conversation history. It just needs to execute against the plan.

What matters here: the plan becomes an artifact, and the execution becomes verification against that artifact.

If you've read about why LLMs are better at verification than first-shot generation, this should sound familiar. Creating a plan first converts the execution task from "generate good code" to "implement according to this plan"—a much clearer, more verifiable objective.

The Full Workflow

The complete picture:

Step 1: Preparation
    │
    ▼
Step 2: Design (Agree on Direction)
    │
    ▼
Step 3: Work Planning  ← The Most Important Step
    │
    ▼
Step 4: Execution
    │
    ▼
Step 5: Verification & Feedback

I'll walk through each step, but Step 3 is where the magic happens.

Step 1: Preparation

Goal: Clarify what you want to achieve, not how.

Create a ticket, issue, or todo document stating the goal in plain language
Point the LLM to AGENTS.md (or CLAUDE.md, depending on your tool) and relevant context files
Don't jump into implementation details yet

This is about setting the stage, not solving the problem.

Step 2: Design (Agree on Direction)

Goal: Align on the approach before any code gets written.

Don't Let It Start Coding Immediately

Instead of "implement this feature," say:

"Before implementing, present a step-by-step plan for how you would approach this."

Review the Plan

Look for:

Contradictions with existing architecture
Simpler alternatives the LLM missed
Misunderstandings of the requirements

At this stage, you're agreeing on what to build and why this approach. The how and in what order come in Step 3.

Step 3: Work Planning (The Most Important Step)

This section is dense. But the payoff is proportional—the more carefully you plan, the smoother execution becomes.

For small tasks, you don't need all of this. See "Scaling to Task Size" at the end.

Goal: Convert the design into executable work units with clear completion criteria.

Why This Step Matters Most

Research shows that decomposing complex tasks into subtasks significantly improves LLM success rates. Step-by-step decomposition produces more accurate results than direct generation.

But there's another reason: the work plan is an artifact.

When the plan exists, the execution task transforms:

Before: "Build this feature" (generation)
After: "Implement according to this plan" (verification)

This is the same principle from Article 1. Creating a plan first means execution becomes verification—and LLMs are better at verification.

What Work Planning Includes

Task decomposition: Break the design into executable units
Dependency mapping: Define order and dependencies between tasks
Completion criteria: What does "done" mean for each task?
Checkpoint design: When do we get external feedback?

Perspectives to Consider

I'll be honest: I learned most of these the hard way. Plans would fall apart mid-implementation, and only later did I realize I'd skipped something obvious in hindsight.

These aren't meant to be followed rigidly for every task. Think of them as a mental checklist. You don't need to get all of these right—if even one of these perspectives changes your plan, it's doing its job.

Perspective 1: Current State Analysis

Understand what exists before planning changes.

What is this code's actual responsibility?
Which parts are essential business logic vs. technical constraints?
What benefits and limitations does the current design provide?
What implicit dependencies or assumptions aren't obvious from the code?

Skipping this leads to plans that don't fit the existing codebase.

Perspective 2: Strategy Selection

Consider how to approach the transition from current to desired state.

Research options:

Look for similar patterns in your tech stack
Check how comparable projects solved this
Review OSS implementations, articles, documentation

Common strategy patterns:

Strangler Pattern: Gradual replacement, incremental migration
Facade Pattern: Hide complexity behind unified interface
Feature-Driven: Vertical slices, user-value first
Foundation-Driven: Build stable base first, then features on top

The key isn't applying patterns dogmatically—it's consciously choosing an approach instead of stumbling into one.

Perspective 3: Risk Assessment

Evaluate what could go wrong with your chosen strategy.

Risk Type	Considerations
Technical	Impact on existing systems, data integrity, performance degradation
Operational	Service availability, deployment downtime, rollback procedures
Project	Schedule delays, learning curve, team coordination

Skipping risk assessment leads to expensive surprises mid-implementation.

Perspective 4: Constraints

Identify hard limits before committing to a strategy.

Technical: Library compatibility, resource capacity, performance requirements
Timeline: Deadlines, milestones, external dependencies
Resources: Team availability, skill gaps, budget
Business: Time-to-market, customer impact, regulations

A strategy that ignores constraints isn't executable.

Perspective 5: Completion Levels

Define what "done" means for each task—this is critical.

Level	Definition	Example
L1: Functional verification	Works as user-facing feature	Search actually returns results
L2: Test verification	New tests added and passing	Type definition tests pass
L3: Build verification	No compilation errors	Interface definition complete

Priority: L1 > L2 > L3. Whenever possible, verify at L1 (actually works in practice).

This directly maps to "external feedback" from the previous articles. Defining completion levels upfront ensures you get external verification at each checkpoint.

Perspective 6: Integration Points

Define when to verify things work together.

Strategy	Integration Point
Feature-driven	When users can actually use the feature
Foundation-driven	When all layers are complete and E2E tests pass
Strangler pattern	At each old-to-new system cutover

Without defined integration points, you end up with "it all works individually but doesn't work together."

Task Decomposition Principles

After considering the perspectives, break down into concrete tasks:

Executable granularity:

Each task = one meaningful commit
Clear completion criteria
Explicit dependencies

Minimize dependencies:

Maximum 2 levels deep (A→B→C is okay, A→B→C→D needs redesign)
Tasks with 3+ chained dependencies should be split
Each task should ideally provide independent value

Build quality in:

Don't make "write tests" a separate task—include testing in the implementation task
Tag each task with its completion level (L1/L2/L3, though in practice L1 is almost always what you want)

Work Planning Anti-Patterns

Anti-Pattern	Consequence
Skip current-state analysis	Plan doesn't fit codebase
Ignore risks	Expensive surprises mid-implementation
Ignore constraints	Plan isn't executable
Over-detail	Lose flexibility, waste planning time
Undefined completion criteria	"Done" is ambiguous, verification impossible

Scaling to Task Size

Not every task needs full work planning.

Scale	Planning Depth
Small (1-2 hours)	Verbal/mental notes or simple TODO list
Medium (1 day to 1 week)	Written work plan, but abbreviated
Large (1+ weeks)	Full work plan covering all perspectives

For a typo fix, you don't need a work plan. For a multi-week refactor, you absolutely do.

Step 4: Execution

Goal: Implement according to the work plan.

Work in Small Steps

Follow the plan. One task at a time. One file, one function at a time where appropriate.

Types-First

When adding new functionality, define interfaces and types before implementing logic. Type definitions become guardrails that help both you and the LLM stay on track.

Why This Changes Everything

With a work plan in place, execution becomes verification. The LLM isn't guessing what to build—it's checking whether the implementation matches the plan.

If you need to deviate from the plan, update the plan first, then continue implementation. Don't let plan and implementation drift apart.

Step 5: Verification & Feedback

Goal: Verify results and externalize learnings.

Feedback Format

When something goes wrong, don't just paste an error. Include the intent:

❌ Just the error
[error log]

✅ Intent + error
Goal: Redirect to dashboard after authentication
Issue: Following error occurs
[error log]

Without intent, the LLM optimizes for "remove the error." With intent, it optimizes for "achieve the goal."

Externalize Learnings

If you find yourself explaining the same thing twice, it's time to write it down.

I covered this in detail in the previous article—where to put rules, what to write, and how to verify they work. The short version: write root causes, not specific incidents, and put them where they'll actually be read.

Referencing Skills and Rules

One common failure mode: you reference a skill or rule file, but the LLM just reads it and moves on without actually applying it.

The Problem

Pattern	Issue
Write "see AGENTS.md"	It's already loaded—redundant reference adds noise
`@file.md` only	LLM reads it, then continues. Reading ≠ applying
"Please reference X"	References it minimally, doesn't apply the content

The Solution: Blocking References

Make the reference a task with verification:

## Required Rules [MANDATORY - MUST BE ACTIVE]

**LOADING PROTOCOL:**
- STEP 1: CHECK if `.agents/skills/coding-rules/SKILL.md` is active
- STEP 2: If NOT active → Execute BLOCKING READ
- STEP 3: CONFIRM skill active before proceeding

Why This Works

Element	Effect
Action verbs	"CHECK", "READ", "CONFIRM"—not just "reference"
STEP numbers	Forces sequence, can't skip
Before proceeding	Blocking—must complete before continuing
If NOT active	Conditional—skips if already loaded (efficiency)

This maps to the task clarity principle: "check if loaded → load if needed → confirm → proceed" is far clearer than "please reference this file."

How This Connects to the Theory

Step	Connection to LLM Characteristics
Step 1: Preparation	Task clarification
Step 2: Design	Artifact-first (design doc is an artifact)
Step 3: Work Planning	Artifact-first (plan is an artifact) + external feedback design
Step 4: Execution	Transform "generation" into "verification against plan"
Step 5: Verification	Obtain external feedback + externalize learnings

The work plan created in Step 3 converts Step 4 from "generate from scratch" to "verify against specification." This is the key mechanism for improving accuracy.

The Research

The practices in this article aren't just workflow opinions—they're backed by research on how LLM agents perform.

ADaPT (Prasad et al., NAACL 2024): Separating planning and execution, with dynamic subtask decomposition when needed, achieved up to 33% higher success rates than baselines (28.3% on ALFWorld, 27% on WebShop, 33% on TextCraft).

Plan-and-Execute (LangChain): Explicit long-term planning enables handling complex tasks that even powerful LLMs struggle with in step-by-step mode.

Multi-Layer Task Decomposition (PMC, 2024): Step-by-step models generate more accurate results than direct generation—task decomposition directly improves output quality.

Task Decomposition (Amazon Science, 2025): With proper task decomposition, smaller specialized models can match the performance of larger general models.

Key Takeaways

Don't let it execute immediately. Ask for a plan first. Even just "present your approach step-by-step before implementing" makes a significant difference.
Work Planning is the superpower. A plan is an artifact. Having it converts execution from generation to verification—and LLMs are better at verification.
Define completion criteria. L1 (works as feature) > L2 (tests pass) > L3 (builds). Know what "done" means before starting.
Scale to task size. Small task = mental note. Large task = full work plan. Don't over-plan trivial work, don't under-plan complex work.
Update plan before deviating. If implementation needs to differ from the plan, update the plan first. Drift kills the verification benefit.
Include intent with errors. "Goal + error" beats "just error." The LLM should know what you're trying to achieve, not just what went wrong.

References

Prasad, A., et al. (2024). "ADaPT: As-Needed Decomposition and Planning with Language Models." NAACL 2024 Findings. arXiv:2311.05772
Wang, L., et al. (2023). "Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models." ACL 2023.
LangChain. "Plan-and-Execute Agents." https://blog.langchain.com/planning-agents/

Stop Guessing If Your Prompt Is Better

Shinsuke KAGAWA — Thu, 22 Jan 2026 13:52:50 +0000

You rewrote your prompt. The output looks different. But is it actually better?

Most of us have been there — reading prompt engineering best practices, tweaking instructions, and hoping the changes help. But without comparison, you're just guessing.

The Problem

When you improve a prompt, you typically:

Run the new version
Look at the output
Think "yeah, this seems better"

But you're comparing against your memory of the old output. Different runs produce different results anyway. How do you know the improvement came from your changes and not just LLM variance?

What I Built

rashomon is a Claude Code plugin that focuses on one practical question: "Did my instruction change actually affect the result?"

It analyzes your prompt, generates an optimized version, runs both in isolated environments, and compares the actual results.

Real Example

I ran this prompt through rashomon:

Add logging to track function calling usage

A reasonable instruction. But vague.

What rashomon detected

Issue	Detail
Vague instructions	What, where, and why to log are unclear
No output format	Log structure not specified
Missing context	No project architecture information

The optimized prompt

## Context

This is a Slack bot using Google Gemini API with function calling. 
The project uses a shared `logger` utility with structured logging.
Function calling flows through:
1. `GeminiService.executeWithRetry()` - detects function calls
2. `FunctionHandler.handleFunctionCall()` - executes them

## Task

Add logging to track function calling usage for analytics and debugging.

## Requirements

At Function Call Detection (GeminiService):
- Function name(s) detected
- Number of function calls in response

At Function Execution (FunctionHandler):
- Parameters passed (sanitized - exclude sensitive data)
- Execution duration
- Result status (success/failure)

## Output Format

logger.info('Function call detected', {
  functionName: 'executeWithRetry',
  detectedFunctions: ['searchNotionPages'],
  functionCallCount: 1
})

What changed

Aspect	Original	Optimized
Logging Scope	1 stage (execution only)	2 stages (detection + execution)
Parameter Sanitization	None	Passwords, tokens, secrets redacted
Files Modified	2	2

The original prompt looked reasonable, but led the agent to log at only one point. The optimized version covered both detection and execution — with security considerations the original didn't address.

Classification: Structural Improvement

About Variance

Not every difference is an improvement. rashomon distinguishes between structural gains and mere variance.

I tried to create a Variance example — a prompt so clear that optimization wouldn't matter. I couldn't. In practice, the same vague prompt sometimes works beautifully, sometimes completely misses the point.

rashomon just makes that inconsistency visible.

Try It

Requires Claude Code.

claude
/plugin marketplace add shinpr/rashomon
/plugin install rashomon@rashomon
# Restart session
/rashomon Your prompt here

shinpr / rashomon

Compare, improve, and verify prompt changes with evidence — not vibes.

See what actually changes when you improve your prompts — not just different wording.

Why rashomon?

Inspired by the Rashomon effect — the idea that the same event can produce different outcomes depending on perspective rashomon makes those differences explicit and comparable.

Spending too much time on trial-and-error with prompts?
Read best practices but not sure how they apply to your case?
Want proof that your changes actually made things better?

rashomon analyzes, improves, and compares prompts—so you can see what actually changed, and whether it matters.

Who Is This For?

rashomon is designed for:

Developers using Claude Code daily
Teams iterating on complex prompts (coding, analysis, writing)
Anyone who wants evidence, not vibes, when improving prompts

Not ideal if:

You don't use git
You want one-shot prompt rewriting without comparison

Quick Example

/rashomon Write a function to sort an array

What You Get

1. Detected Issues

- BP-002

…

View on GitHub

Stop Putting Everything in AGENTS.md

Shinsuke KAGAWA — Mon, 19 Jan 2026 14:13:49 +0000

If you're using Agentic Coding and find yourself explaining the same thing to the LLM over and over, you have a learning externalization problem.

The fix seems obvious: write it down in AGENTS.md (or CLAUDE.md, depending on your tool) and never explain it again.

Note: This article uses "AGENTS.md" as the generic term for root instruction files. Claude Code uses CLAUDE.md, Codex uses AGENTS.md, and other tools have their own conventions. The principles apply regardless of the specific filename.

But here's what actually happens—you keep adding rules, AGENTS.md grows to 200+ lines, and somehow the LLM still ignores half of what you wrote.

This article is about how to actually make your rules stick: where to write them, what to write, and how to verify they work.

The Real Problem

LLMs don't learn across sessions. Every conversation starts fresh. This means:

You explain something once
It works
Next session, you explain it again
And again
Eventually you get frustrated

The solution is to externalize your learnings into rules. But most people do this wrong.

The Common Mistakes

Mistake	What Happens
Put everything in AGENTS.md	It bloats, becomes noise, important rules get buried
Put everything in code comments	LLM doesn't load them into context unless you explicitly reference the file
Don't write it down at all	You repeat yourself forever

The thing is, where you write a rule determines whether the LLM actually follows it.

Where to Write Rules

Not all rules belong in the same place. A simple decision tree:

When is this rule needed?
│
├─ Always, on every task → AGENTS.md
│
├─ When working on a specific feature → Design Doc
│
├─ When using a specific technology → Rule file (skill)
│
└─ When performing a specific task type → Task guidelines

Note: "Skills" are modular rule files used in tools like Codex and Claude Code. They allow you to inject context-specific rules only when relevant. If your tool doesn't have this concept, think of them as separate rule files you reference when needed.

"Task guidelines" refers to rules that apply only during specific operations—like code review, migration, or content generation. Some call these "task rules" or "task-specific constraints."

The Full Picture

Destination	Scope	When Applied	Examples
AGENTS.md	All tasks	Always	Approval flows, stop conditions, project principles
Rule files (skills)	Specific technology area	When using that tech	Type conventions, error handling patterns, function size limits
Task guidelines	Specific task type	When doing that task	Subagent usage rules, review procedures
Design docs	Specific feature	When developing that feature	Feature requirements, API specs, security constraints
Code comments	Specific code location	When modifying that code	Implementation rationale, gotchas

The Key Question

Ask yourself: "Is this needed on every task in this project?"

Yes → AGENTS.md
No → Put it closer to where it's needed

This keeps AGENTS.md lean (around 100 lines) and ensures task-specific rules don't create noise for unrelated work.

You don't need to get this perfect from day one. Start with one thing: keep AGENTS.md small. That alone changes a lot.

What to Write

This is the hard part. Most people write the wrong thing.

The Principle: Write Root Causes, Not Incidents

When something goes wrong, the instinct is to document the specific incident. But this creates bias—the LLM over-fits to that one case.

❌ Bad (specific incident)
"The getUser() function in UserService was missing null check"

✅ Good (root cause / system fix)
"Always null-check return values from external APIs"

The first one only helps if the LLM encounters that exact function again. The second one prevents the entire class of errors.

Specific Incident vs. Root Cause

Aspect	Specific Incident	Root Cause
Applies to	That one location	All similar cases
Prevents recurrence	Weakly (same bug elsewhere)	Strongly (operates as principle)
Bias risk	High (overfitting)	Low (generalizable)

Finding the Root Cause

When you encounter an issue, ask:

Why did this mistake happen? (direct cause)
Why wasn't it prevented? (system gap)
Where else could this same mistake occur? (scope)

Example:

Direct cause: getUser() was missing null check
System gap: We trusted external API return values without validation
Scope: All external API calls

→ Rule to write: "Always null-check return values from external APIs"

How to Verify Rules Work

This is the step most people skip—and it's critical.

The Principle: Fix the System, Then Discard and Retry

When you add or modify a rule in AGENTS.md or a skill file, you need to verify it actually works. The only way to do this:

Add/modify the rule
Discard the current artifact (or stash it in a branch)
Start a new session with the updated rules
Re-run the same task
Verify the issue doesn't recur

Continue with existing artifact after rule change → ❌
Discard and restart with new rules → ✅

Why This Matters

If you keep the existing artifact and just continue, you're still operating in a context polluted by the old system. The new rule might not get properly applied because:

The existing artifact carries biases from before the rule existed
The LLM might try to "reconcile" the new rule with existing work rather than applying it cleanly
You can't tell if the rule actually works or if you just manually fixed the symptom

Verification Checklist

[ ] Modified the rule (AGENTS.md / skill file / task guideline)
[ ] Discarded current artifact (or moved to a branch)
[ ] Started new session with updated rules
[ ] Re-ran the same task
[ ] Confirmed the issue doesn't recur

For small changes, you can stash instead of discard. The key is: test the system in isolation.

When to Write Rules

Not every issue deserves a rule. Some guidance:

Situation	Write a Rule?	Rationale
You explained the same thing twice	Yes	Prevent the third time
Encountered unexpected behavior	Maybe	Find root cause first
Task completed successfully	Maybe	Retrospective—any generalizable insights?
Found a serious bug	Yes	Prevent recurrence

Warning Signs You're Over-Documenting

AGENTS.md exceeds 100 lines
A single rule file exceeds 300 lines (~1,500 tokens)
Rules take more than 1 minute to read through
You find yourself thinking "is this really needed every time?"
Rules contradict each other

If you see these signs, it's time to prune. Rule maintenance includes deletion.

How to Write Rules (Cheat Sheet)

This section is a reference. You don't need to read it all now—come back when you're actually writing a rule. The rest of the article stands on its own.

1. Minimum Viable Length

Context is precious. Same meaning, shorter expression. But don't sacrifice clarity for brevity.

❌ Verbose (38 chars)
If an error occurs, you must always log it

✅ Concise (20 chars)
All errors must be logged

❌ Too short (unclear)
Log errors

2. No Duplication

Same content in multiple places wastes context and creates update drift.

❌ Duplicated
# base.md
Standard error format: { success: false, error: string }

# api.md
Errors use { success: false, error: string } format

✅ Single source
# base.md
Standard error format: { success: false, error: string }

3. Measurable Criteria

Vague instructions create interpretation variance. Use numbers and specific conditions.

✅ Measurable
- Functions: max 30 lines
- Cyclomatic complexity: max 10
- Test coverage: min 80%

❌ Vague
- Readable code
- Sufficient testing

4. Recommendations Over Prohibitions

Banning things without alternatives leaves the LLM guessing. Show the right way.

✅ Recommendation + rationale
【State Management】
Recommended: Zustand or Context API
Reason: Global variables make testing difficult, state tracking complex
Avoid: window.globalState = { ... }

❌ Prohibition list
- Don't use global variables
- Don't store values on window

5. Priority Order

LLMs pay more attention to what comes first. Lead with the most important rules.

## Critical (Must Follow)
1. All APIs require JWT authentication
2. Rate limit: 100 requests/minute

## Standard Specs
- Methods: Follow REST principles
- Body: JSON format

## Edge Cases (Only When Applicable)
- File uploads may use multipart

6. Clear Scope Boundaries

State what the rule covers—and what it doesn't.

## Scope

### Applies To
- REST API endpoints
- GraphQL endpoints

### Does Not Apply To
- Static file serving
- Health checks (/health)

The Feedback Loop

This is how it all fits together in practice:

[Working with LLM]
       │
       ├─ Issue occurs
       │      │
       │      ▼
       │  Find root cause (not just symptom)
       │      │
       │      ▼
       │  Decide where to write (AGENTS.md? Skill? Task guideline?)
       │      │
       │      ▼
       │  Write the rule
       │      │
       │      ▼
       │  Discard current work
       │      │
       │      ▼
       │  New session with updated rules
       │      │
       │      ▼
       │  Verify issue doesn't recur
       │
       ▼
[Continue working]

The goal is to reach a state where you never explain the same thing twice. Every explanation either:

Gets externalized into a rule, or
Was truly a one-off that doesn't need capturing

Passing Feedback Correctly

One more thing: when you give feedback to the LLM, don't just paste error logs. Include your intent.

❌ Just the error
[Stack trace]

✅ Intent + error
Goal: Redirect to dashboard after user authentication
Issue: Following error occurred
[Stack trace]

Without the intent, the LLM optimizes for "make the error go away." With the intent, it optimizes for "achieve the goal while resolving this error."

These are very different things.

Anti-Pattern Summary

Quick reference if you want to check your current practices:

Anti-Pattern	Reference
Put everything in AGENTS.md	→ "Where to Write Rules"
Write specific incidents instead of root causes	→ "What to Write"
Continue with old artifacts after changing rules	→ "How to Verify Rules Work"
List only prohibitions without recommendations	→ "How to Write Rules" #4
Keep explaining instead of writing it down	→ "When to Write Rules"

Key Takeaways

AGENTS.md is not a dumping ground. Only rules needed on every task belong there. Everything else goes closer to where it's used.
Write root causes, not incidents. "Null-check external API returns" beats "UserService.getUser() was missing null check."
Test your rules. After adding a rule, discard current work and re-run. If the issue recurs, the rule isn't working.
Maintenance includes deletion. If AGENTS.md is over 100 lines, you've probably over-documented. Prune ruthlessly.
Explain twice, document once. If you're explaining the same thing for a second time, stop and externalize it.

Once you stop expecting rules alone to do the work, the real question becomes how to design the workflow around them. In practice, that starts with planning—before execution ever begins.

The Research

The practices in this article are grounded in LLM research:

SALAM (Wang et al., 2023): LLM self-feedback is often inaccurate. Structured feedback from external agents (or externalized rules) is more effective.

LEMA (An et al., 2023): Learning from mistakes (error → explanation → correction) improves LLM reasoning ability—but this requires explicit externalization of what was learned.

Feedback Loop for IaC (Palavalli et al., 2024): Feedback loop effectiveness decreases exponentially with each iteration and plateaus. This supports the "discard and restart" approach over endless iteration in the same context.

Reflexion (Shinn et al., 2023): Combining short-term memory (recent trajectory) with long-term memory (past experience) enables effective self-improvement. Externalized rules function as that long-term memory.

References

Wang, D., et al. (2023). "Learning from Mistakes via Cooperative Study Assistant for Large Language Models." arXiv:2305.13829
An, S., et al. (2023). "Learning From Mistakes Makes LLM Better Reasoner." arXiv:2310.20689
Palavalli, M. A., et al. (2024). "Using a Feedback Loop for LLM-based Infrastructure as Code Generation." arXiv:2411.19043
Shinn, N., et al. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS 2023.

Why LLMs Are Bad at "First Try" and Great at Verification

Shinsuke KAGAWA — Mon, 12 Jan 2026 12:46:19 +0000

I used to spend hours crafting the perfect prompt.
Detailed instructions, examples, constraints—the works.

And the AI would still add random features I never asked for.
Or refactor code that was perfectly fine.
Or skip steps it decided were "unnecessary."

Eventually it clicked: I was fighting a losing battle.
So I stopped trying to control generation and started focusing on verification.

The Failure Patterns You've Probably Seen

Before diving into why, these are the common anti-patterns:

The Giant Prompt Syndrome: Cramming requirements, design, implementation, and improvement into a single prompt
Overconfidence in Abstract Instructions: Expecting "think carefully" or "be thorough" to actually improve quality
The Invisible Loop: Thinking you're iterating when you're actually spinning in circles within the same biased context
Context Bloat: Adding "just in case" information until the actually important instructions get buried

If any of these sound familiar, you're in the right place.

The Core Insight

The claim is simple:

LLMs perform better at "verify and improve existing artifacts" than at "controlled first-time generation."

Instead of trying to get the perfect output on the first attempt, you get better results by:

Having the LLM produce something first
Then having it verify and improve that output

This is grounded in how LLMs actually process information.

Why Verification Works Better

At first, I assumed better prompts would lead to better first-shot output. But after enough failures, the pattern became clear: there are three interconnected reasons why LLMs become "smarter" when they have something to work with.

1. External Feedback Changes the Task

When an artifact exists, the task fundamentally transforms:

Without artifact: "Generate something good" (vague, open-ended)
With artifact: "Identify what's wrong with this and fix it" (specific, bounded)

The second task has clearer success criteria. The LLM isn't guessing what "good" means—it can evaluate concrete issues against concrete output.

2. Position Bias (Lost in the Middle)

Research has shown that LLMs exhibit a U-shaped attention pattern: they prioritize information at the beginning and end of their context window, while information in the middle tends to get overlooked.

When you feed an artifact as input to a new session, it naturally occupies a prominent position in the context. The LLM is literally forced to pay attention to it.

This also explains why that really important instruction you buried in paragraph 5 of your mega-prompt keeps getting ignored.

3. Task Clarity Drives Performance

"Improve this code" is a more concrete task than "write good code."

The presence of an artifact provides:

A specific target for evaluation
Clear boundaries for the scope of work
Implicit success criteria (this one matters more than you'd think—"better than before" is much easier to verify than "good enough")

The Externality Spectrum

What made the biggest difference for me was reviewing in a completely separate context.
Once I stopped letting the generator review its own work, the blind spots became obvious.

Not all feedback loops are created equal. Different approaches rank very differently in effectiveness:

The thing is, looping within the same session is fundamentally internal feedback. The LLM is still operating within its original generation biases. Only by separating context do you get true "external" perspective.

In short: if the context doesn't change, neither does the model's perspective.

Practical Implications

So what do you actually do with this?

1. Artifact-First Workflow

Stop trying to get everything right in one shot. Instead:

Generate Phase: Get something out, even if imperfect. Don't over-specify.
External Feedback: Run the code, execute tests, use linters
Verification Phase (new session): Feed the artifact + feedback to a fresh context

[Generation Session]
    │
    ├── Input: Requirements, constraints
    ├── Output: Artifact (code, design, etc.) + brief intent summary (1-3 lines)
    │
    ▼
[External Feedback]
    │
    ├── Code execution
    ├── Test execution
    ├── Linter/static analysis
    │
    ▼
[Verification Session]  ← Fresh context
    │
    ├── Input: Artifact + intent summary + feedback results
    ├── Output: Improved artifact
    │
    ▼
[Repeat or Complete]

2. Know When to Separate Sessions

Session separation isn't always necessary. Use your judgment:

Task Type	Approach
Small, localized fixes (typos, formatting)	Same session is fine
Clear error fixes (with stack trace)	Same session works—external feedback (error log) exists
Design changes, architecture revisions	Separate sessions
Quality improvements, refactoring	Separate sessions
Direction changes, requirement pivots	Separate sessions

Rule of thumb: If you're feeling "something isn't working," that's often a sign to start a fresh session. Your intuition about context pollution is usually right.

3. What to Pass Between Sessions

Not everything from the generation phase should go to the verification phase.

Content	Should Pass?	Why
Full Chain-of-Thought log	No	Verbose, becomes noise. Important info gets lost (position bias)
Intent summary (1-3 lines)	Yes	Preserves the "why" compactly
Final decision + rationale	Yes	Useful for debugging
Rejected alternatives	Maybe	Only when specifically relevant

The principle: Pass the "why," not the "how I thought about it."

Designing Your AGENTS.md

You don't need to redesign your AGENTS.md all at once. But understanding position bias changes how you think about what goes in it.

This insight has direct implications for how you structure AGENTS.md (or whatever root instruction file you use—CLAUDE.md, cursorrules, etc.).

The Position Bias Problem

Context Window Position → Attention Weight

[AGENTS.md]            ← Start: HIGH attention
       ↓
[Middle instructions]  ← Middle: LOW attention (Lost in the Middle)
       ↓
[User prompt]          ← End: HIGH attention

If your AGENTS.md is bloated, the truly important principles get diluted. Adding more "just in case" actually makes everything weaker.

Design Principles

What	Approach
AGENTS.md	Core principles only. ~100 lines. What must be followed on every task
Task-specific info	Inject via skills, command arguments, or reference files when needed
Why separate?	Context separation lets you compose optimal information for each task

What Belongs in AGENTS.md

Project purpose and domain
Non-negotiable constraints (security, naming conventions)
Tech stack overview
Communication style
Error handling behavior

What Doesn't Belong

Individual feature specs
API details
Task-specific workflows
Long code examples
"Nice to have" information

Test: Ask "Is this needed for every task?" If no, it belongs elsewhere.

The Human Role

In Agentic Coding, you're not "using an LLM"—you're designing a system where an LLM operates.

Your Responsibilities

Responsibility	Concrete Actions
Design external feedback	Decide which tests to run, which linters to use, what "success" means
Determine session boundaries	Judge when to cut context, what carries over
Define quality gates	Separate automated checks from human review needs
Maintain AGENTS.md	Keep core principles tight, prevent bloat
Articulate intent	Create or validate the "intent summary" that passes between sessions

Automation vs. Human Review

Good for automation:

Code execution, test execution
Linters, formatters
Type checking
Security scans
Applying formulaic fixes

Requires human review:

Design decision validity
Requirement alignment
Session boundary judgment
Trade-off decisions
Validating the "why"

A Framework: Context Separation at Every Level

It took me a while to realize this wasn't about writing better prompts—it was about where I drew the boundaries.

You don't need to apply all of this rigidly. But when something feels off, one of these levels is usually the culprit:

The Research Behind This

These aren't just opinions—they're grounded in LLM research:

Self-Refine (Madaan et al., 2023)
The generate → feedback → refine loop shows approximately 20% improvement over single-shot generation. Key insight: the improvement comes from the structured iteration, not from the model "trying harder."

Lost in the Middle (Liu et al., 2023)
LLMs show U-shaped attention bias, heavily weighting the beginning and end of context while underweighting the middle. This explains why your carefully crafted instructions in paragraph 5 keep getting ignored.

LLMs Cannot Self-Correct Reasoning Yet (Huang et al., 2023)
Without external feedback, self-correction doesn't work—and can actually make things worse. "Review your work" as an instruction has minimal effect; external signals (test failures, linter errors) are what drive actual improvement.

Key Takeaways

Don't optimize for first-shot perfection. Get something out, then improve it.
Session separation is real. The same context that generated the artifact will struggle to objectively improve it.
External feedback is non-negotiable. Tests, linters, execution results—these are what drive quality, not "think harder" prompts.
Keep AGENTS.md lean. Position bias means bloat actively hurts. If it's not needed for every task, move it out.
Pass intent, not process. Between sessions, transfer the "why" in 1-3 lines, not the full thought log.
You're a system designer. Your job isn't to use the LLM—it's to design the workflow, feedback loops, and context boundaries that let it perform.

What's Next

This article focused on why verification-oriented workflows outperform first-shot generation. In future articles, I'll cover:

How to structure work plans that turn execution into verification
Where to put rules so they actually get followed (hint: not all in AGENTS.md)

If you've been struggling with inconsistent LLM output or finding that your detailed prompts underperform simpler ones, try restructuring around verification. The difference is often dramatic.

What's your experience been? Did switching to a verification-first approach change anything for you?

References

Madaan, A., et al. (2023). "Self-Refine: Iterative Refinement with Self-Feedback." arXiv:2303.17651
Liu, N. F., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." arXiv:2307.03172
Huang, J., et al. (2023). "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024. arXiv:2310.01798
Hsieh, C.-Y., et al. (2024). "Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization." ACL 2024 Findings. arXiv:2406.16008

DEV Community: Shinsuke KAGAWA

Was the Monorepo Really Best for Coding Agents?

The repository stopped being the context boundary

Context closure is not context governance

Better models make conflicting context more expensive

Centralizing before the capability exists

Standardizing after local systems are embedded

Centralize governance, distribute its maintenance

Shared authority needs distributed capability

The harness should need less context over time

Judge governance by the cost of outcomes

The monorepo is not the answer by itself

Reasoning Effort Is Not a Quality Setting

What I actually measured

What high effort changed in Opus

Twenty-two million tokens missed one sentence

The same behavior helped in code review

Sol showed why this is not a medium-versus-high story

The implementations separated at the edges

Resources describe activity, not quality

Effort belongs to the phase

What I can and cannot conclude

Exploration is not the finished product

Complete phase scores

Methodology notes

When Better Models Make Old Agent Workflows Worse

Two kinds of constraint

Why the problem surfaced now

How reliability started manufacturing work

What I changed

Decide the outcome before designing the solution

Give the model decisions, not a predicted route

Let review findings be declined

The constraint I had to put back

What remains strict

The four questions I now ask

A workflow can become technical debt

A second pair of eyes for Claude Code: building Galley, a local runner that checks the work before the PR opens

TL;DR

The setup that made me build this

"Third time, automate it"

What Galley actually does

The decisions I'd actually defend

A Skill installs the tool

Acceptance-criteria test skeletons go in first

quality.yaml is a thing I grow

Claude writes, a second model signs off

The deterministic / non-deterministic seam

Galley building Galley

Where it is

Claude Code

Codex

I Built a Skill Reviewer. Then I Ran It on Itself.

The Setup

The Discovery

Digging Into Logs

Fixing the Criteria, Not the Reviewer

The Retest

What This Comes Down To

The Broader Problem: Skill Quality Is Hard

Try It

Same Framework, Different Engine: Porting AI Coding Workflows from Claude Code to Codex CLI

TL;DR

The 8-Month Wait

What "Near-Zero Migration" Actually Looks Like

Why This Worked: Design Decisions That Paid Off

1. Natural Language as the Interface Layer

2. Context Separation as Architecture

3. Structured Handoffs Over Shared State

The Framework in Action

Getting Started

Letting LLMs Jump — and Then Verifying Ruthlessly

The "First Plausible Answer" Problem

The Factor That Made the Difference: Purpose

A Real Example: Bug Investigation

The Old Approach

What I Changed

Implementation Example

Main Command (diagnose.md)

Sub-agent: Investigator

`quality.yaml` is a thing I grow