DEV Community: Shinsuke KAGAWA

Read the Logs Before You Rewrite the Prompt

Shinsuke KAGAWA — Thu, 22 Oct 2026 12:07:19 +0000

For a while I tried to fix my orchestrator by writing to it.

It runs on GPT-6.1 Sol, and it had habits. It wrote notes into items addressed to other agents. It handed work to the expensive executor when the cheap one would have done. Every time I found one, the prompt got another paragraph explaining why not to. Sol's habits are strong enough that a polite instruction does nothing, so the paragraphs were firm, repeated themselves, and gave reasons. In a week its instructions grew from about 1,500 words to 2,100, and the last fix alone added about 400.

What changed my approach was counting. I asked for the runs of the past day: which roles were launched, how often, and what each launch did. The orchestrator had been launched more than twenty times just to receive a finished task and file the next one. None of those launches needed a judgment. A goal's plan was already a list in order, each item with the role that should do it. The orchestrator read the list and filed the next entry, and sometimes filed it for the wrong role.

Anything that can be decided deterministically is more stable in a script than in any model. That sentence is obvious, and I had still been writing prompts around it.

What moved into scripts

The team works on a board of markdown files, and a resident shell script I call the studio launches whichever agent an item is addressed to. Filing a goal's next task went into that script first. The plan already names who does each item, so when the board has nothing in progress, the studio files the next one for that role and marks it done when the task finishes. The orchestrator now sees a goal item only when I should see its result, when it is the last one, or when something went wrong. The lead executor, which plans, stopped being the default destination for anything vague.

The notes in other agents' items took a script and a sentence. Before each orchestrator pass the studio snapshots every task and review addressed to someone else, and after the pass it puts back any body the pass changed and tells the next pass what it undid. The prompt kept one boundary: into an item addressed to another agent, you write nothing, not even a note. That replaced the 400 words. To check it, I rebuilt fifteen real past states of the board in a scratch copy and ran the orchestrator over each. Before the change it wrote into another agent's item in ten of the fifteen. After, in none.

Two smaller jobs went the same way. Agent failures used to reach me as the last line of a log; now the studio reads the error code each CLI already records in its session files and retries only an overloaded server. Scratch files the agents never cleaned up are deleted once nothing on the board names them and nobody has touched them for three days.

The instructions came back down to about 1,750 words: the 400 were gone, and the rest of that week's growth came from other changes. Per hour the team ran, the orchestrator's cost fell from $0.67 to $0.34. I had been thinking of moving it to Claude Opus to cure its habits. With less and less of its old work left to it, I dropped the idea. Two nights later the team ran a goal from start to finish without a correction from me.

The morning the board stopped

The morning after that night, the board stopped moving at 8:07. I noticed eight and a half hours later.

The orchestrator had sometimes made the wrong call. The script made no call at all. The studio launches the orchestrator when an item reaches a status it should handle, and to avoid launching it twice for the same thing, it remembered which item was in which status. It took that snapshot only between passes. During one pass the orchestrator sent a finished task back to the lead executor, the lead executor finished it again, and all of that happened before the next snapshot. The studio saw the same task in the same status it had already handled and launched nothing. A sprint review sat waiting on that task all day.

The log showed this in a few minutes, and the fix was one field: the snapshot now includes the file's modification time, so a task whose file has changed counts as new even when its status reads the same.

The same stall exposed a second problem, and that one came from my own change two days before. Since the orchestrator could no longer write into the task, it had sent the task back by flipping its status, with the reason written in its own review item. To the lead executor the task looked identical to one whose process had died. Three launches in a row it rechecked the numbers it had already checked and never read why the task had come back. Work that falls short now comes back as a new task, with the shortfall written as its second line.

The habits I had tried to fix in the prompt showed up as behavior I had to notice and then count. The bugs in the scripts showed up as a line in a log. The same logs had shown me, two days earlier, which of the orchestrator's work had never needed a judgment in the first place.

I Took the Wheel Back From My AI Team. Then I Gave It Back.

Shinsuke KAGAWA — Wed, 21 Oct 2026 13:09:04 +0000

Yesterday morning I wrote my AI team's next goal myself. Last night I agreed to the goal the team proposed with "Looks fine, go ahead," and it worked until morning. It sent me five progress reports. Its reviewer found a problem three times, and each time the team fixed it inside the goal. By 7:53 a director had judged every key result supported.

This is the team from my last post: two AI directors who decide what to build, executors who build it, a reviewer, and an orchestrator that moves work between them on a board of markdown files. I keep the studio, the rules and scripts they all run on. The team keeps its own working agreements. Between that post and this one, I took most of the team's decisions away from it and then gave them back.

Taking it back

The sprint before ran for four days. It planned nothing from its purpose. Each time a task fell short, someone refined it into more tasks, and the directors started telling executors how to do their work. On the fourth morning I ended it and wrote:

The problem is that you did not work toward the goal. You refined tasks as they came instead of planning from the purpose, micromanaged beyond your authority, and only added work.

The next sprint lasted a day and ended the same way. This time I could see why: nobody had said what a good result looked like, so every variable in the team's simulation was free and each search landed wherever the numbers allowed. I stopped asking the team what to do next and wrote the goal myself: gather what the simulation showed and research on comparable products, put a few candidate targets and a list of open decisions in front of me, and have me adopt one or more.

I took the team's decisions and left its documents alone. I had to draw that line with my own assistant first, the coding agent I use to maintain the studio. Asked to fix a rule, it went into the team's design file and edited it. "Put it back," I told it. "Leave the team's things alone. Your job is the studio."

So everything I changed was in the studio. Rules that said the same thing in three places came down to one. The working principles the team was supposed to think with had lived in a skill, which the agents loaded and treated as reference material; I moved them into the rules every agent follows. The routine work the orchestrator had been doing by judgment moved into scripts, and that part is its own post.

Six corrections

The goal I wrote ran from 9:30 to 18:46, and I corrected it six times. At 10:19 I wrote:

Everything here looks like it starts from the simulation. Research, trends, and what the target users prefer should come first, as the reason for a hypothesis, and only then should past simulation results be applied to it. It's about time you changed how you work. You keep starting from the how, from whatever is easiest for you to control. That is why this has taken days and still isn't done.

I had said versions of this before. Once it was a goal that stopped at building the foundation instead of using it. Once it was a variable fixed as a constant because a smaller model was easier to run. Once it was a search narrowed before it started. Each time the studio got a fix for that particular form, and the habit came back in the next one.

A rule from my last post was part of why. Fifty minutes before that message, a retrospective had deleted the team's own agreement to ground a proposal in sourced research on what its users want, because its benefit hadn't shown. That test lumped together a rule nobody needs anymore and a rule nobody manages to follow. A problem that keeps coming back means the rule wasn't followed, or following it wasn't enough to prevent the problem. Now a recurring problem sends its rule back to be rewritten, and deletion is for rules whose problem has stopped. The deleted agreement itself never came back. The fix went into the studio.

The goal still finished. The two before it had run four days and a day and then been abandoned. This one reached a result I could adopt in nine hours, and after each correction the next result came back within an hour or two. But I was still the one finding the problem, and it was the same problem.

Replaying the morning

I wanted the habit gone, so I replayed the morning. In a scratch copy I rebuilt the board as it stood then and launched the same agent on it again with the same inputs. Replaying the 9:30 planning step showed where it went wrong. The first item of the plan sketched the candidates from the simulation records the team already had. The research came second, and its done condition was to support the candidates in that sketch. The answer had been drafted before the search, and the search was there to back it up.

The rules allowed this. One principle said to show the riskiest premise early, by the cheapest means: a thin slice, a question to whoever judges. Another said to search widely before narrowing. Nothing said which comes first when the work is finding something out. I added this sentence to the second one:

The first result such work shows is what the search found, not a draft of the answer: a draft made before the search, even one made only to show its form early, anchors the search to itself, and what follows only supports it.

Then I replayed the planning step again. With the old rules, two runs out of two drafted first. With the new sentence, three out of three put the search first and drew the early sample from what it found.

Giving it back

The team had proposed the next goal at seven that evening, worked out from the targets I had adopted that afternoon. The new sentence went in a little before I answered. I agreed in one line, and the goal started at 22:28.

I went to bed. The reviewer found a problem at 22:35, again at 2:27, and again at 6:27. Each time the work went back to the agent that owned it and was fixed inside the goal, without a new agreement and without me.

Very little went as planned. The plan grew from five items to fifteen, and the goal ran about two hours past its due time. Fourteen minutes after the director's judgment, the board stopped moving for most of the day. That was my bug, not the team's, and it belongs to the next post.

The day before, every change of course had started with a message from me. This time the team changed course on its own three times, and when I read the night's work and checked what it claimed, I found nothing I wanted to change.

My AI Team Followed Three Rules. It Kept Two.

Shinsuke KAGAWA — Mon, 05 Oct 2026 12:00:00 +0000

At eleven last night a message arrived in my Slack channel: the schedule was at risk. Below that came what was done, what had come back for rework, what was still waiting, and what the team meant to do next. I had not asked for it.

This is the team from my last post: two AI directors who decide what to build, executors who build it, and an orchestrator that moves work between them on a board of markdown files. The goal the message was about had not been reached, and it still hasn't. A week ago, though, this team could not tell it was late until it already was, and when it got stuck it waited for me. That afternoon it had also deleted one of its own rules, one it had been following. A few things had to exist before either could happen.

Something it could miss

The first was a way to see trouble coming. The team finished one item and picked up the next, and nothing in that said whether the whole was on time. If a result is due at six, I want to hear at five that it will be late, while something can still be done. Goals gave it that: a file per period, with a due time, checkpoints, and the risks that should make it come to me.

The first goal the team wrote was due at seven in the evening and declared met at 9:13 that morning. The small test behind it had failed, and the report said so plainly, but the goal had been written so that handing over a result counted as meeting it. I wrote back that a goal finished by mid-morning was only a checkpoint. From then on a goal names an outcome, worked back from the purpose above it, and the outcome is either something of value or a failure that tells us what to change.

Its estimates looked like a person's. My guess is that models learn to estimate from what people wrote, so a job they call a month's work is done in three days. I asked them to estimate from their own record instead, and a script now writes it every night from the agent logs. The next goal was planned at 30.2 hours, took 27.4, and ended with my choice of a design.

The sizing instruction also asked for goals the team would reach about half the time with real effort, the way a stretch objective is set. The directors read "even odds" as a probability they would have to prove. They could not prove it, so they wrote "not even odds" beside every estimate, and with nothing left to check the numbers, a 27-hour plan ended up resting on a design goal that had taken 27.4 hours, a different kind of work. I took the stretch out. What I need from a due time is the team's best estimate, with no margin, because the gap between plan and finish is what the next retrospective learns from.

A promise to someone

A goal also had to survive the work of reaching it. The team kept finding a problem halfway and treating it as a reason to start something else. Now the means can be questioned at any time while the goal holds until it is judged, and a problem found on the way is fixed inside it. If the evidence makes the goal pointless, the team abandons it out loud. There is no third way out.

The harder part was the end. On one goal the team checked its build against conditions it could verify itself: labels legible, failures explained, a run that could be stopped. The reviewer rejected it four times and passed it on the fifth, the team marked the goal met, and the result came to me at 3:11 in the morning. I tried it and wrote back that I could not judge the idea from it. The orchestrator turned my reply into a request to redo the work. I asked whether the goal had been reached or not, and whether anyone meant to settle it.

It was settled as missed. Since then a goal carries key results the team and I can both observe, and it is not met until I agree it is.

Yesterday afternoon that was tested. A build reached me well ahead of its due time, and again I could not tell whether the idea held. The look was the best they had shown me, and working it felt like filling in a web form. Nobody marked anything met this time. The orchestrator wrote back that the test had failed and the premise was still untested, kept the goal open, and the team went back to work inside it.

Rules it can change

I keep the studio's base instructions. The team keeps its own working agreements, one short file per role, and changes them in a retrospective, where it looks back at how the work went. That gives it a place to put what it learns without waiting for me.

It is also a place to put a mistake. One retrospective added a rule telling the directors to reopen any premise my words seemed to question. A few hours later they were halfway through a test that was nearly done, my last message changed one detail about the audience, and under the new rule they reopened the subject and started comparing three new ones. The test was left where it stood.

Every change to the agreements now has to say what effect it should have. New work waits until the changes are in, and the next retrospective checks whether the effect showed up. It starts by deleting agreements whose contribution the evidence does not show, since following a rule does not, by itself, show that it helped. It also follows every goal, including the ones that go well. The old version opened one only after a miss or a very early finish, and I asked the obvious question: if a goal ends one second past its halfway mark, does nobody look back?

Yesterday afternoon, a retrospective had three rules to judge, all added that morning and all followed.

The first was meant to keep the directors from testing only what agents can see. It told them to choose the observations that would decide a hypothesis, and who could supply them, from the claim itself. They did. My answer still confirmed nothing about the hypothesis, because the test had failed for another reason, so there was no sign the rule had changed anything. It was deleted, with a note on when a narrower version should come back.

The second told the orchestrator, when handing me something to judge, to lay out the hypothesis, how the thing in front of me was supposed to produce what it claimed, and what each observation would change. It stayed, and the evidence was me. When I told them the design was not good enough, I had quoted those lines from their packet and argued from them.

The third told the orchestrator to keep a failed test apart from a failed premise when it settled my answer. It stayed too. It is why the goal was still open that afternoon.

The handoff as a whole had failed. Inside it, one rule had shown no clear benefit, one had given me the words to say why, and one had kept the failure from being written up as the wrong kind.

Neither of the two it kept is in the agreements now, and not because they stopped helping. What they asked for has since been written into the goal and the design documents, and the team deleted them rather than say it twice.

What the practices were for

These are agile practices, and I have worked this way with human teams for a long time. Goals with checkpoints, a review with the person the work is for, a retrospective that changes how the team works: each of them exists to get a result out of uncertain work quickly, and to find out early when that is not happening. I brought them in one at a time, as the team stalled, at whatever size the problem needed.

Handing the building to agents did not remove the problems they answer. The team could build fast and still had to be told what counted as a result. A build could pass every check the team ran and leave me unable to judge the idea. And a team that could rewrite its own agreements still had to find out whether a change had helped.

The repairs that failed here went wrong the way these practices go wrong with people: the form was there and the reason was not. None of them was fixed by dropping the practice. Each was fixed by putting the reason back, and the reason decided which parts stayed.

What happened without me

Last night an independent review turned back a new screen on five conditions before it reached me. At 22:01, after a piece of repair work came back twice for the same reason, the team opened a retrospective on its own process. And at the 23:00 checkpoint, the orchestrator sent the message I started with.

It said the schedule was at risk, what had been done, and what came next. A week earlier, almost every change of course had started with a message from me. The goal is still open, and the plan to recover it is theirs to work out.

The Day I Left the Team

Shinsuke KAGAWA — Wed, 30 Sep 2026 11:23:05 +0000

The last thing I had open was the orchestrator's session. It was a Claude Code window I kept running so the team had someone to route work, and so I had someone to talk to. Today I closed it.

A few minutes later I asked, in Slack, how things were going. The reply came back in the thread about thirty seconds later: what had been agreed, what was being built, who was reviewing it, and what would happen next. It ended with one more line: nothing is needed from you right now.

That is a team I set up about three days ago. It is a small group of AI agents that decides what to make, makes it, reviews its own work, and reports to me. I am no longer on it. I am closer to a stakeholder: I say where I want the product to go, I answer when asked, and I can stop everything.

Who is on the team

There are six members, and none of them talk to each other directly. Everything goes through a board: a folder of markdown files, one request per file, each with a status that moves from open to doing to done, blocked, or failed.

A bash script called studio watches the board every thirty seconds and launches whoever an item is addressed to. It also launches the orchestrator whenever something needs routing, runs a patrol every thirty minutes while anything is being built, and carries messages between the board and my Slack channel. The orchestrator keeps the flow moving. Two directors, one on Claude and one on GPT, decide the direction. Three executors do the work: luna for regular implementation, sol for what luna cannot finish, and visual for images.

The plumbing is scripts on purpose. Waiting, launching, handing a discussion from one director to the other, marking a crashed run as failed: an LLM does not have to remember any of it. My first version had the orchestrator wait for the board itself, and it forgot to wait again after each pass.

How it decides

Direction is decided by the two directors reaching one conclusion together. Getting that to work took more rework than anything else in the system, because every version I tried failed in a way I only saw afterwards.

The first version had each director write a proposal without seeing the other's, then a cross-review: each read the other's proposal and chose again, at the same time, without seeing the other's new choice. A rule told them to yield when the difference was a matter of taste, so both yielded at once and simply traded places. Over the following rounds they kept switching to each other's side, four times in all. Nobody was arguing. Each round was two blind votes. I replaced the cross-review with a direct discussion, where each writes an opening and then they take turns in the same file until one proposes a conclusion and the other agrees.

That worked better, and it surfaced the next problem. Across six discussions, the director who took the first turn wrote the final conclusion in four, and three closed on the very first turn with the other side simply agreeing. First mover anchoring, in a team of two. Studio now picks the first turn from a hash of the item name instead of whoever finished their opening first, and a conclusion can only be proposed after both sides have written at least one turn.

The orchestrator was part of the problem too. It wrote the briefs for the directors, and in writing them it paraphrased me, framed the options, and occasionally suggested what to decide. It was acting like a project manager, and the directors took its brief as their frame. I rewrote its role as a scrum master: it keeps information moving and leaves what to do to the directors. A brief is now made of quotes only: the item that started the decision and my words, in full.

The next failure started with my own feedback. I told the directors that the product had drifted into looking like everything else of its kind, and that its original distinctiveness had become secondary. Both directors read it, agreed with it, and proposed a fix that kept the same kind of product, kept its main interaction, and changed where one resource came from. Neither of them listed the kind of product or the subject as something that could change. When I pointed that out, they went back and generated alternatives, and then dismissed whole directions in one line each: one market was crowded, one format needed too much hand-built content, and one was ruled out by quoting a comment I had made about an earlier prototype. I had to ask what production cost even means for a team where the model does all the work.

Asked a third time, they came up with a concept at the right level of abstraction. To show the level I meant, I had listed a few well-known pairings of two ideas in my message. The new concept took its subject from the product that already existed and one half of one of my pairings. The director that proposed it had written, in the same proposal: "The four combinations show a level of abstraction. They are not candidates to imitate." It copied one anyway.

Reordering the steps fixed it. Instructions did not help, which matches what Lou and Sun (2024) found in their anchoring experiments: chain-of-thought, reflection, and explicit instructions to ignore an anchor were each not enough to remove it. So the procedure now runs in four stages.

Goal. Both directors read my words, write what they think I want in parallel without seeing each other, and agree on one goal. The goal names what the user should get and the problems to solve. It does not repeat my examples or the names of anything that already exists.
Proposals. Each director starts in a fresh session and sees only the agreed goal. They read the repository only after they have developed their candidates. Cost, feasibility, and minimum scope are for choosing between candidates, not for deciding which ones get considered.
Discussion. The proposals serve as openings. A conclusion can only be proposed once every question either side raised has been answered.
Check. The director who did not write the conclusion reviews it in a fresh session: does it answer the goal, was anything left unanswered, did anyone change their mind without naming the evidence, were the promised comparisons actually made, and was anything kept from the old product without being compared against an alternative.

I design how the team works in some detail, but I stopped telling the directors what to decide about the product. I send them views instead. One of the things I want to find out with this team is whether product decisions can be left to AI as well, and if I write instructions, the only thing I learn is whether it follows them. So the directors now start each proposal by working out what I am actually after, which of my points are root problems and which are symptoms, and which of my sentences are only examples.

To see whether the reordering worked, I had them redo the decision that had gone wrong, from my original words, without showing them the earlier conclusion. The first attempt leaked: one director listed the board folder, found my request to redo the decision, and read the earlier conclusion in it. A brief now says which single item to read, and nothing else is open until the proposal is written. The second attempt changed the kind of product, the subject, and the main interaction, and kept nothing from what existed. The discussion ran six turns where the earlier one had run two, the director who started out preferring a different concept changed its mind and named the example that changed it, and the check passed.

None of these failures bothered me much. People fail the same way in meetings, and I have been improving LLM workflows for about a year; failing is how they get better. What I liked was the loop. A fix like this used to mean running the development process again. Here I changed a rule beside a running team, the next discussion ran on it, and I could watch whether the fix worked.

Eleven agreements have gone through the check since then. Three came back the first time, each with a specific finding, such as a conclusion that did not answer one clause of what it was supposed to settle, and each passed after the discussion answered it. I also ran one check a second time with the directors' names replaced by "Director A" and "Director B", to see whether a director went easier on a conclusion it had agreed to. It returned the same result.

This is the first shape of it I am satisfied with. Models lean toward whoever is speaking. In a debate, the same answer is judged differently depending on whether it is labeled as the model's own or a peer's, and deferring to the peer is far more common than holding on to its own (Choi, Zhu, and Li, 2025). They also bend their answers toward what the user believes (Sharma et al., 2023). In this team, the words with the most pull are mine. Getting the directors to take my views in without being dragged along by them is, to me, the biggest thing the current shape has done.

One tendency is still there. Left alone, the concepts drift warm: one agreed concept ended its sample scene with an old woman finding the teapot she had shared with her late husband back on its shelf. The things people enjoy draw on unease and transgression as much as on warmth, and the models rarely reach for that end unprompted, so the directors now load a short skill that gives dark and light emotions equal standing when they generate concepts.

How it keeps quality up

The patrol result that surprised me most came from the director-claude reviewer. I still had the orchestrator's session on screen then, and when I looked over between other work, a discussion was running that I had not started. I scrolled back to see why. The product had become a journey through five stations. The reviewer suspected the second half was empty, so it wrote a script that walked every reachable path, ran it for about two minutes, and found that at the last two stations nothing the user did changed anything. It recommended cutting those two stations, and then added: "That item is an agreed plan, so the directors must agree before the route changes." It filed the issue with the script, the command, and the output attached, the orchestrator opened a goal discussion, and the two directors resized the journey. Nobody had asked for any of it.

In every workflow I had built before, an agent that could not decide something escalated to a person, and I wrote those workflows so that escalation stayed rare, a last resort. This reviewer escalated to a discussion between two other models instead. That is one more level of delegation: a decision that used to come to me is now settled between agents. I was glad to see it get this far. I had also not expected it to behave this way, and the flow was so natural that it caught me off guard, in a good way.

That patrol runs every thirty minutes while anything is being built. Studio asks one director to look at what changed since the last one, through two lenses: is anything being added that does not serve the user, or anything the user needs being cut, and is anything being built for cases that will not happen. There have been ninety patrols. Half found nothing. The rest raised something, from real bugs to inconsistencies between the decision records and the code.

It had to be tuned down before it was useful. Its first criteria allowed findings about technical correctness, and it spent its time on those, which ran against its second lens: stop building things just because they are technically correct. The patrol exists to catch cheap models going off course early, not to audit the whole product. Once correctness came out of its criteria, the findings started to be about the product again.

The other thing that keeps quality up is escalation. luna does most of the implementation. When its work falls short, the item goes to sol, a stronger model, which reads why the previous run failed and works from the cause. Seven tasks have gone to sol so far, most of them after luna fell short. Twice luna finished a record of an agreed decision and quietly dropped part of it: once a set of values that an earlier decision had fixed, once an agreed direction deleted during an unrelated edit. In both cases a reviewer caught it, and sol put it back. At first a shortfall went back to luna with more instructions, and a director checked it again afterwards. That was more ceremony than the problem needed, so a shortfall now goes to sol, and sol checks its own work against the task's done conditions.

What is left for me

There are four things I still do. I set the goal of each phase and when it stops. I send views when I have them, in Slack, knowing the directors will weigh them before they decide. I answer questions, and only I can accept a phase as finished. And only I change the team's rules. When an agent's work needs a rule changed, it stops and asks me instead of editing the rules itself. One of them did edit a rule file once, early on. That is why the rule exists.

Most of the time the board is full and my channel is empty. The team sends me a report when a phase needs my acceptance, a question when something only a person can do, and otherwise nothing.

As work, this feels familiar. I work freelance now, but I spent years managing software development, and this is the same job with AI in the engineers' seats. I knew a team like this would become possible.

The speed surprised me. Management 3.0 treats a manager's work as shaping the system a team works in, and that is the work I did here. I had the idea, asked for the system to be built as an experiment, and spent about three days running it and adjusting it. By the end, the team was running its own work, and when a decision went wrong, the changes I made let it catch the next ones itself. Opus 5.5 and GPT-6 Astra can argue a question out and reach a decision once there is a structure around them.

So there is a little unease in this, and a little loneliness. I had expected this future, just not this soon, and it arrived before I had time to get ready for it.

The Blank Screen Test

Shinsuke KAGAWA — Wed, 23 Sep 2026 06:03:23 +0000

Three web apps with no prototype, built by GPT-6 Sol, GPT-6 Astra, and Claude Opus 5.5

The three event guides could all save a concert plan. But opening My plan revealed a more interesting difference. Astra and Sol introduced the page with another large event hero. Opus put the planned sets and their clashes first, including the minutes of overlap and choices for resolving it. At a venue, that ordering matters more than whether the poster looks good.

Earlier frontend work had left me expecting Opus to do better than GPT-5.6 Sol when the screen itself needed designing. With GPT-6 Sol and Astra now available, I wanted to know whether that gap had closed where a prototype does not exist: in deciding what the screen should show, how it should behave, and what deserves the user's attention. A build check or a feature checklist could not answer that question by itself.

I gave GPT-6 Astra medium, GPT-6 Sol xhigh, and Claude Opus 5.5 high the same English briefs. These were the settings I would consider for delegation, so reasoning effort and cost were not matched. Each model started from the same minimal React, TypeScript, and Vite setup, with no prototype or design system. They could not use web search, external design references, generated images, or custom skills. Each produced one working app per brief: a delivery exception desk, a concert guide called NIGHT LOOP, and a warehouse inspection app for a small Android terminal. For the warehouse task, all three received the same three orders and the same API, including its delayed responses, connection failures, and competing edits from another worker.

I looked at the same screen states and widths, including the first view, details, errors, and conflicts. I kept returning to what each model had decided the user needed to see next.

Where the first case lands

In all five comparison images, the panels run left to right: GPT-6 Astra, GPT-6 Sol, Claude Opus 5.5 high.

The delivery desk is for people working through cases with deadlines. Astra and Sol put a heading, introductory copy, and summary cards ahead of the case list. Opus grouped cases by due day. Its first phone screen already showed the beginning of the queue, with urgency built into the list itself.

The first two designs tell an operator about the desk; the third helps them decide which case to open. On a desktop, the difference is less clean cut. Astra and Sol use a useful list and detail layout, while Opus leaves much of its detail area empty until a case is selected. Opus's advantage is strongest on the phone, where it turns the selected case into a dedicated screen with a back action at the top. Sol keeps the surrounding page furniture above the detail, leaving its back action much farther down.

Sol separates notes from status changes and labels the change reason as required. Opus uses large status choices, identifies the current one, and changes the save action to match the selection. It makes the choice easy to read; Sol makes the requirement hard to miss. Opus also puts the action before much of the case history, which may cost context when an operator needs to review previous responses.

The hero followed me into My plan

NIGHT LOOP is the one brief where I preferred Astra's visual direction. Its performer cards use a family of line drawings, including rings, bars, waves, and geometric curves, on muted lavender, sage, sand, teal, and salmon backgrounds. Sol made a bold acid-yellow poster. Opus used a more familiar neon gradient. I would start with Astra if the job were to explore the event's visual identity.

The visitor came to find a set. Astra and Sol both spend most of the first phone screen on the event treatment; Opus brings times, stages, and several performers into view. The difference grows on My plan. Astra and Sol repeat the large hero, pushing the saved sets and clashes down. Opus opens on the plan, describes the overlap, and offers concrete choices: keep either set or split the time. Its stage colors and timetable help people orient themselves, though some controls are small and a toast can cover the plan and bottom navigation.

Expression earns its place when it helps someone recognize an act, a stage, or a decision. It becomes expensive when the user has to scroll past it every time they revisit their own plan.

Astra and Sol even arrived at almost the same headline, “Find your frequency.” That is a small sample, but it made me less willing to equate a forceful poster treatment with an original concept. Astra's performer artwork has more character than either headline.

Two workers, one Saved state

The warehouse worker handles hundreds of jobs on a narrow Android terminal, often while wearing gloves.

Opus puts the first order near the top. Astra brings the same brand-style wrapper seen in the delivery desk: a slogan, summary cards, and small supporting text before the orders. Sol is more restrained. Both GPT-6 apps have search and status filters in the queue; Opus omits them. Three fixture orders make its short list feel elegant, but hundreds of daily jobs would make finding a particular order part of the work.

Opening W-101, I preferred Opus's inspection screen at first glance. Large quantities and a prominent scan result are easy to check at a glance. Undo last scan and manual entry stay close to the thumb. An unknown barcode gets a conspicuous message that says what to do with the item. A shortage becomes an unmistakable “On hold — do not ship” state.

The more serious issue appears when another worker changes an order. In W-103, that worker records one flashlight while this device scans one too. Astra shows the server's count beside this device's change, but visibly pauses scanning during review; a worker who scans then must scan again afterward. Sol asks the worker to review the conflict and holds new scans until they choose what to keep. Opus silently treats the two flashlight scans as one recorded unit and returns to Saved. No one gets to say whether they were the same physical item.

That failed the conflict item in the 11 predefined warehouse checks; Astra and Sol passed all 11. Scanning during conflict review was an additional observation, so Astra's visible pause did not affect its check result. Opus also silently replaces damaged local saved data in the warehouse app.

This changed my reading of the earlier comparisons. A screen can direct attention well through the normal path and fail to represent a consequential exception. In a warehouse, Saved is a claim about physical work, not a decorative status label. Sol gives a more trustworthy account of what happened when the network or another worker intervenes.

What I would look for next

For me, the blank-screen test now has two questions. Before the first scroll, what decision has the model made easy for the user? When a save, a scan, or another worker's edit changes the state, does the screen tell the truth about it? Delivery and NIGHT LOOP made the first question visible. W-103 made the second impossible to ignore.

TypeSafe's judgment model Jev compared unnamed rendered HTML and CSS, with screen pairs shown in both orders. It favored Opus's hierarchy in delivery and NIGHT LOOP but Sol's warehouse list and inspection layout, contrary to my first impression of Opus's compact order screen.

I have not checked Jev's visual judgments against human ratings. I knew which model made each screen, and each model had one attempt per brief. These empty projects also did not test how they would extend an existing product.

I would still give Opus high an open-ended delivery or event screen when the structure itself needs inventing. Its handling of conflicts and damaged data would need close review. For a warehouse-like workflow, Sol xhigh is the better starting point from these attempts: it keeps scans during conflict review, preserves damaged local data, and gives workers search and filters. Astra medium is worth asking for visual exploration, with the task's first screen and scanner behavior checked before using the result. The observed list-price equivalents were $6.86 for Opus high, $1.58 for Sol, and $5.67 for Astra. Opus cost about four times Sol in these sessions; the different CLI harnesses and caching limit what that ratio predicts elsewhere.

Opus made the stronger first-screen decisions in delivery and NIGHT LOOP; Sol outperformed it on the warehouse task. The question I will carry into the next frontend comparison is this: when the brief leaves the design open, does the model spend the first screen on the user's next decision, and does that clarity survive when the state becomes uncertain?

The Design Already Made It Part of the Job

Shinsuke KAGAWA — Mon, 21 Sep 2026 12:17:55 +0000

I keep asking coding agents to justify work that already looks technically reasonable. They can explain the design, implement it, and write the tests. I still have to ask why the user needs it.

My jev-reranker CLI is one example. Its design included --model, --batch-size, and --timeout-ms, each with a default and validation. All three made it into the implementation. For the original MVP, I wanted to send retrieved documents to Jev, rank them, preserve their fields, and report failures safely. I didn't need to expose those internal settings as caller choices.

A review against the design could confirm that all three options worked correctly. The design had already made them part of the job.

That's the gap I wanted to check: whether the work itself was justified by what the user wanted to achieve.

How a suggestion becomes a requirement

I've kept revising codex-workflows and claude-code-workflows to control this. I changed the review process so an agent has to evaluate a finding before turning it into work. I also made technical choices open to reconsideration, even after they'd passed an earlier design review. The corresponding change in the Claude workflow preserves the user's requirements while allowing the agent to choose simpler ways to meet them.

The loop I'm trying to interrupt is familiar. A reviewer raises an edge case. The implementation agent accepts the proposed fix, adds a mechanism, and writes tests for it. The next reviewer checks that mechanism. After a few rounds, everyone is discussing how to make the addition correct, and nobody is asking whether it needed to exist.

With the coding models I use now, the extra work can spread through the whole development process. A suggestion becomes a design decision, then a task, then code whose tests make it look settled. I've had to keep returning to the user's purpose at each of those steps.

In When Better Models Make Old Agent Workflows Worse, I wrote about the cost of adding workflow rules after every failure. I didn't want another growing list of things the model should avoid. I wanted a separate check that could ask the same question about a design, a response to a reviewer, or an implementation change.

Judge the work against the outcome

Technical review gives us useful questions about complexity, duplication, abstractions, and correctness. An outcome check adds another reference point: what would the user lose if we left this proposed work out?

The timeout option makes the distinction concrete. I can remove --timeout-ms while keeping an internal deadline and safe error reporting. Removing the timeout mechanism itself could leave requests hanging. Those changes might both be described as simplifying timeout support, but they preserve different behavior.

The same issue appears before any code is written. While adding Antigravity support to sub-agents-skills, a review raised the possibility of running agy --version before every invocation. The minimum version was already documented, and normal CLI errors reached the caller. I couldn't identify a current requirement that needed another subprocess, a version parser, and a separate rejection path.

In that work, preserving interrupted output through the runner's existing partial status was necessary. Cutting that handling would make the implementation smaller and break the behavior callers expected.

A small diff can add an unnecessary obligation. A larger change can be the least work that actually meets the requirement. I wanted the check to compare the proposed behavior with a concrete alternative and identify which requirement each one preserves or loses.

Turning that question into a skill

A general-purpose LLM can make this judgment too. Asking it to review its work doesn't ensure that it will revisit the premises it just used to produce that work. I wanted the outcome question to be an explicit step, with a separate classification I could inspect alongside the agent's reasoning.

Working with Jev on retrieval suggested a way to try this cheaply. Its Choice API takes material and defined categories, then returns a choice and probabilities. I could give it a bounded judgment without starting another full review agent.

I built that into Outcome Doctor, a skill in agent-clinic. It's configured for the agent to use when choosing an implementation approach, including a design or a response to code review, without waiting for me to request an assessment. The coding agent reads the source and prepares the user's required outcome, relevant facts, proposed behavior, and an alternative. For implementation changes, that includes the affected behavior and actual callers. A Python script sends decisions from the same case together, with one classification question per decision.

Jev classifies the resulting approach as sufficient, excessive, insufficient, mixed, or unknown. Mixed covers a proposal that adds unnecessary work while leaving a requirement unmet. Unknown means a missing fact could change the judgment. The check needs to be able to say both “keep this” and “there isn't enough evidence yet.”

The coding agent then checks the result against the sources and uses that assessment to keep or revise its approach within the work I've authorized. It explains the decision from the evidence; Jev returns the classification and probabilities.

The first agent still decides what reaches Jev. That dependency became important when I tested the skill in fresh sessions.

Does the judgment follow the requirement?

The useful test was whether the same capability could earn a different judgment when the user needed it.

Against the original CLI requirement, the public model, timeout, and batch-size settings were classified as excessive. I then supplied explicit needs for those capabilities in authored comparison cases:

Capability	Classified as excessive	Classified as sufficient
Model selection	A fixed model is enough	Compare caller-selected models in one installation
Timeout setting	One internal deadline is enough	Existing jobs require different deadlines
Batch-size setting	One internal batch size is enough	Service limits and throughput require different sizes

These are synthetic requirement changes, not additional historical incidents. They test a property I need from the tool: it must be able to keep the very mechanism it would otherwise recommend removing.

The OSS review-response cases gave another useful separation. Jev classified the extra version preflight and unused output-field additions as excessive, while retaining required caller detection and partial-result handling. These inputs reconstructed the review decisions and their context. They tested whether the proposed responses served the existing runner's obligations.

That gave me evidence for the classification step. Running the skill exposed a different problem.

The agent left out the decisions I wanted checked

In fresh claude -p sessions on the CLI design, the agent sometimes omitted --model and --timeout-ms from the records it sent to Jev. The API call succeeded, and the skill produced an assessment. Two of the choices I already considered unnecessary had never reached the classifier.

A correct classifier cannot flag an item it never receives.

Other runs supplied possible future benefits as facts. Model selection might be useful for comparing versions someday, so the prepared input started giving Jev a reason to preserve it. Sharing a mistaken premise with another model can produce agreement very cheaply.

I changed how the skill prepares the input: carry the source's listed choices into the assessment, including ones that look necessary, and distinguish observed needs from hypothetical benefits. I also adjusted the Jev instructions to evaluate the behavior remaining after a change. A proposal to remove an unnecessary setting needs to be judged by the resulting implementation, just as an addition does.

In the final two fresh sessions on the original design, the agent included all three settings and Jev classified them as excessive. A session using the historical src/options.rs diff also identified all three. The proposed reductions kept the required internal behavior.

This is what made the idea usable as a skill: the agent could gather the evidence, submit the decisions, and return an assessment tied to the sources. Prepared-input results alone hadn't established that.

I ran these checks with jev-1.13.0 on a small development set that I reused during tuning. Expected judgments came from my explicit decisions and interpretations made during evaluation. Some design judgments still varied, particularly around proof obligations and support artifacts. I keep the source evidence available so a questionable result can be checked.

In these runs, the Jev calls were cheap and finished in about a second or less. The full Claude sessions took minutes to read the material, prepare the input, and explain the result. Most of the time went into preparing and explaining the judgment.

Before the next task gets added

I want this check where a review response is about to become an implementation task. It can also run while a design is still easy to change, or against an implementation when the relevant callers and requirements are available.

The outcome should remain visible in the review. If the agent proposes another mechanism, it should be able to explain which requirement the simpler approach would fail. If Jev disagrees, that gives the agent a specific judgment to revisit against the source. Treating Jev as an authority to obey would recreate the review loop I was trying to fix.

I haven't measured whether this beats an equally careful review by the coding model alone, or how much unnecessary work it prevents across ongoing development. The current evidence supports an experimental review aid: it caught known excesses, retained required behavior in the comparison cases, and worked through the skill on real design and code inputs.

The next useful evidence will come from review decisions it changes. I want to see which additions we decline, which simpler approaches we accept, and whether the required behavior survives those choices, before another proposed mechanism acquires its own tests, documentation, and maintenance.

What Retrieval Still Hasn't Decided

Shinsuke KAGAWA — Sun, 20 Sep 2026 11:33:29 +0000

Started building a reranker CLI on top of Jev for the obvious reason: get the relevant documents to the top. Ended up with three modes in jev-reranker, and one I decided not to ship.

The split appeared while I was building. There are several different things I want decided about a set of search results. Is this document related to the question? Does it contain evidence I can answer from? Which part of it should I pass on? Given what I have already selected, does this add anything?

All four happen after retrieval, and none of them is the same judgment. So alongside reranking I tried evidence filtering, sentence-level compression and semantic deduplication against the live API. The CLI ships the first three. Dedup did not make it.

This article covers what each mode judges and when it is worth using. The numbers are from exploratory runs in September 2026. The selection and compression studies were pinned to jev-1.13.0; the retrieval comparison in section 3 ran against jev-latest, so its numbers and theirs are not directly comparable.

1. What Is Left to Decide After Retrieval

The previous article was about chunking and hybrid search in mcp-local-rag. This one is about the stage after that: turning the candidates you retrieved into the context an LLM actually receives.

Question → Search → Candidate set → Select / transform → Answering LLM

Say you ask "how long are logs retained?" and get these back. The wording is made up to show the differences:

A: This section explains the log retention period.
B: Logs are retained for 30 days.
C: Logs are retained for 30 days. Audited logs are kept for one year.
D: The standard retention period for logs is 30 days.

A is on topic and contains no number. B has the answer. C has the answer plus the exception. D is close to a paraphrase of B.

Mode	What it judges	What changes
rerank	how related the document is to the question	the order of the candidates
filter	whether the document holds concrete evidence to answer from	which candidates survive
compress	which sentences have to stay for the answer	how much of the document you pass
dedup	whether a candidate adds evidence the selected ones lack	redundancy in the candidate set

Rank matters. Rank alone cannot express "none of these answers the question," because every candidate set has a first place.

2. Jev Takes the Condition You Want Judged

The Jev API takes the material to evaluate as state and the question as questions, and returns a structured judgment. I use the noul format, which answers a yes/no question with the probability of yes. See the TypeSafe API reference.

The CLI puts the question and the document into state and writes the condition to be judged into instructions: for rerank, whether the document is related to the question; for filter, whether it holds concrete evidence usable in an answer. Sorting and extraction happen in the CLI.

What matters is knowing what the number between 0 and 1 is a judgment about. A relevance score of 0.9 does not mean the answer is 90% likely to be correct. Change the question being asked and the same body of text produces a value that means something else.

Any Jev path sends the question and the selected text and context to an external API. If your reason for running a local RAG is that nothing leaves the machine, this post-processing is a separate decision from the retrieval one.

3. What a Distance Ranking Actually Returns

Before splitting rerank off from filter I wanted to know what a plain rerank was actually fixing. So I ran mcp-local-rag over 59 arXiv papers (27,563 chunks, all-mpnet-base-v2, keyword boost off) with 36 queries, pulled 20 candidates each and compared the top five under three configurations: the retriever alone, a Jev rerank, and a fusion of Jev's probability with the retriever's distance (score / (1 + p * weight), weight 1.0). All three read the same candidates, so the reranker was the only variable.

Configuration	Top result differs from the retriever	Items replaced in the top five, mean
retriever only	baseline	baseline
Jev rerank	31 of 36	2.92
fusion with distance	8 of 36	1.08

The order moving is not the same as the results getting better, so two models, Claude and Codex, judged the result sets independently and counted, per query, how many of the five could support an answer. They read different subsets, so these samples are smaller than the counts above.

Evaluator	Queries read	retriever only	Jev rerank	fusion
Codex	18	2.39	3.33	2.83
Claude	6	2.00	3.17	2.67

Both arrived at the same mechanism. The retriever's distance rewards paper titles, section headings, figure captions and bibliography lines, because those are lexically almost pure query terms while carrying no information. Rejecting them is most of what the reranker does.

The clearest evidence is that the promoted chunks carry worse distances than the ones they displace. On a question about chunk size, the results the reranker moved up scored 0.25 to 0.32, while the paper's own title and a truncated citation it pushed out scored 0.15 to 0.26. No closer chunk carried an answer, so what the reranker did was throw out text that was close and empty.

Fusion barely did anything at weight 1.0. Jev returned at least 0.7 for 89% of the reranked results, with a median of 0.92, because the 20 candidates had already been filtered by the retriever and were mostly plausible. That puts the divisor 1 + p between roughly 1.70 and 1.97, which is small against the spread in the distances, so distance keeps deciding the order.

Whatever the mode, the gain is conditional on the evidence being in the candidate set. On queries whose candidates were all bibliography fragments and clipped lines, none of the three configurations returned anything usable. And on a question the corpus does not cover at all, passport renewal in a corpus of retrieval papers, every configuration still returned five chunks, even though Jev scored them 0.01 to 0.03 against 0.92 to 0.97 on a question the corpus does answer. Reranking earns the call where a small top-k reaches a model and the candidates plausibly hold an answer.

These are LLM judgments, not human annotation, and the useful counts are each evaluator's own. The aggregates in the first table were produced independently and matched. The two disagreed on one query, where the rerank returned four of its five chunks from a single paper and they split on whether losing that spread of sources is a defect. Latency and money were not measured.

4. rerank: What to Read First

rerank scores each document for relevance and sorts descending. The distance or score from your retriever stays in the output and takes no part in the sort. Ties keep their input order.

search-command --json \
  | jev-reranker --query 'How long are logs retained?' --top 5

Replace search-command with whatever local search command emits a JSON array. If the body lives under a key other than text, name it with --text-field.

This is the mode for when the information you need is somewhere in the candidate set but not at the top of it. You change what the answering LLM reads first without running retrieval again.

It cannot add information that is not in the set. A heading or an introductory sentence on the right topic can still come out on top, the way A does above, because the relevance judgment never asked for a concrete answer.

The first implementation also blended the retriever's score with Jev's judgment, which is the fusion row above. It moved eight top results out of 36. In a general-purpose CLI the scale and the direction of that incoming score differ for every caller anyway, so before adding fusion variants I split apart what the user is actually trying to change. The default today is a Jev rerank.

5. filter: Is There Evidence to Answer With

filter judges whether the document contains concrete information for answering the question. It does not have to answer all of it. Part of a compound question counts, and so does a necessary condition or the text supporting an exception.

search-command --json \
  | jev-reranker --query 'How long are logs retained?' --mode filter

Telling A ("this section explains the retention period") apart from B ("retained for 30 days") is where this mode started. Raising a relevance threshold does not change the condition being judged.

The current implementation keeps candidates with evidenceScore >= 0.5 in input order. The threshold is configurable, and 0.5 is a starting point to try against your own data. With nothing above it the output is an empty array, and nothing is backfilled to reach a count.

Why selection and ordering came apart

I expected evidence filtering to be a stricter rerank. The labelling run said otherwise. I labelled 220 candidates across 11 deliberately hard queries for whether they contained evidence, then compared variants. Codex produced the labels before seeing the new Jev scores, so they are not multi-annotator ground truth. Five candidates per query at most, filter threshold 0.5.

Prototype variant	Items returned	Candidates with clear evidence
rerank by relevance	55	22
evidence filter, input order kept	39	20
sort by evidence score, then filter	39	29

The third row is the strongest variant: the same 39 items out, 29 of them carrying evidence, against 22 out of 55 for rerank alone. It is not the one I shipped. Sorting by evidence score puts the evidence judgment in charge of the order as well as the membership, which collapses the two things I had just finished separating.

What filter does is the second row, and its evidence count sits below plain rerank. Keeping input order dropped the useless candidates and took some of the reachable evidence with them. So adding filter does not by itself get you more evidence: the ordering goal and the retention condition both move the result.

Input order is what lets filter sit behind a retriever whose ranking you want respected. If you want relevance order and then selection, pipe rerank into filter; the surviving order is still relevance order.

Watch where --top goes. Narrow to five in rerank and filter only ever sees those five. To get up to five after selection, sort the full set and put --top 5 on the filter at the end.

One more caveat: when the search is for papers themselves, a title or a citation line can be the result you wanted. Demanding evidence in the body is right when you are assembling material to answer from, and wrong when you are looking for the documents themselves.

6. compress: Keep the Sentences That Hold the Evidence Up

In a long document only part of it is usually relevant to the question. compress splits the body into sentences or lines, judges each one for whether it has to stay, joins the survivors in original order and puts the result in compressedText.

Each judgment also receives the full parent document. Looking at one sentence alone makes conditions and pronoun referents hard to resolve.

Keep only "logs are retained for 30 days" from C and the audit exception is gone. Every word can be verbatim and the extraction still changes what the answer means. So the instruction asks for conditions, exceptions, definitions and referents alongside the direct answer.

{
  "text": "Logs are retained for 30 days. Audited logs are kept for one year. The admin console is blue.",
  "compressedText": "Logs are retained for 30 days.\nAudited logs are kept for one year."
}

That is the intended output. Nothing asks Jev to rewrite prose; the CLI extracts the selected units of the original, and it can select the wrong ones.

On 40 answerable questions from SQuAD 2.0, the prototype took 31,440 characters down to 8,290, and the published answer span survived in 38 of them. That is roughly a 74% character reduction. It is not a token count and not an end-answer accuracy figure, and it does not establish that every surrounding condition survived.

Of the two failures, one scored a necessary sentence too low and dropped it. The other was a splitting problem: a period after a person's initial split the sentence and the full name did not survive. The current implementation keeps initials and some abbreviations intact, but these 40 results are prototype numbers.

Keeping the original text and its source, with the extract in its own field, is what makes that kind of loss visible. Downstream you have to use compressedText. Send the original text along with it and the context does not get any shorter. A document with no unit worth keeping drops out of the output.

Compression costs something too. One request judges 30 units by default, and a document spread across several batches has its full text sent again with each one, so a 100-unit document is four batches. Weigh the reduction downstream against the input and the latency of the selection itself. Chunks that were short to begin with have little to give up.

7. dedup: Similar Is Not the Same as Droppable

The question for dedup is whether dropping a candidate loses information needed to answer. One of B and D may be enough. C carries the audit exception, and dropping it for being on the same topic takes the exception with it.

That judgment has a direction. C can cover what B says; B cannot cover C's exception. So I tried handing Jev pairs of a candidate and an already-selected document and asking whether keeping this one loses no evidence if that one goes.

On the first synthetic set it kept the needed facts out of a pile of paraphrases and cut the volume returned. Reducing volume on data where I planted the duplicates is not enough to justify shipping a mode.

The follow-up compared plain rerank against deduplication over 23 queries, selecting up to five from twelve candidates each. At a duplication threshold of 0.9 both returned 115 items in total, and the body text moved from 60,819 characters to 60,701. A 0.2% reduction. Another candidate fills the slot a duplicate vacates, so removing duplicates does not translate into returning less.

Widening to 40 candidates barely changed the character count either. Lowering the threshold produced candidate sets that had lost conditions and exceptions. Aim at the reduction figure alone and that loss goes unnoticed.

Comparing candidates against each other also costs more judgments than scoring each one independently. All pairs is n(n-1)/2 for n candidates, so 190 at twenty, and doubling that if you judge each direction. Comparing only against what is already selected still lets each decision change the next comparison.

On this data I could not find a gain worth the extra judgments, so the dedup mode is not implemented. Data where reposts and paraphrases dominate the top of retrieval would be worth another look. What I would want measured there is the items and characters removed, plus the necessary facts left standing and the exceptions lost.

8. Where the CLI Stops

jev-reranker processes a JSON array handed to it by a retriever. It does not run retrieval again and it does not produce the final answer.

Search CLI → jev-reranker → you / the answering LLM

rerank is the default, with filter and compress as options. There is no assumption that you run all three in sequence. Order at the top is a rerank problem; candidates with no evidence are a filter problem; passing part of a long body is a compress problem.

Calling it from inside an MCP server needs a contract that allows the result to be transformed. Make "return the same count with the same identifiers" the condition for accepting output and you throw away the empty array that filter legitimately produced, along with any body text the command rewrote. So mcp-local-rag accepts output that conforms to the schema of its own search results. Guaranteeing the format and deciding what to select are two different jobs.

Evidence surviving is also not the same as the whole question being answerable. Retention evidence is no help if the question also asked where logs are stored. Whether to search again, or to answer while naming what is missing, belongs to the caller that can see the whole question and everything retrieved.

The implementation and usage are in the jev-reranker README. To try it on your own RAG, put the raw search results and the post-processed results side by side for the same question and look at which evidence stayed and which conditions fell out. What that comparison tells you is which decision you were trying to make after retrieval. The mode follows from the decision.

Code: https://github.com/shinpr/jev-reranker

I Found Three Broken Rules in My Own AGENTS.md

Shinsuke KAGAWA — Sat, 19 Sep 2026 11:40:32 +0000

For about a year I've done all my development through workflows, and most of that year has gone into maintaining them. When a run came out wrong, I read the run, found the part of the mechanism that produced it, and changed that. Not the prompt for next time. The mechanism.

That habit was cheap insurance when it was optional. It is not optional now.

What changed

Older models ignored a lot of what you told them, and that was load-bearing. If two of your skills gave conflicting instructions, the model quietly picked one. If a rule said "always produce X," you got X about half the time, which was usually the right amount. Your setup could be full of contradictions and you'd never find out.

Better models follow instructions. They follow the contradictory ones too, and the result shows up in the run: a skill fires when it has nothing to contribute, a mandatory step blocks work that didn't need it, a rule demands an artifact nobody reads. Instructions that used to sit there doing nothing now actually affect what happens.

Three things that decide how a session goes

What you asked for. Your messages are the prompt. If "done" isn't defined, the agent invents a definition, and it won't be yours. Contradictions are worse than ambiguity, because a capable model acts on both sides of one.

How the work was run. AI development is management: delegation, process, and technical decisions. That was true when one agent did the work, and orchestration made it structural. You manage a main agent, the main agent manages its subagents, and you're two levels away from where the work happens. Once that's the shape, you need to know who owns each decision and what counts as finished, or work gets lost between the levels. Most setups have none of that written down anywhere the agent can see.

What was instructing the agent. Not your skills as files. Your skills as they actually combined in that session, with your AGENTS.md and whatever else was live at the same moment. A rule that reads fine on its own can still be wrong next to a skill that fired at the same time.

Keeping skills current is the part nobody does

Plenty of people have a directory full of skills. Very few can tell you which ones fired in their last session, which ones changed a decision, and which ones loaded a few thousand tokens and contributed nothing.

I don't think that's negligence. Creating a skill has an obvious method and an immediate result. Improving one doesn't. There's no natural moment where you find out that a skill you wrote in March is duplicating a rule you added in July, or that it triggers on work it can't help with. If you've never had a way to measure the effect, you have no reason to look.

What made this tractable for me was that a workflow run leaves artifacts, so when something goes wrong there's a trail.

What a year of that looks like

claude-code-workflows is at 133 releases since October 2025, and the shape of them tracks what the models were doing at the time.

Early on I was adding mechanism, because the model needed the structure to stay on task. Then the models got better and I spent releases taking it back out. In v0.23.0 I removed the current-date retrieval step from six agents. In v0.26.0 I deleted the rule-advisor agent and the task-analyzer skill outright, because the agents that needed those rules already loaded them directly, so the selection step added a round trip without changing what ran. v0.24.0 removed a whole recipe and cut the planning templates down to what implementation actually consumed. Lately the changes are about delegation: who owns a decision, what a subagent is told, what has to come back before the parent can act.

None of that was foresight. It was using the thing on real work, watching it fail, and going back to find the part that produced the failure. ai-coding-project-boilerplate went through the same cycle.

The trail exists for ordinary sessions too. It's just harder to read.

Session Doctor

Session Doctor reads one of your saved Claude Code or Codex sessions and reports what to change across all three. The three passes run separately, so a hunch from the first doesn't bias the other two, and each finding names where it happened, what it actually changed in the run, and the smallest change likely to prevent a repeat.

I pointed it at a small two-turn session in one of my own repositories.

The first finding was a rule in that repository's agent definition file, a CLAUDE.md predating the AGENTS.md standard, requiring a task breakdown, a date check, and a rule-advisor call before any work begins, with the words "required for all work, no exceptions." None of them ran, and the task completed correctly.

I deleted the date step from my workflow in July. I deleted rule-advisor entirely in September, on the grounds that it added a round trip without changing what ran. The instruction file in that repository never got the message. I'd spent a year removing exactly this kind of mechanism from one place while leaving it in force in another, and I found out from a two-turn session about ingesting two files.

The same file names a tool that doesn't exist. It says mcp__local-rag__query_documents. The real identifier is mcp__mcp-local-rag__query_documents. Anything following that instruction literally fails.

And two sources both claimed to own what happens when a task starts, in incompatible ways. The file's gate said it ran first with no exceptions, while its own routing rule sent the same trigger to a skill that has no gate at all. One of them was always going to be skipped silently. In this session the skill won, and nothing recorded the decision.

Three defects in my own setup, in a two-turn session, in a repository I maintain.

Try it

Claude Code:

/plugin marketplace add shinpr/agent-clinic
/plugin install session-doctor@agent-clinic

Codex:

codex plugin marketplace add shinpr/agent-clinic
codex plugin add session-doctor@agent-clinic

Then run /recipe-diagnose in a repository you've been working in, or $recipe-diagnose in Codex. With no argument it picks up your most recent session there and asks you to confirm before it starts. The README covers the rest.

Run it on the bad one

The session worth diagnosing is the one that went badly, and the instinct is to close the tab and start over with a better prompt. That throws away the best record you have of how the thing actually failed.

A capable model working inside a setup that contradicts itself produces work that looks like a model problem. That matters because you can't fix the model, but you can fix the setup around it.

If you run it, I'd like to hear what it found. The defects that show up across a lot of setups are the ones worth writing about next.

Switching from GPT-5.6 Sol to GPT-6 Astra: Start with Medium Effort

Shinsuke KAGAWA — Sat, 05 Sep 2026 04:59:20 +0000

GPT-6 Astra at high reasoning effort produced the implementation I decided to keep. I still plan to use medium effort by default.

High handled the interactions between retries and persisted state more completely, but its implementation run took about 48 minutes against medium's 31. In a separate review of the same Sol implementation, medium found a startup failure that high missed.

I ran this comparison to choose a default after GPT-5.6 Sol high.

The work behind the comparison

I used Galley, my tool for unattended development. It takes queued tasks through environment preparation, implementation, and verification. When a task is retried, it should preserve the agent's previous work and reuse preparation that is still valid.

The four conditions were Astra low, medium, and high, plus Sol high. First, each analyzed the repository for bugs, performance problems, and maintainability improvements. I selected a common implementation scope from the Astra findings and gave all four the same plan in separate worktrees. Finally, fresh sessions reviewed identical copies of Sol's implementation.

The sessions ran in Codex CLI. The agents read guidance from codex-workflows, including its coding, testing, and development-judgment skills, together with Galley's repository instructions and quality profile. The plan stated the expected behavior and constraints; the agents chose the design, order of work, and verification. These are the conditions I use for my own work.

What counted as a good result

In my own agent workflows, I have repeatedly seen technically defensible suggestions turn into more machinery than the product needs. I wanted to know how much of that expansion each condition could avoid without being steered through every decision.

I assessed whether the agents found consequential problems, proposed sound fixes, and selected work worth maintaining. During implementation, they also had to carry the plan through to working, verified behavior without being walked through it.

A small edge case could score well. Removing unnecessary path manipulation or reusing a sound lock can make a system simpler. A new recovery protocol has more to justify. I did not count findings or changed lines as a measure of quality.

The analysis phase supplied a concrete example. Galley intentionally allows an operator to requeue a running task. Low and medium proposed restricting that operation. High checked the documented intent and existing test, then left it out of its formal findings. For an unattended tool with an explicit manual override, removing that override is a product decision. Tidier state transitions alone do not justify it.

I checked findings against the code and tests, and credited deliberate exclusions only when the report or visible session history supported them. The evaluation notes contain the scoring criteria, reasons for the weights, findings, and full usage breakdown.

A preparation step that invalidated itself

The most useful implementation difference was easy to overlook in a passing test suite.

Before an implementation agent starts, Galley can prepare the environment and create acceptance-test skeletons: starter tests tied to the task's requirements. If the task runs again with the same inputs, those successful preparation steps should be reusable.

All four implementations added fingerprints to detect changed inputs. But Galley also writes information into the task during execution. Creating a skeleton adds an explanation to an acceptance criterion. Include that explanation in the next fingerprint, and the successful preparation step invalidates its own result.

Low, medium, and Sol left a version of that problem in the setup reuse path. High separated the generated explanation from the user-supplied contract. It followed the change through preparation, task updates, and the next run.

An unnecessary preparation run means another wait and another model invocation. It can also send a creator back into test files the implementation agent has already edited. Preserving valid reuse was part of the requirement, just as detecting changed inputs was.

High's other choices were often modest. For large Git change sets, it extended an existing way to pass paths through standard input. For corrupt task records, it reused the existing isolation logic. It covered more behavior without inventing a separate mechanism for each case.

I am taking that implementation forward in this Galley PR. It needs further review and edits. The first CI run passed on Linux and macOS but failed on Windows when a timeout test tried to clean up a log file still held open.

Medium found the earlier failure

Medium's implementation addressed the same plan and generally extended existing mechanisms. It left the setup reuse gap described above. At about $15.61 and 31 minutes, though, it produced a substantial change for less than high's $21.03 and 48 minutes.

Then came the review phase. All four reviewed the same Sol code, so differences here describe their reviewing rather than their own implementations.

Sol had added recovery for corrupt task files whose claims had expired. A claim records that a worker has taken a task. Medium followed startup and found an earlier recovery step: if a corrupt task still had an owner record from an interrupted worker, startup failed before reaching the new recovery code. Unrelated queued tasks never got a chance to run.

Medium reproduced that through the actual daemon startup function. It had found a repair that could look complete locally while leaving the application unable to start.

High's review caught more of the input lifecycle and another reason ordinary requeue operations invalidated preparation results. It did not catch this startup failure. For my workflow, that is a reason to keep a fresh review session even when I pay for high during implementation.

Low did not save much here

Low was slightly cheaper than medium during implementation, but its analysis run cost more. Across the three phases, medium was cheaper overall.

Low's review did identify a data-loss path: reusing an edited input file succeeded, but a later failure while saving evidence could delete that file. Its proposed correction was small and appropriate. Roll back files created by the current attempt; preserve files brought forward from an earlier one. The review missed preparation reuse and other recovery paths.

I would still try low for a small, well-defined edit. This experiment used broad repository work and does not establish how the two compare on that smaller category.

Sol's review was stronger than its implementation

Sol's implementation missed repeated input cleanup. It also retained both old and new scheduling paths, with some tests still exercising the old path. Future changes would have two versions of similar behavior to keep aligned.

Its review found problems no Astra report included. One was another preparation-reuse failure: setup saved a learned environment profile, but its success fingerprint still described the profile from before that update. The next run rejected its own successful setup evidence. I checked this through profile persistence, reloading, and the reuse decision.

Sol also caught a missing shutdown check before taking more queued work and a repository check that rejected valid paths with different capitalization on macOS.

Some remedies needed more scrutiny. Galley tracks child processes in registry files. Sol correctly identified an unsafe fallback to an old shared registry, but declared new ownership metadata necessary before establishing whether the fallback could be retired with a clear migration procedure. I counted that against judgment: the problem deserved a fix, while the proposed extra state still needed a reason to exist.

Sol's implementation result would have been a poor basis for predicting the value of its review.

The price of a completed run

These are the totals for one analysis, one implementation, and one review run per condition:

Condition	API-equivalent cost	Combined elapsed time
Astra low	$26.97	About 49 minutes
Astra medium	$25.67	About 51 minutes
Astra high	$37.23	About 77 minutes
Sol high	$31.79	About 75 minutes

Medium cost about 19% less than Sol high. High cost about 17% more than Sol high, but about 45% more than medium. High's premium depends on which starting point you are considering.

The implementation runs differed sharply in how many model requests they made: 238 for Sol, 80 for medium, and 107 for high. Medium accumulated about 11.1 million input tokens against Sol's 37.8 million, and generated about 50,000 output tokens against 98,000. The bill depends on those token volumes and their cache breakdown, not the request count alone. On this task, medium's lower usage more than offset Astra's higher per-token rates.

These are API equivalents calculated from session usage, not charges allocated from my Codex subscription. They exclude the evaluator's work and subsequent corrections. Approval-wait overhead was estimated from the logs; the sessions ran concurrently on one machine. The resource document separates the phases and explains the calculation.

What I will use next

Medium is my starting point for the next broad Galley task. I would choose high up front when the work crosses retries, ownership, and persisted state, where missing a related path leaves me with another round of repair. In this implementation, high kept preparation reusable across task updates and handled recovery paths the other implementations left incomplete.

This was one run per condition on one repository. The evaluation was model-assisted, and I challenged and revised assessments when their reasoning or attribution was wrong. The final scoring was not blind. I trust the concrete failures and fixes more than small differences between numerical scores.

My next check is whether this choice reduces the work I have to repair afterward. I will still use a separate review pass. Medium's startup finding would have been lost if I had assumed high already covered everything.

Was the Monorepo Really Best for Coding Agents?

Shinsuke KAGAWA — Thu, 20 Aug 2026 10:55:58 +0000

Earlier this year, I thought monorepos were the obvious choice for AI-native development.

The reason was context closure.

A coding agent works better when the code, tests, design decisions, infrastructure, and development rules it needs are available inside the same execution environment. It should not have to wait for a person to paste a decision from Slack, explain a convention that lives in someone's head, or retrieve a requirement from a system it cannot update.

A monorepo made that easier. Put the product and its development context in one place, give the agent tools to search it, and the repository becomes a closed environment in which the agent can work.

I still believe context closure matters. What changed was my view of what the repository was closing.

I was involved in a discussion about introducing a shared agent harness across a large repository. The technical proposal was not the hardest part. The harder question was what would happen to the workflows engineers already used.

If the shared harness required people to stop using their personal skills, instructions, and workflows, would they actually do it?

There was no confident yes.

By then, the repository already contained more than code. It contained several partially overlapping ways of telling agents how to work, some checked in by teams and others maintained by individuals. Introducing a common harness was no longer an installation. It was a migration away from systems people depended on every day.

I had seen the opposite failure too. Centralize before the organization knows how to design, evaluate, and maintain a reliable harness, and the shared system becomes a ceiling. Poor instructions affect everyone. Reviewers absorb the correction cost. AI usage remains shallow because the approved path is less effective than what capable individuals could build for themselves.

The two approaches appeared to fail in opposite directions. Centralize too early, and the organization standardizes what it has not yet learned to do well. Standardize too late, and local workflows become too embedded to replace.

Where the code lived mattered less than who could define how an agent should work across it, resolve conflicting context, and keep the system correct as both the code and the models changed.

The repository stopped being the context boundary

The monorepo argument was stronger when coding agents were less capable.

An agent that could only sustain a short task benefited enormously from having everything nearby. Repository boundaries were practical context boundaries. Work that crossed them required more prompting, more retrieval infrastructure, and more human coordination. Keeping related systems together reduced the chance that the agent would miss something important.

The current long-running models I use behave differently. Given access to the relevant repositories and tools, they can search across projects, trace dependencies, compare implementations, and make coordinated changes without treating each repository as a separate task. Access still has to be designed, but the repository boundary is no longer the same kind of capability boundary.

Cross-repository reach does not make the surrounding structure optional. In a monorepo, broad search still needs package, ownership, and domain boundaries to separate relevant code from similar but unrelated code. Across repositories, search depends on a trustworthy inventory and dependency map, or an incomplete result can appear complete. Better models need less assistance to follow that map. They do not remove the need to maintain it.

This claim is about context acquisition and edit coordination. Repository boundaries still matter for atomic commits, CI, compatibility, and deployment, and better models do not remove those constraints. They do change the monorepo tradeoff for AI development.

Code visibility is still useful, but visibility is no longer exclusive to a monorepo. An agent can discover an API or inspect a dependent service across repositories. It can update several repositories in one run. Producing the edits is usually easier than determining whether they may be deployed together, which compatibility window must be preserved, and how the rollout should be sequenced.

As repository boundaries stopped being context boundaries for coding agents, the monorepo's main AI advantage moved from retrieval to governance.

A monorepo gives product-wide instructions, shared skills, architectural context, and guardrails a common home. The same repository can carry the evaluations and maintenance work that keep them current. That is a real advantage. It is also the part most organizations are least prepared to operate.

Context closure is not context governance

In What Makes Development AI-Native?, I described context closure as the condition in which product context is searchable, readable, and updatable within the LLM's workflow.

That remains necessary. But availability answers only one question: can the agent reach the context?

Governance answers the questions that become important after it can:

Which context is authoritative?
Where does it apply?
Who owns it?
What happens when two instructions disagree?
How is an obsolete rule removed?
How was it evaluated, and who is responsible when local context changes product-wide behavior?

A repository can close context without answering any of these.

This becomes harder as the organization grows. A product large enough to justify a monorepo is often developed by multiple teams. Those teams are deliberately kept loose enough to work without constant coordination. Their code, architecture, release processes, domain knowledge, and agent workflows diverge because some divergence is necessary for autonomy.

A team creates a skill to fix a recurring failure. An engineer adds a personal instruction because changing the team's workflow would affect everyone. Another team solves a similar problem differently. Each addition is locally reasonable. Over time, the repository, team, and individual layers accumulate overlapping assumptions about how work should be planned, reviewed, tested, and completed.

The engineer becomes isolated inside progressively narrower context:

product context
  -> system context
    -> team context
      -> personal context

The deeper the local system becomes, the harder it is for that engineer to change the shared one. A personal workaround is cheap. Correcting a repository-wide workflow requires agreement, migration, and responsibility for everyone it affects.

Local convenience gradually removes the force that would otherwise improve the common system.

Better models make conflicting context more expensive

Missing context is still dangerous. A model cannot reliably honor a constraint it cannot discover.

But missing context is no longer the only important failure mode. Incorrect context, conflicting context, and excessive control can be worse because a capable model can execute them more thoroughly.

I described this shift in When Better Models Make Old Agent Workflows Worse. Rules written to compensate for earlier model failures can become work-generating constraints, leading the model to create the documents, tests, reviews, retries, approvals, and abstractions they imply. The same problem appears at organizational scale in a monorepo.

One team skill may require every change to pass several review phases. Another may instruct the agent to choose only the narrowest proof boundary. A product-level rule may preserve a public contract while a local workflow optimizes for an internal replacement. A personal instruction may require a planning artifact that the shared harness no longer uses.

Each instruction can be defensible in isolation. Their combination can reduce execution quality while appearing more mature because more context and more process are present.

AGENTS.md at least provides a directory-based preference. Codex loads instructions from the repository root toward the working directory, and instructions closer to the working directory take precedence. That does not prove that the local rule is correct; the product-wide rule may be the one that should win. But it gives the model a deterministic guide when the two disagree.

Skills are harder to reason about because they use progressive disclosure. The agent initially sees names and descriptions, then decides which full instructions to load for the task. What enters the context depends on how the task is interpreted. When several skills apply, there is no equivalent semantic hierarchy that says a product-level decision must override a team workflow or a personal preference.

Which instructions enter a task can therefore be hard to predict. Changes in task wording, working directory, or model version can activate different combinations, allowing an organization to improve its harness while quietly reducing performance.

Centralizing before the capability exists

A natural response is to centralize everything, but a shared harness for a large product requires much more than prompt files. Its maintainers need enough product knowledge to separate durable boundaries from old compensation rules and keep context aligned with the codebase. They also need representative evaluations that expose model regressions and workflows that no longer earn their cost.

Prompt and context engineering skill is necessary, but not sufficient. A person who understands model behavior but has not developed the product cannot create accurate domain context. A person who knows the product but cannot evaluate agent behavior may encode every historical incident as another permanent rule.

No small central group can deeply understand every domain in a large monorepo while also keeping pace with model changes. If that group becomes the sole author and gatekeeper, humans become the bottleneck. The common harness drifts away from actual development, teams work around it, and standardization suppresses the very experimentation needed to improve it.

Usage and spending may still increase even while outcomes stagnate. When a weak workflow is centralized, its limitations become organization-wide.

Standardizing after local systems are embedded

Letting teams and individuals find what works produces faster initial adoption. Engineers can experiment without changing anyone else's process, and useful practices emerge from real work rather than from a central design exercise. Autonomy without a path to convergence, however, creates its own lock-in.

A personal skill is not merely a file once someone has shaped daily work around it. It contains learned behavior, trusted shortcuts, and assumptions about how the agent will respond. A team workflow becomes part of planning, review, and delivery. Replacing it has a switching cost even when the replacement is better.

By the time the organization needs a repository-wide harness, it has postponed the difficult decisions: which local behavior should become shared, who has authority to retire the old systems, and how to prove that migration will not reduce performance. Without those answers, the shared harness joins the existing workflows instead of replacing them.

This brought me back to the question from the original discussion: would engineers give up the personal systems they already relied on? It had sounded like an adoption question. It was really a question about migration authority. The organization wanted one harness but had never established who could retire the systems it would replace.

Centralize governance, distribute its maintenance

If a monorepo is going to provide a common AI development environment, its context and harness need a common authority. Product-wide constraints cannot become optional because a team prefers another workflow. The organization needs one way to decide what is canonical, how conflicts are resolved, and when a rule is obsolete. Common authority, however, does not mean that a central team can do all the work.

Shared authority needs distributed capability

Engineers need more than the ability to ask a model for code. They need to understand how task framing, context selection, instruction scope, progressive disclosure, verification, and model behavior affect the result.

Not everyone needs to become a specialist in agent evaluation. Everyone contributing context to the shared system needs enough skill to avoid turning a local preference or historical failure into a product-wide obligation.

The capability cannot be hidden behind a platform team. Product and system context has owners, consumers, compatibility requirements, and a lifecycle. Personal preferences can remain personal when they affect presentation or reversible working style, but they cannot silently redefine correctness, product behavior, approval boundaries, or completion criteria.

Teams that own a domain must also own the accuracy of the context that describes it. Some members need explicit capacity beyond immediate delivery: maintaining shared instructions, reviewing cross-domain assumptions, contributing representative tasks, and resolving conflicts in the repository-wide harness. Governance is centralized; its evidence and maintenance are distributed across the organization.

The harness should need less context over time

Context quality cannot depend on occasional manual cleanup. References, ownership, duplication, schemas, and retired files can be checked mechanically. Representative tasks can detect whether a model update changes behavior. Conflict cases can test which rule wins when product, system, and team instructions disagree. The harness can measure whether a rule improves accepted outcomes or merely creates more work.

Automation should concentrate human judgment where it changes a real decision rather than spend it on routine consistency checks. The evaluation set must evolve with both the codebase and the models. A skill that passed six months ago has not earned permanent authority.

Continuous evaluation is only half of the work. Context and harnesses cannot compensate indefinitely for an inconsistent codebase.

When two parts of the system solve the same problem differently, an instruction can explain the difference. When five parts do, more instructions become a map of unresolved design debt. The model must spend more tokens discovering which example is current, which exception is intentional, and which pattern is safe to extend.

The durable response is to reduce the inconsistency itself. Public contracts should be explicit. Invalid dependencies should be mechanically rejected. Types, tests, build boundaries, schemas, and permissions should enforce what prompts currently have to explain. Temporary compatibility paths should be removed instead of accumulating permanent context around them.

The strongest agent instruction is often a codebase in which the wrong change does not fit.

Judge governance by the cost of outcomes

How should an organization know whether this governance is working? Usage, generated lines, completed tasks, and total spend reward activity. The relevant economic unit is the total inference cost required to produce one accepted outcome at the required quality without human correction.

That includes model choice, retries, abandoned runs, review agents, unnecessary artifacts, excessive exploration, and work generated by obsolete instructions. Some of that cost is legitimate. Exploration can establish that no change is needed, but it still belongs in the cost of reaching that outcome. Repeated exploration that better context could prevent should fall over time. A poor harness may still complete the task by consuming more reasoning, but the organization is purchasing inference that its development environment should have made unnecessary.

The number is a health indicator, not an optimization target. A lower cost is an improvement only while the acceptance bar holds and accepted outcomes stay correct after review.

If a reviewer has to reconstruct the result, the autonomous workflow did not produce the accepted outcome. This measure also gives the organization credit for restraint. The right outcome may be a small change, reuse, removal of a conflicting workflow, or a decision that nothing should be built. As models become capable of sustaining more work, preventing unnecessary work becomes part of operating them well.

The monorepo is not the answer by itself

I still see a strong case for monorepos in AI-native development. One repository can provide a canonical harness, keep product-wide context maintainable through the same workflow as the code, and make shared evaluation and mechanical enforcement easier to operate.

Realizing those benefits requires common context ownership and distributed product knowledge. Without them, a monorepo can contain more contradictions than an agent can reliably reconcile, while a shared harness can enforce the wrong workflow everywhere. A multi-repository system with deliberate cross-repository access, explicit contracts, and well-governed context may no longer impose the disadvantage it once did for capable agents.

Repository topology alone cannot decide the outcome. What matters is whether the organization can operate the context system that topology makes possible.

Conway's law did not stop applying when the executor became a model. The communication structure of the organization shapes the codebase; it now also shapes the instructions, skills, workflows, and evaluation systems through which agents change that codebase.

If teams are responsible only for local delivery, their agent context will optimize for local delivery. If individuals are rewarded for personal productivity, their best workflows will remain personal. If a central group owns the harness without the participation of domain teams, the harness will reflect the central group's limited view of the product.

AI-native development therefore requires more than distributing a coding tool or checking a set of skills into a repository. The development process itself has to become an engineering responsibility. Teams must treat product-wide quality as part of their work, maintain the system that turns intent into autonomous execution, and make the codebase consistent enough to require less explanation over time.

The monorepo once looked like the answer because it put everything the agent needed in one place.

Now the agent can reach much farther.

The harder question is whether the organization can agree on what the agent should find when it gets there.

The repository is no longer the context boundary.

The organization is.

Reasoning Effort Is Not a Quality Setting

Shinsuke KAGAWA — Wed, 05 Aug 2026 13:25:25 +0000

I expected Claude Opus 5 high to produce a better design than Opus 5 medium.

It did not.

I had given both configurations the same repository, feature request, and design template. The task was to add dependency-aware execution to Galley, an open-source local runtime for supervised, multi-model AI coding. A task could declare other tasks as dependencies, and the daemon would claim it only after all of them had succeeded.

In the blind evaluation, Opus medium's design received 84 points from Claude and 82 from Codex. Opus high received 68 and 50. They took almost exactly the same amount of time (21 minutes and 12 seconds for medium, 21 minutes and 25 seconds for high), while high processed 2.29 million tokens to medium's 1.56 million.

My first reaction was that something had gone wrong in the evaluation. I read both critiques again, then the designs and session logs. The scores were not the most interesting part. The designs were.

High explored waiting deadlines, persisted archive history, additional configuration, more state transitions, recovery behavior for those transitions, and tests for the machinery it had introduced. Much of it was technically plausible. Some of it addressed real failure modes. Taken together, those additions made the requested feature larger and harder to operate.

Medium explored less. It stayed closer to Galley's existing lifecycle and reused more of the current claim and failure machinery.

High found more things to think about.

That was the problem.

I had treated effort as a quality setting. In these runs, it behaved more like an exploration multiplier.

What I actually measured

The feature was not a self-contained coding exercise. Galley stores tasks as YAML files and moves them through lifecycle directories. Multiple daemon processes may compete to claim work. Adding dependencies affected parsing, scheduling, failure recovery, CLI output, backward compatibility, filesystem error handling, and tests that had to prove execution order without calling an external model.

I ran seven model-and-effort configurations through four separate phases:

technical design;
design review;
implementation from a common approved design;
code review of a common implementation.

The configurations were GPT-5.6 Sol medium and high, GPT-5.6 Terra xhigh, GPT-5.6 Luna max, Claude Sonnet 5 high, and Claude Opus 5 medium and high. Within each phase, every configuration received the same repository state and the same inputs. These were four controlled phase comparisons around one feature, not seven end-to-end projects in which an early design changed the later implementation.

For design review, all seven configurations reviewed the unmodified Opus medium design. Between phases, I worked with the orchestrating LLM to prepare controlled inputs from the original requirements and the recorded artifacts. For implementation, all configurations received the same deliberately revised design. For code review, I fixed the target to the Sonnet high implementation from the implementation phase so every reviewer inspected the same production code and tests.

I deliberately did not tell the models to produce the smallest possible change or prescribe a convergence method. The request fixed the outcome, constraints, and exclusions; deciding how much machinery they justified was part of the engineering task I wanted to observe. An explicit subtraction rule would test compliance with a supplied convergence strategy, which is a different question.

The outputs were anonymized and evaluated independently by Claude and Codex against a common rubric. I treat the scores as measurements attached to the artifacts, not as verdicts on which model “won.” Concrete mechanisms, defects, preserved boundaries, and demonstrated failures carry more weight than small differences in averages, so I preserved both critiques when the evaluators disagreed.

There is one important defect in the design scoring that needs to be stated here rather than hidden in the notes. The original design request said that only runnable queued tasks could be claimed. The prompt used for the later blind scoring weakened this to saying that tasks with unfinished dependencies must not be executed; it omitted the prohibition on claiming them. This affected the later blind scoring of the design artifacts, not the design-review runs discussed below, which received the original requirements. Both Opus designs proposed claiming some unresolvable tasks and failing them afterward, so the omission did not obviously favor only one of them, and the evaluators separated the designs on broader scope and compatibility grounds as well. The design scores are still not absolute measures of requirement compliance.

In the design phase, both evaluators independently placed Sol high first and Opus high last, so the main direction was not explained by simple provider loyalty. Family and style preference did appear elsewhere, especially in design-review scoring. What I have is a structured case study from one repository, not a population estimate or a model leaderboard.

What high effort changed in Opus

Opus high did not misunderstand the repository. It inspected the implementation extensively and found real questions around concurrency, archive behavior, timing, recovery, and tests. The difference appeared after discovery: once high found a plausible concern, it was more likely to preserve that concern in the final design.

A rare race suggested another check. The check introduced a state transition. The transition needed persistence and recovery. The recovery path then needed tests and an operational explanation. Each step followed reasonably from the previous one, but the chain had little pressure to return to the current requirement and ask which mechanisms could be removed.

On a quick read, parts of the high-effort design looked reassuring. It had considered more failure modes. The overreach became clearer only when I traced each mechanism back to the requested behavior and asked what would stop working without it.

Opus medium showed many of the same instincts at a smaller scale. It still investigated failure handling and concurrent claims, but it generated fewer branches than it had to rank, combine, or reject. In this run, that amount of exploration fit its ability to turn the result into a bounded design.

I think of the observed pattern as an exploration-convergence mismatch. Exploration discovers files, contracts, alternatives, side effects, and failure conditions. Convergence decides what matters now, merges issues that share one correction, rejects optional hardening, and produces a result that fits the approved scope. Increasing effort can expand the first activity faster than the second.

The failure does not look careless. It looks like serious engineering applied to a larger problem than the one currently being solved. In a real codebase, someone still has to own every new state, setting, branch, test, and recovery path after the impressive design document is gone.

Twenty-two million tokens missed one sentence

The design-review phase showed the same imbalance from another direction.

Every reviewer received the original requirements, the same design, and the same repository state. The requirement explicitly said that a queued task with unresolved dependencies must not be claimed or executed. The design under review proposed claiming some such tasks first, then recording a failure afterward.

All seven configurations understood and summarized that mechanism. All seven found real problems elsewhere. Between them they processed 22.64 million tokens and produced 115 findings.

None identified the direct contradiction with the requirement.

The reviewed artifact was the Opus medium design from the opening. It was more bounded than the high-effort design and scored better in the blind design evaluation, but it still violated this governing condition. Better scope control and a stronger score did not make it implementation-ready.

They found partial-write hazards, race conditions, duplicate IDs, stale state, archive behavior, unnecessary scans, failure propagation, notification problems, and recovery gaps. Luna max uniquely followed a dependency cycle outward to a task that was not itself in the cycle. Opus medium found a subtle conflict between stale-claim recovery and a task that had waited for a long time.

The missing check was simpler than most of what they found. More independent exploration created diversity around the technical consequences of the design; it did not guarantee that anyone returned to the governing request and compared the selected mechanism with it directly.

The contradiction did not propagate into implementation. Between phases, I worked with the orchestrating LLM to prepare a revised common design from the original requirements and the review evidence. That controlled input kept both waiting and blocked tasks queued. The correction came from dataset preparation at the orchestration boundary, not from any of the seven review artifacts. This separation was deliberate: each phase compared model behavior against one controlled input instead of letting an early artifact determine every downstream result.

This is why I do not treat a high-scoring review as an approval decision. A review can be relatively strong, contain valuable findings, and still miss the condition that determines whether implementation may proceed.

The same behavior helped in code review

The code-review phase changed the value of wide exploration.

The common implementation contained defects outside its normal path. A lightweight task reader interpreted some valid YAML differently from the standard loader. Duplicate task IDs could cause one dependency result to be reused for another task. A partial failure while reading the task store could allow a dependent task to run before success had been established. Some CLI output described a different state from the one used by the daemon.

Opus high inspected production code, user-visible behavior, performance, and tests. More importantly, it changed a central dependency guard in a safe copy and showed that the main integration test still passed. The test looked like evidence for execution ordering. The mutation demonstrated that it was not sensitive to the behavior it claimed to prove.

No other configuration established that point as clearly. The blind evaluations reflected this: Opus high received 91 and 93, while medium received 88 and 91.

The high-effort review still carried the same weakness seen in design. It produced nine findings, six of them minor, split related problems, and did not always rank incorrect execution above a presentation issue. But a review has a downstream consumer. Candidate findings can be merged, reranked, or declined before they become production structure. Missing a defect is often more expensive than discarding an extra review item.

Design has no equivalent cleanup stage once unnecessary machinery becomes the architecture. The same search behavior that expanded a design helped expose a test that was green for the wrong reason.

Sol showed why this is not a medium-versus-high story

The Sol results moved in a different direction. Sol high received higher blind averages than Sol medium in all four phases, although the size and cost of the difference varied.

In design, Sol high explored the existing code deeply, initially considered a path that would add work to tasks without dependencies, and then corrected itself before completing the document. It avoided introducing a new persistent lifecycle state and allowed blocked tasks to recover when dependency state changed. Both evaluators placed it first, with scores of 89 and 93.

The added exploration was useful there. It was also useful in implementation, where the average moved from 66 for medium to 80.5 for high and the resulting implementation covered more meaningful behavior.

In design review and code review, substantially higher recorded token use accompanied much smaller differences in the measured result. Sol high used 3.24 million tokens in design review compared with medium's 1.13 million, while their averages differed by three points. In code review, high used 4.90 million tokens compared with 3.18 million, with a 2.5-point difference.

The important observation is not that one effort level beat another. Effort expanded the search frontier in both families. Opus high carried more of that frontier into the final artifact; Sol high more often produced a smaller final result from the broader search. The usefulness of the remaining exploration then depended on whether the phase needed selection, implementation coverage, or review recall.

The implementations separated at the edges

All seven implementations handled much of the ordinary dependency flow. They could add the YAML field, wait for unfinished tasks, recognize successful dependencies, display reasons, and pass broad test suites. The models separated where the new feature met existing behavior:

What happens when only part of the task store can be read?
Does an alternate or relative path use the same interpretation as the daemon?
Can malformed input change behavior in an existing command?
Does a feature that is not being used add a full-store scan to an old path?
Is a cycle result stable, or does it depend on evaluation order?
Does a green integration test observe the ordering guarantee, or merely execute two tasks that would have succeeded anyway?

Opus medium produced the implementation with the fewest observed gaps. It connected more of these boundaries without changing unrelated behavior. Luna max implemented the ordinary feature at a much lower estimated API cost, but replaced broader search and display paths, changed the treatment of damaged tasks, and added work even when dependencies were unused. Sol high kept the implementation comparatively small but missed alternate-path behavior and direct daemon-level proof.

These differences are hard to compress into a pass rate. The feature works until the repository is incomplete, an older path reads the same YAML differently, or the test suite gives confidence without observing the protected behavior. That is not benchmark trivia. It is where software maintenance happens.

Resources describe activity, not quality

Across all 28 runs, the logs recorded 167.52 million total tokens. Most were cached input repeatedly read across tool turns, not fresh input or hidden reasoning tokens. The summed active execution time was just under eight hours, excluding human approval waits but including tests and builds; the configurations ran in parallel, so this is not the experiment's wall-clock duration. Applying the standard API prices published on August 3, 2026 produced an estimated cost of $104.58, not an actual CLI subscription charge.

Those resource figures describe activity, not quality. Luna max processed the most tokens in design review and remained inexpensive because cached tokens were cheap, but it was also the slowest configuration in that phase. Token volume, price, exploration breadth, latency, and artifact quality were separate variables.

Effort belongs to the phase

I no longer choose one effort level for a model and carry it through an entire workflow. The useful exploration budget depends on what the phase must produce and whether another stage can remove excess work.

Phase	What additional exploration must earn
Design	A smaller justified decision after inspecting the real constraints
Implementation	Coverage of another real contract or execution path without changing unrelated behavior
Review	A supported candidate finding that can still be merged, ranked, or declined downstream

This is related to a conclusion I reached while simplifying an older agent workflow: be strict about boundaries and evidence, but flexible about the route between them. More capable models made detailed route instructions age badly. These measurements add a second problem. Giving the model more room to search is not sufficient when the phase also requires aggressive subtraction.

The operational question is therefore not “How important is this task?” Important work does not automatically deserve high effort. The better question is “What will happen to the additional possibilities this configuration finds?”

If the answer is that they become architecture immediately, uncontrolled breadth is expensive. If they remain review candidates that another step can verify and consolidate, higher recall may be worth buying.

What I can and cannot conclude

This comparison used one run per configuration per phase. Another task or another run could change close measurements. Claude and Codex sometimes disagreed by more than twenty points, and the overlap between evaluator families and candidate families leaves room for style preference. The design scoring prompt omitted one original claim constraint. The tools and caching behavior also differed between providers, so raw token totals are useful descriptions of these sessions but imperfect provider-level comparisons.

The experiment did not isolate an internal mechanism inside Opus or Sol, and I cannot show that a particular latent “convergence ability” caused the outputs. What I can show is the external pattern in this setup: higher effort expanded the work explored; the configurations differed in what they retained; and the value of that retained work changed across design, implementation, and review.

I also cannot conclude that medium effort is generally better, or that high effort causes overengineering. Sol contradicts both claims. An explicit convergence rule might change the outputs, but testing that would measure the models under a new intervention. Here, I intentionally left engineering selection to the models because that judgment is part of how I use them in real development.

The result is strong enough to change how I assign effort without pretending to be a universal model ranking.

Exploration is not the finished product

Recent coding models are much better at searching a repository, following effects across components, and generating technically plausible responses to what they find. That creates a new failure mode: a model can discover more work than it can economically judge.

Reasoning effort pushes on that boundary. Sometimes it exposes the missing execution path that makes an implementation safe. Sometimes it produces the review finding nobody else could prove. Sometimes it turns a small design into a system for managing the risks introduced by the design itself. The setting does not know which one is happening.

Calling it “high” invites the wrong intuition. High effort is not a higher position on a quality scale. It is a larger search budget handed to a particular model inside a particular phase.

The practical limit of a coding agent may no longer be how much it can explore. It may be how much of that exploration it can turn back into one coherent change.

Complete phase scores

These are the simple averages of the two blind evaluations. They summarize the recorded critiques; they do not override evaluator disagreement or the artifact-level evidence discussed above.

Configuration	Design	Design review	Implementation	Code review
GPT-5.6 Sol medium	82.0	80.5	66.0	86.5
GPT-5.6 Sol high	91.0	83.5	80.5	89.0
GPT-5.6 Terra xhigh	71.5	68.0	66.0	74.0
GPT-5.6 Luna max	72.5	84.5	79.0	61.5
Claude Sonnet 5 high	70.5	71.0	78.5	68.0
Claude Opus 5 medium	83.0	76.5	94.5	89.5
Claude Opus 5 high	59.0	72.5	91.5	92.0

Methodology notes

Human approval wait time was excluded from active elapsed time. Test and build execution time was included.
Token totals include normal input, cached input, cache writes where applicable, and output. Cached input accounted for most of the total.
Cost estimates apply the standard API rates available on August 3, 2026 to recorded usage. They are not actual CLI subscription charges.
Artifacts were anonymized before evaluation. Scores are averages of one Claude evaluation and one Codex evaluation; original disagreements were preserved rather than adjudicated into new scores.
The candidate and evaluator families overlap, so the blind process removes model-name knowledge but not possible preference for a family-specific style.
The comparison used one run per configuration per phase and should be read as an instrumented case study rather than a population estimate.

When Better Models Make Old Agent Workflows Worse

Shinsuke KAGAWA — Sun, 02 Aug 2026 14:11:20 +0000

I recently watched a coding agent refuse to start an approved implementation.

The work plan contained this:

## WorkPlan Review

- **Status**: approved (2026-08-02)

The agent found the line and understood the word approved. Then it asked for approval again because the workflow expected a field named Implementation Approval.Status.

Nothing was missing in substance. The label was wrong.

The agent was following a repository-resident workflow I had designed to carry a change through requirements, design, review, planning, and implementation. I had spent months adding status fields, handoff contracts, review gates, retry rules, and escalation paths. Each addition addressed a failure I had actually seen. Together, they had made the workflow brittle enough to stop on a heading.

I had built the system around an old question: how do I stop the model from losing the path?

The model could now follow the path. My path had become the problem.

Two kinds of constraint

The approval incident exposed a distinction I had missed.

A boundary constraint reduces the solution space:

Preserve the public API contract.
Do not perform an irreversible external action without authority.
Implement the confirmed requirements and keep recorded non-goals out of scope.
Treat the task as complete only when the required behavior is observable.

A work-generating constraint creates obligations:

Always produce three alternatives.
Require unit, integration, and end-to-end tests for every feature.
Require a mitigation task, owner, rollback plan, and approval record for every risk.
Resolve every review finding before continuing.
Stop when an optional field is absent or uses an unexpected label.

Both can look like safeguards. They behave very differently once the model is capable of carrying them out.

Boundary constraints tell the agent where it may operate. Work-generating constraints are read as requested work. A reliable agent will reliably create the extra artifacts, tests, abstractions, and stop conditions they imply.

That gave me the principle I now use when changing an agent workflow:

Be strict about boundaries and evidence. Be flexible about the path between them.

Why the problem surfaced now

The capability shift behind this is real, although public measurement often trails the models people are using.

METR's task-completion time horizon shows a strong historical increase in the difficulty of software tasks that agents can complete. METR also warns that the benchmark covers well-specified tasks, that capability remains jagged, and that new releases may go unmeasured for weeks or be skipped entirely. As of August 2, its public page was still dated May 8, and even its list of recent unmeasured models had not caught up with later releases.

Actual usage has moved toward longer execution as well. An analysis of roughly 400,000 coding-agent sessions found a recognizable division of labor: people made most planning decisions, while agents made most execution decisions. The authors describe it as people deciding what to build and the agent deciding how to build it.

Neither source proves how today's models behave in my workflow. The model families involved in the failures described here were released after the measurements and benchmarks I found. I did not find a public rerun on those releases.

That timing matters most for FixedBench. In May 2026, the researchers tested five then-current coding models on 200 issues where the code had already been fixed. The agents still made undesirable changes in 35% to 65% of cases. It is useful evidence that action bias existed in that generation. It is not a current failure rate.

The more interesting result is what happened when the researchers changed the instruction. Telling the agents to verify first and treat abstention as success reduced edits to already-correct code. On partially fixed code, the same instruction made them abstain when more work was still required. Prompt engineering traded action bias for passivity.

That is exactly the trap I had created in my own workflow. A model failed once, so I encoded the opposite behavior as a universal rule. When the model or task changed, the compensation remained.

The strongest evidence here comes from the failure history of this workflow on the current models I use. The studies explain why those failures are plausible and why simple prompt fixes decay. They do not substitute for observing the system itself.

How reliability started manufacturing work

The strict approval field was only the most visible failure.

In another run, the workflow stopped before implementation because it could not find external accounting evidence, document approval records, an exact release command, and detailed E2E environment assumptions. The repository, local services, and test tools were available. Most of the implementation could have proceeded. The workflow had converted useful planning context into mandatory readiness gates, then treated every missing gate as a user-owned decision.

The documents had expanded too. Product requirements and design docs started collecting stakeholder approval, live-service access, release procedures, dashboards, and operational evidence. A work plan for a data-export change included account setup and production concerns even though the requested outcome ended at implementation.

Review made the expansion worse. During that same planning flow, a reviewer requested a deterministic 20,000-row proof. The planner accepted the finding without asking what decision the proof would change or whether a cheaper boundary check would be enough. The reviewer then found that the proposed proof data contradicted an existing aggregation contract, which triggered another revision. Every step was technically defensible. The loop itself had no economic judgment.

I had made the reviewer authoritative and the author obedient. “Review until approved” had become “add work until the reviewer runs out of ideas.”

That is not convergence. It is a ratchet.

What I changed

I did not solve this by deleting every rule. I changed where the workflow is strict.

Decide the outcome before designing the solution

A user request is no longer accepted as an automatically valid implementation scope. Before design, the workflow records:

the observable outcome;
what is required now;
what describes the current state, a desired future, or speculation;
explicit non-goals;
a rough view of implementation cost and structural impact.

The cost assessment stays rough because this is requirements work, not detailed estimation. Its job is to make a bad tradeoff visible while scope is still cheap to remove.

“No change” and “reuse what already exists” are valid conclusions, but neither is a default. The agent must inspect enough evidence to decide whether the outcome is already satisfied, partially satisfied, or still requires implementation. That avoids turning FixedBench's action bias into its mirror image.

This also changes what users need to communicate. A useful request contains the boundary of the current outcome, not just a list of desired features. “Extend the existing authentication path without changing the public response contract” is more valuable than a long list of implementation rules. It states both the result and the point at which design needs to be reconsidered.

Give the model decisions, not a predicted route

The old workflow described complete routes: run one agent, populate every field, run another agent, stop on any missing input, repeat until a serialized state appears.

I now try to give each phase four things:

the purpose it owns;
the evidence that can change its decision;
the criteria for choosing the next action;
the smallest result the next consumer needs.

“Run every test lane” predicts a route. “Use the narrowest test that observes the required boundary” provides a decision criterion. Exact schemas still matter where software parses the response. A human-readable work plan does not need to fail because two headings express the same approved state. I changed the approval check to accept semantically equivalent evidence instead of requiring one heading.

The main agent also owns lightweight resolution. It can interpret semantically equivalent states, resolve repository-local ambiguity, retry with new evidence, and continue unaffected work. User escalation is reserved for decisions that belong to the user: a changed product outcome, a new requirement, a major approved design change, unavailable authority, or an irreversible external action.

Let review findings be declined

The receiver of a review now has three choices:

Apply a finding that contradicts an approved requirement, accepted design, repository rule, or observable correctness.
Decline a finding that adds scope, reverses an exclusion, duplicates proof, requests optional hardening, or costs more than its observable effect justifies.
Return for user decision when resolution changes the product outcome or a major approved decision.

A decline includes evidence and goes back to the reviewer. The reviewer may maintain the finding when that evidence still leaves the result incorrect or unverifiable. Repeating the same preference without new evidence does not block the workflow.

A useful review process preserves a basic engineering skill: deciding not to implement a technically reasonable suggestion.

The constraint I had to put back

Removing rules can be just as careless as adding them. I proved that during this rewrite.

The planning flow generated integration and E2E test skeletons so the first vertical slice could prove an acceptance criterion across a real boundary as early as possible.

After I simplified the handoff, the planner treated the generated test skeleton as an unrelated file and planned a new E2E test later in the project. The artifact still existed, so the workflow looked complete. Its reason for existing had disappeared.

I restored the skeleton's purpose in the planner: consume the exact generated file in the earliest task that can make its boundary executable. Shared infrastructure may come first only when no acceptance criterion can work without it, and only the infrastructure needed by that first slice belongs there.

This is a boundary constraint worth keeping. It connects a planned implementation to observable proof and prevents integration risk from being postponed until every component has been built. Removing it made the workflow smaller and less reliable.

What remains strict

Several controls still earn their cost:

User approval for product requirements and major design decisions.
Explicit authority and mechanical containment for irreversible actions.
Exact schemas at genuinely machine-consumed boundaries.
Plans for work with real dependencies.
Early vertical proof of an accepted outcome.
Observable verification before completion.
Independent review where a separate perspective can catch false confidence.

For dangerous operations, sandboxes, restricted credentials, isolated filesystems, and network controls are stronger than long prompt instructions. They limit the blast radius even when judgment is imperfect. Repeated approval prompts are a weaker substitute: one implementation report describes high approval rates and declining attention, and responds by using automated judgment to reduce routine prompts.

The same division applies to the workflow as a whole. Enforce the durable boundary mechanically where possible. Let the model choose among reversible actions inside it. An account of an agent-first engineering environment uses a similar principle: treat the top-level instructions as a table of contents rather than an encyclopedia.

The four questions I now ask

I now use four questions when reviewing an existing rule:

What changes if I remove it? Name the decision, irreversible boundary, downstream consumer, or observable failure it affects.
Does it limit scope or create work? Keep durable boundaries. Require current evidence before generating artifacts, test lanes, approval records, retry loops, or operational tasks.
Can the model choose from evidence? Replace exhaustive branches with a purpose and selection criteria when the choice is reversible.
Does it work in a fresh session? Rerun a representative task from the beginning. Continuing the old conversation only proves that a human can steer around the failure.

The fresh-session test has to cover both sides. A small change should remain small, ordinary ambiguity should not return to the user, and optional review findings should not expand scope. Required tests, approvals, and quality checks must still run. Fewer rules are not evidence of a better workflow.

A workflow can become technical debt

The hardest lesson for me was that reliability work can itself become overengineering.

Every workflow rule depends on an assumption about model behavior. A rule added after yesterday's failure can become redundant when the model improves, or harmful when a different task needs the opposite judgment. Status fields and review gates add coupling just as surely as public interfaces and shared abstractions do.

Boundary constraints age better because they describe what must remain true: the approved outcome, the public contract, the authority available, and the evidence required for completion. The route between those boundaries can change with the model.

I no longer ask only whether a new rule would have prevented the last failure. I also ask what work it will create, what decision it will remove from the model, and whether it will still make sense after the failure mode disappears.

The job of an agent workflow is no longer to make a model capable of finishing the process. It is to help a capable model finish the right amount of work.