DEV Community: Kent Alstad

Building a Self-defining Agent Organization

Kent Alstad — Sat, 31 Oct 2026 16:46:44 +0000

Part of the write-ups behind my Build Stuff 2026 talk, Control Your Fate at the Metal, Vilnius, 2 December.

Most work on agents is about the agents. Better planners, better tools, better prompts, better memory. The organization around them, how goals are set, who owns what, how work is handed off, when a human is asked, is designed once by an engineer and frozen.

That is backwards. Agents are cheap and disposable. The organization is what achieves goals. So the thing to build is not a set of agents but an organization that evolves itself to achieve its goals, inside a structure that humans govern at the most strategic points.

Goals are the reason the organization exists

An organization is not a collection of capabilities. It is a set of goals and whatever it takes to achieve them. Revenue closed. A product shipped. A market entered. Each goal has an owner and criteria that are met in the world or not met at all.

The goals form a graph. A company outcome at the top, function goals beneath it, each supporting the one above. The graph exists so that success can be traced: from a company result, down through the goals that produced it, to the work that moved each one. Any task that cannot be traced to a real-world outcome through that graph is not work. It is motion.

Everything else is instrumental. Roles, procedures, tasks, documents, reviews, the agents themselves: all of it exists to move a goal, and none of it is value on its own. An organization that is busy and achieves nothing has failed, however impressive its activity looks.

So the first question is never "which agents do we need?" It is "which real-world outcomes, and what would it take to produce them?" The answer to the second half is allowed to change.

The organization evolves itself

Human organizations learn by changing their structure. They hire for roles they did not know they needed, write down a procedure after the third time something went wrong, split a team when it owns too much. Nobody files a ticket with engineering to do this. The organization rewrites itself.

An AI organization can do the same if it is written in a language its agents can read and write. Define the organization as documents: what each function is for, what it owns, how work is done, who may do what. The agents read those documents to know who they are. When a goal needs a capability that does not exist, an agent writes the description of the specialist that will provide it. When a procedure fails, the organization proposes a better one. When a goal has no owner, the organization proposes the function that should own it.

The recursion runs all the way up. Specialists are composed for a task and discarded. Functions persist, own goals, and commission specialists. The organization as a whole reviews its own performance against its goals and changes the functions and procedures that fell short. Each layer is a document the layer above can rewrite.

Learning, in this picture, is not remembering what happened. It is changing the definition so the next goal is achieved differently. A lesson that stays in a notes file changes nothing. A lesson that rewrites a procedure changes every cycle after it.

The structure humans govern

An organization that can rewrite itself needs a structure it cannot rewrite, and that structure is where the humans are.

                    HUMANS
                      |
   set direction  .  define what success means  .  decide what may leave the building
   approve spend  .  approve what the organization may become
                      |
                      v
   +------------------------------------------------------------------+
   |  CONSTITUTION  .  PERMISSIONS  .  HUMAN GATES  .  BUDGETS         |   fixed
   +------------------------------------------------------------------+
                      |
                      v
   functions . roles . procedures . knowledge . tasks . artifacts          evolving
                      |
                      v
                    GOALS ACHIEVED

The fixed layer is small. A constitution: what the organization values and may never do. Permissions: what each kind of work may touch, under least privilege. Human gates: the decisions that stop the organization until a person answers. Budgets: how much time and effort a goal may consume before it stops and asks. The organization may not change any of these, however inconvenient they become. Humans may, in the same documents, in the same language.

Everything beneath that layer is the organization's to evolve.

At the most strategic points, not every step

Where humans sit in this structure matters as much as that they sit in it. Put a human on every action and you have built a tool that needs an operator. Remove the human and you have built something nobody governs.

The strategic points are the ones where human intent carries information the organization cannot derive: which direction the company takes, what a goal's success means, what may be published, spent, deployed or committed to, and what the organization may become. Those are gates. The organization stops at them and asks, in one page: the decision, the alternatives, the evidence, a recommendation, and why a human is needed.

Decision: Choose the primary market strategy for next quarter.
A: Agency expansion   B: Direct enterprise   C: Hybrid
Recommendation: C
Why human input is required: the choice depends on risk tolerance and
the direction you want the company to take. Nothing the organization
can read answers that.

Everything that is not a gate, the organization owns. Research, drafting, critique, planning, and the judgment calls inside them are never escalated. A human who is asked to confirm work the organization is competent to do is being wasted, and the organization that asks is failing at its job.

The human's answers are binding, cheap to give, and recorded. The organization re-plans from them. Over time the human spends less attention and the organization achieves more, which is the only direction this relationship should move.

How you know

Not by output. Plans, tasks, artifacts, reviews, cycle counts and token totals are exhaust. The dashboard that makes an agent system look busy says nothing about whether it is working.

By outcomes in the world, traced back through the goals that produced them. Revenue that arrived. Accounts that were created. A product that shipped. Each outcome traces down the goal graph to the functions that owned the goals and the work that moved them, and each change the organization made to itself is judged by whether an outcome moved because of it. Success that cannot be traced to the world is not success.

Within the structure, with human attention spent only at the strategic points. Give the organization a goal its current shape cannot achieve. Watch whether it reshapes itself, achieves the goal, and achieves the next one faster because of what it changed. Watch how often it asked, and whether every question was one only a human could answer.

If the tenth goal is achieved because of what the organization learned on the first nine, and the human was consulted only where it mattered, the organization is evolving. If not, you built a collection of agents with extra steps.

Build Stuff 2026

The organization in this article runs on infrastructure we chose to own rather than rent: our own loop, our own state in Git, our own gates. Why we go that far down the stack is the argument of my talk at Build Stuff in Vilnius this December, the 15th edition, under the theme The Age of Agency.

Control Your Fate at the Metal
Wednesday 2 December, 15:00. 45 minutes.

If you're coming, say hello. If you're not yet, 2026SPEAKER_20 takes 20% off a ticket.

#BuildStuff15 #buildstuffconf #AgeOfAgency @buildstuffconf https://buildstuff.events

I'm Kent Alstad. CEO and innovator at Cloud.Army. I build KandiForge, a toolchain for running software projects with AI agents under an evidence-based process. If you run an organization of agents rather than a set of them, I'd like to hear what you let it change about itself.

Pre-emptive integration analysis: stop putting knowledge into your agents' context, watch their work instead

Kent Alstad — Thu, 29 Oct 2026 20:11:03 +0000

Part of the write-ups behind my Build Stuff 2026 talk, Control Your Fate at the Metal, Vilnius, 2 December.

For a year, most of our effort went into getting knowledge into agents' context: summaries of every file, insights from earlier sessions, briefings on every prompt, notes on every command. The last parts of this series measured what came of it. Reach went up, and behaviour didn't change. Even my own corrections, captured as rules and fired at the moment they apply, didn't cut the repeats in the first days. They went up.

One thing did work, and it isn't a context feature at all.

What worked: watching the branches

I run many agents at once, each on its own issue branch in its own worktree, across six repositories. The shadow merger merges every open branch together in the background whenever one gets a new commit, and works out what collides: which files, which functions. Each affected agent gets one line before its next command or edit: your branch and #489 both change this file. It also relays failures between agents ("another agent hit this exact error three minutes ago") and warns when two live sessions are working in the same area.

We count problems caught before merge. Every week before this shipped, the count was zero, because nothing looked. In its first week: 79.

Compare the two approaches:

	Context features	Shadow merger
What it knows	general knowledge about the repo	a specific fact about this agent's work, right now
When it speaks	on every prompt, read, command	only when something is actually wrong
What it says	"here's what might be relevant"	"this collides with that"
Measured effect	under 1% acted on	79 caught before merge, from zero

The difference isn't the model or the prompt. It's that the shadow merger watches the work itself and speaks only when it has one specific, true thing to say.

The gap: it only sees commits

The shadow merger reacts to commits. Agents often edit for a long time before their first commit, and everything they do in that stretch is invisible to it. Collisions that already exist in the working tree surface late, sometimes only at merge.

The obvious fix is to tell agents to commit more often. We've learned what telling agents things is worth. So the agents won't be told anything.

Part 1: checkpoint commits, without the agent

At the end of every agent turn, the hook makes a checkpoint commit of the worktree's changes on the agent's own issue branch. It's local, nothing is pushed, and it happens only in issue worktrees, never on the integration branch. The agent isn't interrupted and doesn't know it happened. When the agent makes its real commit, the checkpoints are squashed into it, so history stays clean. It works the same in every runtime we support, because it's one entry in the hook wiring list rather than a feature of one tool.

Now the shadow merger sees work in progress within a minute.

Part 2: pre-emptive integration analysis

Once the combined tree reflects what every agent is doing now, it becomes the place to analyse everything, before anything is integrated. I call it pre-emptive integration analysis: integrate everyone's work continuously in the shadow, analyse the result, and tell each agent what its part broke while it is still writing it.

The code watcher: anti-patterns on the syntax tree, duplication, security findings.
The SuperSmall point: structure and complexity of each change, recomputed at every checkpoint, so it's a running score while the agent works rather than a verdict at review time.
Shared memory, decisions and dead ends that every agent shares: does this change repeat a recorded dead end, or contradict a recorded decision?
A build and test of the combined tree on a spare machine, so integration breaks show up before anyone merges.

Each finding goes to the agent whose checkpoint caused it, through the same one-line notice the shadow merger already uses, before that agent's next command or edit. It informs and never blocks. It all runs in the background service, triggered by commits: no watching working trees, no polling, nothing inside the agent's session.

This changes the review step too. By the time an agent reaches "address the ranking", most items have already been raised and fixed while it worked. Review starts from what's left.

Why I think this is the breakthrough

Every context tool, ours included, tries to make the agent smarter before it acts. This does the opposite: it lets the agent act, watches what it actually did, and tells it the one thing that's wrong while it can still fix it. That's how a good senior engineer works with a team. Nobody briefs you on the whole codebase before every task. Someone who can see what everyone is doing says "hold on, that collides with what Sam is doing."

What we'll measure

Problems caught before merge, which should rise as checkpoints expose work earlier.
Fixes after merge, the outcome: changes that later needed a fix because of something another branch did. It should fall.
How soon a finding reaches the agent after the code that caused it was written.
Whether the agent acts on it: the edit that follows a notice.

This is filed as an epic with four parts, in order: checkpoint commits; the watcher and the point on the shadow branch; shared-memory checks; the combined build. Part 1 is in design now. I'll report the first numbers when it's running.

I'm Kent Alstad. I build KandiForge, a toolchain for running software projects with AI agents under an evidence-based process. If you run agents in parallel, I'd like to hear how you find out they collided.

A Case for Evidence Based Agentic Development

Kent Alstad — Fri, 09 Oct 2026 19:31:50 +0000

This is evidence that does not forget. It is not confused by the excitement of the moment. It is not absent because human witnesses are. It is factual evidence. Physical evidence cannot be wrong, it cannot perjure itself, it cannot be wholly absent. Only human failure to find it, study and understand it, can diminish its value.

Paul L. Kirk, 1953 [1]

Kirk was writing about crime scenes. I think he was also writing about software built by agents.

Where we are

Software is more complicated and more feature rich than it has ever been. Agents did that. Point enough of them at a well-defined problem and you get working code, passing tests, and a system that looks almost done, in days.

What we don't have is a viable way to stabilize it and release it.

I found this out the way most people will. I realized I could run five projects at once. Five repos, five sets of agents, five streams of change, all day. It worked, in the sense that code appeared. It also became impossible to monitor every session. I could not read what was being produced, let alone judge it.

That was the moment the framing changed for me. I don't want a co-pilot. I want five dev teams. And a dev team needs something a co-pilot never did: a way for the person in charge to know what is true about the product without having watched every line get written.

The phases of a product

Every product I have built with agents goes through the same three phases, and the proportions are consistent enough that I'll put numbers on them.

The first 80% comes from the spec. You describe what you want, the agents build it. This part is astonishing and it is the part everybody has seen.

The next 15% is vibing. Feedback, iteration, trying things. You use the app, you say "no, like this", the agent changes it. This is where most developers are when they are rapidly creating something. It is interactive, it is the closest to the old way we developed, and it is great for creative solutions. It is also where most people stop, because the app looks done.

The last 5% is the hard part. Bugfixes. Performance. Hardening. Integration across the pieces, and stabilization before release. The trick in this phase is not moving backwards, and not injecting too much change while you fix what is in front of you. Of the five phases a release goes through, integration and stabilize are the difficult ones, and they are the two the first 95% does nothing to prepare you for.

The phases of how people approach it

People's approach to agentic development goes through phases too. I went through all of them.

First, the pair model. For a long time, the pairing model that tools like Cursor promote seemed right. One developer, one agent, a conversation, a diff. But that model is steeped in helping us do it the way we used to do it, only faster. That doesn't work. You end up with more code to stabilize and no reasonable strategy to do it. The pair model is the 15% phase made permanent. It has a place. It is not a viable solution for stabilization, and it does not scale to factory size.

Then, more tests. The next phase was to make the agents write tests for everything, and the problem became how to manage a large volume of change in a way where the tests really guard against going backwards. For the most part, the agent that writes the code also writes the tests. They helped, for APIs. But even there, I found that the complex ways the apps actually used those APIs let problems through that the API tests passed. And what the tests did not catch at all was the requirements mistakes. A test written by the author of the code checks the author's understanding of the requirement against itself. If the understanding is wrong, the test is wrong in the same way, and it passes.

Then, human review. It really seemed like human review was essential. It also seemed impossible, because the volume of review is overwhelming. Five teams produce more pull requests in a day than a person can read in a week.

So that is the situation. The volume and pace of change demands an oversight system that can keep up with "super" developer productivity. Not a faster reviewer. A different kind of gate.

How do we know it is done?

Before anything else, there has to be an answer to that question that was written down before the work started.

For every change, an acceptance plan: the criteria that define done, and the evidence items that will prove each one. If the change has a user interface, the evidence has to narrate the workflow that must be demonstrated. Not "the login works". The fresh install, the one button on the launch screen, the tap, the page the app must open, the consent screen, the picker that appears after it.

I'll describe what these plans look like in practice, because the details are the method.

The criteria are numbered, and each one is a testable statement about behaviour, not about code.
Each evidence item names the criteria it covers and the artifact it produces: a log, a test report, a recording, a screenshot.
Each item says who produces it, and the answer is never the agent that did the work. The wording in ours is blunt: produced only by the evidence service, after the build. No agent runs, copies or edits it.
For a user interface, the item walks the workflow step by step. Every step has a test id. Every step that has to be stubbed is printed into the run log as a stub, so the recording can't pretend.
Test data is a dedicated, human-provisioned test account. Never a production user.
Honest limits are written into the plan. If a step is interactive and cannot run unattended, the plan says so and says how the evidence will be produced instead.
A human accepts the plan before any code is written.

That last point matters more than it looks. The plan is accepted while nobody is attached to an implementation yet. It is the one moment in the process where "what does done mean" gets decided without a diff in the room.

The case: concrete evidence, human focused, end-user driven

Here is the argument.

Not only do we need concrete evidence to check each change against. The evidence has to be human focused and end-user driven. It has to show the product doing what a person would do with it.

It may seem counter-intuitive, but when an agent is tasked with writing a test fixture that works the feature the way a user would, and that fixture captures a video, and another agent uses that video as the working evidence, the agents start to catch errors they could not find in the code. No person records anything. The video is just test output. What makes it different from a test report is its form: it is meaningful to the human who has to accept it, and it turns out to be meaningful to the agent that compares it. The question "has this video changed compared to the baseline?" is a different question from "is this code correct?", and agents are better at the first one. The interpretive skills they bring to comparing two recordings yield a better level of oversight: more attuned to human requirements, and to whether the app is actually usable.

Two things I want to be clear about.

This is in addition to tests, not instead of them. I have thousands of unit tests and I am keeping every one. Videos are strong exactly where unit tests are weak: the whole product, in use, end to end. Unit tests are strong where videos are weak: a single function, every edge case, in milliseconds. Evidence is both.

The video proves more than what it shows. For the fixture to record a user signing in and picking a repository, the build has to succeed, the device has to boot, the network has to be up, the backend has to answer, the data has to be right, and the whole thing has to finish inside the timeout. If any subsystem the user never sees is broken, there is no video. Often the issues the recording exposes are not in the frames at all. They are in the fact that the recording could not be made.

How a recording is made, and what a baseline is

The acceptance plan lists the criteria and the evidence items. An agent writes a test for each item. A separate service, not the agent that wrote the code, builds the branch and runs the test. The test produces the output: a recording, a log, screenshots. A reviewer accepts or rejects.

Accepted output becomes the reference. That is the baseline, and this is the part I want people to take away. The baseline is produced when the evidence is accepted. Nobody records a baseline by hand. Nobody decides in advance what the product should look like. The last thing a human accepted is, by definition, what the product should look like now.

The next change to that area runs the same test, and the new output is compared with the reference. Because every change is issue driven, with an acceptance plan, and ends with evidence accepted, the agent knows what was supposed to change. So it can tell the difference between "this moved because the plan said it would" and "this moved and nothing said it should". It flags the evidence that needs to change as part of the process, and it flags the differences that nothing explains.

People accept the reference. After that, the agents compare against it.

Where the gates sit

Quality sets the schedule, not the calendar. A version is in one phase at a time, and it moves to the next when the evidence says it can.

Before any code. The acceptance plan, and a design spec reviewed by an Architect for the design and a Product Manager for the product. Nothing is built until both are approved.
During the build. Each change is worked in isolation, on its own branch, in its own worktree. The way out is evidence: an artifact produced from a build of the code under test. A pull request is not evidence.
At integration. One change at a time. The evidence is reviewed and accepted by someone who is not the implementing agent, and then the change is merged. Acceptance gates the merge. Where a change can only be proven from an assembled build, because it spans repositories, the order flips: merge first, produce the evidence from the integrated build, and acceptance gates the version leaving integration instead. Either way, nothing leaves integration without accepted evidence, and integration testing runs after every merge.
Acceptance creates the baseline. Evidence produced for review lives in a scratch area. Accepted evidence is committed as the reference. Recordings, logs and screenshots all take the same path.
Stabilize. Slow the rate of change. Only the bugs the release needs. A risky feature gets flagged off rather than fixed under pressure.
Released. Each change is confirmed in the released build before the version is closed. Hotfixes only after that.

Waivers exist. Sometimes a change does not need a recording, or does not need a plan. But a waiver is recorded, with a reason, by a person. It is never a silent skip.

What the humans do

The human still creates the specs and the product definition. That does not go away, and nothing in this method touches it.

In the stabilization process, the human's job has three parts.

Accept the evidence plan up front. Decide what done means before the work starts.
Review the evidence that is ready for acceptance, after it has passed the agent gates. The agents pre-flight the review. They run the comparison against the reference, sweep out most of the problems, and send them back to be fixed before a person ever sees them. What reaches the human is evidence that has already passed, with the agent's account of what it checked.
Review the repo news: the agent's report of the differences it flagged for investigation, with the narrative detail that lets you dig into only what is needed. You open the three recordings that changed when nothing said they should. You do not open the forty that didn't.

There are two human roles in this. The one I just described is a Product Manager. The other is an Architect, for the technical issues: the design, the data, the things a recording cannot show. In general, every review is either a business review or a technical review, and the method routes each piece of evidence to the right one.

I am not stuck on where humans participate. I will continue inserting an agent into the human role, prioritizing or helping with review.

Why would this be true?

I came to this by running out of alternatives, not by reading papers. But the published work points the same way.

Judges, human and machine, are more reliable when comparing two things than when rating one thing on its own. Thurstone showed it for people in 1927, and the finding has held up [2]. Surveys of language models used as evaluators find the same: pairwise comparison is more consistent and agrees with human judgement more often than absolute scoring [3].

Asked to judge correctness from the code, models miss a lot. In one study a frontier model passed half of the incorrect implementations it was shown [4]. In a 2026 code-review benchmark, four of five models found zero performance bugs [5]. Telling the reviewer the change was bug-free cut its detection of real vulnerabilities by as much as 93 percent [6]. That last one is the agent saying "done".

Tests written by agents are weak oracles. In a study of agent-authored test patches, four in five had weak or no explicit assertion: the test ran the code and checked nothing [7]. And models favour their own output. The more a model can recognise its own work, the more it prefers it, even when told the work is its own and it isn't [8]. The author's tests and the author's review are the same judge.

None of that proves that comparing recordings of an app catches regressions. Nobody has published that study. It says the method is pointed in the direction the evidence already points. The proof is what the rest of this series is for.

What I am after

I am an agent product developer trying to find a way to ship great products. The other side of this is the struggle to stabilize, and not shipping. I have been on that side.

Value, for me, is a shipping product that people use. If this method yields that, without slowing down, and without introducing new costs or processes that can't be managed, then it is valuable. If it doesn't, it isn't, and I'll say so.

So I'll end where I started, with Kirk. Evidence that cannot perjure itself, that is not absent because witnesses are, that is only diminished by our failure to look at it. When your agent tells you it is done, what do you have in your hand?

Are others doing this? Is there a better way?

Notes and sources

Paul L. Kirk, Crime Investigation: Physical Evidence and the Police Laboratory (New York: Interscience, 1953), ch. 1, p. 4. Often misattributed to Edmond Locard.2. L. L. Thurstone, "A Law of Comparative Judgment", Psychological Review 34 (1927); D. Laming, Human Judgment: The Eye of the Beholder (2004).
"A Survey on LLM-as-a-Judge", arXiv:2411.15594; Li et al., "A survey on LLM-as-a-judge", ScienceDirect, 2025.
"On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization", arXiv:2507.16587.
"Bigger Isn't Always Better: A Comparative Evaluation of LLMs for Automated Code Review", arXiv:2606.15689.
"Measuring and Exploiting Confirmation Bias in LLM-Assisted Security Code Review", arXiv:2603.18740.
"All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code", arXiv:2606.18168.
Panickssery, Bowman and Feng, "LLM Evaluators Recognize and Favor Their Own Generations", NeurIPS 2024, arXiv:2404.13076.

Research notes: do agents judge differently when comparing than when verifying?

Supports the claim:

Comparing beats rating, for models. Surveys of LLM-as-a-judge find pairwise comparison more consistent (less position bias) and better aligned with human judgement than absolute scoring (arXiv 2411.15594; ScienceDirect S2666675825004564).
Comparing beats rating, for humans. Thurstone (1927), Laming (2004): people are more reliable at relative than absolute judgement. Comparative judgement in assessment rests on this.
Models judging correctness from first principles do badly. GPT-4 passed 50% of wrong Java implementations as correct (arXiv 2507.16587). In a 2026 code-review benchmark four of five models had 0% recall on performance bugs (arXiv 2606.15689). Framing a change as "bug-free" cut vulnerability detection by 16 to 93% (arXiv 2603.18740).
Agent-written tests are weak oracles. 80.2% of agent-authored test patches had weak or no explicit oracle (arXiv 2606.18168, "All Smoke, No Alarm"). Agent-written testing is "a model-dependent process style rather than a dependable driver of success" (arXiv 2602.07900).
Models favour their own output. Self-preference bias rises with self-recognition; models preferred summaries merely labelled as their own (Panickssery, Bowman, Feng, NeurIPS 2024). The author's tests and the author's review are the same judge.
Vision models find non-crash functional bugs from screenshot sequences that crash-based tools cannot: VisionDroid found 29 new bugs on Google Play, 19 confirmed and fixed (Liu et al. 2024). Third-party reruns found far fewer (FuncDroid, GraphDroid), so setup matters.

Cuts against, or limits, the claim:

Pairwise judging is more vulnerable to distractors the judge happens to favour; absolute scoring with a rubric was more robust to that manipulation (arXiv 2504.14716).
Pairwise verdicts can be locally consistent and globally contradictory (arXiv 2602.16610).
Video difference captioning is still hard. ViDiC-1K (1,000 video pairs, 3,720 checklist items) shows "a significant performance gap in comparative description and difference perception" across 17 multimodal models (arXiv 2512.03405). Same-or-different on image pairs: 17.6% for an untuned 4B model (arXiv 2501.04670). Frontier models are better; nobody has published numbers for app recordings.
No published study measures multimodal models on UI regression from recordings against a baseline. It is a gap to fill.

Build Stuff 2026

Control Your Fate at the Metal
Wednesday 2 December, 15:00. 45 minutes.

If you're coming, say hello. If you're not yet, 2026SPEAKER_20 takes 20% off a ticket.

#BuildStuff15 #buildstuffconf #AgeOfAgency @buildstuffconf https://buildstuff.events

We built a context system for our coding agents. They ignored it and used grep.

Kent Alstad — Mon, 05 Oct 2026 06:58:19 +0000

This is the first of a set of write-ups behind my Build Stuff 2026 talk, Control Your Fate at the Metal, Vilnius, 2 December. Details at the end.

A year ago we started giving our AI coding agents something better than search. The idea is the obvious one, and probably the same one you've had: a coding agent wastes an enormous amount of its context window reading files to find out what's in them. So index the repository ahead of time. Write a short description of every file. Build a call graph. Give the agent a tool that answers "which files matter for this task?" in one call instead of twenty.

We built all of it. Then we read 467 of our own agent sessions to see whether it was being used.

What the transcripts said

Over thirty days:

What we counted (30 days) Count
Times agents ran grep or rg through a shell 46,177
Times agents called our retrieval tool ~600
Sessions that called it even once 77 of 467 (16%)
Agents reached for grep about 75 times as often. Five sessions in six never touched the thing we'd built at all.

Three reasons, and none of them were "the model is dumb"

It only got used when something told it to. Nearly every call we could trace back to a cause came from a prompt, a hook or a workflow that named the tool explicitly. Tools that nothing named got almost no calls, regardless of how good they were. Adoption didn't track quality. It tracked whether some upstream instruction said the tool's name out loud.

If you've built an MCP server and wondered why nobody calls it, this is probably why. The agent sees a one-line description and chooses, right then, between a tool it has to reason about and a grep it already knows works.

Our matching rule couldn't fire on real input. Each file had a short pre-written summary, and we matched a request against it by word overlap — roughly a third of the words in the request had to appear in a 280-character summary. Real requests don't look like that. Somebody types "the login redirect is broken on mobile." The summary says "Handles OAuth callback routing and session establishment." Zero overlap, zero results. On one repository, our index put the file the agent went on to actually edit in the top five results zero times out of fifty-five.

We ranked on the summary instead of the thing it summarised. We had full descriptions of every file and were matching against the 280-character abstract of each one. When we scored both ways against real past tasks, ranking on the full text found the right file 17 times to 10.

The part that should worry you

An agent that greps isn't failing. It gets there. It opens four files instead of one, spends more of its window, takes a bit longer, and produces a reasonable answer. No error is thrown. Nothing in your logs looks wrong.

That's the trap. A retrieval system doesn't fail loudly — it gets quietly skipped while every dashboard stays green. Ours reported 100% acceptance and "14× better than baseline" during the month I just described. Both were artifacts of how we'd computed them. The honest acceptance number was 6.7%.

The "tokens saved" graph was worse. It was computed as what a naive agent would have read, minus what we returned. An empty result therefore booked almost the entire hypothetical saving. One tool claimed 16.9 million tokens saved in a month where three-quarters of its answers were empty. We were measuring an imaginary agent. The real one was grepping.

What I'd do differently

Build the thing that records what people actually ask before you build the index.

We spent a year on the supply side — descriptions, graphs, ranking — and never looked at the demand side. We had no record of what our agents were actually asked to do, so we had nothing to test a change against. Every improvement was a guess.

The fix was unglamorous: log each request alongside the files that session went on to read or edit. A backfill over ninety days gave us 5,415 requests and 1,578 request-to-file pairs. That's a test set. Now a change either moves the right file into the top five for real past tasks, or it doesn't ship.

The recorder took a fraction of the effort the index did. We should have built it first.

Talking about this at Build Stuff 2026

The context system in this article is one of the subsystems we chose to own rather than rent. What it took to make it actually work is part of the argument in my talk at Build Stuff in Vilnius this December, the 15th edition, under the theme The Age of Agency.

Control Your Fate at the Metal Wednesday 2 December, 15:00, Human OS Lab room. 45 minutes.

Your tooling depends on somebody's pricing page, rate limiter, and product roadmap. That used to be unavoidable. We went the other way — our own models, our own inference, our own registry, our own protocol implementations — until the only thing under us was hardware. Here's what it costs, what it buys, and where the metal pushes back.

If you're coming, say hello. If you're not yet, 2026SPEAKER_20 takes 20% off a ticket.

BuildStuff15 @buildstuffconf https://buildstuff.events

We used agents to draft this article and document this project as we built.

The Last Mile Problem in Agentic Development

Kent Alstad — Sun, 27 Sep 2026 08:21:01 +0000

Agents have made the first 80% of building software fast. Point enough of them at a well-defined problem, and you get working code, passing tests, and a system that looks almost done, sometimes in hours.

Then you hit the last 10 to 20%.

That's where I think the biggest problem facing agentic developers lives today: getting from 80 or 90% done to 100%. Not generating code, but hardening an idea all the way to completion. Edge cases, integration seams, failure modes, the gap between "it works in the demo" and "it works."

The math stops working

Here's the trouble. When hundreds of agents are building your system, they produce work at a scale no human can match. That's the point. But the last mile has traditionally been human work: review, verification, judgment about what "done" actually means.

So you end up with hundreds of agents producing output and one person, or a small team, trying to do the finishing work for all of them. You don't have the time to do the work of the hundreds. The bottleneck isn't building anymore. It's completing.

I hit this wall at KandiForge. Our agents had produced millions of lines of code across 20 repos, and I needed to ship. But how?

It wasn't a knowledge problem. I understood the requirements. I understood the architecture. What I couldn't do was work through every final detail across a codebase that size. There were simply too many of them, and each one needed the kind of attention that doesn't scale with the number of agents producing the code.

The agents had made building fast. They hadn't made finishing any faster.

Why "just review more" doesn't fully solve it

My first move was the obvious one. I added more reviewers and more testers, both agents and people.

The testers used agents to write tests, but generally after the code was already integrated. So the change gates weren't in the right place. By the time a test caught something, the change was already in.

I don't think anyone has fully solved this yet, myself included. We're all still working on it. That's part of why I'm writing this series: to share what we're trying at KandiForge, and to hear what's working for others.

What comes next

Over the next few posts, I'll share behind-the-scenes notes from how we're tackling this at KandiForge, where we build with an agent factory that follows an evidence-based SDLC across multiple repos. Some of it has worked. Some of it hasn't. I'll share both.

This series leads up to my talk at #BuildStuff15 in Vilnius this December (@buildstuffconf, buildstuff.events).

If you're building with agents: where does your last mile break down? I'd like to hear how others are handling it.