DEV Community: Dani Shemesh

Designing agentic development workflows: where it hurts, and what to build first

Dani Shemesh — Mon, 14 Sep 2026 14:39:10 +0000

Five parts of design, and design flatters itself. This one does not.

What follows is where the shape described so far still hurts, all of it observed in real runs, and then the short list of what to build first if you are starting one. If you read nothing else in the series, read the list.

Follows on from the whole series, but stands alone: every limitation names the thing it broke.

Where it hurts

Honest limitations, all observed in real runs, because a design description without them is marketing.

Enforcement exists only where the check exists. Every rule that mattered and was not a script eventually got skipped: the unchecked gate; the "always write a report" promise that silently covered only the enumerated exits; the "keep going" instruction that lost to an agent's instinct to check in. A single unanswered "shall I carry on?" abandoned a run that was already green, two phases from the finish. Politeness is indistinguishable from a crash. The uncomfortable lesson: an instruction to a language model is a strong default, never a guarantee, and anything load-bearing needs a script behind it.

Figure 1: Every rule that mattered and was not a script eventually got skipped. The amber columns are the ones that had only an instruction behind them.

Uniformity decays. In a family built over months, one workflow guards every seam and another guards none; one dispatches its work as tiered agents while another invokes the same steps inline, where they cannot be tiered, isolated or individually measured. The design stays coherent; the implementations drift from it at different rates, and nothing notices.

Adapters disagree at the schema. One component queried an identifier and never wrote it out; the component that needed it failed for want of it, and failed after the code had already been published. Two components, one implicit schema, no test at the boundary. If two things share a data shape, make the shape explicit and validate it, or it will drift.

History gets overwritten. A loop that rewrites its verdict file each iteration keeps only the final state, so the convergence (twelve failing, then three, then zero) is lost. That trajectory is exactly what you want when asking whether the loop actually works. Append; don't overwrite.

Signals get confused with their proxies. A platform flag meaning "this comment's anchor line moved" was read as "this comment is no longer true". Those are different claims and they came apart badly: items whose anchor had not moved were nonetheless judged against code that no longer existed. Before depending on a platform signal, ask what it literally measures.

Concurrency is limited by the target, not the workflow. Running several instances at once sounds free until you find that the shared working copy is single-occupancy during publishing, that the test database is one container which the first run to finish deletes out from under the others, and that container names are global to the machine. Most of what caps parallelism lives in the system being worked on.

Improvement is genuinely hard to measure. Elapsed time is confounded by caching; token counts by model changes; "number of findings" rewards a noisier reviewer; and an operator who has seen the task before is faster regardless of the tooling. What survives: whether specific, named frictions recur, and counts of expensive operations: environment boots, dependency re-locks, full-suite runs, loop iterations, human waits. Counts cannot be faked by a warm cache.

What no amount of care will fix

Everything above is a defect. Write the missing check, delete the duplicate copy, add the contract test at the seam, and the limitation goes away. The shape of the design does not have to change.

There is a second category that does not work that way, because it follows from running the procedure inside the agent session rather than around it:

The orchestrator cannot choose its own model, its effort or its provider, because a skill's metadata does not change what serves it.
It cannot manage its own context, so its judgment about its own procedure degrades over a long run, and the only remedy is to stop and ask a human to type /compact.
It cannot do two things at once, so a fan-out of parallel reviewers is sequential underneath.
It cannot be started by anything but a person at a terminal, which is the last prerequisite in part five and the one that caps how autonomous a workflow of this shape can become.
It cannot make a step deterministic, only the command inside it. A phase needing no intelligence at all still reaches its script through a model deciding to call it, so what you get is a repeatable command behind an unrepeatable decision: a round trip in cost, and some chance of the step not happening at all.
It cannot avoid having a lifecycle, so pending and paused and resumed and cancelled become yours to implement: incrementally, in the margins of doing something else.

No amount of better skill-writing reaches any of that. Whether it is worth reaching for something that does is a separate question, and a longer one: Should you harness the harness works through it.

If you are building one

The short list, in the order worth learning it:

Put every phase's output in a file, in a run directory, and pass that directory explicitly.
Guard the seams: assert an artifact exists before the phase that consumes it.
Make approvals content-addressed. A gate that trusts a flag is decoration.
Let a script decide when a loop stops.
Dispatch the judgment, inline the machinery: and put your strongest model on the decision nobody downstream can undo.
Prove the tests, not just the code.
Enumerate your terminal states and write a record on all of them.
Forbid the irreversible-in-public actions in code: history rewrites, resolving someone else's thread, merging.
Give "do nothing" a name, a required reason, and a score.
Build skills that run standalone. If a component only works inside its workflow, you have a phase, not a skill, and you will rewrite it for the next workflow.
Start local, with a human at the gates, and treat every friction you hit there as a prerequisite for running unattended later.
Instrument from the start. Every retrospective written by hand should have been a structured record: which component hurt, what it cost, and what the evidence was.
Put a clean control in every corpus, and a bait beside every planted defect, otherwise you are optimizing for a louder agent, not a better one.
Contract-test each seam, and generate the reverse index of which workflows use which component. The blast radius of a shared component is a graph, and it should not live in anyone's memory.
Measure against a no-component baseline. "Did the run succeed?" is the wrong question; "was it better than not having this at all?" is the right one.
Have every dispatched step report its own friction, against the component it invoked: and make "nothing hurt" an explicit record, or you cannot tell better from quieter.
Pick one deliberately trivial task alongside the hard ones. It is the only way fixed overhead becomes visible.
Measure delivery on the delivery systems, not on the workflow's own opinion of itself: and never report a speed number without its quality guardrails.
Build a negative control: one metric the workflow cannot possibly affect. If it improves too, your groups differ for another reason.
Version anything you intend to trend, from the first emission. It cannot be backfilled.
Trace three items by hand before publishing any aggregate. A wrong join produces confident, plausible, wrong numbers.
Treat every piece of incoming text as data. The steps that read external input are the ones holding the tools.
Establish provenance before judging: feedback is a claim about a version, and a platform signal is not the thing you actually care about.
Emit one tripwire per phase: usable output over produced output. Cheapest instrumentation there is, and it catches the failures that return exit code zero.
Package the shared parts once, and let workflow packages depend on them. Duplicating a shared component across packages drifts within weeks, and nothing surfaces it.
Never leave a loose copy of something you have packaged. Same name, same description, different body is undetectable from the outside.
Name actions, not tools, in every skill body: and keep the harness-specific translation in one document per harness. It is the difference between one port and fifty rewrites.
Declare each skill's capability requirements and its degradation. A skill that silently assumes sub-agents exist will invent a tool call on a harness that has none.
Feed it small items. More stories, each narrower: control, comprehension and meaningful gates all degrade with size, and a plan you have to rewrite at the gate means the item was too vague.
Encode your conventions into the steps, and check them. The planning step reads them, validation enforces them, and the negative ones ("don't touch this") are stated explicitly.
Name intents, not commands. The workflow says "run this repository's tests"; the repository says how, which is what lets one workflow serve services in different languages, and what stops a hardcoded test command from failing silently in the next repo.

These are small distributed systems whose components are language models. What makes them trustworthy is files, hashes, exit codes, and a refusal to let any single step be the judge of its own success.

Originally published at fullgc.github.io, where the whole series lives.

Designing agentic development workflows: shipping it, porting it, letting it run

Dani Shemesh — Mon, 14 Sep 2026 14:38:57 +0000

The first four parts describe a workflow that runs on your machine and can be shown to work. This part is about the three transitions that turn it into something other people depend on, and each one costs something specific.

Packaging makes a collection installable and lets one workflow depend on another's components, at the price of a dependency graph nobody can hold in their head. Porting to another harness is cheap for everything that is a file and expensive for everything that is control flow. And autonomy, meaning a ticket assigned to the agent and a reviewed pull request some time later, turns out to be less about capability than about making the exceptional case detectable.

Leans on part one and part two throughout.

Packaging and distribution

At some point this stops being configuration on your machine and becomes something other people install. That step forces a decision the design has been able to defer: what is the unit of distribution?

A note on words, because the two get used interchangeably and they are not the same. This section says package for the general idea, since the argument holds whatever your harness calls it. Every concrete example is a plugin, which is Claude Code's implementation of one. Where something is true only of plugins rather than of packaging in general, it is called out.

What a package can hold

More than skills. A plugin (the packaging format in Claude Code, and the concrete example throughout this section) can carry skills, agents, slash commands, hooks, MCP servers, LSP servers, background monitors, executables that go on PATH while it is enabled, default settings, and a schema for configuration it asks the user for at install time. It can also carry arbitrary payload (dashboards, docs, scripts) and reference its own files through ${CLAUDE_PLUGIN_ROOT}, with ${CLAUDE_PLUGIN_DATA} for state that must survive updates.

That matters more than it sounds. A workflow's whole apparatus can travel as one unit: the orchestrator, the skills it composes, its agents, the hooks that measure it, and the dashboard that reads what they emit. Not just the prompts. The dashboard is the odd one out, riding along as plain files rather than as something the runtime loads, which is what ${CLAUDE_PLUGIN_ROOT} is for. See the plugins guide and reference.

Figure 1: One install brings the whole apparatus. The runtime loads the top row; the bottom row rides along as files, which is why a bundled dashboard has to find itself through ${CLAUDE_PLUGIN_ROOT}.

One package, or one per workflow?

This is a distribution question, not an architecture question, and the answer depends on who consumes it, so the trade-off matters more than any default:

	One package for everything	One package per workflow
Versioning	one number for unrelated changes	independent cadence per workflow
Consumers	everyone gets all of it	a team enables only what it uses
Blast radius of a release	the whole family	one workflow
Shared components	trivially one copy	must live somewhere, the hard part
Hooks	unambiguous	composition across packages is undocumented

Enable/disable is per package, not per component, you cannot ship five workflows in one package and let a team turn four of them off. That alone makes the monolith uncomfortable once more than one team consumes it.

The fact that decides it: packages can depend on packages

A manifest accepts a dependencies array: bare names, or names with a semver range. I verified this against the validator rather than taking it on trust:

{ "name": "workflow-x", "version": "1.2.0",
  "dependencies": ["wf-core", { "name": "wf-telemetry", "version": "~2.1.0" }] }

Install a package whose dependency is missing and the install fails and names what to install rather than half-working. Conflicting ranges from different packages are intersected, and an unsatisfiable combination fails loudly instead of silently picking. Depending across marketplaces is blocked unless the marketplace explicitly allows it. And a package may consist of nothing but a dependency list, which is the documented way to ship a bundle. See plugin dependencies.

That is what makes the split viable. Go one package per workflow and the components several workflows share have to live somewhere, and every obvious home is bad: copy them into each package and they drift, put them in one workflow's package and the others depend on an unrelated workflow, keep everything together and you lose per-team enable/disable. A dependency array removes the dilemma. The shared layer becomes its own package, versioned on its own cadence, and each workflow package declares a dependency on it.

That also settles what granularity to choose. Without dependencies, the shared code dictates it: you keep things together because splitting means duplicating. Once the shared layer can stand alone, that constraint is gone and you are free to decide by who installs what.

Package granularity should follow the consumer boundary, not the code structure.

The question to answer is not "how is this organized?" but "who installs it, and do different installers want different subsets?" That gives a decision rule rather than a preference:

One consumer group, everything used together → one package, one version. Granularity you don't need is pure coordination cost: more manifests, more version numbers, more ways for the set to be half-installed. Do not split a collection because it looks like several things.
Different consumers want different subsets, or parts need different release cadences → split, with the shared layer as its own package that the workflow packages depend on. This is the point at which per-package enable/disable and independent versioning start paying for their overhead.
Either way, the shared layer exists exactly once. Whether it is a directory inside one package or a package of its own, two copies of a shared component is the failure mode below, not a strategy.

Two things make the split cheap if it comes later, and both are worth doing up front even when you don't need them yet:

Reserve the original name for the bundle. If the package everyone installs is X, then when you split, X becomes a package that is nothing but a dependency list. Consumers keep typing the same install command and never learn that the internals changed.
Keep the internal boundaries visible before they are enforced. Directories that already separate "shared" from "this workflow only" split cleanly. A flat pile does not.

When the conditions above do call for a split, this is the shape it takes:

Core: the deterministic primitives (run state, seam guards, source-control isolation), the shared judgment skills (plan, implement, validate, take-approval, write-failing-tests, publish), and the measurement hooks. One copy, one version, one owner.
Per workflow: the orchestrator, the skills only it uses, its agents. Depends on core with a range.
Bundle: dependencies only, under the name consumers already type.

Three traps

1. Duplication drifts silently, and faster than you expect. One observed case: a workflow packaged with twelve skills, eight of them shared with other workflows that were still distributed loose. Three weeks later, five of the six shared skills had diverged: the packaged copies three to seven lines behind, with the same name and a byte-identical description but a different body. Nothing surfaces that. The lesson is not "don't package": it is one source of truth, and packaging is downstream of it. Never edit the packaged copy, and never leave a loose copy of something you have packaged.

2. Skills get namespaced. Agents don't. A packaged skill is addressed as package:skill, which makes collisions impossible, but it is also a rename, so every cross-reference in your instructions has to survive it. Worse, if a loose copy of the same skill still exists, it keeps the short unprefixed name and answers to it, which is exactly how two divergent copies coexist unnoticed.

Agents have no namespace at all, and resolve by a priority ladder in which packaged agents rank lowest, below project-level and user-level ones (sub-agents). So if a workflow installs its agents into the user directory and ships them in a package, the user copies quietly win and the packaged ones are dead weight. Pick one home for agents and stick to it.

3. Hook composition across packages is undocumented. Whether hooks from several packages all fire on the same event, and in what order, is not specified anywhere I could find. Do not build a design that depends on it. Put hooks in exactly one package, which is a good constraint regardless, because measurement is cross-cutting and therefore belongs in core.

What packaging buys beyond convenience

Version identity. A run can record which package version served it, which is what makes the "version anything you intend to trend" rule from part four actually implementable.
Team enforcement. A marketplace declared in project settings plus an enabled-packages list gives everyone the same set, updating on its own, no "did you copy the new one?" ritual (marketplaces, installing).
Opt-in by default. A package can install disabled, and prompt for the configuration it needs at enable time.
A package boundary is also a test boundary. The "without the component" comparison from part four is naturally per package: does this one earn its place?

Context cost is not the deciding factor

A package contributes its skills' names and descriptions to the prompt. Five packages of ten skills and one package of fifty cost about the same, because all fifty names appear either way. Split for ownership and release cadence, not to save context. What genuinely costs context is MCP servers, whose tool definitions are present on every turn, so bundle those deliberately, and separately.

Migration order

Whichever granularity you land on, the order of operations is the same, because each step de-risks the next:

One source of truth in the repository. Whatever copies exist elsewhere (hand-installed directories, an older package, a colleague's machine) reconcile and delete them first. Packaging a collection that already has two divergent copies just makes the divergence permanent, and this is not a packaging problem.
Core package first: shared spine plus hooks, published and consumed by nothing.
One workflow package, depending on core, taken end to end: install, run, measure.
Then the rest, plus the bundle.
Record the package version in every run, so the next question, "did this get better?", is answerable at all.

Portability across harnesses

Everything so far has one unstated dependency: the harness: the agent runtime that loads the skills, dispatches the sub-agents, fires the hooks and enforces the permissions. That is a real lock-in risk, and it is worth knowing which parts of the design carry over and which parts are the port. (Portability across repositories, the same workflow against services in different languages and on different platforms, is the other axis, and it is the "name intents, not commands" rule in part two.)

What is already standardized

More than people assume, as of 2026:

The skill format. Agent Skills is an open specification: a folder with a SKILL.md (YAML frontmatter plus a Markdown body) that can bundle scripts and references. It was published in December 2025 and is stewarded through the Agentic AI Foundation; by 2026 30–40 tools read the same files from the same directories, across competing vendors. Notably, .agents/skills/ is the one location every major tool reads natively.
The instructions file. AGENTS.md is the cross-tool convention for "how to build and test this project", now under the same foundation and read by dozens of agents (field guide). Some harnesses still prefer their own filename and import it.
The tool layer. Anything you build as an MCP server works across compliant runtimes without a rewrite. That is the portable way to expose a capability.
The packaging format. An Agent Plugins standard was announced in August 2026 by several vendors together: a root manifest, fixed locations for skills and MCP configuration, and a namespaced escape hatch for client-specific behavior. Worth tracking against whatever native format you package in today.

What is not standardized, and won't be soon

Capability	Portability
Skills (content, frontmatter, bundled files)	Portable: same format, same directory
Project instructions	Near-portable: one convention, minor filename differences
Tools, via MCP	Portable
Sub-agent dispatch	Divergent: isolated agent files, "personas", TOML definitions, persistent modes: different models of what a sub-agent even is
Hooks / lifecycle events	Divergent: event sets differ sharply, and one harness is far richer than the rest
Slash commands	Fragmented
Model and effort selection per step	Harness-specific
Permissions, sandboxing, background tasks	Harness-specific

The pattern is consistent: content ports, control flow doesn't. The more a capability is about orchestration, the less portable it is.

Figure 2: The file based half ports for free. The orchestration half is the port.

The one rule that buys portability

From obra/superpowers' porting guide, which is the most concrete write-up of this problem I have found, and whose central invariant is worth adopting verbatim:

Skills name actions, not tools.

Write "read the file", "run the command", "invoke the planning skill", "dispatch a sub-agent", never the harness's actual tool name. Porting then means adding a tool-mapping document for the new harness that translates each action into its real tool names, and it never reaches into a skill body to swap names. Their companion rule: everything ships through the harness's own install mechanism, and you never edit the user's files.

This is testable, and worth measuring on your own collection. In one measured collection, skill bodies almost never named a tool (three of fifty-four mentioned one) while the orchestration layer was the opposite: a dozen guides named the skill-invocation tool directly, and two dozen agent definitions carried harness-specific model, effort and tool-allowlist fields. Which is exactly the split you would predict, and it tells you where a port's cost lands.

What a port actually costs

Per harness, roughly four things:

Bootstrap: how the entry-point instructions reach the model. The superpowers guide treats automatic session-start injection as a hard requirement: if the only way to get your framework in front of the model is a per-session opt-in, that harness cannot be properly supported.
Discovery: how skills are found: a native skill mechanism, a plugin registration API, a manifest field, a generated index, or documenting "read the SKILL.md directly" as the sanctioned path.
Tool mapping: one document per harness. Get the tool names from the harness itself, not from its docs.
Distribution: the harness's own install channel.

Degrade explicitly; never fake

When a harness lacks a capability, the skill must say do it inline, or report the missing capability, never invent a tool call that doesn't exist. Their word for this is degradable, and the useful discipline is deciding the degradation in advance:

No sub-agents → run the step inline, and state plainly that the fresh-context property is lost. That is a real reduction in quality, not a cosmetic one.
No hooks → measurement moves to a scheduled reader over the artifacts, which part four argues you need anyway for anything happening after the session.
No task-tracking tool → a plan file.
No per-step model selection → the tiering in part three collapses; say which decisions are now running on the wrong tier rather than pretending the design is intact.

Why this design ports better than most

Not by luck: the earlier principles do the work:

Artifacts are the interface, and the run directory is the only channel. Files are the most portable thing in computing.
Gates are content hashes in a file. No harness feature required, so the whole approval mechanism survives a port untouched.
The primitives are scripts, and the loop referee is a script. Exit codes are universal.
Terminal states and reports are files.

What genuinely doesn't survive: per-step model and effort tiering, hook-based measurement, and the sub-agent isolation that several principles depend on. So the honest summary is that the file-based half ports for free, and the orchestration half is the port.

A portability checklist

Keep skills in the standard format and the standard location.
Name actions, not tools, and grep your own skills to check.
Keep exactly one tool-mapping document per harness, outside the skills.
Keep state in files, not in harness memory.
State each skill's capability requirements explicitly, with a declared degradation for each.
Put everything harness-specific behind a single boundary (the packaging manifest, the hooks file, the agent definitions) so a port touches a known, small set of files rather than fifty skill bodies.

From local to autonomous

Local-first is the starting position, not the ceiling. For some workflows the destination is unattended operation, and the shape of that destination is worth stating concretely.

The target flow

A ticket is assigned to the agent and moved into a queued state. That transition, not a human typing a command, is the trigger. The agent picks the ticket up, runs understand → plan → failing tests → build loop, and opens a pull request. The review and the response to review are part of the chain, not separate human errands: the review workflow reviews the change, the fix workflow triages those findings and addresses the ones that deserve it, and only then does a human look. What they find is a pull request that has already survived a review round and been revised once, with a written record of what was changed, what was argued against, and why.

The gates do not disappear; they move

This is the key transition, and it is a design job rather than a switch:

Today: a human approves every plan.
Tomorrow: a policy approves the ordinary plan and a human approves the exceptional one, and it approves it through the same mechanism, by recording the approval the way a human would (part four), so the gate check itself never learns there was nobody there. "Ordinary" is machine-checkable: the plan touches no protected file, stays inside a size budget, introduces no new dependency, carries no unresolved open question. Anything else escalates to a person.
The workflow's real job becomes making "exceptional" detectable. That is where the design effort goes, and it is why the plan artifact needs to be structured rather than prose.

Some gates stay human permanently

The list is the same one from part two: rewriting shared history, resolving someone else's review thread, merging, and anything that speaks on the team's behalf. Autonomy means removing the approval step from reversible work, never from irreversible-in-public work.

Figure 3: The gates move; they do not disappear. A policy approves the ordinary plan through the same recorded mechanism a human would use, so the gate check never learns there was nobody there, and the publish gate never changes hands at all.

What must be true first

Every one of these is a prerequisite, and every one of them was learned from a run that had a human watching:

Every rule enforced by a script. An unattended run has nobody to notice a skipped gate. The unenforced-gate story in part two is a nuisance with a human present and a silent data-loss event without one.
No question that stops the run. "Shall I carry on?" is fatal without an audience. An autonomous workflow either proceeds, or reaches a terminal state and reports; there is no third option.
Terminal states and a report on every exit, including abandonment. The report is the only thing a human will read, and for most runs it will be the only thing they read.
Environment-vs-test failure classification. Otherwise an infrastructure blip consumes the whole budget and produces a confident, wrong verdict.
Isolation strong enough for concurrency. Separate working copies, no shared mutable services, no two runs racing on one database or one container name.
Restraint that holds without supervision. The review-and-fix leg must be able to decline a wrong suggestion when nobody is there to back it up. This is the hardest prerequisite and the one most worth evaluating explicitly.
Cost and iteration ceilings, because nobody is watching the meter.
Structured telemetry per step, so a fleet of runs can be audited after the fact instead of inspected one at a time.
An eval suite you actually trust (part four), including a clean control. Unattended operation without one is not automation: it is unmonitored production.
Something other than a person to start it. A ticket transition can only be a trigger if some process is listening for it and can begin a run, and that process has to outlive any terminal session. This is the one prerequisite on the list that better skills cannot supply.

The honest reading: most of the work between "useful locally" and "trustworthy unattended" is not model capability. It is enforcement, isolation, classification and reporting: the boring parts, again. The exception is the last item, and it is a real one. Everything above it can be built inside the skills; a runner cannot, because a procedure that lives inside an interactive session has nothing to start it. Part six returns to that.

That is the workflow packaged, ported and, eventually, left to run on its own. What is left is the part a description like this owes the reader, and it is part six: where the design still hurts, and the short list of what to build first.

Originally published at fullgc.github.io, where the whole series lives.

Designing agentic development workflows: is it working, and is it worth it

Dani Shemesh — Mon, 14 Sep 2026 14:38:37 +0000

Everything so far has been design, and design argues for itself. This part is about evidence, which does not.

Three independent ways to know whether a workflow works are worth building, and they answer different questions. Most teams build one of them and then over-claim from it. An external corpus asks whether a step's output is correct. The workflow reporting on itself asks where the process hurts, costs nothing, and is available on day one. Delivery metrics ask whether anything shipped sooner and at what price, which is the question a sponsor asks first and the hardest of the three to answer honestly. Each has a characteristic way of being gamed: a looser grader, a quieter agent, a confounded comparison group.

Assumes the artifacts and terminal states from part two, because every measurement here reads them.

Testing and evaluation

"How do you test this?" is really four questions. A workflow contains four kinds of thing, and each needs a different kind of test.

The thing	Example	How you test it
A script	run state, a seam guard, branch setup	like any other software: run it, check the exit code
An adapter	code that reads or writes an external platform	check it still gives its consumer what that consumer needs
A judgment step	deciding, reviewing, classifying, planning	you can't assert an opinion. Score it on tasks whose answers you know
The workflow itself	the whole thing, end to end	run it on a small task; check the result and the leftovers

The two dead ends are trying to unit-test an opinion, and trying to eyeball a script. Sort the component into one of those rows first and the method follows.

Where to start

In this order. Each step is cheaper than the next.

Validate the structure: does every component parse, and do its file references resolve? A second per commit.
Test the scripts: one test per documented exit code, plus "run it twice and nothing changes the second time." Most of these get re-entered after a failure, so repeating safely matters.
Test the seams: for each pair of components that pass data, check the producer still emits what the consumer needs.
Smoke-test one whole workflow on the smallest real task you have.
Then build a scored task set for the judgment steps. Most valuable, most expensive: do it once you know what failure looks like.

The seams are where things quietly break

Borrow contract testing from microservices: the consumer writes down what it needs from the producer, and that expectation runs as a test on both sides whenever either changes. Only what the consumer actually uses gets pinned, so the producer stays free to change everything else. See Pact or this overview.

Not theoretical: in one observed case a component fetched an identifier and never wrote it out. The component that needed it failed for want of it, after the change had already been published. Two components, one unwritten assumption, no test at the boundary.

This is also the answer to "a component is used by five workflows, what did I just break?" Keep a generated map of which workflows use which component, and let a change select the suites to run.

Scoring a judgment step

You can't write an assertion against an opinion. You can assemble tasks whose right answers you already know, run the step against them, and score it. The task set is the work; the scoring is easy.

Most of this is from Anthropic's Demystifying evals for AI agents, which is worth reading in full:

Start with 20–50 tasks taken from real failures: your bug tracker, not your imagination. Early on, changes have big effects, so a small set discriminates fine.
A task is only usable if two people who know the domain would give the same verdict. Otherwise you are measuring your own ambiguity.
Write down a known-good answer for each task. It proves the task is solvable and that your scoring works: the two problems that otherwise look like a weak agent.
Include tasks where the right answer is "do nothing." Most people skip this. If every task has a problem to find, the best-scoring agent is the one that always finds something. Include correct code that looks suspicious, and one task with nothing wrong at all.
If everything fails, suspect the task. A 0% score across many attempts usually means the task is broken or ambiguous, not that the agent is incapable.
Grade the result, not the steps taken. Checking that an agent followed a particular sequence of tool calls is, in Anthropic's words, "too rigid and results in overly brittle tests", capable models keep finding valid routes nobody planned for. If you do care about the path, check only that it didn't do anything extra.
Read a few transcripts before believing any score. A number can rise because the scoring got looser.

Two task sets are worth having: a planted one, where you introduce problems deliberately and record the answers, as your gate; and a replay one, which costs nothing to write: take real finished work that already has real human feedback, run the workflow against how things looked then, and compare.

If a model does the grading, spot-check its verdicts against human ones now and then, and watch for it drifting lenient, a wrong answer marked correct inflates everything downstream.

Living with randomness

The same input doesn't give the same output twice, so one run tells you little. Run each task a few times and decide which you need: at least one attempt succeeds (fine when you only need one good answer), or every attempt succeeds (when consistency is the point). Those two diverge fast as attempts increase, so say which one you are quoting.

Then two practical rules:

Start every attempt from a clean slate. Leftover files and warm caches cause failures that look real but aren't.
Compare the outputs, not just the pass rate. A step that finds three good things, then three different good things, can score identically each time and still be impossible to depend on.

Cheap checks that punch above their weight

Tripwires. Emit one number per phase: how much of what it produced is actually usable. The example that earned this a permanent place: a review step produced twelve findings, and none of them pointed at a real file and line. Nothing would ever be posted. Every exit code was zero, so no task set would have caught it. One counter made it obvious.

Relations you can assert with no answer key. When you can't say what the right output is, you can often say how two outputs must relate:

shuffle the input order → the decisions should be the same
feed the same item twice → it should be deduplicated
feed something with nothing wrong → the output should be empty

These need no task set, take minutes to write, and catch instability single examples never will. (The formal name is metamorphic testing.)

Testing a workflow that needs human approval. This looks impossible, since something has to approve the plan. The wrong fix is a "skip the gate" flag, which removes the very thing you wanted to cover. Instead, have the test write exactly the approval record a human would have written: same file, same format, same hash. The gate check downstream passes unchanged, because it cannot tell the difference and doesn't need to. Only the source of the approval changes. The same trick is how a gate later becomes an automatic policy (part five).

Where the test cases come from

Before building any of it: read what actually happened. Sample real runs, note what went wrong in plain words, group those notes into named failure categories, and count how often each occurs. Thirty to fifty by hand is the usual starting point.

The one-line version, from Hamel Husain and Shreya Shankar's work on evals, is hard to improve on: look at your actual data, by hand, before building any automated metric. A task set built without this tests the failures you imagined; one built after it tests the failures you have.

What your runtime may already do for you

Check before building anything. A capable agent runtime may offer:

A "without the component" comparison: run the same task set twice, once with the component installed and once with it removed, and compare the scores. This is an ablation, and it is the single most valuable measurement available, because it answers "does this thing earn its place?" rather than "did the model succeed?" A workflow can score 80% on your corpus while your planning skill contributes nothing: the model was getting there anyway. You only find that out by taking the skill away and running the same twenty tasks again.
Separate grading for "did it trigger" and "did it do the right thing": two different bugs, and the first is easy to miss entirely.
Deterministic graders: pattern matches, "was this tool called", "was this file created", alongside model-based scoring.
Repeats, pass thresholds, cost ceilings and machine-readable output, which is all CI needs.

Claude Code has a claude plugin eval command that does exactly this: its --ablation with-without mode runs a no-plugin baseline arm and reports the delta. As of 2.1.259 it is early access and undocumented, so not everyone has it and none of the detail is guaranteed to hold. Worth watching rather than building on, because a runtime that runs the comparison for you turns the most valuable measurement here into a flag.

What they generally don't give you: checks partway through a long workflow, partial credit, or any way to assert which sub-agent ran on which model. For those, split the workflow into one case per phase, or check the files the run left behind, a good reason for phases to write files.

Five checks that need no test set at all

Assert these after any run, with no answer key:

did it finish in one of its declared end states, rather than just stopping?
does every file it promised exist?
was every gate satisfied by a real approval?
did the loop stay inside its limits?
were there any stops that weren't designed: a question that halted it, a missing input?

If you build nothing else, build these five and the component map. They are cheap, need no labeled data, and between them catch most of the ways a workflow quietly rots.

Making the workflow evaluate itself

Every technique so far grades a workflow from the outside: build a corpus, run it, score the output. That is indispensable and expensive: the corpus is the work, and you cannot build one before you know what failure looks like. Which is the well-known bootstrapping problem: the teams who most need evals have no labeled data yet.

There is a second axis, and it inverts the arrangement. Have the workflow report on itself. Each dispatched step records where it hurt; the orchestrator sums up its own run; those records aggregate across many runs, per workflow and per component. No labeled data, no answer key, and it runs on the real distribution of work rather than the one you imagined.

The two are complements, not alternatives:

	External evals	Self-reported friction
Question answered	is the output correct?	where does the process hurt?
Needs a labeled corpus	yes, the expensive part	no
Distribution tested	the one you authored	the one you actually get
Scales with	authoring effort	usage
Available	after you know the failure modes	immediately
Can it be gamed	by a looser grader	by a quieter agent

Crucially, the second bootstraps the first. The taxonomy that self-reporting produces is precisely the open-coding-then-axial-coding step a corpus needs, except the agents do the open coding as they go, instead of a human reading traces weeks later.

Exercise the design deliberately

Passive collection tells you about the work you happened to do. To learn about the design, choose the work so it spans the design's axes. A useful spread for a family of workflows:

one item requiring a schema or data migration (exercises irreversibility and generated artifacts);
one that is pure logic, deliberately trivial;
one that cannot be completed without touching a protected or "ask first" file (does the workflow notice the constraint, or edit through it?);
one that changes a published interface (exercises back-compat judgment, which is what reviewers actually argue about).

The trivial one is not filler; it is the control. If a task with no migration, no new model and no generated files still costs the full apparatus (every phase, every gate) that is fixed overhead, and fixed overhead is invisible unless something cheap is measured beside something expensive.

What a self-report has to contain

Free-text retrospectives do not aggregate. A structured record does, and the field list is where the value is:

The component it is about: not the phase. This is the primary axis; without it every finding collapses to "the workflow is annoying". Validate the name against a known list, or you will get typos and the non-answer "the workflow".
Impact (what it did to this run: blocked / caused rework / merely slow) kept separate from severity (what it would do to a future one). They are orthogonal: a trivial papercut can block, and a serious latent gap can cost nothing today.
Evidence, required and non-empty. It is the only thing separating a finding from an opinion: a command and its output, or a file and a line.
A signature, so the same problem reported by four runs collapses to one row.
Who reported it, which is both a compliance denominator and the difference between a step's own view and the orchestrator's outside view of it.
Optionally a suggested fix, optional on purpose, because requiring one invites invented ones.

The nil protocol

A step that hit no friction still files a report: the same self-report described above, carrying an explicit "nothing hurt" value rather than simply not being written.

Without it, silence and not-bothering are indistinguishable, and those two mean opposite things. A required nil also gives the count a denominator: three frictions across forty steps, rather than three across an unknown number. The failure it prevents is specific. Fix a component, watch the reports drop, and you cannot tell whether the component improved or the agents went quiet. That is the same trap named earlier for corpora without a clean control, and it is the cheapest safeguard in the design.

The rollup, and what its columns must not hide

Aggregate by (component, signature), then print, per group:

count and distinct runs, separately. Four records from four runs is four independent confirmations; four from one run is one run tripping repeatedly. Same count, opposite meanings.
breadth: which workflows reported it, by name. This column is the answer to "is this a shared design fault or one workflow's wording problem", and it is how a fault in a component used by five workflows earns five times the attention.
both raw and per-workflow-normalized severity. Shared components will accumulate more findings simply by being used more; printing only the raw total confirms that hypothesis by construction, and printing only the normalized one hides the real total cost. Print both and say which you are quoting.
a reporting-hygiene section: runs that filed neither findings nor nils, findings with suspiciously thin evidence, unknown component names. This is the part that keeps the rest honest, and it is the only place hard targets belong: reporting rate and evidence present, never friction counts.

Then fix, and run the same work again

The improvement loop closes by re-running the same items on the improved workflows. That comparison is worth something only if the second run is genuinely a second run:

fresh sessions, with no transcript of the first in context;
reset state: the tracker items returned to their starting condition, prior artifacts archived rather than left where an autodetect glob can find them, caches and databases cleared;
the software tagged at both points, and anything installed-by-copy re-installed, or the second wave silently runs the first wave's components;
the task specifications untouched. If the first run revealed that an item was ambiguous, resist clarifying it. That is the most tempting and most invalidating edit available;
the previous attempt's traces removed. Agents have been observed gaining an unfair advantage by reading the version-control history of earlier trials, so a same-task re-run must not leave the first attempt's commits lying around.

What the comparison may honestly claim

Two measures survive the confounds:

Recurrence of named frictions: per item fixed, a binary: did it come back? Most direct, least confounded, and it needs no baseline arithmetic.
Unplanned stops per run: anything that is not a designed gate: a question that halted the run, a stuck terminal state, a missing artifact, a guard firing.

Then, as supporting evidence: operator interventions, terminal states, loop iterations, artifact completeness, gate integrity, and counts of expensive operations (environment boots, dependency re-locks, full-suite runs, human waits), which are cache-proof, unlike elapsed time.

Not evidence: wall-clock, token cost, raw finding counts, lines changed. And report per item; never average across items, because difficulty varies and the average hides the interesting cases.

State the confounds rather than burying them: the sample is small, it is one codebase, and the operator has seen the work before, which no process controls. The claim available is "these specific frictions were removed", not "the workflow is better". Where a claim does reach beyond the workflow's own internals (that delivery got faster, say) it needs the comparison discipline below, especially the negative control.

Failure modes of the method itself

Self-report bias. A step is a poor witness to its own confusion. The orchestrator's outside view of a step, and the step's own view, are different data, collect both and keep them distinguishable.
Only dispatched steps can report as separate voices. A step invoked inline by the orchestrator has no independent context, so its "self-report" is really the orchestrator's. If per-step attribution matters, that is another reason to dispatch the step.
Telemetry must never be able to fail a run. Give the recorder exactly two outcomes (recorded, or bad usage) and keep it off every failure path. A measurement layer that can break a run gets switched off within a week, and then you have neither.
Never set a target on friction counts. "Fewer than ten findings next time" rewards silence.
Separate harness friction from workflow friction. Findings caused by how you ran the experiment (contention between parallel runs, a shared container, an expired credential) are about your harness, not the design. File them under a distinct label or they will misdirect every fix that follows.

Why this is the highest-yield thing to build first

It needs no corpus, so it is available immediately; it produces the failure taxonomy that a corpus later requires; and its output is self-sharpening: a named friction that recurs is a regression test that wrote itself. Fix it, record the signature, and every later run checks for its return at no cost.

Measuring whether it delivers

The first two axes are inward-facing. Corpus evals ask whether a step's output is correct; self-reported friction asks where the process hurts. Neither can tell you whether anything was delivered sooner, or what it cost, and that is the question a sponsor asks first.

Axis	Question	Data needed	Gamed by
External evals	is the output correct?	a labeled corpus	a looser grader
Self-reported friction	where does the process hurt?	nothing; it self-instruments	a quieter agent
Delivery metrics	did it deliver, and at what cost?	the tracker, the forge, runtime telemetry	survivorship and confounded groups

Most teams build exactly one of the three and then over-claim from it. They are not substitutes.

Four streams, and a division of labor that is easy to get wrong

Delivery measurement draws on four independent sources, and each owns something the others cannot supply:

Stream	Owns	Grain
Runtime telemetry	spend and token volume, sessions, active time	session × component × model
Workflow telemetry	runs, outcomes, loop iterations, referee decisions, early exits, component calls	run × work item
Issue tracker	the work item's lifecycle: created, transitions, comments, links	work item
Code host	branches, commits, pull requests, reviews, CI, deployments	pull request / branch

Two rules follow, and both are counter-intuitive enough to state plainly:

Cost only ever comes from the runtime. Nothing in a workflow's own telemetry prices tokens. What the workflow stream contributes is the dimension the cost stream lacks, which workflow, which work item, which outcome. Runtime cost alone has no work-item dimension at all, and that gap is the entire reason both streams exist.
Delivery outcomes only ever come from the tracker and the code host. Workflow telemetry says what a run did; it can never say whether delivery actually changed.

The join, and what breaks it

The spine is the work-item key. The tracker has it natively; the workflow stream carries it; the code host yields it from a branch-name convention (with the PR body and commit messages as fallbacks). Cost joins only indirectly (session → run → work item) which is the join no single system can perform alone, and the reason instrumenting the workflow is a prerequisite for costing anything.

Three consequences worth designing for up front:

A workflow with no work-item key breaks the spine. Anything triggered by a repository or a pull request rather than a ticket must emit an alternative key, or its cost attaches to nothing.
Some joins no system provides. The wait at a human gate (the approval record carries a hash but no timestamp, as noted above); the exact link from a run to the pull request it produced (inferred from a branch name unless the run stamps its id into the PR body or a commit trailer); and the identity map between tracker account, code-host login and commit email, which is a hard prerequisite for anything per-person.
Cardinality is many-to-many. One item may have several runs and several pull requests. Fix the rules once (earliest start, latest merge, summed iterations) and exclude bot authors everywhere.

Figure 1: Cost only ever comes from the runtime, and the runtime has no work item dimension at all. The middle link of that chain exists only if the workflow records it, which is why instrumenting the workflow is a prerequisite for costing anything.

One row per unit of work

Reduce all of it to a single record per work item: created, started, first commit, PR opened, first review, merged, deployed, done; time in each status; reopens; review rounds; size and files touched; CI attempts; runs, outcome, iterations, gate rejections; and spend.

Every metric that follows is then a subtraction or a count over that row, which is what keeps a metric catalog from turning into a pile of bespoke queries.

Four lenses, and one rule about reading them

Usage: is it actually being adopted: share of eligible items with a run; repeat use, which separates "tried it" from "adopted it" in a way a user count hides; funnel drop-off showing where runs die; the outcome mix.
Efficiency: cycle time (started → done, excluding backlog), lead time, build time, time in each status, time to merge, review wait, rework rounds, discussion depth, size.
Quality: first-pass CI success, reopen rate, escape rate (a defect linked back within some window), revert rate, follow-up churn, change failure rate.
Cost: spend per delivered unit, spend on runs that produced nothing, waste ratio, spend per component, spend per loop iteration.

Efficiency is never reported without quality in the same view. A speed number shown alone invites precisely the trade nobody wants, and the invitation is hard to refuse once the number is on a slide.

The elapsed-time paradox

The previous section says elapsed time is not evidence; this section makes cycle time a headline. Both are right, and the distinction matters:

A run's wall-clock is confounded by caches, warm environments and machine load. Use operation counts instead.
A work item's cycle time is measured between events days apart in external systems. No cache touches it.

Same word, different clocks. Never mix them in one chart.

Comparing assisted with unassisted

The moment there is a number, someone will compare it to the unassisted case. The comparison is worth doing and easy to fake, so fix the discipline before the first chart:

Label every work item as unassisted, assisted-any (a run was started, whatever became of it), or assisted-completed.
Compare assisted-any against unassisted. It is tempting to compare only the runs that finished, but runs are abandoned precisely on the items that turned out to be hard, so excluding them removes the workflow's worst cases and manufactures a win. Report the completed subset separately, labeled "when it completes", never as the headline. This is the single most common way these numbers lie.
Compare like with like. Same item type, same window, bots excluded.
Band by size and by repository, and compare within band. A large change in a repository with a slow suite is not comparable to a small one elsewhere. Without banding, a difference in which items people chose looks exactly like a difference the workflow made.
Summarize honestly. Median and p90, never the mean; show the difference with an interval, and if it spans zero, say "noise"; print both group sizes and show nothing for thin cells.
Pair people against themselves. For anyone with items in both groups, compute their own median difference and check whether most people moved the same way. Differences between people (seniority, familiarity, how they scope work) are usually larger than the effect being measured.
Run the same comparison on a metric the workflow cannot possibly affect. The lag between merging and someone finally dragging the item to done is pure bookkeeping; no coding workflow changes it. If the assisted group looks "better" there too, the groups differ for some other reason (most likely easier items being chosen) and the headline cannot be trusted. A negative control is the cheapest protection against fooling yourself, and almost nobody builds one.
Show quality beside speed, always.

Even with all of it, this stays observational: people chose when to use the workflow. You can describe a difference; you cannot claim a cause. A causal claim needs randomization (alternating by week, for instance) which is worth doing once a baseline exists.

Anti-metrics

Tempting	Why not
Run counts, invocation counts	Activity, not delivery. Trivially inflated
Token volume	An input. Use spend per delivered unit
Completed-runs-only timing as the headline	Survivorship; see step 2
Total early exits as a failure count	A gate refusing to proceed is the gate working. Failure is a terminal state, not an exit
Lines of code	Rewards volume, which this tooling produces cheaply
Any per-person KPI	These are program metrics. Per-person views exist for the paired analysis, not for evaluating people

And set no targets before a baseline exists. Targets set in advance are either trivially met or quietly abandoned. Measure for a full delivery cycle, publish the baseline with its spread, then set targets.

Instrumentation: the artifacts are the source; hooks are only the trigger

This closes a loop the design opened much earlier. Part two made artifacts the interface between phases for composability. It turns out they are also the only trustworthy measurement surface: in-session hooks observe tool calls, not workflow semantics, so anything a workflow concluded (an outcome, a verdict, a count) must be read from the file it already writes for its own correctness. No metric should ever depend on parsing prose.

The practical shape: choose the one moment when a run is definitively over, read the artifacts there, and emit everything at once, guarded so a single run cannot emit its terminal batch twice.

Two classes of signal no in-session instrumentation can reach: delayed human reaction (a thread resolved, an item moved days later) and downstream effects (whether the change was reverted, whether a defect was linked to it). Those need a scheduled reader over the artifacts and the external streams. If a workflow's most important number is of that kind, the scheduled reader is part of the measurement design, not a later nicety.

Rules for whoever adds a metric

Increment by exactly one per observed event. Never accumulate a computed quantity into a counter.
Emit numerator and denominator separately, never a ratio. A pre-divided rate cannot be re-aggregated across runs.
Mirror the failure path wherever you handle the success path. A crashed step otherwise leaves a span that never closes and a window that never shuts.
Watch cardinality. Anything unbounded (a file path, a finding title, a branch name) does not belong in a dimension.
Version anything you intend to trend. A prompt or rule change alters what a workflow produces; without a version dimension, a trend line silently splices two different systems together. It cannot be backfilled, so it has to be there from the first emission.

Shared metrics, owned metrics, and the tier that gets over-claimed

Most signals should be shared: one definition emitted by every workflow, sliced by workflow name. Resist the urge to fork them per workflow: a new terminal state belongs in the shared vocabulary, not in a private metric.

But workflows should also carry their own metrics, because the interesting quality signal is usually specific to what that workflow produces: a reviewer has precision and recall over findings; a build loop has iterations to green; a triage step has restraint. Name those distinctly so they cannot be mistaken for a shared series, and keep them scoped to the workflow that emits them.

That gives three tiers, and the distinction governs how each may be read:

Tier	Meaning
Program	One definition across every workflow. Comparable
Owned	Meaningful only for this workflow, measured on its own runs and artifacts
Attributed	A program metric measured on delivery artifacts, restricted to work that used this workflow

Attributed metrics are the ones that get over-claimed. They are measured on work items and pull requests, not on runs, so they would read identically if something else entirely caused the improvement. Report them as "the items that used this workflow", never as evidence about the workflow alone.

One constraint that should govern every dashboard: cost is comparable across workflows; quality is not. Spend per run ranks any two workflows sensibly. Quality metrics are defined against what a workflow produces and have no counterpart in a sibling, so cross-workflow panels should carry only the shared run-and-cost signals, and every quality panel should be scoped to one workflow.

Trace a handful by hand before publishing anything

Before any aggregate goes on a dashboard, follow three to five known items end to end manually (one assisted, one unassisted, one that went wrong) and check that the record matches what actually happened. A single wrong status mapping produces confident, plausible, entirely wrong numbers, and nobody downstream will question a chart.

Continuous cross-checks earn their keep for the same reason: items marked done with no linked pull request, pull requests merged whose item never completed, branches matching no item, negative durations. Each one is a broken join, and a broken join fails silently.

A workflow that works on your machine, and that you can prove works, is still a private tool. Part five is the three transitions that make it something else: packaging it so others can install and depend on it, porting it to a different harness, and taking the human out of the gates that can safely lose one. Part six is where all of it still hurts.

Originally published at fullgc.github.io, where the whole series lives.

Designing agentic development workflows: the cost of control

Dani Shemesh — Mon, 14 Sep 2026 14:38:13 +0000

Two parts in, the machinery is settled: phases in order, artifacts between them, gates that check rather than promise, a script deciding when the loop stops. What is not settled is the arithmetic around it, and the arithmetic decides whether any of the machinery matters.

How large should the item handed to a run be? How many approvals can one person give before approving stops meaning anything? Which steps deserve the strongest model, and which decisions can never be recovered downstream if that choice is wrong? This part is short because the answers are short. They are also the answers most often got wrong, because each one is a place where doing the efficient thing costs you the control the workflow was built for.

Assumes only that gates and loops exist and are enforced, which is part two.

Sizing the work

The most consequential decision about a run happens before it starts: how big is the thing you handed it?

The recommendation is deliberately smaller than feels efficient: more items, each narrower. Not because the workflow cannot handle a large one, but because everything that makes a run trustworthy degrades with size:

Control. A small change is one a person can actually review. You can hold the whole diff in your head, which is the only state in which "approved" means anything.
Comprehension. With a narrow item you can tell what was written and why. With a sprawling one you approve a summary of what was written, which is a different and much weaker act.
Gates become real. A gate on a focused plan is a decision. A gate on a plan spanning nine files and three concerns is a rubber stamp, and the attention arithmetic in the next section only gets worse as items grow.
Plans get concrete. A narrow ask produces a plan naming files and tests. A broad ask produces a plan of intentions, which is unreviewable and unfalsifiable.
Failure gets cheap. A run that ends stuck costs you one small item, not a week.

The cost is real and worth stating: more items means more runs, more gate touches, and more fixed overhead per unit of delivered work (part four argues for measuring exactly that). Accept it deliberately. The alternative is not less work: it is the same work with less control over it.

A usable rule of thumb: if you cannot state the acceptance criteria in a few lines, the item is too big. Split it before running anything.

Plan quality is bounded by input quality

The single largest lever on what a workflow produces is not the workflow. It is the description of the work handed to it.

A vague item produces a confident, vague plan, and then the gate has nothing to push back on, because there is no specific claim to disagree with. Three things make the difference:

Acceptance criteria specific enough that the plan cannot invent scope. If two people would build different things from the description, the plan is a coin flip.
The context the agent cannot discover for itself: why this is wanted, which constraints apply, which decisions are already settled and not up for reconsideration.
Pointers into the codebase: the module this concerns, the convention it must follow, the thing nearby that already does something similar.

A good diagnostic: if you find yourself rewriting the plan at the gate, the item was underspecified. The fix belongs in the description, not in the plan, otherwise the next run makes the same mistake.

Human gates and the economics of attention

Gates are the product, not the overhead. Everything downstream faithfully implements whatever a gate approves, so an error at a gate cannot be recovered later.

Which is also why gate placement is not a free choice: put them where the process already stopped for a human. A gate somewhere the process never had one gets treated as ceremony and rubber-stamped; a missing gate where the process always had one is the one people notice, loudly, after something ships.

The shape worth aiming for, stated as a promise to whoever runs it: two stops. Everything between them is unattended, however many retries it takes. And it never merges. That is a claim a person can hold in their head, which matters more than the exact number: they need to know when they will be needed and what the workflow will never do behind their back.

Which is precisely why the number of gates matters. Three gates in one run is defensible. Twenty-eight across a batch of parallel runs is not. Attention is the scarce resource, and a human asked to approve twenty-eight artifacts approves them without reading.

Batch by phase, not by run

If a gate check only asks "is this artifact's hash in this run's approval log", then several parallel runs can present their plans together and each still records its own approval separately. A handful of sittings instead of dozens, with the enforcement unchanged.

A gate asks exactly one question: approve, or annotate

Never bundle a decision into it. On one run a gate asked for approval and offered a choice between two implementation routes, one of which added a runtime dependency. The reply was "approved", which answered only the first question, and the right branch was taken by luck. Discrete choices belong in an explicit question before the gate, so a bare "approved" cannot be ambiguous.

When batching, require an answer per item; a blanket "all approved" against four artifacts is ambiguous, and ambiguity at a gate should be treated as feedback, not consent.

Measure the attention, not just the elapsed time

Batching four approvals into one sitting can quarter the attention each receives, which shows up later as extra loop iterations that look like workflow friction. Record how long a human actually spent per artifact, separately from how long the machine ran.

Figure 1: Attention is the scarce resource. A human asked to approve twenty eight artifacts approves them without reading.

Model and effort tiering

Because the tier is chosen per dispatched agent, spend deliberately:

Where	Tier	Why
Deciding whether an external claim is correct	strongest model, high effort	the only step that can produce the restraint outcomes, and nothing downstream can recover them
Root-cause investigation	strong model, high effort	every later phase inherits its errors
Writing the tests that define "correct"	strong model	those tests are the specification
Building, proving, wording replies	mid tier, medium effort	machinery: the decisions were already made
Fetching, parsing, classifying a payload	cheapest tier	near-deterministic, high volume

Cost and trust are both configuration. The same file that picks the tier also declares the tools the step may use, and that second field is a trust boundary, not an optimization. A step that writes the run's report needs to read and write files and nothing else: no shell, no search. Denying it a shell is how you know it cannot edit code, regardless of what it decides it wants to do. Grant the narrowest set that lets the step finish, and prefer removing a tool to adding an instruction telling the step not to use it.

One firm rule: keep a floor on effort. Make a step cheaper with a smaller model, never with less thinking. A small model thinking hard beats a large model thinking barely, and the failure mode of minimal effort is confident wrongness rather than visible struggle.

Note the asymmetry that justifies the whole table: the cheapest tier is also the most likely to conclude "the reviewer is right, do what they said": precisely the failure the expensive tier exists to prevent. Tiering is about where the irrecoverable decisions live, not about average difficulty.

What the design buys

That is the design: the components, the shape of a run, the principles that keep that shape honest, how work is sized, where the human stands, what each step costs to run, and how failures are classified.

The largest claim was made at the very start: what the person invoking it stops having to do. That is the adoption argument, and it outweighs everything below. What follows is what the engineering buys.

Interruptibility. A run can be stopped at any phase and either resumed from its cursor or simply read. Nothing important lives only in a transcript.
Attributable failure. Guards and exit codes mean a failure names the step that caused it, instead of an agent quietly proceeding on missing input and failing three phases downstream.
Auditability. The approval log records what a human approved, by content hash and timestamp. The publish record says what actually reached the outside world. The report explains the run to someone who never watched it.
Composability. Because phases talk through files, workflows chain, skills stand alone, and any phase can be replaced by a human doing it by hand.
Compounding improvement. Shared skills mean a fix in one place improves every workflow that uses it, and makes it worth knowing which shared component hurts most.
Bounded cost. Iteration ceilings, retry caps and escalation limits mean a run cannot grind indefinitely; the failure mode is a clear stop, not a runaway bill.
Reviewable output. One commit per unit of work, messages that reference the originating item, and a diff confined to the files the plan named. The reviewer's first question, "why is this file in here?", is answered before they ask it.
The ability to decline. That is the whole difference between a system that addresses feedback and one that merely obeys it.

Every one of those is a claim, and a claim you have not checked is a hope. The rest of this document is about checking them, shipping them, and being honest about where they fail.

That is the design, and the case for it. Whether any of it is true of your workflow is a different question with a different kind of answer: evidence. Part four is the three independent ways to get it, and the way each one is quietly gamed.

Originally published at fullgc.github.io, where the whole series lives.

Designing agentic development workflows: principles for a workflow you can trust

Dani Shemesh — Mon, 14 Sep 2026 13:25:00 +0000

A workflow's phases and its cast are the easy half. The hard half is keeping that order honest when every step is carried out by something that would rather be agreeable than correct.

What follows are the principles that hold that order in place, in two groups. The machinery comes first, then the doctrine that governs what a step does with its own judgment. Many of them exist because something went wrong in a way that produced no error.

This part assumes part one's vocabulary: orchestrator, skill, agent definition, channel adapter, deterministic primitive, and the run directory. Part one introduces them.

Machinery principles

Where state lives, how one phase hands work to the next, what a gate has to be to count as one, and who is allowed to decide that a loop has finished.

1. The orchestrator is thin, because only a sub-agent can carry a model

In current agent runtimes, the model and reasoning effort serving a step can only be set when that step is dispatched as a sub-agent. A skill's own metadata does not change what serves it.

That single platform fact drives the architecture. The orchestrator runs on whatever model the session happens to use, so it stays cheap and mechanical (control flow, gates, dispatch, wiring), and every act of judgment is pushed into an agent that declares its own tier. When a design wants "this decision deserves the strongest model and maximum effort", the only way to express that is to make the decision a dispatched agent.

Generalized: find the unit your runtime lets you configure, and make that unit the boundary of judgment.

2. The run directory is the only channel

Every run gets a durable directory, and that directory is the only channel across a dispatch boundary. Inline steps share the orchestrator's context and could pass things in memory; a dispatched sub-agent cannot. Design for the boundary and the inline case comes free.

The reason is mechanical: environment variables and the working directory do not survive a call into a sub-agent, so the run directory's path is passed as a literal argument into every dispatch, and anything a later phase needs must be on disk. Run state lives in one small JSON file, written atomically (write a temp file, then rename) so a crash cannot leave a half-written record.

That pays off twice. A run becomes inspectable by anyone, including a human with no access to the transcript. And it becomes resumable: if the state file carries a phase cursor, re-entering the workflow continues from where it stopped instead of restarting over a human's earlier approvals.

3. Artifacts are the interface between phases

Each phase produces a named file and the next phase consumes it: the request, the exploration, the plan, the tests, the implementation notes, the validation record, the report.

One small detail worth stealing: when a phase transforms a file, give the output a different name from the input. A step that writes its result over its own input will, on a second run, read its own output, and any deduplication or comparison it was doing silently stops working.

Because the interface is a file, a phase can be re-run, inspected, replaced by a human doing it manually, or swapped for a different implementation without touching its neighbours.

4. Guard the seams

Before a phase runs, assert that the artifacts it depends on exist and are non-empty. A tiny script; exit zero or one; nothing else.

This exists because the characteristic failure of a language model at a seam is not a crash. It is producing the answer in the conversation and never writing the file (or writing it somewhere else) after which the next phase proceeds on absent or stale input and fails somewhere unrelated. A guard turns a silent, mis-attributed failure into a loud, correctly-attributed stop.

The step most worth guarding is the one that writes the final report, because that is the step most likely to hand you prose instead of a file.

Figure 1: A guard does not prevent the failure. It moves the failure to where the cause is.

5. A gate is a check, not a promise

A human gate presents an artifact, takes annotations, and records approval as the artifact's content hash in an append-only log. Before acting on that approval, the workflow re-computes the hash and refuses if it is absent.

Keying on content rather than on a flag buys two properties. A revised artifact has a new hash, so a revision requires fresh approval instead of inheriting the old one. And an approval cannot be manufactured by an agent that believes it was approved.

The workflow that fixes review comments on a pull request had three gates, of which only the first was actually checked, the others in prose. On a real run the third gate, the one guarding everything that reached the outside world, was skipped: the change was pushed to a pull request under review and not one of its ten review threads received a reply. Reviewers were left with silently-changed code and no explanation. The fix was not more prose; it was one shared check called at all three gates, plus the same check inside the two components that publish, so a mis-sequenced publish fails instead of succeeding quietly.

Figure 2: Keying approval to content is what makes a gate a check rather than a promise: a revision cannot inherit an approval, and an agent cannot manufacture one.

6. Judgment runs in fresh context

Any step that judges work done earlier in the same run is dispatched fresh, so it never inherits the context of the step that produced what it is judging. In the workflow that answers pull request review comments, that means the agent deciding whether a comment is correct has not seen the agent that wrote the code under review. The same goes for the agent that proves the tests, and the one that writes the report.

This is a bias control, not a token optimization. An agent that just spent twenty minutes writing a function is the worst available judge of whether that function is wrong.

7. The loop is refereed by a script, not by the agent inside it

The build phase is a two-tier loop: an inner loop of implement→validate, and an outer loop back to re-planning. After each pass, validation writes a structured verdict (status, failure class, tests run, how many of them were the new ones, a reason, the evidence) and a deterministic referee reads it and returns exactly one of:

CONTINUE           run the next iteration
RETRY_MECHANICAL   run the same iteration again; do not re-plan
ESCALATE           go back to planning, re-approve the new plan, resume
STOP_OK            proceed to the next phase
STOP_STUCK         hand back to the human

The referee is a script because a loop that decides its own termination condition does not terminate reliably. It also enforces the guards an agent inside the loop would rationalize away:

False green: no new tests actually ran, so nothing was demonstrated. Escalate.
Same failure signature twice: the approach is wrong, not the attempt. Escalate rather than retry.
An iteration ceiling, and a separate cap on identical retries: bounded work, always.

The failure class in that verdict is what lets the referee pick correctly instead of retrying blindly:

Tactical: the implementation is wrong. Retry with a change.
Mechanical: the same attempt failed for a transient reason. Retry identically, capped.
Strategic: the plan is wrong. Escalate to re-planning, and re-approve the new plan.
Environmental: a credential expired, a container collided, a dependency service is down. This is not a test failure. Stop and surface it.

That last category is the one everybody omits and everybody needs. On one run an expired credential surfaced as a dead test command; validation classified it as a failing test, and the loop spent two iterations "fixing" code that was already correct, grinding against a wall only a human could move.

Figure 3: A loop that decides its own termination condition does not terminate reliably. The failure class picks the arrow: tactical continues, mechanical retries, strategic escalates, environmental stops.

8. Tests come first, and then the tests are proved

Every plan names the failing tests to write. They are written first and confirmed failing before any implementation exists. That part is ordinary test-driven development.

The part that is less ordinary: after the loop goes green, a prover reverts only the production files, re-runs the new tests expecting them to fail, restores the change, and records a verdict: proven, or vacuous. A test that still passes with the change reverted did not test the change.

Without this step, "the tests pass" and "the tests test something" are indistinguishable, and the second is the only one anybody cares about. It is also the step that catches the most human-looking failure mode there is: a test written to satisfy a process rather than to catch a defect.

(The refactor workflow is the instructive exception: it writes no new tests, so its equivalent guarantee is that the existing suite passes unchanged, and any diff in test behavior is a defect.)

Figure 4: A test that still passes with the change reverted did not test the change.

9. Only the adapters know the outside world

The components that reason read and write files and never touch the network. The adapters are the only platform-aware code. The orchestrator wires them together.

That split is what lets the same workflow run driven by a live pull request or by a plain artifact on a branch with no pull request at all, without the second being a degraded path. The deciding and building phases are identical in both, because neither can tell the difference.

10. Terminal states, and a report on every exit

Enumerate the ways a run can end (done; done-but-unproven; nothing-to-do; bad-input; stuck; not-approved) and write a report on every one, including the boring ones. A request whose work turns out to be already complete is a finished run, not an error, and it still deserves a one-line record.

"Done but unproven" earns its place as a distinct state: a run that shipped and explained itself but could not demonstrate that its tests catch the defect is neither done nor stuck. Naming that state is what stops it being rounded up to "done", which is exactly how a vacuous test ships looking verified.

Watch for two things here. A promise to "write a report on every exit path" only ever covers the enumerated exits; a run abandoned mid-flight leaves nothing unless something else writes a record. And if the runtime treats certain filenames specially, a sub-agent may be unable to write the very file it exists to produce: check that the reporting step can actually write to the name you chose.

Doctrine principles

What a step should refuse to do, how it should treat text that arrives from outside, and how it should record a decision so the decision can be audited later.

11. Restraint is a first-class outcome

For any workflow that receives input from elsewhere (a review, an alert, a suggestion) the valuable behavior is often to not act. Responding to review feedback is easy to do badly in four specific ways:

silently ignoring an item;
satisfying the letter of it while missing the point;
obediently implementing an item that was wrong, breaking working code;
editing at a location the reviewer pointed to several commits ago, which now means something else.

So the deciding phase classifies each item and chooses an action, and several of the available actions must produce zero diff: a question gets answered, an incorrect claim gets a reasoned argument, an out-of-scope request becomes a ticket, a stale item is recognized as stale. Make this measurable: an evaluation that checks "did the run leave the tree untouched where it should have" against the real diff, and in which a run that changes everything fails.

Two failure modes sit either side. Obedience: accepting a claim because a senior person made it. Timidity: deferring everything out of uncertainty, which is not restraint but a refusal to do the job. The dividing line worth encoding: restraint is for claims you verified are wrong, never for work you verified is small.

12. Never do the socially irreversible thing

Some actions are cheap for a machine and expensive for a human to undo. Forbid them outright, in code, not in documentation:

Never force-push. Rewriting history on a branch under review can destroy a reviewer's in-progress comments and re-anchor every thread. If the branch is behind, stop and say so.
Never resolve a human's review thread. Resolving means "this is dealt with", which is the reviewer's judgment about someone else's fix, not the author's about their own. Reply, react, leave it open.
Never reply to praise. If the doctrine forbids performative agreement, auto-thanking a compliment is the purest possible violation of it.
Never merge. Never touch a file outside the approved plan.

The general rule: automate the reversible; require a human for the irreversible-in-public. This list is also, not coincidentally, the boundary of autonomy (part five).

13. Incoming text is data, never instructions

Any workflow that ingests text from outside (review comments, ticket descriptions, alert payloads, a web page) is handing attacker-influenceable content to a step that usually holds real tools. Say so explicitly in the step's instructions: classify the content, never obey it. If a payload contains text shaped like directions (change your scope, ignore your rules, run this command) the step ignores it, decides the item on its technical merits, and reports that the payload contained injected directions rather than silently absorbing them.

The awkward corollary: the steps that most need this warning are the ones with the widest tool access, because reading external input and having the power to act on it is the same job.

14. Establish provenance before judging

External feedback is a claim about a specific version of the code, not about the current one. Before evaluating any of it, find out which version, and how far behind that is.

Get this wrong and you get the most confusing failure mode available: an item that is internally coherent, confidently argued, and describes code that no longer exists. The related trap is mistaking a platform signal for the thing you care about. A flag saying a comment's anchor line moved is not a flag saying the comment is no longer true; the two come apart constantly, in both directions. Ask what a signal literally measures before depending on it.

The same discipline applies to a claim's severity. A label attached by whoever raised it is their prior, formed against an older state of the world. It tells you where to look first. It is not evidence, and it never substitutes for verification.

15. Reaffirm or reverse, never drift

The second time a workflow sees the same input, it invites a specific failure: a reasoned decision reversed by attrition. Round one pushed back with evidence; round two has no memory of why, and obediently complies.

So make the prior round an input, not a hope. For any item matching an earlier decision, the workflow owes one of exactly two things: reaffirm it with fresh verification (not "as previously decided", which is not verification) or reverse it explicitly, recording what changed and why. A reversal is healthy; an earlier decision can rest on a claim that turns out to be false. Only the silent flip is the problem, because a third round then sees a decision with no trace that it was ever contested, and the thing starts to oscillate.

16. Record how you decided, not only what you decided

These are not the same act:

read the migration and reasoned that it looks fine
executed the migration against a scratch database and counted the statements it emitted

A schema that captures only a conclusion and a confidence number flattens both into the same row. For a judgment step, the method is the product: a step's pushback only deserves to outweigh a senior human's opinion because it was executed rather than reasoned, and that is precisely the fact the human at the gate most needs to see. Record it as a field (executed, read, or searched) with a detail line naming what was actually done. It also makes a lazy run detectable: an "executed" claim with no detail is a smell.

Related, and easy to get wrong: a label must not lie at the gate. A human scanning a summary counts categories, so if "the tests are missing" is filed under the same label as "the code is broken", the artifact actively misinforms at the moment it is supposed to inform. Give the distinct thing its own category, and require a claim of runtime harm to name a reachable path from a real entry point, otherwise it is latent at most, and must be labeled that way.

17. Report what contradicts the brief you were given

A dispatched step's return contract should lead with whatever changes what its caller believes, not with a summary of what it did.

If a brief asserted that an item was a real defect and the step discovered the function does not exist: that is the headline, and it must be volunteered rather than buried under counts. Obeyed literally, a return contract that says "report the totals per category" produces exactly that burial.

Two consequences for whoever writes the brief:

Do not pre-classify in the brief. "This is a genuine production bug, not a test gap" anchors the one step whose entire purpose is to determine that. Ask for attention, never for a verdict.
Environment facts in a brief are stale by the time they are read. "The credentials are expired, skip that step" may have been true at dispatch and false a minute later. A step should re-probe any environmental assertion at the point of use, and report what it observed rather than what it was told to expect.

18. Encode the standards you want, and then enforce them

A workflow is the highest-leverage place to put "how code is written here", because it applies on every run instead of depending on who happens to be driving. That is one of the main reasons to build one at all, but it only counts if it is implemented rather than assumed:

the planning step must actually read the repository's own conventions (its instructions file, its layering rules, its protected files) and name the relevant ones in the plan;
validation must run the real gates: formatter, linter, type checker, tests;
if there is a review step, one lane of it should be about standards specifically;
and state the negative conventions explicitly: what not to touch, what not to introduce, which files are extend-only. Those are the ones a model will cheerfully violate, because nothing in the code says "don't".

A convention nobody checks is a wish. This is the same rule as everywhere else here: enforcement exists only where the check exists.

Done properly, this is the difference between "the agent produced something that works" and "the agent produced something that looks like ours", and the second is what makes the output reviewable by the people who own the codebase.

19. The workflow knows the procedure; the repository knows itself

A workflow should be writable once and runnable against services in different languages and on different platforms. Nothing about "explore, plan, gate, red tests, implement, validate, prove, report" is language-specific, so nothing language-specific should be written into it.

Which means the concrete facts about a service do not belong in the workflow at all:

how dependencies are installed;
how the full suite runs, and how a single test runs;
how the formatter, linter and type checker are invoked;
how the service is started locally, and how migrations are applied;
the repository's own rules: layering, protected or extend-only files, naming, what not to introduce.

All of that belongs in the repository: in its instructions file (AGENTS.md, by the convention in part five) or in the files that file references. The workflow names the intent; the repository supplies the command. This is the same invariant as "skills name actions, not tools", one level up: name intents, not commands.

So a validation step says "run this repository's test command for the scoped tests" and "run the gates this repository declares", not pytest -k, npm test, go test ./..., or a particular package manager's install line. One useful indirection: if each repository exposes a single well-known entry point (a task runner, a Makefile, a scripts/ directory) then the workflow's "run the tests" is one stable command regardless of the language underneath, and the per-repository answer lives in one file instead of scattered through skill bodies.

The failure this prevents is not a crash, which is the recurring theme here. A hardcoded command works exactly as long as every service resembles the one the workflow was built in; the first service in another language does not fail loudly: the wrong command exits non-zero or prints no tests, and the agent reasons confidently over a signal that means nothing. Guarding the seam catches a missing file, not a misinterpreted one.

Run two cheap tests. Grep your own skills for language- and toolchain-specific tokens: test runners, package managers, build tools, framework entry points; every hit outside a per-repository file is a portability bug. Then run the workflow against a service in another language: the failures are precisely the assumptions you never knew you had written down.

The cost is honest and worth stating: this moves work onto the repositories, and a workflow is only as portable as the weakest instructions file it meets. A wrong answer there is worse than a missing one, because the agent will follow it. So apply the same discipline as a harness port (part five): when the repository does not declare something the workflow needs, stop and say which fact is missing, never guess a command from the shape of the directory tree.

Figure 5: The workflow names the intent; the repository supplies the command. The left column is identical for every service, which is the whole point.

These rules describe how a run should behave. They say nothing about how much work to hand it, how many approvals a person can meaningfully give, or which steps deserve the expensive model. That arithmetic is part three.

Originally published at fullgc.github.io, where the whole series lives.

Designing agentic development workflows: what a workflow is, and what it is made of

Dani Shemesh — Mon, 14 Sep 2026 12:45:42 +0000

What the series is for

Give a team an AI coding tool and you have not given them a way of working. Everyone gets the same agent and uses it differently: one engineer explores the codebase before planning, another goes straight to code, a third writes the tests afterwards if at all. The same engineer does it differently on a different day. The phases live in people's heads and get reconstructed from scratch every time, and nothing in the tool knows that your team reviews a plan before implementation, or what this repository's conventions are, or which mistake the last change of this shape made.

The attention goes wrong in both directions, too. People sit and watch a model reason through a decision they were never going to argue with, then look away at the one point where their judgment was the only thing that mattered, and find out afterwards that it took a route nobody would have approved.

Getting a model to write code is the easy part. The part that needs designing is getting it to do the same thing twice, to stop where you want it to stop, and to leave behind something you can actually review.

This series is about how to design that: a family of agentic development workflows, what they are made of, why the pieces are shaped the way they are, what the shape buys, how you know any of it works, how you ship it to other people, and where it still hurts. It comes out of workflows we have designed, built and run on real work. Nothing here is about a particular product or codebase; the claims are about the pattern.

How the series is organized

	Part	Question it answers
1	What a workflow is, and what it is made of	what the thing is, and what a single run looks like
2	The principles	what keeps the shape honest: the machinery, and the doctrine
3	The cost of control	how big an item, how many approvals, which model, and what the whole apparatus buys
4	Is it working, and is it worth it	is the output correct · where does the process hurt · did it deliver
5	Shipping it, porting it, letting it run	packaging, harness portability, and the road to unattended runs
6	Where it hurts, and what to build first	the limitations, and the short list

What we mean by a workflow

In software development, "workflow" is a heavily overloaded term. Throw "agentic" in front of it and get two opposite things, so it's worth defining exactly what we mean before building anything around the concept.

The distinction that earns its keep is Anthropic's: a workflow is a system where models and tools are orchestrated through predefined code paths, while an agent is a system where the model dynamically directs its own process and tool use. Workflows buy predictability on work you can decompose in advance. Agents buy flexibility on work you cannot.

This series is about workflows in that sense, applied to software development. The part worth noticing is that they are not the opposite of agents. They contain them.

An agentic development workflow is a named, phased procedure that takes a unit of work from a request to a reviewable result, in which three parties each do what they are good at:

The procedure owns the path. Phases and their order are decided before the run, not chosen by a model halfway through it.
Agents own the judgment inside each phase. What to explore, what the plan should be, whether a review comment is correct.
A human owns specific decisions, named in advance, that the workflow cannot make for itself.

None of the three is sufficient alone. A path with no judgment in it is a script, and judgment with no path is a prompt.

Figure 1: A fixed order, a file out of every phase, and a person required at exactly two points.

The interesting claim is not that a model can write code. It is that a procedure makes a model's work inspectable and interruptible at points you pick in advance, and that something other than the model's own discipline enforces those points. How you build such a procedure is a separate question with several answers, and the one used throughout this series is described below.

That combination is what makes the following six properties reliable rather than aspirational. You can ask a bare prompt for most of them, and you will often get them:

Guarantee	Mechanism
A fixed execution path	phases in a declared order, not whatever the model improvises
Scripts over inference	anything that can be decided deterministically is, and by a script
Auditable state	every phase writes a file; the whole run is readable afterwards
Absolute human authority	gates the workflow cannot approve for itself
Isolation	the work happens somewhere that isn't your working copy
Right-sized effort	each step runs at a deliberately chosen model and effort

Two of the six are different in kind, not just degree. Absolute human authority cannot come from an instruction, because the thing you would be instructing is the same thing that would have to enforce it. And right-sized effort is decided by how a step is dispatched, so a step cannot ask for it on its own behalf. The other four you can get out of a well-written prompt on a good day. What the workflow adds is getting them on every run, including the bad days.

Those six run through everything below. Every later section is really an argument about how to keep one of them true under pressure.

The phases come from your development process, not from the tool

A development workflow maps a development process the team already follows. Something is requested; someone works out what it means; they decide an approach; somebody agrees to it; it gets built; it gets verified; it goes out for review; the review gets answered; it merges. The workflow does not invent that sequence, it encodes it.

Three things follow, and they remove most of the guesswork from designing one:

You transcribe phases, you don't design them. If your process investigates before planning, so does the workflow. The order is not a modeling choice.
Gates go where the process already had a human checkpoint. A plan someone signs off, a review someone performs. Don't invent checkpoints your process doesn't have, and don't quietly drop the ones it does.
The artifacts are the ones the process already produces: a plan, a list of tests, a description of the change, a review. That is why people who never ran the workflow can still review its output. It looks like what they already read.

Two more things are worth stating plainly.

If your process is implicit, encoding it forces you to write it down. That is often the most valuable side effect of the whole exercise, and usually the hardest part, because the disagreements about how work should flow only surface when someone tries to make them executable.

And don't invent a process for the agent. A workflow that follows a sequence nobody actually uses produces output nobody trusts, and its gates land where nobody wants to look. If a phase exists only because the workflow needed a phase there, delete it.

This is also why a family of workflows shares a skeleton and diverges in specific places: the underlying process is shared, and the differences are real differences in how the work is done.

One approach among several

This post describes one way to build an agentic development workflow: an orchestrator skill that drives other skills, with the agent runtime as the execution engine and prose as the control flow. It is not the only shape, and the alternatives are not strawmen:

Approach	Strength	Cost
Orchestrator skill (this post)	legible to both the people who own it and the model running it; no build step; any step runnable by hand	prose is a weak way to express control flow
Code-first orchestration: the flow is a program that calls the model	precise, strongly typed at the seams, testable with ordinary tooling	heavier to change; the logic is opaque to the model itself
Graph / state-machine frameworks	excellent when the flow genuinely is a graph	overhead when it is a straight line with a few gates
One instructions file, no phases	the cheapest thing that works, and it often does, for small changes	no gates, no artifacts, nothing to inspect afterwards
Per-task commands or prompt templates	trivial to write and understand	fine until a task needs several steps to be trustworthy
Event-driven bots	no human has to start anything	a trigger, not a procedure; it still needs one of the above to run
Autonomous multi-agent crews	maximum flexibility	minimum auditability, and very hard to gate

The case for prose-orchestrated skills is maintainability: the people who own the process can edit them, not only the people who can modify a program. The price is that prose expresses control flow poorly, which is exactly why every part that has to be deterministic (the loop referee, the seam guards, the gate checks) is a script and not an instruction.

Pick by what your team can actually maintain. A precise flow nobody edits is worse than a legible one they do.

Local-development first

These workflows are local-first by design. They run on a developer's machine, against their checkout (or an isolated working copy beside it), with the developer present at three or four decision points. That is the primary mode, not a stepping stone to tolerate until the infrastructure arrives.

Four reasons to be deliberate about it:

The human is the highest-value component at a handful of specific moments: approving a plan, judging whether a review comment is correct, deciding what gets published. In between, almost worthless. Running locally makes those moments cheap to insert.
Failures are visible and cheap. When something goes wrong the developer is right there, the artifacts are on their disk, and they can open them in their normal editor.
Nothing needs to be built first. No queue, no runner, no service account. A workflow is useful on the day it is written.
It is how you earn the right to run unattended. You learn where a workflow breaks while somebody is watching. Every limitation in part six was found with a human sitting in front of a run.

The end game is autonomy, for some of them

For some workflows (implementing a feature is the clearest case) the destination is unattended operation: a ticket gets assigned to the agent and moved into a queued state, and some time later a human finds a pull request that has already been reviewed and revised once.

That is a real design target, not a fantasy, and it changes what the design in between has to look like. Part five covers the trajectory: which gates become policies, which stay human forever, and what has to be true before you let a run proceed with nobody watching.

A family of workflows, one skeleton

Build workflows as a family rather than one at a time, because most of what they do is the same and the differences are informative. They share a skeleton for the reason given earlier (the development process underneath them is shared) and they diverge exactly where the real processes diverge. Six concrete examples, all real shapes:

Workflow	Trigger	What is distinctive	Ends at
Implement a feature	a ticket	explore the codebase, plan, write failing tests, build loop	a reviewed branch, then a pull request
Fix a bug	a ticket, or an alert-filed report	classify first (is it even a bug?), investigate root cause, then plan/test/build, then prove the regression test	a pull request plus a comment back on the ticket
Refactor code	a named target	plan for behavior preservation; the existing suite is the oracle, so no new tests are written	a pull request that changes no behavior
Review a change	a pull request, a commit range, or a working copy	no build loop at all: package the diff, fan out several reviewer lanes in parallel, synthesize, then verify each finding before publishing	structured findings + a posted review
Fix review comments	a pull request's unresolved threads	triage (decide which comments deserve acting on), then plan/build, then reply in each thread	commits plus a reply on every thread
Generate service docs	a repository	inventory what exists, generate, populate, then a coverage check with a to-do ledger	a pull request

Read the differences as design information:

The bug workflow adds classify at the front (a surprising number of "bugs" are not bugs, and the cheapest possible outcome is a triaged ticket and no code) and prove at the back.
The refactor workflow inverts the test relationship: instead of writing failing tests that define new behavior, it treats the existing suite as the specification and any change in it as a defect.
The review workflow has no loop and no gate on code, because it changes nothing: its output is a judgment, and its risk is publishing a wrong one, so its distinctive phase is verification of its own findings.
The docs workflow has no tests, so its quality gate is a coverage measure instead.

Underneath, all six share the skeleton below, and more importantly they share their parts.

The same six, as a grid. ✓ the workflow has that phase, — it skips it, ● the phase is distinctive to it:

	Feature	Bug	Refactor	Review	PR comments	Docs
Classify	—	●	—	—	●	—
Explore	✓	✓	✓	✓	✓	✓
Plan	✓	✓	✓	—	✓	✓
Red tests	✓	✓	—	—	✓	—
Build loop	✓	✓	✓	—	✓	✓
Self check	—	●	●	●	—	●
Report	✓	✓	✓	✓	✓	✓

Two rows carry most of the information. Classify is present only where a request might turn out not to need code at all (a bug that is not a bug, a review comment that is wrong) and in both of those the cheapest good outcome is no diff. Self check is where each workflow proves its own work, and the oracle differs every time: for a bug it is reverting the fix and watching the new test fail; for a refactor it is the existing suite passing unchanged; for a review it is verifying its own findings before publishing them; for docs it is a coverage ledger. Same slot, four different definitions of "did this actually work", and the workflows with no entry there are the ones whose oracle is simply the tests named in the plan.

The cast

Five kinds of component, and the distinctions earn their keep:

Orchestrator: the workflow itself. Owns control flow, phase order, human gates, dispatch and wiring. Owns no domain logic. It is thin on purpose (part two).

Skill: a unit of work: written instructions plus, usually, scripts. Planning, implementing, validating, exploring, presenting something for approval. A skill knows nothing about which workflow is calling it.

Agent definition: a file declaring a dispatched sub-agent's context window, model, effort level and tool allowlist, registered with the runtime. This is the only place model and effort can be chosen (part two).

Channel adapter: the only component that knows the outside world exists: one that reads from the code-hosting platform, one that writes to it, one that talks to the issue tracker. Everything else is oblivious.

Deterministic primitive: a component that reasons about nothing: durable run state, artifact assertions, source-control isolation. Pure scripts; arguments in, exit codes out; no state of their own.

A skill up close

Three parts, and the split is what keeps a large collection affordable:

The procedure: one file, always read when the skill is used. Keep it to the steps.
References: the detail, in separate files, loaded only when needed. This is the difference between a skill that costs a paragraph and one that costs a chapter.
Scripts: the deterministic work, so the model isn't asked to be a computer.

The description in a skill's metadata does more work than it looks like it does. It is what a model reads when deciding whether this skill applies at all, so write it as "use when … NOT for …". The negative half is the one that gets dropped, and it is the one that prevents the failure nobody notices: a skill firing on work it was never meant for.

Who calls whom

A step can run two ways, and the difference matters more than any other implementation detail:

Invoked inline: same context, same model. The orchestrator simply follows the skill's procedure.
Dispatched: a sub-agent with fresh context and its own model, effort and tool allowlist.

The same skill serves both paths, unchanged. That is what makes a skill reusable rather than a phase.

For dispatched work there is an extra link in the chain: the orchestrator does not call the skill, it calls an agent by name; the agent definition supplies the configuration, and the sub-agent it spawns then invokes the skill. The indirection earns its place because the two files know different things: the agent, in our case, is workflow-aware, the skill is not. The skill stays ignorant of which workflow is using it, while the agent carries what only this workflow knows: which tier this decision deserves, and which tools it may touch.

Figure 2: The same skill serves both paths. The indirection earns its place because the agent is workflow aware and the skill is not.

Skills have to be reusable

This is what makes a family cheaper than the sum of its members, and it rests on one rule:

A skill that only works inside one workflow is not a skill. It is a phase.

So every skill has to be usable at least two ways. One is inside its own workflow, which is a given. The other is some form of reuse beyond it, and that comes in two shapes. Either one is enough; what is not optional is that one of them holds.

Inside a workflow. The ordinary case: an orchestrator invokes it as a phase. This one comes free.

Either: inside several workflows. Planning, implementing, validating, writing failing tests, taking human approval, isolating a branch, filing a ticket. These are shared by nearly every workflow in the family. The reuse is not only economy. Improving one of them improves every workflow at once, and a defect in one is felt everywhere. That asymmetry is worth measuring deliberately: which shared component hurts, and in how many workflows, because a fault in a component used by five workflows deserves five times the attention of a fault in a leaf.

Or: standalone, by a human, with no workflow at all. "Explore how this subsystem works." "Plan this change." "Review this diff." "Set up an isolated branch for me." Standalone usability is a design constraint rather than a bonus feature, because it forces a skill to take its inputs explicitly, resolve its own context, and write its output to a file instead of leaning on an orchestrator's conversation. A skill that can be run alone can be tested alone, debugged alone, and adopted by someone who does not want the whole workflow.

Two further composition properties fall out of the same discipline:

Workflows compose through artifacts. Because a review produces a structured findings file, and the fix workflow can consume either that file or live comments from the platform, review → fix works with no human in between and works before a pull request exists. When one workflow runs inside another, it runs in a "composed" mode where the parent's approvals cover it, and it writes a decision log instead of posting replies: the accountability moves rather than disappearing.

The most reused code is the least clever. Run state, seam guards and branch isolation are shared by everything and reason about nothing. That is on purpose. The components with the widest blast radius should be the ones with no judgment to get wrong.

Anatomy of a run

Whatever the domain, the skeleton repeats. Individual workflows skip phases (review has no build loop, docs has no tests, refactor writes none) but the order never changes:

SETUP        register agents, verify prerequisites        (fail fast, before any work)
INPUT        parse the target; resume if a phase cursor exists
STATE        create or reuse the run directory and its state file
ISOLATE      create or adopt a scratch working copy on its own branch
GATHER       fetch the request / diff / feedback  → an artifact
DECIDE       classify, investigate or explore     → an artifact   ← the expensive judgment
  ══ GATE ══ a human approves the decision, recorded by content hash
PLAN         an ordered plan naming files and tests
  ══ GATE ══ a human approves the approach
BUILD        write failing tests → loop { implement → validate } → referee
PROVE        revert, re-run, restore → proven | vacuous
SCOPE GUARD  the diff touches only what the plan named
  ══ GATE ══ a human approves what is about to be published
PUBLISH      push, reply, comment — through adapters, never forcing
CLOSE        a report, on every exit path

Figure 3: The skeleton. The amber bars are the only places a person is required.

Where the spine comes from. The plan → implement → validate loop in the middle of that skeleton is not original here. It is Cole Medin's PIV loop (plan, implement, validate) which he states as the phased core of his AI-coding methodology (ai-transformation-workshop), downstream of the context-engineering / PRP work that precedes it (context-engineering-intro). Four of his moves carry straight into the design above: prime the context before planning rather than planning cold; a plan that states its own validation strategy; a fresh context window for implementation; and validation as layers: type-check and lint, then unit, then integration, then review, rather than one test command. His CLAUDE.md-style rules file is the same instinct as the "name intents, not commands" rule in part two: the repository declares its own rules, the procedure stays generic.

What this document adds is mostly about making that loop hold when the steps are delegated and nobody is watching every one of them: gates recorded by content hash, a script rather than the agent inside the loop deciding when it stops, files as the only interface between phases, and a prove step after green that tries to make the new tests fail. The loop is his; the refereeing, the artifacts and the paranoia are the parts that get learned the hard way.

A note on SETUP, because that is where the cheapest failures hide: a prerequisite check has to exercise the thing it is checking. Confirming that a build file exists proves nothing about whether its targets can run: the credential they depend on may have expired an hour ago. Probe the actual command, cheaply, before dispatching anything, and derive the probe from the project rather than hard-coding it. Otherwise you discover it ten minutes later, inside a step, dressed up as a test failure.

Two more details matter more than they look. Isolate the work outside the developer's own checkout (a separate working copy on its own branch) so a run's noise never disturbs whatever the human has open, and so several runs can proceed at once. And flatten at the handoff: a loop that commits after every iteration produces a history nobody wants to review, so collapse it into reviewable changes before handing over.

What all of this is for

Worth stating once before the principles arrive, because it is the only part the person invoking a workflow actually feels: what they stop having to do. They bring the unit of work and nothing else. No long prompt to compose. Nothing to remember about which steps come in which order, or which skill to reach for, or what this codebase's conventions are: the practices are embedded in the procedure rather than in the habits of whoever happens to be driving. That is the whole adoption argument, and it is worth more than every engineering property in the rest of this document. Those properties (interruptibility, attributable failure, auditability, bounded cost, the ability to decline) are collected in part three, once there is enough machinery on the table for them to mean something.

Originally published at fullgc.github.io. The rest of the series:

DEV Community: Dani Shemesh

Designing agentic development workflows: where it hurts, and what to build first

Where it hurts

What no amount of care will fix

If you are building one

Designing agentic development workflows: shipping it, porting it, letting it run

Packaging and distribution

What a package can hold

One package, or one per workflow?

The fact that decides it: packages can depend on packages

Three traps

What packaging buys beyond convenience

Context cost is not the deciding factor

Migration order

Portability across harnesses

What is already standardized

What is not standardized, and won't be soon

The one rule that buys portability

What a port actually costs

Degrade explicitly; never fake

Why this design ports better than most

A portability checklist

From local to autonomous

The target flow

The gates do not disappear; they move

Some gates stay human permanently

What must be true first

Designing agentic development workflows: is it working, and is it worth it

Testing and evaluation

Where to start

The seams are where things quietly break

Scoring a judgment step

Living with randomness

Cheap checks that punch above their weight

Where the test cases come from

What your runtime may already do for you

Five checks that need no test set at all

Further reading

Making the workflow evaluate itself

Exercise the design deliberately

What a self-report has to contain

The nil protocol

The rollup, and what its columns must not hide

Then fix, and run the same work again

What the comparison may honestly claim

Failure modes of the method itself

Why this is the highest-yield thing to build first

Measuring whether it delivers

Four streams, and a division of labor that is easy to get wrong

The join, and what breaks it

One row per unit of work

Four lenses, and one rule about reading them

The elapsed-time paradox

Comparing assisted with unassisted

Anti-metrics

Instrumentation: the artifacts are the source; hooks are only the trigger

Rules for whoever adds a metric

Shared metrics, owned metrics, and the tier that gets over-claimed

Trace a handful by hand before publishing anything

Designing agentic development workflows: the cost of control

Sizing the work

Plan quality is bounded by input quality

Human gates and the economics of attention

Batch by phase, not by run

A gate asks exactly one question: approve, or annotate

Measure the attention, not just the elapsed time

Model and effort tiering

What the design buys

Designing agentic development workflows: principles for a workflow you can trust

Machinery principles

1. The orchestrator is thin, because only a sub-agent can carry a model

2. The run directory is the only channel

3. Artifacts are the interface between phases

4. Guard the seams

5. A gate is a check, not a promise

6. Judgment runs in fresh context

7. The loop is refereed by a script, not by the agent inside it

8. Tests come first, and then the tests are proved

9. Only the adapters know the outside world

10. Terminal states, and a report on every exit