DEV Community: Generalist Dev

The best harness is the one you can't see

Generalist Dev — Sun, 23 Aug 2026 07:56:45 +0000

Agent tooling keeps trying to give me a better seat in front of the factory. I don't want a better seat. I want no seat.

Six tools keep showing up in my feed, in three shapes:

firstmate (https://github.com/kunchenguid/firstmate) and HumanLayer (https://humanlayer.dev) put a model in the coordinator seat. You talk to one surface; it fans work out to agents.
herdr (https://herdr.dev) and cmux (https://cmux.com) orchestrate nothing. They own the terminals and tell you which pane is working, blocked, idle, or done.
OpenHands (https://openhands.dev) and Factory (https://factory.ai) are closest to ticket in, PR out. Work can start from GitHub, Linear, Slack, or a headless command instead of an open terminal.

Different products, same tell: each explains what the human should look at while the agents run. The orchestrator gives you a pane to talk to. The multiplexer gives you better panes to watch. The factory gives you a ticket, then a run history or control center.

The interface keeps changing. The human attention budget does not.

Orchestration is a commodity

Part of my skepticism about orchestrators is how little machinery it takes to get one working. A coding agent already knows how to delegate. Add a stop condition and isolated worktrees and you have the interesting half:

# After every turn, evaluate the goal against the transcript.
/goal every issue in docs/issues/ has a merged PR, or stop after 40 turns

# Fan each issue into an isolated worker, then review the result separately.
ultracode: for each file in docs/issues/, implement in a worktree, run
make verify, open a PR, and assign a separate reviewer agent

I've shipped MVPs with roughly this shape: an orchestrator grinding until the output was legible enough for me to inspect. People build cleverer versions: two sessions pointed at each other, one running Matt Pocock's grill-with-docs (https://github.com/mattpocock/skills) as the interviewer and the other playing founder until the ADRs are written.

Fun to watch. Also the problem.

Two things wrong with a model in the driver's seat

It is prompt-shaped. An LLM orchestrator's behavior depends on the initial prompt, the context it found, and whichever instructions happened to load. Hand one feature to two agents and you can get two codebases, not two implementations of one design. Adding a model that normalizes the requirement first moves the prompt one layer up. It does not turn the prompt into policy.

It has gravity. The orchestrator lives in a terminal, so the terminal stays open, so you look at it. Blue rings, unread badges, and blocked/idle tags make the looking more efficient. They do not make it comprehension.

In part 1 I called comprehension a budget: stop at a checkpoint, trace a critical path, and explain the change. Scrollback is the opposite. It spends attention continuously and builds no reliable model of the system. The factory that burned me had a green suite and a terminal I watched for 18 epics. The watching caught nothing.

My bet is that model capability has flattened enough that the harness is now the important variable. Once a model can run for hours, trust has to come from the structure around the run, not from your ability to supervise its transcript.

What the SDLC already knew

Strip the AI out and software teams already look like a harness for non-deterministic workers:

A requirement is written and split into PR-sized issues.
Dependencies decide what can start.
Developers implement without anyone watching them type.
CI, review rules, and a definition of done decide what can land.

The useful property is not that every step is deterministic. People are not deterministic either. The useful property is that authority is explicit: the ticket owns intent, the dependency graph owns order, the worktree isolates implementation, and the gates own admission.

As a function:

S : requirement → [issue]         split into PR-sized units
G : [issue]     → dag(issue)      encode dependencies
I : issue       → diff            implement in isolation
V : diff        → diff | reject   verify with independent evidence

H(x) = fold(merge, map(V ∘ I, G(S(x))))

x and S carry the highest-leverage human judgment. A vague requirement or bad split poisons the graph before a model writes a line. G, I, and the shape of V can be policy even when the workers inside them are probabilistic. V is where the oracle, mutation gate, and scheduled comprehension brake from part 1 belong.

The harness should have a contract, not a cockpit:

input:
  requirement: docs/requirements/refunds.md
  split: docs/issues/
policy:
  verify: make verify
  oracle: testdata/refunds.json
  mutation_threshold: 1.0
output:
  - verified_pr
  - escalation_with_reason

Vanishing as the feature

CI is the obvious comparison. Nobody tails a runner because it might be lonely. You see a check, or a failure with a link. CI became infrastructure when it stopped asking for ambient attention.

Graded on that axis:

	S, G	I	V	what you look at
firstmate	coordinator, per prompt	agents in worktrees	crew policy	tmux or another visible backend
HumanLayer	human, at plan altitude	sessions under a daemon	workflow-defined checks	tasks, artifacts, session UI
herdr	none	whatever you run	none	panes with state tags
cmux	none	whatever you run	none	panes with notification rings
OpenHands	external trigger, usually one task	one agent run	workflow-defined checks	conversation or control surface
Factory	external trigger or mission	Droid sessions	workflow-defined checks	ticket, app, or run history

firstmate's watcher and herdr's detachable daemon reduce the babysitting, then both preserve the pane. HumanLayer detaches further, since agents keep running away from the laptop, while making it easy to jump back into sessions. Those are useful tools. They still treat the run as the product surface.

OpenHands and Factory get closer to the interface I want: work starts where the requirement already lives, and execution can be headless. But hiding I is only one third of the problem. If I still split every requirement by hand, encode no dependency graph, and let the implementation grade its own tests, then S, G, and V are still mine. The agent run vanished; the harness did not.

Invisible must not mean opaque. CI leaves logs, artifacts, exact SHAs, and failed checks. An invisible agent harness should leave the same kind of evidence: which requirement produced which issues, which SHA each gate examined, which oracle disagreed, which mutant survived, and why the run escalated. I should be able to audit it without having to watch it happen.

The harness I want has no pane in its steady state. Input is a requirement and an accepted split. Output is a PR that passed V, or an escalation with a reason and the evidence needed to act. Between those events, it should consume compute, not attention.

It's not loops, graphs, or DAGs. It's what comes after

Generalist Dev — Sun, 09 Aug 2026 21:42:27 +0000

That's the discourse in two screenshots, six weeks apart. First we prompted agents. Then we designed loops that prompt agents. Then loops were apparently over and it was graphs. Somewhere in that quarter, "loop engineering" got its own explainers, its own taxonomy, and its own funeral.

And it's not one guy's timeline. Boris Cherny, who leads Claude Code, says the same thing: "I don't prompt Claude anymore. I have loops running that prompt Claude." Addy Osmani wrote the field guide. The taxonomy posts landed right on schedule — a prompt controls one model response, a loop controls one agent, a graph controls an organization of agents. And the funeral was fast: Steinberger's "did we shift to graphs yet?" went up at 00:34 UTC on July 18; about four and a half hours later an X Article titled "Loop Engineering Is Dead. Enter Graph Engineering." was live, and same-day graph-engineering guides were already reading like settled practice. Receipts, clickable:

The ladder keeps extending: prompt, context, harness, loop, graph. Whatever rung you're standing on, someone is already selling the next one — and every rung lives on the same side of the diagram:

I've been on this treadmill too, and this isn't a takedown. Specs did their job — a fine primitive. Loops and graphs are fine primitives too. But look at what every rung of the ladder has in common: it's all machinery for producing code. Then I ran a real verification pass over the output of my own factory, and came out believing the execution topology is the least interesting part of the problem. Almost nobody is working on what happens after the code exists.

The factory worked

I ran a lights-off factory on a real project: 18 epics, 99 merged PRs, 365 commits, ~30k lines of Go — 17k of them tests. The domain was a financial calculation engine, the unforgiving kind where a number wrong in one direction quietly accumulates risk. The specifics don't matter here; every claim below generalizes.

The output looked like the work of a diligent team. Clean packages, conventional commits, documented invariants. Suite green, go vet clean, race detector clean, every PR agent-reviewed before merge. By every signal the loop could see, this was done. The code even reads well.

That last part matters. This was not slop that escaped because the factory was obviously careless. It was coherent, idiomatic, reviewed software. The failure was more interesting: all of the quality signals were downstream of the same process. The agents produced the implementation, produced most of the tests, reviewed the relationship between the two, and then reported agreement. The factory was very good at proving that its outputs agreed with one another.

Then I tested it for real

After the factory finished, I built a separate verification pass: an oracle transcribed by hand from the primary spec, a ~2,300-case corpus derived from that oracle, and a mutation campaign to test the tests.

It found five classes of bugs, all pointing in the same dangerous direction — the engine confidently outputting numbers that were too small. Shipped as-is, that's the kind of wrong you discover at scale, in production, with money attached. My favorite fits in one character:

-   mult := math.Max(optA.Leg.Mult, optB.Leg.Mult)
+   mult := math.Min(optA.Leg.Mult, optB.Leg.Mult)

Wrong side of a max/min decision. The catch: on the common inputs both sides give the same answer — and those were the only inputs the agent wrote tests for. The suite was green over the bug. Review saw plausible code. Every gate passed.

This is not a domain-specific anecdote. Every codebase has a pair of paths that agree on the common case and diverge on the one that matters — rounding directions, timezone edges, inclusive/exclusive boundaries. An agent optimizing for green will find the test that passes, not the case that breaks.

Green is only a local fact

A passing test proves a narrow proposition: for the examples somebody chose, actual output matched expected output. It does not prove that the examples came from the spec, that the expected values were independently derived, or that the important branch was exercised. “The suite is green” sounds like a statement about the product. Usually it is only a statement about agreement inside the test harness.

If implementation output is copied into expectations, the implementation and the suite can be perfectly consistent and jointly wrong. Add an AI review that asks whether the code and tests look plausible, and you do not have three independent signals. You have one assumption echoed three times.

The missing ingredient is not another reviewer in the same chain. It is a second path to the answer. A useful oracle can be wrong — my hand transcription certainly could have been — but disagreement between independently derived paths creates somewhere concrete to investigate. Agreement inside one path creates confidence without information.

The part that actually scared me

Not the bugs. I went to fix them and realized I couldn't — not quickly. I did not know my own codebase.

Lights-off means lights off: I never read those 99 PRs. Agents wrote them, agents reviewed them, the loop merged them, and I read dashboards. That was the entire appeal — and it worked, right up until the first time something needed a human who could hold the design in their head. Debugging needs recall: where does this value get computed, what invariants does it assume, who else depends on it. I had none of it. The codebase isn't bad — it needs a handful of fixes — but the person who was supposed to make them was never in the room when it was written.

That's comprehension debt. It compounds silently, and no linter measures it.

Comprehension debt is not missing documentation. The factory produced documentation. It is the gap between the system's causal structure and the operator's working model of it. A generated architecture page can tell me that package A imports package B; it cannot give me the practiced ability to predict which invariant breaks when a value changes, or which tests ought to fail. That model is built by tracing, explaining, and occasionally being wrong in public before production forces the lesson.

The usual automation calculus counts the time no longer spent typing or reviewing. It rarely books the liability created when nobody acquires that model. Lights-off development did not remove the cost of understanding the system. It deferred it until the worst possible moment: when the system was already wrong and I was under pressure to change it safely.

What the outer loop is missing

Three things I'm now convinced belong in any lights-off setup, whatever its topology:

Comprehension brakes. The loop halts at checkpoints until the human can explain what changed — not approve it, explain it. Pick a critical path and ask: where does the value enter, where is the authoritative decision made, what invariant must hold, who consumes the result, and which test would fail if the decision flipped? This does not mean reading every generated line. It means sampling until the operator can reconstruct the path without outsourcing the answer back to the factory. Comprehension is a budget you spend deliberately, and if you spend none, you've pre-committed to owning a codebase you can't debug.

Mutation testing. An old pattern — the papers are from the 70s — that answers the question agents made urgent: who tests the tests? Seed single-site bugs into the code and rerun the suite:

// your code
if qty >= positionLimit {
    return ErrLimitExceeded
}

// the mutant: one operator flipped
if qty > positionLimit {
    return ErrLimitExceeded
}

If no test fails, the boundary was never really tested — the suite is green decoration. My campaign ran 220 mutants: 213 killed, 7 provably equivalent, a 100% kill rate on the ones that mattered. But that number only existed after the oracle work. The factory's own suite had an unmeasured kill rate, which is exactly how a green suite shipped max for min.

Mutation score is not a correctness proof. A campaign can only test the faults it knows how to seed, and equivalent mutants need human judgment. But it measures something ordinary coverage does not: whether the suite is sensitive to plausible mistakes in the decisions that matter. Line coverage tells you a test visited the max. Mutation testing asks whether the test would notice if it became min.

Oracles. Expected outputs must come from ground truth that is independent of the implementation — a hand transcription of the spec, a reference implementation, an invariant that must hold. Never from the code under test. The rule I now keep at the top of the corpus package:

// Every expected value in testdata is derived from an independent oracle —
// a transcription of the primary spec — never from engine output.
// Copying the engine's output into an expectation makes the case
// self-grading, and a self-grading test is worthless.

When an agent writes the code and its tests in the same loop iteration, the tests are engine output. Self-grading, worthless. Independence is about derivation, not merely which model or person typed the file. Two agents reading the same implementation and copying the same behavior are not independent. One agent transcribing a normative table from the spec without seeing engine output is.

Side by side, the two shapes:

These three controls are not interchangeable:

Control	Question it answers	Failure it exposes
Oracle	Does the implementation agree with independently derived ground truth?	Wrong product behavior
Mutation gate	Would the suite notice a plausible logic error?	Weak or decorative tests
Comprehension brake	Can a human trace the decision and repair it safely?	Operator knowledge debt

An oracle without mutation testing can be guarded by a suite too weak to localize regressions. Mutation testing without an oracle can make a self-grading suite exquisitely sensitive to the wrong behavior. Both without comprehension leave a verified system that its owner still cannot repair. The controls work because they fail differently.

The shape I'd bet on

MVP → derive oracles from the primary source → development gated by mutation score → CI review that diffs against the oracle, with comprehension brakes between the gates:

Not every package needs a 2,300-case oracle or a perfect mutation score. Start where being wrong is expensive: calculations, permissions, state transitions, money movement, destructive operations. Freeze a small independent corpus. Mutate the decisions on those paths. At each milestone, make the operator trace one of them end to end. Expand the verified surface when production risk justifies it.

That is a less glamorous pitch than “design a graph of agents.” It also gives the graph something objective to optimize against. The topology can change every quarter. The authority boundary should not: agents may propose and produce; independently derived evidence decides whether the result lands.

And note the last step honestly: an AI reviewer without an oracle is just a second opinion with the same blind spot — mine reviewed the max/min bug and saw plausible code.

My draft of this post ended "there are no exact metrics AI-driven development can base itself on." Half wrong. One useful metric already exists and is fifty years old: mutation kill rate. An agent can make it go up, but only by adding tests that notice more seeded faults. That is a much harder number to flatter than lines changed, PRs merged, test count, or coverage.

Comprehension debt still has no gauge. For now the brake is stubbornly human: stop, point at a critical path, and explain it. Until that becomes part of the factory, loops versus graphs is arguing about the delivery truck while nobody checks the cargo — or knows how to open it when it starts leaking.