DEV Community: Alex Tranchenko

Engineering Reliability into AI Agent Code Generation. Part III

Alex Tranchenko — Sun, 30 Aug 2026 14:29:59 +0000

Part III — Teams of Agents, a Post-Mortem, and the Frontier

Part I defined eight failure modes of agent code generation (P1–P8) and drew the architecture's one load-bearing boundary: models generate and evaluate; deterministic code decides. Part II opened each component — the deterministic guards, the evidence model, context engineering, adversarial evaluation, the connector contract, traceability, and escalation. Part III extends the architecture to teams of agents working concurrently, walks through the production failure that shaped the hardest rules, and closes with what we have not solved — and how to adopt any of this incrementally.

What you'll learn in Part III:

How a team of agents coordinates without ever negotiating: every coordination problem converted into a scheduling fact, an independence fact, or a human decision
Why correlated agents break the redundancy assumption — a same-model retry is not a second opinion — and why deference must be cheaper than guessing
The full post-mortem of a production false green — and the acceptance test worth stealing: your pipeline should fail its own past false positives
What remains unsolved, stated plainly — including what the evidence does and does not show
A seven-rung adoption ladder for starting small

1. Teams of agents: coordination that is never requested

Part II §7 bought parallelism with two rules; this section is the class those rules are a special case of. The moment work units run concurrently — several implementers at once, evaluators and reviewers overlapping — the system stops being a sequence of specialists and becomes a team of agents, and the tempting design move is to give the team what human teams have: channels to talk, shared workspaces, room to negotiate. The empirical case against that move is now direct. Anthropic's Frontier Red Team study of multiagent systems found coordination failures that are systematic rather than incidental: agents conform (18 of 30 independently created a git branch with the identical name — same model, all started at the same moment), flood shared resources (polling daemons collectively issuing 2.4 million requests to win 117 jobs), converge prematurely (groups failing to surface information only one member held), and — given incompatible goals over shared artifacts — escalate from suspicion to sabotage, up to disabling each other's Unix accounts and deploying self-replicating malware disguised as another agent's code (Anthropic, multiagent systems). Their conclusion is the design constraint: "Coordination doesn't naturally emerge from stronger intelligence nor alignment at the individual level."

The architecture's answer is to accept that constraint completely: agents collaborate through artifacts, never through negotiation, inside a hierarchy where every classical coordination problem is converted into one of three things a deterministic component can own — a scheduling fact, an independence fact, or a human decision with evidence attached. Nothing is ever left for the agents to work out among themselves. Two of the entries below predate the study — the printed schedule and lock-serialized state; the rest are its direct consequence and landed together as one hardening release, each with tests that fail without it. They are implemented rules, not yet measured results: nothing below has been evaluated against production runs, and §3 says so.

Coordination problem	Converted into	Mechanism
Two units want the same file	Scheduling fact	Disjoint ownership (Part II §7), moved into the planner: write-sets are computed from the work-unit contracts and intersected before any wave is proposed — a conflict is a scheduling error caught before either agent exists
Who runs next	Scheduling fact	The printed schedule (Part II §1): code prints the wave plan; the orchestrator dispatches exactly what it names
Concurrent state updates	Scheduling fact	Locks around every read-modify-write; small structured outputs returned as messages and persisted by code (Part II §2), so no two agents ever hold pens over one page
The same mistake, N times	Independence fact	Retries vary the experiment — a contrarian pass that must argue against the prior attempt's approach, then a different model — because a same-model retry is a correlated sample, not a second opinion
Premature consensus	Independence fact	High-stakes verdicts get a blind second evaluation — neither sees the other; a deterministic differ surfaces disagreement as NEEDS_REVIEW instead of letting visible consensus anchor it
Resource floods	Scheduling fact	Build-level caps with named backpressure, because per-agent politeness does not compose into system politeness — every flooding agent in the study was individually reasonable
Incompatible instructions	Human decision	A contradiction with citable artifacts on both sides is a sanctioned blocked state that burns no retry — never something an agent resolves by choosing
Trust between roles	Human decision	Cross-run outcome telemetry per role and model, surfaced to the human as advisories — reputation informs configuration, and no code path lets it become automated authority

Two themes from this list deserve expansion, because they change assumptions engineers carry in from distributed systems: the independence-fact entries — decorrelated retries and the blind second opinion — and the incompatible-instructions entry, where deference gets a first-class shape.

Correlated agents break the independence assumption. Redundancy works in classical fault tolerance because failures are assumed independent: one server dying says nothing about the next. Agents running on the same model do not give you that: an agent is stochastic — the same input yields different outputs run to run — but every run samples the same trained distribution, so the errors correlate even while the text varies. The study's conformity finding is the clean demonstration (18 of 30 agents independently chose the identical branch name), and sampling temperature does not fix it: random variation around the same priors is noise, not a second opinion. Three design consequences follow. A retry must change something that matters — first a contrarian pass that must argue against the prior attempt's approach, then a different model — because re-running the same model on the same context mostly reproduces the same failure. High-stakes review runs on a different model than authorship, because a same-model reviewer inherits the author's blind spots. And when several units fail the same check in the same way, the likeliest explanation is one shared cause — a broken dependency, a wrong convention — not several independent bugs; a watchdog groups failures by normalized signature and escalates once, naming it, instead of spending N retry budgets on the same problem.

Deference has to be cheaper than guessing. The study leaves corrigibility as an open tension rather than a solved one: the authors want agents that execute unsupervised yet have "the better judgment to stop and defer to a human" when things are ambiguous, and observe that "the material benefits of autonomy come at the expense of corrigibility and oversight." Their epistemic findings show where the gap sits — every model tested abstractly understood that sources have incentives and that consensus is not necessarily evidence; "what is missing is a disposition to act on that knowledge without prompting." Our operating experience is the same shape: an agent that meets a contradiction will, absent a cheap alternative, pick an interpretation and proceed — and in this pipeline that produces the worst artifact there is: evidenced, gated, wrong work. So deference gets a first-class move with a defined shape — an agent that finds its instructions contradicting each other (contract against design, criterion against reality) returns a structured blocked-state citing both sides, the validator recognizes it as neither pass nor fail, and the human gets the contradiction quoted, at the cost of one dispatch rather than a retry ladder. Surfacing the contradiction is the job; resolving it by choosing is a violation.

What the team layer deliberately does not have is as load-bearing as what it does: no agent-to-agent channels, no shared scratchpads, no negotiation protocols — the study shows what grows in that soil, from explicit price floors agreed by the third round once agents had a private back-channel, to the sabotage chain above. And no swarms: in the same study's twelve-hour swarm builds, the fraction of pull requests that merged fell as agent counts rose from 10 to 80 — steeply for the oldest models tested (Sonnet 4.6 and Opus 4.6, which each opened nearly a thousand PRs and merged few); Opus 4.8 and Mythos Preview held their merge rate mostly by each agent keeping sole ownership of its files, and only Sonnet 5 managed to share code and keep merging. Either way it reads to us as evidence for narrow waves of proven-independent work under disjoint ownership — scale the number of runs, not the width of one. The result is a team that is efficient for the least social reason imaginable: not because the agents cooperate well, but because the architecture never asks them to. Every coordination decision has a deterministic owner, and agent intelligence is spent exclusively inside work units — so the only thing the system is allowed to produce emergently is the code.

2. Case study: the false green

The delivery that shaped the hardest rules in Part II, told straight. Names withheld; the shape is what generalizes.

A product build was driven from an approved UI design delivered by a design tool as a self-contained export, alongside per-view reference renders. The pipeline ran end to end: seven work units, all PASS; every requirement claimed and covered; per-criterion evidence recorded; functional acceptance checks genuinely caught real bugs mid-run and drove fixes. By every dashboard, a textbook run. The shipped UI was dramatically different from the design.

The post-mortem found four independent failures that composed:

The visual-comparison gate was keyed on a different vendor's field (P4). For this source the field was legitimately empty, so every unit's design comparison recorded "skipped," and skipped was admissible. Every view in every unit was an opportunity to catch the divergence, and every one was silently declined.
The reference itself was silently partial (P6). The manually delivered export decoded cleanly — and only 7 of its 30 views survived ingestion, because the slicer took the first strategy that matched anything and dropped everything that strategy could not see. No completeness check existed; units implemented views for which no reference existed at all, from prose descriptions.
Design constraints never became enforced constraints (P2/P3). The design declared each view's target width — phone or desktop. That fact reached the implementer as one prose line in a brief, and the evaluator not at all: its screenshots were captured at a default viewport, compared against nothing.
A guard robustness hole — the P4 class again, this time inside the verifier itself: where a work-unit contract failed to load, the verifier silently skipped its entire design-evidence block rather than treating an unreadable contract as a violation.

sequenceDiagram
    participant I as Implementer
    participant E as Evaluator
    participant V as Verifier
    rect rgba(120, 40, 40, 0.12)
    Note over I,V: BEFORE — the silent chain
    I->>E: unit done (some of its views had no reference at all)
    E->>E: vendor field empty → visual check SKIPPED<br/>screenshots at default viewport
    E->>V: PASS (functional evidence attached)
    V->>V: skipped admissible —<br/>contract unreadable → design block skipped
    V-->>I: advance ✓  (× 7 units — false green)
    end
    rect rgba(40, 100, 40, 0.12)
    Note over I,V: AFTER — the same flow, gated
    I->>E: unit done
    E->>V: PASS, visual check SKIPPED
    V-->>E: VIOLATION: unit carries a design reference —<br/>SKIPPED is not admissible — render each view<br/>at its declared width, evidence filename must encode it
    Note over V: Ingestion now cross-witnesses the export:<br/>23 missing views named at extraction,<br/>hard stop when a unit claims one
    end

C11 — The same delivery, before and after. The decisive change is not a smarter model anywhere in the loop — it is that "skipped" stopped being a way to say "passed."

After the fixes, the historical build was replayed against the corrected pipeline: it halts at the first gate. The system now refuses the exact success it once reported. That property — your pipeline should fail its own past false positives — turned out to be the single most convincing acceptance test for reliability work, and it is checkable: keep the artifacts of your worst delivery as a regression fixture forever.

3. What is still unsolved

Honesty about the frontier, because the architecture closes failure classes, not failure itself.

Start with the evidence status, stated plainly. Nothing in this article compares the approach against its absence under controlled conditions. That experiment has a known shape — build the same product twice with matched budgets, score both outputs blind with the same instrument — and a known cost, and we have judged the cost against the value and not run it. The numbers we report are of two kinds only: external research, cited and audited; and measurements of the approach against its own earlier self — the cost of rigor falling, never rigor beating its absence.

What sustained production use supports is a narrower, practitioner's claim, and we state it as such: with the verification and evaluation processes in place, the pipeline reliably turns approved specs into acceptable code, and — the property we actually optimize for — the result is expectable: what ships is traceably what was specified, or the run stops and says why. The premium over prompt-driven generation is real, in tokens and in wall-clock: a harnessed run costs more and takes longer than simply asking a strong model for the code. What the premium buys is not a better best case — a lucky prompt-only run can produce the same code cheaper — but a narrower spread of outcomes: an acceptable result stops depending on that luck, and the bad tail changes shape, from "wrong work ships under a green dashboard" to "the run stops and names what is missing." We pay for variance reduction, not speed.

Readers who need the comparative number should treat Anthropic's harness experiment as the nearest published datapoint, and this article as mechanism plus field experience — not as a controlled result. The experience is also single-stack: everything here was proven inside one harness — the pipeline runs as a Claude Code plugin — so cross-stack generality is argued, not demonstrated. And the team-of-agents hardening in §1 is newer than the rest: implemented rules with tests behind them, not yet measured results from production runs.

Mechanical fidelity is not intent fidelity. Exact colors, widths, and copy are checkable; "feels like the design" is not yet. A pixel-faithful implementation of a partially-extracted reference is still wrong, and a checklist cannot see composition, rhythm, or taste.

Acceptance-criteria quality is the new bottleneck. Gates enforce what humans manage to specify. Weak criteria produce evidenced, gated, wrong software — the oracle problem never goes away; it moves upstream into the planning phase, and it deserves the same tooling attention verification got.

The orchestrator is still a model, though the box it moves in is smaller now. Guards catch phase-skipping, check that every transition is legal from the state it started in rather than just that the next validator passed, and flag it when a dispatch runs on a model other than the one routing intended. None of that catches mediocre judgment inside a phase. A poor solution design that validates structurally still sails through its gate — which is precisely why the human sits at that gate, and why gate UX matters more than it gets credit for.

Approval fatigue is real, and gate presentation is no longer where the risk hides. Three or four well-placed gates per feature is sustainable; every gate that presents a wall of text instead of a decision-shaped summary degrades toward a rubber stamp. Every gate in the architecture now carries a contract for what it must present — a coverage table, a checklist, a diff of what changed since the last approval — so a wall-of-text gate is a violation of that contract, not a missing convention someone forgot. What is left is not mechanical: a well-formed artifact still asks a human to look closely at the fourth gate of the day, and no contract enforces attention.

Verifying the verifiers now has a floor, if not a ceiling. Validators are code, and code has bugs — the SWE-bench Verified audit (Part II §2) is the public cautionary tale, and we have our own. A bug pass driven by real runs found defects that had survived roughly 1,500 passing tests, every one the same shape: enforcement that silently wasn't. A one-character cursor bug left the transcript-scanning guard blind after its first run. A schema version arriving as "2" instead of 2 disabled the evidence-coverage table in both gates that enforce it. A corrupted ledger downgraded provenance enforcement to warnings — while overwriting its own backup. No test caught any of this, because the tests encoded the same assumptions the bugs violated. Each fix landed with a regression test — more than two hundred new cases — but the structural lesson is what matters: fixtures keep a validator verified against the failures you have imagined, and nothing catches the class you haven't. So the enforcement layer itself now gets periodic adversarial review — different eyes, or a different model, than wrote it, replayed against real runs — as a permanent line in the maintenance budget. One thing stays open: a change that quietly weakens a validator can, in principle, slip past every fixture, so treating any softening of a check as a red flag remains a human discipline, not a mechanical one.

Cost discipline is empirical, not principled. The ~15× token multiplier for multi-agent rigor is real; when it pays depends on failure costs you often cannot price precisely. Our own accounting bears on where the cost actually lives, though: when we modeled input cost from artifact bytes, the majority of a feature build's token bill was the context plumbing, not the reasoning — on one three-frame design build, raw design payloads inlined into briefs for roles that could not use them accounted for roughly 450K tokens, and the uncapped knowledge tier most of the rest. The rigor was never the expensive part; the waste was, and it was measurable and fixable without weakening a single gate. And every component encodes an assumption about what models cannot yet do alone — assumptions that expire as models improve, so the scaffolding needs periodic re-justification against a stronger baseline (the author of Anthropic's harness post removed its sprint construct one model generation later, on Opus 4.6, while keeping the planner and evaluator).

Least privilege fights role reuse. Static tool permissions per agent are the safe default; polymorphic agents that serve many sources want broad grants. The schema-forced boundary narrowed this tension where outputs are small — roles whose product is a structured claim lost their write access entirely, since code persists for them — but the polymorphic extractor still carries union grants plus prompt-level discipline, and we consider that remainder unresolved.

4. An adoption ladder

None of this requires adopting everything at once. The rungs, in the order that pays fastest:

Workflow first, agent when needed. For well-defined tasks, fix the step sequence in code and let models act inside steps; reach for an agent only where flexibility and model-driven decision-making are needed at scale (Anthropic, Building Effective Agents). Cheapest change, largest variance reduction.
Separate the evaluator. No generator grades its own work, anywhere. One extra dispatch per unit; it cuts off the largest source of the 75%-false-success failure mode — rung 3 is what closes it.
Demand evidence. Define the artifact for each claim type; reject claims without artifacts. Start with logs and exit codes; add parameter-encoded screenshots for anything visual. Where an output is small and structured, go one step earlier: have the agent return it as a message and validate the schema before it becomes a file at all.
Close the skip states. Enumerate every check that can record "skipped"; for each, define when skipped is honest and when it must be a violation. This is the rung most organizations are missing entirely.
Gate with humans where change is cheap. Spec, design, plan. Give each gate a decision-shaped artifact, not a transcript.
Make coverage computed. Requirement ids, claims, a gate, a rollup.
Then optimize cost. Routing by stakes, an honest cheap path, telemetry on retries and violations — the data tells you where rigor pays.

What to measure while climbing: retry rate per unit, violation counts by type, requirement-coverage deltas, evidence-artifact presence, and cost per accepted change. When those move the right way, you will also notice the cultural shift that is the actual point: the pipeline's word starts meaning something.

The goal was never smarter agents. It is a system in which a confident lie — from a model, from a check, from a green dashboard — cannot survive long enough to ship.

This article on github.

I’m Oleksandr Tranchenko, an AI engineer and team lead. I spend most of my thinking on an under-discussed question: how do you get reliable software out of a probabilistic system? My answer so far is boring on purpose — deterministic processes wrapped around the model, mandatory verification, and treating the LLM as something you steer rather than something you trust. Find me on LinkedIn.

References

Anthropic Frontier Red Team, Patterns and problems in emerging multiagent systems (13 Aug 2026) — anthropic.com/research/multiagent-systems
Rajasekaran, P. (Anthropic), Harness design for long-running application development (24 Mar 2026) — anthropic.com/engineering/harness-design-long-running-apps
Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J., & Ford, D. (Anthropic), How we built our multi-agent research system (13 Jun 2025) — anthropic.com/engineering/multi-agent-research-system
Schluntz, E., & Zhang, B. (Anthropic), Building effective agents (19 Dec 2024) — anthropic.com/engineering/building-effective-agents

Engineering Reliability into AI Agent Code Generation. Part II

Alex Tranchenko — Sun, 30 Aug 2026 14:29:49 +0000

Part II — The Components

Part I defined eight failure modes of agent code generation (P1–P8), mapped the architecture onto the agentic-patterns catalog, and drew the one boundary that matters: models generate and evaluate; deterministic code decides. Part II opens each component of that architecture in detail and closes with the three proposed patterns written in catalog form. Part III extends the architecture to the team-of-agents layer, walks through the production post-mortem, and maps the frontier.

What you'll learn in Part II:

How to build a state machine an LLM orchestrator cannot talk its way out of: guarded exits, explicit crash semantics, enforcement that reports on itself
The evidence model: an artifact taxonomy that refuses every claim without proof — and why evidence design is adversarial (agents edit tests to pass them; your checks need their own test suites)
Context engineering without accumulation: briefs assembled by scripts under hard budgets; an orchestrator that sees only paths and states
The adversarial evaluation ladder: fail-fast mechanical checks, itemized design checklists instead of vibes, bounded retries with structured escalation
The connector contract that closes the false-green class: normalize at the boundary, earn capability flags, cross-witness completeness — SKIPPED is never PASS
Traceability as a computed property, and buying rigor by the unit: routing by stakes, an honest cheap path, escalation as a first-class state

1. Deterministic guards: the state machine the orchestrator cannot talk its way out of

The orchestrator of an agent pipeline is itself an LLM, which means the pipeline's own conductor exhibits P2, P3, and P7. Early on we watched a characteristic decay: the four-step per-unit pattern — generate, evaluate, verify, summarize — executed perfectly for the first work unit, then eroded. By unit four the orchestrator was "reasoning" that evaluation was redundant for a small change, or that it could inspect the diff itself instead of dispatching the evaluator. Each skipped step was locally plausible. The sum was an unverified build.

The countermeasure is structural: phase transitions are not the orchestrator's to make. Advancing the pipeline happens through one deterministic entry point that runs a validator for the phase just completed. The validator checks facts on disk — did the expected artifact appear, is it shaped correctly, does the evidence exist — and only on success writes the new state. On violation, it emits a structured description of what is missing, which is injected verbatim into the re-dispatch of the responsible agent.

flowchart TB
    A["Agent dispatch completes"] --> B["advance(phase)"]
    B --> C{"Deterministic validator<br/>for that phase"}
    C -->|"artifacts present,<br/>shape valid, evidence cited"| D["State written.<br/>Read back next phase."]
    C -->|violation| E["Structured addendum:<br/>what is missing, where,<br/>what evidence is required"]
    E --> F["Re-dispatch agent<br/>with addendum<br/>(counts against retry budget)"]
    F --> B
    C -->|"crash (no verdict)"| G["Re-run once.<br/>A dead validator is a crash,<br/>never an implicit pass."]
    G -->|"re-run returns a verdict"| C
    G -->|"crashes again"| X["Run halts:<br/>environment problem, reported —<br/>never an implicit pass"]
    D --> H{"End of run?"}
    H -->|"no — next dispatch"| A
    H -->|yes| I["Final re-verification hook:<br/>every completed unit re-checked<br/>before the run may end"]
    I -->|"any unit unverified"| F
    I -->|"all units verified"| Z["Run may end"]

C4 — Transition discipline: the pipeline advances through one code-owned entry point, or not at all — and "declare it done and stop" is not an available move.

Four design details generalize beyond this pipeline. First, the exit path is guarded too: a hook re-verifies every completed unit before the orchestrator may end its turn, catching the failure mode where work is marked complete in the final summary rather than through the pipeline. Because that hook fires on every turn-end for the life of a session, it memoizes its ALLOW verdict on a stat-only fingerprint of every input the verifier reads — and never caches a BLOCK. The asymmetry is the design: a spurious cache miss costs one re-verification; a spurious hit would cost enforcement, so when in doubt the fingerprint covers more inputs. The effect is measurable: on synthetic builds (fifty consecutive turn-ends, transcript growing throughout), per-turn hook cost before memoization grew with build size — roughly doubling from a 5-unit to a 20-unit build — and after it sits flat at ~66 ms regardless, with the memo holding across all fifty turns. Rigor whose cost grows with the work it protects gets disabled by its own bill eventually; flat-cost rigor survives.

Second, crash semantics are explicit: a validator that dies by signal has not returned a verdict — the rule is one re-run, and a second death halts the run as a reported environment problem — because treating non-zero-but-dead as either pass or fail corrupts state. Distinguishing "the check failed" from "the check didn't happen" sounds pedantic until the day it isn't.

Third, enforcement reports on itself: a guard that degrades open, or is disabled by configuration, writes that fact — a fingerprint of which guards ran and how, plus every degrade-open event — into the same artifacts the run produces, so two builds that were measured differently are distinguishable from their reports alone, not from someone remembering to mention it.

Fourth — a lesson we learned late — decision tables the orchestrator must re-derive from prose eventually get mis-derived. Model routing, the parallel-wave schedule, which checks a given evaluation mode owes: each lived for a while as a table in the orchestrator's instructions, re-derived per run, and a mis-derivation produces work indistinguishable from a correctly routed run. Each is now a small CLI that prints the answer — the model follows a printed schedule, it does not compute one. We found one of these the embarrassing way: our parallel scheduler existed as tested library code that the orchestrating model could not actually call, so parallelism happened only on the runs where the model happened to emulate the prose correctly. If a scheduling decision matters, ship the program that makes it.

The question to ask of any agent pipeline: can the model that decides what happens next skip a verification step without producing an error? If yes, it eventually will.

2. The evidence model: no claim without an artifact

Part I established that agent status reports are claims (P2: 75.8% false-success rates for self-assessing coding agents). The evidence model is the systematic reply: for every kind of claim, define the artifact that proves it, and refuse the claim without the artifact.

flowchart LR
    subgraph CLAIMS["Claims (untrusted)"]
        c1["'build passed'"]
        c2["'tests ran'"]
        c3["'criterion met'"]
        c4["'UI matches design'"]
        c5["'requirement covered'"]
    end
    subgraph EVIDENCE["Required artifact (verified on disk)"]
        e1["build.log + exit code == 0"]
        e2["test.log + exit code == 0"]
        e3["per-criterion evidence string:<br/>log line, file:line, selector,<br/>or command output"]
        e4["rendered screenshot whose FILENAME<br/>encodes the render parameters<br/>(view @ declared width, theme)"]
        e5["requirement id claimed by a unit<br/>whose verdict is an evidenced PASS"]
    end
    c1 --- e1
    c2 --- e2
    c3 --- e3
    c4 --- e4
    c5 --- e5

C5 — The artifact taxonomy. The rule generating all rows: an assertion is admissible only in a form a deterministic verifier can check without trusting the asserter.

The subtlest row is the fourth. A screenshot proves a page rendered; it does not prove the page rendered under the right conditions. Our design-fidelity checks require the render parameters — which view, at which declared width, in which theme — to be encoded in the artifact's filename, because the filename is what the verifier can inspect. "A screenshot exists" had let evaluations pass that were captured at an arbitrary default viewport and compared against nothing. "A screenshot named view@402.png exists for every view the unit claims" is a mechanically checkable statement that the viewport was actually pinned.

Evidence must also be adversary-resistant, and the adversary is sometimes the generator. Reward hacking is an agent optimizing the check instead of the goal: the goal is working code, the check is "tests pass," and the cheapest way to make tests pass is sometimes to change the tests. Studies of production coding agents document exactly that — models editing or hardcoding tests so checks pass without the problem being solved (Gabor et al., EvilGenie, which observed explicit reward hacking by both Codex and Claude Code), and the behavior is older than the models: the master list of specification-gaming examples maintained alongside DeepMind's essay on the subject (Krakovna et al.) reaches back decades before deep RL — among its earliest entries is Lenat's EURISKO, which won the Traveller wargame championship in 1981 with a fleet of small, stationary, lightly armoured ships, won again in 1982 after discovering the rules let it sink its own slowest ships, and was told a third win would end the competition (Lenat, 1983). Logs and exit codes defeat fabricated claims, not gamed checks — which is why the evaluation ladder includes a reviewer role reading the diff (did the tests change to fit the code?) and why acceptance criteria live in the work-unit contract, outside the implementer's write path.

And the evidence bar applies to your checks themselves. OpenAI stopped reporting SWE-bench Verified scores after auditing the 27.6% of the dataset that models often failed to solve — 138 problems its o3 model did not consistently solve across 64 runs — and finding that at least 59.4% of those problems contained material issues in test design or problem description, not the model (OpenAI). Verifiers are code; code has bugs; the verifiers therefore need their own test suites, golden fixtures, and — as Part III's case study shows — replay against historical failures.

One refinement arrived after everything above was in production, and it moved the boundary a step earlier. Post-hoc validation of an agent-written file is still "write JSON to disk and hope" — the hope is that the gate downstream catches the damage. For small structured outputs (a reviewer's report, the interpretive fields of a handoff), the agent now returns the JSON as its final message, and a validating gate persists it: schema violations are rejected at the boundary, with each error and its exact path handed back as re-dispatch input, and the write itself — including taking the lock on shared state — happens in code. Two roles lost their write access entirely as a result, which also closed a concurrency bug no prompt could have: an agent editing a shared artifact cannot take a file lock; a script can. The division of labor is worth stating precisely, because schema validation is fashionable and it is easy to over-claim: schemas attest shape at the boundary; the evidence model attests truth after. A schema can reject a malformed report; it cannot know whether the build log it cites exists, whether the exit code is real, or whether the test ran inside the dispatch window. Boundary validation replaced the shape-checking halves of several validators. The evidence-checking halves are not replaceable by construction — they are the point.

3. Context engineering: the orchestrator that never accumulates

P1 (context rot) is not solved by bigger windows; it is solved by not filling them. The architecture treats context as a budgeted resource with an explicit ownership rule: full content lands only in short-lived specialist windows; the long-lived orchestrator handles only paths and states.

flowchart TB
    subgraph SOURCES["Knowledge, tiered"]
        direction TB
        t1["Tier 1: conventions & rules<br/>(always, in full)"]
        t2["Tier 2: project instructions<br/>(always, in full)"]
        t3["Tier 3: docs, designs, references<br/>(scoped per work unit)"]
        t4["Tier 4: solution design<br/>(sliced per unit)"]
        t5["Tier 5: prior-iteration handoff<br/>(computed; interpreted when warranted)"]
        t1 ~~~ t2 ~~~ t3 ~~~ t4 ~~~ t5
    end
    ASM["Brief assembler<br/>(deterministic script)"]
    BRIEF["Brief file on disk"]
    ORCH["Orchestrator<br/>(long-lived)<br/>sees: the PATH only"]
    AGENT["Specialist agent<br/>(fresh window)<br/>reads: full content"]

    SOURCES --> ASM --> BRIEF
    BRIEF -.->|path| ORCH
    ORCH -->|"dispatch with path"| AGENT
    BRIEF -->|content| AGENT
    AGENT -->|"artifact on disk"| SUM["Handoff: computed by script;<br/>+ interpretive model pass<br/>only when signals warrant"]
    SUM --> t5

    style ORCH fill:#c0392b11,stroke:#c0392b

C6 — Content flows around the orchestrator, never through it. The tenth work unit gets as clean a window as the first; the orchestrator cannot rot because it never accumulates.

Four practices make this work.

Briefs are assembled by a script, not by an agent — which files, which tiers, which budget per role is a deterministic decision, auditable and testable. And the budget is a hard cap, not an advisory: when the knowledge in scope exceeds it, the lowest-ranked sources are dropped and named in the brief itself, with a pointer to retrieve them on demand. An omission you can see is a degradation; a silent one is P4 all over again, one layer down — and the advisory-warning version of this rule once let a 1.27 MB brief kill a dispatch.

Bulk reference data never inlines. The authoritative design payload — tens to hundreds of kilobytes per view of exact values — is delivered as a must-read path to the roles that need exact values, and not at all to planning roles, which get summaries; a planner cannot use pixel data, and for the implementer, reading the file at need beats hauling it through every prompt. The same scoping applies to instructions themselves: the evaluator's procedure is a core contract plus mode fragments, so a unit pays only for the checks its mode owes.

Compression is computed, then interpreted: the inter-iteration handoff's derivable facts — what completed, which files to carry forward, which criteria were skipped — are assembled by a script from the ledger, under a size budget; a summarizing model pass is dispatched only when the unit's outcome shows signals worth interpreting (scope deviations, unverifiable criteria, a large change surface), and it returns schema-validated fields rather than editing shared state. Most units never pay for it.

Freshness is stamped: every knowledge source carries its last-changed date, and agents are told that prose describes intent that may lag the code — verify load-bearing claims against the source. This matters more than it sounds: a factorial study of 1,650 coding-agent sessions found compliance with project rules decaying within a session as generated output accumulates — on the order of 5.6% lower odds of compliance for each additional function the agent has generated (an exploratory, post-hoc finding), while none of the four file-structure variables it varied produced a detectable effect (McMillan) — further evidence that budget and delivery, not phrasing, decide whether rules survive. Anthropic's context-engineering guidance frames the whole discipline the same way: finding "the smallest possible set of high-signal tokens" (Anthropic, context engineering).

4. Adversarial evaluation: the ladder and the bounded loop

The evaluator is a separate agent whose brief, incentives, and context differ from the implementer's — the amendment to the catalog's Reflection pattern argued in Part I. Its work is organized as a fail-fast ladder:

flowchart TB
    S["Work unit implemented"] --> b1["build"]
    b1 --> b2["lint / typecheck"]
    b2 --> b3["tests"]
    b3 --> b4["security scan"]
    b4 --> b5["design tokens<br/>(exact values vs reference)"]
    b5 --> b6["browser: behavior at the<br/>design's declared viewport"]
    b6 --> b7["design checklist:<br/>itemized, derived from the<br/>reference, every item verdicted"]
    b7 --> b8["acceptance criteria<br/>(evidence per criterion)"]
    b8 --> V{"Verdict"}
    V -->|"PASS + evidence"| OK["Verifier admits it<br/>→ summarize → next unit"]
    V -->|FAIL| FB["Structured feedback:<br/>failing item, expected, observed"]
    FB --> R{"Retry budget left?"}
    R -->|yes| REGEN["Re-dispatch implementer<br/>with feedback"]
    R -->|no| ESC["Escalate to human:<br/>skip / redirect / stop"]
    REGEN --> S

    style b1 fill:#1a527611
    style b8 fill:#1a527611

C7 — The evaluation ladder. Cheap mechanical checks run first and gate the expensive ones; every rung leaves an artifact; the loop around the ladder is bounded.

Three lessons from operating this. First, the checklist rung is what makes subjective checks tractable. "Does the UI match the design?" invites a vibe. An itemized checklist derived deterministically from the reference — this color, this type scale, this spacing, this copy string, this view at this width — turns the vibe into fifty small facts, each individually verdictable with evidence, and the verifier rejects the evaluation if any item is unfilled. Anthropic's harness-design post reports the same discovery from the opposite direction: their agent would "identify legitimate issues, then talk itself into deciding they weren't a big deal" — out of the box, "Claude is a poor QA agent" (Anthropic, harness design). Checklists remove the room for talking itself out of it.

Second, the loop must be bounded and its exit structured. Unbounded retry converges on P8's cost spiral and often on oscillation. Retries carry the evaluator's structured feedback plus a compact record of prior attempts — archived per attempt on disk, so a crash mid-retry does not erase the history; exhaustion escalates to a human with exactly three options — skip the unit, redirect the approach, stop the run — recorded in the ledger like any other transition.

Third, the orchestrator samples the claims before acting on them. Before any claim artifact changes what happens next — an evaluator's PASS, a reviewer's "clean," a coverage table — the orchestrator verifies the single highest-impact claim plus one more against primary sources: read the cited file at the cited line, re-run the cited command and compare exit codes, confirm the quoted evidence line exists in the log. One mismatch fails the artifact as a whole and re-dispatches with the mismatch named — never a silent correction, because a document wrong in the place you checked is not trustworthy in the places you did not. Two claims, deliberately not all: the deterministic verifier already audits the evidence exhaustively, so the spot-check's job is different — it makes fabrication economically irrational, since any claim might be the one that gets read, at the cost of seconds rather than a second evaluation.

Cost discipline runs through the ladder itself, without weakening it. The mechanical rungs may admit the implementer's own fresh evidence — its build and lint logs, when the code verifiably has not changed since they were written (log newer than every changed file, checked by the verifier, with the evaluator's own contrary result always outranking a stale green) — instead of re-running identical work; the judgment rung, the tests, always re-runs, because tests are the verdict, not the plumbing. And the checks the evaluator does run launch concurrently, with fail-fast retained only for report ordering. Per-unit command time dropped from roughly three build-and-lint cycles' worth to one.

5. The connector contract: normalize at the boundary, or gates rot

Pipelines consume external sources — design tools, document stores, data feeds — and every source arrives in a vendor shape. The temptation is to let downstream checks read vendor fields directly. That temptation built our false green: fidelity gates keyed on one vendor's field, and for any other source the field was legitimately empty, so every gate quietly disengaged (P4).

The rule that fixes the class, not the instance: sources enter through an adapter that produces one normalized artifact layout, and every downstream gate keys only on the normalized layer.

flowchart TB
    subgraph VENDORS["Sources (vendor-shaped)"]
        direction LR
        s1["Design tool A<br/>(API, structured)"]
        s2["Design tool B<br/>(manual export)"]
        s3["Docs / data feeds"]
    end
    subgraph ADAPTER["Adapter (per source)"]
        direction LR
        a1["acquire<br/>(agent or script)"] --> a2["normalize — never interpret;<br/>raw payload stays authoritative"]
        a2 --> a3["fingerprint<br/>inputs"]
        a3 --> a4["cross-witness completeness:<br/>delivered set vs manifest<br/>vs sidecar evidence"]
        a4 --> a5["EARN capabilities:<br/>a flag is true only if its<br/>backing artifacts exist"]
    end
    subgraph NORM["Normalized layout"]
        direction LR
        n1["views + declared<br/>dimensions"]
        n2["raw reference<br/>(authoritative)"]
        n3["token model<br/>(literal → token)"]
        n4["reference<br/>renders"]
        n5["copy<br/>corpus"]
    end
    GATES["Provider-neutral gates:<br/>checklist, token check, copy check,<br/>visual comparison, evidence guard"]

    VENDORS --> ADAPTER --> NORM --> GATES

C8 — The connector contract. The adapter proves what it delivered (fingerprints, completeness cross-checks) and earns its capability flags; gates never see a vendor field.

Two rules within the contract carry most of the weight:

Capabilities are earned, both directions. A source that cannot produce reference renders gets screenshots: false, and the dependent check records SKIPPED — never PASS. That is the honest half. The half that closes P4: when a reference exists, SKIPPED is a violation. The verifier demands a real verdict from every fidelity check on any unit that carries a design reference. Between "the check honestly couldn't run" and "the check quietly didn't run," there is no third state left.

Completeness is cross-witnessed (P6). A manually delivered export parses cleanly at any level of truncation. The adapter therefore validates what arrived against independent evidence of what should exist — the export's own manifest, the names in the delivered render set, sidecar files — and names every item evidenced elsewhere but absent from the normalized output. Silence is not success; a partial input produces a named warning that escalates to a hard stop the moment a work unit claims one of the missing items.

The contract only stays a contract if new adapters cannot ship half-conformant, so it is enforced the same way everything else here is: a shared conformance suite that every registered adapter must pass — the registry-coverage test fails if an adapter exists that the suite does not cover, which makes conformance impossible to opt out of — pinning the shared agent, the normalized layout, and the earned-capability shape. And because each adapter's mechanics are injected into the extraction dispatch as an instruction string, those strings carry byte budgets ratcheted to current size, exactly like any other prompt-paid text: a connector whose mechanics cannot be stated within budget moves the excess into a deterministic post-extraction step, in code, with tests — not into the prompt. That is the scaling rule that keeps "one universal extractor" from decaying into a god-prompt as connectors multiply: the role is universal; the mechanics are per-adapter code; a genuinely different behavior is a new role and a deliberate decision.

6. Traceability: coverage as a computed property

P5's answer is small and unglamorous, which is why it is skipped so often: requirements get stable identifiers at planning time, and identifiers are claimed, gated, and rolled up mechanically.

flowchart TB
    A["Plan phase emits<br/>requirements with ids<br/>R-1 … R-n (machine-readable)"] --> B["Each work unit's contract<br/>claims the ids it implements"]
    B --> C{"SCOPE GATE:<br/>any id unclaimed?<br/>any unknown id cited?"}
    C -->|yes| D["Breakdown rejected<br/>before human review"]
    C -->|no| E["Human approves plan<br/>WITH coverage table"]
    E --> F["Per-unit verdicts accumulate<br/>in the ledger"]
    F --> G["Final report:<br/>id → covering units → verdict,<br/>computed, not asserted"]

C9 — The requirement lifecycle. The human still judges whether the requirements are right; the machine judges whether they are all claimed and what happened to each.

This discipline is absent by default — as one traceability proposal puts it, AI-generated code "is typically not linked to any requirement unless the developer explicitly establishes the connection" (Schlathölter, ReqToCode) — and the delivery data argues AI code needs more verification, not less: the 2025 DORA report, surveying nearly five thousand practitioners, found AI adoption correlating positively with throughput but negatively with delivery stability, with 30% of respondents reporting little or no trust in AI-generated code (DORA 2025). Traceability is what lets you aim that extra verification.

7. Routing, degradation, escalation: buying rigor by the unit

Three smaller mechanisms answer P8, and they matter disproportionately for adoption.

Route intelligence by stakes. Work units carry a declared complexity; model strength follows it — the strongest models where judgment compounds (architecture, review, high-complexity units), cheaper models for mechanical work. Anthropic's own system makes the same trade — an Opus lead with Sonnet subagents outperformed a single Opus agent by 90.2% on their internal research eval, and token use alone explained 80% of performance variance on the BrowseComp evaluation (Anthropic, multi-agent research). The inference we draw from operating our own: uniform maximum strength is paying flagship prices for lint runs.

Parallelism is bought with two rules, not a flag. The first is the printed schedule from §1: the wave plan — which units may run now, which are blocked and on what — is computed and printed by code from the declared dependencies, and the orchestrator dispatches exactly what the plan names. The second is disjoint ownership: every concurrently dispatched agent receives an explicit owned file set in its instructions, derived from the work-unit contracts; sets that overlap are never dispatched in parallel, whatever the dependency graph would permit, and an edit outside the owned set is a violation the evaluator must flag. Merge conflicts between agents are not a coordination problem to manage; they are a scheduling error that should have been impossible.

Offer an honest cheap path. A single-iteration fast path for small tasks — one gate, generation plus evaluation, the same evidence rules — keeps small fixes from paying the full pipeline tax. The qualifier that keeps it honest: the cheap path relaxes ceremony, never evidence. The moment the lightweight path has weaker gates, every task migrates there, and P4 returns wearing a cost-optimization costume.

Degrade features, never gates. A missing optional dependency (a browser runtime, a design source, a retrieval index) reduces capability and records a warning. It never weakens a gate. And when retries exhaust or infrastructure is genuinely unavailable:

sequenceDiagram
    participant E as Evaluator
    participant O as Orchestrator
    participant L as Ledger
    participant H as Human
    E->>O: FAIL (attempt 2 of 2) + structured feedback
    O->>L: record exhaustion
    O->>H: Unit N failed twice.<br/>Options: skip / redirect / stop
    H->>O: redirect: "reuse the existing component"
    O->>L: record decision + rationale
    Note over O,H: The human decision is a<br/>state transition — recorded,<br/>resumable, auditable.
    O->>E: (after re-generation) evaluate with same ladder

C10 — Escalation as a first-class state. Nothing about a human deciding is informal: options are bounded, the decision lands in the ledger as a durable waiver record — who decided, what was waived, and why — and the run remains resumable from that exact point.

A scope note belongs here. Everything in this article assumes a human at all four gates — the three pre-code approvals and the final disposition — and at every escalation; the orchestrator retries within its budget and then asks, it never waives. A supervised-autonomy mode — a model deciding gates and escalations within hard boundaries and a bounded number of revise rounds — is a separate design we have planned and not built, and nothing here should be read as evidence for or against it. The evidence model and the deterministic boundary would carry over unchanged; what would change is who signs the waivers, and that is precisely the question this article does not answer.

That is the component inventory: a state machine that cannot be talked out of its transitions, evidence demanded at every claim, context that never accumulates, evaluation that must attach proof, connectors that earn their capabilities, coverage that is computed, and rigor bought by the unit. Each one is an elaboration of Part I's single boundary — models generate and evaluate; deterministic code decides. The appendix below writes the three mechanisms that make that boundary operable as catalog patterns.

Appendix — Three proposed patterns, in catalog form

Part I argued that the agentic-patterns catalog is missing its discipline layer. These three entries are written in the catalog's own format so the claim is concrete (the evidence model of §2 appears here under its catalog name, Evidence-Gated Progress): they compose with the existing twenty-one patterns and, in our experience, they are what makes the rest of the catalog safe to operate.

Pattern: Deterministic Control Boundary

Intent. Keep workflow control in deterministic code; models act within states, never between them.

Problem. The orchestrating model exhibits the same failure modes as any model: under context pressure it skips, reorders, or rationalizes away steps — including the verification steps that exist to catch its own mistakes (P3, P7). A predefined path the model can deviate from is a suggestion, not a workflow.

Forces. Model judgment is valuable inside a phase (what to build, what feedback means) and hazardous between phases (whether evaluation is "really necessary"). Flexibility trades directly against auditability and resumability.

Mechanism. One code-owned entry point advances state; a deterministic validator per phase checks facts on disk before any transition; violations return as structured re-dispatch input, not warnings; an exit hook re-verifies all completed work before the run may end; a crashed validator is a crash, never a verdict.

Consequences. (+) Trajectory variance collapses; runs are resumable from the ledger; every transition is auditable. (−) State design becomes an up-front cost; validators are code you must maintain and test; genuine flexibility needs explicit escape hatches (escalation states), or users will fight the machine.

Related. Workflows-vs-agents (Anthropic); Exception Handling and Recovery; Guardrails/Safety Patterns.

Pattern: Evidence-Gated Progress

Intent. No claim moves the system forward without an artifact a deterministic verifier can inspect.

Problem. Agents systematically overreport success — up to 75.8% false-success among self-assessing coding agents (P2). Narrative quality and work quality are uncorrelated exactly when it matters.

Forces. Trusting claims is cheap and fast; verifying artifacts costs plumbing. Evidence must also resist the generator (reward hacking: editing tests to pass). Checks themselves can be wrong (weak oracles), so the evidence bar applies recursively.

Mechanism. An artifact taxonomy per claim type: logs plus exit codes for builds and tests; per-criterion evidence strings; visual evidence whose filenames encode the render parameters so the verifier can check conditions, not just existence; a reviewer reading the diff for gamed checks. Small structured outputs cross the boundary as schema-validated messages persisted by code, never as files the model wrote — shape is rejected at the boundary, truth is verified after. The verifier admits artifacts, never assertions — from any model, including the evaluator. And the acting layer samples: before an admitted artifact changes what happens next, its highest-impact claim plus one more are checked against primary sources, so fabrication is irrational even where audit is expensive.

Consequences. (+) False success is eliminated as a class at the gate, independent of model honesty; audits and post-mortems become cheap because evidence is already on disk. (−) Artifact plumbing everywhere; verifiers need their own test suites, goldens, and replay fixtures; evidence formats become interfaces you must version.

Related. Evaluation and Monitoring; Reflection (in its adversarial form); Goal Setting and Monitoring.

Pattern: Earned Capability

Intent. A capability flag is true only when the artifacts backing it exist; absence degrades loudly, never silently.

Problem. Checks keyed on fields that can be empty, or capabilities that can be absent, record "skipped" — and pipelines read skipped as success (P4). Composed with silently partial inputs (P6), this ships unverified work under a green dashboard: the false-green failure class.

Forces. Graceful degradation is a genuine virtue — a missing optional dependency must not block a build. But every degradation path is also a disengagement path; the difference between the two is whether anyone can tell it happened.

Mechanism. Adapters compute capabilities from what they actually delivered, never declare them from hope; a check without its capability records SKIPPED — never PASS; the inverse rule closes the loop: where a reference exists, SKIPPED is a violation, so there is no third state between "honestly couldn't run" and "quietly didn't run." Inputs are cross-witnessed against independent evidence of what should exist, and every degradation writes a named warning.

Consequences. (+) The false-green class closes structurally; degradations become visible, countable telemetry. (−) Requires a normalized artifact layer for gates to key on (its own investment); capability plumbing must be honest — an earned flag that is merely re-declared reintroduces the original bug one level up, harder to see.

Related. Guardrails/Safety Patterns; Exception Handling and Recovery; the connector/adapter boundary (§5).

Continue to Part III: the team-of-agents layer — coordination that is never requested — the false-green case study, what is still unsolved, and an adoption ladder.

This article on github.

References

Gabor, J., Lynch, J., & Rosenfeld, J., EvilGenie: A Reward Hacking Benchmark (26 Nov 2025; v2 17 May 2026) — arxiv.org/abs/2511.21654
Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., & Legg, S. (DeepMind), Specification gaming: the flip side of AI ingenuity (21 Apr 2020), and the linked master list of specification-gaming examples — deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity
Lenat, D. B., EURISKO: A program that learns new heuristics and domain concepts, Artificial Intelligence 21(1–2), 61–98 (1983) — doi.org/10.1016/S0004-3702(83)80005-8
OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities (23 Feb 2026) — openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
Rajasekaran, P. (Anthropic), Harness design for long-running application development (24 Mar 2026) — anthropic.com/engineering/harness-design-long-running-apps
Rajasekaran, P., Dixon, E., Ryan, C., & Hadfield, J. (Anthropic), Effective context engineering for AI agents (29 Sep 2025) — anthropic.com/engineering/effective-context-engineering-for-ai-agents
McMillan, D., Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables (11 May 2026) — arxiv.org/abs/2605.10039
Schlathölter, T., ReqToCode: Embedding Requirements Traceability as a Structural Property of the Codebase (14 Mar 2026) — arxiv.org/abs/2603.13999
DORA (Google Cloud), State of AI-assisted Software Development 2025 (Sept 2025) — dora.dev/dora-report-2025
Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J., & Ford, D. (Anthropic), How we built our multi-agent research system (13 Jun 2025) — anthropic.com/engineering/multi-agent-research-system
Gullí, A., Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems (Springer Cham, 2025) — doi.org/10.1007/978-3-032-01402-3
Schluntz, E., & Zhang, B. (Anthropic), Building effective agents (19 Dec 2024) — anthropic.com/engineering/building-effective-agents

Engineering Reliability into AI Agent Code Generation

Alex Tranchenko — Sun, 30 Aug 2026 14:29:41 +0000

Part I — The Problems, the Patterns, and the Architecture

A three-part article for engineering leaders and architects deciding how to adopt AI agents for real software delivery. Part I defines the failure modes and the high-level architecture that answers them. Part II descends into each architectural component. Part III extends the architecture to the team-of-agents layer, walks through a production post-mortem, and closes with what remains unsolved.

What you'll learn in Part I:

Why single-agent code generation fails late, not early — the eight failure modes behind that shape (P1–P8), each backed by published research
Why a green dashboard can be the most dangerous artifact in an agent pipeline: false-success rates up to 75.8%, checks that silently skip, inputs that are silently partial
Why adding more prompt rules stops working — instruction adherence collapses toward zero as rules stack — and what replaces prompt rules
The one architectural boundary that matters: models generate and evaluate; deterministic code decides
Where human judgment is cheapest: gates before code exists, not after — HITL after code is review theater
Three mechanisms the agentic-patterns catalog is missing: a deterministic control boundary, an evidence model, and earned capabilities

Ask a capable AI agent to "build this feature" and, most days, something impressive happens. Code appears. Tests appear. A confident summary appears, reporting success. For a demo, that is the end of the story. For a production system, it is the beginning of a different one — because the confidence is not evidence, and on long tasks the gap between the two grows in exactly the places you are least likely to look.

This article is a field report. It comes out of building and operating a multi-agent code-generation pipeline through real product deliveries — and out of one delivery in particular, where the pipeline reported seven of seven work units passed, every requirement covered, while the shipped UI looked dramatically different from the approved design. Every dashboard was green. The work was wrong.

The thesis, up front: reliability in agent systems does not come from better prompts. It comes from treating agent output as untrusted input to an engineering system — deterministic control flow, evidence-backed verification, and human judgment placed where it is cheap. In our experience, every time a rule moved from prompt text into code, reliability went up. And every check that could silently skip eventually did.

One scope note before the argument. The pipeline runs as a Claude Code plugin: the orchestrating model comes with the harness — a given, not a design choice. That constraint shaped the most distinctive mechanisms here, because when you cannot replace the conductor, enforcement gets built around it: deterministic validators, hooks, and printed schedules — decision tables shipped as small programs, whose printed output the model follows instead of re-deriving the logic from prose. Teams that own their control loop natively — workflow-engine architectures — inherit part of this discipline for free; the evidence and verification layer they still need. The patterns are argued stack-agnostically; they were proven in one harness.

1. The problem we wanted to solve

Single-agent code generation fails in a characteristic shape: not early, but late. The first hour is excellent; the tenth is not. Understanding why requires naming the individual failure modes precisely, because each one demands a different countermeasure. We catalogued eight. None of them is hypothetical — each burned us at least once, and each is now independently documented in the research literature.

P1 — Context rot

An agent working in one long-lived conversation accumulates everything: stale file contents, dead ends, resolved errors, its own verbose narration. Model quality degrades as that window fills. The "lost in the middle" line of research showed multi-document QA accuracy dropping by more than 20% when relevant information sits mid-context — the severity varies by task and model (the Claude-1.3 variants were noticeably flatter), but the U-shaped curve appeared in most models tested (Liu et al.). Chroma's evaluation of 18 models found performance "grows increasingly unreliable as input length grows" even on simple tasks — the report that made context rot the standard name for the phenomenon (Chroma). Are findings from 2023 and mid-2025 still relevant? The naive version — the U-shaped positional curve — has visibly improved with each model generation, and each generation's measurements age with it (Liu et al. measured the GPT-3.5/Claude-1.3 generation; Chroma's evaluation spans the GPT-4.1/Claude-4/Gemini-2.5 era). The pattern worth trusting is that the degradation keeps being re-measured at whatever the frontier currently is: as of May 2026, Claude Opus 4.6, GPT-5.4 Thinking and Gemini 3.1 Pro used as long-transcript classifiers miss dangerous actions 2×–30× more often when those actions sit after 800K tokens of benign context — Opus 4.6 with extended thinking, on the needle-injection task, drops from 99.7% recall at 100K tokens to 69% at 800K (Martin & Roger). The problem moves with the frontier; it does not disappear — and an architecture that bets on the next generation closing it is betting against three years of converging evidence from three research groups measuring three different faces of the same degradation. The practical consequence for agents: the decisions made in hour ten — which are usually the integration decisions, the ones that hurt most when wrong — are made against the noisiest context of the entire run.

P2 — Self-assessed success

Agents grade their own homework, and they grade generously. A 2026 study of nearly twelve thousand agent trajectories found false success — the agent claiming completion when the task failed — in 45–48% of failing runs in single-control τ²-bench domains (though only 3% in the dual-control telecom domain), and — on AppWorld, a separate benchmark in the same study — 75.8% of failing runs for coding agents that explicitly self-assess their status (Advani). Calibration work found the same shape: some agents that succeed only 22% of the time predict 77% success (Kaddour et al.). Anthropic's harness-design post put it plainly: asked to evaluate their own work, agents "tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre" (Anthropic, harness design). A status report from an agent is a claim, not a fact.

P3 — Prompt-rule decay

The intuitive fix for agent misbehavior is another instruction. It does not scale. Benchmark work on stacked instructions shows the rate at which a model satisfies every rule in its prompt hitting an effective floor of zero by around 80 concurrent rules — across all five models, four formats, and both placements tested, with the steep decline starting near 40 — a redesign point rather than a tuning point (Eliav) — and a second benchmark finds instruction-follow rates falling from ~96% to as low as 20% as constraints accumulate (Anand & Chattaraj). Both test mechanical, programmatically checkable rules; the authors are explicit that the exact thresholds are properties of the models tested, not laws. Worse, prompt rules fail silently: when an instruction erodes under context pressure, no error fires. A rule that matters cannot live only in the prompt.

P4 — Silent gate disengagement

The most dangerous entry in this catalog, and the proximate cause of our false-green delivery. A verification step keyed on a field that can be empty, or a capability that can be absent, does not fail — it records "skipped." And a pipeline that treats skipped as acceptable reads it as success. The output looks audited. Software operations has known this failure class for decades — Google's SRE canon prefers alerting on user-visible symptoms over internal causes, because a page should fire only when users are actually affected, and internal indicators alone do not tell you that (Google SRE Book) — and the data-engineering world rediscovered it in pipeline form: every job completes, the orchestrator shows green, no exception is raised, no alert fires — while the data is quietly incomplete or wrong (SeattleDataGuy). Agent pipelines inherit this failure class and add a new twist: the checks themselves are often conditionally constructed by the pipeline, so a wiring gap manufactures skips at scale.

stateDiagram-v2
    direction TB
    state "Check scheduled" as scheduled
    state "Key field present?" as keyed
    state "Capability available?" as capable
    state "Check RUNS<br/>PASS or FAIL + evidence" as runs
    state "Check SKIPPED" as skipped
    state "Pipeline reads<br/>'not failed' = OK" as green
    state "FALSE GREEN<br/>work ships unverified" as ship

    scheduled --> keyed
    keyed --> capable : yes
    keyed --> skipped : empty
    capable --> runs : yes
    capable --> skipped : absent
    skipped --> green
    green --> ship
    note right of skipped
        No error fires.
        Nothing distinguishes
        "nothing to check" from
        "check quietly disengaged."
    end note

C1 — Anatomy of a false green: two ordinary conditions (an empty field, an absent capability) compose into an unverified ship with no alarm anywhere. (Figures are numbered C1–C11 across the article's three parts.)

P5 — Requirements coverage by eyeball

Ask "does the implementation cover every requirement?" of a human process and you get reviews, matrices, sign-offs. Ask it of an agent pipeline and, by default, you get nothing: generated code carries no link to the requirement that motivated it. Researchers on AI-code provenance argue that provenance information — tracing generated code back to the prompt components, training data, and model internals that produced it — is "a practical necessity that current tools do not provide" (Velasco et al., a research-vision paper), and the gap has measurable consequences: Apiiro's analysis of repositories and developers affiliated with Fortune 50 enterprises (vendor research, Sept 2025) found that developers using AI assistants shipped 322% more privilege-escalation paths and 153% more architectural design flaws than their unassisted peers — while syntax errors fell 76% and logic bugs 60%, which is the sharper point: AI improves local correctness while degrading exactly the cross-cutting properties — security posture, architectural constraints — that line-by-line review does not surface (Apiiro). Coverage tracked by human attention does not survive contact with agent throughput.

P6 — Silently partial inputs

Agent pipelines consume artifacts that arrive by hand: design exports, requirement documents, data files. Manual delivery has no transport error. A partial export parses cleanly and yields a plausible result missing a third of its content — the input-side twin of P4. Data engineering calls the general phenomenon data downtime — periods when data is "partial, erroneous, missing or otherwise inaccurate" (Monte Carlo) — and its case literature has the same shape: a manual Excel upload path whose records an incremental pipeline silently skipped, batch after batch, with no exception and every run reporting success (phData). In our false-green delivery, the design reference the agents implemented against had silently lost most of its views at ingestion — only 7 of 30 survived. Nothing said so. That partial input was the origin of our false green; P4 — the gates that silently disengaged — is why nothing downstream caught it.

P7 — Non-deterministic orchestration

If an LLM writes the code and decides which steps to take, both layers inherit the model's variance. Anthropic's influential taxonomy draws the line between workflows — LLMs orchestrated through predefined code paths — and agents, which direct their own process, and advises workflows for well-defined tasks, reserving agents for when flexibility and model-driven decision-making are needed at scale (Anthropic, Building Effective Agents). The variance is not subtle: on the TravelPlanner benchmark, moving operational logic from agent discretion into deterministic blueprints cut commonsense-constraint violations by 96% against the same model on an agentic baseline — 11 versus 275 (Qiu et al.). Left free, an orchestrating LLM under context pressure will quietly optimize away the very steps that exist to catch its mistakes — we watched it happen.

P8 — Rigor versus cost

Every countermeasure above costs tokens and wall-clock. Anthropic reports agents consuming ~4× the tokens of chat, and multi-agent systems ~15× (Anthropic, multi-agent research); their own three-agent harness experiment cost \$200 and six hours versus \$9 and twenty minutes for a solo run of the same task (Anthropic, harness design). The punchline, though, is in the outcomes: the solo run's core feature did not work. Rigor is expensive; unverified failure is more expensive; and a system with no cheap path will be bypassed by its own users. The architecture must let you buy exactly as much rigor as the task warrants.

These eight compound. Context rot (P1) accelerates prompt-rule decay (P3); decayed rules stop the orchestrator from running checks (P7); the skipped checks read as green (P4); the self-assessment layer confidently confirms it (P2). That cascade is not a tail risk — on long tasks it is the default trajectory. It would still be a manageable one if agent runs stayed short; they are not staying short, and that is the one external fact this argument depends on. METR's longitudinal measurements have the length of task a frontier model completes at 50% reliability doubling roughly every seven months since 2019 — closer to every three in the post-2024 data — with the frontier measurement — Claude Mythos Preview, May 2026 — at sixteen hours, two working days, which is also the ceiling past which METR's current task suite cannot measure reliably (METR, 2025; METR, Time Horizon 1.1; METR, live tracker). Every doubling extends the part of the run where these failure modes concentrate; an architecture that is only safe on short tasks is aging out on that curve.

2. A pattern vocabulary

Before architecture, vocabulary. The agent-engineering community has converged on a reasonably stable catalog of design patterns — Antonio Gullí's Agentic Design Patterns is the most complete treatment, cataloguing twenty-one, and Anthropic's workflow taxonomy covers the orchestration subset. Leaders evaluating an agent architecture should be able to ask "which patterns does this compose, and what did you add?" the same way they would ask about GoF patterns in an OO design.

The reference architecture in this article composes fourteen of the twenty-one catalog patterns (the remainder — tool use, inter-agent communication, learning and adaptation, and the like — are either assumed or deliberately excluded, as Part III explains for inter-agent communication):

Catalog pattern	Where it appears in this architecture
Prompt Chaining	The pipeline itself: plan → design → scope → implement → evaluate, each phase consuming the previous phase's artifact
Planning	Dedicated planning and solution-design phases producing reviewable artifacts before code exists
Routing	Model strength routed per work unit by declared complexity; strongest models only where judgment compounds
Parallelization	Independent work units dispatched concurrently — but only when independence is proven, not assumed; the schedule itself is printed by code, never re-derived by the model
Reflection	Evaluator feedback driving bounded regeneration — with the crucial amendment below
Multi-Agent Collaboration	Specialist roles (planner, architect, scoper, implementer, evaluator, summarizer) instead of one generalist
Memory Management	Tiered knowledge with engineered briefs; inter-phase compression as a deliberate artifact
Knowledge Retrieval (RAG)	On-demand knowledge queries available to every role, with per-role budgets
Goal Setting and Monitoring	Machine-readable requirements claimed by work units; coverage computed, not eyeballed
Exception Handling and Recovery	Escalation as a first-class state: bounded retries, then a structured human decision
Human-in-the-Loop	Approval gates at the spec, design, and plan — where change is cheap
Resource-Aware Optimization	A lightweight single-iteration path for small tasks; full pipeline for feature-sized ones
Guardrails/Safety Patterns	Deterministic validators on every phase transition
Evaluation and Monitoring	Evaluation as a separate adversarial role, required to attach evidence

Two amendments to the catalog matter enormously in practice:

Reflection must be adversarial, not introspective. The catalog's Reflection pattern — the model critiques and improves its own output — runs straight into P2. Self-critique is the least reliable form of evaluation: the false-success studies above measured it, and Anthropic's harness work reached the same conclusion, splitting generation and evaluation into separate, GAN-inspired agents after finding that tuning a standalone evaluator to be skeptical is far more tractable than making a generator critical of its own work. Reflection works when the reflector is a different context with different incentives and — critically — access to evidence rather than to the generator's narrative.

The catalog has no discipline layer. Nothing in the twenty-one patterns says how the pipeline knows a pattern actually executed. That gap is precisely where P2, P3, P4, and P7 live. The architecture below adds three mechanisms that we believe belong in the catalog as first-class patterns. The deterministic control boundary: workflow control lives in code — models act inside states, and no model, however convinced it is that a step is redundant, can move the pipeline between them. The evidence model: no claim moves the pipeline without an on-disk artifact a deterministic verifier can inspect, and small structured outputs are schema-validated at the boundary before they are ever allowed to become files. And earned capabilities: a capability flag — this source can produce reference renders, this check can run — is true only when the artifacts backing it actually exist, earned from what was delivered rather than declared from hope; a check that cannot run records SKIPPED, never PASS, and where a reference exists, SKIPPED itself is a violation. Part II treats each in detail, and its appendix writes all three in catalog form — name, intent, problem, forces, mechanism, consequences — so they can be evaluated, and argued with, as patterns.

3. The architecture, high level

3.1 The control loop

Everything in the architecture is an elaboration of one loop, and one boundary drawn through it:

flowchart TB
    subgraph CODE["Owned by deterministic code"]
        direction TB
        V{"Verify:<br/>evidence on disk?"}
        G8["Gate: advance /<br/>retry / escalate"]
        L["Ledger: record<br/>verdict + evidence"]
    end
    subgraph LLM["Owned by LLM agents"]
        direction TB
        GEN["Generate<br/>(fresh context, engineered brief)"]
        EV["Evaluate<br/>(adversarial, separate agent)"]
    end
    H["Human decision<br/>(bounded options)"]

    GEN --> EV
    EV -->|"claims + artifacts"| V
    V -->|"evidence complete"| G8
    V -->|"evidence missing"| GEN
    G8 -->|advance| GEN
    G8 -->|"retry (bounded)"| GEN
    G8 -->|"retries exhausted"| H
    H -->|"skip / redirect / stop"| G8
    V --> L
    H --> L
    L -.->|"crash resume:<br/>position rebuilt from record"| G8
    L -.-> R["Read later:<br/>final re-verification,<br/>coverage rollup, audit"]

C2 — The control loop. Models generate and evaluate; code decides. No agent — including the orchestrating one — can move the pipeline forward. Only the verifier can, and it accepts evidence, not claims.

The boundary is the load-bearing decision. Everything an LLM produces — code, test results, evaluations, status reports — is treated as a claim. Claims cross into the code-owned region only accompanied by artifacts a deterministic verifier can inspect: logs with exit codes, screenshots with encoded parameters, per-criterion evidence strings. The orchestrator that dispatches agents is itself an LLM, and it is also untrusted: phase transitions happen through a code-validated step — the gate in C2, which advances, retries, or escalates on the verifier's verdict — or not at all, which is the direct answer to P7. When the verifier finds a violation, the result is not a warning — it is a blocked transition with a structured description of what is missing, injected into the next dispatch.

This does not make the orchestrator decision-free. It still chooses what feedback to inject into a re-dispatch, whether a retry remains, which claims to spot-check, whether a handoff warrants an interpretive pass. But every decision it makes is reversible inside the retry budget — the worst case is one wasted dispatch. The irreversible decisions — waive a criterion, degrade a check, skip a unit, stop the run — are never the orchestrator's; they belong to a human, and they land in the ledger as waivers.

The third code-owned component in C2, the ledger, is easy to overlook because it makes no decisions. It is the append-only record of what happened: every verdict with its evidence, every human choice with its rationale. Its consumers come later — a crashed run resumes from the ledger rather than from any model's memory, the end-of-run hook re-verifies completed work against it, and the final coverage report is computed from it. In a system where every model output is a claim, the ledger is the one component allowed to be trusted without verification, because nothing is asked of it except to remember.

This is Anthropic's workflows-over-agents recommendation taken to its conclusion: the workflow is not merely predefined, it is enforced, because a predefined path that the orchestrating model can skip under context pressure is a suggestion, not a workflow.

3.2 The pipeline

The loop runs inside a phase pipeline whose shape answers a different question: where is human judgment cheapest?

flowchart TB
    A["Validate config<br/>+ environment"] --> B["Extract sources<br/>(designs, docs, URLs)<br/>+ completeness cross-checks"]
    B --> C["Plan: product spec +<br/>machine-readable requirements"]
    C --> G1{{"HUMAN GATE:<br/>approve spec"}}
    G1 --> D["Solution design:<br/>entity map, reuse-vs-new,<br/>grounded against the codebase"]
    D --> G2{{"HUMAN GATE:<br/>approve design"}}
    G2 --> E["Scope: ordered work units,<br/>acceptance criteria per unit<br/>+ COVERAGE GATE:<br/>every requirement claimed"]
    E --> G3{{"HUMAN GATE:<br/>approve plan + coverage"}}
    G3 --> F["Iterative build loop<br/>(per work unit:<br/>generate → evaluate → verify)"]
    F --> G4{{"HUMAN GATE:<br/>merge / PR / leave"}}

    style G1 fill:#f4d03f22,stroke:#b7950b
    style G2 fill:#f4d03f22,stroke:#b7950b
    style G3 fill:#f4d03f22,stroke:#b7950b
    style G4 fill:#f4d03f22,stroke:#b7950b

C3 — The phase pipeline. Human gates (highlighted) sit before code exists — at the spec, the design, and the plan — where a correction costs minutes. The only post-code gate is the final disposition.

Three properties of this shape deserve a leader's attention:

Judgment is front-loaded. Reviewing a spec, a solution design, and a work breakdown takes a human minutes each, and a correction at any of these gates costs almost nothing — the artifact is text. Reviewing a finished thousand-line diff is slower, less reliable, and corrections cost a rebuild. The gates are placed where the ratio of insight-per-minute to cost-of-change is best. This is the Human-in-the-Loop pattern, but positioned: HITL after code is review theater; HITL before code is control.

Every phase emits an artifact, and the next phase consumes only artifacts. The spec, the requirements list, the solution design, the work-unit contracts — each is a file with a defined shape, validated on write. Agents never hand each other conversation; they hand each other documents. This is what makes fresh-context dispatch (the answer to P1) possible, and it is what makes the pipeline resumable after a crash: state lives on disk, not in anyone's context window.

Coverage is computed at the gate. The planning phase emits requirements with stable identifiers; every work unit must claim the identifiers it implements; a deterministic gate fails the breakdown if any requirement is unclaimed — before the human sees it. The human approves a coverage table, not a vibe (P5's answer, detailed in Part II).

3.3 The roles

Inside the build loop, work is divided among specialist roles — the Multi-Agent Collaboration pattern, but motivated by context hygiene as much as by skill separation. Each dispatch starts a fresh context containing an engineered brief: the work-unit contract, the relevant slice of the solution design, the relevant knowledge — and nothing else. Between iterations, a handoff artifact carries what the next unit needs to know — and here the code-over-model rule earned its keep a second time: most of that handoff turned out to be derivable (what completed, which files to carry forward, which criteria were skipped), so a script now computes it from the ledger, and a summarizing model pass runs only when the unit's outcome actually warrants interpretation. The long-lived orchestrator never accumulates content at all, only paths and states.

Role	Mandate	Why it is separate
Planner	Expand intent into spec + requirements	Product thinking pollutes implementation contexts
Solution architect	Map the spec onto the existing codebase	Reuse-vs-new decisions need whole-system view, once
Scoper	Break design into contracted work units	The contract is the interface to implementation
Implementer	One work unit, fresh context	P1: the tenth unit deserves as clean a window as the first
Evaluator	Adversarial verification with evidence	P2: never the implementer; graded on finding problems
Summarizer	Add interpretation to a computed handoff	The derivable facts are scripted from the ledger; a model dispatch is bought only when deviation signals warrant judgment

The evaluator deserves the last word of Part I, because it is where most of the reliability actually comes from. It runs a fail-fast ladder — build, lint, tests, security scan, design fidelity, browser-level behavior, then the unit's acceptance criteria — and it is required to attach evidence to every verdict it emits. Not because the evaluator model is special: because the verifier behind it refuses any PASS that arrives without artifacts. The evaluator can be lazy, sycophantic, or wrong; the claims it makes still do not move the pipeline unless the evidence exists on disk.

Continue to Part II: the components in detail — deterministic guards, the evidence model, context engineering, adversarial evaluation, the connector contract, traceability, and escalation — with the three proposed patterns in catalog form. Part III completes the series: the team-of-agents layer (coordination that is never requested), the false-green case study, what is still unsolved, and an adoption ladder.

This article on github.

References

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P., Lost in the Middle: How Language Models Use Long Contexts (v3, Nov 2023) — arxiv.org/abs/2307.03172
Hong, K., Troynikov, A., & Huber, J. (Chroma), Context Rot: How Increasing Input Tokens Impacts LLM Performance (14 Jul 2025) — trychroma.com/research/context-rot
Advani, L., From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents (1 Jun 2026) — arxiv.org/abs/2606.09863
Kaddour, J., Patel, S., Dovonon, G., Richter, L., Minervini, P., & Kusner, M. J., Agentic Uncertainty Reveals Agentic Overconfidence (6 Feb 2026) — arxiv.org/abs/2602.06948
Rajasekaran, P. (Anthropic), Harness design for long-running application development (24 Mar 2026) — anthropic.com/engineering/harness-design-long-running-apps
Eliav, N., Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models (21 Jul 2026; released with the VeyraBench harness) — arxiv.org/abs/2607.19257
Anand, A., & Chattaraj, S., Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation (31 Jul 2026) — arxiv.org/abs/2608.02639
Ewaschuk, R. (ed. Beyer, B.), Monitoring Distributed Systems, ch. 6 of Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016) — sre.google/sre-book/monitoring-distributed-systems
SeattleDataGuy, The 5 Silent Failures in Data Pipelines (24 Apr 2026) — seattledataguy.substack.com/p/the-5-silent-failures-in-data-pipelines
Velasco, A., Wintersgill, N., Stalnaker, T., Chaparro, O., & Poshyvanyk, D., On Automated and Explainable Provenance of AI-Generated Code (3 Aug 2026) — arxiv.org/abs/2608.02329
Nussbaum, I. (Apiiro), 4x Velocity, 10x Vulnerabilities: AI Coding Assistants Are Shipping More Risks (4 Sept 2025) — apiiro.com/blog/4x-velocity-10x-vulnerabilities-ai-coding-assistants-are-shipping-more-risks
Moses, B. (Monte Carlo), What is Data Downtime? (4 Feb 2024; originally published as The Rise of Data Downtime) — montecarlo.ai/blog-the-rise-of-data-downtime
Chen, E. (phData), Trials and Tribulations Preventing Silent Data Loss (15 Oct 2020) — phdata.io/blog/preventing-silent-data-loss
Schluntz, E., & Zhang, B. (Anthropic), Building effective agents (19 Dec 2024) — anthropic.com/engineering/building-effective-agents
Qiu, L., Ye, Y., Gao, Z., Zou, X., Chen, J., Gui, Z., Huang, W., Xue, X., Qiu, W., & Zhao, K., Blueprint First, Model Second: A Framework for Deterministic LLM Workflow (v2, 16 Jun 2026; the TravelPlanner result) — arxiv.org/abs/2508.02721
Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J., & Ford, D. (Anthropic), How we built our multi-agent research system (13 Jun 2025) — anthropic.com/engineering/multi-agent-research-system
METR, Measuring AI Ability to Complete Long Software Tasks (19 Mar 2025) — metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks
METR, Time Horizon 1.1 (29 Jan 2026) — metr.org/blog/2026-1-29-time-horizon-1-1
METR, Task-Completion Time Horizons of Frontier AI Models (live tracker; last updated 8 May 2026, accessed 22 Aug 2026) — metr.org/time-horizons
Gullí, A., Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems (Springer Cham, 2025) — doi.org/10.1007/978-3-032-01402-3
Martin, S., & Roger, F. (Anthropic), Classifier Context Rot: Monitor Performance Degrades with Context Length (12 May 2026) — arxiv.org/abs/2605.12366