DEV Community: Joseph Yeo

77 Rules Later: What Graduating Our First Stack Actually Looked Like

Joseph Yeo — Mon, 25 May 2026 08:19:22 +0000

This is Part 8 of the ForgeFlow series. Part 7: The File Modification Boundary documented the constraint that changed how we structure tasks: every autonomous task target should be a new file. We ended Part 7 at 12 projects, roughly 52 failure patterns, and 71 design rules. Part 7 closed with an open question: "Project 13 will be the first real test of whether CL-071 holds under normal conditions."

Quick terms for new readers:

FC = Failure Catalog entry (a documented failure pattern)

CL = Crystallized Lesson (a testable design rule derived from repeated failures)

DEADLOCK = the system gives up after repeated identical failures

ForgeFlow = a fully local, TDD-based autonomous coding system running on Apple Silicon

Part 7 ended with a hypothesis and a bet.

The hypothesis: CL-071 (every task targets a new file, never modifies an existing one) might reduce or remove the dominant failure mode we'd been observing. The bet: we'd set formal graduation criteria and run projects until we met them — or discovered why we couldn't.

We ran five more projects (with one intermediate rerun included in the data). On the seventeenth — a blog API with 14 tasks — all 33 tests passed without intervention or deadlock, completing in approximately 12 minutes.

This post is about the five projects between that hypothesis and this result, what the graduation criteria actually measured, and the failure that appeared after we thought we'd addressed all the known ones.

The Graduation Criteria

Before results, here's what we were measuring. We didn't want "it worked once" to count as graduation. We defined four conditions, all of which had to hold on a qualifying run:

Criterion	Threshold
First-run pass rate (tasks passing on the first TDD cycle, no retry)	≥ 85%
New FC yield per project	≤ 2
Repeat FC rate (previously solved patterns recurring)	≤ 5%
Teacher escalation (human operator interventions mid-task)	Decreasing trend

The logic: a graduated stack should show repeatable autonomous recovery within the tested scope (criterion 1), stop producing novel failure patterns at a high rate (criterion 2), not regress on already-solved problems (criterion 3), and require less human involvement over time (criterion 4).

We chose 85% rather than 100% for the pass rate deliberately. Occasional retries are expected behavior in a TDD loop — in ForgeFlow's architecture, the system is designed to recover from them. What we track is whether it recovers autonomously.

The Five-Project Path

Here's the longitudinal data from Part 7's endpoint (project 12) through the graduation run. Note: this table tracks the autonomous pass rate — tasks that eventually passed without human intervention, including retries. The graduation criterion uses the stricter first-run pass rate (no retries), which we measured separately for the qualifying run.

#	Project	Tasks	Autonomous Pass Rate	New FCs	CL Count (at time)
13	comment-api	12	83%	0	~72
14	order-api	16	56%	2	~74
15	recipe-api	14	57%	1	~75
16	bookmark-api v2	12	83%	0	~76
16.5	catalog-api-v2	12	83%	1	~76
17	blog-api	14	100%	1	77

The trajectory wasn't smooth. Projects 14 and 15 dropped below 60%. Then it recovered. In this sequence, plateaus tended to expose a new failure category; the system dipped, the failure got crystallized into a rule, and the next project incorporated the fix.

What changed between project 15 (57%) and project 17 (100%) was not a model upgrade or an engine rewrite. It was three additional design rules, each derived from a specific failure we observed and diagnosed.

The Dip: What Went Wrong on Projects 14 and 15

Projects 14 (order-api) and 15 (recipe-api) both hovered around 56–57% autonomous pass rate. The failures clustered around a few patterns:

Route endpoint isolation. Tasks that bundled multiple endpoints into a single file — GET list and GET detail in the same route module — showed a notably higher failure rate than single-endpoint tasks. The outputs showed scope-related failures: given two endpoints to implement, the model would sometimes complete one and leave the other as a stub, or attempt both and introduce inconsistencies.

We already had CL-043 (one task, one endpoint) from Part 6. But we'd been applying it loosely — allowing two closely related endpoints to share a task. Projects 14 and 15 showed us that "closely related" was too vague for this local execution loop. The rule needed to be absolute: one endpoint, one file, one task.

Import specification gaps. Route tasks that didn't explicitly list every required import in their task description had a high failure rate. The model would guess import paths, often incorrectly. CL-072 crystallized this: every route task description must include a complete "Required imports" block. For example:

Required imports: from fastapi import APIRouter, Depends;
from sqlalchemy.ext.asyncio import AsyncSession;
from app.database import get_db;
from app.schemas.author import AuthorCreate, AuthorRead

Decimal type mismatches. In project 16.5 (catalog-api-v2), a product model with a Numeric(10,2) price column exposed a subtle testing issue. The model wrote assertions comparing float literals to SQLAlchemy Decimal values — and 999.99 != Decimal('999.99') in Python. CL-076 captured this: any Numeric column test must use Decimal comparisons.

In our diagnosis, these looked less like model-capability failures and more like specification-precision failures — cases where the PRD left enough ambiguity for a 45GB quantized model to make a reasonable-but-wrong choice.

The Failure We Didn't Expect: FC-074

Project 17 (blog-api) was designed as the graduation attempt. We applied all 76 existing rules. The PRD passed our automated validator (50 checks passed, 0 failures). We expected fewer known-pattern failures.

The first three attempts all failed on the very first task — creating the Author model. Same error each time: red_apply_empty — the engine's signal that the RED-phase output contained implementation code rather than a test.

Here's what happened, step by step:

Our setup script created a minimal model stub file — just the class name and primary key column. This was standard practice per CL-066 ("stubs should be PK-only").
Before the RED phase (test generation), the engine runs FC-060 cleanup: it deletes the target implementation file so the model writes it fresh.
FC-060 deleted the stub.
The model didn't need the file to exist at generation time — the surrounding task context still described enough of the intended model structure (via data_models in the PRD and conftest import references) that it produced implementation code during RED instead of a test.
The engine detected this as a scope violation and triggered red_apply_empty.
Three retries. Same result each time.

We called this FC-074: the interaction between two previously validated rules (CL-066: keep stubs minimal, and FC-060: clean target files before RED) producing a new failure when combined.

This is worth pausing on. FC-074 wasn't a gap in any single rule. It was an interaction effect — two rules that had each been validated independently across multiple projects, producing a failure only in a specific sequence of operations.

Rule	Behavior in isolation	Combined behavior
CL-066	Minimal stubs reduce over-complete-stub failures	Creates a target file before RED
FC-060	Deletes implementation target before RED to ensure clean state	Removes the stub CL-066 created
Combined	—	RED sees a missing target but enough context to generate implementation instead of a test

The Fix: Stop Creating Stubs

The first instinct was to adjust the prompt wording — tell the model more explicitly to write a test, not an implementation. We tried that. Same failure. Prompt changes alone didn't resolve it; file-state became the stronger hypothesis.

The second instinct was to refine the stub. But we diagnosed the stub's existence as the likely trigger: FC-060 deleted it, and the residual context information was enough to derail the RED phase.

The third attempt was the simplest: don't create the stub at all.

CL-077: Setup scripts must not create model stub files. Model files are created from scratch by the task that implements them. The conftest wraps model imports in try/except so that earlier tasks can run before the model file exists:

try:
    from app.models.author import Author
except ImportError:
    Author = None

This inverted an assumption we'd held across the previous 16 project iterations. We'd operated under the belief that providing a stub — even a minimal one — helped the model by giving it a starting point. FC-074 suggested that in our current engine architecture, the stub hurt by creating a state that the cleanup logic couldn't handle cleanly.

After applying CL-077, the same blog-api project ran all 14 tasks to completion. 33 tests passed, zero intervention, approximately 12 minutes total.

What the Graduation Run Measured

Here's how project 17 scored against the criteria:

Criterion	Threshold	Project 17 Result
First-run pass rate	≥ 85%	93% (13/14 first-shot, 1 retry)
New FC yield	≤ 2	1 (FC-074)
Repeat FC rate	≤ 5%	0%
Teacher escalation	Decreasing	Zero escalations

Project 17 met all four thresholds. The preceding project (16.5, catalog-api-v2) reached 83% — close but below the ≥85% line. So we are treating project 17 as the graduation point rather than claiming a two-project stable plateau.

To be precise about what this means and what it doesn't:

What it means: On the specific runs we executed — FastAPI + SQLAlchemy async + pytest projects with CRUD-level complexity and 1:N foreign key relationships, using Qwen3-Coder-Next 45GB Q4_K_M on Apple Silicon M5 Max 128GB with 77 design rules — the system completed the full project autonomously within the scope of new-file-creation tasks.

What it doesn't mean: We haven't tested more complex architectural patterns (many-to-many relationships, authentication flows, file uploads, WebSocket endpoints). We haven't tested with different model families or hardware tiers. The 100% figure is for one specific project run; it's a data point, not a guarantee.

77 rules is a lot of rules. Each one was derived from at least one observed problem. But the cumulative load of maintaining 77 interacting rules is substantial. We don't yet know if this scales — whether a 200-rule system would be manageable or would collapse under interaction effects. This matches a concern we are starting to track internally: beyond a certain threshold, adding more constraints may dilute model attention rather than improve output. In our design, we've set a ceiling of 20 CLs per prompt injection bundle to guard against this, but we haven't yet hit a project that tests that limit.

The Rule Accumulation Curve

One pattern we've been tracking across the series is how the rate of new rule discovery changes over time:

Projects  1–3:   CL-001 to CL-020   (~7 per project)
Projects  4–6:   CL-021 to CL-035   (~5 per project)
Projects  7–9:   CL-036 to CL-051   (~5 per project)
Projects 10–12:  CL-052 to CL-071   (~6 per project)
Projects 13–17:  CL-072 to CL-077   (~1 per project)

The yield dropped from roughly 7 new rules per project to roughly 1. We're cautious about reading too much into this — it could mean we're approaching the boundary of what our current project complexity can reveal, rather than the boundary of what rules exist. More complex projects might expose entirely new failure categories.

But within the FastAPI + SQLAlchemy + CRUD scope, the flattening is visible in this dataset. The most notable new failure in this stretch was an interaction effect between existing rules — FC-074 — rather than an entirely novel pattern.

The Interaction Effect Problem

FC-074 taught us something we hadn't articulated before: as the rule set grows, the opportunity for interaction effects between rules increases. Each rule is validated independently, but the system runs them all simultaneously.

This resembles a familiar problem in complex systems: the space of pairwise interactions grows faster than the number of components. We can't test all combinations manually.

We don't have a systematic solution for this yet. What we have is a detection mechanism: when a failure occurs that doesn't match any existing FC pattern, we now check whether it could be an interaction between two rules that had both worked in isolation in prior runs. FC-074 was caught this way.

Whether this can be automated — detecting interaction effects without human diagnosis — is an open question. The engine could potentially track which CLs were active when a novel failure occurs and flag the pairwise candidates, but we haven't built that yet.

What Comes Next

Graduating from the FastAPI stack opens a question: what do we do with a graduated stack?

We see two directions, each answering a different question:

Direction A: Complexity escalation. Stay on FastAPI but increase project complexity — many-to-many relationships, authentication flows, nested resources, pagination. This tests whether the current 77 rules hold at higher complexity or whether new failure categories emerge.

Direction B: Stack transfer. Move to a different framework and measure how many of the 77 rules transfer. Our rules are categorized by stack tags — 29 are marked "universal," 32 are "fastapi"-specific. A new stack would test whether the universal rules actually are universal.

The question we're most interested in now isn't whether we can achieve another 100% run. It's whether a rule-based agent system can keep growing without becoming harder to reason about than the model it was designed to constrain.

Series Links

ForgeFlow runs on a MacBook Pro M5 Max 128GB. Planning uses Claude (cloud API). Execution is fully local — Qwen3-Coder-Next 45GB via Ollama, gemma4:26b for QA, Docker sandbox, no API calls during the coding loop. The methodology and failure data are shared in this series.

If you're building something similar — local AI agents, TDD automation, failure catalog systems — I'd be interested to hear whether you're seeing interaction effects between your own accumulated rules. The comments are open.

The File Modification Boundary We Found After 12 ForgeFlow Projects

Joseph Yeo — Fri, 22 May 2026 15:00:08 +0000

This is Part 7 of the ForgeFlow series. Part 6: The Bug Wasn't in the Model ended at 9 projects, 51 failure patterns, and 70 design rules. Up until that point, failure rates in our setup were declining and the working framework felt like it was converging. Project 12 exposed a structural gap we hadn't yet documented.

Quick terms for new readers:

FC = Failure Catalog entry (a documented failure pattern)

CL = Crystallized Lesson (a testable design rule derived from repeated failures)

Identical GREEN = the model returns an unchanged file during the implementation phase

DEADLOCK = the system gives up after repeated identical failures

Part 6 ended on a high note. Nine projects. A 100% pass rate on the last one. Forty-three crystallized lessons. A working framework in our setup: DCR × Information Quality × Task Complexity. The system felt like it was converging.

Then we tried self-referential foreign keys, and a failure mode we'd only seen sporadically became the dominant pattern.

This post is about project 12 — a department hierarchy API with JWT authentication and self-referential parent-child relationships. It documents the failure pattern that connected several scattered observations into a single engineering constraint. And it discusses why, in our case, the most practical response was to restructure the work rather than retry harder.

The Setup: Department API

Project 12 was designed to test two development vectors simultaneously: JWT authentication (new for ForgeFlow) and self-referential foreign keys (a department can be a child of another department). The tech stack was familiar — FastAPI, SQLAlchemy async, pytest — but the data model was more complex than our previous test projects.

The target execution plan: 13 tasks total. Of these, 4 were new-file creation tasks (schemas, tests), 5 were existing-file modification tasks (models, routes), and 4 were either setup steps or handled outside the autonomous loop.

We ran it five times, redesigning between each iteration. The pattern became hard to ignore.

The Scorecard

The table below shows the task categories from Project 12. The same outcome repeated across five redesign-and-rerun attempts.

Task Type	Count	Pass Rate	Avg Cycles
New file creation (schemas, tests)	4	100%	1.0
Existing file modification (models, routes)	5	0%	DEADLOCK

In our setup, tasks requiring the generation of an entirely new file succeeded on the first attempt. Tasks that required modifying an existing codebase file resulted in a processing deadlock. This held across five separate runs, two different backends (direct Ollama API and Aider), and multiple retry strategies.

To scope these findings: our dataset is constrained to a single model family (Qwen3-Coder-Next, 45GB Q4_K_M) running on a single hardware tier (Apple Silicon M5 Max 128GB). We don't claim these trends apply universally. But the pattern was consistent enough across five runs that we changed how we structure tasks going forward.

What "Identical GREEN" Looks Like

ForgeFlow's TDD loop works in two phases: RED (write a failing test) and GREEN (write code to pass it). The GREEN phase is where modifications happen.

When a task required modifying an existing file, the following loop repeated:

The model receives the existing file content + test requirements
The model outputs code that matches the existing file exactly (detected via SHA-256 hash comparison)
The engine retries with an explicit prompt: "Your output was identical to the current file"
The model outputs the same file again
DEADLOCK after 3 identical cycles

We call this an identical GREEN deadlock. The engine already had detection for it (FC-037, added months ago). But we'd only seen it sporadically before. In project 12, it became the primary failure mode.

Working Hypotheses

We're cautious about attributing "understanding" to the model — we're observing output patterns, not internal reasoning. Here's what we think might be happening:

The whole-file generation pattern (Ollama backend): When generating code via raw completion, the model streams the entire file from the first token. If the existing file is 95% correct and only needs a few lines added, the token history in the context window acts as a statistical attractor — the generation pattern defaults to reproducing the verified, working code rather than deviating to introduce new logic. The smaller the required change relative to the existing file, the stronger this pull appears to be.

The diff generation constraint (Aider backend): Diffs require precise line-matching tokens. When the target file is complex — multiple async routes, mixed dependencies, dense imports — generating accurate unified diff chunks appears to become erratic for our local quantized model. In our tests with this specific model and configuration, this manifested as timeouts (capped at 200 seconds per task) or a fallback to emitting an unchanged version of the source file.

Both pathways showed similar limitations on file modification tasks in our configuration. Whether this is specific to quantized local models or a broader pattern, we can't say.

Connecting Scattered Observations

Before project 12, our tracker had three separate failure patterns that each captured a piece of this:

FC-034 / CL-043: "One task, one endpoint" — adding endpoints to an existing route file often resulted in syntax errors or duplicates
FC-047 / CL-066: "Over-complete stubs" — when a stub had significant boilerplate, the model treated it as finished
FC-039 / CL-058: "POST endpoints need Aider" — some tasks specifically failed on the Ollama backend

Project 12 gave us the data to connect these into a single classification, FC-052:

In our local execution setup, existing file modification tasks demonstrate a high probability of identical GREEN DEADLOCK on both whole-file and diff-based backends. In our observations, identical-output failures appeared more often when the required change was small relative to the existing file.

From FC-052, we derived CL-071:

Every autonomous task target should be a new file. If a workflow step must modify an existing file, that modification should either be handled programmatically during setup or the architecture should be decoupled so that features reside in isolated modules.

This became our 71st crystallized lesson, and it changed how we now structure ForgeFlow projects.

One notable data point: across three complete projects (10, 11, and 12), our failure catalog expanded by only a single new entry. The rule accumulation curve is flattening, which may suggest we're mapping the boundary of our current configuration — or just the boundary of our current project complexity.

The Design Pattern That Emerged

CL-071 pushed us to rethink how we write PRDs.

Before (task-level modifications):

TASK-001: Create User model (stub)        → models/user.py
TASK-002: Add fields to User model        → models/user.py    [DEADLOCK]
TASK-003: Create Department model (stub)  → models/department.py
TASK-004: Add relationship                → models/department.py [DEADLOCK]

After (decoupled new-file generation):

SETUP SCRIPT: Generate complete models with all fields and relationships
TASK-001: Create User schemas    → schemas/user.py      [NEW FILE ✅]
TASK-002: Create Dept schemas    → schemas/department.py [NEW FILE ✅]
TASK-003: Create register route  → routes/auth.py        [NEW FILE ✅]

The pattern: infrastructure is established deterministically during setup, while the model handles clean-sheet file generation.

An important caveat: applying this pattern to project 12 was not a clean autonomous success. We manually implemented the CRUD endpoints (6 routes) to unblock the dependency chain, then tested whether the remaining new-file task would run cleanly under the revised structure. The integration test — creating a fresh test_integration.py — passed on its first autonomous cycle. The important result was narrower than "we solved it": once existing-file modification was removed from the autonomous task path, the remaining new-file task completed cleanly.

We should also note an open concern: forcing every task into a "new file only" pattern shifts complexity from generation-time editing to project-level file organization. At 13 tasks, this is manageable. At 50+, it could create significant file fragmentation and import overhead. We haven't tested at that scale yet.

Where We Are After 12 Projects

Metric	Value
Total projects	12 (11 completed, 1 scrapped)
Failure patterns cataloged (FC)	52
Design rules (CL)	71
Automated rule checks	53 functions in validate_prd.py
Sessions	81

The Honest Assessment

After 12 projects and 81 sessions:

What's working in our setup:

New file generation from detailed specs: reliable across the runs we tested
TDD enforcement (RED must fail, GREEN must pass): useful as a mechanical guardrail
Failure pattern → design rule pipeline: producing diminishing but real returns
Setup-based infrastructure + model-based creation: tested over 3 projects

What isn't working:

Existing file modification: consistently unreliable with our current model and configuration
Non-deterministic results on complex tasks: one task passed in 2 out of 3 runs, failed in 1. Same code, same model, different outcome.
Long dependency chains: a single DEADLOCK blocks everything downstream

Open questions:

Does CL-071 hold on 20+ task projects with complex dependency graphs?
Does the "new file only" constraint create unsustainable file fragmentation at scale?
Will newer local models (Qwen3-Coder v2, Llama 4) shift this boundary?
Is this specific to quantized local models, or do cloud API models show similar patterns on file modification tasks inside TDD loops?

A Request to Readers

If you're running local models — Ollama, llama.cpp, vLLM, or something else — within autonomous execution loops, we'd be interested in learning whether your telemetry shows similar variations between file creation and file modification tasks.

Specifically: how do your local configurations handle incremental diff generation inside structured loops versus generating complete, fresh modules from detailed specs? If you've logged similar boundaries or found alternative designs to work around modification deadlocks, please share your setup and observations in the comments.

We're also curious whether anyone has hard metrics on how cloud models (GPT, Claude) perform on targeted file modifications inside closed-loop TDD environments. Our dataset is one model family on one hardware tier — more data points from different setups would help everyone working in this space.

What's Next

Project 13 will be the first real test of whether CL-071 is a design principle or just a project-12-specific workaround. Every implementation task will target a new file. Setup will handle all infrastructure. The open question isn't whether it passes — it's whether the "new file only" constraint produces a project structure that's actually maintainable at 20+ tasks.

We're also adding automatic CL-071 validation to validate_prd.py — a check that flags any task whose implementation target already exists at execution time. For our workflow, rules that repeatedly affect outcomes should probably be machine-enforced.

The Series So Far

I Built a Local AI Coding Agent on M5 Max 128GB — 164 failures, 35 tests, proof of concept
We Didn't Migrate from n8n to Python Because n8n Failed — The orchestrator rewrite
The Determinism War — Why we stopped chasing better models
The Information Design Gap — Why the agent was coding blind
DCR Wasn't Enough — Adding information quality to the framework
The Bug Wasn't in the Model — Lessons from 9 projects
The File Modification Boundary — You are here. 12 projects, a boundary mapped.

About

I'm Joseph YEO, a solo builder from Seoul, Korea. ForgeFlow runs entirely on a MacBook Pro M5 Max 128GB — no cloud APIs during execution. The planning agent (Claude) designs the specs. The local model (Qwen3-Coder-Next, 45GB Q4_K_M) executes the TDD loop autonomously.

Follow along:

Built over 81 sessions, May 2026. All models run locally via Ollama 0.23.0 on macOS. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me.

The Bug Wasn't in the Model: Lessons from 9 Local AI Coding Agent Projects

Joseph Yeo — Sun, 17 May 2026 14:11:48 +0000

This is Part 6 of the ForgeFlow series. Part 5: DCR Wasn't Enough introduced the two-axis model: DCR × Information Quality. We ended Part 5 at three projects and a 29% pass rate. Here's how we reached 100% on a controlled project — and why we needed a third axis.

Part 5 ended with a framework and a question.

The framework said: System Reliability ≈ DCR × Information Quality. The question was whether that would actually hold up as we kept running projects.

We ran six more. Same model. Same hardware. No cloud APIs during execution. By project nine, the autonomous pass rate hit 100% on that specific project — eight tasks, thirty-one tests, four minutes, zero manual intervention.

This post is about the path from 29% to 100%, and the third variable we didn't expect to find.

The Scoreboard

Here's the full longitudinal data. Nine projects, same 45GB local model, same hardware throughout:

#	Project	Pass Rate	CL Rules	Key Change
1	repo-jwt	0%	0	No design rules existed
2	todo-api	67%	~10	Context files added
3	bookmark-api	100%	~20	Full information pipeline
4	expense-tracker	70%	32	New failure patterns emerged
5	rating-api	73%	32	DB fixture issues
6	library-api	0% → scrapped	35	Fundamental architecture gap*
7	event-api	80%	35	Setup script pattern validated
8	habit-tracker	44%	39	Route tasks collapsed
9	contact-book	100%	43	All axes aligned

The 100% figure refers to Project 9 only, not the aggregate across all nine projects. We used it as a controlled checkpoint: after fixing the route-task failure pattern from Project 8, could the same local model complete a comparable route-heavy project without intervention?

*Project 6 was scrapped because it required an architectural paradigm change (multi-model foreign-key setup scripts) that our orchestrator couldn't state-track at the time. Rather than polluting the loop data with a mismatched setup, we halted execution to redesign our baseline infrastructure scripts. The lessons from that failure directly produced CL-036 through CL-039.

The trajectory isn't a clean line upward. Project 3 hit 100%, then projects 4 through 8 dropped back. Each drop exposed a new category of failure that our rules didn't cover yet.

This is the pattern that mattered most to us: in these nine bounded projects, every failure we investigated had a concrete system-level fix. We did not find a case where replacing the model was the only plausible remedy.

The Crystallization Loop

Each project failure produced what we call a "Crystallized Lesson" (CL) — a concrete, testable rule that prevents that specific failure from recurring. Not a vague principle. A rule precise enough that code could check it.

Examples:

CL-005: Infrastructure files (conftest.py, database.py) must never appear in a task's target files. Origin: Project 3, where the model kept overwriting shared fixtures.
CL-034: DateTime fields with SQLAlchemy's default= must be set in Python's __init__, not relied upon at DB insert time. Origin: Project 5, where unit tests failed because created_at was None before flush.
CL-043: When adding an endpoint to an existing route file, each task must contain exactly one endpoint. Origin: Project 8, where multi-endpoint tasks caused the model to time out trying to understand the existing code.

By project 9, we had 43 of these rules. They're not guidelines — they're checkable constraints on the PRD document that feeds the model. We call the document that holds them the PRD Design Checklist.

Here's what the accumulation looked like:

Projects 1-3:  CL-001 to CL-020  (~7 per project)
Projects 4-6:  CL-021 to CL-035  (~5 per project)
Projects 7-8:  CL-036 to CL-043  (~4 per project)
Project 9:     0 new CLs needed

The rate of new rules slowed, but the depth increased. Early rules were about file placement ("where does conftest.py go?"). Later rules were about engine-level behavior ("how does the correction system handle idempotency?").

What Project 8 Broke

Project 8 (habit-tracker-api) is worth examining because it's where the two-axis model from Part 5 stopped being sufficient.

The project had nine tasks. The first four — model and schema creation — passed autonomously in one cycle each. Then the route tasks (5 through 9) collapsed. Zero of five passed.

The failures fell into four categories:

A pytest configuration warning was being captured as a failure signature. The code was correct, but the orchestrator classified it as broken.
A string-replacement correction was applied twice. client.post( → await client.post( was also applied to lines that already had await, producing await await client.post( — a syntax error.
A schema class was never generated because no test existed for it. The model only builds what it's tested for. No test, no code.
Tasks that modified existing files timed out because the model needed too long to understand the accumulated code.

Notice: none of these are IQ problems in the Part 4/5 sense. The model had all the information it needed. The PRD was well-designed by the standards we had at the time. The failures came from the engine itself — the orchestrator's correction logic, its gate system, its timeout handling.

The Third Variable: Engine Quality

This forced us to extend the two-axis model:

I don't mean this as a measured mathematical product yet, but as a diagnostic model: if any axis collapses, the whole loop collapses.

Or to put it less formally: you can have a perfectly designed PRD and a well-informed model, and still fail because the orchestrator has bugs.

By engine quality, we mean whether the orchestrator preserves the intended semantics of the execution loop: phase isolation (RED writes only tests, GREEN writes only implementation), retry correctness (rollbacks don't destroy infrastructure state), deterministic correction safety (rewrites don't corrupt already-correct code), timeout policy, and commit boundaries.

In our case, the concrete fixes were:

The correction engine's idempotency. Our string-replacement system applied corrections blindly, turning client.post( into await await client.post(. The fix was a line-level guard: if the replacement text already exists on a line, skip it. (We're aware this is a limitation of primitive string matching. A proper AST-based mutation engine using something like LibCST would eliminate this entire class of errors. That's on our roadmap but hasn't been necessary yet at our current project complexity.)
The RED phase scope. If the model outputs an implementation file during the test-writing phase and the orchestrator writes it to disk, the test passes immediately — and the TDD cycle breaks. The fix was restricting the RED phase's file-write scope to test files only.
Router registration resilience. If git reset --hard during a retry also reverts infrastructure changes made by the orchestrator's auto-registration system, the next cycle starts with a broken setup. The fix was committing router registration during the initial setup script, not inside the TDD cycle.

We fixed these with three targeted engine patches, each with its own test suite (16 tests total). After the fixes, project 9 ran eight tasks with zero failures.

The key insight for us: PRD quality and engine quality appeared to be independent variables. Improving one didn't fix the other. Project 8's 44% pass rate wasn't a PRD problem — it was an engine problem that looked like a PRD problem until we traced each failure to its root cause.

What 100% Actually Looked Like

Project 9 was a contact book API with search. Single model (no foreign keys), six CRUD endpoints plus a search-by-query-param feature. We chose it deliberately to test route-task decomposition — the exact pattern that failed in project 8.

The numbers:

Metric	Value
Tasks	8 (model, schemas, 6 endpoints)
Total cycles	8 (every task passed first try)
Total tokens	9,042 (Ollama-reported generated tokens; prompts excluded)
Total time	~4 minutes
Tests generated	31
Manual intervention	0
Cloud API cost	$0

Each task followed the same loop: Ollama writes a failing test → Ollama writes minimum implementation → deterministic corrections applied → pytest runs → commit if green.

The route tasks that had failed repeatedly in project 8 now passed in single cycles. The differences:

Each task added exactly one endpoint (CL-043)
All schema classes had test coverage (CL-042)
The asyncio configuration was pre-set in the setup script (CL-040)
Trailing-slash corrections were applied deterministically (new)
Router registration was committed during setup, not during the TDD cycle (new)

None of these changes required a better model. The model was the same 45GB Qwen3 that produced 0% on project 1.

Why 100% Doesn't Mean "Solved"

I want to be careful here.

100% on a contact book API doesn't mean ForgeFlow can build anything. A contact book API is an architectural sandbox. The project was deliberately chosen to isolate the route-task failure pattern. It had no foreign keys, no authentication, no file uploads. Each endpoint was independent. The success here suggests that our execution loop is stable under these constrained conditions, not that it can refactor a legacy microservice architecture.

The real test is whether the next project — something with two related models and foreign-key relationships — maintains a high pass rate. We don't know yet.

What we do know:

The crystallization loop works. Each failure produces a rule. Rules accumulate. The same failure hasn't recurred in subsequent comparable projects.
Engine fixes matter as much as PRD fixes. Three engine patches in one session unblocked a project that no amount of PRD improvement would have fixed.
The three-variable model explains our data better than two. Projects 4-8 had good PRDs but engine bugs. The two-axis model couldn't explain those drops. The three-variable model can.

The Failure Catalog

Across nine projects, we cataloged 19 distinct failure patterns. Every one was eventually addressed — either through a PRD design rule, an engine fix, or a setup script change.

Category	Count	Resolution
PRD design gap	10	CL rules in checklist
Engine bug	5	forgeflow.py patches + tests
Infrastructure/setup	3	Setup script standardization
Timeout/performance	1	aider_timeout configuration

A few examples from the catalog:

FC-015: Non-idempotent correction rule. Symptom: deterministic correction produced await await client.post(...). Root cause: the string-replacement rule did not check whether the line was already corrected. Fix: line-level idempotency guard — if the replacement text already exists on a given line, skip the replacement.
FC-018: RED phase implementation leakage. Symptom: tests passed immediately during RED because the implementation file was also written to disk. Root cause: the orchestrator's file-write scope included both test and implementation files during the test-writing phase. Fix: restrict RED phase scope to test files only; reject or quarantine non-test files during RED.
FC-019a: Router registration lost across retry. Symptom: every retry started from a broken app state (404 on all endpoints) after git reset --hard. Root cause: the orchestrator's auto-registration system added router imports to main.py during the TDD cycle, but git reset reverted those changes. Fix: commit router registration during the initial setup script, before the TDD loop begins.

Under our postmortem classification criteria, none of the 19 cataloged failures were classified as pure model-capability failures — cases where the model lacked the syntax or logic ability to solve the task. Every failure traced back to something in the system around the model: missing information, incorrect scaffolding, or engine bugs.

This doesn't mean model capability doesn't matter. A stronger model would probably tolerate worse PRDs and buggier engines. But in our limited experience, fixing the system was always cheaper and more permanent than hoping for a smarter model.

What Comes Next

We're designing a diagnostic pipeline that applies the failure catalog automatically. The idea: when a task deadlocks, the engine checks the failure catalog for a matching pattern before giving up.

[DEADLOCK DETECTED]
        │
        ▼
[Pattern Match: Failure Catalog]
        ├──► Match Found ──► Apply Fix (Deterministic) ──► Retry
        └──► No Match   ──► Local LLM Diagnosis (Stage 2)
                                   └──► Fails ──► Human Escalation (Stage 3)

Stage 1 is pure pattern matching — deterministic, no LLM needed. Stage 2 would use the local model to diagnose novel failures. Stage 3 remains human review.

The goal isn't to eliminate human involvement entirely. It's to ensure that each human intervention produces a rule that prevents the same intervention next time. The system should get cheaper to operate with every project it runs.

The Thesis, Updated

Part 3: "The bottleneck is not model capability, but the verifiability of specifications."

Part 4: "Even after verifiability is constructed, the bottleneck shifts to information delivery."

Part 5: "An AI coding agent's reliability is a product of its deterministic coverage and its information quality."

Now, the working version after nine projects:

"In our experience, an AI coding agent's reliability is bounded by three independent variables: the determinism of its scaffolding, the quality of information it receives, and the correctness of its own engine. Improving any two without the third produced a system that failed in ways that looked like model limitations but weren't."

The practical diagnostic is now threefold: measure your deterministic coverage, inspect your information quality, and test the engine itself. Fix the axis that's actually broken. In our nine projects, that diagnostic kept pointing to the system, not the model.

Whether that pattern holds at higher complexity is something we're still finding out.

About

I'm Joseph YEO, building ForgeFlow from Seoul, Korea — a local AI coding agent that runs entirely on Apple Silicon, no cloud inference during execution.

What's your experience with orchestrator-level bugs masquerading as model limitations? Have you seen cases where the system around the model was the actual bottleneck? I'd love to compare notes.

Follow along:

Previous parts: Part 1: 164 Failures · Part 2: n8n to Python · Part 3: The Determinism War · Part 4: The Information Design Gap · Part 5: DCR Wasn't Enough

9 projects. 43 rules. 19 failure patterns. 48 development sessions. Same 45GB model throughout. All models run locally via Ollama 0.23.0 on Apple Silicon M5 Max 128GB. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me.

DCR Wasn't Enough: Why AI Coding Agents Also Need Information Quality

Joseph Yeo — Thu, 14 May 2026 14:02:17 +0000

This is Part 5 of the ForgeFlow series. Part 3: The Determinism War introduced DCR. Part 4: The Information Design Gap showed how information delivery moved our pass rate from 0% to 67%.

We thought we had the answer.

In Part 4, we showed that fixing our information pipeline — zero code changes, just better PRD design — moved ForgeFlow's autonomous pass rate from 0% to 67%. We thought the lesson was clear: give the model enough context and it delivers.

Then we ran a third project. Same model. Same engine. Same careful PRD design. The pass rate dropped to 29%.

This post is about why that happened and what it taught us about measuring AI coding agents.

What Went Wrong on Project C

Project C was a bookmark API with many-to-many relationships — more complex than a simple CRUD, but not wildly different in structure. We applied everything we learned from Project B: explicit context files per task, detailed descriptions, proper test scenario format.

The PRD had sixteen tasks. In the first seven executed tasks, only two passed autonomously — 2 / 7, or 29%. The other five required manual intervention.

The failures weren't the same as Project A's. In Project A, the model hallucinated imports and invented fixtures because it couldn't see the project. In Project C, it could see the project — but it kept hitting runtime patterns our prompt pipeline had not exposed clearly enough:

Pydantic's HttpUrl adds a trailing slash that breaks equality checks
FastAPI's router prefix with "/" vs "" causes 307 redirects
SQLAlchemy async many-to-many relationships trigger MissingGreenlet errors
create_async_engine lives in sqlalchemy.ext.asyncio, not sqlalchemy

I would not classify these primarily as intelligence failures. They looked more like behavioral knowledge gaps — framework-specific quirks that context files alone didn't expose.

Why Context Files Weren't Enough

This forced us to confront a limitation in our Part 4 approach.

Context files are static. They're defined when you write the PRD — before any code exists. By TASK-007, the project has files that weren't there when the PRD was written. The model can't see them unless someone manually updates the list.

For example, TASK-007 needed to create a route that depended on models and schemas generated by TASK-003 and TASK-005. But those files didn't exist when the PRD was written. The context file list was correct at design time — and stale by execution time.

And even when context files are current, they show the model what code exists. They don't teach the model how the runtime behaves. No amount of reading bookmark.py will tell you that bookmark.tags.append(tag) triggers a synchronous database call inside an async context.

We realized we needed two different kinds of information, not one:

Type	What it provides	Source	Example
Structural information	What files exist, what they export, how they relate	Context files, repo maps	"BookmarkCreate has a field `url: HttpUrl`"
Behavioral knowledge	How the runtime actually works, framework quirks, patterns to avoid	Accumulated experience, failure analysis	"HttpUrl adds a trailing slash — use `str(url).rstrip('/')` for comparison"

Project B's success came from fixing structural information. Project C's failures seemed to come largely from missing behavioral knowledge. Both feed into what the model needs, but they come from different places and accumulate differently.

Two Axes, Not One

This is what led us to think about AI coding agent performance along two axes instead of one.

After Part 3, we had DCR — the ratio of decisions handled deterministically. At 85%, the model's job was narrow: just write the code.

After Part 4, we had the Information Design Gap — the model needs enough context to do that narrow job.

Now, after three projects, we're working with a slightly more structured version:

Axis 1: DCR (Deterministic Coverage Ratio) — how much of the decision surface is handled without the model. This is the scaffolding.

Axis 2: Information Quality (IQ) — how well the model is equipped for the decisions it does handle. This is the fuel.

This is not a measured equation — just the mental model that best explains our runs so far:

System Reliability ≈ DCR × Information Quality

DCR narrows the blast radius. IQ determines how well the model performs inside it. In our data so far, you need both. Neither alone has been sufficient.

Three Dimensions of Information Quality

After analyzing our failures across three projects and reading through recent work on LLM-based code generation, we've found that "the model didn't get enough information" breaks down into at least three problems:

Dimension 1 — Availability: Does the information exist in the context window?

This was Project A's problem. The model received ~240 tokens of task-relevant content. The information existed on disk — the orchestrator just never loaded it.

Dillon & Varanasi (2026) observed something similar. They measured whether generated code follows team-level architectural decisions. When a decision was visible in the files the model received, compliance was near-perfect. When it existed only in documents the model never saw, compliance dropped to zero.

Dimension 2 — Selection: Is irrelevant information excluded?

More context isn't always better. Alonso et al. (2026) found that adding procedural TDD instructions increased regressions, while a targeted test map reduced them significantly. The practical lesson for us was simple: token budget is finite.

Hu et al. (2026) quantified this from the other direction. In their cross-file code generation benchmark, 62% of functions didn't need cross-file context at all. The skill is knowing which information to include.

Dimension 3 — Structure: Is the information formatted for the model to use?

This is the counterintuitive one. Information can be available and selected correctly but still fail because of how it's structured.

Hu et al.'s ablation showed this clearly. "Inlined" context (dependencies inserted at relevant code locations) versus "prepended" context (same information at the top of the prompt) — same information, different structure. Removing the inlining degraded performance to nearly the same level as removing the context entirely.

Chinthareddy (2026) found a similar pattern with code retrieval. On a set of architectural queries, a deterministic AST-derived knowledge graph scored 100% correctness while a vector-similarity approach on the same codebase scored 40% (on the Shopizer benchmark suite). The gap came from how relationships were structured, not what information was available.

How the Two Axes Interact

Here's why we think you need both:

Scenario	DCR	IQ	What happens
A	High	High	System has a chance to work. Deterministic decisions are correct, model has what it needs.
B	High	Low	Deterministic decisions are correct, but the model is flying blind in its narrow lane.
C	Low	High	Model generates good code, but the system mishandles it — wrong task order, broken gates, environment failures.
D	Low	Low	Failures become hard to diagnose. This may be what "the model isn't smart enough" often looks like.

Our Project A was Scenario B. DCR was 85% — the harness was solid. But Information Quality was ~15%. The model couldn't do its job because it couldn't see the project.

Project B was closer to Scenario A. Same DCR. At least on the availability dimension, the delivered context moved closer to ~80%. The model had enough context to complete most of the tasks that fit the orchestration loop.

Project C showed us that IQ itself has layers. Structural availability was good (~80%), but behavioral knowledge was missing. The two-axis model held — DCR was fine, IQ was the bottleneck — but the nature of the IQ problem was different from Project A.

A Practical Diagnostic

If you're building or evaluating an AI coding agent, here's the check we now run on our own system:

Step 1: Measure your DCR. List every decision point in one execution cycle. For each one, ask: is this resolved by deterministic code, or does it depend on model output? Count the ratio. If it's below 50%, the scaffolding likely needs reinforcement before the model can succeed.

Step 2: Dump the prompt. Not the template — the actual string that reaches the model at inference time. Read it as if you're a developer seeing this codebase for the first time. Can you write the code from this prompt alone? If you can't, the model can't either.

Step 3: Diagnose by axis.

High DCR, low pass rate → Information Quality problem. Check: are context files loaded? Are descriptions specific enough? Are test assertions reaching the model?
Low DCR, inconsistent results → Structural problem. The model is making decisions that should be deterministic. Move those decisions into code.
Both seem fine, still failing → Might be a genuine model capability limit. Only after that does a model upgrade become the next reasonable hypothesis.

We jumped to Step 3 after Project A. "Qwen3 can't handle JWT auth" was our first diagnosis. Our initial diagnosis was premature. The bigger problem was that the information pipeline was effectively empty. We could have saved ourselves weeks.

Related Work Pointing in a Similar Direction

I didn't set out to build a framework. I set out to figure out why Project A failed. But a consistent pattern kept showing up in recent work:

Alonso et al. (2026) — TDD procedure instructions hurt. Contextual test maps helped. Procedure without context was counterproductive.

Midolo et al. (2026) — Surveyed 50 developers. 14% independently reported "contextual information about other system components" as a missing factor.

Jalil et al. (2025) — Smaller models with TDD and code execution surpassed larger models without those supports.

Dillon & Varanasi (2026) — Decision compliance went from 46% to 95% by adding product context and structured specs. Cost per merge-ready task dropped 68%.

Hu et al. (2026) — Cross-file inlining improved exact match by a reported average of 29.73% on RepoExec across three backbone models. The result was model-independent.

Chinthareddy (2026) — Deterministic AST-derived code graphs achieved 100% correctness vs. 40% for vector-only retrieval on architectural queries (Shopizer suite). LLM-based graph extraction missed 31% of files entirely.

These studies don't prove our framework. But they point in a consistent direction, and our three internal runs are consistent with that direction.

What We're Not Claiming

I want to be precise about the boundaries here.

We're not claiming that model capability doesn't matter. It does — for the non-deterministic slice. A stronger model will generate better code from the same information.

We're not claiming these two metrics capture everything. Latency, cost, context window size, tool use ability — all matter. But in our limited experience, DCR and IQ have explained the largest share of variance in autonomous pass rates.

We're also not claiming this is proven. ForgeFlow is a sample size of one. We have three data points (0%, 67%, 29%) and they're consistent with the two-axis model, but three points don't make a proof.

If anyone has run similar experiments — different scaffolding levels, different context strategies, measured pass rates — I'd genuinely love to compare notes.

The Thesis, So Far

Part 3: "The bottleneck is not model capability, but the verifiability of specifications."

Part 4: "Even after verifiability is constructed, the bottleneck shifts to information delivery."

Now the version we're working with:

"An AI coding agent's reliability seems to be a product of its deterministic coverage and its information quality. Improving either without the other produces a system that is either structurally sound but informationally blind, or well-informed but structurally fragile."

Two axes. One product. Neither alone has been sufficient in our experience.

Measure your DCR. Dump your prompt. Fix the axis that's actually broken. Only after that does a model upgrade become the next reasonable hypothesis. That's the diagnostic that's worked for us. Whether it generalizes is something we're still finding out.

About

I'm Joseph YEO, building ForgeFlow from Seoul, Korea — a local AI coding agent that runs entirely on Apple Silicon, no cloud inference during execution.

This post synthesizes what I've learned from running three projects end-to-end and reading 40+ papers on LLM-based code generation. The two-axis model isn't a proven theory — it's the working diagnostic I use every time a cycle fails. I'm sharing it because it's been useful, and because I'm curious whether others are seeing the same patterns.

How are you handling the "stale context" problem as your agent modifies the codebase? Are you using repo maps, re-indexing on every task, or something else entirely? I'd love to hear what's working.

Follow along:

All models run locally via Ollama 0.23.0 on Apple Silicon M5 Max 128GB. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me.

The Information Design Gap: Why Our AI Agent Was Coding Blind

Joseph Yeo — Wed, 13 May 2026 15:23:44 +0000

This is Part 4 of the ForgeFlow series. Part 3: The Determinism War introduced DCR (Deterministic Coverage Ratio) and why we stopped chasing better models.

In Part 3, I proposed a hypothesis:

"The bottleneck of LLM-driven software engineering is not model capability, but the verifiability of specifications."

Then I said: "We're building the system to test it."

We ran two projects. Same model. Same engine. Same orchestrator. The autonomous pass rate went from 0% to 67%.

In our case, the fix wasn't a better model. It was giving the model enough information to do its job.

Two Projects, One Model, One Engine

ForgeFlow is a TDD orchestrator that runs entirely locally. No cloud API calls during execution. The cycle is simple: generate test (RED) → generate implementation (GREEN) → run pytest → commit or retry.

We ran it against two internal projects:

	Project A: repo-jwt	Project B: todo-api
Domain	JWT authentication API	Todo CRUD API
Tasks	18	12
Model	Qwen3-Coder-Next Q4_K_M (45GB)	Same
Engine	forgeflow.py v2	Same
Autonomous passes	0 / 18 (0%)	8 / 12 (67%)
Manual intervention	18 tasks (100%)	4 tasks (33%)

Same model. Same engine. The pass rate changed from 0% to 67%.

What changed between Project A and Project B was not the model or the orchestrator. It was the information structure of the PRD — the spec document that tells the model what to build.

The Prompt Dump That Exposed the Problem

After Project A failed across all 18 tasks, we did something we should have done much earlier: we dumped the actual prompt the model received at inference time.

Not the prompt template. Not the system prompt spec. The literal string that arrived at the model's context window.

Here's what we expected to find: a rich prompt containing the task spec, relevant source files, test fixtures, data model definitions, and dependency context.

What we actually found was a prompt of about ~720 tokens — and only ~240 of those were task-relevant project information. The rest was role text, formatting rules, and boilerplate.

No source code. No test fixtures. No existing implementation files. The model was being asked to generate code for a project it could barely see.

In hindsight, this was an embarrassing oversight. The information pipeline existed in the code — it just wasn't wired up. But we didn't notice until we read the raw prompt.

The Five Gaps

We listed out every piece of information the model should have received but didn't. Five gaps emerged:

Gap 1 — No context in RED phase. During test generation, the context parameter was hardcoded to an empty dictionary. The model wrote tests for modules it couldn't see, importing functions that didn't exist yet, guessing at fixture structures it had no way to know.

Gap 2 — No context file list in the PRD. The orchestrator had a function ready to read context files from a task-level field. But the PRD never defined that field. So the function returned an empty list every time.

Gap 3 — Module names without signatures. The prompt listed available modules by name: todo.py, database.py. But not their contents, not their function signatures, not their class fields. The model knew that modules existed, but not what they contained.

Gap 4 — Test assertions not forwarded. The PRD included test assertion fields with precise expected behavior. The prompt builder read a different field name. The assertions existed in the spec but never reached the model.

Gap 5 — No conftest.py. In a pytest project, conftest.py defines shared fixtures — test database sessions, HTTP clients, factory functions. The model never saw it. Every task that required a test client, the model invented its own from scratch, often incompatibly.

Quantifying the Gap

We measured how much relevant information actually reached the model by counting task-specific tokens:

Source	Task-relevant tokens	What it contains
What the model received	~240	Task ID, one-line description, module names
What a human developer would reference	~1,640	Above + conftest.py, database.py, models, schemas, existing routes

The model was operating on roughly 15% of the information a developer would use for the same task.

We started calling this the Information Design Gap: the difference between what a model could use and what the system actually delivers at inference time. Whether this framing is useful beyond our system is something we're still figuring out — but for us, it immediately clarified what to fix.

The Fix: No Code Changes

Here's the part that surprised us.

The orchestrator already had the machinery to deliver context files. A function to resolve which files a task needs — existed. A function to read those files from disk — existed. A function to format them into the prompt — existed. The prompt builder had a slot for context.

The pipeline existed. The PRD just wasn't feeding it.

For Project B (todo-api), we made three changes — all in the PRD, none in the engine:

1. Added a context file list to every task. Each task now listed exactly which existing files the model should see. Early tasks had empty lists. CRUD endpoint tasks included the test fixture file, the relevant model file, and the schema file.

Here's what a task spec looked like after the fix:

- id: TASK-007
  description: >
    Create POST /api/todos.
    Use the client fixture from conftest.py.
    Return 201 with TodoResponse schema.
  context_files:
    - tests/conftest.py
    - app/models/todo.py
    - app/schemas/todo.py

2. Made descriptions explicit. Instead of "Create the POST endpoint", we wrote "Create POST /api/todos. Use the client fixture from conftest.py. Return 201 with TodoResponse schema." One sentence, but it told the model where to look.

3. Unified test scenario format. Aligned PRD field names with what the prompt builder actually read, so test assertions reached the model.

Total lines of code changed in the engine: zero.

What 67% Actually Means

Eight of twelve tasks passed autonomously — the model generated code, tests passed, and ForgeFlow committed without human intervention for those tasks.

Based on manual inspection, I would not classify the four manual tasks as model-capability failures. They were structural mismatches with the TDD cycle:

Task	Why manual	Category
TASK-001	Infrastructure setup — test and implementation must be created together	RED phase incompatible
TASK-003	Fixture-only — conftest.py defines fixtures, nothing to "fail" in RED	No failing test possible
TASK-010	Validation already handled by Pydantic schema from an earlier task	RED unexpected pass
TASK-012	Integration test — no implementation file, test-only task	Engine assumes impl file exists

That said, I should be honest about what we can't fully measure here. "Model-capability failure" is hard to distinguish from "subtle information gap we didn't notice." Our classification is based on manual inspection, not a controlled experiment. What we can say with confidence is that the type of failure changed completely — from hallucinated imports and invented fixtures in Project A to structural mismatches in Project B.

The Lesson: Intelligence Gap vs. Information Gap

After Project A, our diagnosis was: "The model isn't smart enough. Qwen3 at Q4 quantization can't handle multi-file JWT authentication."

That diagnosis was wrong — or at least, premature.

The model appeared to have more usable capability than our system was exposing. In this run, the difference between 0% and 67% looked less like intelligence and more like context delivery.

This completely changed how we thought about local model limitations:

	Intelligence Gap	Information Gap
Symptom	Model generates plausible but wrong code	Model generates structurally incompatible code
Diagnosis	"Model too small / too quantized"	"Prompt missing critical context"
Fix	Upgrade model (expensive, diminishing returns)	Improve information design (free, compounding)
Testable?	Hard — model capability is a black box	Easy — dump the prompt, count what's missing

The Information Design Gap is testable. Dump the prompt. Read it as if you're a developer seeing this project for the first time. If you couldn't write the code from that prompt alone, the model can't either.

Similar Patterns in Recent Research

While writing this post, we surveyed recent research on TDD-based code generation and found similar patterns appearing independently. These don't prove our framework, but the convergence seemed worth noting.

Alonso et al. (2026) tested TDD prompting on SWE-bench Verified with a 30B local model. Adding procedural TDD instructions ("write tests first, then implement") increased regressions. Adding a graph-derived test map ("here are the specific tests at risk") reduced them significantly. Their conclusion: agents don't need to be told how to do TDD — they need to be told which tests to check.

We saw the same mechanism: telling the model what process to follow consumed context tokens that could carry actual project information.

Midolo et al. (2026) surveyed 50 developers about what makes code generation prompts succeed. Their top factors: algorithmic details (57%) and I/O format specification (44%). When asked what else was missing, 14% independently reported "contextual information about other components in the system" — which sounds a lot like the gap our per-task context file list was designed to close.

Jalil et al. (2025) showed that smaller models with TDD and a code interpreter could surpass larger models without those supports. The pattern held across model families: tests as structured context beat model scale.

Different benchmarks, different teams, different setups. They all point toward the same practical lesson: before blaming the model, it might be worth inspecting the information pipeline. Our data adds one more point in that direction.

Implications for DCR

In Part 3, I defined DCR as the ratio of deterministic decisions in an agent loop. A reader asked whether DCR should be tracked like test coverage — not just reviewed once at architecture time.

Running two projects gave us a partial answer: DCR alone wasn't enough.

ForgeFlow's DCR didn't change between Project A and Project B. It was 85% both times — same 11 of 13 decisions handled deterministically. Yet performance went from 0% to 67%.

What changed was the quality of information feeding the non-deterministic decisions. DCR tells you how narrow the model's role is. It doesn't tell you whether the model is equipped to play that role.

This is why we're now thinking about DCR in two layers:

Static DCR: how many decision points are designed to be deterministic. (Architecture metric.)
Observed DCR: how many decisions were actually resolved deterministically during real runs. (Runtime metric.)

And alongside both: Information Delivery Rate — how much of the available, relevant context actually reaches the model at inference time. Using task-relevant token delivery as a rough proxy, Project A was around 15%. Project B was much closer to the information a human developer would expect to see.

We're still working out whether these are the right abstractions — but they've been useful for diagnosing our own failures so far.

What We're Building Next

The immediate roadmap based on these findings:

RED phase context delivery. The RED phase (test generation) was still sending an empty context when we ran these projects. We've since fixed this in the engine — the model now sees existing fixtures before writing new tests.

Automatic context inference. Right now, context files are manually specified per task in the PRD. The next step is deriving them from the dependency graph: if TASK-007 depends on TASK-005 and TASK-006, automatically include their implementation files as context. We're exploring tree-sitter-based approaches for this.

Structural mismatch detection. Four of twelve tasks didn't fit the RED-GREEN cycle. We want ForgeFlow to detect these patterns (infrastructure setup, fixture-only, test-only) during PRD validation and handle them with a separate path — not force them through TDD.

The Thesis, Updated

Part 3's thesis was about structure:

"The bottleneck is not model capability, but verifiability of specifications."

Two projects later, I'd extend it:

"In our runs, even after we built verifiability, the bottleneck seemed to shift to information delivery — whether the model receives enough context to use that verifiability."

DCR gave us the harness. Information design made that harness useful. Both seem to be required. Neither alone was sufficient in our experience.

Same model. Same engine. Zero code changes. 0% → 67%. In our case, the difference was information.

Several recent studies point in a similar direction, though from different setups. The practical suggestion I'd offer: if your AI coding agent is underperforming, it might be worth checking what it's receiving before swapping the model. That's what worked for us.

About

I'm Joseph YEO, a solo builder from Seoul, Korea. ForgeFlow is my experiment in pushing local AI agents toward more reliable autonomous execution — no cloud inference during execution, no hand-holding mid-cycle.

This post covers what happened when we actually ran the system from Part 3 against real projects and discovered the gap between having a verification harness and feeding the generator enough context. I'm sharing this because I wish someone had told me to dump the raw prompt before I spent weeks blaming the model.

If you've run into similar issues — or found different solutions — I'd love to hear about it in the comments.

Follow along:

Built over ~33 sessions, May 2026. All models run locally via Ollama 0.23.0 on Apple Silicon. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me.

The Determinism War: Why We Stopped Chasing Better Models

Joseph Yeo — Tue, 12 May 2026 13:56:15 +0000

The biggest upgrade to our AI coding system was not a better model.

It was deleting model calls.

Everyone in AI infrastructure asks: "Which model should we use?" That question is a distraction. After 22 development sprints and a deep dive into 35 research papers while building ForgeFlow — our fully local, TDD-based autonomous implementation system — we landed on a different question:

"How many of our system's decisions can we replace with deterministic rules?"

The answer to that question often matters more than whether you use a frontier model or a 45GB local one.

The Insight That Changed Everything

LLM-based generation is probabilistic unless tightly constrained. Even when you pin temperature and seed, model calls remain poor substitutes for explicit rules when a decision can be mechanically verified.

Consider what a typical agentic coding loop actually decides:

Which task to implement next
Whether the environment is healthy
Whether the generated code has syntax errors
Whether the import paths are correct
Whether the generated file is in scope
Whether the tests pass
Whether to commit, retry, or declare deadlock

None of these require a language model. They're deterministic operations: dependency graph traversal, docker health checks, py_compile, ruff, allow-list comparison, pytest exit code, SHA-256 hashing.

The only judgment that genuinely requires a model is code generation itself — one step in the entire cycle.

We call this ratio the DCR: Deterministic Coverage Ratio.

DCR = deterministic decision points / total decision points

DCR is not a benchmark score. It is a design accounting tool: count every branch in the agent loop, then classify whether it is resolved by code or by model judgment.

ForgeFlow's current DCR: ≈ 85% — 11 of 13 decision points per cycle are deterministic.

Decision	Deterministic?	Mechanism
Next task selection	✅	dependency DAG traversal
Dependency satisfied?	✅	status field check
Syntax valid?	✅	py_compile
Style gate pass?	✅	ruff
Import paths correct?	✅	Phase 0.5 AST correction
File in scope?	✅	allow-list comparison
Environment healthy?	✅	docker / ollama / disk check
Test code generation	❌	local LLM
Implementation generation	❌	local LLM
Tests pass?	✅	pytest exit code
Commit / Retry / Deadlock?	✅	gate_decision (rule-based)
Failure type?	✅	stderr pattern matching
Deadlock detected?	✅	failure signature × 3

Why DCR Is the Right Metric

Recent inference-time scaling research keeps pointing to the same condition: repeated sampling becomes powerful when there is an automatic verifier.

Without an oracle, more samples just create more candidates to rank. With an oracle, more samples create more chances to find a correct answer.

A few papers that converge on this pattern:

Large Language Monkeys (Brown et al., ICLR 2025): Coverage scales with sample count. In coding tasks with automatic verification, repeated sampling with a cheaper model consistently outperforms a frontier model run once.
"The Larger the Better?" (Hassid et al., 2024): Under a fixed compute budget, smaller code models sampled multiple times can match or outperform larger models — when unit tests are available to select the correct answer.
"Do We Truly Need So Many Samples?" (2025): Multi-model repeated sampling improves sample efficiency. Smaller model combinations approach larger-model performance with far fewer samples, given a verifier.

The critical shared condition: automatic verification must exist.

Unit tests. pytest. An oracle that says "yes" or "no" without human judgment.

This is where most real-world projects diverge from benchmarks. HumanEval and MBPP ship with test suites. Your JWT authentication service doesn't.

The bottleneck isn't model capability. It's verifiability.

Verifiability Is Constructed, Not Given

Existing research operates in "Verifiability as Given" mode: benchmarks provide the oracle, models generate code, oracle decides.

ForgeFlow operates in "Verifiability as Constructed" mode: start with a natural-language requirement and systematically transform it into something testable before any model is ever called.

We call this the Verifiability Transformation Pipeline:

Stage 1: Specification Crystallization
Natural language → numerically precise spec. Zero ambiguous terms ("appropriately", "as needed", "etc." are banned). Every requirement is anchored by a concrete test assertion:

POST {title: 'Buy milk'} → 201, body contains {id, title}

Stage 2: Atomic Decomposition
One task = one test file + one implementation file. An ordered dependency DAG ensures each task's dependencies are already verified before it runs. This eliminates "cross-task contamination" — the pattern where one task's test setup silently breaks another task's environment.

Stage 3: Assertion Embedding
The test contract is written into the spec itself, at pytest-assertion granularity. TDD RED phase uses this to generate the test first. The oracle exists before any code does.

The goal is not to claim that benchmark scaling laws automatically transfer to every real-world project. The goal is to reshape your project until enough of it has benchmark-like properties: explicit specs, executable tests, and a deterministic oracle. Once you have that, repeated sampling and local inference become much more interesting.

The Substitution Paradox

Here's the part that feels contradictory:

Replacing non-deterministic judgment with deterministic rules often requires the most creative kind of intelligence.

Designing the rule "run py_compile before the LLM sees the output" seems obvious in retrospect. But reaching that design requires prior reasoning: that syntax errors are a deterministic category, that they can be caught pre-LLM, and that relying on the LLM to catch its own syntax errors is inherently fragile.

That's inference. Creative, structural inference.

This is why ForgeFlow has a three-tier intelligence structure:

Tier	Role	Actor	Frequency
C — Design	"Which decisions can become deterministic?"	Claude + Joseph	Once per system
B — Execute	"Generate code matching this spec"	Qwen3 45GB (local)	Every cycle
A — Verify	pytest, py_compile, ruff, gate_decision	forgeflow.py	Every cycle, ≈ free

The cloud model designs the harness. The local model runs inside it. The deterministic verifier judges the result.

The strongest model's job is to make the weakest model capable.

The Economics

Cloud model, direct execution:

Cost = (1 / P(success)) × price_per_call

As the problem gets harder, tries increase. As the model gets stronger, price increases. Cost diverges in both directions.

ForgeFlow (DCR maximization):

Cost = design_cost (one-time) + N × marginal_local_inference_cost

The marginal cost is not zero — electricity, time, and thermal throttling all exist. But it is low enough that repeated attempts become operationally feasible in a way that repeated cloud calls are not. Design cost is high, but it is a constant.

This is the same computation the "Large Language Monkeys" paper runs — applied to a system where the oracle is constructed rather than assumed.

What This Means in Practice

For local AI users: The question isn't "is my model good enough?" It's "have I built a harness where my model only needs to do code generation, and everything else is handled deterministically?"

For AI system designers: DCR is a design maturity metric. DCR 30% means you're relying on the model for most decisions. DCR 85% means the model is a narrow specialist in a well-guarded context. Measure yours.

For teams debating cloud vs local: Once DCR is high enough, model selection becomes an economic and security variable — not a technical one. The same harness runs with a local model overnight or a cloud model during the day. Same gates. Same oracle. Different inference cost.

The Thesis in One Sentence

"The bottleneck of LLM-driven software engineering is not model capability, but the verifiability of specifications — and once verifiability is systematically constructed, a 45GB local model running overnight can match a frontier cloud model running once."

We think this is true. We're building the system to prove it.

What's your system's DCR?

About

I'm Joseph YEO, a solo builder from Seoul, Korea. ForgeFlow is my experiment in pushing local AI agents toward full autonomy — no cloud inference during execution, no hand-holding mid-cycle.

This post is about the design principle behind that experiment: maximizing the ratio of decisions that don't require a model at all.

Follow along:

Built over ~26 sessions, May 2026. All models run locally via Ollama 0.23.0 on Apple Silicon. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me. I use AI tools to write, just like I use them to code. That's kind of the whole point.

We Didn't Migrate from n8n to Python Because n8n Failed

Joseph Yeo — Mon, 11 May 2026 14:58:28 +0000

We migrated because our AI orchestrator became something that needed tests, trust boundaries, and deterministic behavior.

ForgeFlow is a fully local, multi-agent TDD system — a planning agent (Claude) designs a spec document set, and a local 45GB LLM executes TDD cycles overnight on Apple Silicon. During our initial development iterations, n8n was the orchestrator. It felt like the right call. Visual workflow, no-code glue, easy HTTP nodes, built-in error routing. What's not to like?

Turns out, quite a lot.

What n8n Got Right

To be fair, n8n earned its place early on. When we were validating the core loop — call LLM → apply files → run pytest → branch on result — the visual canvas was genuinely useful. You could see the entire decision tree at a glance. Non-engineers could follow it. Debugging a single node failure was fast.

And for a prototype, that matters. We got to a working TDD loop in far fewer iterations than a pure Python approach would have required.

Where It Started Breaking

Friction mounted as the system's architectural requirements became more stringent.

Problem 1: The fetch() wall

n8n's Code node runs in a sandboxed Node.js environment. When we tried to read existing source files before passing them as context to the LLM — a critical step for our TDD approach — fetch() to our local exec server failed silently. In our setup, $helpers.httpRequest() didn't give us the control we needed either. Whether this was configuration-specific or sandbox-related, the practical result was the same: file reads became workflow plumbing instead of ordinary code. The workaround was to spin up a separate HTTP Request node just for file reads.

This wasn't a bug. It was the architecture saying: "Code nodes are for data transformation, not I/O." Which is correct! But we needed I/O.

Problem 2: The trust boundary was wrong

In our system, the rule is absolute: only the orchestrator may modify files or run git commands. The LLM proposes. The orchestrator decides and acts.

In n8n, enforcing this required careful node discipline. Nothing in the framework prevents you from firing off a git commit in any arbitrary Code node. The constraint was cultural, not structural. And cultural constraints erode.

In Python, the constraint is structural: apply_files(), git_commit(), gate_decision() are functions in a specific layer (L1/L2). Nothing in L3 (the LLM layer) can call them directly. The layer separation is enforced by code organization, not discipline.

Problem 3: Deterministic logic deserves deterministic code

The most critical parts of our system — syntax validation, failure signature hashing, gate decision logic — are 100% deterministic. Not a single LLM call. Pure logic.

Expressing that logic in n8n's Code nodes felt like writing a chess engine in PowerPoint. It worked, technically. But every if/elif chain lived in a text field, untestable and unversionable in any meaningful way.

We wanted to write this:

def compute_failure_signature(failure_type: str, target_file: str, stderr_text: str) -> str:
    payload = f"{failure_type}:{target_file}:{stderr_text[:200]}"
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

And then test it directly:

def test_failure_signature_is_stable():
    sig1 = compute_failure_signature("patch", "auth.py", "IndentationError line 42")
    sig2 = compute_failure_signature("patch", "auth.py", "IndentationError line 42")
    assert sig1 == sig2

def test_failure_signature_differs_by_file():
    sig1 = compute_failure_signature("patch", "auth.py", "IndentationError line 42")
    sig2 = compute_failure_signature("patch", "models.py", "IndentationError line 42")
    assert sig1 != sig2

You can approximate that with a Code node, but not as a first-class, importable, directly testable artifact. The difference matters when this logic is what determines whether a 45GB model gets another retry or hits deadlock.

Problem 4: Cross-model verification became essential

By our 21st development iteration, we brought in an independent reasoning model (different vendor, no shared context) to write test_forgeflow.py — 110 tests for our orchestrator's behavior, written solely from our spec documents.

It found 7 real mismatches between our spec and our implementation.

To run this process, we needed: a well-defined function contract, stable function signatures, and a test suite that could run in CI. None of that is natural in n8n. All of it is natural in Python.

The Migration Decision

The turning point was a single question: "Can we write a unit test for this logic?"

If the answer was no — and with n8n, it often was — that was a design smell. Critical, deterministic logic that can't be tested independently is a liability.

We rewrote the orchestrator as forgeflow.py with a three-layer trust model:

L1: TRUST (immutable)      — File I/O, Git, Docker, State, Cycle Control
L2: VERIFY (deterministic) — Syntax Gate, correction engine, gate_decision
L3: DOUBT (always suspect) — LLM backends, JSON extraction, prompt assembly

The design invariant: L1 and L2 never call an LLM. All LLM interaction is isolated in L3.

This made the migration feel less like a rewrite and more like making implicit structure explicit.

What We'd Tell Our Past Selves

Use n8n for: Prototyping agent loops, visualizing complex branching logic for stakeholders, integrating heterogeneous external APIs with minimal glue code.

Don't use n8n for: Any logic you'll want to unit-test. Any constraint that needs to be structurally enforced rather than culturally agreed upon. Any system where the orchestrator's behavior is itself the thing being verified.

The moment your AI orchestrator has enough opinions to deserve a test suite, it's time to write real code.

The Result

The important change wasn't "Python instead of n8n." It was that the orchestrator became verifiable.

106 behavioral tests pass against stable function contracts
Three execution paths (COMMIT / RETRY / DEADLOCK) are validated independently via dry-run
Trust boundaries are structural: the LLM layer has no reference to file mutation or git operations
Cross-model verification is repeatable: any external model can re-run tests against the same spec

n8n helped us discover the loop. Python let us verify it.

ForgeFlow runs entirely on local hardware — Apple Silicon M5 Max 128G, 45GB model, no cloud inference during execution. If you're building AI agent systems and hitting the same orchestration walls, whether with n8n, LangGraph, or CrewAI — what was your inflection point? The trust boundary question is one most teams hit eventually.

About

I'm Joseph YEO, a solo builder from Seoul, Korea. ForgeFlow is my experiment in pushing local AI agents toward full autonomy — no cloud inference during execution, no hand-holding mid-cycle.

This post is about one architectural decision that turned out to matter more than I expected: moving the orchestrator from n8n to Python, and why testability was the forcing function.

Follow along:

Built over ~22 sessions, May 2026. All models run locally via Ollama 0.23.0 on Apple Silicon. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me. I use AI tools to write, just like I use them to code. That's kind of the whole point.

I Built a Local AI Coding Agent on M5 Max 128GB — It Failed 164 Times Before Passing 35 Tests

Joseph Yeo — Sat, 09 May 2026 11:31:07 +0000

Fully local. No cloud APIs during execution. TDD-enforced. 35 tests passing.

To be clear: I used Claude for the initial architecture and rule design. The experiment was strictly focused on whether a local LLM could survive the autonomous execution loop without phoning home. Planning, docs, and correction rule design I handled with Claude (a cloud API). The coding agent loop (Brain → Coder → Tester) ran 100% locally — no API calls during execution.

The Actual Setup

Most AI coding agent posts you see rely on GPT-4o or Claude via API. The model lives in a data center, and you're paying per token. That's fine — but it means your code, your architecture decisions, and your project context are all leaving your machine.

I wanted something different: a multi-agent system that runs entirely on my MacBook Pro M5 Max 128GB. It autonomously writes code, runs tests in a Docker sandbox, and only commits when tests pass. No internet required once it's running.

This is the story of ForgeFlow — what I built, what broke, and what the data showed.

Hardware Context

The M5 Max 128GB is unusual hardware for this kind of work. Most local LLM setups top out at 32GB or 64GB unified memory, which forces you to choose between model quality and running multiple models simultaneously. At 128GB, that constraint disappears.

What I ran simultaneously:

Model	Size (Q4_K_M)	Role
Qwen3-Coder-Next	~45GB	Brain + Coder
gemma4:26b	~17GB	QA
nomic-embed-text	~0.3GB	RAG embeddings

Total: ~62GB loaded, ~66GB headroom for OS + KV cache. Both models stay warm in memory with keep_alive: 24h — no reload latency between cycles.

This isn't a flex. It's context: the architectural decisions I made (same model for Brain and Coder, both models always loaded) are only feasible at this memory tier. At 64GB, you'd need to make different tradeoffs.

The Architecture: What ForgeFlow Actually Does

ForgeFlow is an n8n workflow that runs every 10 minutes, autonomously picks the next coding task, writes tests first, writes code second, and only commits if all tests pass.

The full loop:

Schedule Trigger (10 min)
  → Load Context (working memory + results log + project rules)
  → Brain (Qwen3-Coder-Next): pick next task from PRD
  → Localization: RAG search for relevant existing code
  → Coder RED (same model): write a failing test
  → Verify RED: pytest must FAIL — if it passes, the test is wrong
  → Coder GREEN: write minimum code to pass the test
  → Phase 0 Gate: py_compile + ruff (deterministic, no LLM)
  → QA (gemma4:26b): run full test suite in Docker sandbox
  → Gate Decision: COMMIT / RETRY / DEADLOCK / ESCALATE
  → Commit & Update (on pass)

Three design principles drove every decision:

1. pytest exit code is the only truth. I don't care if the LLM thinks the code is "clean." If the pytest exit code isn't 0, the code is garbage.

2. The LLM proposes, n8n disposes. No model has write access to the filesystem or git. n8n is the only actor that applies files, runs git commands, and updates state.

3. Deterministic gates before LLM gates. py_compile and ruff run in under 0.5 seconds. If they catch the error, there's no reason to spend 30 seconds calling gemma4.

The Memory System

One of the underrated problems in autonomous coding agents is state management across cycles. The agent can't remember what it did last cycle unless you explicitly store it.

ForgeFlow keeps track of state across six memory layers:

Layer	Storage	Scope
Git history	`.git`	Permanent
Code summaries	ChromaDB (RAG)	Project lifetime
results.tsv	TSV file	Session
AGENTS.md	Markdown	Cross-session
Working memory	JSON file	Current loop
Failure patterns	AGENTS.md auto-update	Generalized

Each layer operates at a different time scale. Git is permanent. Working memory resets every cycle. AGENTS.md accumulates lessons across sessions — when the same failure type occurs 3+ times, a rule gets written: "always include from app.database import get_db — the model consistently forgets this."

TDD Enforcement: Red-Green-Refactor as a System Constraint

The TDD loop isn't a suggestion — it's mechanically enforced by the workflow:

RED phase: Coder writes a test. n8n runs pytest. If it passes, the test is rejected — it's testing something that already works, which means it's the wrong test.
GREEN phase: Coder writes minimum code to pass the test. n8n applies the files, runs the full test suite (not just the new test), checks for regressions.
Commit: Only happens if exit code is 0 across the entire test suite.

Enforcing this mechanically means the model can't shortcut. It can't write "good enough" code and hope the reviewer misses it. The test either passes or it doesn't.

Failure Handling: Bounded Repair

Blind retries are a token-burn trap. Instead, ForgeFlow fingerprints every failure:

failure_signature = SHA256(failure_type + file_path + first_50_chars_of_stderr)[:12]

If I see the same SHA256 signature three times, the agent hits a DEADLOCK and walks away. It's better to skip a task than to let a model hallucinate in a loop.

Failure classification:

Type	Description
`patch`	Code logic error, syntax error
`environment`	Import error, missing module
`localization`	Wrong file referenced
`deadlock`	Same signature 3×

The Data: What Actually Happened

I ran ForgeFlow on a Todo REST API (FastAPI + SQLAlchemy + pytest) — 12 tasks, classic CRUD.

Overall:

Metric	Value
Total attempts	164
PASS (committed)	11 (6.7%)
FAIL (discarded)	116 (70.7%)
DEADLOCK (skipped)	37 (22.6%)
Manual interventions	3
Final test count	35 passing

The 6.7% raw PASS rate sounds bad. But that number is misleading — it includes the early cycles before deterministic corrections were added.

The real signal is in the pass rate as the system "learned" (via manual rules):

Corrections active	PASS rate
0–5 corrections	5.6%
6 corrections	0%
7–10 corrections	40.0%
11–13 corrections	62.5%

Each "correction" is a deterministic rule applied before the LLM output reaches the filesystem. Examples:

from app.db.session import → auto-rewrite to from app.database import
@router.post(...) without status_code=201 → auto-insert
File not in target_files → reject with error message

As corrections accumulated, PASS rate went from 5.6% to 62.5%. The corrections are essentially a hand-built knowledge base of the model's systematic errors. It turns out those errors are highly consistent and predictable.

Failure type distribution:

Type	Count	%
patch	99	64.7%
environment	54	35.3%

At 35.3%, my environment failure rate is triple the standard benchmarks (~13%). That's the "quantization tax" you pay for running Q4 models locally. The deterministic corrections target exactly these failure types.

Hardest tasks:

Task	Attempts	Primary failure
TASK-002 (DB model)	41	PytestDeprecationWarning + ImportError
TASK-006 (GET list)	26	ImportError conftest
TASK-012 (integration)	22	Regression (previous code overwritten)

TASK-002 taking 41 attempts is the starkest number. Most failures were the same PytestDeprecationWarning signature — the model couldn't fix a pytest configuration issue that required understanding the test infrastructure, not just the code under test. Eventually, a manual intervention resolved it.

What Broke (Honestly)

3 manual interventions were required:

TASK-008 (PUT endpoint): The Coder kept generating tests with wrong status codes. Added correction #13 (PUT 201→200 auto-fix) after diagnosing the pattern.
TASK-011 (filtering): The Coder overwrote routes/todo.py while working on filtering, destroying previously committed code. The target_files violation detection wasn't blocking writes — only logging them.
TASK-012 (integration test): DEADLOCK 3 times. The model couldn't figure out that test_integration.py needed to use the existing client fixture from conftest.py rather than creating its own TestClient.

All three were fixed in the session after they occurred by adding deterministic corrections. The system learned — just not automatically.

What This Is (and Isn't)

This is:

A proof that a fully local multi-agent TDD loop is viable on consumer hardware
Evidence that deterministic corrections significantly outperform raw LLM retry for systematic errors
A framework for thinking about autonomous coding at the task level

This isn't:

A "set it and forget it" system. It's a force multiplier that still requires a human to untangle the logic when the model hits a wall.
A system that works without oversight (3 interventions in 12 tasks is not zero)
Generalizable beyond the hardware tier that makes it feasible

The 62.5% PASS rate in the final correction set is meaningful. But the 3 required manual interventions mean the system isn't yet fully autonomous.

What's Next

Second project: A more complex backend (20+ tasks, non-trivial dependencies) to validate that the correction set generalizes and the dependency resolution logic holds under pressure. The goal is a two-project dataset for a proper write-up.

Phase 0.5 Gate: I'm looking at implementing AST-based checks — inspired by the Khati et al. (2026) paper — to kill hallucinations before they even hit the Docker sandbox. The goal is to catch app.routes.todo.get_todo_by_id (doesn't exist) before it reaches pytest.

Automatic correction learning: Right now, corrections are written manually after pattern identification. The next step is having n8n automatically identify recurring failure signatures and propose corrections for human approval.

Hardware Note for Other M-Series Users

If you're on M2/M3/M4 Pro (36–48GB), the same architecture works with tradeoffs:

Run one model at a time (swap between Brain/Coder and QA)
Use smaller QA model (gemma4:9b instead of 26b)
Expect higher latency per cycle (~15–20 min instead of 10)

The fundamental approach — deterministic orchestration + LLM proposal + test-as-truth — doesn't require 128GB. It just runs faster and with better models at that tier.

About

I'm Joseph YEO, a solo builder from Seoul, Korea. I run multiple projects in parallel using AI agents — local AI automation (ForgeFlow), supply chain security (DevRadar Guard), and a few things currently under wraps.

What I'm really interested in is how autonomous these agents can actually become before I have to step in as the human. ForgeFlow is one experiment. There will be more.

Follow along:

Built over ~7 sessions, May 2026. All models run locally via Ollama 0.23.0 on macOS. No cloud APIs were used during autonomous execution.

This post was drafted with Claude and edited by me. I use AI tools to write, just like I use them to code. That's kind of the whole point.

Case Study: How I Dogfood DevRadar Guard on a 954-Dependency Project

Joseph Yeo — Mon, 06 Apr 2026 13:25:37 +0000

This is a follow-up to my earlier post: Axios Was Compromised. Here's What It Means for Your Repo.

The Setup

GloriaPPT is a presentation tool I built and maintain. It's a fairly typical modern JavaScript app: a Next.js frontend, a Node.js backend, and deployment on Vercel. What makes it interesting for this case study is its dependency tree: 954 npm packages in the lockfile.

Most of those packages are transitive. I haven't read the source code for most of them, and realistically, neither do most small teams. If one of them were compromised tomorrow, I probably wouldn't know right away.

That's the problem I built DevRadar Guard to solve.

Before DevRadar Guard

Dependency monitoring: Manual npm audit when I remembered to run it
Supply chain alerts: None — I found out about incidents from social feeds and security threads
.npmrc hardening: Default settings
CLAUDE.md security section: Didn't exist
Pre-install hooks: None
CI/CD security checks: Basic Dependabot, no custom policy
Response time to incidents: Hours to days, depending on when I saw the news

After DevRadar Guard

I deployed DevRadar Guard's hosted monitoring on a small VPS that checks every 30 minutes. Here's what changed.

Signal Collection

The Signal Engine collects threat intelligence from GitHub Security Advisories every 30 minutes. In the first 24 hours, it ingested 467 raw events — advisories affecting npm packages — and normalized all of them into structured threat candidates with confidence scores.

Each signal is scored across five dimensions, including source quality, technical specificity, cross-reference validation, discussion velocity, and ecosystem relevance.

Exposure Matching

Out of 467 normalized signals, the Exposure Engine matched 1 against GloriaPPT's actual dependency tree: axios.

Package: axios
Installed version: 1.14.1
Confidence score: 65/100
Exposure score: 50/100
Final risk: 57/100 (alert threshold: 50)

The confidence score reflects signal quality: a high-trust source, a named package and version, and enough technical detail to treat the advisory seriously.

The exposure score reflects how directly the issue touched this repo: axios was a direct dependency, and the affected version was present in the lockfile.

The Alert

At 00:24 KST (UTC+9) on April 6, a Slack alert landed in #devradar-alerts:

🛡️ DevRadar Guard Alert

Package: axios
Version: 1.14.1
Risk Score: 57/100
Confidence: 65
Exposure: 50
Path: direct

Signal: Compromised axios versions 1.14.1 and 0.30.4 were
reported to deliver a remote access trojan...

I didn't find out about the axios compromise from social media. The alert was waiting for me when I checked Slack.

Guardrail Bundle

DevRadar Guard generates a guardrail bundle — a set of files you can drop into a repo to harden installs, guide AI coding agents, and surface risky dependency changes during review:

File	Purpose
`CLAUDE.md` security section	Security policy for AI coding agents
`.npmrc` hardening	`ignore-scripts=true`, `audit=true`, registry pinning
Pre-install hook	Warns before installing packages younger than 7 days
GitHub Actions workflow	PR check that flags risky dependency changes (alert-only)
`devradar-policy.json`	Machine-readable policy for CI/CD integration

GloriaPPT now uses all 8 generated guardrail files. The pre-install hook would likely have flagged the malicious plain-crypto-js dependency used in the attack, since it had been published less than 24 hours earlier.

The Numbers

Metric	Value
Dependencies monitored	954
Raw threat signals collected (first 24h)	467
Normalization success rate (first 24h sample)	100%
Signals matched to GloriaPPT	1 (axios)
False positives in this case study	0
Time from advisory to alert	< 30 minutes (cron cycle)
Guardrail files generated	8
Manual intervention during detection	0

What This Doesn't Prove

I want to be honest about what this case study shows and what it doesn't.

It shows: A real supply chain threat was detected, matched to a real project, and surfaced as an actionable alert — automatically, without manual intervention.

It doesn't show: DevRadar Guard catching a zero-day before anyone else. The axios advisory was already published when my pipeline picked it up. I'm not claiming to be faster than GitHub Advisory. I'm claiming to be faster than manual monitoring — finding out from social feeds, security threads, or a post after the fact.

It doesn't show: Protection against all supply chain attacks. The Signal Engine currently monitors GitHub Advisories only. Reddit, npm registry anomaly detection, and other sources are planned but not yet active in Alpha.

It doesn't show: Automatic blocking. DevRadar Guard Alpha is alert-only. No PR failures, no install blocks, no surprises. You get the information; you decide what to do.

Try It

DevRadar Guard is still in Alpha, and I'm testing it with a small number of pilot teams. Right now that includes hosted monitoring on a 30-minute cycle, matched alerts in Slack or Discord, a generated guardrail bundle for the repo, and a weekly threat briefing. All I ask in return is a few minutes of feedback each week.

If your project has a package-lock.json and you want earlier, repo-specific visibility into supply chain incidents, the starter kit and waitlist are below:

DevRadar Guard Alpha — alert-only, no automatic blocking. You stay in control.

Axios Was Compromised. Here's What It Means for Your Repo.

Joseph Yeo — Mon, 06 Apr 2026 03:58:03 +0000

On March 31, 2026, the axios npm package — with over 100 million weekly downloads — was compromised and used to distribute malware.

A threat actor took over the lead maintainer's npm account, published two backdoored versions (1.14.1 and 0.30.4), and added a hidden dependency that deployed a cross-platform remote access trojan. The payload targeted Windows, macOS, and Linux. The malicious versions were live for only about three hours before they were removed.

In practice, three hours was enough.

Microsoft attributed the attack to Sapphire Sleet, a North Korean state actor. Google's Threat Intelligence Group confirmed UNC1069 involvement. This was a coordinated, pre-staged operation — the malicious dependency was planted 18 hours before activation.

Why This Matters to You

If your package.json uses caret ranges like ^1.x, a routine npm install could have pulled the compromised version automatically. No unusual action required. Just your normal CI/CD pipeline doing what it was designed to do.

Most teams would not have caught this in time.

Not because they're careless, but because the tooling gap is real:

npm audit looks for known CVEs. This wasn't a CVE when it hit.
Dependabot follows published advisories. This version came from the real maintainer account.
Lockfiles help, but only if they're pinned and not being updated automatically.

The teams that stayed safe had one thing in common: they treated dependency management as part of their security practice, not just routine package maintenance.

What Happened in Our Setup

I maintain a project called GloriaPPT — a typical Next.js app with 954 npm dependencies. When the axios advisory dropped, I wasn't refreshing Twitter. I got a Slack alert.

I built DevRadar Guard to answer one practical question fast: does this incident actually touch one of my repos? In this case, the flow looked like this:

Signal Engine picked up the GitHub Advisory within its 30-minute collection cycle.
Exposure Engine matched it against GloriaPPT's package-lock.json, where axios was a direct dependency.
Guardrail Engine sent a Slack alert with the risk score, confidence level, and affected version.

No manual checking. No scrolling through threads or advisories. The alert landed with the information I needed to decide what to do next.

Axios Was One Incident. The Pattern Keeps Repeating.

Axios will be patched. Credentials will be rotated. Postmortems will be published.

But the pattern repeats. Before axios, it was event-stream. Before that, ua-parser-js. The attack surface keeps growing with every install that pulls in packages your team didn't explicitly choose or review.

The question isn't whether the next supply chain attack will happen. It's whether your repo will know about it before your CI/CD pipeline installs it.

What You Can Do Today

Even without new tooling, these steps can reduce your risk right away:

Pin your dependencies. Remove ^ and ~ from critical packages. Use exact versions.
Set ignore-scripts=true in .npmrc. In this incident, that setting would have blocked the malicious install script.
Review your lockfile after every install. If a new transitive dependency appears that you didn't add, investigate.
Audit your CI/CD pipeline permissions. Does your build environment need network access during npm install?

What We're Building

I'm also building DevRadar Guard around this workflow: early signal collection, repo exposure checks, and guardrail generation. Part of it is open source, including starter config for .npmrc, pre-install hooks, CLAUDE.md (security policy for AI coding agents), and GitHub Actions.

DevRadar Guard is still in Alpha and runs in alert-only mode. No automatic blocking, and no surprise PR failures. You stay in control.

If your team depends on npm and this workflow sounds useful, take a look at the starter kit or join the waitlist: