Building the AI Dark Factory — Issue #20
Preface
I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.
Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com
Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video
This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.
This is the issue that earns the honest in "learning in public."
Nineteen sessions. Fifty-four hours of estimated work. Twenty Gherkin scenarios. A working CI/CD pipeline. A skills library. ADRs for every load-bearing decision. Three evals. Two runbooks. An order cancellation endpoint that passed 6/6 on first attempt.
And the question I have been seeding since Issue #6: was it worth it?
Here are the numbers.
The time audit
Every finding file across nineteen sessions, reviewed and estimated.
| Issue | Topic | Impl | Spec | Infra | Debug | Total |
|---|---|---|---|---|---|---|
| #2 | WireMock + Gherkin | 60 | 30 | 60 | 90 | 240 |
| #3 | Agent implementation | 90 | 30 | 10 | 30 | 160 |
| #4 | Pact contract testing | 60 | 30 | 60 | 90 | 240 |
| #5 | Spec quality demo | 20 | 90 | 10 | 20 | 140 |
| #6 | CI/CD pipeline | 20 | 10 | 90 | 90 | 210 |
| #7 | Scope + notification | 50 | 90 | 20 | 30 | 190 |
| #8 | Spec audit | 10 | 120 | 30 | 20 | 180 |
| #9 | Skills infrastructure | 10 | 30 | 90 | 10 | 140 |
| #10 | 3-tier architecture | 10 | 30 | 100 | 10 | 150 |
| #11 | Non-human callers | 10 | 30 | 70 | 20 | 130 |
| #12 | Skill review | 10 | 20 | 80 | 60 | 170 |
| #13 | Skill audit | 10 | 20 | 100 | 10 | 140 |
| #14 | Memory wall | 10 | 30 | 70 | 10 | 120 |
| #15 | Production CLAUDE.md | 20 | 20 | 80 | 10 | 130 |
| #16 | ADRs | 60 | 20 | 70 | 40 | 190 |
| #17 | Evals | 10 | 20 | 100 | 20 | 150 |
| #18 | Runbooks | 10 | 10 | 100 | 30 | 150 |
| #19 | Full stack | 90 | 60 | 30 | 90 | 270 |
| Total | 650 | 740 | 1,170 | 680 | 3,240 | |
| % | 20% | 23% | 36% | 21% | 54 hrs |
Implementation plus specification — features and their contracts — is 43% of total time. Infrastructure alone is 36%. Debug is 21%.
If you collapse specification and infrastructure together as "non-feature work," the split is 20% feature code versus 80% everything else.
That is the J-curve in a single ratio.
The curve
The J-curve is visible in infrastructure ratio across the five phases of the project:
| Phase | Issues | Infra ratio | New scenarios |
|---|---|---|---|
| Foundation | #2–#5 | 27% | 11 |
| CI + Spec | #6–#8 | 46% | 2 |
| Skills (Layer 2) | #9–#13 | 59% | 0 |
| Stewardship (Layer 3) | #14–#18 | 71% | 0 |
| Assembly | #19 | 21% | 5 |
Infrastructure ratio went 27% → 46% → 59% → 71% → 21%. The trough is Issues #14–#18. Issue #19 is the uptick.
Ten consecutive sessions with zero new features. On a traditional productivity graph, this is a canyon between two spikes.
One pattern worth naming before the honest answers: as infrastructure investment increased, debug time as a percentage of total session time decreased. Issues #9–#18 averaged 13% debug. Issues #2–#5 averaged 30%. The infrastructure did not eliminate discovering new failure modes when new code is written — Issue #19's 33% debug proves that. What it eliminated was re-deriving the same failure modes across sessions.
The overhead taxonomy
Not all overhead is equal. Three categories.
Necessary overhead is overhead that directly prevented a production failure or caught a real bug.
Pact contract testing: Issue #6's deliberate breaking change proved it. Renaming status to result in the payment stub passed all 11 Gherkin scenarios and failed only the Pact job. No behavioral test caught it. Pact catches field-level contract drift; Gherkin catches behavioral drift. Both categories of drift reach production if only one layer exists.
ADRs and evals for un-testable invariants: the notification-synchronous change passes all 20 tests. There is no behavioral test that asserts the notification call is asynchronous. ADR-002 and the Operation Scope eval Q3 are the only protection. This is not overhead that could be offloaded to tests — it is protecting an invariant that tests cannot express.
Transitional overhead is overhead that was high while being built and declined as infrastructure was established.
The Gherkin quality skill took three sessions to build (Issue #9: created, Issue #11: stress-tested, Issue #12: reviewed). After Issue #12, every subsequent session that touches a feature file benefits from v2.0's four guards. Issue #19 demonstrated the return: 10 debt items caught in drafts before implementation, zero post-implementation spec rewrites.
The CI/CD pipeline took one 210-minute session. It runs automatically on every subsequent push. The cost is paid once; the benefit compounds indefinitely.
Avoidable overhead is overhead that better sequencing would have prevented.
Building v1.1 before running the five-dimension review cost approximately 90 minutes across Issues #11–#12. If the review framework had existed before v1.1 was published, the idempotency failure modes and the routing signal length would have been caught before the stress tests required them.
The CI port conflict cost 30 minutes of debugging. If the mock server lifecycle invariant had been documented before writing ci.yml, the double-start step would not have been written.
The Issue #5 intentional test failure left on main required a baseline fix in Issue #6 before CI could be enabled. Avoidable if the failing test had been quarantined to a branch.
Six honest answers
Q1: When did the infrastructure start paying for itself?
Issue #19 is the first measurable payoff. Five scenarios written with zero spec rewrites after implementation. The Gherkin skill caught 10 debt items in drafts first. The evals confirmed the notification invariant before code was written. The step-definition-style skill conventions were followed correctly on the first pass.
The infrastructure paid for itself in a single session. That does not mean it was efficient — 17 sessions of investment for 1 session of payoff is not a favorable ratio if the project ends here. The infrastructure becomes a good investment when Issue #19 is not the last feature session.
Q2: Under what conditions is this approach worth it?
The break-even point is approximately 8 implementation-heavy sessions after the infrastructure is in place. The full skills and constraints infrastructure took 10 sessions to build. Each implementation session with the infrastructure in place saves approximately 60–90 minutes compared to a session without it — fewer re-derived patterns, fewer spec rewrites, fewer dangerous improvements that pass tests. At 75 minutes of savings per session, the math requires ~20 implementation sessions to recover the infrastructure cost in direct efficiency alone.
The value case that does not appear in that arithmetic: dangerous improvements prevented. The notification-synchronous change that passes all 20 tests would have caused an ordering outage on the first notification service incident. ADR-002 and the Operation Scope eval are worth preventing one production incident, which typically costs far more than 10 infrastructure sessions.
Q3: Under what conditions is this approach NOT worth it?
If the project is a prototype with a defined end date, a one-off tool, or a codebase that will run fewer than 15 agent sessions, most of this infrastructure is overhead that a good CLAUDE.md and a test suite can approximate at lower cost.
The minimum viable investment for a short-lived project: Gherkin scenarios, WireMock stubs, one Pact consumer test per external service, CI pipeline. Everything else is optional until you feel the pain of not having it.
Q4: What was the most valuable investment?
The dangerous improvement experiment in Issue #16. The concurrent inventory-and-payment implementation was committed, the tests ran, three passed and two failed. The failure was caught before merge. The experiment cost 40 minutes. The failure mode it demonstrates — charging customers for out-of-stock orders because payment and inventory run concurrently — would cost far more in production. This is the most direct evidence that the ADR infrastructure prevents real mistakes rather than hypothetical ones.
Q5: What was the most avoidable overhead?
Building v1.1 before running the review. Publish v2.0 (or nothing) from the start. The review framework from Issue #12 would have caught the v1.1 failure modes at creation rather than requiring three sessions of stress-testing and remediation. The lesson generalizes: build the review process before publishing the first skill, not after stress-testing reveals the failure modes.
Q6: What is the irreducible human contribution?
Five things could not have been generated by the agent from the existing infrastructure.
The decision to stop implementing features and invest in infrastructure. After Issue #8, the project had a working API, a full test suite, and a CI pipeline. A pure-output mindset would have continued adding endpoints. The decision to spend Issues #9–#18 on skills, constraints, and failure mode documentation was a strategic bet on future sessions. Agents optimize for the task at hand. Humans set the investment horizon.
The original Gherkin scenarios. Every scenario was written by the human author and reflects product decisions — which failure modes matter, what "partial availability" means, how payment timeouts should behave — that agents cannot infer from code alone.
The four failure modes in Issue #14. The agent could document the failure modes once told to look for them. The choice to look — and the framing of the problem as "what does an agent not know that it doesn't know?" — is not a question the agent would generate unprompted.
The fire-and-forget notification design. This was a deliberate architectural choice made before any agent was involved. The agent implemented it correctly because it was specified. The specification was the human's.
The meaning of the work. This retrospective is only possible because twenty sessions of structured documentation exist. But the decision to document in this format, to track overhead categories, and to ask "was it worth it?" required editorial judgment about what matters to the reader. The agent can execute. The human decides what the work means.
The honest ROI verdict
Three ways to measure it.
On a pure-efficiency basis — minutes of implementation work per total session minute — the ROI is negative through Issue #19. The infrastructure investment has not been recovered in session efficiency alone.
On a risk-adjusted basis — dangerous improvements prevented, invariants documented before they're violated, contracts formalized before they drift — the ROI is positive by Issue #17. The dangerous improvement demonstration in Issue #16, the notification-synchronous prevention in Issue #17, and the payment contract enforcement that Pact provides are each worth the infrastructure that enables them.
On a knowledge-transfer basis — a future agent starting Issue #21 inherits a project where every decision is documented, every invariant is named, and every eval is in place — the ROI is strongly positive and compounding. The infrastructure converts session-held context into durable, queryable, machine-readable artifacts. The next session starts better than this one.
Which measure you use depends on whether you think your future sessions will encounter the failure modes the infrastructure prevents.
The seeded moments — were they right?
Three sessions were deliberately seeded with honest J-curve observations before this issue.
Issue #6: the YAML took 20 minutes, the session took 90. This was the first explicit acknowledgment that infrastructure investment costs real time. It was right — CI + Spec (Issues #6–#8) had a 29% debug ratio, similar to the Foundation phase, before the infrastructure started returning value.
Issue #10: skill maintenance cost observation. This was right but incomplete. The skill maintenance cost was real (three sessions to get from v1.1 to v2.0) but the maintenance cost per skill declined sharply once the review framework existed. The seeded observation described the cost; it did not predict the decline.
Issue #13: "more in memory than expected." This was the most accurate seed. The Layer 2 audit found 17 items, only 4 of which were already properly converted to skills. The implicit knowledge problem was larger than expected at Issue #13's writing, and it remained the most persistent gap through Issue #19 (the stub URL convention was still an implicit decision).
The question the J-curve asks
The J-curve is not a graph of whether this approach works. It is a graph of what this approach costs and when the cost is paid.
The engineer reading this has to decide: is the risk I'm managing with this infrastructure real for my project? Am I building something that will run for 20+ implementation sessions? Do the failure modes the infrastructure prevents — contract drift, invariant violations, dangerous improvements that pass tests — exist in my production environment?
If the answers are yes, the infrastructure is worth the canyon.
If the project is a one-off tool or a prototype with a defined end date, the answers may be no. In that case, a good CLAUDE.md, a Gherkin test suite, and one Pact consumer test per external service will give you most of the protection at a fraction of the cost.
The J-curve's most honest output is not the framework or the time audit. It is the specific question: what failure modes does your project need to prevent, and are they the kind that tests can catch?
Next issue: The Human Role at Level 4 — what it actually feels like day to day, the flow state question, and what Level 5 means for the engineer who builds the system rather than writes the code.
Sources & Further Reading
- Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
- Nate B. Jones — natebjones.com
- Building the AI Dark Factory - start here with Issue #1
- Project repository
- J-curve conditions framework
- Session findings — Issue #20
This article was written with the assistance of AI tools.
Top comments (0)