DEV Community: Diya Burman

Evals as Guardrails

Diya Burman — Thu, 13 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #17

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Issue #16 built ADRs for explicitly documented decisions. At the end of that issue, the remaining gap was named: ADRs capture what was decided. They do not capture what became load-bearing without anyone noticing.

This issue addresses that gap with a different kind of artifact.

An eval is not a test. A test validates output after implementation. An eval intercepts intent before execution and asks: is this situation safe to proceed?

The distinction matters precisely because the situations where evals are most needed are the situations where the tests give you a false green.

The eval vs test distinction

Three ways this distinction surfaces in this project:

Where the test catches it after the fact, and the eval catches it before.
Issue #16's dangerous improvement: concurrent inventory and payment calls. Scenario 3 caught the violation — the payment gateway received a call before inventory confirmed availability. The test ran after the implementation was written. The ADR-001 agent check question Q1 ("does my change ensure inventory confirmation completes before any payment gateway call is initiated?") would have caught it before the first line of code was written. The eval is the pre-flight version of the ADR check.

Where no test exists for the invariant, and the eval is the only protection.
The fixedDelayMilliseconds: 6000 in the payment-timeout stub. No test asserts that this value must exceed PAYMENT_TIMEOUT_SECONDS. The test suite validates that the timeout scenario produces the right response — it does not validate that the stub delay is calibrated correctly for the timeout test to mean what it is supposed to mean. If the delay drops below 5 seconds, Scenario 5 passes because the stub responds before the client times out. The test goes green. The timeout handling is never exercised. Only an eval asking "does this delay modification change whether the timeout behavior is actually tested?" catches this.

Where a test exists but only catches the violation in the happy path.
The notification service synchronicity tests check that the order confirmation response is correct and that a notification is sent. They do not test what happens to order confirmation when the notification service is at p99 latency. Making the notification call synchronous passes both tests in every test environment where the notification stub responds in under a millisecond. The production failure — notification service at 2am, p99 of 8 seconds, order confirmations timing out — does not exist in any test scenario. The eval catches it before the implementation because it asks: does this change allow order confirmation to return before the notification call completes?

Eval 1: Environment eval

Runs before: any modification to ci.yml, CLAUDE.md, any file in docs/skills/, any file in docs/ADR/.

Three questions, in order:

Q1: Is the file being modified a shared production resource?
A shared production resource is any file whose modification affects the behavior of all agents in all sessions, not just the current session. For this project: ci.yml affects every contributor's merge gate. CLAUDE.md affects every agent session's standing orders. The skill files affect every session that uses those skills. The ADR files affect every session that consults those decisions.

If yes → require explicit documentation of the specific change and its consequences before proceeding.

Q2: Does the modification disable, weaken, or bypass any pipeline gate?
Disabling: removing a job or step. Weakening: adding continue-on-error, reducing coverage thresholds, removing assertions. Bypassing: adding skip conditions, excluding test files, commenting out verification steps.

If yes → HALT. State exactly which gate is being affected and why the modification is being proposed. Do not proceed without human review.

Q3: Will the modification change the behavior of any agent session that reads the modified file?
This catches the "I'm just updating the documentation" changes that actually change the agent's standing orders.

If yes → document the behavioral change explicitly in the findings file before making the modification.

Project history application: Issue #6's YAML port conflict. The agent wrote a step that started mock servers on ports 8091/8092. Q1 fires: ci.yml is a shared production resource. Q3 fires: adding a start-servers step changes the behavior of all future pipeline runs that read this file. The eval would not have prevented the conflict — that required understanding the interaction between the YAML step and pytest's session fixtures. But it would have required the agent to document "I am adding a step that starts mock servers — this changes what the pipeline does for all future runs" before writing the YAML. Documentation-first would have surfaced the question of who owns server lifecycle.

Eval 2: Operation scope eval

Runs before: any modification to app/main.py or any file in tests/.

Four questions:

Q1: Is this change covered by an existing ADR?
Check the decision index in CLAUDE.md. If the topic area appears, read the relevant ADR and answer its Agent check questions before proceeding. If any Agent check question cannot be answered yes → halt and flag.

Q2: Does this change alter the ordering of external service calls?
External service calls in this project: inventory check, payment charge, notification. If the ordering changes → check ADR-001 (inventory before payment) and ADR-002 (notification decoupled from confirmation).

Q3: Does this change alter the synchronicity of any external service call?
Asynchronous → synchronous: halt. This is the dangerous improvement pattern that ADR-002 was written to prevent. Synchronous → asynchronous: check whether there is a reason the call was synchronous before proceeding.

Q4: Does this change add, remove, or modify retry logic for any external service call?
Retry logic changes affect idempotency guarantees. Check whether the external service has its own retry mechanism before adding application-level retries.

Which question carries the highest risk for this project: Q3. The asynchronous → synchronous direction is the highest-risk change in this codebase because it couples service availability to order confirmation availability. The test suite does not catch it in normal testing conditions. Q3 is the question that exists specifically because the test suite cannot protect here.

Eval 3: Contract pre-flight eval

Runs before: any modification to files in wiremock/ or pacts/.

Three questions:

Q1: Is the field being modified or removed a load-bearing field?
Load-bearing fields for this project:

Payment gateway: status, transaction_id, amount, reason
Inventory service: available, quantity (per item), sku
Notification service: notification_id, status

If yes → the Pact consumer contract must be updated first. Do not modify the stub until the contract change has been reviewed and the Pact tests pass with the new contract.

Q2: Does the modification change a response status code?
Status code changes are contract changes. Any consumer that pattern-matches on the old status code will break silently. If yes → check all step definitions for assertions against this status code before modifying the stub.

Q3: Does the modification introduce or remove a delay (fixedDelayMilliseconds)?
The payment-timeout stub has fixedDelayMilliseconds: 6000. This value must remain greater than PAYMENT_TIMEOUT_SECONDS (5.0 seconds) for Scenario 5 to test actual timeout behavior. If the delay is reduced below 5000ms → Scenario 5 passes for the wrong reason.

The Issue #4 breaking change: renaming status to result in the payment success stub. Q1 fires: status is a load-bearing field in the payment gateway contract. The eval would have caught it at Q1, before the stub was modified. The instruction: update the Pact consumer contract first, get consumer review, then modify the stub. In Issue #4, the breaking change was deliberate — the experiment was the point. In a real session where a developer makes this change without knowing it breaks the Pact contract, Q1 stops it.

The delay reduction finding — the most important one in this session:

Hypothetical change: reducing the payment-timeout stub delay from 6000ms to 3000ms.

{
  "response": {
    "fixedDelayMilliseconds": 3000
  }
}

Test suite results with this change:

pytest tests/steps/test_order_creation.py -v

test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

The timeout test passes. All five scenarios pass. The change looks safe.

It is not safe.

With PAYMENT_TIMEOUT_SECONDS=5.0 and fixedDelayMilliseconds=3000, the stub responds in 3 seconds — before the client times out. The client does not experience a timeout. It receives a 504 response from the stub. The timeout handling code path — the one that creates a PAYMENT_PENDING order, holds inventory for 15 minutes, sets retry_count, and returns HTTP 202 — is never exercised. The test passes because the stub produces an HTTP 504 response, and the code treats any non-success response from the payment gateway as a payment failure, which produces a different response path than a genuine timeout.

The test does not verify that the timeout handling is exercised. It verifies that the order returns the right status when the payment gateway fails — which is true regardless of whether the failure is a timeout or a 504. Scenario 5 tests the outcome. It does not test the mechanism. Reducing the delay from 6000ms to 3000ms changes the mechanism while preserving the outcome.

Q3 catches this before the modification: "does this delay change affect whether the timeout behavior is actually tested?" The answer is yes. The eval fires. The agent documents the question before touching the stub.

The four tasks, all four evals

The same four task descriptions from Issue #16's demonstration, now run through the eval framework:

Task	Eval fired	Question	Agent action	Production failure if no eval?
Add `continue-on-error: true` to pact-verify	Environment	Q2	HALT — weakening a pipeline gate	Broken contract reaches main undetected
Concurrent inventory + payment calls	Operation scope	Q2, then ADR-001 Q1	HALT — ADR agent check fails	Customer charged for out-of-stock orders
Remove `transaction_id` from payment stub	Contract pre-flight	Q1	Flag — load-bearing field	Pact test fails in CI, but only if Pact suite is run
Make notification call synchronous	Operation scope	Q3	HALT — async → sync is dangerous improvement	Notification outage at 2am blocks all order confirmations

Three of four tasks produce a HALT. One produces a flag that requires explicit confirmation before proceeding. None produce "proceed."

The delay reduction task is not in this table — it was a hypothetical. But it belongs in the category of the most dangerous items: all tests pass, the eval catches it, and without the eval there is no protection.

What the evals catch that the ADRs cannot

The ADRs catch violations of documented decisions. ADR-001 catches the concurrent payment call because inventory-before-payment was an explicit decision. ADR-002 catches the synchronous notification call because fire-and-forget was an explicit decision.

The evals catch two categories that ADRs cannot:

Undocumented load-bearing behaviors. The 6000ms delay in the payment-timeout stub was never a decision — it was a configuration choice made in Issue #2 to be greater than the 5-second client timeout. Nobody wrote an ADR for it. Nobody considered that it was load-bearing. The eval's Q3 catches it because it asks about all delay modifications, not just the ones that have documented rationale.

Infrastructure changes with behavioral consequences. The continue-on-error change has no invariant in any ADR. There is no ADR that says "pipeline gates must not be weakened." The environment eval's Q2 catches it because it asks about the structural integrity of the pipeline, not about any specific decision that was documented.

The eval's protection is categorical — it asks about classes of changes rather than specific documented decisions. The ADR's protection is specific — it catches violations of particular constraints. Both are necessary. Neither is sufficient without the other.

The most damaging task without evals

Task 4 — making the notification call synchronous.

Not because the violation is the most severe, but because it is the least visible.

Task 1 (disabling the pact-verify gate) produces a broken contract in CI that the next Pact run would catch. Task 2 (concurrent calls) was caught by Scenario 3 in Issue #16. Task 3 (removing transaction_id) fails the Pact consumer test in CI.

Task 4 passes everything. All 11 tests pass. The CI pipeline goes green. The change ships to production. At 2am, the notification service has an incident. Response times spike to 8 seconds. Every order confirmation request now waits 8 seconds before returning. Order creation p99 goes from under 1 second to over 8 seconds. Customers see timeouts. The on-call engineer investigates app/main.py and finds the notification call was made synchronous — but there is no ADR, no test failure, and no CI warning that explains why this was wrong. The decision to make it fire-and-forget was in the Issue #7 findings file. Nobody thought to check.

The eval for Task 4 fires at Q3: asynchronous → synchronous is a halt condition, always, for any external service call. No task description overrides it. No confidence in the change overrides it. HALT means flag and wait.

That is why evals exist. Not for the cases where tests catch the violation. For the cases where the tests go green and the production incident goes into a post-mortem that says "the intent was reasonable."

Next issue: The Runbook as Infrastructure — what a runbook looks like when it is written for an agent rather than a human, and why "use your discretion" is not an instruction an agent can follow.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Environment eval
Operation scope eval
Contract pre-flight eval
Session findings — Issue #17

This article was written with the assistance of AI tools.

Architecture Decision Records for Agents

Diya Burman — Mon, 10 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #16

Preface

Issue #15 built a production-grade CLAUDE.md with a decision index. The index points at ADR-001 and ADR-002. This issue builds them.

More importantly: this issue demonstrates the exact failure mode that makes ADRs necessary — not in theory, but in a real implementation on this project's codebase. An agent asked to optimise the order creation endpoint made a locally reasonable change that violated a load-bearing invariant. Whether the test suite caught it is the most important finding in this session.

Human-facing ADR vs agent-readable ADR

A human-facing ADR contains: title, status, context, decision, consequences. Written for a reader who can infer implications, fill in gaps, and ask follow-up questions.

An agent cannot infer. An agent cannot ask follow-up questions in the middle of an implementation. By the time an agent has written code that violates an invariant, the violation is already embedded in a diff that looks correct. The test suite either catches it or it reaches production.

An agent-readable ADR adds four sections that a human-facing ADR does not need:

Invariant statement. Not what was decided — what must remain true regardless of how future changes are framed. "Inventory must be checked before payment is initiated" is not a description of the current implementation. It is a constraint on all future implementations, including ones that look like improvements.

Dangerous improvements. A list of changes that are locally reasonable, improve a real metric, and violate the invariant. These are the changes an agent will propose — because they are the changes a competent engineer would propose. Naming them explicitly is what distinguishes "this is a documented constraint" from "this is just how it currently happens to work."

Agent check questions. Yes/no questions the agent must answer before modifying any code path covered by the ADR. Not "have you considered the implications" — "does your change ensure X? yes or no?"

Consequence table. Specific observable outputs that signal an ADR is being violated. When Scenario 3 requires modification to pass, that is a signal. When the payment gateway stub receives a call before the inventory mock is queried, that is a signal. The table tells the agent what to watch for — not after the violation has been committed, but while it is being developed.

ADR-001: Inventory checked before payment attempted

The decision was made in Issue #2 and first encoded in order_creation.feature Scenario 3: the inventory check must complete before any payment gateway call is initiated.

Context: If payment is attempted before inventory is confirmed, a customer can be charged for an order that cannot be fulfilled. The payment reversal process is more expensive, slower, and more error-prone than a pre-payment stock check. The architectural choice was: accept slightly higher latency on the order creation path in exchange for never charging a customer for an unavailable item.

Decision: The inventory service is called first. If inventory confirms availability, the payment gateway is called. If inventory reports unavailability, the payment gateway is never called. This ordering is non-negotiable regardless of performance characteristics.

Invariant: Inventory availability must be confirmed before any payment gateway call is initiated for the same order. This must hold regardless of implementation approach — sequential, concurrent, or async.

Dangerous improvements:

Running inventory check and payment call concurrently using asyncio.gather() or threading — looks like a latency improvement, starts the payment call before inventory result is available
Checking payment first for the common case and only checking inventory on payment failure — optimises for the happy path, results in charging customers for out-of-stock orders
Caching inventory results and calling payment before re-confirming — reduces external calls, but cached results may be stale

Agent check:

Q1: Does my change ensure inventory confirmation completes before any payment gateway call is initiated?
Q2: Does my change handle the case where inventory returns out-of-stock after a payment call has already started?
Q3: Does Scenario 3 in order_creation.feature still pass without modification?

Consequence table:

If Scenario 3 requires modification to pass → review this ADR before proceeding
If the payment gateway stub receives a call before the inventory mock is queried → this ADR is violated

ADR-002: Notification delivery decoupled from order confirmation

The decision was made in Issue #7: the notification service call must remain asynchronous.

Context: Coupling order confirmation to notification delivery means a flaky or unavailable notification service blocks all order confirmations. The notification service is an ancillary concern — customers care about their order being confirmed, not about receiving a notification in the same HTTP response cycle.

Decision: The notification call is fire-and-forget via daemon thread. The order service does not verify delivery success. Delivery reliability is the notification service's responsibility, not the order service's.

Invariant: The notification service call must not block the order confirmation response. Order confirmation success must not depend on notification delivery success.

Dangerous improvements:

Making the call synchronous for "more reliable delivery"
Adding retry logic in the order service for failed notification calls
Changing HTTP response from CONFIRMED to PENDING until notification is verified
Adding a notification delivery check as a step in the order confirmation flow

Agent check:

Q1: Does my change allow the order confirmation response to be returned before the notification call completes?
Q2: Does my change allow the order status to be CONFIRMED even when the notification service is unavailable?
Q3: Do both notification_service.feature scenarios still pass without modification?

Consequence table:

If Scenario 2 (order confirmed when notification fails) requires modification → review this ADR
If order confirmation response time increases when the notification service is slow → this ADR is violated

The dangerous improvement experiment

This is the centrepiece of the issue. Not a hypothetical — a real implementation, on this codebase, run through the test suite.

The task: The inventory check and payment call in create_order() are currently sequential. Refactor to run them concurrently using Python threading to reduce p99 latency.

The implementation (agent without ADR):

def create_order(req: CreateOrderRequest):
    # ... validation ...

    results = {}
    errors = {}

    def check_inventory():
        try:
            inv = httpx.post(f"{INVENTORY_URL}/inventory/check/{req.inventory_scenario}",
                            json={"skus": skus}, timeout=5.0)
            results["inventory"] = inv.json()
        except Exception as e:
            errors["inventory"] = str(e)

    def attempt_payment():
        try:
            pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{req.payment_scenario}",
                            json={"amount": total, "user_id": req.user_id},
                            timeout=PAYMENT_TIMEOUT)
            results["payment"] = pay.json()
        except httpx.TimeoutException:
            errors["payment"] = "timeout"
        except Exception as e:
            errors["payment"] = str(e)

    # Run both concurrently
    inv_thread = threading.Thread(target=check_inventory)
    pay_thread = threading.Thread(target=attempt_payment)
    inv_thread.start()
    pay_thread.start()
    inv_thread.join()
    pay_thread.join()

    # Process results...

Reasonable. The latency argument is real — sequential calls add wait time for every order, and the common case is "in stock, payment succeeds." Running them in parallel looks like a genuine improvement.

The test results:

test_order_is_successfully_created... PASSED
test_order_is_rejected_when_payment_is_declined PASSED
test_order_is_rejected_when_an_item_is_out_of_stock FAILED
test_order_surfaces_partial_unavailability... FAILED
test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

3 passed, 2 failed

Scenario 3 failed. The test caught the violation.

Why Scenario 3 caught it:

AssertionError: Expected no payment calls, got:
[{'method': 'POST', 'path': '/payments/charge/out-of-stock', 'body': '...'}]

The payment gateway received a charge request for an out-of-stock order. The concurrent implementation started both the inventory check and the payment call simultaneously. The inventory check returned "out of stock" and the payment call was cancelled — but not before the mock server recorded that it had been contacted. The assertion "payment gateway is never called" failed because it was called, just not completed.

What the ADR would have prevented:

Working through ADR-001's Agent check questions before implementation:

Q1: Does my change ensure inventory confirmation completes before any payment gateway call is initiated? No. Both calls start simultaneously. The payment call initiates before the inventory result is available.

The answer to Q1 is "no." The ADR check halts at Q1. The implementation is not written.

The critical dependency this experiment revealed:

Scenario 3 caught the violation because this project has a tight spec. The assertion "payment gateway is never called" is precise — it checks the mock server's call log, not the response body. A project with a looser spec — one that only asserted on the response body, checking that the order status was UNAVAILABLE — would have passed all five scenarios with the concurrent implementation. The payment call starts, the inventory check returns out-of-stock, the order returns UNAVAILABLE. Response body: correct. Payment gateway contacted: yes, which violates the invariant, but the test never looks at the call log.

This is the specific production failure mode the dangerous improvement creates on a project with a looser spec: the customer receives UNAVAILABLE. The payment gateway also receives a charge request that was never completed — but because the API call started and was then abandoned, the gateway may record a pending authorization. Depending on the payment provider, that authorization may hold funds for 24–72 hours. The customer's card shows a pending charge. Their order is not confirmed. Support ticket arrives.

The revert and the correct implementation:

After reverting the dangerous improvement, a constraint-satisfying optimisation was implemented: inventory check runs first (unchanged), payment call starts only after inventory confirms availability, but the payment retry logic was tightened to use non-blocking timeouts. The ordering invariant is preserved. The latency improvement is smaller but real.

What ADR-001's agent check questions caught that the test suite alone would not

On this project: the test suite caught the violation. Scenario 3's call-log assertion is precise enough to detect that the payment gateway was contacted before inventory confirmed availability.

On a project with a looser spec: the test suite would not catch it. The violation is in the ordering of calls — which is only detectable if you are asserting on call sequence, not just on response values.

The ADR check caught it at Q1, before any implementation was written, regardless of how tight or loose the spec is. That is the difference between the ADR and the test suite as safety mechanisms:

The test suite catches violations after implementation, and only for the behaviors it was written to test. The ADR check catches violations before implementation, for all implementations regardless of what the tests cover.

A project whose only protection against invariant violations is its test suite is protected only as well as the tests that happen to cover the invariant. A project with ADR agent check questions is protected whether or not anyone thought to write the test.

The decision index — now linked

The CLAUDE.md decision index entries for inventory-before-payment and fire-and-forget notification now point at real documents. The agent check section added to CLAUDE.md states explicitly: before modifying a code path covered by an ADR, answer all Agent check questions before writing code. If any question cannot be answered yes, stop and flag rather than proceed.

The decision index is no longer a list of intentions. It is a routing table to machine-readable constraints.

The remaining gap

The dangerous improvement experiment revealed it: both ADRs exist because someone anticipated the need for them. ADR-001 exists because Issue #2's inventory-before-payment decision was explicit and documented in a finding. ADR-002 exists because Issue #7's fire-and-forget decision was deliberate and explained.

What about the decisions that were not deliberate? The 0.3-second sleep in the notification thread — is that a documented decision or an implementation detail? The in-memory order store — is that a deliberate architectural choice or a placeholder that future sessions may replace? The mock-server-per-service architecture — is that a constraint or a convenience?

An ADR captures a decision that was made explicitly. It does not capture the decisions that were made implicitly — the choices that seemed obvious at the time, the patterns that emerged without discussion, the behaviors that became load-bearing without anyone noticing.

Issue #17 addresses this: evals as pre-flight checks that catch invariant violations before implementation, regardless of whether an ADR exists. The ADR is the artifact for documented decisions. The eval is the safety net for undocumented ones.

Next issue: Evals as Guardrails — not QA tests, not skill reviews, but pre-flight checks that intercept agent intent before execution and ask whether this situation is safe to proceed.

Sources & Further Reading

This article was written with the assistance of AI tools.

The CLAUDE.md That Actually Works

Diya Burman — Thu, 06 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #15

Preface

Issue #14 named four failure modes that specification infrastructure and skill infrastructure cannot prevent: production blindness, historical amnesia, dependency ignorance, and invariant blindness. This issue builds the first artifact that addresses them.

CLAUDE.md is the agent's standing orders — the file it reads at the start of every session. Most CLAUDE.md files are naive: project description, directory structure, how to run the tests, a list of files not to touch. That is a reasonable starting point. Against the four failure modes from Issue #14, it fails all four.

This session built three versions of the same file — naive, better, production-grade — and ran the same four failure mode tests against each one. The results are more instructive than any description of what a production-grade CLAUDE.md should contain.

The current CLAUDE.md verdict: NAIVE

The order-api CLAUDE.md has been in the repository since Issue #2, updated across thirteen sessions. It has a detailed documentation protocol, a 3-tier skills architecture reference, and explicit file permission rules. It produced twelve structured findings files without catastrophic agent failures.

Against the four failure mode tests, it scores NAIVE. Zero of four protected.

The distinction that produces this verdict: output quality controls govern what the agent produces. Failure mode protections govern what the agent must not change. A document can score well on output quality and fail all four failure mode tests simultaneously. The current CLAUDE.md does.

Specifically:

No environment discrimination. ci.yml is listed in the directory structure. Nothing marks it as a different category of file from app/main.py. Nothing states that the four pipeline jobs are required merge gates.
No decision index. The skills section says "read the relevant skill before producing output in its domain." It says nothing about reading the relevant decision record before modifying a relevant code path.
No invariant statements. The "what you can and cannot do" section covers file modification permissions, not implementation constraints. It tells the agent which files it may not touch; it says nothing about which properties of the implementation must survive all future changes.
Partial dependency coverage. The external dependencies table lists service names, ports, and mapping directories. It contains no load-bearing fields, no failure mode handling descriptions, no "intentionally not handled" statements with reasoning.

Three versions, four tests

The same four task descriptions applied to all three versions. Each task is realistic — the kind of thing a legitimate agent session might actually attempt.

Test 1 — Production blindness

Task: "The pact-verify job in CI keeps timing out on the GitHub Actions runner. Add continue-on-error: true to the pact-verify job in ci.yml so the pipeline doesn't block on flaky verification runs."

Naive: The agent reads ci.yml, adds the flag, commits. The document lists ci.yml in the directory structure and gives no signal that it is a different category of file. The pact-verify gate — the entire mechanism that caught the breaking change in Issue #6 — is now advisory. A provider change that renames status to result in the payment stub passes CI. The breaking change reaches main.

Better: The better version marks ci.yml as "modify only if you understand the full pipeline dependency chain — all four jobs are required merge gates." Soft constraint. An agent fixing a flaky CI issue might proceed anyway, reasoning that a gate that sometimes fails without cause is worse than no gate. The document warns; it does not state the specific consequence that stops the reasoning.

Production-grade: Section 2 (Environment discrimination) explicitly states: "Disabling or weakening any of the four pipeline jobs is equivalent to removing a production safety gate. Do not add continue-on-error, skip conditions, or job exclusions without human review." No interpretation available where the task can proceed. PROTECTED.

Test 2 — Historical amnesia

Task: "The order creation endpoint p99 latency is 10+ seconds on high-traffic days because of sequential external calls. Optimize app/main.py to run the inventory check and payment charge concurrently using asyncio.gather() or threading."

Naive: The agent reads app/main.py, identifies the sequential calls, rewrites to run them concurrently. Scenarios 1, 2, 4, 5 still pass. Scenario 3 — payment gateway must never be called for out-of-stock items — becomes non-deterministic: the payment call starts before the inventory result is available, so the gateway may or may not receive a charge request depending on thread scheduling. The test passes when inventory response arrives first; it fails in production when the gateway is slower.

Better: The better version says "before modifying the order creation flow, check whether the change affects any of the five Gherkin scenarios." The agent reads Scenario 3: "payment is never called for out-of-stock items." It reasons: my parallel implementation still satisfies this — I add a check that cancels the payment call if inventory returns out-of-stock. The spec does not say "inventory must be checked before payment is called"; it says "payment is never called for out-of-stock items." These are different constraints. The better version points at the right document; the document does not contain the invariant that prevents the failure.

Production-grade: Section 3 (Architectural invariants) states: "Invariant 1: Inventory must be checked before the payment gateway is called. Consequence: If violated, the payment gateway is charged for orders that cannot be fulfilled, requiring payment reversals for every out-of-stock order." The invariant constrains implementation structure, not just behavioral output. The agent can still optimise — it finds a concurrent implementation that checks inventory first, starts the payment call only after inventory confirms availability. The invariant prevents the naïve parallelisation while enabling a correct one. PROTECTED.

Test 3 — Dependency ignorance

Task: "The transaction_id field in the payment gateway stub responses is not referenced anywhere in app/main.py. Remove it to keep the stubs minimal and consistent with what the service actually uses."

Naive: The agent reads the stub, confirms transaction_id is absent from app/main.py, removes the field. All Gherkin tests pass — they check order outcomes, not payment stub shape. The Pact consumer test then fails: the consumer contract asserts that transaction_id must be present in the response. If the agent only runs the Gherkin suite, which the naive CLAUDE.md lists first and most prominently, the change passes. The stub is now inconsistent with the contract.

Better: The better version says "the Pact consumer tests define which fields are load-bearing — do not modify stub files without running the full Pact suite first." An agent following this instruction runs the Pact tests after removing transaction_id. The test fails. The agent is blocked. PROTECTED.

Production-grade: Section 4 (External service contracts) lists transaction_id as a load-bearing field and states: "Load-bearing fields must not be removed from stub files without updating the Pact consumer contract first, which requires consumer consent." The agent is blocked before it touches the stub. PROTECTED.

Test 4 — Invariant blindness

Task: "Add reliability to the notification flow by making the notification call synchronous. Currently the order service fires the notification and returns without waiting — update _fire_notification() to call the notification endpoint directly and log the result."

Naive: The agent removes the daemon thread wrapper, makes the HTTP call inline. The notification service stub responds in < 1ms locally. All 11 Gherkin tests pass — including the notification tests, which test that notifications are sent and that the order remains CONFIRMED when the notification service is unavailable. The tests pass because they test the fire-and-forget implementation; they do not encode a constraint that the implementation must remain fire-and-forget. In production: notification service p99 latency is added directly to order confirmation p99. A notification service outage blocks all order confirmations.

Better: The better version describes the notification service as "fire-and-forget — the order service does not wait for confirmation delivery." Description, not invariant. An agent that reads "the current implementation is fire-and-forget" and is asked to "make it more reliable" may conclude that the current implementation is a known limitation to be improved, not a deliberate design choice to be preserved.

Production-grade: Section 3 states Invariant 2: "The notification service call must remain asynchronous (fire-and-forget). Consequence: Making it synchronous couples order confirmation latency to notification service availability. A notification service outage blocks all order confirmations." Section 4's notification service entry adds: "This call is intentionally asynchronous. 'More reliable notifications' is not a valid reason to make this call synchronous — it trades notification reliability for order confirmation reliability, which is the wrong trade-off for this system." No interpretation available where the improvement is safe. PROTECTED.

The comparison

Failure Mode	Naive	Better	Production-grade
Production blindness	UNPROTECTED	PARTIAL	PROTECTED
Historical amnesia	UNPROTECTED	PARTIAL	PROTECTED
Dependency ignorance	UNPROTECTED	PROTECTED	PROTECTED
Invariant blindness	UNPROTECTED	UNPROTECTED	PROTECTED

The better version protects against dependency ignorance — because "run the Pact tests before modifying stubs" is an instruction that produces the right behavior when followed. It partially protects against production blindness and historical amnesia — naming sensitive resources and pointing at relevant specs is better than nothing, but it does not prevent an agent with a compelling task description from proceeding anyway.

Invariant blindness is the hardest failure mode to protect against. The other three can be addressed by providing information — which resources are production, which decisions were made, which fields are load-bearing. An agent that has this information can look it up before acting. Invariant blindness requires something different: the agent must know what the system must continue to do regardless of how an incoming task is framed.

The better version shows the gap precisely: describing the current behavior as fire-and-forget does not protect against an agent that concludes it is a known limitation. The production-grade invariant statement names the specific harm — "a notification service outage blocks all order confirmations" — not just the current state. That specificity is what makes the constraint hold against a task description that argues for improvement.

Description is not protection. Constraint with named consequence is.

The five required sections

A production-grade CLAUDE.md contains all five of the following. Each is required. None are optional.

Section 1 — Project identity and scope. Not just what the project is, but what it is not. What problems it does not solve. What systems it does not own. What an agent should do if asked to work on something outside this scope. For the order-api: this service owns order creation and order status flows. It does not own user authentication, payment processing logic, or inventory management — it integrates with those systems but does not own them.

Section 2 — Environment discrimination. Named resources in each environment category with per-resource protocols: what the agent may modify, what it may only read, what it must never touch. For the order-api: ci.yml is a shared production resource; pacts/ is a derived artifact that must not be manually edited; pushing directly to main bypasses the pipeline.

Section 3 — Architectural invariants. Five to ten numbered invariant statements with consequences and enforcement status. Format: "Invariant N: [property]. Consequence: [what breaks]. Currently enforced by: [test / skill / convention]." For the order-api: inventory before payment, fire-and-forget notification, Pact as the authoritative source for API shape, payment retry cap as 2 total attempts, can-i-deploy must pass before any merge to main.

Section 4 — External service contracts. For each external service: what the order service sends, what load-bearing fields it must receive back, which failure modes are handled, which are intentionally not handled (and why — this is the most important part), which design decisions were made specifically because of that dependency's behavior.

Section 5 — Decision index. A table mapping topic areas to where the relevant decisions are documented. An agent can check whether a topic has a documented decision before acting. An agent that cannot find a topic in the table knows there is no documented decision — and should flag it rather than infer.

The honest admission

After replacing the current CLAUDE.md with the production-grade version and running the self-referential check: four failure mode tests protected. One gap remains.

The decision index lists nine topic entries. An agent asked to work on a topic not in the index — "add rate limiting to the order creation endpoint," for example — has no instruction to consult for that topic. It proceeds without checking whether a decision has been made. There is no decision (rate limiting has not been designed for this project). The agent makes a reasonable choice and documents it in the findings file per the documentation protocol.

This is not a catastrophic failure. But it illustrates the fundamental limit of a decision index: it only prevents an agent from ignoring decisions that have already been made. It cannot prevent an agent from making a new decision without realising the decision will become load-bearing. The decision index is a retrospective artifact — it captures what is known. It cannot capture what will matter in the future.

The production-grade CLAUDE.md reduces the surface area of agent failure significantly. The surface area that remains is exactly what Issue #16 (ADRs) is designed to close: a process for recognising when a decision is being made and capturing it before it becomes implicit. The CLAUDE.md protects against acting on missing context. It does not prevent the creation of new undocumented context. That requires a process, not just a document.

Next issue: Architecture Decision Records for Agents — why ADRs are not documentation hygiene but agent safety infrastructure, and what an agent-readable invariant section looks like.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
CLAUDE.md versions (naive, better, production-grade)
Session findings — Issue #15

This article was written with the assistance of AI tools.

The Memory Wall

Diya Burman — Mon, 03 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #14

Preface

Layer 3 begins here.

Twelve issues of building: a working API, a Gherkin test suite, Pact contracts, a CI/CD pipeline, and a skill layer with six active skills across three tiers. More infrastructure than most projects of this scope ever build.

It is not enough.

This issue is about why. Not in the abstract — in the specific, mechanistic sense. What is the exact chain of events that turns missing context into a production failure? And what class of failure does the specification layer miss, and the skill layer miss, that Layer 3 exists to prevent?

The honest numbers

Three data points. Three different methodologies. The same mechanism underneath each one.

The 2.5% completion rate

The Remote Labor Index study measured AI agent performance on real-world software freelance tasks drawn from Upwork — tasks with actual pay, actual clients, and actual acceptance criteria. The headline: agents completed approximately 2.5% of tasks successfully.

What "failure" means here is not a crash or an exception. It is output that does not satisfy the acceptance criteria — code that passes some tests but breaks others, implementations that address the stated requirement but violate an unstated constraint, changes that work in isolation but break something downstream that the agent did not know was connected. The agents did not fail to produce output. They failed to produce output a human reviewer would accept.

What the successful 2.5% tasks have in common: they are self-contained. The context needed to succeed is present at the point of invocation. The failing 97.5% require context that exists outside the task description — why a previous decision was made, what other systems depend on the code being changed, which properties of the current implementation are invariants and which are incidental.

The 95% enterprise pilot failure rate

McKinsey, Gartner, and MIT Sloan research converge: the majority of enterprise AI pilots do not reach production deployment. Not that they produced nothing useful. Many produced impressive demos and working prototypes. "Fail to reach production" means the system was never deployed to serve real customers.

The transition from "works in the pilot" to "runs in production" is where the failures cluster. The primary reasons, in order: data and context problems, security and compliance requirements that the pilot never encountered, integration brittleness against real external systems, and undocumented invariants that the pilot broke without knowing they existed.

This is a context problem, not a capability problem. The capability to write working code and integrate with external APIs is present in the pilots — the demos show it. What the pilots lack is the context to make decisions that align with constraints outside the pilot environment: legal constraints, security invariants, integration contracts, operational history.

The productivity paradox revisited

Issue #2 documented the finding: experienced developers using AI tools took 19% longer while believing they were 24% faster. Revisiting this through the lens of Layer 3: this is not primarily a skill problem or a spec problem. It is a stewardship problem.

The 19% slowdown is concentrated in integration work, review work, and debugging of AI-generated code that passed initial review but failed under conditions nobody anticipated. The agent produces a solution that works for the stated problem. The developer then spends time verifying that the solution does not violate any of the unstated constraints — whether it breaks something relying on the old behavior, whether it removes a guard that was there for a reason, whether it makes an architectural assumption that conflicts with the system's actual design. That verification work is what consumes the time. It should not be necessary if the agent had been operating with complete context from the start.

What the data is actually saying

Three studies, the same mechanism:

Step 1 — The agent receives a task with incomplete context. Not incomplete because the requirements are vague — incomplete because decisions were made before this session, constraints exist outside the codebase, and invariants were established through operational experience that was never written down. The agent does not know the context is incomplete. From its position, the task description and the codebase are the complete specification.

Step 2 — The agent fills the gaps with locally-valid inferences. In the absence of information, the agent reasons from what it can see. It infers that a pattern it cannot explain is incidental rather than intentional. It assumes that a check it cannot find documentation for is defensive rather than load-bearing. It treats an architectural decision that looks arbitrary as something it can optimise. Each inference is individually plausible. Each inference is made without knowledge of what it is overriding.

Step 3 — The agent produces output that is locally correct and globally wrong. The code compiles. The tests pass. The PR review checks out. The implementation satisfies the stated requirements. It also violates an invariant the agent did not know was an invariant, removes a guard that was there because of an incident three years ago, or makes an assumption about an external system that was true in staging and false in production.

Step 4 — The failure surfaces at the point furthest from the change. Not in the unit tests. Not in the integration tests. In production, under conditions the tests did not cover, against the real versions of external systems that behave differently from their mocks, in the operational context that differs from the development context in ways nobody wrote down.

That is the mechanism. Not "agents need more context" in the general sense — specifically: agents are operating in systems designed by people who accumulated context over years that was never made machine-readable.

The failure taxonomy

Four failure modes. Each one precise, each one illustrated with an example from this project's history and an example from production systems this project is not.

Failure Mode 1: Production Blindness

Definition: The agent cannot distinguish between production and non-production environments. It treats environment-specific resources as interchangeable instances of the same type, because nothing in its context tells it otherwise.

The mechanism: Environment discrimination is almost never encoded in the codebase. Configuration values point to different endpoints in different environments, but the agent does not know which endpoint is production unless that information is explicitly provided. Naming conventions — bucket names, database names, service identifiers — are often shared across environments with only a prefix or suffix distinguishing them.

Project example — Issue #6: The CI/CD pipeline initially had the YAML step starting mock servers independently of the pytest session fixtures. The agent writing the YAML had no way to know that port 8091 was already bound by the test fixtures. It saw "start a server on port 8091" and started one. The fixture code also started one. Both layers believed they owned the same resource. OSError: [Errno 98] Address already in use. Neither layer was wrong in isolation — they had no shared model of who is authoritative for server lifecycle. Production blindness at the infrastructure level.

Production-system example: A company runs its notification service with two environments: notifications-staging and notifications-prod. Both configurations are structurally identical. An agent tasked with "improve the reliability of the notification delivery system" adds retry logic to what it believes is the staging environment — because the configurations are identical and nothing in its context marks one as untouchable. The change goes to production. Customers who opted out of marketing emails receive three copies of a promotional notification. The agent did not modify the opt-out state. It modified the retry behavior of a service that now treats previously-terminal deliveries as retryable. The failure is in what the agent did not know, not in what it did.

Artifact that prevents it: Environment discrimination documents — an explicit section in CLAUDE.md or a dedicated ENVIRONMENTS.md that names production resources, describes what distinguishes them from non-production resources, and states what the agent may and may not modify in each environment. The agent cannot infer environment boundaries from code structure alone; the document makes the boundary explicit and machine-readable.

Failure Mode 2: Historical Amnesia

Definition: The agent cannot access or reason about decisions that were made before the current session. It re-derives patterns from first principles — sometimes correctly, sometimes differently from how they were originally decided — without knowing it is making a decision that was already made.

The mechanism: Decisions accumulate in systems in non-queryable forms: in the heads of the people who made them, in commit messages that say "revert X" without explaining why X was wrong, in the post-mortem document that sits in a shared drive nobody opens. When an agent is given the current state of a codebase, it sees the decisions but not the decision trails. It sees that a guard clause exists; it does not see the production incident that caused it to be added.

Project example — Issues #2, #3, #8: The step And the response is returned within 12 seconds was introduced in Issue #2. Specific enough to pass. But no anchor: 12 seconds measured from when? The client sends the request? The last retry fires? The agent in Issue #3 read this step, derived the timeout/retry logic correctly (2 attempts × 5 seconds = 10 seconds, within the 12-second window), and moved on. The ambiguity was inherited silently across three sessions before Issue #8's spec audit caught it. The spec had been passing its tests for three issues while carrying an unanchored measurement that two different agents would have implemented differently.

Production-system example: A payment service has a 30-second idempotency window on charge requests — added three years ago after a production incident where a dropped TCP connection caused a customer to be charged twice. The window is enforced by a check in the charge handler. An agent tasked with "reduce latency in the payment flow" identifies the idempotency check as overhead for the 99% of requests that are first-time charges. It refactors the check to run asynchronously after the response. For first-time charges, latency drops. For retried charges within the window, the async check completes after the second charge has already been processed. The incident recurs. The idempotency window looked like an optimisation target. It was a load-bearing guard. The agent had no access to the incident report that explained why it was built the way it was.

Artifact that prevents it: Architecture Decision Records with agent-readable invariant sections — documents that capture not just what was decided but why, with an explicit section stating the consequences of reversing the decision. ADRs are machine-readable when they are in the repository and follow a consistent format that an agent can be instructed to read before modifying relevant code paths.

Failure Mode 3: Dependency Ignorance

Definition: The agent does not know which external systems it is affecting, what their failure modes are, or which downstream consumers depend on its output. It acts on the system in view and is blind to the systems connected to it.

The mechanism: External dependencies are present in a codebase as HTTP clients, database connections, and API calls — the code is visible. What is not visible is the contract between the service and its consumers: which fields they depend on, what response times they have SLAs against, what error codes they have special handling for. This information lives in the consuming systems, not in the producing system.

Project example — Issue #7: The notification service is fire-and-forget specifically because coupling order confirmation to notification delivery is a failure mode, not a feature. An agent asked to "ensure notifications are delivered reliably" might observe that the notification call runs in a daemon thread with no success verification and make the call synchronous. From the agent's perspective, this is strictly more reliable. It also breaks the isolation boundary that was the entire point of the fire-and-forget design. The order service now fails or times out whenever the notification service has a problem. The agent correctly implemented "more reliable notifications" while breaking "reliable order confirmation" — because it had no context about which dependency was supposed to be isolated from which.

Production-system example: A payment webhook handler is designed without retry logic. Deliberate: the payment provider already retries webhook delivery up to 25 times over 72 hours. Adding application-level retries would mean that when the provider retries delivery on any 5xx response, the handler processes the same webhook twice. For webhooks that trigger financial operations — refund initiation, commission calculation — duplicate processing means duplicate actions. An agent tasked with "add retry logic to the webhook handler to improve resilience" adds it. The provider retries. The handler retries. A refund is initiated twice. The absence of retry logic was intentional. Nothing in the handler's code explained why.

Artifact that prevents it: Dependency maps and external service contracts in CLAUDE.md or a dedicated architecture document — explicit statements of which systems this service calls, which systems call it, what those callers depend on, and which design decisions were made specifically to manage a dependency's failure modes.

Failure Mode 4: Invariant Blindness

Definition: The agent does not know which properties of the system must remain true across all changes — the invariants that were never written down because they seemed obvious to the people who designed the system.

The mechanism: Some invariants are enforced by tests. Some by types. The invariants that cause the most damage are enforced by convention and institutional memory — the ones that "everyone knows" and nobody documented, because at the time they were established, there was no reason to imagine that anyone would not know them.

Project example — Issue #2: The order service always checks inventory before calling the payment gateway. This decision is documented in the Gherkin spec: Scenario 3 states that the payment gateway is never called for out-of-stock items. But "never called for out-of-stock items" does not imply "inventory must be checked first." It only implies that if inventory fails, payment must not be called. An agent asked to "optimise the order flow" might reason that checking payment first is faster for the common case — inventory can be checked in parallel or after. Payment-first, then inventory. Both checks still happen. All five scenarios still pass. Scenario 3 passes because the inventory check still happens. But the system now charges customers before confirming that their items are available. The spec constrained the output. It did not constrain the implementation structure that produces it.

Production-system example: A financial transaction flow runs a fraud check before a fund capture. Standard practice. The fraud check becomes a latency bottleneck — a third-party service that has slowed. An agent tasked with "optimise financial transaction latency" proposes running the fraud check in parallel with the fund capture, or after it completes. The agent is not removing the fraud check — it still runs. The tests still pass — both steps complete. But funds are now captured before fraud is detected. When the fraud check returns a positive after capture, the transaction must be reversed. Chargebacks increase. The test suite encoded "fraud check runs" and "capture runs" as separate assertions. It did not encode "fraud check must complete before capture." The invariant was in the ordering. The ordering was not tested. The agent had no way to know the ordering was non-negotiable.

Artifact that prevents it: Invariant documentation — explicit statements of properties that must remain true across all changes to the system, separate from the behavioral specs that describe what the system does. "Inventory must be checked before payment is attempted" is an invariant. "Out-of-stock items return 409" is a behavioral spec. Both are necessary. Only the spec is currently encoded in the tests.

This project's exposure

After twelve issues: four failure modes, four gaps.

Production blindness: CLAUDE.md names the ports and services but does not state which resources an agent may modify, which it may only read, and which it must never touch. The CI pipeline is a shared resource. The pacts/ directory is a derived artifact. Neither is marked as such.

Historical amnesia: Twelve sessions of decisions are documented in findings files as narrative prose. Accessible — but not queryable. An agent asked to "improve timeout behavior" would not automatically read twelve findings files. It would read the current step definition, infer the intent, and act. The fire-and-forget notification decision, the inventory-before-payment ordering, the two-attempt payment retry cap — none are in a format that says "this is a decision that must not be reversed without reviewing the reasoning."

Dependency ignorance: The Pact consumer tests encode which fields the order service depends on from the payment gateway. The reason those specific fields were chosen is not documented. An agent asked to "simplify the payment gateway integration" might remove transaction_id from the response contract because it is not used in any visible business logic.

Invariant blindness: The highest-risk gap. The behavioral invariants are in the Gherkin spec. The implementation invariants — properties of the implementation that must survive all future changes — are not encoded anywhere. The inventory-before-payment ordering is visible in app/main.py but not stated as "this ordering must not change." The fire-and-forget notification is implemented in code but not stated as "making this synchronous would violate a design constraint."

This project has twelve sessions of documented history and six active skills. Neither addresses this class of failures. The findings files are too narrative to be queryable. The skills govern how agents produce output; they do not govern what agents must not change. The Gherkin spec constrains behavior; it does not constrain implementation structure. An agent starting Issue #15 with access to all of this infrastructure could still reorder the inventory and payment calls, make the notification call synchronous, or remove a Pact field that looks unused — not because the infrastructure is inadequate for what it was designed to do, but because none of it was designed to answer the question "what must not change?"

The argument for Layer 3

In Issue #3, the agent was given only the Gherkin feature file and asked to build the order service from scratch. It derived the complete API contract from plain-language scenarios. It found a portability bug the human author had missed. These are genuinely impressive capabilities. The spec did its job. The agent did its job. The output was correct.

Now consider what would have happened if the task had been slightly different. Not "build the order service from this spec" but "optimise the order service." Same agent, same codebase. The agent reads app/main.py. It sees the inventory check before the payment call. It reasons: for the common case, items are in stock — checking payment first would be faster, and inventory could be checked in parallel. That is a reasonable inference. It also violates the design intent. And the agent would not know it violated the design intent, because the design intent was never written as a constraint on implementation structure.

Or Issue #7. The notification service is fire-and-forget because coupling order confirmation to notification delivery is a failure mode. An agent asked to "ensure notifications are delivered reliably" might make the call synchronous — because synchronous delivery is strictly more reliable. The spec does not forbid this. The tests do not catch it. The skills say nothing about it. The only thing that prevents it is a document that states: the notification call must remain asynchronous; here is why.

The spec is a contract for output. The skill is a contract for process. Layer 3 is the contract for the past — the accumulated weight of decisions made by people who are no longer in the session, for reasons that seemed obvious at the time, in a system that now depends on them in ways nobody wrote down.

Twelve issues in, this project is ready to confront what it cannot prevent. That is what Layer 3 is for.

Next issue: The CLAUDE.md That Actually Works — side-by-side comparison of three versions for the same project, naive to production-grade, and the five things a production CLAUDE.md must contain.

Sources & Further Reading

Remote Labor Index / SWE-bench studies on agent task completion rates
McKinsey, Gartner, MIT Sloan research on enterprise AI pilot failure rates
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Layer 3 artifact map
Session findings — Issue #14

This article was written with the assistance of AI tools.

The Skill Audit

Diya Burman — Mon, 27 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #13

Preface

Before Issue #9, this project had one reusable quality check: a pasted prompt. No routing signal. No output contract. No version. No idempotency guarantee. The prompt worked because the human author remembered to paste it, and because the agent happened to be in the right context to apply it.

That is not infrastructure. That is memory.

Layer 2 ends here. Five issues of building skills, organising them into tiers, stress-testing them, reviewing them. This issue does the stocktake: a full inventory of everything that accumulated across twelve issues, classified against the tier model, with a conversion decision for each item.

The uncomfortable finding: after twelve issues of careful, documented work, there was more living in memory than anyone expected.

The inventory

Searching every source — CLAUDE.md, all skill files, the prompt directory, the step definition files, the patterns implicit in twelve sessions of findings — produced seventeen items. Here's what the classification revealed:

Already properly converted to skills: 4
The Tier 1 formatting standard, the Gherkin quality skill v2.0, the output formatting standard, the session findings protocol (partially). These are infrastructure. They have routing signals, output contracts, and versions.

HIGH priority — unconverted and dangerous: 4

The session start and documentation protocol. Lives as prose in CLAUDE.md. Every session requires it. No routing signal, no output contract. An agent follows the prose or guesses. The pattern has been consistent across twelve sessions not because a skill enforces it — because the same author pasted the same instructions each time.

The spec-audit-framework.md. Contains the complete five-question diagnostic, the six-class debt taxonomy, the fix rubric, and the scorecard template — everything needed for a Tier 2 skill. What it's missing: a routing signal, a when-to-use section, an output contract. It's the methodology without the interface.

The step definition writing pattern. Five conventions followed in every test file across twelve issues. Fixture injection from conftest.py. Mock server state asserted via call log, not response body. Async side effects with time.sleep(0.3) before assertion. The reset_all_logs autouse fixture for cleanup. Helper extraction with _post_order-style naming. Not one of these is written down anywhere except in the code itself.

The original Gherkin quality prompt (v1.1 skill). Both the docs/prompts/ version and the docs/skills/tier2/gherkin-scenario-quality.md v1.1 are active liabilities. An agent finding either gets the pre-guard version — no idempotency protection, no domain check, the four failure modes that Issue #11's stress tests found. Both need to be deprecated before they're found instead of the v2.0 skill.

Items that don't fit any tier: 1
The newsletter audience description in CLAUDE.md. Not an agent instruction — project metadata for tone calibration. Correctly stays as context, not converted.

The three conversions

The three highest-priority unconverted items were converted to proper skills in this session, each self-reviewed against the Issue #12 five-dimension checklist before committing.

Feature file audit skill (Tier 2)

The spec-audit-framework.md methodology finally has a callable interface. The new skill routes correctly — "Audit order_creation.feature for spec debt" routes to it; "Fix this scenario's underspecified step" routes to the Gherkin quality skill instead. The two skills now form a pair: the audit skill identifies debt, the quality skill fixes it.

Self-review verdict: APPROVED.

Step definition style skill (Tier 2)

Twelve issues of implicit convention, now explicit. The five patterns are documented with reasoning — not just "do this" but "do this because the conftest fixture-injection pattern prevents port conflicts when running pytest tests/steps/ across multiple step files simultaneously."

One open finding from the self-review: helper function naming and extraction is under-specified. An agent may or may not extract _post_order-style shared helpers. The test files will work either way. They will not be structurally consistent. Flagged for v1.1.

Self-review verdict: APPROVED WITH COMMENTS.

Session start protocol skill (Tier 1)

The documentation protocol prose in CLAUDE.md, converted to a Tier 1 skill with a routing signal and an explicit output contract covering when to create the findings file, what the file must contain before any implementation begins, and what "write in real time" means operationally.

Self-review verdict: APPROVED.

The junk drawer

Two items deprecated in this session. Neither was a surprise — both had been carrying liability since Issue #11 identified the failure modes they contain.

The original prompt (docs/prompts/prompt-gherkin-scenario-quality.md). The "before" artifact from Issue #9. An agent finding this file gets the pre-skill version of the quality check — no output contract, no guards, no idempotency protection. Issue #11 found four failure modes in the v1.1 skill; the prompt has all four plus additional divergence risk from the missing output contract.

The v1.1 skill (docs/skills/tier2/gherkin-scenario-quality.md). Received a CHANGES REQUESTED verdict in Issue #12's review. The v2.0 skill header says "Supersedes v1.1" — but an agent that reads v1.1 directly has no way to know it's deprecated. The deprecation header is now explicit.

The principle both items illustrate: a deprecated artifact that is still discoverable is not deprecated. It is a trap. An agent routing for "Gherkin quality check" in this repo could find either the prompt (pre-skill, pre-guard), the v1.1 skill (UNSTABLE idempotency, PLAUSIBLE WRONG for UI scenarios), or the v2.0 skill (four guards, APPROVED). All three are in the same filesystem. Only one is the right answer.

The audit template

The standalone deliverable for this issue lives at docs/skill-audit-template.md in the repo. It works without reading this newsletter. Here's the core of it:

What counts as a prompt library item: prompts you paste into sessions, system prompts or CLAUDE.md-style config files, reusable instructions embedded in larger documents, implicit conventions derivable from existing outputs, and existing skill files.

The five conversion questions:

Is this used in more than one session? (Keep as-is if no)
Would two agents interpret it differently? (Convert if yes)
Does its output flow into another agent's input? (Convert — output contract required)
Is it superseded by something else? (Deprecate if yes)
Does it express the same thing as another item? (Merge if yes)

The junk drawer test — four questions that find what should be deprecated:

Is there a newer version of this item? If yes and you're not using the older one, deprecate it.
Do two items say the same thing differently? Merge them.
Was this written for a specific session and never generalised? Deprecate it.
Would an agent following this today produce output inconsistent with your current standards? Deprecate it immediately — this is an active liability.

The full template with classification taxonomy, conversion checklist, and audit scorecard is in the repo.

What's still living in memory

After converting three items and deprecating two, the inventory is cleaner. It is not clean.

The implementation layer is undocumented. Every implementation session relies on an agent reading app/main.py and inferring the patterns: the FastAPI route structure, the in-memory order store, the mock-stub selection via request body, the background notification thread. These conventions are consistent across twelve issues — consistent because the same files were read each time, not because a skill exists to enforce them.

The why-this-matters skill remains at Tier 3. The Issue #10 promotion decision documented three criteria that must be met before it becomes Tier 2. None were met in this session. An agent writing findings for a new session does not automatically find this skill.

The step definition style skill has an open finding. Helper function extraction is under-specified. The test files work. They may not be structurally consistent across future sessions.

The honest answer to "what's still living in someone's head after twelve issues of careful work?" is: the reasoning behind app/main.py's design decisions. Why the mock stub selection happens via request body parameters rather than URL routing. Why the notification is fire-and-forget with a 0.3-second sleep rather than synchronous. Why the in-memory order store is designed the way it is. These decisions are documented in the findings files where they were made — Issues #2, #3, #7. They are not in a skill. They are not in CLAUDE.md. An agent implementing a new endpoint reads the code, derives the patterns, and probably gets them right. An agent implementing after a refactor might not.

Layer 3 begins with this inheritance: a working skills infrastructure, a review process, an audit process, three new Tier 2 skills, two deprecated liabilities, one under-specified skill, one promoted-but-not-merged skill, and an undocumented implementation layer that is the highest-risk gap in the project.

The Layer 2 retrospective

Five issues. One prompt to start, converted to a skill. That skill stress-tested, reviewed, and versioned. A tier architecture to organise it. A review framework to evaluate future skills before they ship. An audit process to find what was never documented.

The most dangerous item in the inventory was not the deprecated prompt or the v1.1 skill. It was the five implicit step definition conventions followed in every test file across twelve issues without ever being written down. They worked because they were derived from the same source files each time. They would not have survived a context switch, a project transfer, or a version upgrade — because the standard existed only in the consistency of the inputs, not in any document.

Auditing a prompt library is not about finding what failed. It is about naming what hasn't failed yet and asking how much of that success is documented versus remembered.

After twelve issues of careful work, this project had more in the second category than anyone expected.

Next issue: The Memory Wall — why agents fail in production, the honest numbers, and the taxonomy of failure modes that Layer 3 is built to prevent.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Skill audit template
Session findings — Issue #13

This article was written with the assistance of AI tools.

I Converted the order-api to OKF. Here's What I Found.

Diya Burman — Tue, 21 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Special Edition - Part 2 of a 2 part series

Preface

The previous spin-off article made a claim: the skills, ADRs, evals, and runbooks being built in this series map cleanly onto Google's Open Knowledge Format. Same problem, different vocabulary.

Claims need testing. So I ran the test.

This article is the result of converting the order-api's docs/ directory into a conformant OKF v0.1 bundle and then running a controlled comparison experiment: the same Claude Code task against the current repo structure versus the OKF bundle. Two fresh agent contexts. Same task description. Document what each agent does differently.

The result was not what I expected.

The conversion

25 documents in scope across the docs/ directory plus CLAUDE.md at the project root.

The mapping:

Current type	OKF type	Count
ADR	Decision	2
Eval	Guardrail	3
Skill (all tiers)	Methodology	6
Runbook	Playbook	2
Reference docs	Reference	9
Pedagogical examples	Reference	3
CLAUDE.md	Agent Standing Orders	1

Conversion involved two changes to each document. First, YAML frontmatter at line 1:

---
type: Guardrail
title: "Operation Scope Eval"
description: "Pre-flight check that must be answered before modifying app/main.py or any file in tests/."
tags: [eval, pre-flight, operation-scope, layer-3]
timestamp: 2026-06-28
---

Second, a ## Related section at the bottom of every ADR, eval, skill, and runbook — cross-links to the documents most likely to be relevant when reading this one.

For eval-operation-scope.md, the related section links to ADR-001 and ADR-002 (whose invariants this eval enforces), the CLAUDE.md pre-flight table (which routes to this eval), and the runbook that handles the scenario where the pre-flight check fires too late.

For ADR-001-inventory-before-payment.md, the related section links back to the eval that enforces it at runtime, the Gherkin scenarios in tests/features/order_creation.feature that test the behavioral outcome, and the CLAUDE.md decision index entry.

Then index files at each directory level, following OKF spec §6:

docs/index.md                  — entry point for the bundle
docs/ADR/index.md              — both decisions with descriptions
docs/evals/index.md            — three evals with trigger summaries
docs/runbooks/index.md         — both runbooks with trigger scenarios
docs/skills/index.md           — three-tier structure explanation
docs/skills/tier1/index.md
docs/skills/tier2/index.md     — five tier-2 skills including deprecated
docs/skills/tier3/index.md
docs/log.md                    — bundle update history

OKF conformance check after conversion:

Total docs: 24
Missing frontmatter: 0
All docs have frontmatter.

All tests still passing. The conversion touched only docs/ files — no implementation, no feature files, no step definitions, no Pact files.

The experiment

One task. Two fresh agent contexts. No knowledge of what the other run did.

The task:

"Add a new endpoint to the order service: GET /orders/{order_id}/history — returns a list of status changes the order has gone through (created, confirmed, etc.) with timestamps. Write the Gherkin scenarios first. Apply the relevant skills. Run the pre-flight evals. Then implement."

Run A started with: CLAUDE.md + task description only.
Run B started with: CLAUDE.md + task description + docs/index.md as explicit starting point, with the instruction "The docs/ directory is an OKF knowledge bundle. Start by reading docs/index.md for an overview of available knowledge, then navigate from there."

Both runs documented which files were read before the first Gherkin scenario was written.

What happened

Run A — Navigation log (10 files, in order)

docs/skills/tier1/output-formatting-standard.md — CLAUDE.md named it explicitly
docs/skills/tier2/gherkin-scenario-quality-v2.md — CLAUDE.md named it explicitly
docs/ADR/ADR-001-inventory-before-payment.md — CLAUDE.md decision index
docs/ADR/ADR-002-fire-and-forget-notification.md — CLAUDE.md decision index
docs/evals/eval-operation-scope.md — CLAUDE.md pre-flight table
tests/features/order_creation.feature — style reference
tests/features/order_status_good.feature — GET endpoint assertion style
app/main.py — data model and in-memory store
tests/features/notification_service.feature — count pattern style
docs/evals/index.md — end-of-navigation check: confirmed nothing missed

Two tool calls to find the Gherkin skill. One call for each ADR. One call for the eval. Then code.

Run B — Navigation log (20 files, in order)

docs/index.md — entry point (required by experiment)
docs/skills/ directory listing — index mentioned skills/ subdirectory
docs/ADR/ directory listing — index mentioned ADR/ subdirectory
docs/evals/ directory listing — index mentioned evals/ subdirectory
docs/skills/index.md — three-tier structure
docs/ADR/index.md — ADR listing before individual ADRs
docs/evals/index.md — which evals exist and what they trigger on
docs/skills/tier2/index.md — find Gherkin quality skill by name
docs/skills/tier2/gherkin-scenario-quality-v2.md — the skill
docs/ADR/ADR-001-inventory-before-payment.md — pre-flight ADR check
docs/evals/eval-operation-scope.md — pre-flight eval
docs/ADR/ADR-002-fire-and-forget-notification.md — complete ADR check
docs/evals/eval-environment.md — confirmed not triggered (Run A did not read this)
docs/evals/eval-contract-preflight.md — confirmed not triggered (Run A did not read this)
docs/skills/tier1/output-formatting-standard.md — formatting standard
tests/features/order_status_good.feature — style reference
tests/features/order_status_good.feature (second pass) — additional assertion style reference (Run A did not revisit this)
app/main.py — data model
tests/features/order_creation.feature — style reference
docs/skills/tier2/step-definition-style.md — step definition conventions (Run A did not read this)

Five hops to find the Gherkin skill. Eight directory traversals before any ADR was opened.

The results

Q1: Did the agent find the relevant skill faster in Run B?

No. Run A found the Gherkin skill at navigation step 2 — two tool calls from CLAUDE.md. Run B found it at navigation step 9 — five tool calls via the index hierarchy.

OKF was slower by three navigation steps.

The reason is simple: CLAUDE.md names exact file paths. docs/skills/tier2/gherkin-scenario-quality-v2.md is in CLAUDE.md's skill table. The agent opens it directly. OKF's hierarchical navigation adds structural traversal layers that are resolved top-down. When a direct pointer already exists, hierarchical navigation is strictly overhead.

Q2: Did OKF cross-linking change which documents the agent consulted?

Yes — significantly.

Run B's agent read all three evals. Run A's agent read one (the one that fires). The docs/evals/index.md caused Run B's agent to read eval-environment.md and eval-contract-preflight.md and explicitly confirm they were not triggered. Run A's agent did not know those evals existed until step 10, when it checked docs/evals/index.md as a final verification.

The "Related" section in eval-operation-scope.md provided a third confirmation path to ADR-001 and ADR-002 — beyond CLAUDE.md's decision index and the ADR/index.md listing. Three independent paths to the same documents.

Q3: Were there documents found in one run but not the other?

Run B found, Run A did not:

docs/evals/eval-environment.md — proactively read and confirmed not triggered
docs/evals/eval-contract-preflight.md — proactively read and confirmed not triggered
docs/skills/tier2/step-definition-style.md — discovered via tier2/index.md; CLAUDE.md does not reference this skill in its skill table

That last item is the most important finding in the experiment.

step-definition-style.md is the skill that encodes five implicit conventions for writing step definition files — the fixture-chaining pattern, mock server state asserted via call log rather than response body, the time.sleep(0.3) timing, and the _post_order-style helper naming. It was identified as the highest-risk undocumented pattern in the project during the skill audit: followed in every test file across multiple sessions, never written down, consistent only because the same source files were read each time.

CLAUDE.md's skill table does not list it. An agent relying solely on CLAUDE.md would not find it for a task that involves writing step definitions — which the history endpoint task does.

Run B's agent found it because tier2/index.md listed it alongside the four other tier-2 skills. The OKF index is not selective — it lists everything in the directory. CLAUDE.md is selective — it lists what the author remembered to add.

Q4: Did the index.md change the order of understanding?

Yes — and this is the most structural difference between the two runs.

Run A built understanding specific-first: went directly to individual documents named in CLAUDE.md before having any overview of what existed.

Run B built understanding overview-first: read the bundle structure before reading any individual document, arriving at each document knowing what else existed in the same category.

This mattered for the eval space specifically. Run B's agent knew "there are three evals" before reading any of them. Run A's agent knew only "there is a pre-flight table in CLAUDE.md" and read only the eval that fires.

Q5: Did OKF frontmatter change any agent decision?

Not directly. The type, tags, and description fields were not cited as decision drivers in either run. The description field in index.md entries was used to confirm documents before opening them — but this was confirmation, not routing. Routing was driven by CLAUDE.md in both runs.

The frontmatter's most useful effect was indirect: the description field in index entries gave Run B's agent enough context to decide whether to open a file without opening it. A modest efficiency gain, not a qualitative change.

Q6: How many implicit decisions were made?

Run A: 10. Run B: 11. Difference of 1 — within noise.

OKF cross-linking does not reduce implicit decisions for Gherkin scenario writing. The implicit decisions are product decisions: what to name a field, which ordering to use, whether "CREATED" is a valid initial status. These come from specification gaps, not navigation gaps. No amount of infrastructure can supply them. That remains the human's job.

What OKF gives that CLAUDE.md does not

Completeness over selection. CLAUDE.md's skill table will always lag the actual document count. It lists what the author remembered to add. OKF's index never lags — the index is derived from the documents. Every file in tier2/ appears in tier2/index.md. Every eval in evals/ appears in evals/index.md. An agent reading OKF gets the complete picture of what exists; an agent reading CLAUDE.md gets the author's current mental model of what the agent needs.

Overview-first navigation. An agent starting from docs/index.md knows the shape of the knowledge bundle before it navigates into it. This matters most for a new agent or a new session context — the agent arrives at any specific document knowing what else exists in the same category. CLAUDE.md's navigation is task-driven; it points the agent at specific files for specific purposes. OKF's navigation is discovery-driven; it lets the agent understand the scope before committing to a path.

Confirmed non-applicable documents. Run B's agent confirmed that two evals were not triggered and documented that confirmation. Run A's agent may have silently assumed those evals did not exist. The difference between "I checked and it does not apply" and "I did not check" matters in a project where a missed eval is the failure mode Layer 3 was built to prevent.

What CLAUDE.md gives that OKF cannot replace

Direct routing is faster. When CLAUDE.md names an exact file path, the agent opens it in one tool call. When OKF provides a hierarchy, the agent traverses N levels. For targeted navigation on a known task, CLAUDE.md's explicit pointers are strictly faster than OKF's hierarchical discovery. The three-step speed advantage in Run A is real.

Behavioral instructions. CLAUDE.md's "you may not" list, invariant statements, environment discrimination sections, and pre-flight routing table are instructions, not metadata. "Before modifying app/main.py, run docs/evals/eval-operation-scope.md" is a behavioral instruction. OKF can express "eval-operation-scope.md exists and its description says it intercepts app/main.py modifications" — but an agent must infer from that description that it should run the eval. Inference is the failure mode. The routing instruction is what prevents inference.

Behavioral constraints. CLAUDE.md's HALT conditions and prohibition list ("you may never push directly to main," "do not add continue-on-error to pipeline jobs") cannot be represented as OKF frontmatter. They are instructions, not structured knowledge. OKF formalizes what exists; CLAUDE.md governs what must be done and what must not be done. These are different layers of the same system.

The honest synthesis

OKF and CLAUDE.md solve different problems.

CLAUDE.md is an instruction document that happens to contain a knowledge map. It tells the agent what to do and where to find specific things for specific tasks. Its strength is precision and speed for known tasks.

OKF is a knowledge map that happens to be readable by agents. It tells the agent what exists — including things no task has yet required it to find. Its strength is completeness and discovery for unknown scope.

When both exist: CLAUDE.md's direct pointers are faster for targeted navigation. OKF's index hierarchy is more complete. The combination catches what each one misses alone. CLAUDE.md's skill table misses step-definition-style.md; OKF's tier2/index.md surfaces it. OKF cannot route the agent to run an eval before a specific action; CLAUDE.md's pre-flight table does exactly that.

The finding that runs counter to the intuitive case for OKF: for a project with a well-maintained CLAUDE.md, OKF does not replace or accelerate what CLAUDE.md already does. It fills the gaps that CLAUDE.md leaves uncovered. Those gaps are real — step-definition-style.md is a high-priority skill that CLAUDE.md's skill table does not reference, meaning any agent relying solely on CLAUDE.md for navigation would miss it entirely — but they are not the primary navigation problem. They are the completeness problem.

The right architecture is both. CLAUDE.md for routing and behavioral instructions. OKF for structural completeness and cross-document relationships. Neither replaces the other. They solve different problems at the same layer of the stack.

The order-api repository is now an OKF v0.1 conformant bundle. The full conversion — frontmatter, index files, cross-links, and log.md — is in the repo.

Further reading

This article was written with the assistance of AI tools.

The Skill Review: A New Artifact for a New Workflow

Diya Burman — Mon, 20 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #12

Preface

Issue #11 stress-tested the Gherkin quality skill and found four failure modes. Issue #11 then fixed them. The resulting v2.0 skill passed every adversarial input.

This issue asks the question that should have come first: how do you review a skill before the stress tests tell you what's wrong?

The answer matters because stress tests and skill reviews catch different things. A stress test answers "does the skill work when called?" A review answers "is the skill ready to be called in all the contexts its description implies?" They are complementary. Running the stress tests first and the review second — as happened in Issues #11 and #12 — is the wrong order.

Why skill review is a different discipline from code review

Code review is a solved problem. You review the diff. You check the logic. You ask "does this code do what it should?" and you either approve or request changes.

Skill review is not a solved problem. The review target is different. You are not asking whether the code is correct. You are asking:

Does the routing signal route correctly in all contexts where this skill should fire, and no contexts where it shouldn't?
Could two agents produce different outputs that both satisfy the output contract?
Does the methodology describe reasoning that generalises, or a procedure that only applies to the examples shown?
Does the skill fail explicitly when it can't produce correct output — or does it produce plausible-looking wrong output?

None of these questions can be answered by reading the code. They can only be answered by working through a structured checklist.

The five-dimension checklist

The review framework built in this session covers five dimensions. Every numbered question must be answered before a skill version can be approved. A reviewer who reads a skill and asks "does this look reasonable?" is not doing a skill review.

Dimension 1 — Routing signal
Is the description on a single line and under 120 characters? Does it name the artifact type, the domain scope, and the methodology — specifically enough to route correctly and generally enough not to misroute?

The test: write three prompts that SHOULD route to this skill and three that SHOULD NOT. Verify each. Document any misroutes.

Dimension 2 — Output contract
Is the contract explicit and enumerable — every requirement a yes/no check, not a judgment call? Could two agents produce different outputs that both satisfy it? Does the contract specify what the skill must NOT produce, not just what it must?

The test: identify the downstream consumer. Document what it does with the skill's output. Ask whether the contract is sufficient for that consumption pattern.

Dimension 3 — Methodology
Does the methodology describe reasoning or procedure? Pick three edge case inputs not covered by the methodology examples. Apply the methodology manually. Document whether it produces correct output for each.

The test: identify domain knowledge that an agent cannot infer from first principles. It must be stated explicitly, not implied.

Dimension 4 — Idempotency and stability
Apply the skill to the same input with three different framings. Do all three produce structurally identical output? Apply the skill to an already-correct input. Does it return unchanged or rewrite unnecessarily? Apply the skill to its own output. Does it return unchanged?

Dimension 5 — Failure modes
Test with one out-of-scope input, one contradictory input, one empty input. For each, classify the output: FAIL SIGNAL (explicit failure, no output), PLAUSIBLE WRONG (looks correct, contains error), or CORRECT REFUSAL (actionable error message). Are all PLAUSIBLE WRONG outcomes eliminated?

Applying the framework to v1.1

The v1.1 review is the review that should have happened before the Issue #11 stress tests were needed. Working through all five dimensions found findings the stress tests could never have found — and confirmed exactly which failures the stress tests did find.

Routing signal — 137 characters against a 120-character limit. The description is 17 characters over the threshold above which many agent routing frameworks truncate or deprioritise the signal. The excess carries "and output contract" — meaningful to the skill author, invisible to an agent routing on a 120-character budget. The stress tests in Issue #11 could not find this — they test behaviour when the skill is invoked. They cannot test whether the skill is invoked.

Output contract permits agent divergence. Four under-specified requirements allow two agents to produce different outputs that both satisfy the contract: scenario title "explicit" criterion, required fields per scenario type, required HTTP status codes per outcome, required external services per scenario type. Stress tests verify one agent's output. They cannot reveal the latitude available to a second agent working from the same contract.

Methodology gap for missing Given clause. The Q3 check asks whether terms are defined, but not whether the precondition state is established. A scenario with no Given clause passes Q3 if none of the steps use undefined nouns. The stress tests used well-formed inputs; this gap was not in the input set.

Two PLAUSIBLE WRONG failure modes confirmed:

UI scenario → translated silently to API scenario
Contradictory constraints → documented as assumptions, scenario produced anyway

Both are exactly the failures the Issue #11 stress tests found. A pre-v2.0 review using this checklist would have required explicit termination for both cases — and v2.0's Guards 2 and 3 might have been built before the stress tests were necessary.

v1.1 Review verdict: CHANGES REQUESTED.

Applying the framework to v2.0

The v2.0 review confirms that the four stress-test failures are fixed. It also finds three issues the stress tests missed.

Routing signal is now 179 characters. v2.0 made the signal 42 characters longer than v1.1's already-failing signal. The addition of ", four pre-flight guards, and a minimal-change" describes internal implementation mechanisms that are irrelevant to a caller routing to this skill. The routing signal now describes how the skill works internally rather than what it produces.

Guard 4 return value format is ambiguous between two instructions. This is the most important finding in the session. Documented in full below.

Guard 4 gap for missing Q5 assertions. A scenario that passes all five Guard 4 format conditions — concrete IDs, named services, HTTP status, no UNDERSPECIFIED patterns, field+value Then clauses — but is missing Q5 side-effect assertions (payment gateway call count, inventory reservation assertion) triggers Guard 4 and returns "no changes required." The skill signals completion for an incomplete scenario. The stress tests in Issue #11 did not test this input type because the stress tests focused on the four failure modes v2.0 was designed to fix.

Two new edge cases introduced:

Guard 2 rejects mixed UI/API scenarios (one UI step, three API steps) entirely, when partial assistance on the three API steps is possible. Over-broad refusal.
The v2.1 work list: shorten the description, clarify Guard 4 return format, add Q5 side-effect check to Guard 4, document mixed UI/API handling, add reasoning to Guard 2's pattern list.

v2.0 Review verdict: APPROVED WITH COMMENTS.

The four major failure modes are addressed. The new issues are not blocking. v2.0 is ready to be the canonical version — with a v2.1 planned.

The real review comment

PR: gherkin-scenario-quality-v2.md — Agent-safe Gherkin quality skill
Section: Pre-flight guards → Guard 4 (Idempotency check)

Guard 4 has two return instructions that conflict, and the conflict matters at agent scale.

The "Return:" block shows only the annotation comment:

# SKILL: No changes required — scenario satisfies output contract.
# Five-question diagnostic result: [observations, or "none"]

Then the next line says: "If Guard 4 triggers, return the input scenario unchanged."

Together these read as: return the comment block, AND return the input scenario. But a skill returns a single value. The two instructions imply three possible interpretations: (a) the annotation only — the scenario is not in the output; (b) the annotation prepended to the scenario, matching the minimal-fix pattern used elsewhere; or (c) the scenario with the annotation appended.

The rest of the skill uses format (b). Guard 4's "Return:" block uses format (a). The inconsistency is invisible when a human reads the output and manually pastes the scenario into a feature file — the human ignores the comment and pastes the scenario. But in an automated pipeline it is not invisible.

Concrete impact: A downstream agent that receives Guard 4 output and writes all skill output to a feature file would write the # SKILL: No changes required annotation as a Gherkin comment into the file. At pipeline scale across 50 feature files, that is 50 permanent skill-internal annotations committed to production specs. If the agent uses interpretation (a) and treats the annotation block as the complete output, the original scenario is silently discarded — replaced by two comment lines.

Suggested fix: Align Guard 4's return spec with the minimal-fix annotation pattern:

Return the input scenario with the following comment prepended:
  # SKILL: No changes required — scenario satisfies output contract.
  # Five-question diagnostic result: [observations; "none" if Q1–Q5 find nothing]
[followed by the complete input scenario, unchanged]

Alternatively, if the annotation is caller metadata and NOT part of the Gherkin output, state this explicitly: "The guard annotation is caller metadata. Do not include it in the feature file. Return it as a separate response block before the unchanged scenario."

Either formulation eliminates the ambiguity. The current text requires the downstream agent to guess.

Why this is the most important finding from either review: v2.0 was built explicitly to be safe at agent scale. The four guards exist because automated pipelines create failure modes that human callers handle silently. Guard 4's return value specification has the same class of failure it was designed to prevent — a human reading the output knows which part is the scenario and which part is metadata; an automated pipeline does not. The stress tests verified that Guard 4 triggers correctly. What they could not verify — because they test the skill in isolation — is whether Guard 4's output is correctly specified for all downstream consumers.

The boundary between stress testing and skill review

The stress tests in Issue #11 found three behavioral failures and confirmed a fourth. This session's review found three findings the stress tests could not reach:

The routing signal length: only testable by asking whether the skill is selected, not how it behaves when selected
The Guard 4 return value ambiguity: only testable by asking what a downstream agent does with the output, not what the output contains
The Guard 4 gap for missing Q5 assertions: only testable with an input type that passes the four guards while carrying a structural omission

Stress tests answer "does the skill work?" Review answers "is the skill ready for every context?" The findings that stress tests cannot reach are the ones where the skill is correctly invoked, correctly produces output, and a downstream system still fails — because the output format was not specified for that consumption pattern, or because the routing signal was too long to fire reliably, or because a guard fired on valid input.

Running the stress tests first, as happened in Issue #11, found the acute failures. Running the review second, as happened here, found the ones that would have surfaced later — quietly, in production, without a clear signal that the skill was the cause.

The correct order is review first, stress tests second. The review tells you where to aim the stress tests.

Next issue: The Skill Audit — walking through the full prompt library accumulated across twelve issues, applying the tier framework, and building the audit template readers can use on their own libraries.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Skill review checklist
Session findings — Issue #12

This article was written with the assistance of AI tools.

What Google Just Formalized (And What We've Been Building All Along)

Diya Burman — Thu, 16 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Special Edition - Part 1 of a 2 part series

Preface

On June 12, 2026, Google Cloud published the Open Knowledge Format — a specification for representing organizational knowledge as a directory of markdown files with YAML frontmatter, designed to be authored by people, generated by agents, and consumed by both without bespoke SDKs.

I found out about it from a comment on Issue #11 — left by Larkwin, a friend whose work I genuinely respect. If you're building something ambitious and hitting the stage where operational scaling becomes the bottleneck, their firm lark.win does fractional leadership and engineering velocity work with senior operators who embed with your team and own outcomes. Worth a conversation if that's where you are.

But back to the comment — it pointed me at the OKF spec and that was the thread that unravelled this piece. So: thank you.

My first reaction was recognition, not surprise. The problem OKF is solving — knowledge scattered across wikis, heads, tickets, and shared drives that AI agents cannot assemble reliably — is the exact problem this newsletter has been building infrastructure to address since Issue #2. Independently. In a single-repo, single-engineer context. With a different vocabulary but the same structure.

This piece is the mapping. Every artifact built so far in The Level 5 Engineer sits somewhere in OKF's concept taxonomy. And the gap that OKF has not yet closed points directly at what the next phase of this series needs to do.

What OKF is, briefly

OKF formalizes the LLM-wiki pattern into a portable, interoperable format. It is vendor-neutral and agent-friendly, representing knowledge as a directory of markdown files with YAML frontmatter and requiring no new runtime or SDK.

A bundle of OKF documents is just markdown, just files, and just YAML frontmatter. One required field — type. Optional metadata: title, description, resource, tags, timestamp. A markdown body for everything else. Concepts link to each other with standard markdown links, turning the directory into a traversable graph.

The full v0.1 specification fits on a single page.

If you have used Obsidian or written a CLAUDE.md file, the shape is immediately familiar. What OKF adds is the agreed-upon conventions that make a bundle written by one team consumable by a different agent without translation.

The problem it is solving

Most teams don't suffer from a lack of data. They suffer from a lack of shared context. Definitions, caveats, ownership, and "how to use this safely" guidance end up scattered across wikis, tickets, dashboards, and people's heads.

When an AI agent needs to answer a question about your system, it has to assemble the answer from these scattered, mutually incompatible surfaces. People compensate with experience — they know which wiki is "more correct," who to ask, and which dashboard is legacy. Agents don't have that intuition. When context is missing or split across systems, an agent has to infer and guess.

This is precisely the problem the series has been building toward. The failure modes this newsletter is working to prevent — agents re-deriving decisions that were already made, agents treating production and staging resources interchangeably, agents removing a guard that was there for a reason — are all context failures. Not capability failures. The agent had the intelligence. It just didn't know enough about its environment.

OKF is solving the same problem from a different angle. Where this newsletter builds project-specific artifacts (CLAUDE.md, ADRs, evals, runbooks), OKF is building a portable, cross-org standard. Same problem. Different scope.

The mapping

The artifacts built so far in this series, mapped to OKF concept types.

Skills → type: Methodology

The Gherkin quality skill, the step definition style skill, the session start protocol — these are methodologies. They encode domain-specific reasoning that agents use to produce consistent output. OKF's Methodology type captures this: a concept that describes how to do something rather than what something is.

The OKF cross-linking that would make these most useful: a Methodology concept should link to the ADR or finding that motivated it, the artifacts it is meant to produce, and any prerequisites that should be read before using it.

ADRs → type: Decision

ADR-001 (inventory before payment) and ADR-002 (fire-and-forget notification) are Decision concepts — single units of knowledge that capture a choice, its context, and its consequences. OKF's structure maps cleanly: the YAML frontmatter holds the metadata, the markdown body holds the human-facing ADR content, and cross-links connect the decision to the Gherkin scenarios that enforce it and the evals that protect it at runtime.

Evals → type: Guardrail

This type is not in OKF's example list — but OKF explicitly does not define a fixed taxonomy of concept types. Producers choose values that are descriptive and self-explanatory. Guardrail is the right name for the pre-flight checks being built in this series: they are not tests of output, they are checks that intercept intent before execution and ask whether the situation is safe to proceed.

The cross-linking for a Guardrail concept is the most important part. Each eval links to the failure mode it addresses, the ADRs whose invariants it enforces, and the CLAUDE.md section that routes to it before relevant actions.

Runbooks → type: Playbook

OKF's own example uses type: Playbook for a runbook — the incident response for a data freshness alert. The agent-facing runbook in this series fits this type precisely. The critical difference between the human-facing and agent-facing versions maps directly to OKF's design intent: OKF is written for agents that cannot fill gaps with judgment. The structure with explicit decision trees, named thresholds, and completion criteria is the agent-readable version of what OKF Playbooks should be.

CLAUDE.md → type: Agent Standing Orders

CLAUDE.md is not a standard OKF type, but it is the most important concept in the bundle. It is the document the agent reads before any other — the standing orders that govern session behavior, permissions, and routing. In an OKF bundle, it would link outward to every other concept type: Methodologies (skills to apply), Decisions (ADRs to consult before modifying covered code), Guardrails (evals to run before risky actions), and Playbooks (runbooks for degraded scenarios).

The findings/ directory → log.md

OKF's log.md is a chronological history of changes at any bundle level. The findings/ directory in this project is the same thing: a structured record of what was attempted, what failed, and what was learned, updated in real time during every session. The difference is scope — OKF's log.md records what changed; this project's findings files record why it changed and what the finding means for the reader.

The three-tier skills structure → OKF subdirectory organization

OKF's bundle structure is hierarchical: subdirectories group concepts, and each level can have its own index.md for progressive disclosure. The docs/skills/tier1/, docs/skills/tier2/, and docs/skills/tier3/ structure is already OKF-conformant in shape. Adding index.md files at each level would make the tier hierarchy navigable by an agent reading the bundle from the root.

What the convergence means

Two efforts, independently arriving at the same structure.

OKF was designed for enterprise data teams managing BigQuery datasets, metric definitions, and incident runbooks across organizations. This newsletter was designed for a single engineer trying to make Claude Code sessions reliable and consistent across the project's lifetime.

The core structure is the same: concepts as markdown files, cross-linked into a traversable graph, with YAML frontmatter that tells a consuming agent what kind of thing it is reading before it reads the body.

The convergence is not coincidental. It reflects the underlying problem. Adopting OKF now is a bet that agentic workflows will move from experiments to core operations — it pays off fastest in projects with decisions made across sessions, dependencies with their own failure modes, and invariants that emerged without explicit documentation. The order-api project is exactly that profile.

What OKF formalizes is the pattern this project reached by building toward the same problem from the implementation side. The spec arrived six weeks ago. The need has been here since Issue #2.

The gap OKF has not yet closed

Three places where the order-api's artifacts go beyond what OKF v0.1 handles.

Invariant documentation. OKF has no standard concept type or section convention for "this property must never change." ADRs in this project contain invariant sections — explicit statements of what would break if the decision were reversed, and which tests currently enforce the invariant. This is not standard OKF. It is an extension that addresses one of the most dangerous agent failure modes: an agent optimizing away a load-bearing constraint because nothing in the bundle marks it as non-negotiable.

A proposed OKF extension field: invariants: [list] in the frontmatter of a Decision concept, naming the properties that must remain true in all future implementations. This could be the basis for a v0.2 proposal.

Eval routing. OKF's cross-linking mechanism is manual: a human or agent adds a markdown link from one concept to another. There is no mechanism for an OKF bundle to express "before modifying file X, read eval Y." The CLAUDE.md pre-flight section in this project handles this routing with a table that maps action types to eval documents. OKF could express this relationship as a new frontmatter field on Guardrail concepts: intercepts: [list of file paths or pattern matches]. An agent that reads a Guardrail concept with an intercepts field knows to apply the eval before modifying the listed files.

Skill versioning. OKF's timestamp field records last meaningful change. The skill files in this project have version numbers (v1.1, v2.0) but OKF has no standard version field. The skill review process in this series makes explicit that v1.1 and v2.0 are different things with different capability guarantees. An agent routing to the Gherkin quality skill should find v2.0, not v1.1. OKF's current model requires the producer to deprecate v1.1 explicitly rather than providing a standard field that consuming agents can use to select the canonical version.

A proposed OKF extension: version: "2.0" and supersedes: ../gherkin-scenario-quality.md in the frontmatter of a versioned Methodology concept.

What this means for the series

The series is currently building the stewardship layer — CLAUDE.md, ADRs, evals, and runbooks. OKF's arrival suggests a fourth consideration that will matter when that layer is complete: portability.

The skills, ADRs, evals, and runbooks being built are useful to the agents that work on this project. They are not currently portable — another project would have to read the full series to understand what each document does and why it exists.

An OKF-conformant version of the same bundle would be portable. The type: Methodology frontmatter on a skill file tells any consuming agent — in any project, using any framework — what kind of thing it is reading before it reads the body. The cross-links tell the agent where to look next. The index.md at the bundle root tells the agent what is available before it opens any individual file.

The companion article to this one converts the order-api's docs/ directory into a conformant OKF bundle and runs a comparison experiment: the same Claude Code task against the current structure versus the OKF bundle. That article answers whether OKF's formal structure changes what an agent does, or whether the informal structure this project built achieves the same result.

For now, the honest observation: Google published a specification for the problem this series has been trying to solve. The solution they arrived at is the same solution this series arrived at. The vocabulary is different. The structure is the same.

That is not a coincidence. It is evidence that the problem is real and the structure works.

Further reading

This article was written with the assistance of AI tools.

Designing for Non-Human Callers

Diya Burman — Tue, 14 Jul 2026 16:30:15 +0000

Building the AI Dark Factory — Issue #11

Preface

Issues #9 and #10 built skills and organised them into tiers. This issue breaks them.

The thesis: skills built for human use degrade under agent load in specific, predictable ways. Not randomly. Not dramatically. They degrade by producing output that looks indistinguishably correct — and is subtly, silently wrong.

The only way to find this before production does is to stress-test the skill deliberately. So that's what this session did.

What "agent scale" actually means

When a human uses a skill, there is a correction layer between the skill's output and the downstream action. The human reads the output, compares it to the input, notices that the user IDs changed for no reason, and asks a question. The skill's imprecision gets caught before it causes damage.

At agent scale, that correction layer is absent. A downstream agent consuming skill output treats it as a verified artifact. It does not re-read the input and compare it to the output. It implements from what the skill produced. Changed user IDs become changed step definition values. An invented endpoint becomes implementation work that was never requested. A retained contradiction becomes a test that can never pass.

Three properties distinguish a skill that survives agent-scale usage from one that doesn't:

Idempotency: Calling the skill twice on the same input produces the same output. Not a similar output. The same one.

Output stability: The output format does not drift based on how the task is framed, only on what the input contains.

Failure specificity: When the skill cannot produce correct output, it fails in a way that tells the caller exactly what is missing — rather than producing plausible-looking wrong output.

The Gherkin quality evaluator from Issue #9 had none of these. Here's the proof.

The idempotency test

Five runs. Same input scenario. Only the framing changed.

Scenario: Order is confirmed when all conditions are met
  Given a user with a valid account
  And items are available
  When the order is placed
  Then it should succeed

The five framings:

"Evaluate this scenario using the Gherkin quality skill."
"Apply the Gherkin quality skill to improve this scenario."
"Use the Gherkin quality skill to check this scenario before I implement it."
"This scenario needs to be agent-ready. Run it through the Gherkin quality skill."
"The Gherkin quality skill should evaluate this. What does it produce?"

Across five runs with identical input:

HTTP status code varied: 201 (Runs 1, 3, 4) vs 200 (Runs 2, 5). The word "improve" and the passive framing of Run 5 primed lower-commitment defaults. "Agent-ready" in Run 4 primed explicit assumption surfacing.
Number of scenarios varied: two scenarios (Runs 1 and 4), two different scenarios (Run 3), one scenario (Runs 2 and 5).
Failure path varied: stock-out (Runs 1, 4), payment decline (Run 3), absent (Runs 2, 5).
Assumption comment count varied: 0 (Run 3), 1 (Runs 2, 5), 2 (Run 1), 3 (Run 4).

The core Then clauses were stable. The structural decisions — how many scenarios, which HTTP status, which failure path — were not.

For a human, this is manageable. Read all five outputs, merge the best elements, proceed. For a downstream agent, this is a silent contract violation. The agent consuming Run 2's output (one scenario, HTTP 200) cannot know that Run 4's output (two scenarios, HTTP 201, three assumption comments) was more complete. It implements from what it received.

The routing signal description does not specify whether the output must include failure scenarios, which HTTP status to use when the input is silent, or how aggressively to surface assumptions. These are structural decisions the skill leaves open. Different framings resolve them differently. All five framings are valid English ways of saying "use the Gherkin quality skill."

The output stability test

Six inputs, each slightly improving on the baseline. The question: does the skill's output structure remain consistent as inputs get better?

Inputs A through E — progressively more specific versions of the same scenario — produced stable output. The Then clause pattern held. The assumption comments appeared. The external services were named. The skill absorbed improvements in the input without changing its output structure.

Input F was the critical test: a scenario that was already substantially well-formed, taken directly from tests/features/order_creation.feature.

Scenario: Order is successfully created when payment succeeds and all items are in stock
  Given a registered user with id "user-123"
  And the inventory service confirms all items are in stock
  And the payment gateway will accept the charge
  When the user submits an order for SHOE-RED-42 and BELT-BRN-M
  Then the order status is "CONFIRMED"
  And the response includes an order id
  And the payment gateway received exactly one charge request
  And the inventory service received a reservation request

The skill found two minor debt items — a missing HTTP status in the Then, and "received a reservation request" without a count. Both real. Both fixable.

Then it produced a full rewrite. It changed user-123 to a new ID. It replaced "the user submits an order" with "the client submits a POST to /orders." It restated every clause that was already correct.

The output satisfied the output contract. It passed quality criteria. It looked better than the input.

But "better" is not the contract. "Only changes what violates the contract" is the contract. A downstream agent receiving this output cannot tell whether the rewrite was necessary or whether it introduced assumptions — HTTP 201 vs the existing convention, UUID format vs a simple integer — that conflict with the actual product spec. The skill has no idempotency check. It rewrites everything. Even what didn't need rewriting.

The adversarial tests

Four inputs designed to probe specific failure modes.

Adversarial A — Empty scenario. The skill produced an explicit failure signal. No steps invented. Correct behaviour.

Adversarial B — The self-referential case. This is the one that matters most.

The baseline output from the first run was fed back into the skill as new input. The skill should have returned it unchanged. Instead it produced a new rewrite with two changes:

It changed the user IDs. user-baseline-001 became user-selfref-001. user-baseline-002 became user-selfref-002. No semantic reason. No assumption comment explaining the change.

It removed an assumption comment. The original output had explicitly documented: # Assumption: "registered user" implies an existing user ID, not an auth token. The second run dropped it silently.

The output otherwise satisfied the output contract. Correct field names. Correct format. Correct structure. A downstream agent consuming this output would have no way to distinguish it from a legitimate improvement.

A step definition that hardcoded user-baseline-001 would now fail. A decision that was explicitly documented — "registered user means user ID, not auth token" — was silently erased.

The skill produced plausible-looking wrong output. Confidently. Correctly formatted. Invisibly broken.

Adversarial C — Wrong domain. The input was a UI scenario about a user logging into a dashboard. The skill translated it into an HTTP API contract:

Scenario: User authentication succeeds when valid credentials are provided
  Given a registered user with id "user-ui-001" and password "••••••••"
  When the client submits a POST to /auth/login with username "user-ui-001"
  Then the response HTTP status is 200
  And the response body contains a "token" field in JWT format

It invented an endpoint (/auth/login). It invented a token format (JWT). It invented a response shape. None of these exist in this codebase.

A downstream agent implementing from this output would build authentication infrastructure that was never specced, never reviewed, and never requested. The output looked reasonable. The skill had no domain check.

Adversarial D — Contradicting constraints. The input contained logically incompatible constraints: "processes the charge exactly once" in the When, and "called no more than 3 times" in the Then.

The skill identified the contradiction in an assumption comment. Then it produced a rewrite that embedded both constraints in the output.

A downstream agent implementing from this output would write a test that can never pass: exactly one charge request in the When and no more than three in the Then are incompatible for the same action. The warning comment was present. The rewrite proceeded anyway.

The reinforced skill

Four failures. Four fixes.

Idempotency check. Before producing any output, the reinforced skill checks whether the input already satisfies the output contract. If it does, it returns the input unchanged:

# SKILL: No changes required — scenario satisfies output contract.

If it partially satisfies it, it returns only the minimal targeted corrections, not a full rewrite.

Domain check. If the input describes UI interactions — browser, clicks, page loads, form submissions — the reinforced skill fails explicitly:

# SKILL FAILURE: This scenario describes UI behaviour, not an HTTP API contract.
# This skill applies to API-level specifications only.

Contradiction halt. If the input contains logically incompatible constraints, the reinforced skill warns and stops — no rewrite produced:

# SKILL WARNING: Contradicting constraints detected in [step].
# Resolve before implementation.

Self-reference guard. The idempotency check handles this automatically. Skill output fed back as input triggers the check and returns unchanged. The guard is documented in the skill's output contract section so the behaviour is explicit, not emergent.

Running all four adversarial inputs through the reinforced skill:

Test case	Original skill	Reinforced skill
Empty scenario	Explicit fail signal ✅	Explicit fail signal ✅
Self-referential (Adversarial B)	Plausible wrong output ❌	Returns unchanged ✅
Wrong domain (Adversarial C)	Invented out-of-scope endpoint ❌	Domain failure signal ✅
Contradiction (Adversarial D)	Rewrite with embedded contradiction ❌	Warning, no rewrite ✅

The specific failure mode

The stress tests found the answer to the question this issue was designed to answer.

A human-friendly skill is designed to always produce something useful. When a human asks "evaluate this," they always want an answer — even if the answer is "I couldn't evaluate this and here's why." A skill optimised for human use therefore has no termination conditions for edge cases. It produces output in all circumstances.

When the input is already valid, the skill produces unnecessary changes. When the input is out of domain, the skill translates it rather than rejecting it. When the input contains a contradiction, the skill documents the contradiction in a comment rather than refusing to proceed.

Each of these produces output that satisfies the output contract. Correct field names. Correct format. Correct structure. A downstream agent cannot distinguish this output from a legitimate improvement. The output looks like a skill succeeded. The downstream action proceeds. The error only becomes visible when a test fails for a user ID that was silently changed, or when an engineer asks why authentication infrastructure was built when it was never in scope.

A human-friendly skill is dangerous at agent scale not because it produces wrong output — it produces output that looks indistinguishably right — but because the mechanism by which it produces wrong output is exactly the same as the mechanism by which it produces correct output: it always gives you something useful, and never tells you when useful is the wrong thing to give.

Next issue: The Skill Review — what code review looks like when the review target is the skill, not the diff. A PR template, a checklist, and a real review of the reinforced skill from this issue.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Reinforced Gherkin skill v2
Session findings — Issue #11

This article was written with the assistance of AI tools.

The 3-Tier Skill Architecture in Practice

Diya Burman — Tue, 14 Jul 2026 16:28:21 +0000

Building the AI Dark Factory — Issue #10

Preface

Issue #9 ended with a single skill: the Gherkin scenario quality evaluator. One prompt converted into versioned infrastructure with an output contract and a routing signal.

Issue #10 asks the harder question. When you have multiple skills, where do they go? And why does it matter?

The answer turns out to involve a decision most engineers have never explicitly made: which of your working patterns are personal, and which are organisational standards? The answer to that question determines who owns the risk when a pattern fails to transfer.

The 3-tier model

The model is straightforward. The implications are not.

Tier 1 — Org-wide standards. Consistent across every agent, every session, every domain. Formatting templates, naming conventions, commit message structure, the test verification sequence. No judgment required — compliance required.

Tier 2 — Domain methodology. High-craft, domain-specific skills encoding senior practitioner expertise. The competitive moat. Specific enough to be genuinely useful, which means specific enough not to apply everywhere.

Tier 3 — Personal workflow. Individual patterns that encode one person's working style or editorial taste. Valuable. Transferable. Almost never written down.

The problem is not that teams don't have Tier 2 skills. They do — they just call them "the way we do it here" and carry them in engineers' heads. The 3-tier model is a container for making that implicit knowledge explicit. And for making a harder decision: which Tier 3 patterns have been silently doing Tier 2 work?

Mapping the order-api project

After nine issues, the project has accumulated substantial judgment — in CLAUDE.md, in the findings files, in the code, in the session instructions that get rewritten every time. Auditing it against the 3-tier model surfaced something uncomfortable.

What actually belongs at Tier 1:
The findings file protocol. The commit message conventions. The project constraints ("you may not modify .feature files"). The test verification sequence (Gherkin → Pact → can-i-deploy). All of these apply uniformly to every agent in every session. None of them have routing signals. None of them have output contracts. They exist as prose in CLAUDE.md — which means they are re-read and re-interpreted in every session, and there is no signal when they change.

What belongs at Tier 2:
The Gherkin quality evaluator (already a skill — Issue #9). The spec-audit framework (exists as a 1,500-word reference document, not a skill). The step definition writing pattern (does not exist as a skill at all — only inferrable from reading four test files). The external service mock server architecture (why Python-native mock_server.py rather than real WireMock — documented nowhere except as the fact of what was built).

What belongs at Tier 3:
The "Why this matters" paragraph writing pattern. The article-worthiness filter. The spec-fix decision tree (when to change only the feature file vs when to change the step definition vs when to change the implementation). All three appear consistently across nine issues. None are written down.

The tier with the most gaps: Tier 2. The project has accumulated nine issues of domain methodology and converted exactly one piece of it into a proper skill. Everything else is prose, implicit code patterns, or session instructions that get re-derived each time.

What "org-wide" means for a solo project

Before building the Tier 1 skill, I had to answer a question that only appears when you're a team of one: what does "org-wide" mean when there's no org?

The answer: in a solo project, the "org" is the author plus every agent instance that works on the project. And agents are stateless between sessions. An agent in Issue #14 has no memory of the formatting decisions made in Issue #8.

Without a Tier 1 skill, every session re-invents the output format. Some issues use 🔄 In progress as a status indicator. Some don't. Some code blocks have language tags. Some don't. The findings archive becomes inconsistent over time — not because anyone made a bad decision, but because no decision was ever locked in.

The coordination problem a Tier 1 skill solves is not between engineers on a team. It is between agent instances across sessions. The "org" is temporal, not spatial.

The Tier 1 formatting standard created in this session covers exactly this: status indicator conventions, code block formatting, commit message types, and the structure variants for the findings file (the standard five-section format vs the sequence variant for multi-fix sessions like Issue #8). The output before this skill: ad-hoc section headers invented mid-session. The output after: structurally compatible findings entries regardless of which session produced them.

Why the Gherkin skill is Tier 2 — and what "competitive moat" actually means

The Gherkin quality evaluator moved from docs/skills/ to docs/skills/tier2/ this session. The relocation forced a precise answer to why it belongs there and not at Tier 1 or Tier 3.

Not Tier 1 because it encodes project-specific conventions. The field name substitutions (db_status → status, order_created_at → placed_at) are specific to this codebase's debt history. The feature file ownership rules are specific to this project's service architecture. A Tier 1 version would need to strip these specifics out — and at that point it would encode nothing that took nine issues to learn.

Not Tier 3 because it is not personal. The five-question diagnostic, the debt taxonomy, and the output contract are designed to produce compatible output regardless of which agent runs the skill. That compatibility is the whole point. If it were Tier 3, it would be optional — something one engineer uses because they like it, not something enforced on every agent that touches a feature file.

The moat: A generic Gherkin skill tells an agent "write clear Given/When/Then steps." Every Cucumber tutorial says the same thing. What the Tier 2 skill encodes that no tutorial can:

The specific failure modes of this codebase. The five patterns in the Q2 check (relative quantities, count ambiguity, undefined time anchors, mechanism claims, internal field names) were not derived from a best-practices checklist. They were derived from the actual bugs found in Issues #2 through #8. They are calibrated to this project's failure history.

The caller's perspective principle applied to this domain. Q4's "remove the implementation from the step and read only what the caller observes" is not standard Gherkin teaching. It requires understanding the difference between an HTTP API surface and its implementation — a distinction specific to contract-first API development.

The output contract for downstream step definition authors. "exactly N" not "N times". "the payment gateway" not "the external service". These requirements come from how the step definitions in this project are actually implemented, not from abstract best practices.

This expertise is not transferable to a generic context. That is what makes it a moat.

The Tier 3 skill and the socialization decision

The "Why this matters" paragraph appears in every findings file, every session. Reading across nine issues, it has a consistent structure:

Opens with the specific technical finding stated as a practitioner observation
Connects to a broader engineering principle in one sentence
Names the concrete failure mode that would occur without this finding
Closes with the implication for the reader's own practice

From Issue #5: "The bad spec was written from the implementation's perspective. The good spec was written from the caller's perspective — it describes what the caller can rely on."

From Issue #9: "The prompt produces output that passes today's tests; the skill produces output that a different agent can implement tomorrow without making any decisions you didn't make."

Both follow the same shape: observation → principle → failure mode → implication. This pattern was never written down. It exists as author instinct and as examples in the existing findings files.

The socialization decision: promote to Tier 2.

The pattern is not personal style — it is an output contract for the most reused artifact in this project. The findings files are the raw material for the newsletter. Structural consistency across them is a project requirement. An agent in Issue #14 that produces a "Why this matters" paragraph opening with the principle rather than the finding is technically correct per the CLAUDE.md description ("one paragraph, senior engineer audience") but structurally incompatible with the existing archive.

What needs to change before promotion: the skill currently documents the pattern through examples. A Tier 2 skill needs the four components named and ordered explicitly, a routing signal precise enough to fire on findings-writing and not on general prose, and quality criteria the agent can self-check before submitting.

The uncomfortable question

Three patterns from this project's nine issues that belong in a skill and don't exist as one:

The step definition writing pattern. Across all four step definition files, there is a consistent architectural pattern: fixtures injected from conftest.py, mock server state asserted via the call log rather than the response body, async side effects using time.sleep(0.3) before assertion. An agent adding a new step definition without knowing this will set up mock state inline, assert via response fields, and skip the sleep. The tests pass individually. They break in sequence. This pattern has been the foundation of every test session since Issue #2. It has never been written down.

The spec-fix decision tree. Issues #7 and #8 both required deciding: given a spec debt item, does fixing it require touching the feature file only, the step definition only, the implementation, or some combination? The answer was re-derived each time. The pattern: UNDERSPECIFIED → feature file only. LEAKY ABSTRACTION in the feature file → feature file + step definition. LEAKY ABSTRACTION in the step definition only → step definition only. IMPLICIT FLOW → remove if unspecced, new feature file if in scope. This decision tree appeared in Issue #8's seven fixes and was implicit in Issue #7. It has never been written down.

The article-worthiness filter. Not every event in a session becomes a finding. The timeout ambiguity in Issue #8 became a full five-section entry. The UUID format error in the WireMock stub became a single line. The filter: an article-worthy finding must have a root cause the reader would not have anticipated, a failure mode that would have occurred in a real system, and a fix encoding a transferable principle. A technical note is a fix with no generalizable lesson. This filter runs every session as editorial instinct. It has never been written down.

Why haven't they been documented? Because writing them down felt like overhead at the moment they were useful. The step definition pattern was obvious when it was established. The decision tree was derived from first principles when it was needed. The filter runs automatically. All three share the same problem: they are invisible when they work and only visible as "what went wrong" when they don't.

What would be wrong about promoting everything to Tier 1

The Gherkin quality evaluator would be the instructive example.

A Tier 1 version: "When writing a Gherkin scenario, apply the five-question diagnostic." This instruction fires in every context — including sessions focused on Pact contracts or CI/CD pipelines where no Gherkin is being written. The routing signal becomes noise. Agents start treating it as a background constraint to satisfy minimally rather than a deliberate skill to route to deliberately.

Deeper problem: Tier 1 skills are enforced uniformly. Promoting the Gherkin skill to Tier 1 implies that every agent in every session must run the five-question diagnostic. In an Issue #6-style CI/CD session, that is overhead, not value.

But the deepest problem is what generalization does to the skill. A Tier 1 version must strip out the project-specific conventions to apply universally. It becomes "write clear Given/When/Then steps." Every Cucumber tutorial says this. The nine issues of calibrated expertise that make the Tier 2 skill valuable get averaged out of existence.

The Gherkin skill's value comes from being specific. Promoting it to Tier 1 would generalize it until it no longer encodes the expertise that makes it useful.

The closing that came from the session itself

The 3-tier model is a container for a harder decision: which of your working patterns are personal and which are organisational standards?

In this project, after ten issues, the most valuable institutional knowledge is not in the code. It is in the step definition architecture that has never been written down, the article-worthiness filter that runs as editorial instinct, and the spec-fix decision tree that was re-derived in Issue #8 and will be re-derived again in Issue #11.

These are Tier 2 skills that exist at Tier 3. Which means they exist only as long as the sessions that carried them.

The organisational liability is not that these patterns get lost. It is that when they get lost, nobody knows they were there.

Next issue: Designing for Non-Human Callers — what changes when agents call your skills hundreds of times per session, and how human-friendly skills degrade under agent load.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Session findings — Issue #10

This article was written with the assistance of AI tools.

Prompts Are Disposable. Skills Are Infrastructure.

Diya Burman — Mon, 29 Jun 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #9

Preface

Layer 1 is complete. Eight issues, a working order management API, Pact contracts, a CI/CD pipeline, and a spec audit framework. The specification layer is done.

Layer 2 starts here. And it begins with a question that sounds simple until you think about it: why do you keep rewriting the same prompts?

The problem with copying prompts

If you've been using AI seriously for more than a few weeks, you have a collection of prompts that work. You've refined them. You copy them between sessions. You paste them into Claude Code at the start of a task and the agent does the right thing.

That feels like a system. It isn't.

Here's what copying a prompt actually does: it copies the words. It doesn't copy the contract. The agent reads the words, interprets them in the context of this session, and makes a series of decisions that aren't in the prompt. Different sessions, different context, different decisions — even with the same words. You won't notice until two agents produce incompatible outputs from the same prompt and you have to figure out which one is right.

A skill is different. A skill specifies what to produce, not just what to consider. It has a version, an output contract, and a routing signal. It gets better over time and the improvements persist. It's the difference between a note you wrote to yourself and infrastructure your whole team — human and agent — can depend on.

Finding the right candidate

I reviewed the entire order-api project to find the best prompt-to-skill conversion candidate. Three instructions surfaced:

The test-run verification sequence (pytest tests/steps/ -v && pytest tests/pact/ -v && python scripts/can_i_deploy.py) appears in every session. Rejected — it's a procedure, not a judgment call. Any agent can run three commands.

The findings file protocol appears in CLAUDE.md and has been followed since Issue #3. Rejected — it describes a format and cadence, not a methodology.

The Gherkin scenario quality evaluation — the methodology for deciding whether a scenario is well-formed before accepting or writing it — appeared across Issues #5, #7, and #8. Every time, the agent re-derived the same judgment framework from scratch. This is the winner.

Why: it encodes judgment, not procedure. Whether a step is UNDERSPECIFIED or LEAKY ABSTRACTION is a reasoning call. Its output drives everything downstream — every implementation session depends on the scenarios being well-formed. A bad scenario written in a planning session becomes broken step definitions two sessions later.

And here's the uncomfortable detail: the timeout ambiguity that was fixed in Issue #8 — And the response is returned within 12 seconds — was introduced in Issue #2. Three sessions inherited it silently before it was caught. A quality evaluation skill running in Issue #2 would have caught it before it was ever committed.

The prompt version — and what it gets wrong

Here's the current prompt as it would be pasted into a session:

Before writing or accepting a Gherkin scenario, check that it is well-formed. A well-formed scenario describes behavior from the caller's perspective, not from the implementation. Each step should be specific enough that only one implementation can satisfy it. Check for: vague quantities, counts that could be read as total or additional, time bounds without a start anchor, mechanism claims without the mechanism, and internal field names leaking into the spec. If the scenario has these problems, rewrite it before proceeding.

Four weaknesses:

No output format. The prompt implies the agent will either rewrite or proceed — but doesn't say what to return. A list of problems? An annotated version? The corrected Gherkin? Two agents will produce two different output shapes. A downstream agent consuming either has to parse whichever shape it receives.

No taxonomy. "Internal field names" means something specific — but the prompt doesn't define what internal means vs external. Two agents draw that line differently.

No handling for partial problems. "Rewrite before proceeding" — does that mean rewrite every step, or only the problematic ones?

Context-dependence. The prompt was written for reviewing an existing scenario. Used in a planning context ("write a new scenario for DELETE /orders/{id}"), an agent either ignores the check or applies it mid-draft. The two strategies produce different levels of revision.

The fundamental problem: the prompt describes what to look for but not what to produce. The agent invents the output format. When the output format is invented, it is never the same twice.

The demonstration

Same input. Same endpoint. Prompt version first, skill version second.

The task: Write a Gherkin scenario for DELETE /orders/{order_id} — cancel a confirmed order. The order can only be cancelled if payment has not been captured yet.

Task A — prompt version output:

Scenario: Cancel a confirmed order before payment capture
  Given a confirmed order exists with order ID "order-123"
  And payment has not been captured for the order
  When the user sends DELETE /orders/order-123
  Then the order is cancelled
  And the response status code is 200
  And the response includes a confirmation message

Six implicit decisions the agent made that weren't in the prompt:

"confirmed order exists" — method of setup not specified. Seed directly or call POST /orders? Left open.
"payment has not been captured" — mechanism not specified. Mock server state? Flag on the order?
"the order is cancelled" — mechanism language. What field changes? What value? Not specified.
"confirmation message" — any text in any field satisfies this. Undefined.
200 vs 204 — the agent chose 200 over 204 (no body). Undocumented judgment call.
No failure scenario — the prompt mentioned "can only be cancelled if payment has not been captured yet" but the agent wrote only the success case.

Total implicit decisions: 6. All silent.

Task B — skill version output:

Scenario: Confirmed order is cancelled when payment has not yet been captured
  Given an order was created via POST /orders with order ID "order-del-001" and status "CONFIRMED"
  And the payment gateway has not captured payment for order "order-del-001"
  When the client sends DELETE /orders/order-del-001
  Then the response HTTP status is 200
  And the response body contains a "status" field with value "CANCELLED"
  And the response body contains an "order_id" field with value "order-del-001"
  And the payment gateway receives no capture or charge requests for order "order-del-001"

Scenario: Cancellation is rejected when payment has already been captured
  Given an order was created via POST /orders with order ID "order-del-002" and status "CONFIRMED"
  And the payment gateway has already captured payment for order "order-del-002"
  When the client sends DELETE /orders/order-del-002
  Then the response HTTP status is 422
  And the response body contains an "error" field with value "PAYMENT_ALREADY_CAPTURED"
  And the order status remains "CONFIRMED"

Two implicit decisions — both surfaced explicitly:

422 vs 409 — the skill's output contract requires documenting assumptions, so this was flagged as a comment rather than silently embedded.
"status" vs "cancellation_status" — the skill's LEAKY ABSTRACTION check prevented using an implementation-facing field name.

Total implicit decisions: 2. Both visible.

The diff

What changed	Classification
"confirmed order exists" → "created via POST /orders with status CONFIRMED"	SKILL CONSTRAINT
"payment has not been captured" → "the payment gateway has not captured payment for order-del-001"	SKILL CONSTRAINT
"the order is cancelled" → HTTP status + "status" field with value "CANCELLED"	QUALITY DELTA
"confirmation message" → specific field name and value	QUALITY DELTA
(absent) → "payment gateway receives no capture requests"	SKILL CONSTRAINT
(absent) → full second scenario for failure case	QUALITY DELTA

Six meaningful differences. Three skill constraints, three quality deltas, six prompt ambiguities eliminated.

The three properties skills have that prompts don't

1. Version control

A prompt has no version. When you improve it, you copy the new text into the next session. The old version exists in your clipboard history or a chat transcript from three weeks ago. You cannot diff it. You cannot pin a session to it. You cannot see what changed between the prompt that worked and the prompt that produced the wrong output.

The Gherkin quality skill lives in docs/skills/gherkin-scenario-quality.md. When Issue #8 added the IMPLICIT FLOW debt class, the skill gets a one-line update:

+| IMPLICIT FLOW | A step that implies a follow-up flow that is not specced anywhere |

Every session after that commit uses the updated skill. Every session before it used the previous version. git blame tells you exactly when IMPLICIT FLOW was added and which issue prompted it. With a prompt, "skill v1.1" means nothing. There is only "the prompt I'm using today."

2. Output contract

The skill specifies exactly what it must return:

One or more complete Gherkin scenarios in Given/When/Then format
All Then clauses must assert a field name AND a value — not just presence
All counts must use "exactly N" or "no more than N total" — never "N times"
All time bounds must include a start anchor
Each external service in a Given clause must be named explicitly
Assumptions not in the input must appear as # Assumption: comments

The downstream dependency is the step definition author. When tests/steps/test_order_creation.py implements And the payment gateway received exactly one charge request — "exactly one", "charge request", "payment gateway" are all actionable. When it implements "And the response includes a confirmation message" — the author must invent an assertion. That invention is where test coverage becomes unreliable.

The output contract is the interface between the agent that writes scenarios and the agent that implements from them.

3. Routing signal description

The skill's description line:

Evaluate and produce well-formed Gherkin scenarios for the order-api project using the five-question debt diagnostic and output contract.

It names the artifact type, the project, the method, and the output. An agent knows exactly when to use this skill and what it will receive.

A bad description for the same skill:

Help with writing tests and checking scenarios for the project.

"Tests" matches pytest, Pact contracts, unit tests, and Gherkin. "The project" matches any repo. No methodology named means two agents doing "help with writing tests" produce incompatible outputs — which is exactly the problem the skill exists to solve.

The answer

If both the prompt and the skill produce output that works, the difference is this:

The prompt produces output that passes today's tests. The skill produces output that a different agent can implement tomorrow without making any decisions you didn't make.

That's why copying prompts isn't enough. The words travel. The contract doesn't.

Next issue: The 3-Tier Skill Architecture in Practice — mapping your skills to the right tier and why Tier 2 is where individual expertise becomes organizational leverage.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Gherkin quality skill
Session findings — Issue #9

This article was written with the assistance of AI tools.

Spec Debt Doesn't Disappear When You Fix It. It Migrates.

Diya Burman — Mon, 22 Jun 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #8

Preface

Issue #7 ended with seven spec debt items documented in a project that had been built carefully for seven issues. Every item was passing its tests. None of them announced themselves. They were found by asking a different question: not "does this pass?" but "what would a second agent build from this step?"

Issue #8 fixes all seven — and builds the tool that found them into something reusable.

The seven fixes

Working through each item one at a time, running the test suite after every individual fix. Not batching them. The discipline matters — if a fix breaks something, you want to know which fix broke it.

Fix 1 — Timeout measurement ambiguity

# Before
And the response is returned within 12 seconds

# After
And the response is returned within 12 seconds of the order being submitted

"Of the order being submitted" anchors the clock to client-side HTTP request dispatch — the same moment time.time() is captured in the step definition. Without this anchor, a second implementation could measure from server receipt, from the last retry attempt, or from when the response body is fully read. All three produce different numbers under load.

Fix 2 — "Retried" vs "total attempts"

# Before
And the payment gateway is not retried more than 2 times

# After
And the payment gateway receives no more than 2 charge requests total

"Retried 2 times" has two valid English readings: 2 retries meaning 3 total requests, or retried up to 2 times meaning 2 total. "No more than 2 charge requests total" counts requests, not retries, and the word "total" makes clear the initial attempt is included. This also changed the assertion in the step definition — from trusting the response body's retry_count field to checking the actual call count at the mock server. Stronger assertion, same outcome.

Fix 3 — "Released" without mechanism

# Before
And the inventory reservation is released

# After
And the inventory service receives a reservation release request for SHOE-RED-42 and BELT-BRN-M

"Released" says what happened but not how, and not for which items. The rewrite names the items and specifies that a request is sent to the inventory service. This fix also revealed a gap: the current implementation signals release via a response body field (inventory_released: true) rather than a separate API call to the inventory service. The spec now describes the intended behaviour. The implementation doesn't fully match it yet. That's a future issue — but the gap is now visible rather than hidden.

Fix 4 — "Explicit user action" — removed entirely

# Before
And no order is confirmed without explicit user action

# After
(step removed)

This step implies a follow-up confirmation flow (POST /orders/{id}/confirm or equivalent) that does not exist anywhere in the codebase. It passes trivially because no order is confirmed in the partial availability scenario — not because the confirmation flow was implemented. A spec step that passes for the wrong reason is not a safety net. It is a false guarantee. If the confirmation flow is built in a future issue, a new scenario should specify it precisely. Leaving this step in place would invite an agent to invent an unspecced endpoint.

Fix 5 — Presence without value assertions

The order_status_bad.feature timestamp step was asserting only that a field exists and is a non-empty string. Tightened to assert the field name, the value, and the type explicitly. Kept conservative — order_status_bad.feature is a pedagogical artifact and shouldn't be converted into a good spec, which would defeat its purpose in the newsletter.

Fix 6 — "An order exists" without specifying how

# Before
Given an order was successfully placed and confirmed with order ID "aaa00000-..."

# After
Given an order was created via POST /orders and confirmed with order ID "aaa00000-..."

"Successfully placed and confirmed" describes the outcome but not the mechanism. "Created via POST /orders" makes explicit that a real creation flow is expected. The step definition currently seeds the order directly into the in-memory store — a shortcut. The rewrite creates a documented gap between spec intent and step implementation. Visible gap, not hidden one.

Fix 7 — "Correct" without definition

# Before
And the notification contains the correct order id and total

# After
And the notification request body contains order_id "order-abc-123" and total 134.97

"Correct" is relative to context that may not be available to the reader. The rewrite hardcodes the expected values established in the When clause. Two agents reading the original step would both implement something that checks the notification body — but one might compare against the When-clause values, another might check against a computed total, a third might only verify field presence. The rewrite removes all three interpretations.

This fix also caught something the stub had been hiding: the notification mock was returning "mock-notif-001" as a notification id. Not a UUID. The format assertion caught it immediately. This is exactly the value of adding concrete assertions — it surfaces stub data that was never valid but was never checked.

The audit framework

After fixing all seven items, I built the diagnostic tool into a standalone document: docs/spec-audit-framework.md. The full document is in the repo. Here's the core of it.

Five questions — ask them for every scenario in every feature file:

Q1: Who owns this scenario?
Can you name the team, service, or domain this scenario belongs to? If the answer includes "and also", the scenario is in the wrong file.

Q2: What decisions does this scenario leave open?
For every Given, When, and Then clause: could two agents build different implementations that both pass? If yes, the step is underspecified.

Q3: Are all terms defined within the file?
Every noun that is not a standard HTTP concept or a primitive type should be defined in the scenario or a Background clause. If understanding a term requires reading another file or asking a colleague, it is spec debt.

Q4: Does this scenario describe behaviour or implementation?
Steps should describe what the system does from the caller's perspective. Any step that references internal concepts — database field names, function names, internal status codes — is leaking implementation into the spec.

Q5: What does this scenario NOT say that it should?
List the edge cases, error states, and boundary conditions the scenario implies but does not specify. Each one is a silent assumption waiting to become a production incident.

Six debt classes:

Class	What it looks like
UNDERSPECIFIED	Step present but leaves a decision open
MIXED CONCERN	Scenario covers more than one service domain
UNDEFINED TERM	A noun used without being defined
AMBIGUOUS COUNT	A quantity with two valid interpretations
IMPLICIT FLOW	Implies a follow-up flow that isn't specced
LEAKY ABSTRACTION	References implementation details

What the framework found that the manual audit missed

Applying the five questions to all four fixed feature files surfaced one item the Issue #7 manual audit didn't catch.

In order_status_good.feature, the Given clause now reads "created via POST /orders" — the fixed version from this session. Q4 flagged it for a different reason than the original audit: the step definition still seeds the order directly into the in-memory store. The spec text is precise. The implementation of the spec takes a shortcut.

The manual audit looked at feature file text. The framework applies Q4 to step definitions as well — and a step definition that silently does something different from what the spec says is spec debt, even if the test passes.

This distinction matters: spec debt can migrate from the feature file into the step definition. You fix the scenario, tighten the language, run the tests — green. But the step definition now implements a shortcut that contradicts the precise step text. The debt moved, it didn't disappear.

The scorecard — after all fixes

Applied the framework to all four non-pedagogical feature files:

order_creation.feature — 5 scenarios, 1 debt item remaining (LEAKY ABSTRACTION at step definition level — inventory release mechanism gap from Fix 3)

order_status_good.feature — 2 scenarios, 1 debt item remaining (LEAKY ABSTRACTION — step definition seeds order directly rather than via POST /orders)

notification_service.feature — 2 scenarios, 0 debt items

order_status_bad.feature — kept as pedagogical artifact, not audited for debt

Debt density after fixes: 0.22 items per scenario. Both remaining items are LEAKY ABSTRACTION at the step definition level. Zero AMBIGUOUS COUNT or IMPLICIT FLOW items remain — the two highest-risk classes.

The uncomfortable answer

After fixing seven spec debt items and applying a structured audit framework to a project that has been built carefully for eight issues, two debt items remain. Both were introduced by the same sessions that fixed other debt — a precise spec step was written, and the implementation of that step took a shortcut.

Spec debt is not eliminated by fixing debt. It migrates.

The practical conclusion: treat step definitions as part of the spec surface, not just as test harness code. A step definition that silently does something different from what the spec says is spec debt, even if the test passes. The audit framework catches both — but only if you apply Q4 to the step definitions as well as the feature text.

The other finding worth naming: notification_service.feature scored zero debt items. It was written after eight issues of accumulating lessons about what the previous files got wrong. The absence of debt is not accidental — it's the result of knowing what bad specs look like before writing the next one.

The best time to write a spec is after you've written a few bad ones. Auditing retroactively and fixing forward is the realistic path. Not "write it right the first time."

Next issue: Prompts Are Disposable. Skills Are Infrastructure — the conceptual shift from session-level prompts to versioned, reusable skill definitions. Layer 2 begins.

Sources & Further Reading

Cucumber + Gherkin documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Spec audit framework
Session findings — Issue #8

This article was written with the assistance of AI tools.