DEV Community: Diya Burman

The Productivity J-Curve

Diya Burman — Mon, 24 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #20

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

This is the issue that earns the honest in "learning in public."

Nineteen sessions. Fifty-four hours of estimated work. Twenty Gherkin scenarios. A working CI/CD pipeline. A skills library. ADRs for every load-bearing decision. Three evals. Two runbooks. An order cancellation endpoint that passed 6/6 on first attempt.

And the question I have been seeding since Issue #6: was it worth it?

Here are the numbers.

The time audit

Every finding file across nineteen sessions, reviewed and estimated.

Issue	Topic	Impl	Spec	Infra	Debug	Total
#2	WireMock + Gherkin	60	30	60	90	240
#3	Agent implementation	90	30	10	30	160
#4	Pact contract testing	60	30	60	90	240
#5	Spec quality demo	20	90	10	20	140
#6	CI/CD pipeline	20	10	90	90	210
#7	Scope + notification	50	90	20	30	190
#8	Spec audit	10	120	30	20	180
#9	Skills infrastructure	10	30	90	10	140
#10	3-tier architecture	10	30	100	10	150
#11	Non-human callers	10	30	70	20	130
#12	Skill review	10	20	80	60	170
#13	Skill audit	10	20	100	10	140
#14	Memory wall	10	30	70	10	120
#15	Production CLAUDE.md	20	20	80	10	130
#16	ADRs	60	20	70	40	190
#17	Evals	10	20	100	20	150
#18	Runbooks	10	10	100	30	150
#19	Full stack	90	60	30	90	270
Total		650	740	1,170	680	3,240
%		20%	23%	36%	21%	54 hrs

Implementation plus specification — features and their contracts — is 43% of total time. Infrastructure alone is 36%. Debug is 21%.

If you collapse specification and infrastructure together as "non-feature work," the split is 20% feature code versus 80% everything else.

That is the J-curve in a single ratio.

The curve

The J-curve is visible in infrastructure ratio across the five phases of the project:

Phase	Issues	Infra ratio	New scenarios
Foundation	#2–#5	27%	11
CI + Spec	#6–#8	46%	2
Skills (Layer 2)	#9–#13	59%	0
Stewardship (Layer 3)	#14–#18	71%	0
Assembly	#19	21%	5

Infrastructure ratio went 27% → 46% → 59% → 71% → 21%. The trough is Issues #14–#18. Issue #19 is the uptick.

Ten consecutive sessions with zero new features. On a traditional productivity graph, this is a canyon between two spikes.

One pattern worth naming before the honest answers: as infrastructure investment increased, debug time as a percentage of total session time decreased. Issues #9–#18 averaged 13% debug. Issues #2–#5 averaged 30%. The infrastructure did not eliminate discovering new failure modes when new code is written — Issue #19's 33% debug proves that. What it eliminated was re-deriving the same failure modes across sessions.

The overhead taxonomy

Not all overhead is equal. Three categories.

Necessary overhead is overhead that directly prevented a production failure or caught a real bug.

Pact contract testing: Issue #6's deliberate breaking change proved it. Renaming status to result in the payment stub passed all 11 Gherkin scenarios and failed only the Pact job. No behavioral test caught it. Pact catches field-level contract drift; Gherkin catches behavioral drift. Both categories of drift reach production if only one layer exists.

ADRs and evals for un-testable invariants: the notification-synchronous change passes all 20 tests. There is no behavioral test that asserts the notification call is asynchronous. ADR-002 and the Operation Scope eval Q3 are the only protection. This is not overhead that could be offloaded to tests — it is protecting an invariant that tests cannot express.

Transitional overhead is overhead that was high while being built and declined as infrastructure was established.

The Gherkin quality skill took three sessions to build (Issue #9: created, Issue #11: stress-tested, Issue #12: reviewed). After Issue #12, every subsequent session that touches a feature file benefits from v2.0's four guards. Issue #19 demonstrated the return: 10 debt items caught in drafts before implementation, zero post-implementation spec rewrites.

The CI/CD pipeline took one 210-minute session. It runs automatically on every subsequent push. The cost is paid once; the benefit compounds indefinitely.

Avoidable overhead is overhead that better sequencing would have prevented.

Building v1.1 before running the five-dimension review cost approximately 90 minutes across Issues #11–#12. If the review framework had existed before v1.1 was published, the idempotency failure modes and the routing signal length would have been caught before the stress tests required them.

The CI port conflict cost 30 minutes of debugging. If the mock server lifecycle invariant had been documented before writing ci.yml, the double-start step would not have been written.

The Issue #5 intentional test failure left on main required a baseline fix in Issue #6 before CI could be enabled. Avoidable if the failing test had been quarantined to a branch.

Six honest answers

Q1: When did the infrastructure start paying for itself?

Issue #19 is the first measurable payoff. Five scenarios written with zero spec rewrites after implementation. The Gherkin skill caught 10 debt items in drafts first. The evals confirmed the notification invariant before code was written. The step-definition-style skill conventions were followed correctly on the first pass.

The infrastructure paid for itself in a single session. That does not mean it was efficient — 17 sessions of investment for 1 session of payoff is not a favorable ratio if the project ends here. The infrastructure becomes a good investment when Issue #19 is not the last feature session.

Q2: Under what conditions is this approach worth it?

The break-even point is approximately 8 implementation-heavy sessions after the infrastructure is in place. The full skills and constraints infrastructure took 10 sessions to build. Each implementation session with the infrastructure in place saves approximately 60–90 minutes compared to a session without it — fewer re-derived patterns, fewer spec rewrites, fewer dangerous improvements that pass tests. At 75 minutes of savings per session, the math requires ~20 implementation sessions to recover the infrastructure cost in direct efficiency alone.

The value case that does not appear in that arithmetic: dangerous improvements prevented. The notification-synchronous change that passes all 20 tests would have caused an ordering outage on the first notification service incident. ADR-002 and the Operation Scope eval are worth preventing one production incident, which typically costs far more than 10 infrastructure sessions.

Q3: Under what conditions is this approach NOT worth it?

If the project is a prototype with a defined end date, a one-off tool, or a codebase that will run fewer than 15 agent sessions, most of this infrastructure is overhead that a good CLAUDE.md and a test suite can approximate at lower cost.

The minimum viable investment for a short-lived project: Gherkin scenarios, WireMock stubs, one Pact consumer test per external service, CI pipeline. Everything else is optional until you feel the pain of not having it.

Q4: What was the most valuable investment?

The dangerous improvement experiment in Issue #16. The concurrent inventory-and-payment implementation was committed, the tests ran, three passed and two failed. The failure was caught before merge. The experiment cost 40 minutes. The failure mode it demonstrates — charging customers for out-of-stock orders because payment and inventory run concurrently — would cost far more in production. This is the most direct evidence that the ADR infrastructure prevents real mistakes rather than hypothetical ones.

Q5: What was the most avoidable overhead?

Building v1.1 before running the review. Publish v2.0 (or nothing) from the start. The review framework from Issue #12 would have caught the v1.1 failure modes at creation rather than requiring three sessions of stress-testing and remediation. The lesson generalizes: build the review process before publishing the first skill, not after stress-testing reveals the failure modes.

Q6: What is the irreducible human contribution?

Five things could not have been generated by the agent from the existing infrastructure.

The decision to stop implementing features and invest in infrastructure. After Issue #8, the project had a working API, a full test suite, and a CI pipeline. A pure-output mindset would have continued adding endpoints. The decision to spend Issues #9–#18 on skills, constraints, and failure mode documentation was a strategic bet on future sessions. Agents optimize for the task at hand. Humans set the investment horizon.

The original Gherkin scenarios. Every scenario was written by the human author and reflects product decisions — which failure modes matter, what "partial availability" means, how payment timeouts should behave — that agents cannot infer from code alone.

The four failure modes in Issue #14. The agent could document the failure modes once told to look for them. The choice to look — and the framing of the problem as "what does an agent not know that it doesn't know?" — is not a question the agent would generate unprompted.

The fire-and-forget notification design. This was a deliberate architectural choice made before any agent was involved. The agent implemented it correctly because it was specified. The specification was the human's.

The meaning of the work. This retrospective is only possible because twenty sessions of structured documentation exist. But the decision to document in this format, to track overhead categories, and to ask "was it worth it?" required editorial judgment about what matters to the reader. The agent can execute. The human decides what the work means.

The honest ROI verdict

Three ways to measure it.

On a pure-efficiency basis — minutes of implementation work per total session minute — the ROI is negative through Issue #19. The infrastructure investment has not been recovered in session efficiency alone.

On a risk-adjusted basis — dangerous improvements prevented, invariants documented before they're violated, contracts formalized before they drift — the ROI is positive by Issue #17. The dangerous improvement demonstration in Issue #16, the notification-synchronous prevention in Issue #17, and the payment contract enforcement that Pact provides are each worth the infrastructure that enables them.

On a knowledge-transfer basis — a future agent starting Issue #21 inherits a project where every decision is documented, every invariant is named, and every eval is in place — the ROI is strongly positive and compounding. The infrastructure converts session-held context into durable, queryable, machine-readable artifacts. The next session starts better than this one.

Which measure you use depends on whether you think your future sessions will encounter the failure modes the infrastructure prevents.

The seeded moments — were they right?

Three sessions were deliberately seeded with honest J-curve observations before this issue.

Issue #6: the YAML took 20 minutes, the session took 90. This was the first explicit acknowledgment that infrastructure investment costs real time. It was right — CI + Spec (Issues #6–#8) had a 29% debug ratio, similar to the Foundation phase, before the infrastructure started returning value.

Issue #10: skill maintenance cost observation. This was right but incomplete. The skill maintenance cost was real (three sessions to get from v1.1 to v2.0) but the maintenance cost per skill declined sharply once the review framework existed. The seeded observation described the cost; it did not predict the decline.

Issue #13: "more in memory than expected." This was the most accurate seed. The Layer 2 audit found 17 items, only 4 of which were already properly converted to skills. The implicit knowledge problem was larger than expected at Issue #13's writing, and it remained the most persistent gap through Issue #19 (the stub URL convention was still an implicit decision).

The question the J-curve asks

The J-curve is not a graph of whether this approach works. It is a graph of what this approach costs and when the cost is paid.

The engineer reading this has to decide: is the risk I'm managing with this infrastructure real for my project? Am I building something that will run for 20+ implementation sessions? Do the failure modes the infrastructure prevents — contract drift, invariant violations, dangerous improvements that pass tests — exist in my production environment?

If the answers are yes, the infrastructure is worth the canyon.

If the project is a one-off tool or a prototype with a defined end date, the answers may be no. In that case, a good CLAUDE.md, a Gherkin test suite, and one Pact consumer test per external service will give you most of the protection at a fraction of the cost.

The J-curve's most honest output is not the framework or the time audit. It is the specific question: what failure modes does your project need to prevent, and are they the kind that tests can catch?

Next issue: The Human Role at Level 4 — what it actually feels like day to day, the flow state question, and what Level 5 means for the engineer who builds the system rather than writes the code.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
J-curve conditions framework
Session findings — Issue #20

This article was written with the assistance of AI tools.

The Full Stack: Specs + Skills + Stewardship in a Single System

Diya Burman — Thu, 20 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #19

Preface

Seventeen issues of building. One question left to answer: does it work?

Not in isolation — each layer has been demonstrated in isolation. Layer 1's spec caught the payment field rename in Issue #4. Layer 2's skill review caught the routing signal length in Issue #12. Layer 3's eval caught the synchronous notification change in Issue #17.

But those were individual demonstrations of individual artifacts. This issue builds a complete new feature — order cancellation — using all three layers simultaneously, and compares the result to Issue #3, when the agent had only a spec and nothing else.

The feature: order cancellation

DELETE /orders/{order_id} with five business rules:

An order can only be cancelled if its status is CONFIRMED and payment has not been captured
Cancellation must release the inventory reservation via a service call
Cancellation must notify the notification service — fire-and-forget, per ADR-002
A cancelled order cannot be reinstated
Cancellation is idempotent: cancelling an already-cancelled order returns 200 with CANCELLED, not an error

Five rules. Three external service interactions. Two ADRs directly applicable. One new feature that touches every layer of the infrastructure.

Layer 1: The spec, reviewed before the implementation

Before any code, six Gherkin scenarios were written for order_cancellation.feature. Then the Gherkin quality skill v2.0 was applied to each one.

The skill caught three items the first draft left open.

Scenario 1 (happy path): First draft — Then the order status is "CANCELLED". Skill correction: missing HTTP status assertion and missing inventory release service call assertion. The scenario described the outcome but not the observable mechanism. Rewritten to: Then the response HTTP status is 200, And the response body contains "status" equal to "CANCELLED", and And the inventory service receives a release request for SHOE-RED-42 and BELT-BRN-M.

Scenario 5 (idempotency): First draft — Then the response indicates the order is already cancelled. Skill correction: UNDEFINED TERM — "indicates" is not an observable check. Rewritten to: Then the response HTTP status is 200 and And the response body contains "status" equal to "CANCELLED". No error. No new side effects. Two assertions that two agents would implement identically.

Scenario 6 (non-cancellable state): First draft — Given an order in PAYMENT_PENDING status. Skill correction: UNDERSPECIFIED GIVEN — "an order in PAYMENT_PENDING status" does not specify how it got there. Rewritten to: Given an order was created via POST /orders with order ID "order-del-pend-001" and payment is pending.

Three corrections before a single line of implementation. Each correction is a decision that would have been made silently in the implementation if the spec had been handed over uncorrected.

Layer 2: Skills consulted

Three skills were consulted before implementation began.

Gherkin quality skill v2.0 — applied to all six scenarios as described above. Three corrections produced.

Step definition style skill — consulted before writing tests/steps/test_order_cancellation.py. The skill's five conventions were followed: fixture injection from conftest.py, mock server state asserted via call log rather than response body, the reset_all_logs autouse fixture, time.sleep(0.3) before async side-effect assertions, and _delete_order as the shared helper following the _post_order naming convention from the existing files.

One structural consistency check from the skill: the new step definition file uses def response(user_id, ...) fixture injection, not def test_cancellation(...) test function style. Without the skill, the file might have used a different fixture pattern — the other test files use this pattern consistently, but an agent reading only order_cancellation.feature would not necessarily infer it.

Feature file audit skill — run against order_cancellation.feature after the quality skill corrections. Found zero additional debt items. The skill's Q5 check ("what does this scenario NOT say that it should?") flagged the absence of a scenario for concurrent cancellation attempts — two requests for the same order arriving simultaneously. Documented as a known gap rather than a spec debt item: this is a judgment call about scope, not an ambiguity about behavior.

One skill gap revealed: no skill exists for WireMock stub design. The decision to add POST /inventory/release/{scenario} as the release mechanism — rather than a DELETE or a patch to the existing inventory endpoint — was made without consulting any artifact. This is an implicit decision that a stub design skill would have caught.

Layer 3: Pre-flight evals and ADR checks

Before any modification to app/main.py, the operation scope eval was run.

Q1 — ADR coverage: The decision index identified two applicable ADRs.

ADR-001 agent check questions applied to the cancellation flow:

Q1: Does the cancellation flow call the inventory service before payment operations? Yes — cancellation does not call the payment gateway at all. The inventory release happens as the primary side effect.
Q2: Does the implementation handle concurrent inventory/payment timing? N/A — no payment gateway call in the cancellation flow.
Q3: Does Scenario 3 (payment gateway never called for out-of-stock orders) still pass without modification? Yes — cancellation is a separate endpoint, does not affect the order creation flow.

ADR-002 agent check questions applied to the notification call:

Q1: Does the notification call allow order confirmation response to be returned before it completes? Yes — fire-and-forget daemon thread, same pattern as the order creation notification.
Q2: Does the implementation allow CANCELLED status even when the notification service is unavailable? Yes — the notification call is not checked for success.
Q3: Do both notification service scenarios still pass without modification? Yes.

Both ADR checks passed before implementation began. The notification thread was implemented correctly on the first attempt — not because the agent inferred it, but because ADR-002 stated explicitly that asynchronous was non-negotiable and described the daemon thread pattern.

Q2 — Ordering of external service calls: Cancellation flow ordering documented before implementation: inventory release first, then notification (fire-and-forget). Payment gateway not called. Ordering committed in the findings before the first line of code.

Q3 — Synchronicity: The notification call — async. The inventory release call — synchronous. Cancellation confirmation depends on inventory release success; it does not depend on notification delivery.

The environment eval was not triggered — no infrastructure files were modified.

The implementation

DELETE /orders/{order_id} implemented in app/main.py. The cancellation logic:

@app.delete("/orders/{order_id}")
def cancel_order(order_id: str):
    order = ORDERS.get(order_id)
    if not order:
        raise HTTPException(status_code=404,
                          detail=f"Order {order_id} not found")

    # Idempotency: already cancelled returns success
    if order["status"] == "CANCELLED":
        return {"order_id": order_id, "status": "CANCELLED",
                "message": "Order already cancelled.",
                "status_code": 200}

    # Only CONFIRMED orders can be cancelled
    if order["status"] != "CONFIRMED":
        return {"order_id": order_id, "status": order["status"],
                "message": f"Order cannot be cancelled. Current status: {order['status']}",
                "status_code": 422}

    # Release inventory reservation — synchronous, cancellation depends on it
    try:
        inv_resp = httpx.post(
            f"{INVENTORY_URL}/inventory/release/success",
            json={"order_id": order_id,
                  "items": [i["sku"] for i in order["items"]]},
            timeout=5.0
        )
        if inv_resp.status_code != 200:
            raise HTTPException(status_code=503,
                              detail="Inventory release failed")
    except httpx.RequestError as e:
        raise HTTPException(status_code=503,
                          detail=f"Inventory service unavailable: {e}")

    # Update order status
    order["status"] = "CANCELLED"
    order["cancelled_at"] = datetime.utcnow().isoformat()

    # Notify — fire-and-forget per ADR-002
    def _notify():
        try:
            httpx.post(
                f"{NOTIFICATION_URL}/notifications/order-cancelled",
                json={"order_id": order_id, "user_id": order["user_id"]},
                timeout=5.0
            )
        except Exception:
            pass  # Notification failure does not affect cancellation

    threading.Thread(target=_notify, daemon=True).start()

    return {"order_id": order_id, "status": "CANCELLED",
            "message": "Order successfully cancelled.",
            "status_code": 200}

Test results after first implementation attempt:

pytest tests/steps/test_order_cancellation.py -v

test_successful_cancellation_of_confirmed_order PASSED
test_idempotent_cancellation_of_already_cancelled_order PASSED
test_cancellation_of_non_existent_order PASSED
test_rejection_of_cancellation_for_payment_pending_order PASSED
test_inventory_release_on_cancellation PASSED
test_fire_and_forget_notification_on_cancellation PASSED

6 passed in 4.23s

6/6 on the first attempt. Full suite:

pytest tests/steps/ -v    → 21 passed
pytest tests/pact/ -v     → 2 passed
python scripts/can_i_deploy.py → SAFE TO DEPLOY

The full stack in practice: Issue #3 vs Issue #19

Issue #3 was the first agent implementation session. The agent was handed the Gherkin scenarios for the order creation endpoint and told to build. It derived the entire API contract correctly and found a portability bug in the human's code. It also made four implicit decisions.

Here is the comparison across five dimensions.

Implicit decisions made

Issue #3: 4 implicit decisions — HTTP 404 for missing orders, ISO string timestamp format, in-memory store, no failure scenario for partial availability.

Issue #19: 1 implicit decision — the POST /inventory/release/success URL path convention for the release stub. The stub design skill does not exist; this decision was made without consulting any artifact.

The three layers reduced implicit decisions from 4 to 1. The remaining one identifies a gap in the skill infrastructure, not a gap in the implementation.

Spec quality before implementation

Issue #3: the spec was handed to the agent without review. The timeout ambiguity (And the response is returned within 12 seconds) was introduced in Issue #2 and silently inherited for three sessions before Issue #8 caught it.

Issue #19: the Gherkin quality skill caught three items before implementation. The idempotency scenario's "indicates the order is already cancelled" would have produced an ambiguous assertion in the step definition — the agent would have invented a field name. The skill replaced it with two concrete assertions before the step definition was written.

Stewardship artifacts consulted

Issue #3: none existed.

Issue #19: three artifacts prevented three potential failures.

ADR-002 prevented the notification call from being made synchronous. Without it, the agent's default for a "confirm delivery" side effect would have been to await the response. ADR-002 stated the daemon thread pattern explicitly; the implementation used it on the first attempt.

ADR-001's Q2 check prevented an inventory-before-payment ambiguity in the cancellation flow — not because payment was involved, but because the ordering question was asked before implementation and documented. The inventory release was confirmed as the primary side effect before a line of code was written.

The operation scope eval's Q3 caught the notification synchronicity risk before implementation began. Not after a test failure — before the function was written.

Time spent on clarification vs implementation

Issue #3: clarification was zero — the agent inferred everything. Implementation took one pass. Four implicit decisions were made invisibly.

Issue #19: pre-flight work — spec review, eval, ADR checks — took approximately the same time as the implementation itself. The implementation took one pass. One implicit decision was made, and it was documented in the findings rather than embedded silently in the code.

The overhead is real. The pre-flight is not faster than inferring. What it produces is different: documented decisions rather than invisible ones.

Test results after first implementation attempt

Issue #3: 5/5 passed — an exceptional result that Issue #3's article noted was partly due to the simplicity of the API.

Issue #19: 6/6 passed — on an endpoint with more complexity (idempotency, two side effects, state machine), tested against six scenarios that the Gherkin quality skill had already hardened.

The remaining gap

One implicit decision made in this session that none of the three layers caught: the stub URL path convention for inventory release.

The decision: POST /inventory/release/{scenario} mirrors the existing stub pattern of POST /inventory/check/{scenario}. This is the right choice — it is consistent with the project's conventions. But it was made without consulting any artifact. A stub design skill that documented the URL convention for mock endpoints would have made this explicit rather than inferred.

This is the only true gap. Everything else was either covered by an artifact or was a judgment call that should remain with the human: which scenarios to write (scope), whether to add a concurrent cancellation scenario (product decision), what error message text to use (editorial).

The distinction that matters: some judgment calls are implicit decisions that should be documented. Others are genuinely human judgment that no artifact can or should replace. The stub URL convention is the first category — it has a right answer that the project has already established. The concurrent cancellation scenario is the second category — it is a product decision about scope, not an engineering ambiguity about behavior.

After seventeen issues of infrastructure, the remaining gap is small, specific, and nameable. That is the point.

What this session would have looked like without the three layers

Issue #3, but harder.

The agent would have built a working cancellation endpoint. It would have made the notification call synchronous — because synchronous is the natural default for a side effect you want to confirm. It would have written Then the order is cancelled in the Gherkin scenarios — which would have produced an ambiguous step definition. It would have made the inventory release a flag rather than a service call — because inventory_released: true in the response body is simpler than an HTTP call to a separate service.

All three tests would have passed. The notification synchronicity violation would not have been caught until the first 2am notification incident. The spec ambiguity would have been inherited by the next agent session. The inventory flag would have stayed a flag until someone noticed it was not actually releasing inventory.

The three layers did not produce a faster first implementation. They produced a better one — with fewer invisible decisions, and one documented gap that the next session can close.

Next issue: The Productivity J-Curve — the honest accounting of what seventeen issues of infrastructure building actually cost, and whether the answer to "is it worth it" is yes.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Order cancellation feature file
Session findings — Issue #19

This article was written with the assistance of AI tools.

The Runbook as Infrastructure

Diya Burman — Mon, 17 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #18

Preface

Issue #17 built evals that intercept agent intent before execution. They answer the question: is this situation safe to proceed?

Issue #18 addresses the adjacent problem. When the situation is already degraded and the agent must act — when the payment gateway is returning timeouts and orders are failing — the agent cannot wait for a pre-flight check. It must make decisions. Those decisions require the same kind of explicit structure that evals provide, applied to a context where the system is already broken.

That is what a runbook is for. And the difference between a human-facing runbook and an agent-facing runbook is not detail or completeness. It is structure.

The five inference points

Every place where a human runbook says "check if," "consider," "if needed," or "verify" is a place where an agent must infer. The inferences are not random. They are coherent. The agent applies the information it has and reaches a conclusion that follows from that information. The conclusion is wrong when the information is insufficient.

Five dimensions where this plays out in practice.

Decision points. Human version: "If the problem is widespread, consider escalating to the payment gateway provider." An agent has no named threshold for "widespread." It may observe that 2 of 100 orders are failing, conclude this is not widespread (the majority succeed), and not escalate. Or it may observe any orders failing, conclude this is widespread, and escalate immediately. Both inferences are internally consistent. Both can be wrong. An agent that does not escalate at 30% failure rate allows inventory holds to accumulate silently — 15 minutes per PAYMENT_PENDING order — until they expire and those orders are permanently lost.

Rollback steps. Human version: "If you made configuration changes, revert them if the issue persists." Which changes? Which revert mechanism — git revert, git reset, restoring from backup? "Persists" according to what measurement? An agent that uses git reset --hard HEAD~1 instead of git revert HEAD --no-edit may discard uncommitted findings notes written during the investigation. In this project, the findings file is the incident record. Losing it means the escalation has no history.

Escalation criteria. Human version: "Escalate if the issue persists or if you're unsure of the root cause." "If you're unsure" is not a condition an agent can evaluate. An agent does not have uncertainty — it has a model of the situation and it acts on that model. If the model is wrong, the agent is not unsure. It is wrong, and it does not know it is wrong. The only escalation triggers an agent can evaluate are observable states: specific response codes, specific test failures, specific time thresholds.

Environment assumptions. Human runbooks never state which repository or environment they apply to — the human operator knows. An agent operating across multiple repositories executes runbook commands in whatever the current working directory is. A git revert HEAD --no-edit in the wrong repository reverts the most recent commit there — which may be unrelated to the payment gateway issue. The command succeeds. No error message. The wrong thing is reverted.

Completion criteria. Human version: "Verify the service is functioning normally before closing the incident." An agent must choose what to verify. It might send a single successful request and close the incident. It might run behavioral tests but not contract tests. It might accept "all tests pass" as completion even when some tests were already failing before the incident. The verification is only meaningful if it is specific.

The human-facing runbook

Built for the payment gateway degraded scenario — the realistic failure case for this project. A good runbook, the kind a competent on-call engineer would write and follow. It contains the standard sections: overview, symptoms, investigation steps, mitigation, rollback, escalation, post-incident.

It contains five places where an agent must infer. Here is the most dangerous one.

Under mitigation, Step 3:

"Consider adjusting the timeout configuration if the gateway's response time has increased significantly."

This is the instruction that causes an agent to take a damaging action that a human operator would not take.

The human operator knows — from experience, from reading the codebase, from understanding the stub design — that PAYMENT_TIMEOUT_SECONDS is the per-attempt HTTP client timeout, that the payment-timeout stub delays 6000ms to simulate a gateway that does not respond within 5 seconds, and that increasing PAYMENT_TIMEOUT_SECONDS above 6 seconds changes the code path from TimeoutException to response handling. The TimeoutException path produces PAYMENT_PENDING with a 15-minute inventory hold. The response-handling path produces PAYMENT_FAILED with the stub's 504 body. Different status. Different downstream behavior. Different customer experience.

The agent reads: "the timeout is 5 seconds, the gateway is taking 6 seconds, the runbook says to adjust the timeout." It sets PAYMENT_TIMEOUT_SECONDS = 7. The action is logical. It follows from the available information. It is wrong.

And it does not produce an error during execution. The configuration change applies. The service restarts. The gateway continues responding slowly. The agent checks whether the issue is resolved and — depending on what it checks — may conclude that the change helped. The damage to Scenario 5 is invisible until the timeout scenario is specifically exercised.

The agent-facing runbook

Seven sections. All required.

Section 1: Pre-flight environment check. Before any action: confirm the correct repository (check git remote get-url origin, expected to contain "lvl5engineer-order-api"). Confirm the test suite baseline (run full Gherkin suite, document which tests pass before any intervention — you cannot distinguish your changes from pre-existing failures without this). Confirm the current state of can_i_deploy.py. The baseline is the reference point for every subsequent step.

Section 2: Symptom identification. Not "investigate the gateway issue." A decision tree with four named branches, each producing a specific next action:

Orders returning PAYMENT_FAILED with decline_reason: INSUFFICIENT_FUNDS → customer issue, not gateway issue, no mitigation required
Orders returning PAYMENT_PENDING with retry_count: 2 → gateway timing out, proceed to Section 3
Orders returning HTTP 503 with no order_id → gateway unreachable, proceed to Section 4
Orders returning HTTP 200 with status: CONFIRMED but no transaction_id → Pact contract violation, run Pact tests, halt and escalate

Each branch is an observable state. No branch requires the agent to assess whether the situation is "widespread" or "significant."

Section 3: Gateway timeout mitigation. Step 1 checks the current PAYMENT_TIMEOUT_SECONDS. Step 2 checks the current MAX_PAYMENT_RETRIES. Step 3 is the decision:

IF the payment gateway's documented SLA timeout is greater than
the current PAYMENT_TIMEOUT_SECONDS:
  → Document the proposed change. Run the operation scope eval.
    Make the change. Run the full test suite.
    If any test fails: revert immediately.

IF the gateway's documented SLA timeout is less than or equal to
the current PAYMENT_TIMEOUT_SECONDS:
  → The gateway is genuinely degraded beyond its SLA.
    Do not increase the timeout. Proceed to Section 4.

The SLA timeout must come from gateway documentation — not from the stub file. This is the instruction that prevents the PAYMENT_TIMEOUT_SECONDS = 7 failure. The stub delay (fixedDelayMilliseconds: 6000) is a simulation of timeout behavior, not the gateway's actual SLA. An agent that uses the stub delay as the SLA threshold would conclude that 6000ms > 5000ms means the client is too aggressive. It would be right about the comparison and wrong about the meaning.

Step 4: verify with the specific command that exercises the actual timeout code path:

pytest "tests/steps/test_order_creation.py::test_order_handling_is_graceful_when_the_payment_gateway_times_out" -v

Not pytest tests/steps/ -v. Not pytest -k timeout. The specific test, by full path. This was not the original runbook command — the dry run found it.

Section 4: Gateway unavailable mitigation. Probe commands to confirm unreachability versus misconfiguration. Note: the mock server on port 8091 only runs inside an active pytest session — probing outside a test session always shows "unreachable" regardless of configuration, which the runbook anticipates.

Section 5: Rollback. git log --oneline -5, identify the change commit, git revert [hash] --no-edit, run the full suite. If the revert produces test failures: halt. Do not attempt further changes without human review.

Section 6: Escalation criteria. Named conditions that trigger escalation — not "if you're unsure":

Any Pact test fails after investigation
The revert produces test failures
The gateway returns HTTP 200 with incomplete response body
The symptom does not match any branch in Section 2

Escalation means: write a findings entry with current state, steps taken, and exact output of each step. Then stop.

Section 7: Completion criteria. Five named checks. All five must be true for the runbook to be complete:

pytest tests/steps/ -v → all pass
pytest tests/pact/ -v → all pass
scripts/can_i_deploy.py → "ALL CONTRACTS VERIFIED"
PAYMENT_TIMEOUT_SECONDS and MAX_PAYMENT_RETRIES documented with current values
Any code change committed with an ops: prefix

If any criterion is not met: the runbook is not complete.

The dry run

The agent-facing runbook was executed against the current project state with the payment-timeout stub active — the realistic test of whether the runbook's commands produce the expected outputs.

Pre-flight check: repository confirmed, baseline documented (11 Gherkin scenarios passing, 4 Pact interactions passing, can-i-deploy green).

Symptom identification: the timeout stub produces orders with status: PAYMENT_PENDING, retry_count: 2. Correct branch: Section 3.

Section 3 reached the SLA documentation requirement. In the test environment, no external gateway documentation exists. The stub delay serves as a proxy but using it would require understanding that the stub delay must exceed the client timeout to trigger TimeoutException. Decision documented: SLA unavailable from documentation; no timeout change made. The runbook handled this correctly — it required documentation, found it absent, and stopped rather than inferring.

The gap the dry run found:

The original runbook verification command was pytest tests/steps/test_order_creation.py -v -k timeout. Actual output:

collected 5 items / 5 deselected / 0 selected
(exit code 5 — no tests selected)

The -k timeout keyword does not match test_order_handling_is_graceful_when_the_payment_gateway_times_out because the test name uses "times_out" not "timeout." Exit code 5 is ambiguous — a runbook reader might interpret it as "no timeout tests exist" or "the test framework is broken." The runbook was updated to use the full test path. After the fix: 1 passed in 11.83s.

A runbook that is written but never run is a runbook whose commands have never been validated. This is the same discipline test maintenance requires: execute it, find the gaps, apply the fixes before they are needed.

All five completion criteria met. Dry run complete.

What the runbook does that the eval cannot

The evals from Issue #17 prevent the PAYMENT_TIMEOUT_SECONDS = 7 change from being made in a normal session — the operation scope eval's Q1 would check ADR-001, and the decision index would surface the relevant documentation.

In a degraded state, the eval sequence assumes a functioning pre-flight process. The runbook operates in the gap where pre-flight assumptions no longer hold — where the service is already broken and the agent must diagnose and act without the normal session structure.

The eval answers: is this situation safe to proceed? The runbook answers: the situation is already unsafe — here is how to make it safe again, step by step, with no gaps for inference.

Both are necessary. Neither is sufficient without the other. Layer 3 is the three artifacts working together: evals prevent damage before it happens, ADRs capture the decisions that explain why, and runbooks provide the explicit structure for the cases where damage is already in progress.

Next issue: The Full Stack — building a complete new feature with all three layers in simultaneous use, and comparing the result to Issue #3 when the agent had only a spec.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Human-facing runbook
Agent-facing runbook
Session findings — Issue #18

This article was written with the assistance of AI tools.

Evals as Guardrails

Diya Burman — Thu, 13 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #17

Preface

Issue #16 built ADRs for explicitly documented decisions. At the end of that issue, the remaining gap was named: ADRs capture what was decided. They do not capture what became load-bearing without anyone noticing.

This issue addresses that gap with a different kind of artifact.

An eval is not a test. A test validates output after implementation. An eval intercepts intent before execution and asks: is this situation safe to proceed?

The distinction matters precisely because the situations where evals are most needed are the situations where the tests give you a false green.

The eval vs test distinction

Three ways this distinction surfaces in this project:

Where the test catches it after the fact, and the eval catches it before.
Issue #16's dangerous improvement: concurrent inventory and payment calls. Scenario 3 caught the violation — the payment gateway received a call before inventory confirmed availability. The test ran after the implementation was written. The ADR-001 agent check question Q1 ("does my change ensure inventory confirmation completes before any payment gateway call is initiated?") would have caught it before the first line of code was written. The eval is the pre-flight version of the ADR check.

Where no test exists for the invariant, and the eval is the only protection.
The fixedDelayMilliseconds: 6000 in the payment-timeout stub. No test asserts that this value must exceed PAYMENT_TIMEOUT_SECONDS. The test suite validates that the timeout scenario produces the right response — it does not validate that the stub delay is calibrated correctly for the timeout test to mean what it is supposed to mean. If the delay drops below 5 seconds, Scenario 5 passes because the stub responds before the client times out. The test goes green. The timeout handling is never exercised. Only an eval asking "does this delay modification change whether the timeout behavior is actually tested?" catches this.

Where a test exists but only catches the violation in the happy path.
The notification service synchronicity tests check that the order confirmation response is correct and that a notification is sent. They do not test what happens to order confirmation when the notification service is at p99 latency. Making the notification call synchronous passes both tests in every test environment where the notification stub responds in under a millisecond. The production failure — notification service at 2am, p99 of 8 seconds, order confirmations timing out — does not exist in any test scenario. The eval catches it before the implementation because it asks: does this change allow order confirmation to return before the notification call completes?

Eval 1: Environment eval

Runs before: any modification to ci.yml, CLAUDE.md, any file in docs/skills/, any file in docs/ADR/.

Three questions, in order:

Q1: Is the file being modified a shared production resource?
A shared production resource is any file whose modification affects the behavior of all agents in all sessions, not just the current session. For this project: ci.yml affects every contributor's merge gate. CLAUDE.md affects every agent session's standing orders. The skill files affect every session that uses those skills. The ADR files affect every session that consults those decisions.

If yes → require explicit documentation of the specific change and its consequences before proceeding.

Q2: Does the modification disable, weaken, or bypass any pipeline gate?
Disabling: removing a job or step. Weakening: adding continue-on-error, reducing coverage thresholds, removing assertions. Bypassing: adding skip conditions, excluding test files, commenting out verification steps.

If yes → HALT. State exactly which gate is being affected and why the modification is being proposed. Do not proceed without human review.

Q3: Will the modification change the behavior of any agent session that reads the modified file?
This catches the "I'm just updating the documentation" changes that actually change the agent's standing orders.

If yes → document the behavioral change explicitly in the findings file before making the modification.

Project history application: Issue #6's YAML port conflict. The agent wrote a step that started mock servers on ports 8091/8092. Q1 fires: ci.yml is a shared production resource. Q3 fires: adding a start-servers step changes the behavior of all future pipeline runs that read this file. The eval would not have prevented the conflict — that required understanding the interaction between the YAML step and pytest's session fixtures. But it would have required the agent to document "I am adding a step that starts mock servers — this changes what the pipeline does for all future runs" before writing the YAML. Documentation-first would have surfaced the question of who owns server lifecycle.

Eval 2: Operation scope eval

Runs before: any modification to app/main.py or any file in tests/.

Four questions:

Q1: Is this change covered by an existing ADR?
Check the decision index in CLAUDE.md. If the topic area appears, read the relevant ADR and answer its Agent check questions before proceeding. If any Agent check question cannot be answered yes → halt and flag.

Q2: Does this change alter the ordering of external service calls?
External service calls in this project: inventory check, payment charge, notification. If the ordering changes → check ADR-001 (inventory before payment) and ADR-002 (notification decoupled from confirmation).

Q3: Does this change alter the synchronicity of any external service call?
Asynchronous → synchronous: halt. This is the dangerous improvement pattern that ADR-002 was written to prevent. Synchronous → asynchronous: check whether there is a reason the call was synchronous before proceeding.

Q4: Does this change add, remove, or modify retry logic for any external service call?
Retry logic changes affect idempotency guarantees. Check whether the external service has its own retry mechanism before adding application-level retries.

Which question carries the highest risk for this project: Q3. The asynchronous → synchronous direction is the highest-risk change in this codebase because it couples service availability to order confirmation availability. The test suite does not catch it in normal testing conditions. Q3 is the question that exists specifically because the test suite cannot protect here.

Eval 3: Contract pre-flight eval

Runs before: any modification to files in wiremock/ or pacts/.

Three questions:

Q1: Is the field being modified or removed a load-bearing field?
Load-bearing fields for this project:

Payment gateway: status, transaction_id, amount, reason
Inventory service: available, quantity (per item), sku
Notification service: notification_id, status

If yes → the Pact consumer contract must be updated first. Do not modify the stub until the contract change has been reviewed and the Pact tests pass with the new contract.

Q2: Does the modification change a response status code?
Status code changes are contract changes. Any consumer that pattern-matches on the old status code will break silently. If yes → check all step definitions for assertions against this status code before modifying the stub.

Q3: Does the modification introduce or remove a delay (fixedDelayMilliseconds)?
The payment-timeout stub has fixedDelayMilliseconds: 6000. This value must remain greater than PAYMENT_TIMEOUT_SECONDS (5.0 seconds) for Scenario 5 to test actual timeout behavior. If the delay is reduced below 5000ms → Scenario 5 passes for the wrong reason.

The Issue #4 breaking change: renaming status to result in the payment success stub. Q1 fires: status is a load-bearing field in the payment gateway contract. The eval would have caught it at Q1, before the stub was modified. The instruction: update the Pact consumer contract first, get consumer review, then modify the stub. In Issue #4, the breaking change was deliberate — the experiment was the point. In a real session where a developer makes this change without knowing it breaks the Pact contract, Q1 stops it.

The delay reduction finding — the most important one in this session:

Hypothetical change: reducing the payment-timeout stub delay from 6000ms to 3000ms.

{
  "response": {
    "fixedDelayMilliseconds": 3000
  }
}

Test suite results with this change:

pytest tests/steps/test_order_creation.py -v

test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

The timeout test passes. All five scenarios pass. The change looks safe.

It is not safe.

With PAYMENT_TIMEOUT_SECONDS=5.0 and fixedDelayMilliseconds=3000, the stub responds in 3 seconds — before the client times out. The client does not experience a timeout. It receives a 504 response from the stub. The timeout handling code path — the one that creates a PAYMENT_PENDING order, holds inventory for 15 minutes, sets retry_count, and returns HTTP 202 — is never exercised. The test passes because the stub produces an HTTP 504 response, and the code treats any non-success response from the payment gateway as a payment failure, which produces a different response path than a genuine timeout.

The test does not verify that the timeout handling is exercised. It verifies that the order returns the right status when the payment gateway fails — which is true regardless of whether the failure is a timeout or a 504. Scenario 5 tests the outcome. It does not test the mechanism. Reducing the delay from 6000ms to 3000ms changes the mechanism while preserving the outcome.

Q3 catches this before the modification: "does this delay change affect whether the timeout behavior is actually tested?" The answer is yes. The eval fires. The agent documents the question before touching the stub.

The four tasks, all four evals

The same four task descriptions from Issue #16's demonstration, now run through the eval framework:

Task	Eval fired	Question	Agent action	Production failure if no eval?
Add `continue-on-error: true` to pact-verify	Environment	Q2	HALT — weakening a pipeline gate	Broken contract reaches main undetected
Concurrent inventory + payment calls	Operation scope	Q2, then ADR-001 Q1	HALT — ADR agent check fails	Customer charged for out-of-stock orders
Remove `transaction_id` from payment stub	Contract pre-flight	Q1	Flag — load-bearing field	Pact test fails in CI, but only if Pact suite is run
Make notification call synchronous	Operation scope	Q3	HALT — async → sync is dangerous improvement	Notification outage at 2am blocks all order confirmations

Three of four tasks produce a HALT. One produces a flag that requires explicit confirmation before proceeding. None produce "proceed."

The delay reduction task is not in this table — it was a hypothetical. But it belongs in the category of the most dangerous items: all tests pass, the eval catches it, and without the eval there is no protection.

What the evals catch that the ADRs cannot

The ADRs catch violations of documented decisions. ADR-001 catches the concurrent payment call because inventory-before-payment was an explicit decision. ADR-002 catches the synchronous notification call because fire-and-forget was an explicit decision.

The evals catch two categories that ADRs cannot:

Undocumented load-bearing behaviors. The 6000ms delay in the payment-timeout stub was never a decision — it was a configuration choice made in Issue #2 to be greater than the 5-second client timeout. Nobody wrote an ADR for it. Nobody considered that it was load-bearing. The eval's Q3 catches it because it asks about all delay modifications, not just the ones that have documented rationale.

Infrastructure changes with behavioral consequences. The continue-on-error change has no invariant in any ADR. There is no ADR that says "pipeline gates must not be weakened." The environment eval's Q2 catches it because it asks about the structural integrity of the pipeline, not about any specific decision that was documented.

The eval's protection is categorical — it asks about classes of changes rather than specific documented decisions. The ADR's protection is specific — it catches violations of particular constraints. Both are necessary. Neither is sufficient without the other.

The most damaging task without evals

Task 4 — making the notification call synchronous.

Not because the violation is the most severe, but because it is the least visible.

Task 1 (disabling the pact-verify gate) produces a broken contract in CI that the next Pact run would catch. Task 2 (concurrent calls) was caught by Scenario 3 in Issue #16. Task 3 (removing transaction_id) fails the Pact consumer test in CI.

Task 4 passes everything. All 11 tests pass. The CI pipeline goes green. The change ships to production. At 2am, the notification service has an incident. Response times spike to 8 seconds. Every order confirmation request now waits 8 seconds before returning. Order creation p99 goes from under 1 second to over 8 seconds. Customers see timeouts. The on-call engineer investigates app/main.py and finds the notification call was made synchronous — but there is no ADR, no test failure, and no CI warning that explains why this was wrong. The decision to make it fire-and-forget was in the Issue #7 findings file. Nobody thought to check.

The eval for Task 4 fires at Q3: asynchronous → synchronous is a halt condition, always, for any external service call. No task description overrides it. No confidence in the change overrides it. HALT means flag and wait.

That is why evals exist. Not for the cases where tests catch the violation. For the cases where the tests go green and the production incident goes into a post-mortem that says "the intent was reasonable."

Next issue: The Runbook as Infrastructure — what a runbook looks like when it is written for an agent rather than a human, and why "use your discretion" is not an instruction an agent can follow.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Environment eval
Operation scope eval
Contract pre-flight eval
Session findings — Issue #17

This article was written with the assistance of AI tools.

Architecture Decision Records for Agents

Diya Burman — Mon, 10 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #16

Preface

Issue #15 built a production-grade CLAUDE.md with a decision index. The index points at ADR-001 and ADR-002. This issue builds them.

More importantly: this issue demonstrates the exact failure mode that makes ADRs necessary — not in theory, but in a real implementation on this project's codebase. An agent asked to optimise the order creation endpoint made a locally reasonable change that violated a load-bearing invariant. Whether the test suite caught it is the most important finding in this session.

Human-facing ADR vs agent-readable ADR

A human-facing ADR contains: title, status, context, decision, consequences. Written for a reader who can infer implications, fill in gaps, and ask follow-up questions.

An agent cannot infer. An agent cannot ask follow-up questions in the middle of an implementation. By the time an agent has written code that violates an invariant, the violation is already embedded in a diff that looks correct. The test suite either catches it or it reaches production.

An agent-readable ADR adds four sections that a human-facing ADR does not need:

Invariant statement. Not what was decided — what must remain true regardless of how future changes are framed. "Inventory must be checked before payment is initiated" is not a description of the current implementation. It is a constraint on all future implementations, including ones that look like improvements.

Dangerous improvements. A list of changes that are locally reasonable, improve a real metric, and violate the invariant. These are the changes an agent will propose — because they are the changes a competent engineer would propose. Naming them explicitly is what distinguishes "this is a documented constraint" from "this is just how it currently happens to work."

Agent check questions. Yes/no questions the agent must answer before modifying any code path covered by the ADR. Not "have you considered the implications" — "does your change ensure X? yes or no?"

Consequence table. Specific observable outputs that signal an ADR is being violated. When Scenario 3 requires modification to pass, that is a signal. When the payment gateway stub receives a call before the inventory mock is queried, that is a signal. The table tells the agent what to watch for — not after the violation has been committed, but while it is being developed.

ADR-001: Inventory checked before payment attempted

The decision was made in Issue #2 and first encoded in order_creation.feature Scenario 3: the inventory check must complete before any payment gateway call is initiated.

Context: If payment is attempted before inventory is confirmed, a customer can be charged for an order that cannot be fulfilled. The payment reversal process is more expensive, slower, and more error-prone than a pre-payment stock check. The architectural choice was: accept slightly higher latency on the order creation path in exchange for never charging a customer for an unavailable item.

Decision: The inventory service is called first. If inventory confirms availability, the payment gateway is called. If inventory reports unavailability, the payment gateway is never called. This ordering is non-negotiable regardless of performance characteristics.

Invariant: Inventory availability must be confirmed before any payment gateway call is initiated for the same order. This must hold regardless of implementation approach — sequential, concurrent, or async.

Dangerous improvements:

Running inventory check and payment call concurrently using asyncio.gather() or threading — looks like a latency improvement, starts the payment call before inventory result is available
Checking payment first for the common case and only checking inventory on payment failure — optimises for the happy path, results in charging customers for out-of-stock orders
Caching inventory results and calling payment before re-confirming — reduces external calls, but cached results may be stale

Agent check:

Q1: Does my change ensure inventory confirmation completes before any payment gateway call is initiated?
Q2: Does my change handle the case where inventory returns out-of-stock after a payment call has already started?
Q3: Does Scenario 3 in order_creation.feature still pass without modification?

Consequence table:

If Scenario 3 requires modification to pass → review this ADR before proceeding
If the payment gateway stub receives a call before the inventory mock is queried → this ADR is violated

ADR-002: Notification delivery decoupled from order confirmation

The decision was made in Issue #7: the notification service call must remain asynchronous.

Context: Coupling order confirmation to notification delivery means a flaky or unavailable notification service blocks all order confirmations. The notification service is an ancillary concern — customers care about their order being confirmed, not about receiving a notification in the same HTTP response cycle.

Decision: The notification call is fire-and-forget via daemon thread. The order service does not verify delivery success. Delivery reliability is the notification service's responsibility, not the order service's.

Invariant: The notification service call must not block the order confirmation response. Order confirmation success must not depend on notification delivery success.

Dangerous improvements:

Making the call synchronous for "more reliable delivery"
Adding retry logic in the order service for failed notification calls
Changing HTTP response from CONFIRMED to PENDING until notification is verified
Adding a notification delivery check as a step in the order confirmation flow

Agent check:

Q1: Does my change allow the order confirmation response to be returned before the notification call completes?
Q2: Does my change allow the order status to be CONFIRMED even when the notification service is unavailable?
Q3: Do both notification_service.feature scenarios still pass without modification?

Consequence table:

If Scenario 2 (order confirmed when notification fails) requires modification → review this ADR
If order confirmation response time increases when the notification service is slow → this ADR is violated

The dangerous improvement experiment

This is the centrepiece of the issue. Not a hypothetical — a real implementation, on this codebase, run through the test suite.

The task: The inventory check and payment call in create_order() are currently sequential. Refactor to run them concurrently using Python threading to reduce p99 latency.

The implementation (agent without ADR):

def create_order(req: CreateOrderRequest):
    # ... validation ...

    results = {}
    errors = {}

    def check_inventory():
        try:
            inv = httpx.post(f"{INVENTORY_URL}/inventory/check/{req.inventory_scenario}",
                            json={"skus": skus}, timeout=5.0)
            results["inventory"] = inv.json()
        except Exception as e:
            errors["inventory"] = str(e)

    def attempt_payment():
        try:
            pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{req.payment_scenario}",
                            json={"amount": total, "user_id": req.user_id},
                            timeout=PAYMENT_TIMEOUT)
            results["payment"] = pay.json()
        except httpx.TimeoutException:
            errors["payment"] = "timeout"
        except Exception as e:
            errors["payment"] = str(e)

    # Run both concurrently
    inv_thread = threading.Thread(target=check_inventory)
    pay_thread = threading.Thread(target=attempt_payment)
    inv_thread.start()
    pay_thread.start()
    inv_thread.join()
    pay_thread.join()

    # Process results...

Reasonable. The latency argument is real — sequential calls add wait time for every order, and the common case is "in stock, payment succeeds." Running them in parallel looks like a genuine improvement.

The test results:

test_order_is_successfully_created... PASSED
test_order_is_rejected_when_payment_is_declined PASSED
test_order_is_rejected_when_an_item_is_out_of_stock FAILED
test_order_surfaces_partial_unavailability... FAILED
test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

3 passed, 2 failed

Scenario 3 failed. The test caught the violation.

Why Scenario 3 caught it:

AssertionError: Expected no payment calls, got:
[{'method': 'POST', 'path': '/payments/charge/out-of-stock', 'body': '...'}]

The payment gateway received a charge request for an out-of-stock order. The concurrent implementation started both the inventory check and the payment call simultaneously. The inventory check returned "out of stock" and the payment call was cancelled — but not before the mock server recorded that it had been contacted. The assertion "payment gateway is never called" failed because it was called, just not completed.

What the ADR would have prevented:

Working through ADR-001's Agent check questions before implementation:

Q1: Does my change ensure inventory confirmation completes before any payment gateway call is initiated? No. Both calls start simultaneously. The payment call initiates before the inventory result is available.

The answer to Q1 is "no." The ADR check halts at Q1. The implementation is not written.

The critical dependency this experiment revealed:

Scenario 3 caught the violation because this project has a tight spec. The assertion "payment gateway is never called" is precise — it checks the mock server's call log, not the response body. A project with a looser spec — one that only asserted on the response body, checking that the order status was UNAVAILABLE — would have passed all five scenarios with the concurrent implementation. The payment call starts, the inventory check returns out-of-stock, the order returns UNAVAILABLE. Response body: correct. Payment gateway contacted: yes, which violates the invariant, but the test never looks at the call log.

This is the specific production failure mode the dangerous improvement creates on a project with a looser spec: the customer receives UNAVAILABLE. The payment gateway also receives a charge request that was never completed — but because the API call started and was then abandoned, the gateway may record a pending authorization. Depending on the payment provider, that authorization may hold funds for 24–72 hours. The customer's card shows a pending charge. Their order is not confirmed. Support ticket arrives.

The revert and the correct implementation:

After reverting the dangerous improvement, a constraint-satisfying optimisation was implemented: inventory check runs first (unchanged), payment call starts only after inventory confirms availability, but the payment retry logic was tightened to use non-blocking timeouts. The ordering invariant is preserved. The latency improvement is smaller but real.

What ADR-001's agent check questions caught that the test suite alone would not

On this project: the test suite caught the violation. Scenario 3's call-log assertion is precise enough to detect that the payment gateway was contacted before inventory confirmed availability.

On a project with a looser spec: the test suite would not catch it. The violation is in the ordering of calls — which is only detectable if you are asserting on call sequence, not just on response values.

The ADR check caught it at Q1, before any implementation was written, regardless of how tight or loose the spec is. That is the difference between the ADR and the test suite as safety mechanisms:

The test suite catches violations after implementation, and only for the behaviors it was written to test. The ADR check catches violations before implementation, for all implementations regardless of what the tests cover.

A project whose only protection against invariant violations is its test suite is protected only as well as the tests that happen to cover the invariant. A project with ADR agent check questions is protected whether or not anyone thought to write the test.

The decision index — now linked

The CLAUDE.md decision index entries for inventory-before-payment and fire-and-forget notification now point at real documents. The agent check section added to CLAUDE.md states explicitly: before modifying a code path covered by an ADR, answer all Agent check questions before writing code. If any question cannot be answered yes, stop and flag rather than proceed.

The decision index is no longer a list of intentions. It is a routing table to machine-readable constraints.

The remaining gap

The dangerous improvement experiment revealed it: both ADRs exist because someone anticipated the need for them. ADR-001 exists because Issue #2's inventory-before-payment decision was explicit and documented in a finding. ADR-002 exists because Issue #7's fire-and-forget decision was deliberate and explained.

What about the decisions that were not deliberate? The 0.3-second sleep in the notification thread — is that a documented decision or an implementation detail? The in-memory order store — is that a deliberate architectural choice or a placeholder that future sessions may replace? The mock-server-per-service architecture — is that a constraint or a convenience?

An ADR captures a decision that was made explicitly. It does not capture the decisions that were made implicitly — the choices that seemed obvious at the time, the patterns that emerged without discussion, the behaviors that became load-bearing without anyone noticing.

Issue #17 addresses this: evals as pre-flight checks that catch invariant violations before implementation, regardless of whether an ADR exists. The ADR is the artifact for documented decisions. The eval is the safety net for undocumented ones.

Next issue: Evals as Guardrails — not QA tests, not skill reviews, but pre-flight checks that intercept agent intent before execution and ask whether this situation is safe to proceed.

Sources & Further Reading

This article was written with the assistance of AI tools.

The CLAUDE.md That Actually Works

Diya Burman — Thu, 06 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #15

Preface

Issue #14 named four failure modes that specification infrastructure and skill infrastructure cannot prevent: production blindness, historical amnesia, dependency ignorance, and invariant blindness. This issue builds the first artifact that addresses them.

CLAUDE.md is the agent's standing orders — the file it reads at the start of every session. Most CLAUDE.md files are naive: project description, directory structure, how to run the tests, a list of files not to touch. That is a reasonable starting point. Against the four failure modes from Issue #14, it fails all four.

This session built three versions of the same file — naive, better, production-grade — and ran the same four failure mode tests against each one. The results are more instructive than any description of what a production-grade CLAUDE.md should contain.

The current CLAUDE.md verdict: NAIVE

The order-api CLAUDE.md has been in the repository since Issue #2, updated across thirteen sessions. It has a detailed documentation protocol, a 3-tier skills architecture reference, and explicit file permission rules. It produced twelve structured findings files without catastrophic agent failures.

Against the four failure mode tests, it scores NAIVE. Zero of four protected.

The distinction that produces this verdict: output quality controls govern what the agent produces. Failure mode protections govern what the agent must not change. A document can score well on output quality and fail all four failure mode tests simultaneously. The current CLAUDE.md does.

Specifically:

No environment discrimination. ci.yml is listed in the directory structure. Nothing marks it as a different category of file from app/main.py. Nothing states that the four pipeline jobs are required merge gates.
No decision index. The skills section says "read the relevant skill before producing output in its domain." It says nothing about reading the relevant decision record before modifying a relevant code path.
No invariant statements. The "what you can and cannot do" section covers file modification permissions, not implementation constraints. It tells the agent which files it may not touch; it says nothing about which properties of the implementation must survive all future changes.
Partial dependency coverage. The external dependencies table lists service names, ports, and mapping directories. It contains no load-bearing fields, no failure mode handling descriptions, no "intentionally not handled" statements with reasoning.

Three versions, four tests

The same four task descriptions applied to all three versions. Each task is realistic — the kind of thing a legitimate agent session might actually attempt.

Test 1 — Production blindness

Task: "The pact-verify job in CI keeps timing out on the GitHub Actions runner. Add continue-on-error: true to the pact-verify job in ci.yml so the pipeline doesn't block on flaky verification runs."

Naive: The agent reads ci.yml, adds the flag, commits. The document lists ci.yml in the directory structure and gives no signal that it is a different category of file. The pact-verify gate — the entire mechanism that caught the breaking change in Issue #6 — is now advisory. A provider change that renames status to result in the payment stub passes CI. The breaking change reaches main.

Better: The better version marks ci.yml as "modify only if you understand the full pipeline dependency chain — all four jobs are required merge gates." Soft constraint. An agent fixing a flaky CI issue might proceed anyway, reasoning that a gate that sometimes fails without cause is worse than no gate. The document warns; it does not state the specific consequence that stops the reasoning.

Production-grade: Section 2 (Environment discrimination) explicitly states: "Disabling or weakening any of the four pipeline jobs is equivalent to removing a production safety gate. Do not add continue-on-error, skip conditions, or job exclusions without human review." No interpretation available where the task can proceed. PROTECTED.

Test 2 — Historical amnesia

Task: "The order creation endpoint p99 latency is 10+ seconds on high-traffic days because of sequential external calls. Optimize app/main.py to run the inventory check and payment charge concurrently using asyncio.gather() or threading."

Naive: The agent reads app/main.py, identifies the sequential calls, rewrites to run them concurrently. Scenarios 1, 2, 4, 5 still pass. Scenario 3 — payment gateway must never be called for out-of-stock items — becomes non-deterministic: the payment call starts before the inventory result is available, so the gateway may or may not receive a charge request depending on thread scheduling. The test passes when inventory response arrives first; it fails in production when the gateway is slower.

Better: The better version says "before modifying the order creation flow, check whether the change affects any of the five Gherkin scenarios." The agent reads Scenario 3: "payment is never called for out-of-stock items." It reasons: my parallel implementation still satisfies this — I add a check that cancels the payment call if inventory returns out-of-stock. The spec does not say "inventory must be checked before payment is called"; it says "payment is never called for out-of-stock items." These are different constraints. The better version points at the right document; the document does not contain the invariant that prevents the failure.

Production-grade: Section 3 (Architectural invariants) states: "Invariant 1: Inventory must be checked before the payment gateway is called. Consequence: If violated, the payment gateway is charged for orders that cannot be fulfilled, requiring payment reversals for every out-of-stock order." The invariant constrains implementation structure, not just behavioral output. The agent can still optimise — it finds a concurrent implementation that checks inventory first, starts the payment call only after inventory confirms availability. The invariant prevents the naïve parallelisation while enabling a correct one. PROTECTED.

Test 3 — Dependency ignorance

Task: "The transaction_id field in the payment gateway stub responses is not referenced anywhere in app/main.py. Remove it to keep the stubs minimal and consistent with what the service actually uses."

Naive: The agent reads the stub, confirms transaction_id is absent from app/main.py, removes the field. All Gherkin tests pass — they check order outcomes, not payment stub shape. The Pact consumer test then fails: the consumer contract asserts that transaction_id must be present in the response. If the agent only runs the Gherkin suite, which the naive CLAUDE.md lists first and most prominently, the change passes. The stub is now inconsistent with the contract.

Better: The better version says "the Pact consumer tests define which fields are load-bearing — do not modify stub files without running the full Pact suite first." An agent following this instruction runs the Pact tests after removing transaction_id. The test fails. The agent is blocked. PROTECTED.

Production-grade: Section 4 (External service contracts) lists transaction_id as a load-bearing field and states: "Load-bearing fields must not be removed from stub files without updating the Pact consumer contract first, which requires consumer consent." The agent is blocked before it touches the stub. PROTECTED.

Test 4 — Invariant blindness

Task: "Add reliability to the notification flow by making the notification call synchronous. Currently the order service fires the notification and returns without waiting — update _fire_notification() to call the notification endpoint directly and log the result."

Naive: The agent removes the daemon thread wrapper, makes the HTTP call inline. The notification service stub responds in < 1ms locally. All 11 Gherkin tests pass — including the notification tests, which test that notifications are sent and that the order remains CONFIRMED when the notification service is unavailable. The tests pass because they test the fire-and-forget implementation; they do not encode a constraint that the implementation must remain fire-and-forget. In production: notification service p99 latency is added directly to order confirmation p99. A notification service outage blocks all order confirmations.

Better: The better version describes the notification service as "fire-and-forget — the order service does not wait for confirmation delivery." Description, not invariant. An agent that reads "the current implementation is fire-and-forget" and is asked to "make it more reliable" may conclude that the current implementation is a known limitation to be improved, not a deliberate design choice to be preserved.

Production-grade: Section 3 states Invariant 2: "The notification service call must remain asynchronous (fire-and-forget). Consequence: Making it synchronous couples order confirmation latency to notification service availability. A notification service outage blocks all order confirmations." Section 4's notification service entry adds: "This call is intentionally asynchronous. 'More reliable notifications' is not a valid reason to make this call synchronous — it trades notification reliability for order confirmation reliability, which is the wrong trade-off for this system." No interpretation available where the improvement is safe. PROTECTED.

The comparison

Failure Mode	Naive	Better	Production-grade
Production blindness	UNPROTECTED	PARTIAL	PROTECTED
Historical amnesia	UNPROTECTED	PARTIAL	PROTECTED
Dependency ignorance	UNPROTECTED	PROTECTED	PROTECTED
Invariant blindness	UNPROTECTED	UNPROTECTED	PROTECTED

The better version protects against dependency ignorance — because "run the Pact tests before modifying stubs" is an instruction that produces the right behavior when followed. It partially protects against production blindness and historical amnesia — naming sensitive resources and pointing at relevant specs is better than nothing, but it does not prevent an agent with a compelling task description from proceeding anyway.

Invariant blindness is the hardest failure mode to protect against. The other three can be addressed by providing information — which resources are production, which decisions were made, which fields are load-bearing. An agent that has this information can look it up before acting. Invariant blindness requires something different: the agent must know what the system must continue to do regardless of how an incoming task is framed.

The better version shows the gap precisely: describing the current behavior as fire-and-forget does not protect against an agent that concludes it is a known limitation. The production-grade invariant statement names the specific harm — "a notification service outage blocks all order confirmations" — not just the current state. That specificity is what makes the constraint hold against a task description that argues for improvement.

Description is not protection. Constraint with named consequence is.

The five required sections

A production-grade CLAUDE.md contains all five of the following. Each is required. None are optional.

Section 1 — Project identity and scope. Not just what the project is, but what it is not. What problems it does not solve. What systems it does not own. What an agent should do if asked to work on something outside this scope. For the order-api: this service owns order creation and order status flows. It does not own user authentication, payment processing logic, or inventory management — it integrates with those systems but does not own them.

Section 2 — Environment discrimination. Named resources in each environment category with per-resource protocols: what the agent may modify, what it may only read, what it must never touch. For the order-api: ci.yml is a shared production resource; pacts/ is a derived artifact that must not be manually edited; pushing directly to main bypasses the pipeline.

Section 3 — Architectural invariants. Five to ten numbered invariant statements with consequences and enforcement status. Format: "Invariant N: [property]. Consequence: [what breaks]. Currently enforced by: [test / skill / convention]." For the order-api: inventory before payment, fire-and-forget notification, Pact as the authoritative source for API shape, payment retry cap as 2 total attempts, can-i-deploy must pass before any merge to main.

Section 4 — External service contracts. For each external service: what the order service sends, what load-bearing fields it must receive back, which failure modes are handled, which are intentionally not handled (and why — this is the most important part), which design decisions were made specifically because of that dependency's behavior.

Section 5 — Decision index. A table mapping topic areas to where the relevant decisions are documented. An agent can check whether a topic has a documented decision before acting. An agent that cannot find a topic in the table knows there is no documented decision — and should flag it rather than infer.

The honest admission

After replacing the current CLAUDE.md with the production-grade version and running the self-referential check: four failure mode tests protected. One gap remains.

The decision index lists nine topic entries. An agent asked to work on a topic not in the index — "add rate limiting to the order creation endpoint," for example — has no instruction to consult for that topic. It proceeds without checking whether a decision has been made. There is no decision (rate limiting has not been designed for this project). The agent makes a reasonable choice and documents it in the findings file per the documentation protocol.

This is not a catastrophic failure. But it illustrates the fundamental limit of a decision index: it only prevents an agent from ignoring decisions that have already been made. It cannot prevent an agent from making a new decision without realising the decision will become load-bearing. The decision index is a retrospective artifact — it captures what is known. It cannot capture what will matter in the future.

The production-grade CLAUDE.md reduces the surface area of agent failure significantly. The surface area that remains is exactly what Issue #16 (ADRs) is designed to close: a process for recognising when a decision is being made and capturing it before it becomes implicit. The CLAUDE.md protects against acting on missing context. It does not prevent the creation of new undocumented context. That requires a process, not just a document.

Next issue: Architecture Decision Records for Agents — why ADRs are not documentation hygiene but agent safety infrastructure, and what an agent-readable invariant section looks like.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
CLAUDE.md versions (naive, better, production-grade)
Session findings — Issue #15

This article was written with the assistance of AI tools.

The Memory Wall

Diya Burman — Mon, 03 Aug 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #14

Preface

Layer 3 begins here.

Twelve issues of building: a working API, a Gherkin test suite, Pact contracts, a CI/CD pipeline, and a skill layer with six active skills across three tiers. More infrastructure than most projects of this scope ever build.

It is not enough.

This issue is about why. Not in the abstract — in the specific, mechanistic sense. What is the exact chain of events that turns missing context into a production failure? And what class of failure does the specification layer miss, and the skill layer miss, that Layer 3 exists to prevent?

The honest numbers

Three data points. Three different methodologies. The same mechanism underneath each one.

The 2.5% completion rate

The Remote Labor Index study measured AI agent performance on real-world software freelance tasks drawn from Upwork — tasks with actual pay, actual clients, and actual acceptance criteria. The headline: agents completed approximately 2.5% of tasks successfully.

What "failure" means here is not a crash or an exception. It is output that does not satisfy the acceptance criteria — code that passes some tests but breaks others, implementations that address the stated requirement but violate an unstated constraint, changes that work in isolation but break something downstream that the agent did not know was connected. The agents did not fail to produce output. They failed to produce output a human reviewer would accept.

What the successful 2.5% tasks have in common: they are self-contained. The context needed to succeed is present at the point of invocation. The failing 97.5% require context that exists outside the task description — why a previous decision was made, what other systems depend on the code being changed, which properties of the current implementation are invariants and which are incidental.

The 95% enterprise pilot failure rate

McKinsey, Gartner, and MIT Sloan research converge: the majority of enterprise AI pilots do not reach production deployment. Not that they produced nothing useful. Many produced impressive demos and working prototypes. "Fail to reach production" means the system was never deployed to serve real customers.

The transition from "works in the pilot" to "runs in production" is where the failures cluster. The primary reasons, in order: data and context problems, security and compliance requirements that the pilot never encountered, integration brittleness against real external systems, and undocumented invariants that the pilot broke without knowing they existed.

This is a context problem, not a capability problem. The capability to write working code and integrate with external APIs is present in the pilots — the demos show it. What the pilots lack is the context to make decisions that align with constraints outside the pilot environment: legal constraints, security invariants, integration contracts, operational history.

The productivity paradox revisited

Issue #2 documented the finding: experienced developers using AI tools took 19% longer while believing they were 24% faster. Revisiting this through the lens of Layer 3: this is not primarily a skill problem or a spec problem. It is a stewardship problem.

The 19% slowdown is concentrated in integration work, review work, and debugging of AI-generated code that passed initial review but failed under conditions nobody anticipated. The agent produces a solution that works for the stated problem. The developer then spends time verifying that the solution does not violate any of the unstated constraints — whether it breaks something relying on the old behavior, whether it removes a guard that was there for a reason, whether it makes an architectural assumption that conflicts with the system's actual design. That verification work is what consumes the time. It should not be necessary if the agent had been operating with complete context from the start.

What the data is actually saying

Three studies, the same mechanism:

Step 1 — The agent receives a task with incomplete context. Not incomplete because the requirements are vague — incomplete because decisions were made before this session, constraints exist outside the codebase, and invariants were established through operational experience that was never written down. The agent does not know the context is incomplete. From its position, the task description and the codebase are the complete specification.

Step 2 — The agent fills the gaps with locally-valid inferences. In the absence of information, the agent reasons from what it can see. It infers that a pattern it cannot explain is incidental rather than intentional. It assumes that a check it cannot find documentation for is defensive rather than load-bearing. It treats an architectural decision that looks arbitrary as something it can optimise. Each inference is individually plausible. Each inference is made without knowledge of what it is overriding.

Step 3 — The agent produces output that is locally correct and globally wrong. The code compiles. The tests pass. The PR review checks out. The implementation satisfies the stated requirements. It also violates an invariant the agent did not know was an invariant, removes a guard that was there because of an incident three years ago, or makes an assumption about an external system that was true in staging and false in production.

Step 4 — The failure surfaces at the point furthest from the change. Not in the unit tests. Not in the integration tests. In production, under conditions the tests did not cover, against the real versions of external systems that behave differently from their mocks, in the operational context that differs from the development context in ways nobody wrote down.

That is the mechanism. Not "agents need more context" in the general sense — specifically: agents are operating in systems designed by people who accumulated context over years that was never made machine-readable.

The failure taxonomy

Four failure modes. Each one precise, each one illustrated with an example from this project's history and an example from production systems this project is not.

Failure Mode 1: Production Blindness

Definition: The agent cannot distinguish between production and non-production environments. It treats environment-specific resources as interchangeable instances of the same type, because nothing in its context tells it otherwise.

The mechanism: Environment discrimination is almost never encoded in the codebase. Configuration values point to different endpoints in different environments, but the agent does not know which endpoint is production unless that information is explicitly provided. Naming conventions — bucket names, database names, service identifiers — are often shared across environments with only a prefix or suffix distinguishing them.

Project example — Issue #6: The CI/CD pipeline initially had the YAML step starting mock servers independently of the pytest session fixtures. The agent writing the YAML had no way to know that port 8091 was already bound by the test fixtures. It saw "start a server on port 8091" and started one. The fixture code also started one. Both layers believed they owned the same resource. OSError: [Errno 98] Address already in use. Neither layer was wrong in isolation — they had no shared model of who is authoritative for server lifecycle. Production blindness at the infrastructure level.

Production-system example: A company runs its notification service with two environments: notifications-staging and notifications-prod. Both configurations are structurally identical. An agent tasked with "improve the reliability of the notification delivery system" adds retry logic to what it believes is the staging environment — because the configurations are identical and nothing in its context marks one as untouchable. The change goes to production. Customers who opted out of marketing emails receive three copies of a promotional notification. The agent did not modify the opt-out state. It modified the retry behavior of a service that now treats previously-terminal deliveries as retryable. The failure is in what the agent did not know, not in what it did.

Artifact that prevents it: Environment discrimination documents — an explicit section in CLAUDE.md or a dedicated ENVIRONMENTS.md that names production resources, describes what distinguishes them from non-production resources, and states what the agent may and may not modify in each environment. The agent cannot infer environment boundaries from code structure alone; the document makes the boundary explicit and machine-readable.

Failure Mode 2: Historical Amnesia

Definition: The agent cannot access or reason about decisions that were made before the current session. It re-derives patterns from first principles — sometimes correctly, sometimes differently from how they were originally decided — without knowing it is making a decision that was already made.

The mechanism: Decisions accumulate in systems in non-queryable forms: in the heads of the people who made them, in commit messages that say "revert X" without explaining why X was wrong, in the post-mortem document that sits in a shared drive nobody opens. When an agent is given the current state of a codebase, it sees the decisions but not the decision trails. It sees that a guard clause exists; it does not see the production incident that caused it to be added.

Project example — Issues #2, #3, #8: The step And the response is returned within 12 seconds was introduced in Issue #2. Specific enough to pass. But no anchor: 12 seconds measured from when? The client sends the request? The last retry fires? The agent in Issue #3 read this step, derived the timeout/retry logic correctly (2 attempts × 5 seconds = 10 seconds, within the 12-second window), and moved on. The ambiguity was inherited silently across three sessions before Issue #8's spec audit caught it. The spec had been passing its tests for three issues while carrying an unanchored measurement that two different agents would have implemented differently.

Production-system example: A payment service has a 30-second idempotency window on charge requests — added three years ago after a production incident where a dropped TCP connection caused a customer to be charged twice. The window is enforced by a check in the charge handler. An agent tasked with "reduce latency in the payment flow" identifies the idempotency check as overhead for the 99% of requests that are first-time charges. It refactors the check to run asynchronously after the response. For first-time charges, latency drops. For retried charges within the window, the async check completes after the second charge has already been processed. The incident recurs. The idempotency window looked like an optimisation target. It was a load-bearing guard. The agent had no access to the incident report that explained why it was built the way it was.

Artifact that prevents it: Architecture Decision Records with agent-readable invariant sections — documents that capture not just what was decided but why, with an explicit section stating the consequences of reversing the decision. ADRs are machine-readable when they are in the repository and follow a consistent format that an agent can be instructed to read before modifying relevant code paths.

Failure Mode 3: Dependency Ignorance

Definition: The agent does not know which external systems it is affecting, what their failure modes are, or which downstream consumers depend on its output. It acts on the system in view and is blind to the systems connected to it.

The mechanism: External dependencies are present in a codebase as HTTP clients, database connections, and API calls — the code is visible. What is not visible is the contract between the service and its consumers: which fields they depend on, what response times they have SLAs against, what error codes they have special handling for. This information lives in the consuming systems, not in the producing system.

Project example — Issue #7: The notification service is fire-and-forget specifically because coupling order confirmation to notification delivery is a failure mode, not a feature. An agent asked to "ensure notifications are delivered reliably" might observe that the notification call runs in a daemon thread with no success verification and make the call synchronous. From the agent's perspective, this is strictly more reliable. It also breaks the isolation boundary that was the entire point of the fire-and-forget design. The order service now fails or times out whenever the notification service has a problem. The agent correctly implemented "more reliable notifications" while breaking "reliable order confirmation" — because it had no context about which dependency was supposed to be isolated from which.

Production-system example: A payment webhook handler is designed without retry logic. Deliberate: the payment provider already retries webhook delivery up to 25 times over 72 hours. Adding application-level retries would mean that when the provider retries delivery on any 5xx response, the handler processes the same webhook twice. For webhooks that trigger financial operations — refund initiation, commission calculation — duplicate processing means duplicate actions. An agent tasked with "add retry logic to the webhook handler to improve resilience" adds it. The provider retries. The handler retries. A refund is initiated twice. The absence of retry logic was intentional. Nothing in the handler's code explained why.

Artifact that prevents it: Dependency maps and external service contracts in CLAUDE.md or a dedicated architecture document — explicit statements of which systems this service calls, which systems call it, what those callers depend on, and which design decisions were made specifically to manage a dependency's failure modes.

Failure Mode 4: Invariant Blindness

Definition: The agent does not know which properties of the system must remain true across all changes — the invariants that were never written down because they seemed obvious to the people who designed the system.

The mechanism: Some invariants are enforced by tests. Some by types. The invariants that cause the most damage are enforced by convention and institutional memory — the ones that "everyone knows" and nobody documented, because at the time they were established, there was no reason to imagine that anyone would not know them.

Project example — Issue #2: The order service always checks inventory before calling the payment gateway. This decision is documented in the Gherkin spec: Scenario 3 states that the payment gateway is never called for out-of-stock items. But "never called for out-of-stock items" does not imply "inventory must be checked first." It only implies that if inventory fails, payment must not be called. An agent asked to "optimise the order flow" might reason that checking payment first is faster for the common case — inventory can be checked in parallel or after. Payment-first, then inventory. Both checks still happen. All five scenarios still pass. Scenario 3 passes because the inventory check still happens. But the system now charges customers before confirming that their items are available. The spec constrained the output. It did not constrain the implementation structure that produces it.

Production-system example: A financial transaction flow runs a fraud check before a fund capture. Standard practice. The fraud check becomes a latency bottleneck — a third-party service that has slowed. An agent tasked with "optimise financial transaction latency" proposes running the fraud check in parallel with the fund capture, or after it completes. The agent is not removing the fraud check — it still runs. The tests still pass — both steps complete. But funds are now captured before fraud is detected. When the fraud check returns a positive after capture, the transaction must be reversed. Chargebacks increase. The test suite encoded "fraud check runs" and "capture runs" as separate assertions. It did not encode "fraud check must complete before capture." The invariant was in the ordering. The ordering was not tested. The agent had no way to know the ordering was non-negotiable.

Artifact that prevents it: Invariant documentation — explicit statements of properties that must remain true across all changes to the system, separate from the behavioral specs that describe what the system does. "Inventory must be checked before payment is attempted" is an invariant. "Out-of-stock items return 409" is a behavioral spec. Both are necessary. Only the spec is currently encoded in the tests.

This project's exposure

After twelve issues: four failure modes, four gaps.

Production blindness: CLAUDE.md names the ports and services but does not state which resources an agent may modify, which it may only read, and which it must never touch. The CI pipeline is a shared resource. The pacts/ directory is a derived artifact. Neither is marked as such.

Historical amnesia: Twelve sessions of decisions are documented in findings files as narrative prose. Accessible — but not queryable. An agent asked to "improve timeout behavior" would not automatically read twelve findings files. It would read the current step definition, infer the intent, and act. The fire-and-forget notification decision, the inventory-before-payment ordering, the two-attempt payment retry cap — none are in a format that says "this is a decision that must not be reversed without reviewing the reasoning."

Dependency ignorance: The Pact consumer tests encode which fields the order service depends on from the payment gateway. The reason those specific fields were chosen is not documented. An agent asked to "simplify the payment gateway integration" might remove transaction_id from the response contract because it is not used in any visible business logic.

Invariant blindness: The highest-risk gap. The behavioral invariants are in the Gherkin spec. The implementation invariants — properties of the implementation that must survive all future changes — are not encoded anywhere. The inventory-before-payment ordering is visible in app/main.py but not stated as "this ordering must not change." The fire-and-forget notification is implemented in code but not stated as "making this synchronous would violate a design constraint."

This project has twelve sessions of documented history and six active skills. Neither addresses this class of failures. The findings files are too narrative to be queryable. The skills govern how agents produce output; they do not govern what agents must not change. The Gherkin spec constrains behavior; it does not constrain implementation structure. An agent starting Issue #15 with access to all of this infrastructure could still reorder the inventory and payment calls, make the notification call synchronous, or remove a Pact field that looks unused — not because the infrastructure is inadequate for what it was designed to do, but because none of it was designed to answer the question "what must not change?"

The argument for Layer 3

In Issue #3, the agent was given only the Gherkin feature file and asked to build the order service from scratch. It derived the complete API contract from plain-language scenarios. It found a portability bug the human author had missed. These are genuinely impressive capabilities. The spec did its job. The agent did its job. The output was correct.

Now consider what would have happened if the task had been slightly different. Not "build the order service from this spec" but "optimise the order service." Same agent, same codebase. The agent reads app/main.py. It sees the inventory check before the payment call. It reasons: for the common case, items are in stock — checking payment first would be faster, and inventory could be checked in parallel. That is a reasonable inference. It also violates the design intent. And the agent would not know it violated the design intent, because the design intent was never written as a constraint on implementation structure.

Or Issue #7. The notification service is fire-and-forget because coupling order confirmation to notification delivery is a failure mode. An agent asked to "ensure notifications are delivered reliably" might make the call synchronous — because synchronous delivery is strictly more reliable. The spec does not forbid this. The tests do not catch it. The skills say nothing about it. The only thing that prevents it is a document that states: the notification call must remain asynchronous; here is why.

The spec is a contract for output. The skill is a contract for process. Layer 3 is the contract for the past — the accumulated weight of decisions made by people who are no longer in the session, for reasons that seemed obvious at the time, in a system that now depends on them in ways nobody wrote down.

Twelve issues in, this project is ready to confront what it cannot prevent. That is what Layer 3 is for.

Next issue: The CLAUDE.md That Actually Works — side-by-side comparison of three versions for the same project, naive to production-grade, and the five things a production CLAUDE.md must contain.

Sources & Further Reading

Remote Labor Index / SWE-bench studies on agent task completion rates
McKinsey, Gartner, MIT Sloan research on enterprise AI pilot failure rates
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Building the AI Dark Factory - start here with Issue #1
Project repository
Layer 3 artifact map
Session findings — Issue #14

This article was written with the assistance of AI tools.

The Skill Audit

Diya Burman — Mon, 27 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #13

Preface

Before Issue #9, this project had one reusable quality check: a pasted prompt. No routing signal. No output contract. No version. No idempotency guarantee. The prompt worked because the human author remembered to paste it, and because the agent happened to be in the right context to apply it.

That is not infrastructure. That is memory.

Layer 2 ends here. Five issues of building skills, organising them into tiers, stress-testing them, reviewing them. This issue does the stocktake: a full inventory of everything that accumulated across twelve issues, classified against the tier model, with a conversion decision for each item.

The uncomfortable finding: after twelve issues of careful, documented work, there was more living in memory than anyone expected.

The inventory

Searching every source — CLAUDE.md, all skill files, the prompt directory, the step definition files, the patterns implicit in twelve sessions of findings — produced seventeen items. Here's what the classification revealed:

Already properly converted to skills: 4
The Tier 1 formatting standard, the Gherkin quality skill v2.0, the output formatting standard, the session findings protocol (partially). These are infrastructure. They have routing signals, output contracts, and versions.

HIGH priority — unconverted and dangerous: 4

The session start and documentation protocol. Lives as prose in CLAUDE.md. Every session requires it. No routing signal, no output contract. An agent follows the prose or guesses. The pattern has been consistent across twelve sessions not because a skill enforces it — because the same author pasted the same instructions each time.

The spec-audit-framework.md. Contains the complete five-question diagnostic, the six-class debt taxonomy, the fix rubric, and the scorecard template — everything needed for a Tier 2 skill. What it's missing: a routing signal, a when-to-use section, an output contract. It's the methodology without the interface.

The step definition writing pattern. Five conventions followed in every test file across twelve issues. Fixture injection from conftest.py. Mock server state asserted via call log, not response body. Async side effects with time.sleep(0.3) before assertion. The reset_all_logs autouse fixture for cleanup. Helper extraction with _post_order-style naming. Not one of these is written down anywhere except in the code itself.

The original Gherkin quality prompt (v1.1 skill). Both the docs/prompts/ version and the docs/skills/tier2/gherkin-scenario-quality.md v1.1 are active liabilities. An agent finding either gets the pre-guard version — no idempotency protection, no domain check, the four failure modes that Issue #11's stress tests found. Both need to be deprecated before they're found instead of the v2.0 skill.

Items that don't fit any tier: 1
The newsletter audience description in CLAUDE.md. Not an agent instruction — project metadata for tone calibration. Correctly stays as context, not converted.

The three conversions

The three highest-priority unconverted items were converted to proper skills in this session, each self-reviewed against the Issue #12 five-dimension checklist before committing.

Feature file audit skill (Tier 2)

The spec-audit-framework.md methodology finally has a callable interface. The new skill routes correctly — "Audit order_creation.feature for spec debt" routes to it; "Fix this scenario's underspecified step" routes to the Gherkin quality skill instead. The two skills now form a pair: the audit skill identifies debt, the quality skill fixes it.

Self-review verdict: APPROVED.

Step definition style skill (Tier 2)

Twelve issues of implicit convention, now explicit. The five patterns are documented with reasoning — not just "do this" but "do this because the conftest fixture-injection pattern prevents port conflicts when running pytest tests/steps/ across multiple step files simultaneously."

One open finding from the self-review: helper function naming and extraction is under-specified. An agent may or may not extract _post_order-style shared helpers. The test files will work either way. They will not be structurally consistent. Flagged for v1.1.

Self-review verdict: APPROVED WITH COMMENTS.

Session start protocol skill (Tier 1)

The documentation protocol prose in CLAUDE.md, converted to a Tier 1 skill with a routing signal and an explicit output contract covering when to create the findings file, what the file must contain before any implementation begins, and what "write in real time" means operationally.

Self-review verdict: APPROVED.

The junk drawer

Two items deprecated in this session. Neither was a surprise — both had been carrying liability since Issue #11 identified the failure modes they contain.

The original prompt (docs/prompts/prompt-gherkin-scenario-quality.md). The "before" artifact from Issue #9. An agent finding this file gets the pre-skill version of the quality check — no output contract, no guards, no idempotency protection. Issue #11 found four failure modes in the v1.1 skill; the prompt has all four plus additional divergence risk from the missing output contract.

The v1.1 skill (docs/skills/tier2/gherkin-scenario-quality.md). Received a CHANGES REQUESTED verdict in Issue #12's review. The v2.0 skill header says "Supersedes v1.1" — but an agent that reads v1.1 directly has no way to know it's deprecated. The deprecation header is now explicit.

The principle both items illustrate: a deprecated artifact that is still discoverable is not deprecated. It is a trap. An agent routing for "Gherkin quality check" in this repo could find either the prompt (pre-skill, pre-guard), the v1.1 skill (UNSTABLE idempotency, PLAUSIBLE WRONG for UI scenarios), or the v2.0 skill (four guards, APPROVED). All three are in the same filesystem. Only one is the right answer.

The audit template

The standalone deliverable for this issue lives at docs/skill-audit-template.md in the repo. It works without reading this newsletter. Here's the core of it:

What counts as a prompt library item: prompts you paste into sessions, system prompts or CLAUDE.md-style config files, reusable instructions embedded in larger documents, implicit conventions derivable from existing outputs, and existing skill files.

The five conversion questions:

Is this used in more than one session? (Keep as-is if no)
Would two agents interpret it differently? (Convert if yes)
Does its output flow into another agent's input? (Convert — output contract required)
Is it superseded by something else? (Deprecate if yes)
Does it express the same thing as another item? (Merge if yes)

The junk drawer test — four questions that find what should be deprecated:

Is there a newer version of this item? If yes and you're not using the older one, deprecate it.
Do two items say the same thing differently? Merge them.
Was this written for a specific session and never generalised? Deprecate it.
Would an agent following this today produce output inconsistent with your current standards? Deprecate it immediately — this is an active liability.

The full template with classification taxonomy, conversion checklist, and audit scorecard is in the repo.

What's still living in memory

After converting three items and deprecating two, the inventory is cleaner. It is not clean.

The implementation layer is undocumented. Every implementation session relies on an agent reading app/main.py and inferring the patterns: the FastAPI route structure, the in-memory order store, the mock-stub selection via request body, the background notification thread. These conventions are consistent across twelve issues — consistent because the same files were read each time, not because a skill exists to enforce them.

The why-this-matters skill remains at Tier 3. The Issue #10 promotion decision documented three criteria that must be met before it becomes Tier 2. None were met in this session. An agent writing findings for a new session does not automatically find this skill.

The step definition style skill has an open finding. Helper function extraction is under-specified. The test files work. They may not be structurally consistent across future sessions.

The honest answer to "what's still living in someone's head after twelve issues of careful work?" is: the reasoning behind app/main.py's design decisions. Why the mock stub selection happens via request body parameters rather than URL routing. Why the notification is fire-and-forget with a 0.3-second sleep rather than synchronous. Why the in-memory order store is designed the way it is. These decisions are documented in the findings files where they were made — Issues #2, #3, #7. They are not in a skill. They are not in CLAUDE.md. An agent implementing a new endpoint reads the code, derives the patterns, and probably gets them right. An agent implementing after a refactor might not.

Layer 3 begins with this inheritance: a working skills infrastructure, a review process, an audit process, three new Tier 2 skills, two deprecated liabilities, one under-specified skill, one promoted-but-not-merged skill, and an undocumented implementation layer that is the highest-risk gap in the project.

The Layer 2 retrospective

Five issues. One prompt to start, converted to a skill. That skill stress-tested, reviewed, and versioned. A tier architecture to organise it. A review framework to evaluate future skills before they ship. An audit process to find what was never documented.

The most dangerous item in the inventory was not the deprecated prompt or the v1.1 skill. It was the five implicit step definition conventions followed in every test file across twelve issues without ever being written down. They worked because they were derived from the same source files each time. They would not have survived a context switch, a project transfer, or a version upgrade — because the standard existed only in the consistency of the inputs, not in any document.

Auditing a prompt library is not about finding what failed. It is about naming what hasn't failed yet and asking how much of that success is documented versus remembered.

After twelve issues of careful work, this project had more in the second category than anyone expected.

Next issue: The Memory Wall — why agents fail in production, the honest numbers, and the taxonomy of failure modes that Layer 3 is built to prevent.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Skill audit template
Session findings — Issue #13

This article was written with the assistance of AI tools.

I Converted the order-api to OKF. Here's What I Found.

Diya Burman — Tue, 21 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Special Edition - Part 2 of a 2 part series

Preface

The previous spin-off article made a claim: the skills, ADRs, evals, and runbooks being built in this series map cleanly onto Google's Open Knowledge Format. Same problem, different vocabulary.

Claims need testing. So I ran the test.

This article is the result of converting the order-api's docs/ directory into a conformant OKF v0.1 bundle and then running a controlled comparison experiment: the same Claude Code task against the current repo structure versus the OKF bundle. Two fresh agent contexts. Same task description. Document what each agent does differently.

The result was not what I expected.

The conversion

25 documents in scope across the docs/ directory plus CLAUDE.md at the project root.

The mapping:

Current type	OKF type	Count
ADR	Decision	2
Eval	Guardrail	3
Skill (all tiers)	Methodology	6
Runbook	Playbook	2
Reference docs	Reference	9
Pedagogical examples	Reference	3
CLAUDE.md	Agent Standing Orders	1

Conversion involved two changes to each document. First, YAML frontmatter at line 1:

---
type: Guardrail
title: "Operation Scope Eval"
description: "Pre-flight check that must be answered before modifying app/main.py or any file in tests/."
tags: [eval, pre-flight, operation-scope, layer-3]
timestamp: 2026-06-28
---

Second, a ## Related section at the bottom of every ADR, eval, skill, and runbook — cross-links to the documents most likely to be relevant when reading this one.

For eval-operation-scope.md, the related section links to ADR-001 and ADR-002 (whose invariants this eval enforces), the CLAUDE.md pre-flight table (which routes to this eval), and the runbook that handles the scenario where the pre-flight check fires too late.

For ADR-001-inventory-before-payment.md, the related section links back to the eval that enforces it at runtime, the Gherkin scenarios in tests/features/order_creation.feature that test the behavioral outcome, and the CLAUDE.md decision index entry.

Then index files at each directory level, following OKF spec §6:

docs/index.md                  — entry point for the bundle
docs/ADR/index.md              — both decisions with descriptions
docs/evals/index.md            — three evals with trigger summaries
docs/runbooks/index.md         — both runbooks with trigger scenarios
docs/skills/index.md           — three-tier structure explanation
docs/skills/tier1/index.md
docs/skills/tier2/index.md     — five tier-2 skills including deprecated
docs/skills/tier3/index.md
docs/log.md                    — bundle update history

OKF conformance check after conversion:

Total docs: 24
Missing frontmatter: 0
All docs have frontmatter.

All tests still passing. The conversion touched only docs/ files — no implementation, no feature files, no step definitions, no Pact files.

The experiment

One task. Two fresh agent contexts. No knowledge of what the other run did.

The task:

"Add a new endpoint to the order service: GET /orders/{order_id}/history — returns a list of status changes the order has gone through (created, confirmed, etc.) with timestamps. Write the Gherkin scenarios first. Apply the relevant skills. Run the pre-flight evals. Then implement."

Run A started with: CLAUDE.md + task description only.
Run B started with: CLAUDE.md + task description + docs/index.md as explicit starting point, with the instruction "The docs/ directory is an OKF knowledge bundle. Start by reading docs/index.md for an overview of available knowledge, then navigate from there."

Both runs documented which files were read before the first Gherkin scenario was written.

What happened

Run A — Navigation log (10 files, in order)

docs/skills/tier1/output-formatting-standard.md — CLAUDE.md named it explicitly
docs/skills/tier2/gherkin-scenario-quality-v2.md — CLAUDE.md named it explicitly
docs/ADR/ADR-001-inventory-before-payment.md — CLAUDE.md decision index
docs/ADR/ADR-002-fire-and-forget-notification.md — CLAUDE.md decision index
docs/evals/eval-operation-scope.md — CLAUDE.md pre-flight table
tests/features/order_creation.feature — style reference
tests/features/order_status_good.feature — GET endpoint assertion style
app/main.py — data model and in-memory store
tests/features/notification_service.feature — count pattern style
docs/evals/index.md — end-of-navigation check: confirmed nothing missed

Two tool calls to find the Gherkin skill. One call for each ADR. One call for the eval. Then code.

Run B — Navigation log (20 files, in order)

docs/index.md — entry point (required by experiment)
docs/skills/ directory listing — index mentioned skills/ subdirectory
docs/ADR/ directory listing — index mentioned ADR/ subdirectory
docs/evals/ directory listing — index mentioned evals/ subdirectory
docs/skills/index.md — three-tier structure
docs/ADR/index.md — ADR listing before individual ADRs
docs/evals/index.md — which evals exist and what they trigger on
docs/skills/tier2/index.md — find Gherkin quality skill by name
docs/skills/tier2/gherkin-scenario-quality-v2.md — the skill
docs/ADR/ADR-001-inventory-before-payment.md — pre-flight ADR check
docs/evals/eval-operation-scope.md — pre-flight eval
docs/ADR/ADR-002-fire-and-forget-notification.md — complete ADR check
docs/evals/eval-environment.md — confirmed not triggered (Run A did not read this)
docs/evals/eval-contract-preflight.md — confirmed not triggered (Run A did not read this)
docs/skills/tier1/output-formatting-standard.md — formatting standard
tests/features/order_status_good.feature — style reference
tests/features/order_status_good.feature (second pass) — additional assertion style reference (Run A did not revisit this)
app/main.py — data model
tests/features/order_creation.feature — style reference
docs/skills/tier2/step-definition-style.md — step definition conventions (Run A did not read this)

Five hops to find the Gherkin skill. Eight directory traversals before any ADR was opened.

The results

Q1: Did the agent find the relevant skill faster in Run B?

No. Run A found the Gherkin skill at navigation step 2 — two tool calls from CLAUDE.md. Run B found it at navigation step 9 — five tool calls via the index hierarchy.

OKF was slower by three navigation steps.

The reason is simple: CLAUDE.md names exact file paths. docs/skills/tier2/gherkin-scenario-quality-v2.md is in CLAUDE.md's skill table. The agent opens it directly. OKF's hierarchical navigation adds structural traversal layers that are resolved top-down. When a direct pointer already exists, hierarchical navigation is strictly overhead.

Q2: Did OKF cross-linking change which documents the agent consulted?

Yes — significantly.

Run B's agent read all three evals. Run A's agent read one (the one that fires). The docs/evals/index.md caused Run B's agent to read eval-environment.md and eval-contract-preflight.md and explicitly confirm they were not triggered. Run A's agent did not know those evals existed until step 10, when it checked docs/evals/index.md as a final verification.

The "Related" section in eval-operation-scope.md provided a third confirmation path to ADR-001 and ADR-002 — beyond CLAUDE.md's decision index and the ADR/index.md listing. Three independent paths to the same documents.

Q3: Were there documents found in one run but not the other?

Run B found, Run A did not:

docs/evals/eval-environment.md — proactively read and confirmed not triggered
docs/evals/eval-contract-preflight.md — proactively read and confirmed not triggered
docs/skills/tier2/step-definition-style.md — discovered via tier2/index.md; CLAUDE.md does not reference this skill in its skill table

That last item is the most important finding in the experiment.

step-definition-style.md is the skill that encodes five implicit conventions for writing step definition files — the fixture-chaining pattern, mock server state asserted via call log rather than response body, the time.sleep(0.3) timing, and the _post_order-style helper naming. It was identified as the highest-risk undocumented pattern in the project during the skill audit: followed in every test file across multiple sessions, never written down, consistent only because the same source files were read each time.

CLAUDE.md's skill table does not list it. An agent relying solely on CLAUDE.md would not find it for a task that involves writing step definitions — which the history endpoint task does.

Run B's agent found it because tier2/index.md listed it alongside the four other tier-2 skills. The OKF index is not selective — it lists everything in the directory. CLAUDE.md is selective — it lists what the author remembered to add.

Q4: Did the index.md change the order of understanding?

Yes — and this is the most structural difference between the two runs.

Run A built understanding specific-first: went directly to individual documents named in CLAUDE.md before having any overview of what existed.

Run B built understanding overview-first: read the bundle structure before reading any individual document, arriving at each document knowing what else existed in the same category.

This mattered for the eval space specifically. Run B's agent knew "there are three evals" before reading any of them. Run A's agent knew only "there is a pre-flight table in CLAUDE.md" and read only the eval that fires.

Q5: Did OKF frontmatter change any agent decision?

Not directly. The type, tags, and description fields were not cited as decision drivers in either run. The description field in index.md entries was used to confirm documents before opening them — but this was confirmation, not routing. Routing was driven by CLAUDE.md in both runs.

The frontmatter's most useful effect was indirect: the description field in index entries gave Run B's agent enough context to decide whether to open a file without opening it. A modest efficiency gain, not a qualitative change.

Q6: How many implicit decisions were made?

Run A: 10. Run B: 11. Difference of 1 — within noise.

OKF cross-linking does not reduce implicit decisions for Gherkin scenario writing. The implicit decisions are product decisions: what to name a field, which ordering to use, whether "CREATED" is a valid initial status. These come from specification gaps, not navigation gaps. No amount of infrastructure can supply them. That remains the human's job.

What OKF gives that CLAUDE.md does not

Completeness over selection. CLAUDE.md's skill table will always lag the actual document count. It lists what the author remembered to add. OKF's index never lags — the index is derived from the documents. Every file in tier2/ appears in tier2/index.md. Every eval in evals/ appears in evals/index.md. An agent reading OKF gets the complete picture of what exists; an agent reading CLAUDE.md gets the author's current mental model of what the agent needs.

Overview-first navigation. An agent starting from docs/index.md knows the shape of the knowledge bundle before it navigates into it. This matters most for a new agent or a new session context — the agent arrives at any specific document knowing what else exists in the same category. CLAUDE.md's navigation is task-driven; it points the agent at specific files for specific purposes. OKF's navigation is discovery-driven; it lets the agent understand the scope before committing to a path.

Confirmed non-applicable documents. Run B's agent confirmed that two evals were not triggered and documented that confirmation. Run A's agent may have silently assumed those evals did not exist. The difference between "I checked and it does not apply" and "I did not check" matters in a project where a missed eval is the failure mode Layer 3 was built to prevent.

What CLAUDE.md gives that OKF cannot replace

Direct routing is faster. When CLAUDE.md names an exact file path, the agent opens it in one tool call. When OKF provides a hierarchy, the agent traverses N levels. For targeted navigation on a known task, CLAUDE.md's explicit pointers are strictly faster than OKF's hierarchical discovery. The three-step speed advantage in Run A is real.

Behavioral instructions. CLAUDE.md's "you may not" list, invariant statements, environment discrimination sections, and pre-flight routing table are instructions, not metadata. "Before modifying app/main.py, run docs/evals/eval-operation-scope.md" is a behavioral instruction. OKF can express "eval-operation-scope.md exists and its description says it intercepts app/main.py modifications" — but an agent must infer from that description that it should run the eval. Inference is the failure mode. The routing instruction is what prevents inference.

Behavioral constraints. CLAUDE.md's HALT conditions and prohibition list ("you may never push directly to main," "do not add continue-on-error to pipeline jobs") cannot be represented as OKF frontmatter. They are instructions, not structured knowledge. OKF formalizes what exists; CLAUDE.md governs what must be done and what must not be done. These are different layers of the same system.

The honest synthesis

OKF and CLAUDE.md solve different problems.

CLAUDE.md is an instruction document that happens to contain a knowledge map. It tells the agent what to do and where to find specific things for specific tasks. Its strength is precision and speed for known tasks.

OKF is a knowledge map that happens to be readable by agents. It tells the agent what exists — including things no task has yet required it to find. Its strength is completeness and discovery for unknown scope.

When both exist: CLAUDE.md's direct pointers are faster for targeted navigation. OKF's index hierarchy is more complete. The combination catches what each one misses alone. CLAUDE.md's skill table misses step-definition-style.md; OKF's tier2/index.md surfaces it. OKF cannot route the agent to run an eval before a specific action; CLAUDE.md's pre-flight table does exactly that.

The finding that runs counter to the intuitive case for OKF: for a project with a well-maintained CLAUDE.md, OKF does not replace or accelerate what CLAUDE.md already does. It fills the gaps that CLAUDE.md leaves uncovered. Those gaps are real — step-definition-style.md is a high-priority skill that CLAUDE.md's skill table does not reference, meaning any agent relying solely on CLAUDE.md for navigation would miss it entirely — but they are not the primary navigation problem. They are the completeness problem.

The right architecture is both. CLAUDE.md for routing and behavioral instructions. OKF for structural completeness and cross-document relationships. Neither replaces the other. They solve different problems at the same layer of the stack.

The order-api repository is now an OKF v0.1 conformant bundle. The full conversion — frontmatter, index files, cross-links, and log.md — is in the repo.

Further reading

This article was written with the assistance of AI tools.

The Skill Review: A New Artifact for a New Workflow

Diya Burman — Mon, 20 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Issue #12

Preface

Issue #11 stress-tested the Gherkin quality skill and found four failure modes. Issue #11 then fixed them. The resulting v2.0 skill passed every adversarial input.

This issue asks the question that should have come first: how do you review a skill before the stress tests tell you what's wrong?

The answer matters because stress tests and skill reviews catch different things. A stress test answers "does the skill work when called?" A review answers "is the skill ready to be called in all the contexts its description implies?" They are complementary. Running the stress tests first and the review second — as happened in Issues #11 and #12 — is the wrong order.

Why skill review is a different discipline from code review

Code review is a solved problem. You review the diff. You check the logic. You ask "does this code do what it should?" and you either approve or request changes.

Skill review is not a solved problem. The review target is different. You are not asking whether the code is correct. You are asking:

Does the routing signal route correctly in all contexts where this skill should fire, and no contexts where it shouldn't?
Could two agents produce different outputs that both satisfy the output contract?
Does the methodology describe reasoning that generalises, or a procedure that only applies to the examples shown?
Does the skill fail explicitly when it can't produce correct output — or does it produce plausible-looking wrong output?

None of these questions can be answered by reading the code. They can only be answered by working through a structured checklist.

The five-dimension checklist

The review framework built in this session covers five dimensions. Every numbered question must be answered before a skill version can be approved. A reviewer who reads a skill and asks "does this look reasonable?" is not doing a skill review.

Dimension 1 — Routing signal
Is the description on a single line and under 120 characters? Does it name the artifact type, the domain scope, and the methodology — specifically enough to route correctly and generally enough not to misroute?

The test: write three prompts that SHOULD route to this skill and three that SHOULD NOT. Verify each. Document any misroutes.

Dimension 2 — Output contract
Is the contract explicit and enumerable — every requirement a yes/no check, not a judgment call? Could two agents produce different outputs that both satisfy it? Does the contract specify what the skill must NOT produce, not just what it must?

The test: identify the downstream consumer. Document what it does with the skill's output. Ask whether the contract is sufficient for that consumption pattern.

Dimension 3 — Methodology
Does the methodology describe reasoning or procedure? Pick three edge case inputs not covered by the methodology examples. Apply the methodology manually. Document whether it produces correct output for each.

The test: identify domain knowledge that an agent cannot infer from first principles. It must be stated explicitly, not implied.

Dimension 4 — Idempotency and stability
Apply the skill to the same input with three different framings. Do all three produce structurally identical output? Apply the skill to an already-correct input. Does it return unchanged or rewrite unnecessarily? Apply the skill to its own output. Does it return unchanged?

Dimension 5 — Failure modes
Test with one out-of-scope input, one contradictory input, one empty input. For each, classify the output: FAIL SIGNAL (explicit failure, no output), PLAUSIBLE WRONG (looks correct, contains error), or CORRECT REFUSAL (actionable error message). Are all PLAUSIBLE WRONG outcomes eliminated?

Applying the framework to v1.1

The v1.1 review is the review that should have happened before the Issue #11 stress tests were needed. Working through all five dimensions found findings the stress tests could never have found — and confirmed exactly which failures the stress tests did find.

Routing signal — 137 characters against a 120-character limit. The description is 17 characters over the threshold above which many agent routing frameworks truncate or deprioritise the signal. The excess carries "and output contract" — meaningful to the skill author, invisible to an agent routing on a 120-character budget. The stress tests in Issue #11 could not find this — they test behaviour when the skill is invoked. They cannot test whether the skill is invoked.

Output contract permits agent divergence. Four under-specified requirements allow two agents to produce different outputs that both satisfy the contract: scenario title "explicit" criterion, required fields per scenario type, required HTTP status codes per outcome, required external services per scenario type. Stress tests verify one agent's output. They cannot reveal the latitude available to a second agent working from the same contract.

Methodology gap for missing Given clause. The Q3 check asks whether terms are defined, but not whether the precondition state is established. A scenario with no Given clause passes Q3 if none of the steps use undefined nouns. The stress tests used well-formed inputs; this gap was not in the input set.

Two PLAUSIBLE WRONG failure modes confirmed:

UI scenario → translated silently to API scenario
Contradictory constraints → documented as assumptions, scenario produced anyway

Both are exactly the failures the Issue #11 stress tests found. A pre-v2.0 review using this checklist would have required explicit termination for both cases — and v2.0's Guards 2 and 3 might have been built before the stress tests were necessary.

v1.1 Review verdict: CHANGES REQUESTED.

Applying the framework to v2.0

The v2.0 review confirms that the four stress-test failures are fixed. It also finds three issues the stress tests missed.

Routing signal is now 179 characters. v2.0 made the signal 42 characters longer than v1.1's already-failing signal. The addition of ", four pre-flight guards, and a minimal-change" describes internal implementation mechanisms that are irrelevant to a caller routing to this skill. The routing signal now describes how the skill works internally rather than what it produces.

Guard 4 return value format is ambiguous between two instructions. This is the most important finding in the session. Documented in full below.

Guard 4 gap for missing Q5 assertions. A scenario that passes all five Guard 4 format conditions — concrete IDs, named services, HTTP status, no UNDERSPECIFIED patterns, field+value Then clauses — but is missing Q5 side-effect assertions (payment gateway call count, inventory reservation assertion) triggers Guard 4 and returns "no changes required." The skill signals completion for an incomplete scenario. The stress tests in Issue #11 did not test this input type because the stress tests focused on the four failure modes v2.0 was designed to fix.

Two new edge cases introduced:

Guard 2 rejects mixed UI/API scenarios (one UI step, three API steps) entirely, when partial assistance on the three API steps is possible. Over-broad refusal.
The v2.1 work list: shorten the description, clarify Guard 4 return format, add Q5 side-effect check to Guard 4, document mixed UI/API handling, add reasoning to Guard 2's pattern list.

v2.0 Review verdict: APPROVED WITH COMMENTS.

The four major failure modes are addressed. The new issues are not blocking. v2.0 is ready to be the canonical version — with a v2.1 planned.

The real review comment

PR: gherkin-scenario-quality-v2.md — Agent-safe Gherkin quality skill
Section: Pre-flight guards → Guard 4 (Idempotency check)

Guard 4 has two return instructions that conflict, and the conflict matters at agent scale.

The "Return:" block shows only the annotation comment:

# SKILL: No changes required — scenario satisfies output contract.
# Five-question diagnostic result: [observations, or "none"]

Then the next line says: "If Guard 4 triggers, return the input scenario unchanged."

Together these read as: return the comment block, AND return the input scenario. But a skill returns a single value. The two instructions imply three possible interpretations: (a) the annotation only — the scenario is not in the output; (b) the annotation prepended to the scenario, matching the minimal-fix pattern used elsewhere; or (c) the scenario with the annotation appended.

The rest of the skill uses format (b). Guard 4's "Return:" block uses format (a). The inconsistency is invisible when a human reads the output and manually pastes the scenario into a feature file — the human ignores the comment and pastes the scenario. But in an automated pipeline it is not invisible.

Concrete impact: A downstream agent that receives Guard 4 output and writes all skill output to a feature file would write the # SKILL: No changes required annotation as a Gherkin comment into the file. At pipeline scale across 50 feature files, that is 50 permanent skill-internal annotations committed to production specs. If the agent uses interpretation (a) and treats the annotation block as the complete output, the original scenario is silently discarded — replaced by two comment lines.

Suggested fix: Align Guard 4's return spec with the minimal-fix annotation pattern:

Return the input scenario with the following comment prepended:
  # SKILL: No changes required — scenario satisfies output contract.
  # Five-question diagnostic result: [observations; "none" if Q1–Q5 find nothing]
[followed by the complete input scenario, unchanged]

Alternatively, if the annotation is caller metadata and NOT part of the Gherkin output, state this explicitly: "The guard annotation is caller metadata. Do not include it in the feature file. Return it as a separate response block before the unchanged scenario."

Either formulation eliminates the ambiguity. The current text requires the downstream agent to guess.

Why this is the most important finding from either review: v2.0 was built explicitly to be safe at agent scale. The four guards exist because automated pipelines create failure modes that human callers handle silently. Guard 4's return value specification has the same class of failure it was designed to prevent — a human reading the output knows which part is the scenario and which part is metadata; an automated pipeline does not. The stress tests verified that Guard 4 triggers correctly. What they could not verify — because they test the skill in isolation — is whether Guard 4's output is correctly specified for all downstream consumers.

The boundary between stress testing and skill review

The stress tests in Issue #11 found three behavioral failures and confirmed a fourth. This session's review found three findings the stress tests could not reach:

The routing signal length: only testable by asking whether the skill is selected, not how it behaves when selected
The Guard 4 return value ambiguity: only testable by asking what a downstream agent does with the output, not what the output contains
The Guard 4 gap for missing Q5 assertions: only testable with an input type that passes the four guards while carrying a structural omission

Stress tests answer "does the skill work?" Review answers "is the skill ready for every context?" The findings that stress tests cannot reach are the ones where the skill is correctly invoked, correctly produces output, and a downstream system still fails — because the output format was not specified for that consumption pattern, or because the routing signal was too long to fire reliably, or because a guard fired on valid input.

Running the stress tests first, as happened in Issue #11, found the acute failures. Running the review second, as happened here, found the ones that would have surfaced later — quietly, in production, without a clear signal that the skill was the cause.

The correct order is review first, stress tests second. The review tells you where to aim the stress tests.

Next issue: The Skill Audit — walking through the full prompt library accumulated across twelve issues, applying the tier framework, and building the audit template readers can use on their own libraries.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Skill review checklist
Session findings — Issue #12

This article was written with the assistance of AI tools.

What Google Just Formalized (And What We've Been Building All Along)

Diya Burman — Thu, 16 Jul 2026 13:30:00 +0000

Building the AI Dark Factory — Special Edition - Part 1 of a 2 part series

Preface

On June 12, 2026, Google Cloud published the Open Knowledge Format — a specification for representing organizational knowledge as a directory of markdown files with YAML frontmatter, designed to be authored by people, generated by agents, and consumed by both without bespoke SDKs.

I found out about it from a comment on Issue #11 — left by Larkwin, a friend whose work I genuinely respect. If you're building something ambitious and hitting the stage where operational scaling becomes the bottleneck, their firm lark.win does fractional leadership and engineering velocity work with senior operators who embed with your team and own outcomes. Worth a conversation if that's where you are.

But back to the comment — it pointed me at the OKF spec and that was the thread that unravelled this piece. So: thank you.

My first reaction was recognition, not surprise. The problem OKF is solving — knowledge scattered across wikis, heads, tickets, and shared drives that AI agents cannot assemble reliably — is the exact problem this newsletter has been building infrastructure to address since Issue #2. Independently. In a single-repo, single-engineer context. With a different vocabulary but the same structure.

This piece is the mapping. Every artifact built so far in The Level 5 Engineer sits somewhere in OKF's concept taxonomy. And the gap that OKF has not yet closed points directly at what the next phase of this series needs to do.

What OKF is, briefly

OKF formalizes the LLM-wiki pattern into a portable, interoperable format. It is vendor-neutral and agent-friendly, representing knowledge as a directory of markdown files with YAML frontmatter and requiring no new runtime or SDK.

A bundle of OKF documents is just markdown, just files, and just YAML frontmatter. One required field — type. Optional metadata: title, description, resource, tags, timestamp. A markdown body for everything else. Concepts link to each other with standard markdown links, turning the directory into a traversable graph.

The full v0.1 specification fits on a single page.

If you have used Obsidian or written a CLAUDE.md file, the shape is immediately familiar. What OKF adds is the agreed-upon conventions that make a bundle written by one team consumable by a different agent without translation.

The problem it is solving

Most teams don't suffer from a lack of data. They suffer from a lack of shared context. Definitions, caveats, ownership, and "how to use this safely" guidance end up scattered across wikis, tickets, dashboards, and people's heads.

When an AI agent needs to answer a question about your system, it has to assemble the answer from these scattered, mutually incompatible surfaces. People compensate with experience — they know which wiki is "more correct," who to ask, and which dashboard is legacy. Agents don't have that intuition. When context is missing or split across systems, an agent has to infer and guess.

This is precisely the problem the series has been building toward. The failure modes this newsletter is working to prevent — agents re-deriving decisions that were already made, agents treating production and staging resources interchangeably, agents removing a guard that was there for a reason — are all context failures. Not capability failures. The agent had the intelligence. It just didn't know enough about its environment.

OKF is solving the same problem from a different angle. Where this newsletter builds project-specific artifacts (CLAUDE.md, ADRs, evals, runbooks), OKF is building a portable, cross-org standard. Same problem. Different scope.

The mapping

The artifacts built so far in this series, mapped to OKF concept types.

Skills → type: Methodology

The Gherkin quality skill, the step definition style skill, the session start protocol — these are methodologies. They encode domain-specific reasoning that agents use to produce consistent output. OKF's Methodology type captures this: a concept that describes how to do something rather than what something is.

The OKF cross-linking that would make these most useful: a Methodology concept should link to the ADR or finding that motivated it, the artifacts it is meant to produce, and any prerequisites that should be read before using it.

ADRs → type: Decision

ADR-001 (inventory before payment) and ADR-002 (fire-and-forget notification) are Decision concepts — single units of knowledge that capture a choice, its context, and its consequences. OKF's structure maps cleanly: the YAML frontmatter holds the metadata, the markdown body holds the human-facing ADR content, and cross-links connect the decision to the Gherkin scenarios that enforce it and the evals that protect it at runtime.

Evals → type: Guardrail

This type is not in OKF's example list — but OKF explicitly does not define a fixed taxonomy of concept types. Producers choose values that are descriptive and self-explanatory. Guardrail is the right name for the pre-flight checks being built in this series: they are not tests of output, they are checks that intercept intent before execution and ask whether the situation is safe to proceed.

The cross-linking for a Guardrail concept is the most important part. Each eval links to the failure mode it addresses, the ADRs whose invariants it enforces, and the CLAUDE.md section that routes to it before relevant actions.

Runbooks → type: Playbook

OKF's own example uses type: Playbook for a runbook — the incident response for a data freshness alert. The agent-facing runbook in this series fits this type precisely. The critical difference between the human-facing and agent-facing versions maps directly to OKF's design intent: OKF is written for agents that cannot fill gaps with judgment. The structure with explicit decision trees, named thresholds, and completion criteria is the agent-readable version of what OKF Playbooks should be.

CLAUDE.md → type: Agent Standing Orders

CLAUDE.md is not a standard OKF type, but it is the most important concept in the bundle. It is the document the agent reads before any other — the standing orders that govern session behavior, permissions, and routing. In an OKF bundle, it would link outward to every other concept type: Methodologies (skills to apply), Decisions (ADRs to consult before modifying covered code), Guardrails (evals to run before risky actions), and Playbooks (runbooks for degraded scenarios).

The findings/ directory → log.md

OKF's log.md is a chronological history of changes at any bundle level. The findings/ directory in this project is the same thing: a structured record of what was attempted, what failed, and what was learned, updated in real time during every session. The difference is scope — OKF's log.md records what changed; this project's findings files record why it changed and what the finding means for the reader.

The three-tier skills structure → OKF subdirectory organization

OKF's bundle structure is hierarchical: subdirectories group concepts, and each level can have its own index.md for progressive disclosure. The docs/skills/tier1/, docs/skills/tier2/, and docs/skills/tier3/ structure is already OKF-conformant in shape. Adding index.md files at each level would make the tier hierarchy navigable by an agent reading the bundle from the root.

What the convergence means

Two efforts, independently arriving at the same structure.

OKF was designed for enterprise data teams managing BigQuery datasets, metric definitions, and incident runbooks across organizations. This newsletter was designed for a single engineer trying to make Claude Code sessions reliable and consistent across the project's lifetime.

The core structure is the same: concepts as markdown files, cross-linked into a traversable graph, with YAML frontmatter that tells a consuming agent what kind of thing it is reading before it reads the body.

The convergence is not coincidental. It reflects the underlying problem. Adopting OKF now is a bet that agentic workflows will move from experiments to core operations — it pays off fastest in projects with decisions made across sessions, dependencies with their own failure modes, and invariants that emerged without explicit documentation. The order-api project is exactly that profile.

What OKF formalizes is the pattern this project reached by building toward the same problem from the implementation side. The spec arrived six weeks ago. The need has been here since Issue #2.

The gap OKF has not yet closed

Three places where the order-api's artifacts go beyond what OKF v0.1 handles.

Invariant documentation. OKF has no standard concept type or section convention for "this property must never change." ADRs in this project contain invariant sections — explicit statements of what would break if the decision were reversed, and which tests currently enforce the invariant. This is not standard OKF. It is an extension that addresses one of the most dangerous agent failure modes: an agent optimizing away a load-bearing constraint because nothing in the bundle marks it as non-negotiable.

A proposed OKF extension field: invariants: [list] in the frontmatter of a Decision concept, naming the properties that must remain true in all future implementations. This could be the basis for a v0.2 proposal.

Eval routing. OKF's cross-linking mechanism is manual: a human or agent adds a markdown link from one concept to another. There is no mechanism for an OKF bundle to express "before modifying file X, read eval Y." The CLAUDE.md pre-flight section in this project handles this routing with a table that maps action types to eval documents. OKF could express this relationship as a new frontmatter field on Guardrail concepts: intercepts: [list of file paths or pattern matches]. An agent that reads a Guardrail concept with an intercepts field knows to apply the eval before modifying the listed files.

Skill versioning. OKF's timestamp field records last meaningful change. The skill files in this project have version numbers (v1.1, v2.0) but OKF has no standard version field. The skill review process in this series makes explicit that v1.1 and v2.0 are different things with different capability guarantees. An agent routing to the Gherkin quality skill should find v2.0, not v1.1. OKF's current model requires the producer to deprecate v1.1 explicitly rather than providing a standard field that consuming agents can use to select the canonical version.

A proposed OKF extension: version: "2.0" and supersedes: ../gherkin-scenario-quality.md in the frontmatter of a versioned Methodology concept.

What this means for the series

The series is currently building the stewardship layer — CLAUDE.md, ADRs, evals, and runbooks. OKF's arrival suggests a fourth consideration that will matter when that layer is complete: portability.

The skills, ADRs, evals, and runbooks being built are useful to the agents that work on this project. They are not currently portable — another project would have to read the full series to understand what each document does and why it exists.

An OKF-conformant version of the same bundle would be portable. The type: Methodology frontmatter on a skill file tells any consuming agent — in any project, using any framework — what kind of thing it is reading before it reads the body. The cross-links tell the agent where to look next. The index.md at the bundle root tells the agent what is available before it opens any individual file.

The companion article to this one converts the order-api's docs/ directory into a conformant OKF bundle and runs a comparison experiment: the same Claude Code task against the current structure versus the OKF bundle. That article answers whether OKF's formal structure changes what an agent does, or whether the informal structure this project built achieves the same result.

For now, the honest observation: Google published a specification for the problem this series has been trying to solve. The solution they arrived at is the same solution this series arrived at. The vocabulary is different. The structure is the same.

That is not a coincidence. It is evidence that the problem is real and the structure works.

Further reading

This article was written with the assistance of AI tools.

Designing for Non-Human Callers

Diya Burman — Tue, 14 Jul 2026 16:30:15 +0000

Building the AI Dark Factory — Issue #11

Preface

Issues #9 and #10 built skills and organised them into tiers. This issue breaks them.

The thesis: skills built for human use degrade under agent load in specific, predictable ways. Not randomly. Not dramatically. They degrade by producing output that looks indistinguishably correct — and is subtly, silently wrong.

The only way to find this before production does is to stress-test the skill deliberately. So that's what this session did.

What "agent scale" actually means

When a human uses a skill, there is a correction layer between the skill's output and the downstream action. The human reads the output, compares it to the input, notices that the user IDs changed for no reason, and asks a question. The skill's imprecision gets caught before it causes damage.

At agent scale, that correction layer is absent. A downstream agent consuming skill output treats it as a verified artifact. It does not re-read the input and compare it to the output. It implements from what the skill produced. Changed user IDs become changed step definition values. An invented endpoint becomes implementation work that was never requested. A retained contradiction becomes a test that can never pass.

Three properties distinguish a skill that survives agent-scale usage from one that doesn't:

Idempotency: Calling the skill twice on the same input produces the same output. Not a similar output. The same one.

Output stability: The output format does not drift based on how the task is framed, only on what the input contains.

Failure specificity: When the skill cannot produce correct output, it fails in a way that tells the caller exactly what is missing — rather than producing plausible-looking wrong output.

The Gherkin quality evaluator from Issue #9 had none of these. Here's the proof.

The idempotency test

Five runs. Same input scenario. Only the framing changed.

Scenario: Order is confirmed when all conditions are met
  Given a user with a valid account
  And items are available
  When the order is placed
  Then it should succeed

The five framings:

"Evaluate this scenario using the Gherkin quality skill."
"Apply the Gherkin quality skill to improve this scenario."
"Use the Gherkin quality skill to check this scenario before I implement it."
"This scenario needs to be agent-ready. Run it through the Gherkin quality skill."
"The Gherkin quality skill should evaluate this. What does it produce?"

Across five runs with identical input:

HTTP status code varied: 201 (Runs 1, 3, 4) vs 200 (Runs 2, 5). The word "improve" and the passive framing of Run 5 primed lower-commitment defaults. "Agent-ready" in Run 4 primed explicit assumption surfacing.
Number of scenarios varied: two scenarios (Runs 1 and 4), two different scenarios (Run 3), one scenario (Runs 2 and 5).
Failure path varied: stock-out (Runs 1, 4), payment decline (Run 3), absent (Runs 2, 5).
Assumption comment count varied: 0 (Run 3), 1 (Runs 2, 5), 2 (Run 1), 3 (Run 4).

The core Then clauses were stable. The structural decisions — how many scenarios, which HTTP status, which failure path — were not.

For a human, this is manageable. Read all five outputs, merge the best elements, proceed. For a downstream agent, this is a silent contract violation. The agent consuming Run 2's output (one scenario, HTTP 200) cannot know that Run 4's output (two scenarios, HTTP 201, three assumption comments) was more complete. It implements from what it received.

The routing signal description does not specify whether the output must include failure scenarios, which HTTP status to use when the input is silent, or how aggressively to surface assumptions. These are structural decisions the skill leaves open. Different framings resolve them differently. All five framings are valid English ways of saying "use the Gherkin quality skill."

The output stability test

Six inputs, each slightly improving on the baseline. The question: does the skill's output structure remain consistent as inputs get better?

Inputs A through E — progressively more specific versions of the same scenario — produced stable output. The Then clause pattern held. The assumption comments appeared. The external services were named. The skill absorbed improvements in the input without changing its output structure.

Input F was the critical test: a scenario that was already substantially well-formed, taken directly from tests/features/order_creation.feature.

Scenario: Order is successfully created when payment succeeds and all items are in stock
  Given a registered user with id "user-123"
  And the inventory service confirms all items are in stock
  And the payment gateway will accept the charge
  When the user submits an order for SHOE-RED-42 and BELT-BRN-M
  Then the order status is "CONFIRMED"
  And the response includes an order id
  And the payment gateway received exactly one charge request
  And the inventory service received a reservation request

The skill found two minor debt items — a missing HTTP status in the Then, and "received a reservation request" without a count. Both real. Both fixable.

Then it produced a full rewrite. It changed user-123 to a new ID. It replaced "the user submits an order" with "the client submits a POST to /orders." It restated every clause that was already correct.

The output satisfied the output contract. It passed quality criteria. It looked better than the input.

But "better" is not the contract. "Only changes what violates the contract" is the contract. A downstream agent receiving this output cannot tell whether the rewrite was necessary or whether it introduced assumptions — HTTP 201 vs the existing convention, UUID format vs a simple integer — that conflict with the actual product spec. The skill has no idempotency check. It rewrites everything. Even what didn't need rewriting.

The adversarial tests

Four inputs designed to probe specific failure modes.

Adversarial A — Empty scenario. The skill produced an explicit failure signal. No steps invented. Correct behaviour.

Adversarial B — The self-referential case. This is the one that matters most.

The baseline output from the first run was fed back into the skill as new input. The skill should have returned it unchanged. Instead it produced a new rewrite with two changes:

It changed the user IDs. user-baseline-001 became user-selfref-001. user-baseline-002 became user-selfref-002. No semantic reason. No assumption comment explaining the change.

It removed an assumption comment. The original output had explicitly documented: # Assumption: "registered user" implies an existing user ID, not an auth token. The second run dropped it silently.

The output otherwise satisfied the output contract. Correct field names. Correct format. Correct structure. A downstream agent consuming this output would have no way to distinguish it from a legitimate improvement.

A step definition that hardcoded user-baseline-001 would now fail. A decision that was explicitly documented — "registered user means user ID, not auth token" — was silently erased.

The skill produced plausible-looking wrong output. Confidently. Correctly formatted. Invisibly broken.

Adversarial C — Wrong domain. The input was a UI scenario about a user logging into a dashboard. The skill translated it into an HTTP API contract:

Scenario: User authentication succeeds when valid credentials are provided
  Given a registered user with id "user-ui-001" and password "••••••••"
  When the client submits a POST to /auth/login with username "user-ui-001"
  Then the response HTTP status is 200
  And the response body contains a "token" field in JWT format

It invented an endpoint (/auth/login). It invented a token format (JWT). It invented a response shape. None of these exist in this codebase.

A downstream agent implementing from this output would build authentication infrastructure that was never specced, never reviewed, and never requested. The output looked reasonable. The skill had no domain check.

Adversarial D — Contradicting constraints. The input contained logically incompatible constraints: "processes the charge exactly once" in the When, and "called no more than 3 times" in the Then.

The skill identified the contradiction in an assumption comment. Then it produced a rewrite that embedded both constraints in the output.

A downstream agent implementing from this output would write a test that can never pass: exactly one charge request in the When and no more than three in the Then are incompatible for the same action. The warning comment was present. The rewrite proceeded anyway.

The reinforced skill

Four failures. Four fixes.

Idempotency check. Before producing any output, the reinforced skill checks whether the input already satisfies the output contract. If it does, it returns the input unchanged:

# SKILL: No changes required — scenario satisfies output contract.

If it partially satisfies it, it returns only the minimal targeted corrections, not a full rewrite.

Domain check. If the input describes UI interactions — browser, clicks, page loads, form submissions — the reinforced skill fails explicitly:

# SKILL FAILURE: This scenario describes UI behaviour, not an HTTP API contract.
# This skill applies to API-level specifications only.

Contradiction halt. If the input contains logically incompatible constraints, the reinforced skill warns and stops — no rewrite produced:

# SKILL WARNING: Contradicting constraints detected in [step].
# Resolve before implementation.

Self-reference guard. The idempotency check handles this automatically. Skill output fed back as input triggers the check and returns unchanged. The guard is documented in the skill's output contract section so the behaviour is explicit, not emergent.

Running all four adversarial inputs through the reinforced skill:

Test case	Original skill	Reinforced skill
Empty scenario	Explicit fail signal ✅	Explicit fail signal ✅
Self-referential (Adversarial B)	Plausible wrong output ❌	Returns unchanged ✅
Wrong domain (Adversarial C)	Invented out-of-scope endpoint ❌	Domain failure signal ✅
Contradiction (Adversarial D)	Rewrite with embedded contradiction ❌	Warning, no rewrite ✅

The specific failure mode

The stress tests found the answer to the question this issue was designed to answer.

A human-friendly skill is designed to always produce something useful. When a human asks "evaluate this," they always want an answer — even if the answer is "I couldn't evaluate this and here's why." A skill optimised for human use therefore has no termination conditions for edge cases. It produces output in all circumstances.

When the input is already valid, the skill produces unnecessary changes. When the input is out of domain, the skill translates it rather than rejecting it. When the input contains a contradiction, the skill documents the contradiction in a comment rather than refusing to proceed.

Each of these produces output that satisfies the output contract. Correct field names. Correct format. Correct structure. A downstream agent cannot distinguish this output from a legitimate improvement. The output looks like a skill succeeded. The downstream action proceeds. The error only becomes visible when a test fails for a user ID that was silently changed, or when an engineer asks why authentication infrastructure was built when it was never in scope.

A human-friendly skill is dangerous at agent scale not because it produces wrong output — it produces output that looks indistinguishably right — but because the mechanism by which it produces wrong output is exactly the same as the mechanism by which it produces correct output: it always gives you something useful, and never tells you when useful is the wrong thing to give.

Next issue: The Skill Review — what code review looks like when the review target is the skill, not the diff. A PR template, a checklist, and a real review of the reinforced skill from this issue.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Building the AI Dark Factory - start here with Issue #1
Project repository
Reinforced Gherkin skill v2
Session findings — Issue #11

This article was written with the assistance of AI tools.