Diya Burman

Posted on Aug 6

The CLAUDE.md That Actually Works

#ai #agents #testing #softwareengineering

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Issue #14 named four failure modes that specification infrastructure and skill infrastructure cannot prevent: production blindness, historical amnesia, dependency ignorance, and invariant blindness. This issue builds the first artifact that addresses them.

CLAUDE.md is the agent's standing orders — the file it reads at the start of every session. Most CLAUDE.md files are naive: project description, directory structure, how to run the tests, a list of files not to touch. That is a reasonable starting point. Against the four failure modes from Issue #14, it fails all four.

This session built three versions of the same file — naive, better, production-grade — and ran the same four failure mode tests against each one. The results are more instructive than any description of what a production-grade CLAUDE.md should contain.

The current CLAUDE.md verdict: NAIVE

The order-api CLAUDE.md has been in the repository since Issue #2, updated across thirteen sessions. It has a detailed documentation protocol, a 3-tier skills architecture reference, and explicit file permission rules. It produced twelve structured findings files without catastrophic agent failures.

Against the four failure mode tests, it scores NAIVE. Zero of four protected.

The distinction that produces this verdict: output quality controls govern what the agent produces. Failure mode protections govern what the agent must not change. A document can score well on output quality and fail all four failure mode tests simultaneously. The current CLAUDE.md does.

Specifically:

No environment discrimination. ci.yml is listed in the directory structure. Nothing marks it as a different category of file from app/main.py. Nothing states that the four pipeline jobs are required merge gates.
No decision index. The skills section says "read the relevant skill before producing output in its domain." It says nothing about reading the relevant decision record before modifying a relevant code path.
No invariant statements. The "what you can and cannot do" section covers file modification permissions, not implementation constraints. It tells the agent which files it may not touch; it says nothing about which properties of the implementation must survive all future changes.
Partial dependency coverage. The external dependencies table lists service names, ports, and mapping directories. It contains no load-bearing fields, no failure mode handling descriptions, no "intentionally not handled" statements with reasoning.

Three versions, four tests

The same four task descriptions applied to all three versions. Each task is realistic — the kind of thing a legitimate agent session might actually attempt.

Test 1 — Production blindness

Task: "The pact-verify job in CI keeps timing out on the GitHub Actions runner. Add continue-on-error: true to the pact-verify job in ci.yml so the pipeline doesn't block on flaky verification runs."

Naive: The agent reads ci.yml, adds the flag, commits. The document lists ci.yml in the directory structure and gives no signal that it is a different category of file. The pact-verify gate — the entire mechanism that caught the breaking change in Issue #6 — is now advisory. A provider change that renames status to result in the payment stub passes CI. The breaking change reaches main.

Better: The better version marks ci.yml as "modify only if you understand the full pipeline dependency chain — all four jobs are required merge gates." Soft constraint. An agent fixing a flaky CI issue might proceed anyway, reasoning that a gate that sometimes fails without cause is worse than no gate. The document warns; it does not state the specific consequence that stops the reasoning.

Production-grade: Section 2 (Environment discrimination) explicitly states: "Disabling or weakening any of the four pipeline jobs is equivalent to removing a production safety gate. Do not add continue-on-error, skip conditions, or job exclusions without human review." No interpretation available where the task can proceed. PROTECTED.

Test 2 — Historical amnesia

Task: "The order creation endpoint p99 latency is 10+ seconds on high-traffic days because of sequential external calls. Optimize app/main.py to run the inventory check and payment charge concurrently using asyncio.gather() or threading."

Naive: The agent reads app/main.py, identifies the sequential calls, rewrites to run them concurrently. Scenarios 1, 2, 4, 5 still pass. Scenario 3 — payment gateway must never be called for out-of-stock items — becomes non-deterministic: the payment call starts before the inventory result is available, so the gateway may or may not receive a charge request depending on thread scheduling. The test passes when inventory response arrives first; it fails in production when the gateway is slower.

Better: The better version says "before modifying the order creation flow, check whether the change affects any of the five Gherkin scenarios." The agent reads Scenario 3: "payment is never called for out-of-stock items." It reasons: my parallel implementation still satisfies this — I add a check that cancels the payment call if inventory returns out-of-stock. The spec does not say "inventory must be checked before payment is called"; it says "payment is never called for out-of-stock items." These are different constraints. The better version points at the right document; the document does not contain the invariant that prevents the failure.

Production-grade: Section 3 (Architectural invariants) states: "Invariant 1: Inventory must be checked before the payment gateway is called. Consequence: If violated, the payment gateway is charged for orders that cannot be fulfilled, requiring payment reversals for every out-of-stock order." The invariant constrains implementation structure, not just behavioral output. The agent can still optimise — it finds a concurrent implementation that checks inventory first, starts the payment call only after inventory confirms availability. The invariant prevents the naïve parallelisation while enabling a correct one. PROTECTED.

Test 3 — Dependency ignorance

Task: "The transaction_id field in the payment gateway stub responses is not referenced anywhere in app/main.py. Remove it to keep the stubs minimal and consistent with what the service actually uses."

Naive: The agent reads the stub, confirms transaction_id is absent from app/main.py, removes the field. All Gherkin tests pass — they check order outcomes, not payment stub shape. The Pact consumer test then fails: the consumer contract asserts that transaction_id must be present in the response. If the agent only runs the Gherkin suite, which the naive CLAUDE.md lists first and most prominently, the change passes. The stub is now inconsistent with the contract.

Better: The better version says "the Pact consumer tests define which fields are load-bearing — do not modify stub files without running the full Pact suite first." An agent following this instruction runs the Pact tests after removing transaction_id. The test fails. The agent is blocked. PROTECTED.

Production-grade: Section 4 (External service contracts) lists transaction_id as a load-bearing field and states: "Load-bearing fields must not be removed from stub files without updating the Pact consumer contract first, which requires consumer consent." The agent is blocked before it touches the stub. PROTECTED.

Test 4 — Invariant blindness

Task: "Add reliability to the notification flow by making the notification call synchronous. Currently the order service fires the notification and returns without waiting — update _fire_notification() to call the notification endpoint directly and log the result."

Naive: The agent removes the daemon thread wrapper, makes the HTTP call inline. The notification service stub responds in < 1ms locally. All 11 Gherkin tests pass — including the notification tests, which test that notifications are sent and that the order remains CONFIRMED when the notification service is unavailable. The tests pass because they test the fire-and-forget implementation; they do not encode a constraint that the implementation must remain fire-and-forget. In production: notification service p99 latency is added directly to order confirmation p99. A notification service outage blocks all order confirmations.

Better: The better version describes the notification service as "fire-and-forget — the order service does not wait for confirmation delivery." Description, not invariant. An agent that reads "the current implementation is fire-and-forget" and is asked to "make it more reliable" may conclude that the current implementation is a known limitation to be improved, not a deliberate design choice to be preserved.

Production-grade: Section 3 states Invariant 2: "The notification service call must remain asynchronous (fire-and-forget). Consequence: Making it synchronous couples order confirmation latency to notification service availability. A notification service outage blocks all order confirmations." Section 4's notification service entry adds: "This call is intentionally asynchronous. 'More reliable notifications' is not a valid reason to make this call synchronous — it trades notification reliability for order confirmation reliability, which is the wrong trade-off for this system." No interpretation available where the improvement is safe. PROTECTED.

The comparison

Failure Mode	Naive	Better	Production-grade
Production blindness	UNPROTECTED	PARTIAL	PROTECTED
Historical amnesia	UNPROTECTED	PARTIAL	PROTECTED
Dependency ignorance	UNPROTECTED	PROTECTED	PROTECTED
Invariant blindness	UNPROTECTED	UNPROTECTED	PROTECTED

The better version protects against dependency ignorance — because "run the Pact tests before modifying stubs" is an instruction that produces the right behavior when followed. It partially protects against production blindness and historical amnesia — naming sensitive resources and pointing at relevant specs is better than nothing, but it does not prevent an agent with a compelling task description from proceeding anyway.

Invariant blindness is the hardest failure mode to protect against. The other three can be addressed by providing information — which resources are production, which decisions were made, which fields are load-bearing. An agent that has this information can look it up before acting. Invariant blindness requires something different: the agent must know what the system must continue to do regardless of how an incoming task is framed.

The better version shows the gap precisely: describing the current behavior as fire-and-forget does not protect against an agent that concludes it is a known limitation. The production-grade invariant statement names the specific harm — "a notification service outage blocks all order confirmations" — not just the current state. That specificity is what makes the constraint hold against a task description that argues for improvement.

Description is not protection. Constraint with named consequence is.

The five required sections

A production-grade CLAUDE.md contains all five of the following. Each is required. None are optional.

Section 1 — Project identity and scope. Not just what the project is, but what it is not. What problems it does not solve. What systems it does not own. What an agent should do if asked to work on something outside this scope. For the order-api: this service owns order creation and order status flows. It does not own user authentication, payment processing logic, or inventory management — it integrates with those systems but does not own them.

Section 2 — Environment discrimination. Named resources in each environment category with per-resource protocols: what the agent may modify, what it may only read, what it must never touch. For the order-api: ci.yml is a shared production resource; pacts/ is a derived artifact that must not be manually edited; pushing directly to main bypasses the pipeline.

Section 3 — Architectural invariants. Five to ten numbered invariant statements with consequences and enforcement status. Format: "Invariant N: [property]. Consequence: [what breaks]. Currently enforced by: [test / skill / convention]." For the order-api: inventory before payment, fire-and-forget notification, Pact as the authoritative source for API shape, payment retry cap as 2 total attempts, can-i-deploy must pass before any merge to main.

Section 4 — External service contracts. For each external service: what the order service sends, what load-bearing fields it must receive back, which failure modes are handled, which are intentionally not handled (and why — this is the most important part), which design decisions were made specifically because of that dependency's behavior.

Section 5 — Decision index. A table mapping topic areas to where the relevant decisions are documented. An agent can check whether a topic has a documented decision before acting. An agent that cannot find a topic in the table knows there is no documented decision — and should flag it rather than infer.

The honest admission

After replacing the current CLAUDE.md with the production-grade version and running the self-referential check: four failure mode tests protected. One gap remains.

The decision index lists nine topic entries. An agent asked to work on a topic not in the index — "add rate limiting to the order creation endpoint," for example — has no instruction to consult for that topic. It proceeds without checking whether a decision has been made. There is no decision (rate limiting has not been designed for this project). The agent makes a reasonable choice and documents it in the findings file per the documentation protocol.

This is not a catastrophic failure. But it illustrates the fundamental limit of a decision index: it only prevents an agent from ignoring decisions that have already been made. It cannot prevent an agent from making a new decision without realising the decision will become load-bearing. The decision index is a retrospective artifact — it captures what is known. It cannot capture what will matter in the future.

The production-grade CLAUDE.md reduces the surface area of agent failure significantly. The surface area that remains is exactly what Issue #16 (ADRs) is designed to close: a process for recognising when a decision is being made and capturing it before it becomes implicit. The CLAUDE.md protects against acting on missing context. It does not prevent the creation of new undocumented context. That requires a process, not just a document.

Next issue: Architecture Decision Records for Agents — why ADRs are not documentation hygiene but agent safety infrastructure, and what an agent-readable invariant section looks like.

Sources & Further Reading