DEV Community: Diya Burman

Prompts Are Disposable. Skills Are Infrastructure.

Diya Burman — Mon, 29 Jun 2026 13:30:00 +0000

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Layer 1 is complete. Eight issues, a working order management API, Pact contracts, a CI/CD pipeline, and a spec audit framework. The specification layer is done.

Layer 2 starts here. And it begins with a question that sounds simple until you think about it: why do you keep rewriting the same prompts?

The problem with copying prompts

If you've been using AI seriously for more than a few weeks, you have a collection of prompts that work. You've refined them. You copy them between sessions. You paste them into Claude Code at the start of a task and the agent does the right thing.

That feels like a system. It isn't.

Here's what copying a prompt actually does: it copies the words. It doesn't copy the contract. The agent reads the words, interprets them in the context of this session, and makes a series of decisions that aren't in the prompt. Different sessions, different context, different decisions — even with the same words. You won't notice until two agents produce incompatible outputs from the same prompt and you have to figure out which one is right.

A skill is different. A skill specifies what to produce, not just what to consider. It has a version, an output contract, and a routing signal. It gets better over time and the improvements persist. It's the difference between a note you wrote to yourself and infrastructure your whole team — human and agent — can depend on.

Finding the right candidate

I reviewed the entire order-api project to find the best prompt-to-skill conversion candidate. Three instructions surfaced:

The test-run verification sequence (pytest tests/steps/ -v && pytest tests/pact/ -v && python scripts/can_i_deploy.py) appears in every session. Rejected — it's a procedure, not a judgment call. Any agent can run three commands.

The findings file protocol appears in CLAUDE.md and has been followed since Issue #3. Rejected — it describes a format and cadence, not a methodology.

The Gherkin scenario quality evaluation — the methodology for deciding whether a scenario is well-formed before accepting or writing it — appeared across Issues #5, #7, and #8. Every time, the agent re-derived the same judgment framework from scratch. This is the winner.

Why: it encodes judgment, not procedure. Whether a step is UNDERSPECIFIED or LEAKY ABSTRACTION is a reasoning call. Its output drives everything downstream — every implementation session depends on the scenarios being well-formed. A bad scenario written in a planning session becomes broken step definitions two sessions later.

And here's the uncomfortable detail: the timeout ambiguity that was fixed in Issue #8 — And the response is returned within 12 seconds — was introduced in Issue #2. Three sessions inherited it silently before it was caught. A quality evaluation skill running in Issue #2 would have caught it before it was ever committed.

The prompt version — and what it gets wrong

Here's the current prompt as it would be pasted into a session:

Before writing or accepting a Gherkin scenario, check that it is well-formed. A well-formed scenario describes behavior from the caller's perspective, not from the implementation. Each step should be specific enough that only one implementation can satisfy it. Check for: vague quantities, counts that could be read as total or additional, time bounds without a start anchor, mechanism claims without the mechanism, and internal field names leaking into the spec. If the scenario has these problems, rewrite it before proceeding.

Four weaknesses:

No output format. The prompt implies the agent will either rewrite or proceed — but doesn't say what to return. A list of problems? An annotated version? The corrected Gherkin? Two agents will produce two different output shapes. A downstream agent consuming either has to parse whichever shape it receives.

No taxonomy. "Internal field names" means something specific — but the prompt doesn't define what internal means vs external. Two agents draw that line differently.

No handling for partial problems. "Rewrite before proceeding" — does that mean rewrite every step, or only the problematic ones?

Context-dependence. The prompt was written for reviewing an existing scenario. Used in a planning context ("write a new scenario for DELETE /orders/{id}"), an agent either ignores the check or applies it mid-draft. The two strategies produce different levels of revision.

The fundamental problem: the prompt describes what to look for but not what to produce. The agent invents the output format. When the output format is invented, it is never the same twice.

The demonstration

Same input. Same endpoint. Prompt version first, skill version second.

The task: Write a Gherkin scenario for DELETE /orders/{order_id} — cancel a confirmed order. The order can only be cancelled if payment has not been captured yet.

Task A — prompt version output:

Scenario: Cancel a confirmed order before payment capture
  Given a confirmed order exists with order ID "order-123"
  And payment has not been captured for the order
  When the user sends DELETE /orders/order-123
  Then the order is cancelled
  And the response status code is 200
  And the response includes a confirmation message

Six implicit decisions the agent made that weren't in the prompt:

"confirmed order exists" — method of setup not specified. Seed directly or call POST /orders? Left open.
"payment has not been captured" — mechanism not specified. Mock server state? Flag on the order?
"the order is cancelled" — mechanism language. What field changes? What value? Not specified.
"confirmation message" — any text in any field satisfies this. Undefined.
200 vs 204 — the agent chose 200 over 204 (no body). Undocumented judgment call.
No failure scenario — the prompt mentioned "can only be cancelled if payment has not been captured yet" but the agent wrote only the success case.

Total implicit decisions: 6. All silent.

Task B — skill version output:

Scenario: Confirmed order is cancelled when payment has not yet been captured
  Given an order was created via POST /orders with order ID "order-del-001" and status "CONFIRMED"
  And the payment gateway has not captured payment for order "order-del-001"
  When the client sends DELETE /orders/order-del-001
  Then the response HTTP status is 200
  And the response body contains a "status" field with value "CANCELLED"
  And the response body contains an "order_id" field with value "order-del-001"
  And the payment gateway receives no capture or charge requests for order "order-del-001"

Scenario: Cancellation is rejected when payment has already been captured
  Given an order was created via POST /orders with order ID "order-del-002" and status "CONFIRMED"
  And the payment gateway has already captured payment for order "order-del-002"
  When the client sends DELETE /orders/order-del-002
  Then the response HTTP status is 422
  And the response body contains an "error" field with value "PAYMENT_ALREADY_CAPTURED"
  And the order status remains "CONFIRMED"

Two implicit decisions — both surfaced explicitly:

422 vs 409 — the skill's output contract requires documenting assumptions, so this was flagged as a comment rather than silently embedded.
"status" vs "cancellation_status" — the skill's LEAKY ABSTRACTION check prevented using an implementation-facing field name.

Total implicit decisions: 2. Both visible.

The diff

What changed	Classification
"confirmed order exists" → "created via POST /orders with status CONFIRMED"	SKILL CONSTRAINT
"payment has not been captured" → "the payment gateway has not captured payment for order-del-001"	SKILL CONSTRAINT
"the order is cancelled" → HTTP status + "status" field with value "CANCELLED"	QUALITY DELTA
"confirmation message" → specific field name and value	QUALITY DELTA
(absent) → "payment gateway receives no capture requests"	SKILL CONSTRAINT
(absent) → full second scenario for failure case	QUALITY DELTA

Six meaningful differences. Three skill constraints, three quality deltas, six prompt ambiguities eliminated.

The three properties skills have that prompts don't

1. Version control

A prompt has no version. When you improve it, you copy the new text into the next session. The old version exists in your clipboard history or a chat transcript from three weeks ago. You cannot diff it. You cannot pin a session to it. You cannot see what changed between the prompt that worked and the prompt that produced the wrong output.

The Gherkin quality skill lives in docs/skills/gherkin-scenario-quality.md. When Issue #8 added the IMPLICIT FLOW debt class, the skill gets a one-line update:

+| IMPLICIT FLOW | A step that implies a follow-up flow that is not specced anywhere |

Every session after that commit uses the updated skill. Every session before it used the previous version. git blame tells you exactly when IMPLICIT FLOW was added and which issue prompted it. With a prompt, "skill v1.1" means nothing. There is only "the prompt I'm using today."

2. Output contract

The skill specifies exactly what it must return:

One or more complete Gherkin scenarios in Given/When/Then format
All Then clauses must assert a field name AND a value — not just presence
All counts must use "exactly N" or "no more than N total" — never "N times"
All time bounds must include a start anchor
Each external service in a Given clause must be named explicitly
Assumptions not in the input must appear as # Assumption: comments

The downstream dependency is the step definition author. When tests/steps/test_order_creation.py implements And the payment gateway received exactly one charge request — "exactly one", "charge request", "payment gateway" are all actionable. When it implements "And the response includes a confirmation message" — the author must invent an assertion. That invention is where test coverage becomes unreliable.

The output contract is the interface between the agent that writes scenarios and the agent that implements from them.

3. Routing signal description

The skill's description line:

Evaluate and produce well-formed Gherkin scenarios for the order-api project using the five-question debt diagnostic and output contract.

It names the artifact type, the project, the method, and the output. An agent knows exactly when to use this skill and what it will receive.

A bad description for the same skill:

Help with writing tests and checking scenarios for the project.

"Tests" matches pytest, Pact contracts, unit tests, and Gherkin. "The project" matches any repo. No methodology named means two agents doing "help with writing tests" produce incompatible outputs — which is exactly the problem the skill exists to solve.

The answer

If both the prompt and the skill produce output that works, the difference is this:

The prompt produces output that passes today's tests. The skill produces output that a different agent can implement tomorrow without making any decisions you didn't make.

That's why copying prompts isn't enough. The words travel. The contract doesn't.

Next issue: The 3-Tier Skill Architecture in Practice — mapping your skills to the right tier and why Tier 2 is where individual expertise becomes organizational leverage.

Sources & Further Reading

Nate B. Jones — Agent-First Skills Architecture · natebjones.com
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Project repository
Gherkin quality skill
Session findings — Issue #9

This article was written with the assistance of AI tools.

Spec Debt Doesn't Disappear When You Fix It. It Migrates.

Diya Burman — Mon, 22 Jun 2026 13:30:00 +0000

Preface

Issue #7 ended with seven spec debt items documented in a project that had been built carefully for seven issues. Every item was passing its tests. None of them announced themselves. They were found by asking a different question: not "does this pass?" but "what would a second agent build from this step?"

Issue #8 fixes all seven — and builds the tool that found them into something reusable.

The seven fixes

Working through each item one at a time, running the test suite after every individual fix. Not batching them. The discipline matters — if a fix breaks something, you want to know which fix broke it.

Fix 1 — Timeout measurement ambiguity

# Before
And the response is returned within 12 seconds

# After
And the response is returned within 12 seconds of the order being submitted

"Of the order being submitted" anchors the clock to client-side HTTP request dispatch — the same moment time.time() is captured in the step definition. Without this anchor, a second implementation could measure from server receipt, from the last retry attempt, or from when the response body is fully read. All three produce different numbers under load.

Fix 2 — "Retried" vs "total attempts"

# Before
And the payment gateway is not retried more than 2 times

# After
And the payment gateway receives no more than 2 charge requests total

"Retried 2 times" has two valid English readings: 2 retries meaning 3 total requests, or retried up to 2 times meaning 2 total. "No more than 2 charge requests total" counts requests, not retries, and the word "total" makes clear the initial attempt is included. This also changed the assertion in the step definition — from trusting the response body's retry_count field to checking the actual call count at the mock server. Stronger assertion, same outcome.

Fix 3 — "Released" without mechanism

# Before
And the inventory reservation is released

# After
And the inventory service receives a reservation release request for SHOE-RED-42 and BELT-BRN-M

"Released" says what happened but not how, and not for which items. The rewrite names the items and specifies that a request is sent to the inventory service. This fix also revealed a gap: the current implementation signals release via a response body field (inventory_released: true) rather than a separate API call to the inventory service. The spec now describes the intended behaviour. The implementation doesn't fully match it yet. That's a future issue — but the gap is now visible rather than hidden.

Fix 4 — "Explicit user action" — removed entirely

# Before
And no order is confirmed without explicit user action

# After
(step removed)

This step implies a follow-up confirmation flow (POST /orders/{id}/confirm or equivalent) that does not exist anywhere in the codebase. It passes trivially because no order is confirmed in the partial availability scenario — not because the confirmation flow was implemented. A spec step that passes for the wrong reason is not a safety net. It is a false guarantee. If the confirmation flow is built in a future issue, a new scenario should specify it precisely. Leaving this step in place would invite an agent to invent an unspecced endpoint.

Fix 5 — Presence without value assertions

The order_status_bad.feature timestamp step was asserting only that a field exists and is a non-empty string. Tightened to assert the field name, the value, and the type explicitly. Kept conservative — order_status_bad.feature is a pedagogical artifact and shouldn't be converted into a good spec, which would defeat its purpose in the newsletter.

Fix 6 — "An order exists" without specifying how

# Before
Given an order was successfully placed and confirmed with order ID "aaa00000-..."

# After
Given an order was created via POST /orders and confirmed with order ID "aaa00000-..."

"Successfully placed and confirmed" describes the outcome but not the mechanism. "Created via POST /orders" makes explicit that a real creation flow is expected. The step definition currently seeds the order directly into the in-memory store — a shortcut. The rewrite creates a documented gap between spec intent and step implementation. Visible gap, not hidden one.

Fix 7 — "Correct" without definition

# Before
And the notification contains the correct order id and total

# After
And the notification request body contains order_id "order-abc-123" and total 134.97

"Correct" is relative to context that may not be available to the reader. The rewrite hardcodes the expected values established in the When clause. Two agents reading the original step would both implement something that checks the notification body — but one might compare against the When-clause values, another might check against a computed total, a third might only verify field presence. The rewrite removes all three interpretations.

This fix also caught something the stub had been hiding: the notification mock was returning "mock-notif-001" as a notification id. Not a UUID. The format assertion caught it immediately. This is exactly the value of adding concrete assertions — it surfaces stub data that was never valid but was never checked.

The audit framework

After fixing all seven items, I built the diagnostic tool into a standalone document: docs/spec-audit-framework.md. The full document is in the repo. Here's the core of it.

Five questions — ask them for every scenario in every feature file:

Q1: Who owns this scenario?
Can you name the team, service, or domain this scenario belongs to? If the answer includes "and also", the scenario is in the wrong file.

Q2: What decisions does this scenario leave open?
For every Given, When, and Then clause: could two agents build different implementations that both pass? If yes, the step is underspecified.

Q3: Are all terms defined within the file?
Every noun that is not a standard HTTP concept or a primitive type should be defined in the scenario or a Background clause. If understanding a term requires reading another file or asking a colleague, it is spec debt.

Q4: Does this scenario describe behaviour or implementation?
Steps should describe what the system does from the caller's perspective. Any step that references internal concepts — database field names, function names, internal status codes — is leaking implementation into the spec.

Q5: What does this scenario NOT say that it should?
List the edge cases, error states, and boundary conditions the scenario implies but does not specify. Each one is a silent assumption waiting to become a production incident.

Six debt classes:

Class	What it looks like
UNDERSPECIFIED	Step present but leaves a decision open
MIXED CONCERN	Scenario covers more than one service domain
UNDEFINED TERM	A noun used without being defined
AMBIGUOUS COUNT	A quantity with two valid interpretations
IMPLICIT FLOW	Implies a follow-up flow that isn't specced
LEAKY ABSTRACTION	References implementation details

What the framework found that the manual audit missed

Applying the five questions to all four fixed feature files surfaced one item the Issue #7 manual audit didn't catch.

In order_status_good.feature, the Given clause now reads "created via POST /orders" — the fixed version from this session. Q4 flagged it for a different reason than the original audit: the step definition still seeds the order directly into the in-memory store. The spec text is precise. The implementation of the spec takes a shortcut.

The manual audit looked at feature file text. The framework applies Q4 to step definitions as well — and a step definition that silently does something different from what the spec says is spec debt, even if the test passes.

This distinction matters: spec debt can migrate from the feature file into the step definition. You fix the scenario, tighten the language, run the tests — green. But the step definition now implements a shortcut that contradicts the precise step text. The debt moved, it didn't disappear.

The scorecard — after all fixes

Applied the framework to all four non-pedagogical feature files:

order_creation.feature — 5 scenarios, 1 debt item remaining (LEAKY ABSTRACTION at step definition level — inventory release mechanism gap from Fix 3)

order_status_good.feature — 2 scenarios, 1 debt item remaining (LEAKY ABSTRACTION — step definition seeds order directly rather than via POST /orders)

notification_service.feature — 2 scenarios, 0 debt items

order_status_bad.feature — kept as pedagogical artifact, not audited for debt

Debt density after fixes: 0.22 items per scenario. Both remaining items are LEAKY ABSTRACTION at the step definition level. Zero AMBIGUOUS COUNT or IMPLICIT FLOW items remain — the two highest-risk classes.

The uncomfortable answer

After fixing seven spec debt items and applying a structured audit framework to a project that has been built carefully for eight issues, two debt items remain. Both were introduced by the same sessions that fixed other debt — a precise spec step was written, and the implementation of that step took a shortcut.

Spec debt is not eliminated by fixing debt. It migrates.

The practical conclusion: treat step definitions as part of the spec surface, not just as test harness code. A step definition that silently does something different from what the spec says is spec debt, even if the test passes. The audit framework catches both — but only if you apply Q4 to the step definitions as well as the feature text.

The other finding worth naming: notification_service.feature scored zero debt items. It was written after eight issues of accumulating lessons about what the previous files got wrong. The absence of debt is not accidental — it's the result of knowing what bad specs look like before writing the next one.

The best time to write a spec is after you've written a few bad ones. Auditing retroactively and fixing forward is the realistic path. Not "write it right the first time."

Next issue: Prompts Are Disposable. Skills Are Infrastructure — the conceptual shift from session-level prompts to versioned, reusable skill definitions. Layer 2 begins.

Sources & Further Reading

Cucumber + Gherkin documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Spec audit framework
Session findings — Issue #8

This article was written with the assistance of AI tools.

Your Spec Files Are Lying to You. Mine Were Too.

Diya Burman — Mon, 15 Jun 2026 17:30:55 +0000

Preface

Every issue so far has worked with one service and one spec file. Issue #7 changes that. A second service enters the picture — a notification service that the order service calls after a confirmed payment — and with it comes the question that every growing system eventually forces: where do spec file boundaries go?

The answer turns out to matter more than it looks. And the audit at the end of this issue found seven spec debt items in files we've been running since Issue #2. All passing. All carrying risk.

The notification service — and a design decision that has spec implications

The new service is minimal: POST /notifications/order-confirmed accepts an order id, user id, and total, and returns a notification id and a QUEUED status. Simple enough. The interesting part is how the order service calls it.

The call is fire-and-forget.

When an order is confirmed, the order service starts a daemon thread, fires the notification request, and returns the CONFIRMED response immediately — without waiting for the notification to succeed. If the notification service is down, slow, or returning errors, the order is still confirmed. The customer gets their confirmation. The notification may or may not arrive.

This is a deliberate design decision. The order service owns the transaction. The notification service owns delivery. Coupling the order confirmation response to notification delivery would mean a flaky notification service could block order creation — which is a much worse failure mode than a missed notification.

But the decision has a direct spec implication: any scenario that asserts Then the order status is "CONFIRMED" must remain true regardless of what the notification service does. The spec cannot simultaneously require CONFIRMED and make CONFIRMED depend on notification success. That would be a hidden coupling — the spec would look independent but the implementation would not be.

This is the kind of architectural decision that should be in the spec before it's in the code. Once it's in the code it becomes folklore.

The wrong way first: one big spec file

Before doing it right I did it wrong deliberately. I added two notification scenarios to the bottom of order_creation.feature — the existing file that's been covering order creation since Issue #2.

All 7 tests passed. Green across the board. pytest has no opinion about spec architecture.

The problems are structural, not functional:

Mixed ownership. order_creation.feature line 1 says Feature: Order Creation. By line 48 it's testing notification delivery. If the notification team changes their contract — say, adding a channel field to the request — they have to open order_creation.feature to update it. That file is not theirs. The filename, the feature declaration, and the existing scenarios all signal "this belongs to the order team." The notification scenarios are squatters.

The growing file problem. At 5 scenarios the file is readable. At 7 it starts to smell. Extrapolate to a real system: 10 downstream services, 5–10 scenarios each, all appended to the originating feature file because each was "triggered by" an order creation event. The file becomes a catch-all that nobody owns and everybody edits. Ownership dissolves into "whoever last touched it."

The agent routing problem. When an agent is handed order_creation.feature to build against, it must now implement both order logic and notification logic. It cannot know from the file whether the notification call belongs in POST /orders or in a separate endpoint. It will make a decision — probably the wrong one — and that decision will be baked into the implementation before anyone notices.

Spec debt seed. The scenario "Order confirmation succeeds even if notification fails" uses the step "the notification service is unavailable" without defining what unavailable means. TCP connection refused? 503? A 30-second hang? Each is a different failure mode with different implications for retry logic. An agent will pick one interpretation silently. Two agents will pick different ones. Both implementations will pass the spec. This is spec debt: it forms quietly, passes its tests, and surfaces as a production incident months later.

The right way: bounded spec files

After documenting what was wrong, I moved the notification scenarios into their own file: tests/features/notification_service.feature. Rewrote both scenarios to:

Precisely define "unavailable" as 503 Service Unavailable — not a timeout, not a connection refused, not an ambiguous network failure
Describe the notification contract from the notification service's perspective
Make the file self-contained — a notification service team reading it wouldn't need to open order_creation.feature to understand it

The result:

order_creation.feature: 5 scenarios, all about order creation. No references to notifications.
notification_service.feature: 2 scenarios, all about notification delivery behaviour.

The file boundary is now a contract boundary. They can be versioned, owned, and handed to different agents independently.

Bounded spec files are not a tidiness preference. They are a precision tool for multi-agent systems. When a spec file is bounded to one service, an agent can be assigned exactly that file and nothing else. It builds one surface, tests one contract, returns. When the spec bleeds across services, the agent must make decisions about service ownership that were never written down. Those decisions accumulate as hidden assumptions in the implementation.

The spec debt audit

With the bounded file structure in place, I audited all four feature files in the project for spec debt — places where the spec passes its tests but leaves decisions that should have been made explicitly.

Seven items. All passing. All carrying risk.

1. Ambiguous timeout measurement

File: order_creation.feature — Scenario: payment gateway times out
Step: And the response is returned within 12 seconds

From when? The client sends the request? The server receives it? The last retry fires? Two agents will instrument this differently and both will pass. "Within 12 seconds of the order being submitted" — defining "submitted" as the moment the HTTP request body is sent — removes the ambiguity.

2. "Retried" vs "total attempts"

File: order_creation.feature — Scenario: payment gateway times out
Step: And the payment gateway is not retried more than 2 times

Does this mean 2 total attempts (1 original + 1 retry) or 2 retries on top of the original (3 total)? The English is genuinely ambiguous. An agent will pick one. The test will pass. The production system will behave differently than intended.

Fix: And the payment gateway receives no more than 2 charge requests total — "requests total" removes all ambiguity about whether the first attempt counts.

3. "Released" is not a mechanism

File: order_creation.feature — Scenario: payment declined
Step: And the inventory reservation is released

"Released" is not defined. Does the inventory service receive a DELETE? A POST to a release endpoint? Does a TTL fire? An agent will implement whichever mechanism seems natural. Two agents will produce incompatible implementations that both pass the spec.

Fix: Name the items and the mechanism: And the inventory service receives a reservation release request for SHOE-RED-42 and BELT-BRN-M.

4. "Explicit user action" describes a flow that doesn't exist

File: order_creation.feature — Scenario: partial availability
Step: And no order is confirmed without explicit user action

"Explicit user action" is not defined anywhere in the spec. A second API call? A UI confirmation? A webhook? This step passes trivially because no order is confirmed — the negative condition is true by absence. But it implies a follow-up confirmation flow that was never built, never specced, and never reviewed. If a future agent reads this step and builds a confirmation flow to satisfy it, it will invent something that was never intended.

Fix: Remove it if the follow-up flow is out of scope. Or replace it with a concrete step: And a subsequent POST to /orders/{order_id}/confirm is required to complete the order.

5. Presence without value

File: order_status_bad.feature
Step: field-name assertions without value or type assertions

Asserting that a field exists only catches absence — not incorrect presence. An agent can return {"status": null} and pass. The spec catches the wrong thing.

Fix: Assert the full expected shape with explicit values rather than just field names.

6. "An order exists" doesn't say how

File: order_status_good.feature
Step: Given an order exists with status "CONFIRMED"

"An order exists" doesn't specify how it got there — full creation flow, or directly seeded into the store. The two methods produce different side effects. An agent building a test harness may seed the order directly, bypassing the creation flow entirely, which means the status endpoint tests never verify that a real confirmed order is actually readable via the API.

Fix: Given a previously confirmed order created via POST /orders with id "{order_id}" — or explicitly state that direct seeding is acceptable.

7. "Correct" is relative

File: notification_service.feature
Step: And the notification contains the correct order id and total

"Correct" compared to what? If the order total is computed, two agents may compute it differently and both pass "correct" against their own computation.

Fix: Hardcode the expected value: And the notification request body contains order_id matching the confirmed order and total of 134.97.

Why all seven of these matter even though they're all green

Every item in that audit passes its test. That is the point.

Spec debt is not visible in a green CI run. It is visible only when you ask: what would a second agent build from this spec? The step "the payment gateway is not retried more than 2 times" has been in the codebase since Issue #2. It has passed every run. But it encodes an ambiguity that will be resolved differently by every agent that implements it fresh. The "no order is confirmed without explicit user action" step describes a flow that does not exist anywhere in the codebase. It passes because the negative condition is trivially true.

If a future agent reads that step and builds a confirmation flow to satisfy it, it will build something that was never specced, never reviewed, and never integrated. The spec invited it. The tests blessed it. Nobody noticed.

This is the exact failure mode that makes AI-assisted development unreliable at scale. Specs that look precise, pass their tests, and silently invite incompatible implementations. The debt doesn't announce itself. It compounds.

Where the project stands

Fifteen tests passing across four bounded feature files. The notification service is integrated. The Pact contracts — which existed before this session — remain unbroken because the notification call happens after the transaction completes. Adding a new service boundary didn't require touching existing contracts.

Seven spec debt items documented. None fixed yet. The fixes are the next issue.

Next issue: The Spec Audit — applying the debt framework to a real existing service and building the diagnostic tool readers can use on their own codebases.

Sources & Further Reading

Cucumber + Gherkin documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #7

This article was written with the assistance of AI tools.

My Tests Passed. My Pipeline Caught What They Missed.

Diya Burman — Sat, 13 Jun 2026 18:59:27 +0000

A Level 5 Engineer — Issue #6

Preface

Five issues in, everything we've built lives on one machine. The Gherkin scenarios, the WireMock stubs, the Pact contracts, the can-i-deploy script — all of it runs locally, passes locally, and means nothing the moment someone else touches the codebase.

Issue #6 fixes that. A GitHub Actions pipeline now runs on every push, executes the full specification stack in dependency order, and blocks merges to main if anything breaks. The pipeline is the guardrail. From this point on, a broken contract or a failing scenario cannot reach main undetected.

Getting there took ninety minutes and two interventions I didn't plan for. Both are worth documenting.

Before the YAML: deciding what "green" means

The first thing Claude Code did before touching any pipeline config was run the full test suite to establish a baseline. The instruction was explicit: everything must pass before a single line of YAML gets written.

It found a failure immediately — and it wasn't from the breaking change experiment. It was from Issue #5.

The bad-spec test (test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order) was still asserting db_status in the response body. That was intentional in Issue #5 — the failure was the finding. The session ended with it red because the point was to show what bad specs produce. But on main, with CI incoming, that means the pipeline would have been red on day one before a single feature change.

The fix was adding backward-compat aliases to the response:

return {
    "order_id": order_id,
    "status": order["db_status"],            # good spec field
    "db_status": order["db_status"],         # bad spec alias — keeps Issue #5 test passing
    "placed_at": order["order_created_at"],  # good spec field
    "order_created_at": order["order_created_at"],  # bad spec alias
}

Neither test file was modified. No feature files were touched. The aliases kept both the good-spec and bad-spec tests passing against the same endpoint.

The reason this matters before the pipeline exists: a team that starts CI with a known failure trains itself to ignore red. The cost of normalising a red CI is much higher than the cost of fixing the baseline first. Claude Code made the right call and documented it before moving on.

The pipeline structure

Four jobs, in dependency order:

test → pact-consumer → pact-verify → can-i-deploy

Each job only runs if its predecessor passes. If Gherkin breaks, Pact never runs. If the consumer tests fail, verification never runs. If verification fails, can-i-deploy is skipped. The pipeline fails fast and tells you exactly which layer broke.

The artifact chain is what makes it a pipeline rather than four parallel scripts. The pact-consumer job generates the .pact files and uploads them as a GitHub Actions artifact. The pact-verify job downloads that artifact and verifies it — the same files, not freshly regenerated ones. Without this, each job would build its own consumer contract from scratch, and verification would be proving that the contract matches the code rather than proving it matches what pact-consumer actually produced.

One non-obvious piece: mock_server.py is a library module with no command-line entry point. The pipeline needed servers running as background processes. The fix was an inline Python invocation:

- name: Start mock servers
  run: |
    . .venv/bin/activate
    python -c "
    import time
    from mock_server import start_mock_server
    start_mock_server(8091, 'wiremock/payment-mappings')
    start_mock_server(8092, 'wiremock/inventory-mappings')
    time.sleep(86400)
    " &
    sleep 2

The time.sleep(86400) keeps the process alive for the duration of the job. Inelegant but functional. A proper if __name__ == "__main__" entry point with argparse is the obvious cleanup for a future session.

The first CI run — and why I had to intervene manually

The YAML was committed, pushed to main, and the pipeline ran. All three runs failed on the test job:

OSError: [Errno 98] Address already in use

Ports 8091 and 8092. Every test in test_order_creation.py errored at setup. The order status tests — which don't use the mock servers — passed fine.

Claude Code didn't catch this on its own. Here's why that's worth explaining.

When Claude Code wrote the pipeline, it was working from the codebase and its own knowledge of GitHub Actions patterns. It knew the mock servers needed to be running before pytest started, so it added an explicit start-servers step to the YAML — a reasonable decision based on the information it had. What it couldn't see was the runtime interaction between that YAML step and pytest's session-scoped fixtures, because that interaction only manifests in the CI environment, not locally.

Locally, running pytest tests/steps/ -v has always worked correctly because the session fixture starts the servers and nothing else competes. Claude Code had only ever seen local runs succeed. It had no signal that the YAML step was creating a conflict — because the conflict doesn't exist locally.

This is a fundamental limit of the "paste and walk away" approach at the boundary between local and remote environments: the agent can reason about the codebase and about CI patterns, but it can't observe the CI run itself. The failure was on GitHub. Claude Code was in a terminal. Those two things weren't connected.

I diagnosed the error from the GitHub Actions log, explained the root cause, and pasted new instructions. Claude Code fixed it in one step — removing the redundant YAML steps entirely:

# Removed from both test and pact-verify jobs:
- name: Start mock servers
  run: |
    . .venv/bin/activate
    python -c "..." &
    sleep 2

The pytest session fixtures already own server lifecycle correctly. scope="session" means pytest starts the servers once per test run and keeps them alive. The YAML step was duplicating a responsibility that was already handled. The fix wasn't a workaround — it was removing the wrong layer.

The root cause in plain terms: the YAML step and the pytest fixture both thought they were responsible for starting the servers. The port was already bound when the fixture tried to bind it again. Works on my machine. Breaks in CI. Classic.

The breaking change experiment — in the pipeline

With the pipeline green, the breaking change test ran as designed.

Branch test/breaking-change-pipeline, commit 76c0d89: renamed "status" to "result" in wiremock/payment-mappings/payment-success.json. Same change as Issue #4, now running through CI instead of local verification.

The expected failure:

a successful payment charge (FAILED)

Failures:
1) Verifying a pact between OrderService and PaymentGateway
   1.1) has a matching body
          $ -> Actual map is missing the following keys: status
   {
     "amount": 134.97,
  -  "status": "ACCEPTED",
  +  "result": "ACCEPTED",
     "transaction_id": "txn-abc-123"
   }

pact-verify fails. can-i-deploy is skipped. The merge is blocked.

And the key point from Issue #4 holds at the pipeline level: the test job — the Gherkin suite — would pass with the breaking change in place. The order creation scenarios check HTTP status codes and business outcomes. They never read pay_resp.json()["status"]. A stub returning result instead of status still returns HTTP 200. Gherkin passes. Pact catches it.

This is the division of labour. Gherkin proves the system does the right thing. Pact proves the contracts don't drift. You need both, and now both run automatically on every push.

The one step that requires the GitHub UI

Claude Code cannot configure branch protection rules — that requires the GitHub web UI or admin API. This step is non-negotiable and must be done manually:

Repo → Settings → Branches → Add branch protection rule
Branch name pattern: main
Enable Require status checks to pass before merging
Add all four status checks: test, pact-consumer, pact-verify, can-i-deploy
Enable Require branches to be up to date before merging
Save

Without this, the pipeline is advisory. A push to main can still happen even if all four jobs are red. The pipeline becomes a dashboard — it shows you the problem but doesn't stop anything. Branch protection is what turns "CI failed" from a notification into enforcement. The pipeline is only a guardrail if something stops you going around it.

The honest part

The YAML took about twenty minutes to write. The session took ninety minutes total — because the baseline fix and the port conflict ate the rest.

The instinct during the baseline audit was to skip past the known failure. It's a demo test, we know why it's there, configure CI to skip that file and move on. That would have been thirty seconds. It also would have been wrong — a pipeline with documented exceptions is a pipeline people route around.

The instinct during the port conflict was to blame the CI environment. Ubuntu runs things differently, ports work differently, it's a platform quirk. That framing would have sent the debugging in the wrong direction. The actual cause was simpler: two layers both thought they owned the same responsibility, and nobody had written down which one was actually in charge.

Both of those moments are the J-curve. Not the YAML — the discipline of not skipping and not blaming the environment. The overhead of CI is not the config file. It's every decision about what "green" actually means and who's responsible for what.

The pipeline is now real infrastructure. The breaking change can't reach main. That's worth ninety minutes.

Next issue: The Scope Problem — scaling Gherkin across a multi-service system. What happens when one spec file isn't enough, and how spec debt forms.

Sources & Further Reading

GitHub Actions documentation
Pact documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #6

This article was written with the assistance of AI tools.

The AI Built the Wrong Thing. Every Test Passed.

Diya Burman — Wed, 10 Jun 2026 02:40:42 +0000

A Level 5 Engineer — Issue #5

Preface

Every issue so far has assumed something I haven't said out loud: that the specs are good. Issue #2 wrote them carefully. Issue #3 handed them to an agent and watched it build correctly. Issue #4 proved the contracts survive provider drift.

But what happens when the spec isn't good? Not broken — Gherkin syntax is fine, tests pass, the agent builds something. Just imprecise. Vague in ways that feel precise when you're writing them.

This issue answers that question by doing the thing deliberately. I wrote bad Gherkin on purpose, handed it to the agent, watched what it built — and then rewrote the spec and did it again. The difference between the two implementations is the article.

The hardest thing about bad specs

Bad specs are hard to spot when you're writing them because they feel complete.

A scenario that references implementation details sounds like reasonable description — you wrote the implementation, so the details feel like specifics. A Given clause that feels obvious to you will be interpreted differently by every reader who hasn't seen the code. The Gherkin is syntactically correct. The tests pass. Nothing in the output signals that anything is wrong.

This is the trap. It's not that bad specs break things. It's that they don't.

The endpoint

I added a new endpoint to the order-api project: GET /orders/{order_id}/status. It returns the current status of an order and relevant metadata. Simple enough that the spec should be easy to write well. Which makes it a good target for writing it badly on purpose.

The bad specs

Two scenarios. Both syntactically valid. Both produce passing tests. Both wrong in different ways.

# BAD SPEC 1 — The leaky spec
# Problem: references internal implementation concepts (db_status, order_created_at)
# rather than describing what a caller observes. The agent uses these names literally
# in the response body, leaking storage terminology into the public API contract.

Scenario: Retrieving status for a confirmed order
  Given an order exists in the system with db_status "CONFIRMED"
  When I request GET /orders/{order_id}/status
  Then the response should contain the db_status field set to "CONFIRMED"
  And the order_created_at field should be populated from the order record

# BAD SPEC 2 — The vague Given
# Problem: "an order that has not been placed" is underspecified. The agent must
# guess what this means — a malformed ID? A well-formed UUID with no record?
# A deleted order? Each interpretation is plausible and produces different behavior.

Scenario: Retrieving status for an order that does not exist
  Given an order that has not been placed
  When I request GET /orders/{order_id}/status
  Then the response should indicate the order was not found

Both passed immediately:

tests/steps/test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order PASSED
tests/steps/test_order_status_bad.py::test_retrieving_status_for_an_order_that_does_not_exist PASSED

2 passed in 0.34s

Green. No warnings. No hint that anything is wrong.

What the agent built from the bad specs

Here's the implementation the agent produced:

@app.get("/orders/{order_id}/status")
def get_order_status(order_id: str):
    order = _orders.get(order_id)
    if order is None:
        raise HTTPException(status_code=404, detail="Order not found")
    return {
        "order_id": order_id,
        "db_status": order["db_status"],
        "order_created_at": order["order_created_at"],
    }

It satisfies the spec completely. It also made four decisions the spec never made:

Decision 1: The field is named db_status in the response.
The spec said db_status so the agent used db_status. It never questioned whether this was an internal name leaking into a public API. It satisfied the spec literally.

Decision 2: A missing order returns 404.
The spec says "indicate the order was not found." 404 is a defensible interpretation. So is 422, 403, or a 200 with a NOT_FOUND status field. The agent picked the most conventional option — but the spec never mandated it, and FastAPI's default 404 body is {"detail": "Order not found"}, not {"error": "Order not found"}. A client checking response.json()["error"] gets a KeyError.

Decision 3: The timestamp field is named order_created_at with no format requirement.
The spec says "populated from the order record." The agent chose order_created_at and returned an ISO string because that's what datetime.utcnow().isoformat() produces. The step definition checked only that the field is non-empty and a string — so any format would have passed. A Unix timestamp integer would have passed. A human-readable string like "June 2nd" would have passed.

Decision 4: The order store is in-memory.
The spec says nothing about persistence. An in-memory dict is the simplest thing that makes the tests pass. In production, orders are persisted. The in-memory store vanishes on restart and isn't shared across worker processes.

Every one of these decisions is plausible. The agent made the reasonable call every time. That's not the problem. The problem is that a different agent, given the same spec, might have made different reasonable calls — and both implementations would pass the same test suite.

The rewrite

Writing the good spec forced every decision the bad spec had silently delegated:

# GOOD SPEC 1 — Caller's perspective, not implementation's
# Fixed: field names describe what the caller observes (status, placed_at)
# not what the storage layer calls them (db_status, order_created_at).
# The format of placed_at is now a contract obligation, not an assumption.

Scenario: Confirmed order status is returned with placement timestamp
  Given a confirmed order with id "order-abc-123" exists in the system
  When I request GET /orders/order-abc-123/status
  Then the response status code is 200
  And the response body contains "order_id" equal to "order-abc-123"
  And the response body contains "status" equal to "CONFIRMED"
  And the response body contains "placed_at" as a valid ISO 8601 timestamp

# GOOD SPEC 2 — Precise Given, explicit 404 body shape
# Fixed: "a well-formed UUID with no corresponding record" is now unambiguous.
# The 404 response body shape is now a contract obligation, not a guess.

Scenario: Unknown order id returns 404 with error message
  Given no order with id "order-xyz-999" exists in the system
  When I request GET /orders/order-xyz-999/status
  Then the response status code is 404
  And the response body contains an "error" field

Notice what changed. The scenarios describe the same two situations. The intent is identical. But now every decision is in the spec rather than in the agent's interpretation of the spec.

What the agent built from the good spec

@app.get("/orders/{order_id}/status")
def get_order_status(order_id: str):
    order = _orders.get(order_id)
    if order is None:
        return JSONResponse(status_code=404, content={"error": "Order not found"})
    return {
        "order_id": order_id,
        "status": order["db_status"],
        "placed_at": order["order_created_at"],
    }

Same endpoint. Same logic. Different API.

db_status became status. order_created_at became placed_at. The 404 body now contains error not detail. The timestamp is now asserted to be ISO 8601 — not just non-empty.

These are not cosmetic differences. They are different contracts that clients build against.

The cross-run

After building from the good spec, I ran the bad-spec tests against the new implementation:

tests/steps/test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order FAILED
tests/steps/test_order_status_bad.py::test_retrieving_status_for_an_order_that_does_not_exist PASSED

E   KeyError: 'db_status'

The leaky test failed. The field db_status doesn't exist in the good implementation — it's been renamed to status, which is what a caller should see. The test that was checking for an internal name is now broken, correctly.

The vague test passed. Both implementations return a 404 for a missing order — the good implementation just happened to reach the same conclusion, but for an explicit reason this time.

That asymmetry is instructive. The vague Given produced the right answer by coincidence. The leaky Then produced the wrong field name by construction. One was luck. One was baked in.

Why this matters

Both implementations pass their own test suites. That is the trap.

If you run the bad-spec tests against the bad-spec implementation: green. If you run the good-spec tests against the good-spec implementation: green. The difference only surfaces when you cross-run — and in production, you never cross-run. You ship the bad implementation, it passes CI, and the problem lands in a client exception report six months later.

Here's the concrete difference: the bad-spec implementation returns db_status and order_created_at with no format guarantee. The good-spec implementation returns status and placed_at with a mandatory ISO 8601 format. An agent given the bad spec had no way to know that db_status was wrong — the spec said db_status. An agent given the good spec had no choice but to produce status — the spec said status.

Spec quality is not about whether tests pass. It is about how much of the implementation the spec author wrote versus how much was silently delegated to the agent. Every silent delegation is a place where two agents given the same spec produce different code — code that both passes, but disagrees on the contract.

At scale — dozens of endpoints, hundreds of scenarios — that disagreement is the system.

The practical test for a good spec

Before handing any scenario to an agent, ask one question: what decisions does this scenario leave open?

If the answer is "none — every field name, format, response code, and body shape is specified," the spec is ready. If the answer is "a few reasonable ones," those are the places where your implementation and the next agent's implementation will silently diverge.

The agent will always make reasonable decisions. That's not the problem. The problem is that reasonable is not the same as specified — and at Level 4, specified is the only thing that counts.

Next issue: Wiring the Guardrails — GitHub Actions, the Pact Broker, and the pipeline that turns contract violations into blocked merges automatically.

Sources & Further Reading

Cucumber + Gherkin documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #5

This article was written with the assistance of AI tools.

Green CI. Broken Contract. Nobody Noticed.

Diya Burman — Wed, 10 Jun 2026 02:24:34 +0000

A Level 5 Engineer — Issue #4

Preface

If you've been following along, you know where we are. Issue #2 introduced WireMock and Gherkin — write the behavioral contract before the code, stub your dependencies, run a real test suite. Issue #3 handed that spec to an AI agent and walked away. Five scenarios passed. The agent even found a bug in my code.

Everything worked. And that's exactly the problem this issue is about.

Because the WireMock stubs working perfectly is not the same thing as the real services working. The gap between those two statements is where production incidents are born.

The confidence trap

Here's the scenario nobody talks about until it happens to them.

Your order service calls a payment gateway. You've stubbed it with WireMock. Your Gherkin scenarios pass. Your agent builds against those stubs. Five for five, green across the board.

Meanwhile, the payment gateway team — a different squad, a different repo, maybe a different company entirely — ships a cleanup. They've been inconsistent about field naming across their API. status in one endpoint, result in another. They standardize. They rename status to result in the charge response. Their tests pass. They deploy.

Your tests still pass too. The stub hasn't changed. The stub will never change unless you change it.

The first time you learn about the rename is a production incident.

This is the confidence trap: a mock that can drift from the real service makes you feel safe right up until production proves you weren't. The tests are green. The contract is broken. You just don't know it yet.

What Pact does differently

WireMock is a behavioral double — it simulates a service so your tests can run in isolation. You define what it returns. You maintain it. You can make it say anything you want, which means it can silently lie about what the real service actually does.

Pact inverts the trust relationship.

Instead of you maintaining a stub that you hope reflects reality, your consumer tests declare what they need from the provider. Those declarations get written into a .pact file — a machine-readable contract. The provider then runs verification against that contract before it ships. If the provider no longer satisfies what the consumer declared, verification fails and the deploy is blocked.

The consumer defines the need. The provider proves delivery. No human has to remember to update a stub.

Building it — and what the docs didn't tell me

I added Pact to the order-api project this issue, covering both downstream dependencies — the payment gateway and the inventory service — with consumer tests matching the same five scenarios from the Gherkin feature file.

It was less smooth than I expected.

The pact-python v3 FFI surprise

Every tutorial for pact-python shows the same pattern: create a module-scoped Pact fixture, run multiple tests against it, write the pact file at the end. I wrote exactly that. The first test in each class passed. Every subsequent test failed with this:

RuntimeError: The provider state could not be specified.

No hint of what was actually wrong. After digging into the source, the root cause: pact-python 3.x is a complete rewrite backed by a Rust FFI binary. The Rust handle is consumed by the first serve() call — you cannot add new interactions to a handle after that point. The v2-style module-scoped pattern violates this constraint in a way the error message doesn't explain at all.

The fix was restructuring the consumer tests so all interactions are defined upfront before any serve() call:

# ❌ v2-style — breaks in pact-python v3
class TestPaymentConsumer:
    @pytest.fixture(scope="module")
    def pact(self):
        return Consumer("OrderService").has_pact_with(Provider("PaymentGateway"))

    def test_success(self, pact):
        pact.given("payment succeeds").upon_receiving("a charge")...
        with pact:
            # test

    def test_declined(self, pact):
        pact.given("payment declined").upon_receiving("a decline")...
        # RuntimeError — handle already consumed

# ✅ v3 correct pattern — all interactions before serve()
def test_payment_gateway_consumer():
    pact = Consumer("OrderService").has_pact_with(Provider("PaymentGateway"), ...)
    (pact
        .given("the payment gateway will accept the charge")
        .upon_receiving("a successful payment charge")
        .with_request("POST", "/payments/charge/success")
        .will_respond_with(200, body={"status": "ACCEPTED",
                                      "transaction_id": "txn-abc-123",
                                      "amount": 134.97}))
    (pact
        .given("the payment gateway will decline the charge")
        .upon_receiving("a declined payment charge")
        .with_request("POST", "/payments/charge/declined")
        .will_respond_with(402, body={"status": "DECLINED",
                                      "reason": "INSUFFICIENT_FUNDS"}))
    # ... all interactions defined ...
    with pact.serve() as srv:
        # exercise all interactions against srv.url
    pact.write_file("pacts/")

If you're upgrading from pact-python 1.x or 2.x: expect to rewrite your test fixtures. This isn't a syntax change — it's a different mental model of how the mock server lifecycle works.

The Verifier transport configuration gap

Provider verification had its own friction. The Verifier constructor in pact-python v3 takes a hostname, not a full URL. Passing a full URL causes a silent host mismatch when you later configure the transport:

# ❌ Causes "Host mismatch: localhost != http://localhost:8291"
Verifier("PaymentGateway", "http://localhost:8291")
    .add_transport(url="http://localhost:8291")

# ✅ Correct
Verifier("PaymentGateway", "localhost")
    .add_transport(protocol="http", port=8291, scheme="http")
    .add_source(pact_file)
    .set_request_timeout(10000)  # needed for the 6s timeout stub

The set_request_timeout(10000) line is also non-obvious: the payment timeout stub uses fixedDelayMilliseconds: 6000 to simulate a slow response. The verifier's default timeout is 5 seconds. Without the explicit timeout extension, the timeout interaction fails verification with a connection error rather than a clean pass.

Neither of these are in the main documentation. Both took real time to find. They're in the findings file for this session — linked at the bottom.

The breaking change experiment

All the Pact setup is preamble. This is the proof.

Step 1: Baseline — all contracts verified

pytest tests/pact/test_provider_verification.py -v

Verifying a pact between OrderService and PaymentGateway
  a declined payment charge         (OK)
  a successful payment charge       (OK)
  a timed-out payment charge        (OK)
PASSED

Verifying a pact between OrderService and InventoryService
  [3 interactions — all OK]
PASSED

2 passed in 8.19s

Step 2: Introduce the breaking change

In wiremock/payment-mappings/payment-success.json, one field rename:

// Before
{"status": "ACCEPTED", "transaction_id": "txn-abc-123", "amount": 134.97}

// After — "status" renamed to "result"
{"result": "ACCEPTED", "transaction_id": "txn-abc-123", "amount": 134.97}

Step 3: Provider verification with the breaking change

pytest tests/pact/test_provider_verification.py -v

  a successful payment charge (FAILED)

Failures:
  1.1) has a matching body
         $ -> Actual map is missing the following keys: status
  {
    "amount": 134.97,
  -  "status": "ACCEPTED",
  +  "result": "ACCEPTED",
    "transaction_id": "txn-abc-123"
  }

1 failed in 7.22s

Pact caught it. Exact field. Exact diff. No ambiguity about what broke or why.

Step 4: The same breaking change against the WireMock test suite

pytest tests/steps/test_order_creation.py -v

test_order_is_successfully_created... PASSED
test_order_is_rejected_when_payment_is_declined PASSED
test_order_is_rejected_when_an_item_is_out_of_stock PASSED
test_order_surfaces_partial_unavailability... PASSED
test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

5 passed in 13.01s

Five for five. All green. The breaking change is completely invisible.

Step 5: Revert and confirm

2 passed in 8.19s

Why the WireMock tests stayed green

This isn't a flaw in the Gherkin approach — it's a precise boundary on what any behavioral test can and can't see.

The Gherkin scenarios test the order service's behavior: does the order get confirmed? Does the right status come back to the caller? In app/main.py, when the payment gateway responds, the code checks the HTTP status code and returns {"status": "CONFIRMED"} — it never reads the status field from the payment gateway body. So from the test harness's perspective, nothing changed. The right HTTP code came back, the order was confirmed, all assertions passed.

Pact caught it because the consumer test had explicitly declared that the order service expects a status field in the payment response. That expectation is encoded in the .pact file. When provider verification ran against the modified stub, the Rust verifier compared the actual response against the contract and found the key missing.

The Gherkin test and the Pact consumer test are testing different things. Gherkin tests the system's behavior end-to-end. Pact tests the shape of the conversation between services. You need both. They're not competing — they're covering different failure modes.

The can-i-deploy gate

The final piece was a local can-i-deploy simulation — a script that reads the generated .pact files, checks each interaction's expected response shape against the WireMock stub mappings, and exits 0 (safe) or 1 (blocked).

With contracts intact:

python scripts/can_i_deploy.py

Pact: OrderService → PaymentGateway
  PASS  a declined payment charge
  PASS  a successful payment charge
  PASS  a timed-out payment charge

Pact: OrderService → InventoryService
  PASS  [3 interactions]

RESULT: ALL CONTRACTS VERIFIED — safe to deploy
Exit: 0

With the breaking change in place:

  FAIL  a successful payment charge
        stub is missing fields expected by consumer: ['status']

RESULT: CONTRACT VIOLATIONS DETECTED — do not deploy
Exit: 1

In a real Pact Broker setup, this check queries a central record of which consumer versions have verified which provider versions. The local simulation does something simpler but teaches the same pattern: before you deploy, prove the contract is still satisfied. The exit code is what a CI pipeline reads. A non-zero exit stops the merge.

The full GitHub Actions wiring — where this becomes an automated gate on every PR — is Issue #6. The local simulation is enough to feel how it works.

Where we are

Four issues in, the specification layer is taking shape. Gherkin and WireMock proved the agent builds reliably against a well-written spec. The agent session proved that clean specs produce clean implementations and expose your assumptions. Pact closes the loop — the contract now survives beyond the stub and catches provider drift before it reaches production.

The stack is starting to look like something real. But there's a question I've been putting off since Issue #2 that can't wait any longer: what actually makes a Gherkin scenario good? Because not all specs are equal, and an agent that builds from a loose spec produces something very different from one that builds from a tight one. Next issue I'm going to prove that by deliberately writing bad Gherkin, handing it to the agent, and showing you what comes out.

Next issue: The Spec That Doesn't Lie — deliberately writing bad Gherkin, seeing what the agent builds from it, then rewriting it and comparing the output.

Sources & Further Reading

Pact documentation
pact-python v3 migration guide
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #4

This article was written with the assistance of AI tools.

The Agent Found What Code Review Missed.

Diya Burman — Wed, 10 Jun 2026 01:02:49 +0000

A Level 5 Engineer — Issue #3

If you've been following along, you know what we've built so far. Issue #1 introduced the five levels framework and the Dark Factory concept. Issue #2 got concrete — we wrote five Gherkin scenarios for an order management API before touching any implementation code, stubbed out two external dependencies with WireMock, and ran a real test suite against the whole thing.

At the end of Issue #2 I made a promise: hand the spec to an AI agent, spec only, no implementation hints, and see what it builds.

This is that issue.

The setup

The instruction I gave Claude Code at the start of the session was exactly this:

"The Gherkin scenarios in tests/features/order_creation.feature define the full behavioural contract for this API. Do not read the existing implementation in app/main.py. Build a fresh implementation that makes all 5 scenarios pass. Document your findings in FINDINGS.md as you go."

That's it. No architecture hints. No "use FastAPI." No "here's how the mock servers work." Just the spec and a documentation instruction.

The CLAUDE.md in the repo handled the rest — the guardrails, the project context, the constraint that the .feature files cannot be touched, and the format the FINDINGS.md should follow. If you missed the deep dive on CLAUDE.md in Issue #2, that file is essentially the agent's standing orders. It reads it at the start of every session.

Then I sat back and watched.

What the agent derived from the spec alone

Here's what I found interesting. Before writing a single line of code, the agent read the Gherkin scenarios and derived the entire API contract from them. Unprompted. It produced this:

POST /inventory/check/{inventory_scenario}
  → all available      → POST /payments/charge/{payment_scenario}
  → partial available  → return 207 PARTIAL_UNAVAILABLE (no charge)
  → all out of stock   → return 409 UNAVAILABLE (no charge)

And the full response shape for all five scenarios:

Scenario	status	status_code	Key fields
Success	CONFIRMED	—	`order_id`
Payment declined	PAYMENT_FAILED	402	`decline_reason`, `inventory_released: true`
Out of stock	UNAVAILABLE	409	`unavailable_items`
Partial stock	PARTIAL_UNAVAILABLE	207	`available_items`, `unavailable_items`
Payment timeout	PAYMENT_PENDING	202	`inventory_hold_minutes: 15`, `retry_count`

This is exactly right. The agent read five plain-language scenarios and extracted a precise technical contract — the order of operations, the response codes, the body fields, the retry behaviour — without being told any of it explicitly.

That's not nothing. That's the spec doing its job.

Where it got interesting — the timeout scenario

Scenario 5 is the one I was most curious about. Timeout behaviour is notoriously hard to test and easy to get wrong. The agent worked through it carefully and documented its reasoning:

PAYMENT_TIMEOUT_SECONDS=5 — per-attempt HTTP client timeout
MAX_PAYMENT_RETRIES=2 — total attempt cap, not a retry count on top of the first attempt
Worst-case wall time with 2 attempts at 5 seconds each: 10 seconds — comfortably inside the 12-second contract from the scenario
The WireMock timeout stub uses fixedDelayMilliseconds: 6000 — deliberately longer than the client timeout so the client always times out before the mock responds

That last detail is subtle and correct. If the mock delay were shorter than the client timeout, the test would be testing the wrong thing — the mock responding slowly rather than the client giving up. The agent caught this without being prompted. It's in the FINDINGS.md.

The bug it found that I had written

This is my favourite part of this issue.

The original test setup — the code I pointed Claude Code at — had a hard-coded path:

sys.path.insert(0, "/home/claude/order-api")

On my machine this would silently start mock servers with no stubs loaded. Every payment call would return a 404. Every inventory call would return a 404. The tests would fail in ways that looked like logic errors rather than a configuration problem.

The agent caught it, diagnosed the root cause, and fixed it:

# Before — hard-coded, breaks on any machine but the original
sys.path.insert(0, "/home/claude/order-api")

# After — computed dynamically, works everywhere
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))

To be clear: this bug was in my code. Code I had written and shipped to the repo. The agent found it during implementation because it was trying to run the tests on a different environment and they failed in a way that forced the diagnosis.

This is a thing that happens at Level 4 that doesn't happen at Level 2. When you're implementing yourself, you don't notice the hard-coded paths because everything works on your machine. When an agent implements on a clean environment, your assumptions get exposed immediately.

My honest reaction

I'll be transparent about something. This API isn't complex. It's an order endpoint with two downstream dependencies and five scenarios. I didn't expect the agent to struggle with it, and it didn't. It hit errors, diagnosed them promptly, and moved on. Five scenarios, all passing.

What struck me wasn't the capability — it was the texture of the experience.

Watching Claude Code work, I found myself doing something I don't usually do when I'm implementing: I was evaluating. Not writing, not debugging, not context-switching. Just reading the agent's reasoning and deciding whether I agreed with it. That's a different cognitive posture entirely. It felt closer to a code review than a coding session.

I also noticed I spent the entire session approving individual commands — every file edit, every pytest run, every pip install. Claude Code asks for permission before each action by default. For this first session I let it. From the next task onward I'm going to configure it to run basic commands without checking in every thirty seconds. There's a trust-building curve here, and I'm on the early part of it.

What this proves — and what it doesn't

Five passing scenarios on a moderately simple API is not proof that Level 5 is solved. It's proof that the approach works at this scale and this complexity.

The honest question — the one this newsletter is actually tracking — is whether it holds as the system grows. Pact tests across services. CI/CD pipelines. Evals as guardrails. Contextual stewardship documents for systems with years of history and undocumented decisions baked into the architecture.

That's where the real test is. And that's where we're going next.

What I'd do differently

One thing the exercise exposed: the spec was good enough for the agent to build correctly, but I had one implicit assumption that didn't make it into the scenarios. The response shape for the success case doesn't specify that status_code should be absent — it just checks for order_id. The agent inferred this correctly, but if it hadn't, the test would have passed anyway.

That's a gap in the spec, not a gap in the agent. The lesson is the same one from Issue #2: every implicit assumption is a decision waiting to cause a bug in production. Write it down. Make it a scenario. Make the machine prove it.

Next issue: Phase 3 — adding Pact contract testing between the order service and its dependencies. What happens when the service contract and the mock stub disagree?

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Claude Code documentation
pytest-bdd documentation
Project repository
Session findings - Issue #3

This article was written with the assistance of AI tools.

Senior Developers Using AI Are Getting Slower. The Data Says So.

Diya Burman — Wed, 10 Jun 2026 00:37:33 +0000

A Level 5 Engineer — Issue #2

Preface

If you read Issue 1, you walked away with the map. Six levels, a plateau most engineers never escape, and a Dark Factory that a handful of teams are quietly running in production. If you missed it, go read it first — this one builds directly on it.

This issue is about the single most important shift that happens when you try to move from Level 3 to Level 4. Not the tools. Not the mindset. The bottleneck.

Because it moved. And most of us didn't notice.

When speed stops being the problem

For most of our careers, the bottleneck in software development was implementation speed. You had the idea, you had the design, you had the ticket — the constraint was how fast fingers could turn it into working code. That's the world we optimized for. That's why we measured velocity. That's why standups exist. That's why "10x engineer" was ever a phrase people said out loud without embarrassment.

AI blew that bottleneck wide open.

At Level 2, implementation stops being the constraint almost overnight. You're pairing with an agent and the code just... appears. Features that used to take days take hours. Hours take minutes. It feels like the problem is solved.

Except you haven't solved it. You've just exposed the one that was hiding behind it.

The new bottleneck is specification quality.

The agent can build anything you can describe precisely enough. The operative word is precisely. The moment you try to hand off a vague, half-formed idea — the kind a human developer would fill in with reasonable assumptions and a quick Slack message — the agent either hallucinates something plausible-looking that isn't what you wanted, or it freezes, or worse, it confidently builds the wrong thing all the way to completion.

The constraint is no longer your ability to implement. It's your ability to specify.

What a bad spec actually looks like

Here's the uncomfortable truth — most "requirements" we write as engineers are not specifications. They are vibes dressed up in Jira tickets.

"Add pagination to the users endpoint." That's not a spec. How many results per page? Is the default configurable? What happens when the page number exceeds the total — empty array or 404? What's the sort order? Cursor-based or offset-based? What happens to existing API consumers who aren't sending page parameters yet?

A human developer asks those questions in standup or figures them out from context. An agent working autonomously at Level 4 cannot do that. It will make a choice — silently, confidently, and consistently wrong in a way you won't catch until production.

This is why Dan Shapiro's insight about specification quality isn't just a productivity tip. It's a prerequisite for moving up the ladder at all. You cannot reach Level 4 with Level 2 specs. The system won't let you.

So I built one. Here's what happened.

I wanted to do something concrete this issue rather than just theorize. So I picked a real-world-shaped scenario — an e-commerce order management API with two external dependencies — and built it end to end with WireMock simulating the dependencies and Gherkin scenarios written before the code.

The full project is on GitHub so you can clone it and run the exact same setup on your machine. Everything below is reproducible.

The scenario

A POST /orders endpoint that talks to two external services:

A payment gateway (think Stripe) that can succeed, decline, or time out
An inventory service that can confirm stock, report out-of-stock, or report partial availability

Realistic enough to be relatable. Scoped enough to finish in an afternoon. The kind of integration complexity every backend engineer deals with.

Step 1 — Write the spec first. Actually first.

Here are the five Gherkin scenarios I wrote before a single line of implementation code:

Feature: Order Creation

  Scenario: Order is successfully created when payment succeeds and all items are in stock
    Given a registered user with id "user-123"
    And the inventory service confirms all items are in stock
    And the payment gateway will accept the charge
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "CONFIRMED"
    And the response includes an order id
    And the payment gateway received exactly one charge request
    And the inventory service received a reservation request

  Scenario: Order is rejected when payment is declined
    Given a registered user with id "user-456"
    And the inventory service confirms all items are in stock
    And the payment gateway will decline the charge
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "PAYMENT_FAILED"
    And the response status code is 402
    And the response includes the decline reason "INSUFFICIENT_FUNDS"
    And the inventory reservation is released
    And no order id is issued

  Scenario: Order is rejected when an item is out of stock
    Given a registered user with id "user-789"
    And the inventory service reports SHOE-RED-42 is out of stock
    When the user submits an order for SHOE-RED-42
    Then the order status is "UNAVAILABLE"
    And the response status code is 409
    And the payment gateway is never called

  Scenario: Order surfaces partial unavailability without auto-confirming
    Given a registered user with id "user-321"
    And the inventory service reports SHOE-RED-42 as available but BELT-BRN-M as unavailable
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "PARTIAL_UNAVAILABLE"
    And the response status code is 207
    And the payment gateway is never called
    And no order is confirmed without explicit user action

  Scenario: Order handling is graceful when the payment gateway times out
    Given a registered user with id "user-654"
    And the inventory service confirms all items are in stock
    And the payment gateway will not respond within the timeout window
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the response is returned within 12 seconds
    And the order status is "PAYMENT_PENDING"
    And the inventory is held for 15 minutes
    And the payment gateway is not retried more than 2 times

Notice what these are. Not implementation documents. Not pseudocode. They're a behavioural contract — plain-language descriptions of exactly what the system should do in specific situations, written in a format any teammate, PM, or yes — agent — can read.

The discipline of writing them first forced me to make decisions I would normally have postponed:

Do we check inventory before charging, or charge first? (Inventory first. The fourth scenario locks this in.)
What happens during partial availability — auto-fulfill what's available, or ask the user? (Ask. Encoded in scenario 4.)
What's the timeout SLA on the payment gateway? (5 seconds, with a max of 2 retries. Scenario 5 makes this testable.)
What's the response code for partial availability? (207 Multi-Status.)

Without these scenarios, every one of those decisions would have been made silently by whoever wrote the code first.

Step 2 — Stub the dependencies with WireMock

Before writing any tests you can actually run, the external services need to be simulated. This is what Dan Shapiro calls a digital twin universe — a fully simulated version of your dependencies that behaves like the real thing without the real thing's unpredictability, cost, or rate limits.

WireMock is the industry standard for this. A WireMock stub is just a JSON file describing how a service should respond:

{
  "request": {
    "method": "POST",
    "url": "/payments/charge/success"
  },
  "response": {
    "status": 200,
    "body": "{\"status\": \"ACCEPTED\", \"transaction_id\": \"txn-abc-123\", \"amount\": 134.97}"
  }
}

For the payment timeout scenario, WireMock has a built-in fixedDelayMilliseconds parameter. One line and the mock takes 6 seconds to respond:

{
  "request": {
    "method": "POST",
    "url": "/payments/charge/timeout"
  },
  "response": {
    "status": 504,
    "body": "{\"status\": \"TIMEOUT\"}",
    "fixedDelayMilliseconds": 6000
  }
}

That tiny config line is what makes scenario 5 testable. Without it, you cannot exercise timeout behaviour in a local environment without disabling network connectivity at the OS level — which I have done in the past, and it is exactly as miserable as it sounds.

Step 3 — Wire the scenarios to real assertions

Gherkin by itself is just text. To turn it into an executable test suite I used pytest-bdd, which lets each Given/When/Then line map to a Python function:

@given("the payment gateway will decline the charge", target_fixture="payment_scenario")
def pay_declined():
    return "declined"

@when("the user submits an order for SHOE-RED-42 and BELT-BRN-M", target_fixture="response")
def submit_two(user_id, payment_scenario, inventory_scenario):
    items = [
        {"sku": "SHOE-RED-42", "quantity": 1, "unit_price": 89.99},
        {"sku": "BELT-BRN-M",  "quantity": 1, "unit_price": 44.98},
    ]
    r = requests.post(f"http://localhost:{API_PORT}/orders",
                      json={"user_id": user_id, "items": items,
                            "payment_scenario": payment_scenario,
                            "inventory_scenario": inventory_scenario})
    return {"response": r}

@then("the payment gateway is never called")
def payment_not_called():
    assert not payment_log.all(), f"Expected no payment calls, got: {payment_log.all()}"

That last assertion — the payment gateway is never called — is the kind of thing that's almost impossible to verify with traditional unit tests but trivial with WireMock. WireMock records every call it receives. You assert against that log directly.

Step 4 — Run the suite

$ pytest tests/steps/test_order_creation.py -v
============================= test session starts ==============================
collected 5 items

tests/steps/test_order_creation.py::test_order_is_successfully_created_when_payment_succeeds_and_all_items_are_in_stock PASSED
tests/steps/test_order_creation.py::test_order_is_rejected_when_payment_is_declined PASSED
tests/steps/test_order_creation.py::test_order_is_rejected_when_an_item_is_out_of_stock PASSED
tests/steps/test_order_creation.py::test_order_surfaces_partial_unavailability_without_autoconfirming PASSED
tests/steps/test_order_creation.py::test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

============================== 5 passed in 13.53s ==============================

Five for five. But getting there was educational.

What broke along the way

I want to be honest about the failures because that's where the actual learning happened. The five tests didn't pass on the first run. They didn't pass on the second run either.

Failure 1 — The "no stub matched" silent success

When a request comes in that no WireMock stub knows how to handle, the default behaviour is to return a 404. My API code did this:

try:
    pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{scenario}", ...)
    payment_result = pay.json()
except Exception:
    raise HTTPException(503, "Payment service error")

A 404 is not an exception in httpx. It's just a response. So the API would happily call pay.json(), get {"error": "No stub matched"}, and treat the entire interaction as a success — issuing an order id and confirming the order even though no real payment had been processed.

This is genuinely dangerous. A misconfigured mock would have made all my tests pass while hiding that the real service path was broken. Lesson: always explicitly check the response status from a mock. The fix:

try:
    pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{scenario}", ...)
    if pay.status_code == 404:
        raise HTTPException(503, f"Payment scenario not found: {scenario}")
    payment_result = pay.json()

Failure 2 — The shared call log bug

I started with one MockServer class that held a single class-level call log. Both the payment and inventory mocks recorded into the same list. When the test asserted "the payment gateway received exactly one charge request," the inventory call was in the log but no payment call was — because of failure 1 — and the assertion was looking at the combined log.

The fix was conceptually small but architecturally important — each mock server instance gets its own call log:

def start_mock_server(port: int, mappings_dir: str) -> tuple[HTTPServer, MockCallLog]:
    stubs = [json.loads(f.read_text()) for f in Path(mappings_dir).glob("*.json")]
    log = MockCallLog()                  # ← per-instance log
    handler = make_handler(stubs, log)
    server = HTTPServer(("localhost", port), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server, log

This mirrors how real WireMock works in production — you run separate WireMock instances per service, each with its own request log. The bug was a direct consequence of cutting that corner.

Failure 3 — The fixture wiring gap

Scenarios 3 and 4 don't define a payment scenario in their Given clauses, because the payment gateway should never be called in those cases. But pytest-bdd was still expecting the payment_scenario fixture — and erroring out before the test even ran.

This is a subtle distinction worth naming. The Gherkin spec was correct. It said exactly what it should say. The error was in the test wiring that connected the spec to the assertions. The fix was a default fixture:

@pytest.fixture
def payment_scenario():
    """Default — overridden by specific Given steps."""
    return "success"

The spec stays clean. The wiring handles the case where a scenario doesn't care about a particular setup.

What this exercise actually proved to me

A few things that are now visceral rather than abstract:

Specs that the AI cannot see during the build are uniquely powerful. My scenarios live in tests/features/order_creation.feature. The implementation lives in app/main.py. When I asked an agent to modify the API, I could give it the implementation only. The spec stayed external. The agent had to make the test pass against behaviour it couldn't reverse-engineer from the assertions themselves. This is the part that genuinely changes things at Level 4.

WireMock's 404-on-no-match is a feature, not a bug. It exposes integration mistakes that would otherwise hide forever. The first time I saw a test silently succeed because of the 404 passthrough I was annoyed. Now I think it should be louder.

Writing the scenarios first changed what I built. Scenario 4 — partial availability — would not have existed if I'd written the code first. I would have implemented "all available or fail" and shipped it. Writing the spec first made me confront the question. The answer became part of the system.

Try this yourself

Everything above is in a project you can clone, run, and break. Five scenarios, two mock services, one API. Total setup time: under fifteen minutes if you have Python and pip installed.

git clone <repo-url> order-api
cd order-api
pip install fastapi uvicorn httpx pytest pytest-bdd requests
pytest tests/steps/test_order_creation.py -v

If you want to use real WireMock instead of the Python-based mock:

# Download WireMock standalone
curl -L -o wiremock.jar \
  https://repo1.maven.org/maven2/org/wiremock/wiremock-standalone/3.3.1/wiremock-standalone-3.3.1.jar

# Run two instances — the JSON mappings work as-is
java -jar wiremock.jar --port 8081 --root-dir wiremock/payment-mappings &
java -jar wiremock.jar --port 8082 --root-dir wiremock/inventory-mappings &

The WireMock mapping JSON files I wrote work in real WireMock with zero changes. That was deliberate. The Python mock is for getting started fast. The real WireMock is for when you want to scale this pattern across an actual service mesh.

Next issue: I take this same setup and hand it to an AI agent. Spec only — no implementation hints. We see what it builds, what it gets wrong, and how the spec acts as a guardrail.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Cucumber + Gherkin documentation
WireMock documentation
pytest-bdd documentation
Session findings - Issue #2

This article was written with the assistance of AI tools.

There Are 5 Levels of AI Coding. Most Engineers Are Stuck on Level 2.

Diya Burman — Tue, 09 Jun 2026 19:12:34 +0000

The Level 5 Engineer Newsletter — Issue #1

Preface

Before you panic and close this tab — yes, I see you hovering!

I’m not naively chasing the thing that puts me out of a job — I’m aware of the irony. The tech world is in full meltdown mode about AI taking developer jobs right now, and, I think, most of that noise misses the point entirely.

A Dark Factory still needs someone who understands the system deeply enough to define what it should build, catch what it shouldn’t touch, and course-correct when it goes sideways. That’s not a developer writing code anymore — that’s a steward. The role doesn’t disappear; it transforms. From worker bee to architect. From implementer to the person who holds the mental model of the entire system and encodes that judgment into infrastructure that the agents can operate safely within. That shift is actually what this newsletter is about — and it deserves its own deep dive, which is coming. For now, just know:

Level 5 is the destination, not the cliff edge.

So. The Five Levels.

Dan Shapiro borrowed the structure from the NHTSA’s autonomous driving classification — five levels from “human does everything” to “machine does everything, humans not required.” Applied to software development, it maps out like this:

Level 0 — Spicy Autocomplete. You’re still writing every character. Maybe you use AI as a search engine or accept the occasional tab suggestion. The code is yours. You’re also losing ground every day to the people who aren’t at Level 0.

Level 1 — The Coding Intern. You’re delegating discrete tasks. “Write a unit test for this.” “Add a docstring here.” You’re seeing some speedup. Your job is essentially unchanged. YOU are still the bottleneck.

Level 2 — Junior Developer. This is where it starts to feel like something. You’re pairing with AI like a colleague. Flow state. Productivity you haven’t felt in years. You hand off the boring stuff and focus on the interesting parts. Here’s the trap though — Level 2 feels like you’re done. You’re not done. Most people who think they’re “using AI” are living here permanently and calling it Level 5.

Level 3 — Developer as Manager. You’re not writing much code anymore. Your AI agent has multiple tabs running at all times. Your life is diffs. You review everything at the PR level. For a lot of people, this actually feels worse than Level 2 — more overhead, less flow. And this is where almost everyone tops out. Not because they can’t go further. Because Level 3 feels like the ceiling.

Level 4 — Developer as Product Manager. The code is a black box. You write specs. You argue about the specs. You set up the right tools, define the right constraints, and then you leave for 12 hours. You come back and check if the tests passed. Dan Shapiro says he’s here. I believe him because of how he describes it — it doesn’t sound glamorous, it sounds like a different kind of hard work.

Level 5 — The Dark Factory. Named after Fanuc’s robot factory — staffed entirely by robots, lights off, no humans needed or welcome. At Level 5, you’re not really running a software process anymore. You have a system that turns specs into software, autonomously. A handful of teams are doing this today. Small teams, less than five people, shipping production software with no human-written or human-reviewed code. A prominent example would be Claude’s own engineering team, who claim that Claude Code wrote most of Claude Code.

Like what you read thus far? This post is public, so feel free to share it.

The Part That Actually Stings

Here’s what Nate B. Jones added to this framework, which made me sit with it for a while.

There’s a rigorous study showing that experienced developers using AI tools took 19% longer on tasks — while believing they were 24% faster. The gap between perceived productivity and actual productivity even has a name — the AI confidence gap. You feel faster. The clock disagrees.

The teams that are pulling away aren’t using more AI tools. They’re using AI differently. The bottleneck has moved. It’s no longer about how fast you can implement. It’s about how precisely you can specify.

The Dark Factory teams aren’t superhuman coders. They’ve built infrastructure around judgment — external behavioural scenarios that the AI cannot see during the build process, digital twin environments that simulate production dependencies safely, testing architectures designed specifically so the AI can’t reverse-engineer the passing criteria.

The rest of the industry is plateaued at Level 3, reviewing diffs, and measuring velocity in story points on a Jira board that hasn’t been groomed since Q2.

Why I’m Writing This

I’ve been a software engineer for nearly a decade. I’ve done the serverless architectures, the MLOps pipelines, the distributed systems. I’ve been to Re:Invent (3 times. Ooooh, fancy schmancy). I know the stack.

And I watched these videos and realized I was at Level 2. Maybe Level 3 on a good week. And I had been calling it “using AI effectively.”

This newsletter is the documentation of that climb — starting with understanding what the levels actually mean and why most of us have been misreading where we stand. Or at least that’s where I’m starting. I wouldn’t be surprised if my own understanding shifts incrementally as I make progress. That’s kind of the point.

Not the theory — the practice. The tools, the habits, the org-level thinking, the moments where something clicks. I’ll be honest when I’m stuck and honest when something works. No performance, no hype.

The climb starts with understanding the map. Now you have it.

Next issue: The specification quality problem — why the bottleneck has shifted and what “writing a good spec” actually means in practice.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2) · natebjones.com · Substack

Thanks for reading The Level 5 Engineer! Subscribe for free to receive new posts and support my work.

This article was written with the assistance of AI tools.