DEV Community: Diya Burman

Wiring the Guardrails

Diya Burman — Wed, 10 Jun 2026 03:04:02 +0000

A Level 5 Engineer — Issue #6

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Five issues in, everything we've built lives on one machine. The Gherkin scenarios, the WireMock stubs, the Pact contracts, the can-i-deploy script — all of it runs locally, passes locally, and means nothing the moment someone else touches the codebase.

Issue #6 fixes that. A GitHub Actions pipeline now runs on every push, executes the full specification stack in dependency order, and blocks merges to main if anything breaks. The pipeline is the guardrail. From this point on, a broken contract or a failing scenario cannot reach main undetected.

Getting there took ninety minutes and two interventions I didn't plan for. Both are worth documenting.

Before the YAML: deciding what "green" means

The first thing Claude Code did before touching any pipeline config was run the full test suite to establish a baseline. The instruction was explicit: everything must pass before a single line of YAML gets written.

It found a failure immediately — and it wasn't from the breaking change experiment. It was from Issue #5.

The bad-spec test (test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order) was still asserting db_status in the response body. That was intentional in Issue #5 — the failure was the finding. The session ended with it red because the point was to show what bad specs produce. But on main, with CI incoming, that means the pipeline would have been red on day one before a single feature change.

The fix was adding backward-compat aliases to the response:

return {
    "order_id": order_id,
    "status": order["db_status"],            # good spec field
    "db_status": order["db_status"],         # bad spec alias — keeps Issue #5 test passing
    "placed_at": order["order_created_at"],  # good spec field
    "order_created_at": order["order_created_at"],  # bad spec alias
}

Neither test file was modified. No feature files were touched. The aliases kept both the good-spec and bad-spec tests passing against the same endpoint.

The reason this matters before the pipeline exists: a team that starts CI with a known failure trains itself to ignore red. The cost of normalising a red CI is much higher than the cost of fixing the baseline first. Claude Code made the right call and documented it before moving on.

The pipeline structure

Four jobs, in dependency order:

test → pact-consumer → pact-verify → can-i-deploy

Each job only runs if its predecessor passes. If Gherkin breaks, Pact never runs. If the consumer tests fail, verification never runs. If verification fails, can-i-deploy is skipped. The pipeline fails fast and tells you exactly which layer broke.

The artifact chain is what makes it a pipeline rather than four parallel scripts. The pact-consumer job generates the .pact files and uploads them as a GitHub Actions artifact. The pact-verify job downloads that artifact and verifies it — the same files, not freshly regenerated ones. Without this, each job would build its own consumer contract from scratch, and verification would be proving that the contract matches the code rather than proving it matches what pact-consumer actually produced.

One non-obvious piece: mock_server.py is a library module with no command-line entry point. The pipeline needed servers running as background processes. The fix was an inline Python invocation:

- name: Start mock servers
  run: |
    . .venv/bin/activate
    python -c "
    import time
    from mock_server import start_mock_server
    start_mock_server(8091, 'wiremock/payment-mappings')
    start_mock_server(8092, 'wiremock/inventory-mappings')
    time.sleep(86400)
    " &
    sleep 2

The time.sleep(86400) keeps the process alive for the duration of the job. Inelegant but functional. A proper if __name__ == "__main__" entry point with argparse is the obvious cleanup for a future session.

The first CI run — and why I had to intervene manually

The YAML was committed, pushed to main, and the pipeline ran. All three runs failed on the test job:

OSError: [Errno 98] Address already in use

Ports 8091 and 8092. Every test in test_order_creation.py errored at setup. The order status tests — which don't use the mock servers — passed fine.

Claude Code didn't catch this on its own. Here's why that's worth explaining.

When Claude Code wrote the pipeline, it was working from the codebase and its own knowledge of GitHub Actions patterns. It knew the mock servers needed to be running before pytest started, so it added an explicit start-servers step to the YAML — a reasonable decision based on the information it had. What it couldn't see was the runtime interaction between that YAML step and pytest's session-scoped fixtures, because that interaction only manifests in the CI environment, not locally.

Locally, running pytest tests/steps/ -v has always worked correctly because the session fixture starts the servers and nothing else competes. Claude Code had only ever seen local runs succeed. It had no signal that the YAML step was creating a conflict — because the conflict doesn't exist locally.

This is a fundamental limit of the "paste and walk away" approach at the boundary between local and remote environments: the agent can reason about the codebase and about CI patterns, but it can't observe the CI run itself. The failure was on GitHub. Claude Code was in a terminal. Those two things weren't connected.

I diagnosed the error from the GitHub Actions log, explained the root cause, and pasted new instructions. Claude Code fixed it in one step — removing the redundant YAML steps entirely:

# Removed from both test and pact-verify jobs:
- name: Start mock servers
  run: |
    . .venv/bin/activate
    python -c "..." &
    sleep 2

The pytest session fixtures already own server lifecycle correctly. scope="session" means pytest starts the servers once per test run and keeps them alive. The YAML step was duplicating a responsibility that was already handled. The fix wasn't a workaround — it was removing the wrong layer.

The root cause in plain terms: the YAML step and the pytest fixture both thought they were responsible for starting the servers. The port was already bound when the fixture tried to bind it again. Works on my machine. Breaks in CI. Classic.

The breaking change experiment — in the pipeline

With the pipeline green, the breaking change test ran as designed.

Branch test/breaking-change-pipeline, commit 76c0d89: renamed "status" to "result" in wiremock/payment-mappings/payment-success.json. Same change as Issue #4, now running through CI instead of local verification.

The expected failure:

a successful payment charge (FAILED)

Failures:
1) Verifying a pact between OrderService and PaymentGateway
   1.1) has a matching body
          $ -> Actual map is missing the following keys: status
   {
     "amount": 134.97,
  -  "status": "ACCEPTED",
  +  "result": "ACCEPTED",
     "transaction_id": "txn-abc-123"
   }

pact-verify fails. can-i-deploy is skipped. The merge is blocked.

And the key point from Issue #4 holds at the pipeline level: the test job — the Gherkin suite — would pass with the breaking change in place. The order creation scenarios check HTTP status codes and business outcomes. They never read pay_resp.json()["status"]. A stub returning result instead of status still returns HTTP 200. Gherkin passes. Pact catches it.

This is the division of labour. Gherkin proves the system does the right thing. Pact proves the contracts don't drift. You need both, and now both run automatically on every push.

The one step that requires the GitHub UI

Claude Code cannot configure branch protection rules — that requires the GitHub web UI or admin API. This step is non-negotiable and must be done manually:

Repo → Settings → Branches → Add branch protection rule
Branch name pattern: main
Enable Require status checks to pass before merging
Add all four status checks: test, pact-consumer, pact-verify, can-i-deploy
Enable Require branches to be up to date before merging
Save

Without this, the pipeline is advisory. A push to main can still happen even if all four jobs are red. The pipeline becomes a dashboard — it shows you the problem but doesn't stop anything. Branch protection is what turns "CI failed" from a notification into enforcement. The pipeline is only a guardrail if something stops you going around it.

The honest part

The YAML took about twenty minutes to write. The session took ninety minutes total — because the baseline fix and the port conflict ate the rest.

The instinct during the baseline audit was to skip past the known failure. It's a demo test, we know why it's there, configure CI to skip that file and move on. That would have been thirty seconds. It also would have been wrong — a pipeline with documented exceptions is a pipeline people route around.

The instinct during the port conflict was to blame the CI environment. Ubuntu runs things differently, ports work differently, it's a platform quirk. That framing would have sent the debugging in the wrong direction. The actual cause was simpler: two layers both thought they owned the same responsibility, and nobody had written down which one was actually in charge.

Both of those moments are the J-curve. Not the YAML — the discipline of not skipping and not blaming the environment. The overhead of CI is not the config file. It's every decision about what "green" actually means and who's responsible for what.

The pipeline is now real infrastructure. The breaking change can't reach main. That's worth ninety minutes.

Next issue: The Scope Problem — scaling Gherkin across a multi-service system. What happens when one spec file isn't enough, and how spec debt forms.

Sources & Further Reading

GitHub Actions documentation
Pact documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #6

This article was written with the assistance of AI tools.

The Spec That Doesn't Lie

Diya Burman — Wed, 10 Jun 2026 02:40:42 +0000

A Level 5 Engineer — Issue #5

Preface

Every issue so far has assumed something I haven't said out loud: that the specs are good. Issue #2 wrote them carefully. Issue #3 handed them to an agent and watched it build correctly. Issue #4 proved the contracts survive provider drift.

But what happens when the spec isn't good? Not broken — Gherkin syntax is fine, tests pass, the agent builds something. Just imprecise. Vague in ways that feel precise when you're writing them.

This issue answers that question by doing the thing deliberately. I wrote bad Gherkin on purpose, handed it to the agent, watched what it built — and then rewrote the spec and did it again. The difference between the two implementations is the article.

The hardest thing about bad specs

Bad specs are hard to spot when you're writing them because they feel complete.

A scenario that references implementation details sounds like reasonable description — you wrote the implementation, so the details feel like specifics. A Given clause that feels obvious to you will be interpreted differently by every reader who hasn't seen the code. The Gherkin is syntactically correct. The tests pass. Nothing in the output signals that anything is wrong.

This is the trap. It's not that bad specs break things. It's that they don't.

The endpoint

I added a new endpoint to the order-api project: GET /orders/{order_id}/status. It returns the current status of an order and relevant metadata. Simple enough that the spec should be easy to write well. Which makes it a good target for writing it badly on purpose.

The bad specs

Two scenarios. Both syntactically valid. Both produce passing tests. Both wrong in different ways.

# BAD SPEC 1 — The leaky spec
# Problem: references internal implementation concepts (db_status, order_created_at)
# rather than describing what a caller observes. The agent uses these names literally
# in the response body, leaking storage terminology into the public API contract.

Scenario: Retrieving status for a confirmed order
  Given an order exists in the system with db_status "CONFIRMED"
  When I request GET /orders/{order_id}/status
  Then the response should contain the db_status field set to "CONFIRMED"
  And the order_created_at field should be populated from the order record

# BAD SPEC 2 — The vague Given
# Problem: "an order that has not been placed" is underspecified. The agent must
# guess what this means — a malformed ID? A well-formed UUID with no record?
# A deleted order? Each interpretation is plausible and produces different behavior.

Scenario: Retrieving status for an order that does not exist
  Given an order that has not been placed
  When I request GET /orders/{order_id}/status
  Then the response should indicate the order was not found

Both passed immediately:

tests/steps/test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order PASSED
tests/steps/test_order_status_bad.py::test_retrieving_status_for_an_order_that_does_not_exist PASSED

2 passed in 0.34s

Green. No warnings. No hint that anything is wrong.

What the agent built from the bad specs

Here's the implementation the agent produced:

@app.get("/orders/{order_id}/status")
def get_order_status(order_id: str):
    order = _orders.get(order_id)
    if order is None:
        raise HTTPException(status_code=404, detail="Order not found")
    return {
        "order_id": order_id,
        "db_status": order["db_status"],
        "order_created_at": order["order_created_at"],
    }

It satisfies the spec completely. It also made four decisions the spec never made:

Decision 1: The field is named db_status in the response.
The spec said db_status so the agent used db_status. It never questioned whether this was an internal name leaking into a public API. It satisfied the spec literally.

Decision 2: A missing order returns 404.
The spec says "indicate the order was not found." 404 is a defensible interpretation. So is 422, 403, or a 200 with a NOT_FOUND status field. The agent picked the most conventional option — but the spec never mandated it, and FastAPI's default 404 body is {"detail": "Order not found"}, not {"error": "Order not found"}. A client checking response.json()["error"] gets a KeyError.

Decision 3: The timestamp field is named order_created_at with no format requirement.
The spec says "populated from the order record." The agent chose order_created_at and returned an ISO string because that's what datetime.utcnow().isoformat() produces. The step definition checked only that the field is non-empty and a string — so any format would have passed. A Unix timestamp integer would have passed. A human-readable string like "June 2nd" would have passed.

Decision 4: The order store is in-memory.
The spec says nothing about persistence. An in-memory dict is the simplest thing that makes the tests pass. In production, orders are persisted. The in-memory store vanishes on restart and isn't shared across worker processes.

Every one of these decisions is plausible. The agent made the reasonable call every time. That's not the problem. The problem is that a different agent, given the same spec, might have made different reasonable calls — and both implementations would pass the same test suite.

The rewrite

Writing the good spec forced every decision the bad spec had silently delegated:

# GOOD SPEC 1 — Caller's perspective, not implementation's
# Fixed: field names describe what the caller observes (status, placed_at)
# not what the storage layer calls them (db_status, order_created_at).
# The format of placed_at is now a contract obligation, not an assumption.

Scenario: Confirmed order status is returned with placement timestamp
  Given a confirmed order with id "order-abc-123" exists in the system
  When I request GET /orders/order-abc-123/status
  Then the response status code is 200
  And the response body contains "order_id" equal to "order-abc-123"
  And the response body contains "status" equal to "CONFIRMED"
  And the response body contains "placed_at" as a valid ISO 8601 timestamp

# GOOD SPEC 2 — Precise Given, explicit 404 body shape
# Fixed: "a well-formed UUID with no corresponding record" is now unambiguous.
# The 404 response body shape is now a contract obligation, not a guess.

Scenario: Unknown order id returns 404 with error message
  Given no order with id "order-xyz-999" exists in the system
  When I request GET /orders/order-xyz-999/status
  Then the response status code is 404
  And the response body contains an "error" field

Notice what changed. The scenarios describe the same two situations. The intent is identical. But now every decision is in the spec rather than in the agent's interpretation of the spec.

What the agent built from the good spec

@app.get("/orders/{order_id}/status")
def get_order_status(order_id: str):
    order = _orders.get(order_id)
    if order is None:
        return JSONResponse(status_code=404, content={"error": "Order not found"})
    return {
        "order_id": order_id,
        "status": order["db_status"],
        "placed_at": order["order_created_at"],
    }

Same endpoint. Same logic. Different API.

db_status became status. order_created_at became placed_at. The 404 body now contains error not detail. The timestamp is now asserted to be ISO 8601 — not just non-empty.

These are not cosmetic differences. They are different contracts that clients build against.

The cross-run

After building from the good spec, I ran the bad-spec tests against the new implementation:

tests/steps/test_order_status_bad.py::test_retrieving_status_for_a_confirmed_order FAILED
tests/steps/test_order_status_bad.py::test_retrieving_status_for_an_order_that_does_not_exist PASSED

E   KeyError: 'db_status'

The leaky test failed. The field db_status doesn't exist in the good implementation — it's been renamed to status, which is what a caller should see. The test that was checking for an internal name is now broken, correctly.

The vague test passed. Both implementations return a 404 for a missing order — the good implementation just happened to reach the same conclusion, but for an explicit reason this time.

That asymmetry is instructive. The vague Given produced the right answer by coincidence. The leaky Then produced the wrong field name by construction. One was luck. One was baked in.

Why this matters

Both implementations pass their own test suites. That is the trap.

If you run the bad-spec tests against the bad-spec implementation: green. If you run the good-spec tests against the good-spec implementation: green. The difference only surfaces when you cross-run — and in production, you never cross-run. You ship the bad implementation, it passes CI, and the problem lands in a client exception report six months later.

Here's the concrete difference: the bad-spec implementation returns db_status and order_created_at with no format guarantee. The good-spec implementation returns status and placed_at with a mandatory ISO 8601 format. An agent given the bad spec had no way to know that db_status was wrong — the spec said db_status. An agent given the good spec had no choice but to produce status — the spec said status.

Spec quality is not about whether tests pass. It is about how much of the implementation the spec author wrote versus how much was silently delegated to the agent. Every silent delegation is a place where two agents given the same spec produce different code — code that both passes, but disagrees on the contract.

At scale — dozens of endpoints, hundreds of scenarios — that disagreement is the system.

The practical test for a good spec

Before handing any scenario to an agent, ask one question: what decisions does this scenario leave open?

If the answer is "none — every field name, format, response code, and body shape is specified," the spec is ready. If the answer is "a few reasonable ones," those are the places where your implementation and the next agent's implementation will silently diverge.

The agent will always make reasonable decisions. That's not the problem. The problem is that reasonable is not the same as specified — and at Level 4, specified is the only thing that counts.

Next issue: Wiring the Guardrails — GitHub Actions, the Pact Broker, and the pipeline that turns contract violations into blocked merges automatically.

Sources & Further Reading

Cucumber + Gherkin documentation
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #5

This article was written with the assistance of AI tools.

The Contract That Survives the Agent

Diya Burman — Wed, 10 Jun 2026 02:24:34 +0000

A Level 5 Engineer — Issue #4

Preface

If you've been following along, you know where we are. Issue #2 introduced WireMock and Gherkin — write the behavioral contract before the code, stub your dependencies, run a real test suite. Issue #3 handed that spec to an AI agent and walked away. Five scenarios passed. The agent even found a bug in my code.

Everything worked. And that's exactly the problem this issue is about.

Because the WireMock stubs working perfectly is not the same thing as the real services working. The gap between those two statements is where production incidents are born.

The confidence trap

Here's the scenario nobody talks about until it happens to them.

Your order service calls a payment gateway. You've stubbed it with WireMock. Your Gherkin scenarios pass. Your agent builds against those stubs. Five for five, green across the board.

Meanwhile, the payment gateway team — a different squad, a different repo, maybe a different company entirely — ships a cleanup. They've been inconsistent about field naming across their API. status in one endpoint, result in another. They standardize. They rename status to result in the charge response. Their tests pass. They deploy.

Your tests still pass too. The stub hasn't changed. The stub will never change unless you change it.

The first time you learn about the rename is a production incident.

This is the confidence trap: a mock that can drift from the real service makes you feel safe right up until production proves you weren't. The tests are green. The contract is broken. You just don't know it yet.

What Pact does differently

WireMock is a behavioral double — it simulates a service so your tests can run in isolation. You define what it returns. You maintain it. You can make it say anything you want, which means it can silently lie about what the real service actually does.

Pact inverts the trust relationship.

Instead of you maintaining a stub that you hope reflects reality, your consumer tests declare what they need from the provider. Those declarations get written into a .pact file — a machine-readable contract. The provider then runs verification against that contract before it ships. If the provider no longer satisfies what the consumer declared, verification fails and the deploy is blocked.

The consumer defines the need. The provider proves delivery. No human has to remember to update a stub.

Building it — and what the docs didn't tell me

I added Pact to the order-api project this issue, covering both downstream dependencies — the payment gateway and the inventory service — with consumer tests matching the same five scenarios from the Gherkin feature file.

It was less smooth than I expected.

The pact-python v3 FFI surprise

Every tutorial for pact-python shows the same pattern: create a module-scoped Pact fixture, run multiple tests against it, write the pact file at the end. I wrote exactly that. The first test in each class passed. Every subsequent test failed with this:

RuntimeError: The provider state could not be specified.

No hint of what was actually wrong. After digging into the source, the root cause: pact-python 3.x is a complete rewrite backed by a Rust FFI binary. The Rust handle is consumed by the first serve() call — you cannot add new interactions to a handle after that point. The v2-style module-scoped pattern violates this constraint in a way the error message doesn't explain at all.

The fix was restructuring the consumer tests so all interactions are defined upfront before any serve() call:

# ❌ v2-style — breaks in pact-python v3
class TestPaymentConsumer:
    @pytest.fixture(scope="module")
    def pact(self):
        return Consumer("OrderService").has_pact_with(Provider("PaymentGateway"))

    def test_success(self, pact):
        pact.given("payment succeeds").upon_receiving("a charge")...
        with pact:
            # test

    def test_declined(self, pact):
        pact.given("payment declined").upon_receiving("a decline")...
        # RuntimeError — handle already consumed

# ✅ v3 correct pattern — all interactions before serve()
def test_payment_gateway_consumer():
    pact = Consumer("OrderService").has_pact_with(Provider("PaymentGateway"), ...)
    (pact
        .given("the payment gateway will accept the charge")
        .upon_receiving("a successful payment charge")
        .with_request("POST", "/payments/charge/success")
        .will_respond_with(200, body={"status": "ACCEPTED",
                                      "transaction_id": "txn-abc-123",
                                      "amount": 134.97}))
    (pact
        .given("the payment gateway will decline the charge")
        .upon_receiving("a declined payment charge")
        .with_request("POST", "/payments/charge/declined")
        .will_respond_with(402, body={"status": "DECLINED",
                                      "reason": "INSUFFICIENT_FUNDS"}))
    # ... all interactions defined ...
    with pact.serve() as srv:
        # exercise all interactions against srv.url
    pact.write_file("pacts/")

If you're upgrading from pact-python 1.x or 2.x: expect to rewrite your test fixtures. This isn't a syntax change — it's a different mental model of how the mock server lifecycle works.

The Verifier transport configuration gap

Provider verification had its own friction. The Verifier constructor in pact-python v3 takes a hostname, not a full URL. Passing a full URL causes a silent host mismatch when you later configure the transport:

# ❌ Causes "Host mismatch: localhost != http://localhost:8291"
Verifier("PaymentGateway", "http://localhost:8291")
    .add_transport(url="http://localhost:8291")

# ✅ Correct
Verifier("PaymentGateway", "localhost")
    .add_transport(protocol="http", port=8291, scheme="http")
    .add_source(pact_file)
    .set_request_timeout(10000)  # needed for the 6s timeout stub

The set_request_timeout(10000) line is also non-obvious: the payment timeout stub uses fixedDelayMilliseconds: 6000 to simulate a slow response. The verifier's default timeout is 5 seconds. Without the explicit timeout extension, the timeout interaction fails verification with a connection error rather than a clean pass.

Neither of these are in the main documentation. Both took real time to find. They're in the findings file for this session — linked at the bottom.

The breaking change experiment

All the Pact setup is preamble. This is the proof.

Step 1: Baseline — all contracts verified

pytest tests/pact/test_provider_verification.py -v

Verifying a pact between OrderService and PaymentGateway
  a declined payment charge         (OK)
  a successful payment charge       (OK)
  a timed-out payment charge        (OK)
PASSED

Verifying a pact between OrderService and InventoryService
  [3 interactions — all OK]
PASSED

2 passed in 8.19s

Step 2: Introduce the breaking change

In wiremock/payment-mappings/payment-success.json, one field rename:

// Before
{"status": "ACCEPTED", "transaction_id": "txn-abc-123", "amount": 134.97}

// After — "status" renamed to "result"
{"result": "ACCEPTED", "transaction_id": "txn-abc-123", "amount": 134.97}

Step 3: Provider verification with the breaking change

pytest tests/pact/test_provider_verification.py -v

  a successful payment charge (FAILED)

Failures:
  1.1) has a matching body
         $ -> Actual map is missing the following keys: status
  {
    "amount": 134.97,
  -  "status": "ACCEPTED",
  +  "result": "ACCEPTED",
    "transaction_id": "txn-abc-123"
  }

1 failed in 7.22s

Pact caught it. Exact field. Exact diff. No ambiguity about what broke or why.

Step 4: The same breaking change against the WireMock test suite

pytest tests/steps/test_order_creation.py -v

test_order_is_successfully_created... PASSED
test_order_is_rejected_when_payment_is_declined PASSED
test_order_is_rejected_when_an_item_is_out_of_stock PASSED
test_order_surfaces_partial_unavailability... PASSED
test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

5 passed in 13.01s

Five for five. All green. The breaking change is completely invisible.

Step 5: Revert and confirm

2 passed in 8.19s

Why the WireMock tests stayed green

This isn't a flaw in the Gherkin approach — it's a precise boundary on what any behavioral test can and can't see.

The Gherkin scenarios test the order service's behavior: does the order get confirmed? Does the right status come back to the caller? In app/main.py, when the payment gateway responds, the code checks the HTTP status code and returns {"status": "CONFIRMED"} — it never reads the status field from the payment gateway body. So from the test harness's perspective, nothing changed. The right HTTP code came back, the order was confirmed, all assertions passed.

Pact caught it because the consumer test had explicitly declared that the order service expects a status field in the payment response. That expectation is encoded in the .pact file. When provider verification ran against the modified stub, the Rust verifier compared the actual response against the contract and found the key missing.

The Gherkin test and the Pact consumer test are testing different things. Gherkin tests the system's behavior end-to-end. Pact tests the shape of the conversation between services. You need both. They're not competing — they're covering different failure modes.

The can-i-deploy gate

The final piece was a local can-i-deploy simulation — a script that reads the generated .pact files, checks each interaction's expected response shape against the WireMock stub mappings, and exits 0 (safe) or 1 (blocked).

With contracts intact:

python scripts/can_i_deploy.py

Pact: OrderService → PaymentGateway
  PASS  a declined payment charge
  PASS  a successful payment charge
  PASS  a timed-out payment charge

Pact: OrderService → InventoryService
  PASS  [3 interactions]

RESULT: ALL CONTRACTS VERIFIED — safe to deploy
Exit: 0

With the breaking change in place:

  FAIL  a successful payment charge
        stub is missing fields expected by consumer: ['status']

RESULT: CONTRACT VIOLATIONS DETECTED — do not deploy
Exit: 1

In a real Pact Broker setup, this check queries a central record of which consumer versions have verified which provider versions. The local simulation does something simpler but teaches the same pattern: before you deploy, prove the contract is still satisfied. The exit code is what a CI pipeline reads. A non-zero exit stops the merge.

The full GitHub Actions wiring — where this becomes an automated gate on every PR — is Issue #6. The local simulation is enough to feel how it works.

Where we are

Four issues in, the specification layer is taking shape. Gherkin and WireMock proved the agent builds reliably against a well-written spec. The agent session proved that clean specs produce clean implementations and expose your assumptions. Pact closes the loop — the contract now survives beyond the stub and catches provider drift before it reaches production.

The stack is starting to look like something real. But there's a question I've been putting off since Issue #2 that can't wait any longer: what actually makes a Gherkin scenario good? Because not all specs are equal, and an agent that builds from a loose spec produces something very different from one that builds from a tight one. Next issue I'm going to prove that by deliberately writing bad Gherkin, handing it to the agent, and showing you what comes out.

Next issue: The Spec That Doesn't Lie — deliberately writing bad Gherkin, seeing what the agent builds from it, then rewriting it and comparing the output.

Sources & Further Reading

Pact documentation
pact-python v3 migration guide
Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Project repository
Session findings — Issue #4

This article was written with the assistance of AI tools.

I Gave the Agent the Spec and Walked Away. Here's What It Built.

Diya Burman — Wed, 10 Jun 2026 01:02:49 +0000

A Level 5 Engineer — Issue #3

If you've been following along, you know what we've built so far. Issue #1 introduced the five levels framework and the Dark Factory concept. Issue #2 got concrete — we wrote five Gherkin scenarios for an order management API before touching any implementation code, stubbed out two external dependencies with WireMock, and ran a real test suite against the whole thing.

At the end of Issue #2 I made a promise: hand the spec to an AI agent, spec only, no implementation hints, and see what it builds.

This is that issue.

The setup

The instruction I gave Claude Code at the start of the session was exactly this:

"The Gherkin scenarios in tests/features/order_creation.feature define the full behavioural contract for this API. Do not read the existing implementation in app/main.py. Build a fresh implementation that makes all 5 scenarios pass. Document your findings in FINDINGS.md as you go."

That's it. No architecture hints. No "use FastAPI." No "here's how the mock servers work." Just the spec and a documentation instruction.

The CLAUDE.md in the repo handled the rest — the guardrails, the project context, the constraint that the .feature files cannot be touched, and the format the FINDINGS.md should follow. If you missed the deep dive on CLAUDE.md in Issue #2, that file is essentially the agent's standing orders. It reads it at the start of every session.

Then I sat back and watched.

What the agent derived from the spec alone

Here's what I found interesting. Before writing a single line of code, the agent read the Gherkin scenarios and derived the entire API contract from them. Unprompted. It produced this:

POST /inventory/check/{inventory_scenario}
  → all available      → POST /payments/charge/{payment_scenario}
  → partial available  → return 207 PARTIAL_UNAVAILABLE (no charge)
  → all out of stock   → return 409 UNAVAILABLE (no charge)

And the full response shape for all five scenarios:

Scenario	status	status_code	Key fields
Success	CONFIRMED	—	`order_id`
Payment declined	PAYMENT_FAILED	402	`decline_reason`, `inventory_released: true`
Out of stock	UNAVAILABLE	409	`unavailable_items`
Partial stock	PARTIAL_UNAVAILABLE	207	`available_items`, `unavailable_items`
Payment timeout	PAYMENT_PENDING	202	`inventory_hold_minutes: 15`, `retry_count`

This is exactly right. The agent read five plain-language scenarios and extracted a precise technical contract — the order of operations, the response codes, the body fields, the retry behaviour — without being told any of it explicitly.

That's not nothing. That's the spec doing its job.

Where it got interesting — the timeout scenario

Scenario 5 is the one I was most curious about. Timeout behaviour is notoriously hard to test and easy to get wrong. The agent worked through it carefully and documented its reasoning:

PAYMENT_TIMEOUT_SECONDS=5 — per-attempt HTTP client timeout
MAX_PAYMENT_RETRIES=2 — total attempt cap, not a retry count on top of the first attempt
Worst-case wall time with 2 attempts at 5 seconds each: 10 seconds — comfortably inside the 12-second contract from the scenario
The WireMock timeout stub uses fixedDelayMilliseconds: 6000 — deliberately longer than the client timeout so the client always times out before the mock responds

That last detail is subtle and correct. If the mock delay were shorter than the client timeout, the test would be testing the wrong thing — the mock responding slowly rather than the client giving up. The agent caught this without being prompted. It's in the FINDINGS.md.

The bug it found that I had written

This is my favourite part of this issue.

The original test setup — the code I pointed Claude Code at — had a hard-coded path:

sys.path.insert(0, "/home/claude/order-api")

On my machine this would silently start mock servers with no stubs loaded. Every payment call would return a 404. Every inventory call would return a 404. The tests would fail in ways that looked like logic errors rather than a configuration problem.

The agent caught it, diagnosed the root cause, and fixed it:

# Before — hard-coded, breaks on any machine but the original
sys.path.insert(0, "/home/claude/order-api")

# After — computed dynamically, works everywhere
PROJECT_ROOT = Path(__file__).parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))

To be clear: this bug was in my code. Code I had written and shipped to the repo. The agent found it during implementation because it was trying to run the tests on a different environment and they failed in a way that forced the diagnosis.

This is a thing that happens at Level 4 that doesn't happen at Level 2. When you're implementing yourself, you don't notice the hard-coded paths because everything works on your machine. When an agent implements on a clean environment, your assumptions get exposed immediately.

My honest reaction

I'll be transparent about something. This API isn't complex. It's an order endpoint with two downstream dependencies and five scenarios. I didn't expect the agent to struggle with it, and it didn't. It hit errors, diagnosed them promptly, and moved on. Five scenarios, all passing.

What struck me wasn't the capability — it was the texture of the experience.

Watching Claude Code work, I found myself doing something I don't usually do when I'm implementing: I was evaluating. Not writing, not debugging, not context-switching. Just reading the agent's reasoning and deciding whether I agreed with it. That's a different cognitive posture entirely. It felt closer to a code review than a coding session.

I also noticed I spent the entire session approving individual commands — every file edit, every pytest run, every pip install. Claude Code asks for permission before each action by default. For this first session I let it. From the next task onward I'm going to configure it to run basic commands without checking in every thirty seconds. There's a trust-building curve here, and I'm on the early part of it.

What this proves — and what it doesn't

Five passing scenarios on a moderately simple API is not proof that Level 5 is solved. It's proof that the approach works at this scale and this complexity.

The honest question — the one this newsletter is actually tracking — is whether it holds as the system grows. Pact tests across services. CI/CD pipelines. Evals as guardrails. Contextual stewardship documents for systems with years of history and undocumented decisions baked into the architecture.

That's where the real test is. And that's where we're going next.

What I'd do differently

One thing the exercise exposed: the spec was good enough for the agent to build correctly, but I had one implicit assumption that didn't make it into the scenarios. The response shape for the success case doesn't specify that status_code should be absent — it just checks for order_id. The agent inferred this correctly, but if it hadn't, the test would have passed anyway.

That's a gap in the spec, not a gap in the agent. The lesson is the same one from Issue #2: every implicit assumption is a decision waiting to cause a bug in production. Write it down. Make it a scenario. Make the machine prove it.

Next issue: Phase 3 — adding Pact contract testing between the order service and its dependencies. What happens when the service contract and the mock stub disagree?

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Claude Code documentation
pytest-bdd documentation
Project repository
Session findings - Issue #3

This article was written with the assistance of AI tools.

The Bottleneck Moved. Did You Notice?

Diya Burman — Wed, 10 Jun 2026 00:37:33 +0000

A Level 5 Engineer — Issue #2

Preface

If you read Issue 1, you walked away with the map. Six levels, a plateau most engineers never escape, and a Dark Factory that a handful of teams are quietly running in production. If you missed it, go read it first — this one builds directly on it.

This issue is about the single most important shift that happens when you try to move from Level 3 to Level 4. Not the tools. Not the mindset. The bottleneck.

Because it moved. And most of us didn't notice.

When speed stops being the problem

For most of our careers, the bottleneck in software development was implementation speed. You had the idea, you had the design, you had the ticket — the constraint was how fast fingers could turn it into working code. That's the world we optimized for. That's why we measured velocity. That's why standups exist. That's why "10x engineer" was ever a phrase people said out loud without embarrassment.

AI blew that bottleneck wide open.

At Level 2, implementation stops being the constraint almost overnight. You're pairing with an agent and the code just... appears. Features that used to take days take hours. Hours take minutes. It feels like the problem is solved.

Except you haven't solved it. You've just exposed the one that was hiding behind it.

The new bottleneck is specification quality.

The agent can build anything you can describe precisely enough. The operative word is precisely. The moment you try to hand off a vague, half-formed idea — the kind a human developer would fill in with reasonable assumptions and a quick Slack message — the agent either hallucinates something plausible-looking that isn't what you wanted, or it freezes, or worse, it confidently builds the wrong thing all the way to completion.

The constraint is no longer your ability to implement. It's your ability to specify.

What a bad spec actually looks like

Here's the uncomfortable truth — most "requirements" we write as engineers are not specifications. They are vibes dressed up in Jira tickets.

"Add pagination to the users endpoint." That's not a spec. How many results per page? Is the default configurable? What happens when the page number exceeds the total — empty array or 404? What's the sort order? Cursor-based or offset-based? What happens to existing API consumers who aren't sending page parameters yet?

A human developer asks those questions in standup or figures them out from context. An agent working autonomously at Level 4 cannot do that. It will make a choice — silently, confidently, and consistently wrong in a way you won't catch until production.

This is why Dan Shapiro's insight about specification quality isn't just a productivity tip. It's a prerequisite for moving up the ladder at all. You cannot reach Level 4 with Level 2 specs. The system won't let you.

So I built one. Here's what happened.

I wanted to do something concrete this issue rather than just theorize. So I picked a real-world-shaped scenario — an e-commerce order management API with two external dependencies — and built it end to end with WireMock simulating the dependencies and Gherkin scenarios written before the code.

The full project is on GitHub so you can clone it and run the exact same setup on your machine. Everything below is reproducible.

The scenario

A POST /orders endpoint that talks to two external services:

A payment gateway (think Stripe) that can succeed, decline, or time out
An inventory service that can confirm stock, report out-of-stock, or report partial availability

Realistic enough to be relatable. Scoped enough to finish in an afternoon. The kind of integration complexity every backend engineer deals with.

Step 1 — Write the spec first. Actually first.

Here are the five Gherkin scenarios I wrote before a single line of implementation code:

Feature: Order Creation

  Scenario: Order is successfully created when payment succeeds and all items are in stock
    Given a registered user with id "user-123"
    And the inventory service confirms all items are in stock
    And the payment gateway will accept the charge
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "CONFIRMED"
    And the response includes an order id
    And the payment gateway received exactly one charge request
    And the inventory service received a reservation request

  Scenario: Order is rejected when payment is declined
    Given a registered user with id "user-456"
    And the inventory service confirms all items are in stock
    And the payment gateway will decline the charge
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "PAYMENT_FAILED"
    And the response status code is 402
    And the response includes the decline reason "INSUFFICIENT_FUNDS"
    And the inventory reservation is released
    And no order id is issued

  Scenario: Order is rejected when an item is out of stock
    Given a registered user with id "user-789"
    And the inventory service reports SHOE-RED-42 is out of stock
    When the user submits an order for SHOE-RED-42
    Then the order status is "UNAVAILABLE"
    And the response status code is 409
    And the payment gateway is never called

  Scenario: Order surfaces partial unavailability without auto-confirming
    Given a registered user with id "user-321"
    And the inventory service reports SHOE-RED-42 as available but BELT-BRN-M as unavailable
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the order status is "PARTIAL_UNAVAILABLE"
    And the response status code is 207
    And the payment gateway is never called
    And no order is confirmed without explicit user action

  Scenario: Order handling is graceful when the payment gateway times out
    Given a registered user with id "user-654"
    And the inventory service confirms all items are in stock
    And the payment gateway will not respond within the timeout window
    When the user submits an order for SHOE-RED-42 and BELT-BRN-M
    Then the response is returned within 12 seconds
    And the order status is "PAYMENT_PENDING"
    And the inventory is held for 15 minutes
    And the payment gateway is not retried more than 2 times

Notice what these are. Not implementation documents. Not pseudocode. They're a behavioural contract — plain-language descriptions of exactly what the system should do in specific situations, written in a format any teammate, PM, or yes — agent — can read.

The discipline of writing them first forced me to make decisions I would normally have postponed:

Do we check inventory before charging, or charge first? (Inventory first. The fourth scenario locks this in.)
What happens during partial availability — auto-fulfill what's available, or ask the user? (Ask. Encoded in scenario 4.)
What's the timeout SLA on the payment gateway? (5 seconds, with a max of 2 retries. Scenario 5 makes this testable.)
What's the response code for partial availability? (207 Multi-Status.)

Without these scenarios, every one of those decisions would have been made silently by whoever wrote the code first.

Step 2 — Stub the dependencies with WireMock

Before writing any tests you can actually run, the external services need to be simulated. This is what Dan Shapiro calls a digital twin universe — a fully simulated version of your dependencies that behaves like the real thing without the real thing's unpredictability, cost, or rate limits.

WireMock is the industry standard for this. A WireMock stub is just a JSON file describing how a service should respond:

{
  "request": {
    "method": "POST",
    "url": "/payments/charge/success"
  },
  "response": {
    "status": 200,
    "body": "{\"status\": \"ACCEPTED\", \"transaction_id\": \"txn-abc-123\", \"amount\": 134.97}"
  }
}

For the payment timeout scenario, WireMock has a built-in fixedDelayMilliseconds parameter. One line and the mock takes 6 seconds to respond:

{
  "request": {
    "method": "POST",
    "url": "/payments/charge/timeout"
  },
  "response": {
    "status": 504,
    "body": "{\"status\": \"TIMEOUT\"}",
    "fixedDelayMilliseconds": 6000
  }
}

That tiny config line is what makes scenario 5 testable. Without it, you cannot exercise timeout behaviour in a local environment without disabling network connectivity at the OS level — which I have done in the past, and it is exactly as miserable as it sounds.

Step 3 — Wire the scenarios to real assertions

Gherkin by itself is just text. To turn it into an executable test suite I used pytest-bdd, which lets each Given/When/Then line map to a Python function:

@given("the payment gateway will decline the charge", target_fixture="payment_scenario")
def pay_declined():
    return "declined"

@when("the user submits an order for SHOE-RED-42 and BELT-BRN-M", target_fixture="response")
def submit_two(user_id, payment_scenario, inventory_scenario):
    items = [
        {"sku": "SHOE-RED-42", "quantity": 1, "unit_price": 89.99},
        {"sku": "BELT-BRN-M",  "quantity": 1, "unit_price": 44.98},
    ]
    r = requests.post(f"http://localhost:{API_PORT}/orders",
                      json={"user_id": user_id, "items": items,
                            "payment_scenario": payment_scenario,
                            "inventory_scenario": inventory_scenario})
    return {"response": r}

@then("the payment gateway is never called")
def payment_not_called():
    assert not payment_log.all(), f"Expected no payment calls, got: {payment_log.all()}"

That last assertion — the payment gateway is never called — is the kind of thing that's almost impossible to verify with traditional unit tests but trivial with WireMock. WireMock records every call it receives. You assert against that log directly.

Step 4 — Run the suite

$ pytest tests/steps/test_order_creation.py -v
============================= test session starts ==============================
collected 5 items

tests/steps/test_order_creation.py::test_order_is_successfully_created_when_payment_succeeds_and_all_items_are_in_stock PASSED
tests/steps/test_order_creation.py::test_order_is_rejected_when_payment_is_declined PASSED
tests/steps/test_order_creation.py::test_order_is_rejected_when_an_item_is_out_of_stock PASSED
tests/steps/test_order_creation.py::test_order_surfaces_partial_unavailability_without_autoconfirming PASSED
tests/steps/test_order_creation.py::test_order_handling_is_graceful_when_the_payment_gateway_times_out PASSED

============================== 5 passed in 13.53s ==============================

Five for five. But getting there was educational.

What broke along the way

I want to be honest about the failures because that's where the actual learning happened. The five tests didn't pass on the first run. They didn't pass on the second run either.

Failure 1 — The "no stub matched" silent success

When a request comes in that no WireMock stub knows how to handle, the default behaviour is to return a 404. My API code did this:

try:
    pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{scenario}", ...)
    payment_result = pay.json()
except Exception:
    raise HTTPException(503, "Payment service error")

A 404 is not an exception in httpx. It's just a response. So the API would happily call pay.json(), get {"error": "No stub matched"}, and treat the entire interaction as a success — issuing an order id and confirming the order even though no real payment had been processed.

This is genuinely dangerous. A misconfigured mock would have made all my tests pass while hiding that the real service path was broken. Lesson: always explicitly check the response status from a mock. The fix:

try:
    pay = httpx.post(f"{PAYMENT_URL}/payments/charge/{scenario}", ...)
    if pay.status_code == 404:
        raise HTTPException(503, f"Payment scenario not found: {scenario}")
    payment_result = pay.json()

Failure 2 — The shared call log bug

I started with one MockServer class that held a single class-level call log. Both the payment and inventory mocks recorded into the same list. When the test asserted "the payment gateway received exactly one charge request," the inventory call was in the log but no payment call was — because of failure 1 — and the assertion was looking at the combined log.

The fix was conceptually small but architecturally important — each mock server instance gets its own call log:

def start_mock_server(port: int, mappings_dir: str) -> tuple[HTTPServer, MockCallLog]:
    stubs = [json.loads(f.read_text()) for f in Path(mappings_dir).glob("*.json")]
    log = MockCallLog()                  # ← per-instance log
    handler = make_handler(stubs, log)
    server = HTTPServer(("localhost", port), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server, log

This mirrors how real WireMock works in production — you run separate WireMock instances per service, each with its own request log. The bug was a direct consequence of cutting that corner.

Failure 3 — The fixture wiring gap

Scenarios 3 and 4 don't define a payment scenario in their Given clauses, because the payment gateway should never be called in those cases. But pytest-bdd was still expecting the payment_scenario fixture — and erroring out before the test even ran.

This is a subtle distinction worth naming. The Gherkin spec was correct. It said exactly what it should say. The error was in the test wiring that connected the spec to the assertions. The fix was a default fixture:

@pytest.fixture
def payment_scenario():
    """Default — overridden by specific Given steps."""
    return "success"

The spec stays clean. The wiring handles the case where a scenario doesn't care about a particular setup.

What this exercise actually proved to me

A few things that are now visceral rather than abstract:

Specs that the AI cannot see during the build are uniquely powerful. My scenarios live in tests/features/order_creation.feature. The implementation lives in app/main.py. When I asked an agent to modify the API, I could give it the implementation only. The spec stayed external. The agent had to make the test pass against behaviour it couldn't reverse-engineer from the assertions themselves. This is the part that genuinely changes things at Level 4.

WireMock's 404-on-no-match is a feature, not a bug. It exposes integration mistakes that would otherwise hide forever. The first time I saw a test silently succeed because of the 404 passthrough I was annoyed. Now I think it should be louder.

Writing the scenarios first changed what I built. Scenario 4 — partial availability — would not have existed if I'd written the code first. I would have implemented "all available or fail" and shipped it. Writing the spec first made me confront the question. The answer became part of the system.

Try this yourself

Everything above is in a project you can clone, run, and break. Five scenarios, two mock services, one API. Total setup time: under fifteen minutes if you have Python and pip installed.

git clone <repo-url> order-api
cd order-api
pip install fastapi uvicorn httpx pytest pytest-bdd requests
pytest tests/steps/test_order_creation.py -v

If you want to use real WireMock instead of the Python-based mock:

# Download WireMock standalone
curl -L -o wiremock.jar \
  https://repo1.maven.org/maven2/org/wiremock/wiremock-standalone/3.3.1/wiremock-standalone-3.3.1.jar

# Run two instances — the JSON mappings work as-is
java -jar wiremock.jar --port 8081 --root-dir wiremock/payment-mappings &
java -jar wiremock.jar --port 8082 --root-dir wiremock/inventory-mappings &

The WireMock mapping JSON files I wrote work in real WireMock with zero changes. That was deliberate. The Python mock is for getting started fast. The real WireMock is for when you want to scale this pattern across an actual service mesh.

Next issue: I take this same setup and hand it to an AI agent. Spec only — no implementation hints. We see what it builds, what it gets wrong, and how the spec acts as a guardrail.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — natebjones.com
Cucumber + Gherkin documentation
WireMock documentation
pytest-bdd documentation
Session findings - Issue #2

This article was written with the assistance of AI tools.

The Level 5 Engineer - The Map I Didn't Know I Needed

Diya Burman — Tue, 09 Jun 2026 19:12:34 +0000

The Level 5 Engineer Newsletter — Issue #1

Preface

Before you panic and close this tab — yes, I see you hovering!

I’m not naively chasing the thing that puts me out of a job — I’m aware of the irony. The tech world is in full meltdown mode about AI taking developer jobs right now, and, I think, most of that noise misses the point entirely.

A Dark Factory still needs someone who understands the system deeply enough to define what it should build, catch what it shouldn’t touch, and course-correct when it goes sideways. That’s not a developer writing code anymore — that’s a steward. The role doesn’t disappear; it transforms. From worker bee to architect. From implementer to the person who holds the mental model of the entire system and encodes that judgment into infrastructure that the agents can operate safely within. That shift is actually what this newsletter is about — and it deserves its own deep dive, which is coming. For now, just know:

Level 5 is the destination, not the cliff edge.

So. The Five Levels.

Dan Shapiro borrowed the structure from the NHTSA’s autonomous driving classification — five levels from “human does everything” to “machine does everything, humans not required.” Applied to software development, it maps out like this:

Level 0 — Spicy Autocomplete. You’re still writing every character. Maybe you use AI as a search engine or accept the occasional tab suggestion. The code is yours. You’re also losing ground every day to the people who aren’t at Level 0.

Level 1 — The Coding Intern. You’re delegating discrete tasks. “Write a unit test for this.” “Add a docstring here.” You’re seeing some speedup. Your job is essentially unchanged. YOU are still the bottleneck.

Level 2 — Junior Developer. This is where it starts to feel like something. You’re pairing with AI like a colleague. Flow state. Productivity you haven’t felt in years. You hand off the boring stuff and focus on the interesting parts. Here’s the trap though — Level 2 feels like you’re done. You’re not done. Most people who think they’re “using AI” are living here permanently and calling it Level 5.

Level 3 — Developer as Manager. You’re not writing much code anymore. Your AI agent has multiple tabs running at all times. Your life is diffs. You review everything at the PR level. For a lot of people, this actually feels worse than Level 2 — more overhead, less flow. And this is where almost everyone tops out. Not because they can’t go further. Because Level 3 feels like the ceiling.

Level 4 — Developer as Product Manager. The code is a black box. You write specs. You argue about the specs. You set up the right tools, define the right constraints, and then you leave for 12 hours. You come back and check if the tests passed. Dan Shapiro says he’s here. I believe him because of how he describes it — it doesn’t sound glamorous, it sounds like a different kind of hard work.

Level 5 — The Dark Factory. Named after Fanuc’s robot factory — staffed entirely by robots, lights off, no humans needed or welcome. At Level 5, you’re not really running a software process anymore. You have a system that turns specs into software, autonomously. A handful of teams are doing this today. Small teams, less than five people, shipping production software with no human-written or human-reviewed code. A prominent example would be Claude’s own engineering team, who claim that Claude Code wrote most of Claude Code.

Like what you read thus far? This post is public, so feel free to share it.

The Part That Actually Stings

Here’s what Nate B. Jones added to this framework, which made me sit with it for a while.

There’s a rigorous study showing that experienced developers using AI tools took 19% longer on tasks — while believing they were 24% faster. The gap between perceived productivity and actual productivity even has a name — the AI confidence gap. You feel faster. The clock disagrees.

The teams that are pulling away aren’t using more AI tools. They’re using AI differently. The bottleneck has moved. It’s no longer about how fast you can implement. It’s about how precisely you can specify.

The Dark Factory teams aren’t superhuman coders. They’ve built infrastructure around judgment — external behavioural scenarios that the AI cannot see during the build process, digital twin environments that simulate production dependencies safely, testing architectures designed specifically so the AI can’t reverse-engineer the passing criteria.

The rest of the industry is plateaued at Level 3, reviewing diffs, and measuring velocity in story points on a Jira board that hasn’t been groomed since Q2.

Why I’m Writing This

I’ve been a software engineer for nearly a decade. I’ve done the serverless architectures, the MLOps pipelines, the distributed systems. I’ve been to Re:Invent (3 times. Ooooh, fancy schmancy). I know the stack.

And I watched these videos and realized I was at Level 2. Maybe Level 3 on a good week. And I had been calling it “using AI effectively.”

This newsletter is the documentation of that climb — starting with understanding what the levels actually mean and why most of us have been misreading where we stand. Or at least that’s where I’m starting. I wouldn’t be surprised if my own understanding shifts incrementally as I make progress. That’s kind of the point.

Not the theory — the practice. The tools, the habits, the org-level thinking, the moments where something clicks. I’ll be honest when I’m stuck and honest when something works. No performance, no hype.

The climb starts with understanding the map. Now you have it.

Next issue: The specification quality problem — why the bottleneck has shifted and what “writing a good spec” actually means in practice.

Sources & Further Reading

Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
Nate B. Jones — The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2) · natebjones.com · Substack

Thanks for reading The Level 5 Engineer! Subscribe for free to receive new posts and support my work.

This article was written with the assistance of AI tools.