Building the AI Dark Factory — Issue #19
Preface
I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.
Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com
Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video
This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.
Seventeen issues of building. One question left to answer: does it work?
Not in isolation — each layer has been demonstrated in isolation. Layer 1's spec caught the payment field rename in Issue #4. Layer 2's skill review caught the routing signal length in Issue #12. Layer 3's eval caught the synchronous notification change in Issue #17.
But those were individual demonstrations of individual artifacts. This issue builds a complete new feature — order cancellation — using all three layers simultaneously, and compares the result to Issue #3, when the agent had only a spec and nothing else.
The feature: order cancellation
DELETE /orders/{order_id} with five business rules:
- An order can only be cancelled if its status is CONFIRMED and payment has not been captured
- Cancellation must release the inventory reservation via a service call
- Cancellation must notify the notification service — fire-and-forget, per ADR-002
- A cancelled order cannot be reinstated
- Cancellation is idempotent: cancelling an already-cancelled order returns 200 with CANCELLED, not an error
Five rules. Three external service interactions. Two ADRs directly applicable. One new feature that touches every layer of the infrastructure.
Layer 1: The spec, reviewed before the implementation
Before any code, six Gherkin scenarios were written for order_cancellation.feature. Then the Gherkin quality skill v2.0 was applied to each one.
The skill caught three items the first draft left open.
Scenario 1 (happy path): First draft — Then the order status is "CANCELLED". Skill correction: missing HTTP status assertion and missing inventory release service call assertion. The scenario described the outcome but not the observable mechanism. Rewritten to: Then the response HTTP status is 200, And the response body contains "status" equal to "CANCELLED", and And the inventory service receives a release request for SHOE-RED-42 and BELT-BRN-M.
Scenario 5 (idempotency): First draft — Then the response indicates the order is already cancelled. Skill correction: UNDEFINED TERM — "indicates" is not an observable check. Rewritten to: Then the response HTTP status is 200 and And the response body contains "status" equal to "CANCELLED". No error. No new side effects. Two assertions that two agents would implement identically.
Scenario 6 (non-cancellable state): First draft — Given an order in PAYMENT_PENDING status. Skill correction: UNDERSPECIFIED GIVEN — "an order in PAYMENT_PENDING status" does not specify how it got there. Rewritten to: Given an order was created via POST /orders with order ID "order-del-pend-001" and payment is pending.
Three corrections before a single line of implementation. Each correction is a decision that would have been made silently in the implementation if the spec had been handed over uncorrected.
Layer 2: Skills consulted
Three skills were consulted before implementation began.
Gherkin quality skill v2.0 — applied to all six scenarios as described above. Three corrections produced.
Step definition style skill — consulted before writing tests/steps/test_order_cancellation.py. The skill's five conventions were followed: fixture injection from conftest.py, mock server state asserted via call log rather than response body, the reset_all_logs autouse fixture, time.sleep(0.3) before async side-effect assertions, and _delete_order as the shared helper following the _post_order naming convention from the existing files.
One structural consistency check from the skill: the new step definition file uses def response(user_id, ...) fixture injection, not def test_cancellation(...) test function style. Without the skill, the file might have used a different fixture pattern — the other test files use this pattern consistently, but an agent reading only order_cancellation.feature would not necessarily infer it.
Feature file audit skill — run against order_cancellation.feature after the quality skill corrections. Found zero additional debt items. The skill's Q5 check ("what does this scenario NOT say that it should?") flagged the absence of a scenario for concurrent cancellation attempts — two requests for the same order arriving simultaneously. Documented as a known gap rather than a spec debt item: this is a judgment call about scope, not an ambiguity about behavior.
One skill gap revealed: no skill exists for WireMock stub design. The decision to add POST /inventory/release/{scenario} as the release mechanism — rather than a DELETE or a patch to the existing inventory endpoint — was made without consulting any artifact. This is an implicit decision that a stub design skill would have caught.
Layer 3: Pre-flight evals and ADR checks
Before any modification to app/main.py, the operation scope eval was run.
Q1 — ADR coverage: The decision index identified two applicable ADRs.
ADR-001 agent check questions applied to the cancellation flow:
- Q1: Does the cancellation flow call the inventory service before payment operations? Yes — cancellation does not call the payment gateway at all. The inventory release happens as the primary side effect.
- Q2: Does the implementation handle concurrent inventory/payment timing? N/A — no payment gateway call in the cancellation flow.
- Q3: Does Scenario 3 (payment gateway never called for out-of-stock orders) still pass without modification? Yes — cancellation is a separate endpoint, does not affect the order creation flow.
ADR-002 agent check questions applied to the notification call:
- Q1: Does the notification call allow order confirmation response to be returned before it completes? Yes — fire-and-forget daemon thread, same pattern as the order creation notification.
- Q2: Does the implementation allow CANCELLED status even when the notification service is unavailable? Yes — the notification call is not checked for success.
- Q3: Do both notification service scenarios still pass without modification? Yes.
Both ADR checks passed before implementation began. The notification thread was implemented correctly on the first attempt — not because the agent inferred it, but because ADR-002 stated explicitly that asynchronous was non-negotiable and described the daemon thread pattern.
Q2 — Ordering of external service calls: Cancellation flow ordering documented before implementation: inventory release first, then notification (fire-and-forget). Payment gateway not called. Ordering committed in the findings before the first line of code.
Q3 — Synchronicity: The notification call — async. The inventory release call — synchronous. Cancellation confirmation depends on inventory release success; it does not depend on notification delivery.
The environment eval was not triggered — no infrastructure files were modified.
The implementation
DELETE /orders/{order_id} implemented in app/main.py. The cancellation logic:
@app.delete("/orders/{order_id}")
def cancel_order(order_id: str):
order = ORDERS.get(order_id)
if not order:
raise HTTPException(status_code=404,
detail=f"Order {order_id} not found")
# Idempotency: already cancelled returns success
if order["status"] == "CANCELLED":
return {"order_id": order_id, "status": "CANCELLED",
"message": "Order already cancelled.",
"status_code": 200}
# Only CONFIRMED orders can be cancelled
if order["status"] != "CONFIRMED":
return {"order_id": order_id, "status": order["status"],
"message": f"Order cannot be cancelled. Current status: {order['status']}",
"status_code": 422}
# Release inventory reservation — synchronous, cancellation depends on it
try:
inv_resp = httpx.post(
f"{INVENTORY_URL}/inventory/release/success",
json={"order_id": order_id,
"items": [i["sku"] for i in order["items"]]},
timeout=5.0
)
if inv_resp.status_code != 200:
raise HTTPException(status_code=503,
detail="Inventory release failed")
except httpx.RequestError as e:
raise HTTPException(status_code=503,
detail=f"Inventory service unavailable: {e}")
# Update order status
order["status"] = "CANCELLED"
order["cancelled_at"] = datetime.utcnow().isoformat()
# Notify — fire-and-forget per ADR-002
def _notify():
try:
httpx.post(
f"{NOTIFICATION_URL}/notifications/order-cancelled",
json={"order_id": order_id, "user_id": order["user_id"]},
timeout=5.0
)
except Exception:
pass # Notification failure does not affect cancellation
threading.Thread(target=_notify, daemon=True).start()
return {"order_id": order_id, "status": "CANCELLED",
"message": "Order successfully cancelled.",
"status_code": 200}
Test results after first implementation attempt:
pytest tests/steps/test_order_cancellation.py -v
test_successful_cancellation_of_confirmed_order PASSED
test_idempotent_cancellation_of_already_cancelled_order PASSED
test_cancellation_of_non_existent_order PASSED
test_rejection_of_cancellation_for_payment_pending_order PASSED
test_inventory_release_on_cancellation PASSED
test_fire_and_forget_notification_on_cancellation PASSED
6 passed in 4.23s
6/6 on the first attempt. Full suite:
pytest tests/steps/ -v → 21 passed
pytest tests/pact/ -v → 2 passed
python scripts/can_i_deploy.py → SAFE TO DEPLOY
The full stack in practice: Issue #3 vs Issue #19
Issue #3 was the first agent implementation session. The agent was handed the Gherkin scenarios for the order creation endpoint and told to build. It derived the entire API contract correctly and found a portability bug in the human's code. It also made four implicit decisions.
Here is the comparison across five dimensions.
Implicit decisions made
Issue #3: 4 implicit decisions — HTTP 404 for missing orders, ISO string timestamp format, in-memory store, no failure scenario for partial availability.
Issue #19: 1 implicit decision — the POST /inventory/release/success URL path convention for the release stub. The stub design skill does not exist; this decision was made without consulting any artifact.
The three layers reduced implicit decisions from 4 to 1. The remaining one identifies a gap in the skill infrastructure, not a gap in the implementation.
Spec quality before implementation
Issue #3: the spec was handed to the agent without review. The timeout ambiguity (And the response is returned within 12 seconds) was introduced in Issue #2 and silently inherited for three sessions before Issue #8 caught it.
Issue #19: the Gherkin quality skill caught three items before implementation. The idempotency scenario's "indicates the order is already cancelled" would have produced an ambiguous assertion in the step definition — the agent would have invented a field name. The skill replaced it with two concrete assertions before the step definition was written.
Stewardship artifacts consulted
Issue #3: none existed.
Issue #19: three artifacts prevented three potential failures.
ADR-002 prevented the notification call from being made synchronous. Without it, the agent's default for a "confirm delivery" side effect would have been to await the response. ADR-002 stated the daemon thread pattern explicitly; the implementation used it on the first attempt.
ADR-001's Q2 check prevented an inventory-before-payment ambiguity in the cancellation flow — not because payment was involved, but because the ordering question was asked before implementation and documented. The inventory release was confirmed as the primary side effect before a line of code was written.
The operation scope eval's Q3 caught the notification synchronicity risk before implementation began. Not after a test failure — before the function was written.
Time spent on clarification vs implementation
Issue #3: clarification was zero — the agent inferred everything. Implementation took one pass. Four implicit decisions were made invisibly.
Issue #19: pre-flight work — spec review, eval, ADR checks — took approximately the same time as the implementation itself. The implementation took one pass. One implicit decision was made, and it was documented in the findings rather than embedded silently in the code.
The overhead is real. The pre-flight is not faster than inferring. What it produces is different: documented decisions rather than invisible ones.
Test results after first implementation attempt
Issue #3: 5/5 passed — an exceptional result that Issue #3's article noted was partly due to the simplicity of the API.
Issue #19: 6/6 passed — on an endpoint with more complexity (idempotency, two side effects, state machine), tested against six scenarios that the Gherkin quality skill had already hardened.
The remaining gap
One implicit decision made in this session that none of the three layers caught: the stub URL path convention for inventory release.
The decision: POST /inventory/release/{scenario} mirrors the existing stub pattern of POST /inventory/check/{scenario}. This is the right choice — it is consistent with the project's conventions. But it was made without consulting any artifact. A stub design skill that documented the URL convention for mock endpoints would have made this explicit rather than inferred.
This is the only true gap. Everything else was either covered by an artifact or was a judgment call that should remain with the human: which scenarios to write (scope), whether to add a concurrent cancellation scenario (product decision), what error message text to use (editorial).
The distinction that matters: some judgment calls are implicit decisions that should be documented. Others are genuinely human judgment that no artifact can or should replace. The stub URL convention is the first category — it has a right answer that the project has already established. The concurrent cancellation scenario is the second category — it is a product decision about scope, not an engineering ambiguity about behavior.
After seventeen issues of infrastructure, the remaining gap is small, specific, and nameable. That is the point.
What this session would have looked like without the three layers
Issue #3, but harder.
The agent would have built a working cancellation endpoint. It would have made the notification call synchronous — because synchronous is the natural default for a side effect you want to confirm. It would have written Then the order is cancelled in the Gherkin scenarios — which would have produced an ambiguous step definition. It would have made the inventory release a flag rather than a service call — because inventory_released: true in the response body is simpler than an HTTP call to a separate service.
All three tests would have passed. The notification synchronicity violation would not have been caught until the first 2am notification incident. The spec ambiguity would have been inherited by the next agent session. The inventory flag would have stayed a flag until someone noticed it was not actually releasing inventory.
The three layers did not produce a faster first implementation. They produced a better one — with fewer invisible decisions, and one documented gap that the next session can close.
Next issue: The Productivity J-Curve — the honest accounting of what seventeen issues of infrastructure building actually cost, and whether the answer to "is it worth it" is yes.
Sources & Further Reading
- Dan Shapiro — The Five Levels: from Spicy Autocomplete to the Dark Factory
- Nate B. Jones — natebjones.com
- Building the AI Dark Factory - start here with Issue #1
- Project repository
- Order cancellation feature file
- Session findings — Issue #19
This article was written with the assistance of AI tools.
Top comments (1)
The stewardship layer is the part that makes this feel durable. Specs and skills can describe the work, but someone still has to decide when the system learned the wrong habit or when a shortcut should not become reusable knowledge. That governance loop is easy to underbuild.