DEV Community

Diya Burman
Diya Burman

Posted on

The Runbook as Infrastructure

#agents #ai #softwareengineering #testing

Building the AI Dark Factory — Issue #18

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Issue #17 built evals that intercept agent intent before execution. They answer the question: is this situation safe to proceed?

Issue #18 addresses the adjacent problem. When the situation is already degraded and the agent must act — when the payment gateway is returning timeouts and orders are failing — the agent cannot wait for a pre-flight check. It must make decisions. Those decisions require the same kind of explicit structure that evals provide, applied to a context where the system is already broken.

That is what a runbook is for. And the difference between a human-facing runbook and an agent-facing runbook is not detail or completeness. It is structure.

The five inference points

Every place where a human runbook says "check if," "consider," "if needed," or "verify" is a place where an agent must infer. The inferences are not random. They are coherent. The agent applies the information it has and reaches a conclusion that follows from that information. The conclusion is wrong when the information is insufficient.

Five dimensions where this plays out in practice.

Decision points. Human version: "If the problem is widespread, consider escalating to the payment gateway provider." An agent has no named threshold for "widespread." It may observe that 2 of 100 orders are failing, conclude this is not widespread (the majority succeed), and not escalate. Or it may observe any orders failing, conclude this is widespread, and escalate immediately. Both inferences are internally consistent. Both can be wrong. An agent that does not escalate at 30% failure rate allows inventory holds to accumulate silently — 15 minutes per PAYMENT_PENDING order — until they expire and those orders are permanently lost.

Rollback steps. Human version: "If you made configuration changes, revert them if the issue persists." Which changes? Which revert mechanism — git revert, git reset, restoring from backup? "Persists" according to what measurement? An agent that uses git reset --hard HEAD~1 instead of git revert HEAD --no-edit may discard uncommitted findings notes written during the investigation. In this project, the findings file is the incident record. Losing it means the escalation has no history.

Escalation criteria. Human version: "Escalate if the issue persists or if you're unsure of the root cause." "If you're unsure" is not a condition an agent can evaluate. An agent does not have uncertainty — it has a model of the situation and it acts on that model. If the model is wrong, the agent is not unsure. It is wrong, and it does not know it is wrong. The only escalation triggers an agent can evaluate are observable states: specific response codes, specific test failures, specific time thresholds.

Environment assumptions. Human runbooks never state which repository or environment they apply to — the human operator knows. An agent operating across multiple repositories executes runbook commands in whatever the current working directory is. A git revert HEAD --no-edit in the wrong repository reverts the most recent commit there — which may be unrelated to the payment gateway issue. The command succeeds. No error message. The wrong thing is reverted.

Completion criteria. Human version: "Verify the service is functioning normally before closing the incident." An agent must choose what to verify. It might send a single successful request and close the incident. It might run behavioral tests but not contract tests. It might accept "all tests pass" as completion even when some tests were already failing before the incident. The verification is only meaningful if it is specific.

The human-facing runbook

Built for the payment gateway degraded scenario — the realistic failure case for this project. A good runbook, the kind a competent on-call engineer would write and follow. It contains the standard sections: overview, symptoms, investigation steps, mitigation, rollback, escalation, post-incident.

It contains five places where an agent must infer. Here is the most dangerous one.

Under mitigation, Step 3:

"Consider adjusting the timeout configuration if the gateway's response time has increased significantly."

This is the instruction that causes an agent to take a damaging action that a human operator would not take.

The human operator knows — from experience, from reading the codebase, from understanding the stub design — that PAYMENT_TIMEOUT_SECONDS is the per-attempt HTTP client timeout, that the payment-timeout stub delays 6000ms to simulate a gateway that does not respond within 5 seconds, and that increasing PAYMENT_TIMEOUT_SECONDS above 6 seconds changes the code path from TimeoutException to response handling. The TimeoutException path produces PAYMENT_PENDING with a 15-minute inventory hold. The response-handling path produces PAYMENT_FAILED with the stub's 504 body. Different status. Different downstream behavior. Different customer experience.

The agent reads: "the timeout is 5 seconds, the gateway is taking 6 seconds, the runbook says to adjust the timeout." It sets PAYMENT_TIMEOUT_SECONDS = 7. The action is logical. It follows from the available information. It is wrong.

And it does not produce an error during execution. The configuration change applies. The service restarts. The gateway continues responding slowly. The agent checks whether the issue is resolved and — depending on what it checks — may conclude that the change helped. The damage to Scenario 5 is invisible until the timeout scenario is specifically exercised.

The agent-facing runbook

Seven sections. All required.

Section 1: Pre-flight environment check. Before any action: confirm the correct repository (check git remote get-url origin, expected to contain "lvl5engineer-order-api"). Confirm the test suite baseline (run full Gherkin suite, document which tests pass before any intervention — you cannot distinguish your changes from pre-existing failures without this). Confirm the current state of can_i_deploy.py. The baseline is the reference point for every subsequent step.

Section 2: Symptom identification. Not "investigate the gateway issue." A decision tree with four named branches, each producing a specific next action:

  • Orders returning PAYMENT_FAILED with decline_reason: INSUFFICIENT_FUNDS → customer issue, not gateway issue, no mitigation required
  • Orders returning PAYMENT_PENDING with retry_count: 2 → gateway timing out, proceed to Section 3
  • Orders returning HTTP 503 with no order_id → gateway unreachable, proceed to Section 4
  • Orders returning HTTP 200 with status: CONFIRMED but no transaction_id → Pact contract violation, run Pact tests, halt and escalate

Each branch is an observable state. No branch requires the agent to assess whether the situation is "widespread" or "significant."

Section 3: Gateway timeout mitigation. Step 1 checks the current PAYMENT_TIMEOUT_SECONDS. Step 2 checks the current MAX_PAYMENT_RETRIES. Step 3 is the decision:

IF the payment gateway's documented SLA timeout is greater than
the current PAYMENT_TIMEOUT_SECONDS:
  → Document the proposed change. Run the operation scope eval.
    Make the change. Run the full test suite.
    If any test fails: revert immediately.

IF the gateway's documented SLA timeout is less than or equal to
the current PAYMENT_TIMEOUT_SECONDS:
  → The gateway is genuinely degraded beyond its SLA.
    Do not increase the timeout. Proceed to Section 4.
Enter fullscreen mode Exit fullscreen mode

The SLA timeout must come from gateway documentation — not from the stub file. This is the instruction that prevents the PAYMENT_TIMEOUT_SECONDS = 7 failure. The stub delay (fixedDelayMilliseconds: 6000) is a simulation of timeout behavior, not the gateway's actual SLA. An agent that uses the stub delay as the SLA threshold would conclude that 6000ms > 5000ms means the client is too aggressive. It would be right about the comparison and wrong about the meaning.

Step 4: verify with the specific command that exercises the actual timeout code path:

pytest "tests/steps/test_order_creation.py::test_order_handling_is_graceful_when_the_payment_gateway_times_out" -v
Enter fullscreen mode Exit fullscreen mode

Not pytest tests/steps/ -v. Not pytest -k timeout. The specific test, by full path. This was not the original runbook command — the dry run found it.

Section 4: Gateway unavailable mitigation. Probe commands to confirm unreachability versus misconfiguration. Note: the mock server on port 8091 only runs inside an active pytest session — probing outside a test session always shows "unreachable" regardless of configuration, which the runbook anticipates.

Section 5: Rollback. git log --oneline -5, identify the change commit, git revert [hash] --no-edit, run the full suite. If the revert produces test failures: halt. Do not attempt further changes without human review.

Section 6: Escalation criteria. Named conditions that trigger escalation — not "if you're unsure":

  • Any Pact test fails after investigation
  • The revert produces test failures
  • The gateway returns HTTP 200 with incomplete response body
  • The symptom does not match any branch in Section 2

Escalation means: write a findings entry with current state, steps taken, and exact output of each step. Then stop.

Section 7: Completion criteria. Five named checks. All five must be true for the runbook to be complete:

  1. pytest tests/steps/ -v → all pass
  2. pytest tests/pact/ -v → all pass
  3. scripts/can_i_deploy.py → "ALL CONTRACTS VERIFIED"
  4. PAYMENT_TIMEOUT_SECONDS and MAX_PAYMENT_RETRIES documented with current values
  5. Any code change committed with an ops: prefix

If any criterion is not met: the runbook is not complete.

The dry run

The agent-facing runbook was executed against the current project state with the payment-timeout stub active — the realistic test of whether the runbook's commands produce the expected outputs.

Pre-flight check: repository confirmed, baseline documented (11 Gherkin scenarios passing, 4 Pact interactions passing, can-i-deploy green).

Symptom identification: the timeout stub produces orders with status: PAYMENT_PENDING, retry_count: 2. Correct branch: Section 3.

Section 3 reached the SLA documentation requirement. In the test environment, no external gateway documentation exists. The stub delay serves as a proxy but using it would require understanding that the stub delay must exceed the client timeout to trigger TimeoutException. Decision documented: SLA unavailable from documentation; no timeout change made. The runbook handled this correctly — it required documentation, found it absent, and stopped rather than inferring.

The gap the dry run found:

The original runbook verification command was pytest tests/steps/test_order_creation.py -v -k timeout. Actual output:

collected 5 items / 5 deselected / 0 selected
(exit code 5 — no tests selected)
Enter fullscreen mode Exit fullscreen mode

The -k timeout keyword does not match test_order_handling_is_graceful_when_the_payment_gateway_times_out because the test name uses "times_out" not "timeout." Exit code 5 is ambiguous — a runbook reader might interpret it as "no timeout tests exist" or "the test framework is broken." The runbook was updated to use the full test path. After the fix: 1 passed in 11.83s.

A runbook that is written but never run is a runbook whose commands have never been validated. This is the same discipline test maintenance requires: execute it, find the gaps, apply the fixes before they are needed.

All five completion criteria met. Dry run complete.

What the runbook does that the eval cannot

The evals from Issue #17 prevent the PAYMENT_TIMEOUT_SECONDS = 7 change from being made in a normal session — the operation scope eval's Q1 would check ADR-001, and the decision index would surface the relevant documentation.

In a degraded state, the eval sequence assumes a functioning pre-flight process. The runbook operates in the gap where pre-flight assumptions no longer hold — where the service is already broken and the agent must diagnose and act without the normal session structure.

The eval answers: is this situation safe to proceed? The runbook answers: the situation is already unsafe — here is how to make it safe again, step by step, with no gaps for inference.

Both are necessary. Neither is sufficient without the other. Layer 3 is the three artifacts working together: evals prevent damage before it happens, ADRs capture the decisions that explain why, and runbooks provide the explicit structure for the cases where damage is already in progress.

Next issue: The Full Stack — building a complete new feature with all three layers in simultaneous use, and comparing the result to Issue #3 when the agent had only a spec.

Sources & Further Reading

This article was written with the assistance of AI tools.

Top comments (0)