Every team I have worked on has a folder full of runbooks that nobody trusts. The runbooks were written during or right after an incident, by someone who was tired and wanted to make sure the next person did not have to suffer the same way. They were accurate the day they were written. Then the system changed, the runbook did not, and the gap between the two grew quietly until the next incident, when the person on call discovered that step four now points at a service that was renamed six months ago.
The problem with runbooks is not that people do not write them. The problem is that a runbook is a piece of documentation pretending to be a piece of software. It describes a procedure that a human is supposed to execute perfectly under pressure, at the exact moment when a human is least able to execute anything perfectly. The format is wrong for the job. A procedure that matters enough to write down is a procedure that should be able to run itself, or at least check its own steps as a human walks through them.
Claude Code changed how I think about runbooks by letting me treat them as executable artifacts instead of static text. A runbook is no longer a wiki page. It is a structured document that a skill can read, validate, and step through, verifying preconditions before each action and confirming outcomes after each one. The human stays in the loop for the decisions that need judgment, and the tedious verification work happens automatically. Here is how the workflow came together.
Why Static Runbooks Fail Exactly When You Need Them
A static runbook fails in a specific and predictable way. It works fine during the calm review when someone reads it and nods. It fails during the incident, when the person following it hits a step that no longer matches reality and has to decide, in the middle of an outage, whether to trust the runbook or trust their own read of the situation.
That decision is the worst possible thing to ask of someone at 3 a.m. They are already stressed, already short on context, already worried about making things worse. Handing them a document that might be wrong in ways they cannot easily detect adds a whole second layer of uncertainty on top of the incident itself. The runbook was supposed to reduce cognitive load. Instead it becomes another thing to second guess.
The root cause is that runbooks drift and nothing catches the drift. Code has tests. Infrastructure has drift detection. Runbooks have neither. They sit in a wiki, slowly diverging from the system they describe, and the only detector of that divergence is a human discovering it the hard way during the next incident.
A runbook that has not been executed since it was written is not a runbook. It is a guess about a procedure, and guesses are exactly what you do not want during an incident. The value of a runbook is proportional to how recently it was proven to work.
The workflow below is the one I built to close the drift, keep the runbooks honest, and turn them from documents people distrust into procedures people lean on. The Claude Code skills carry the verification burden so the human can carry the judgment.
If you want the broader context on how I handle the moment things break, the Claude Code for Incident Response workflow describes the wider response process that these runbooks plug into.
The Structure Skill
The first skill turns a freeform runbook into a structured one. Given a wiki page or a rough set of notes, it produces a runbook where every step has a defined shape.
Each step has a precondition, an action, and a verification. The precondition is the state the system must be in before the step is safe to run. The action is the concrete thing that happens. The verification is the observable signal that confirms the action did what it was supposed to do. A step without all three is incomplete, and the skill flags it.
The structure forces the author to think through parts of the procedure that freeform text lets them skip. It is easy to write "restart the worker" in a paragraph. It is harder to write the precondition that makes a restart safe, and harder still to write the verification that confirms the restart actually helped rather than just cycling the problem. The structure surfaces those gaps while the author still has the context to fill them.
The skill also captures the intent of each step. Alongside the mechanical action, it records why the step exists and what would go wrong if the step were skipped. That intent is what lets a human make a good decision when the runbook meets a situation its author did not anticipate.
The Validation Skill
The second skill keeps the runbook honest over time. On a schedule, it walks each runbook and checks that the references still resolve.
Every runbook is full of references to the real world. Service names, dashboard links, metric queries, config keys, host groups, escalation contacts. Each of those references can rot. The validation skill checks them against the live system and reports the ones that no longer resolve. A service that was renamed, a dashboard that was deleted, a metric that no longer exists, an on call contact who left the company.
The report is the drift detector that static runbooks never had. Instead of a human discovering the broken reference during an incident, the validation skill discovers it during a quiet Tuesday and files it as ordinary maintenance work. The runbook gets fixed while the fix is cheap.
The skill does not try to fix the references itself. Automated repair of a runbook is exactly the kind of change that needs a human to confirm intent, because a renamed service might mean the step should point somewhere new, or it might mean the step is now obsolete. The skill surfaces the drift and leaves the judgment to a person.
The validation run is also a coverage report. It shows which runbooks have been validated recently and which have gone stale, which makes it easy to see at a glance where the operational documentation is trustworthy and where it is not.
If you want to see how this same idea applies to code rather than procedures, the approach in Claude Code for Dependency Management uses a similar continuous verification loop to keep a moving target under control.
The Execution Skill
The third skill walks a human through a runbook during an actual incident. It reads the structured runbook and drives the procedure one step at a time.
Before each step, the skill checks the precondition against the live system. If the precondition does not hold, the skill stops and explains what it found rather than blindly proceeding. This is the safety property that a static runbook cannot offer. A human under pressure will skip a precondition check. The skill will not.
The skill executes the action only with human confirmation for anything that changes state. For read only steps that gather information, it proceeds automatically and presents the results. For steps that restart, scale, fail over, or otherwise mutate the system, it presents the action and waits for a person to approve it. The division keeps the human in control of the consequential decisions while removing the tedium from the informational ones.
After each step, the skill runs the verification and reports whether the expected signal appeared. If the verification fails, the skill halts and surfaces the discrepancy. A runbook that proceeds past a failed verification is a runbook that turns a small problem into a large one, and the skill is built to refuse that.
The whole execution is logged with timestamps. When the incident is over, the log is a precise record of what was done, when, and what the system looked like at each step. That record feeds the postmortem and becomes evidence for improving the runbook.
The Feedback Skill
The fourth skill closes the loop after the runbook is used. It takes the execution log and compares what happened to what the runbook expected.
Every real execution is a test of the runbook. If a step took longer than expected, if a verification was ambiguous, if a human had to improvise around a gap, the feedback skill captures it. The output is a set of proposed improvements to the runbook, grounded in what actually happened rather than in what someone imagined at the desk.
The proposals go into review the same way any change does. A human decides whether to accept them, and the runbook improves with each use instead of decaying with each use. This is the inversion that matters. A static runbook gets worse every time the system changes. An executable runbook that captures feedback gets better every time it is run.
The feedback skill also flags runbooks that are used often, because a frequently used runbook is a signal about the system. If a procedure runs every week, the underlying problem is probably worth fixing so the procedure is never needed again. The runbook usage data points at the automation opportunities that would remove the toil entirely.
How the Workflow Runs in Practice
A new runbook starts as rough notes, usually written right after an incident while the memory is fresh. The structure skill turns the notes into a structured runbook with preconditions, actions, and verifications. A human reviews it and fills the gaps the structure surfaced.
From that point, the validation skill checks the runbook on a schedule and files drift as maintenance work. The runbook stays aligned with the system without anyone having to remember to check it. The drift that used to hide until the next incident now shows up as routine tickets.
When an incident hits, the person on call runs the runbook through the execution skill. The skill verifies preconditions, walks the steps, confirms outcomes, and logs everything. The human makes the judgment calls and approves the state changing actions, while the skill handles the checking.
After the incident, the feedback skill compares the execution to the runbook and proposes improvements. The runbook gets a little better, and the next person to run it inherits a procedure that is more accurate than the one before. Over many cycles, the runbook library becomes a set of procedures that people actually trust, because they have been proven to work recently and repeatedly.
What This Workflow Did to My Practice
The first change is that runbooks stopped rotting. The validation loop catches drift while it is cheap, so the library no longer contains the landmines that used to detonate during incidents. When I open a runbook now, I trust it, because I know it was validated against the live system within the last cycle.
The second change is in how incidents feel. Walking a structured runbook that checks its own preconditions is a fundamentally calmer experience than reading a wiki page and hoping. The skill carries the verification anxiety, which frees the human to think about the actual problem rather than about whether step four still applies.
The third change is that the runbook library became a map of operational toil. The usage data from the execution skill shows exactly which procedures run often, and those are the procedures worth automating out of existence. The runbooks stopped being just documentation and became a prioritized list of the work that would make the on call rotation quieter.
The fourth change is cultural. When runbooks are trustworthy and improve with use, people write more of them, because writing one is no longer a thankless act of documentation that will decay. The library grows because the incentives finally point the right way.
For the full picture of how I run production systems with Claude Code, the complete series on DEV.to covers every workflow I depend on, from incident response to dependency management to runbook automation.
FAQ
Does this replace the human on call?
No, and it is not meant to. The skill handles verification and tedium. The human handles judgment, approves every state changing action, and decides what to do when the runbook meets a situation its author did not foresee. The goal is to remove the parts of incident work that a human does badly under pressure, not the parts a human does well.
What about procedures that are too risky to automate at all?
Those are exactly the procedures that benefit most from the precondition and verification checks, even if every action stays manual. A risky procedure walked step by step, with each precondition confirmed before the human acts, is far safer than the same procedure read off a wiki page. Automation of the checking is valuable even when automation of the actions is not.
How do I start if I have fifty stale runbooks already?
Do not try to convert all of them at once. Run the validation skill across the whole library first to find out which runbooks are least rotted, and convert those, because they are closest to correct. Convert the rest as they get used. A runbook that has not been used in a year can wait until someone needs it.
What if a runbook needs a step the skill cannot execute?
Then that step stays manual, with the skill handling the precondition and verification around it. Not every action needs to be automated for the workflow to add value. A runbook that is half automated and half checked is still enormously better than a static document that is neither.
The shift from static runbooks to executable ones is one of those changes that seems small until you have lived with it for a few months. The incidents that used to involve a frantic hunt through a wiki now involve a calm walk through a procedure that checks itself. The runbooks that used to rot now improve with every use. The trade cost me a few weeks of building the skills, and it has paid that back in every incident since, in the form of nights where the procedure just worked and I got to go back to sleep.
Top comments (0)