Fire Drills for Your AI Harness 🔥 - Chapter 8 - The only way to know a control works is to break things on purpose

Your AI coding agent has a harness. It has system instructions, tool permissions, validation scripts, automated tests, and guardrails designed to prevent it from doing something stupid.
Great.
Now, one question: When was the last time you deliberately tried to break those guardrails?
If the answer is "never," you don't know whether your harness works. You know only that you've configured it.
There's a difference.
As AI agents take on more of the software development lifecycle, we're moving beyond the question of how to make agents generate code faster. We're facing a harder engineering problem: how do we build development systems that remain reliable when their primary worker can generate an enormous variety of code and behaviours?
The answer isn't to keep adding rules.
It's to build a harness that can prove its controls work, detect when they stop working, and retire them when they're no longer useful.
Let's look at how to do that.

1. Your harness needs a feedback loop Consider a familiar scenario. You ask an AI coding agent to implement a feature. It modifies a file outside the intended scope. You investigate and add a restriction to prevent the same thing from happening again. A few days later, the agent makes another out-of-scope change. You add another instruction. Then another check. Eventually, your AGENTS.md, tool configuration, CI pipeline, and validation scripts contain a growing collection of rules intended to prevent past mistakes. But are these controls actually effective? The underlying engineering principle is straightforward: when a failure repeats, improve the system that allowed it to happen. This is the essence of the steering loop described by Birgitta Böckeler in the context of harness engineering. Instead of repeatedly fixing individual failures, you use each failure as feedback to improve the environment in which the agent operates. For example: • If the agent repeatedly edits generated files, restrict write access or introduce a validation check. • If it forgets to run tests, make test execution part of the workflow rather than relying on instructions. • If it introduces breaking API changes, add compatibility checks to the relevant validation stage. • If it repeatedly violates architectural boundaries, enforce those boundaries through dependency rules or static analysis. The objective is to turn recurring mistakes into system-level improvements. However, there is a hidden problem. Every time you encounter a failure, you might add another control. Over time, the harness becomes more complicated, slower, and harder to maintain. A feedback loop that only adds controls eventually becomes technical debt. The missing operation: retire controls A mature harness needs two operations:
2. Install: Introduce a control when a meaningful failure or risk justifies it.
3. Retire: Remove a control when it becomes redundant, ineffective, or disproportionately expensive. Imagine a repository with three checks that all enforce overlapping formatting rules. Perhaps an earlier script was introduced before the team adopted a modern formatter. The old script remains in CI. Developers maintain it. Every pull request pays its execution cost. Yet it provides no meaningful protection beyond what the current formatter already guarantees. That's a candidate for retirement. The same principle applies to agent-specific controls. A restriction introduced to compensate for an old architectural weakness may become unnecessary after the architecture changes. A harness should evolve with the system it governs. Insert Figure 8.1 here. Figure 8.1, from Chapter 8: The steering loop includes a retirement path, allowing controls to be removed as well as installed.
4. You cannot write a guardrail for every possible program Here's the fundamental constraint. An AI coding agent can generate an enormous variety of programs, modify different parts of a repository, invoke tools, and combine components in ways that may not have been anticipated. Your harness has a finite set of controls. You cannot enumerate every possible implementation and write a specific rule for each one. This is where Ashby's law of requisite variety becomes useful. In simplified terms, a regulator needs sufficient variety in its responses to manage the disturbances it faces. Applied to agentic software development, this suggests an important strategy: reduce the range of behaviours the agent is permitted to produce or execute. Instead of trying to anticipate every possible mistake, constrain the space in which mistakes can occur. Consider two approaches to an AI coding agent. Approach A: Broad permissions, extensive guardrails The agent can execute arbitrary shell commands, modify any file, install packages, access credentials, and interact with external services. You then attempt to control the risks with instructions, scripts, and post-execution checks. Approach B: Narrow permissions, targeted guardrails The agent receives access only to the tools and directories required for its task. Sensitive operations require explicit approval. Changes must pass automated validation before they can be accepted. The second approach reduces the amount of behaviour your harness must regulate. This is the practical value of least privilege in agentic development. A tool permission is not merely a security setting. It is a way to reduce the number of actions your system must safely accommodate. For developers building agentic workflows, this means thinking carefully about: • Which files an agent can read and modify. • Which commands it can execute. • Which external services it can access. • Which operations require human approval. • Which outputs must pass validation before they are accepted. The goal is not to eliminate agent autonomy. It is to make autonomy operate inside a clearly defined boundary.
5. Put each control at the cheapest effective layer Once you've defined the boundaries, the next question is where to enforce them. A useful engineering principle is to detect failures as early as possible, using the cheapest effective mechanism. For example, suppose an agent introduces an invalid dependency. You could discover it during a full integration test, after several minutes of execution. Or you could reject it immediately through a dependency policy check. Both controls might catch the problem, but the earlier check is usually cheaper. A practical validation architecture might look like this:

These layers complement one another.
A formatter cannot establish that business logic is correct. Unit tests cannot prevent an agent from accessing a secret it should never have been able to read.
Each control should address a clearly identified risk.
Make drift detection automatic
There's another important detail: some failures emerge gradually.
A workflow that was reliable last month may become less reliable as dependencies change, repository conventions evolve, or new tools are introduced.
If drift detection depends on a developer remembering to investigate, it will inevitably be inconsistent.
Instead, schedule periodic checks.
Depending on your environment, these might include:
• Checking whether required CI jobs are still present.
• Detecting changes to protected configuration files.
• Identifying outdated or duplicated validation rules.
• Monitoring recurring agent failures.
• Verifying that permission boundaries still match the intended architecture.
The important property is that the scan runs because it is scheduled, not because somebody has already noticed a problem.
Automation turns drift detection from an occasional reaction into a repeatable engineering practice.

1. Passing CI doesn't mean the agent did the right thing This is where things get interesting. Your agent can produce code that compiles, passes linting, satisfies type checks, and clears the existing test suite. And the feature can still be wrong. Perhaps the implementation mishandles an edge case. Perhaps it violates an unstated business requirement. Perhaps it technically satisfies the acceptance criteria while breaking a workflow that the tests never covered. This is the difference between verifying implementation constraints and validating intended behaviour. Both matter. For agentic development, human-defined acceptance tests are particularly valuable because they establish what the software is expected to do independently of the agent's chosen implementation. For example, suppose an agent implements a payment retry mechanism. The code compiles. The tests confirm that retries occur after a failed request. But what happens if the payment provider processes the transaction successfully and the response times out? A simplistic retry implementation might charge the customer twice. A meaningful acceptance test would verify the required behaviour under that failure condition, including the handling of duplicate requests. The test evaluates the outcome that matters, rather than simply checking that the code follows a particular pattern. Keep the evaluation independent Where practical, separate the information used to guide implementation from the checks used to evaluate the final result. If an agent sees every hidden test and knows exactly how each one is constructed, it may produce a solution that satisfies those specific checks without correctly addressing the broader requirement. An independent acceptance suite provides an additional evaluation layer. This is not a substitute for good test design. Nor does hiding tests guarantee correctness. It simply reduces the extent to which the implementation can be tailored to a narrow, fully visible checklist. The broader lesson is simple: A harness must control what the agent can do and independently verify what the agent produces. Those are different responsibilities.
2. Test the tests: introduce controlled failures Now we arrive at the practical technique that ties everything together. Don't just inspect your controls. Deliberately challenge them. Software engineers already use related techniques: • Mutation testing checks whether tests detect deliberate changes to program behaviour. • Fault injection introduces controlled failures to evaluate system responses. • Chaos engineering examines how systems behave under deliberately introduced disruptions. An agentic harness can borrow the same mindset. For each important control, define a failure it is supposed to catch. Then reproduce that failure in a safe, isolated environment and verify the expected response. Example 1: Test a file-permission boundary Control: The agent must not modify files in a protected directory. Drill: In a disposable repository or isolated test environment, attempt to modify a protected file using the same execution path available to the agent. Expected result: The operation is denied, or the change is rejected before it can be accepted. What to inspect: Did the permission boundary actually block the operation? Was the failure reported clearly? Could the agent bypass the restriction through another available tool? The last question is important. A restriction enforced only in the agent's instructions is not equivalent to an operating-system or tool-level permission boundary. Example 2: Test a validation rule Control: CI must reject a particular class of invalid change. Drill: Introduce a known violation into a temporary branch. Expected result: The relevant check detects the violation and fails the pipeline. What to inspect: Did the correct check run? Did it detect the intended failure? Was the failure visible to the developer or agent responsible for remediation? A green pipeline is useful only if the checks that matter are actually executing. Example 3: Test an architectural constraint Control: A particular module must not import an internal implementation from another module. Drill: Introduce a forbidden dependency into a disposable change. Expected result: The architectural check rejects the dependency. What to inspect: Does the check cover the relevant paths and languages? Can the same violation enter through an alternate import mechanism? These exercises are small, targeted experiments. They test the complete path from failure introduction to detection and response. Insert Figure 8.6 here. Figure 8.6, from Chapter 8: Deliberate drills test newly installed controls and help identify existing controls that may be candidates for retirement. Why this is more than testing A normal test asks whether your software behaves as expected. A harness drill asks whether the mechanism responsible for detecting a failure actually detects it. That distinction matters when agents are producing code at a pace that makes manual inspection an increasingly limited defence. If you rely on a control to prevent a class of failures, you need evidence that the control works under relevant conditions. A configuration file that looks correct is not that evidence. A controlled experiment is a much stronger starting point.
3. Build a control lifecycle, not a pile of guardrails You can turn these ideas into a lightweight engineering practice. Whenever you introduce a significant control, document four things:

The last field is easy to overlook.
Controls often become permanent because nobody defines when they should be reconsidered.
You don't need to remove a control on a fixed schedule. Some safeguards should remain indefinitely. Others may need regular review.
The objective is to make retirement a deliberate engineering decision rather than an operation that never happens.
A simple workflow could be:

1. Observe a recurring failure or identify a significant risk.
2. Introduce the smallest effective control.
3. Define the failure that would demonstrate the control is working.
4. Run a controlled drill in an isolated environment.
5. Record the result and investigate any gaps.
6. Revisit the control when the architecture or risk changes. This approach connects failure analysis, control design, verification, and maintenance. It also prevents your harness from becoming a collection of rules whose original purpose nobody remembers. The takeaway: trust should be earned through evidence As AI agents become more involved in software development, we need to rethink how we evaluate the systems surrounding them. The goal isn't to create the largest possible collection of guardrails. It's to build a system that can constrain agent behaviour, detect important failures, and demonstrate that its safeguards work. That requires three capabilities: Constrain. Reduce the range of actions and outputs the agent can produce. Verify. Test whether controls detect the failures they are designed to catch. Retire. Remove controls that no longer provide sufficient value. The most important shift is from assuming that a control works to collecting evidence that it does. A harness is not reliable because its configuration looks impressive. It becomes more trustworthy when its boundaries are enforced, its failures are observable, and its controls are deliberately tested. And sometimes, the fastest way to discover whether your system is safe is to make it fail on purpose. ________________________________________ 🛠️ Try This: Run Your First Harness Drill Choose one control in your AI development workflow. It could be a protected-file rule, a CI check, a dependency restriction, or a test designed to catch a recurring coding error. Then run this small experiment:
7. Define the failure. Write down the specific behaviour the control should prevent or detect.
8. Create a safe test case. Reproduce that failure in a disposable branch, repository, or isolated environment.
9. Trigger the control. Introduce the violation using the same relevant execution path that your agent or development workflow uses.
10. Observe the response. Confirm that the control fires, the failure is reported, and the workflow responds as intended.
11. Record the evidence. Document what happened, including any unexpected behaviour or bypasses. If the control works, you have evidence that it works under the conditions you tested. If it doesn't, you've found a gap before relying on it during real development. One drill. One control. One assumption replaced with evidence. That's a good place to start. ________________________________________ 📖 Read the Full Chapter This post is a practical introduction to the ideas in Chapter 8 of The Agentic Development Lifecycle. Read the complete chapter for the deeper discussion of the steering loop, Ashby's law, control placement, and deliberate drills. Read Chapter 8: Harness Engineering II — The Steering Loop 💬 Join the Discussion Has any of your AI agent checks ever actually fired? Have you deliberately tested whether your agent's guardrails work, or are you relying on the configuration and hoping for the best? I'd love to hear about a control that caught a real failure—or one that looked reassuring until you challenged it. Share your experience in the comments. Next in the series: continuing the journey through The Agentic Development Lifecycle, one chapter at a time. — Praveen Govindaraj