AI Made Code Cheap. Trust Is Still Expensive.

What it takes to let an AI assistant change code that stops databases and reboots production hosts.

Every engineering team now has a tool that writes code faster than anyone can review it. That quietly flips what's scarce. For decades the expensive part of software was writing it. Now it's knowing whether what was written is safe to run.

Nowhere is that gap wider than in infrastructure. A bug in a web app shows someone a broken page. A bug in infrastructure automation stops the wrong database, or reboots a host halfway through an upgrade. An AI assistant can produce a plausible change to that code in seconds. Proving it wrong can take an outage.

Most teams respond in one of two ways. They keep AI away from production automation, or they let it in and review every line by hand. The first gives up the speed. The second turns senior engineers into the bottleneck, and tired reviewers miss things.

We took a third route. A one-line bug shows why it works.

During a maintenance run, our automation refused to touch a database cluster because it found "no snapshot in the last 48 hours". The log line directly above it showed a backup taken 2.9 hours earlier. I handed the log to an AI assistant, and it traced the failure to a single condition. A snapshot counted only if its name used the long form of the node's role, and this host's snapshots used the abbreviation. The comment above the function promised both spellings. The code had implemented one.

The fix was one line. Writing it was never the hard part. The hard part was the question that came next:

How do you trust a one-line change to code that stops databases, takes snapshots and reboots production hosts, and make that trust cheap enough to earn on every change?

That question reshaped how we build our automation. This article covers the five principles that came out of it. All five are in use today, and two still have pieces being built. None of them are about AI. All of them are why the AI is useful.

What "AI-native" means for infrastructure

"AI-native" usually describes products. For an infrastructure codebase, it means something narrower and more useful: a codebase where an AI can make a change and you can know, quickly and cheaply, whether that change is safe.

That comes down to four properties. Notice that generating code isn't one of them.

Each principle below builds one of these properties. None depends on a particular model or tool, and every one of them also makes the codebase better for the humans who work in it.

The codebase nobody can "just run"

The system runs rolling maintenance on self-managed, replicated database clusters spread across more than one public cloud. Apache Airflow orchestrates each workflow as a DAG, Ansible executes commands on hosts over SSH, and a Python layer handles discovery, health checks and cloud snapshots.

The scheduler never touches a host directly. Every action crosses two more layers and, often, a cloud API, and each boundary is somewhere a bug can hide.

A rolling upgrade, simplified:

1. Discover the cluster and verify its topology.
2. Confirm a recent backup exists, or take one.
3. For each secondary: mute alerts, stop the database, upgrade, start, check replication lag, unmute.
4. Repeat for the arbiter.
5. Step the primary down, then upgrade it the same way.
6. Run post-checks and return the primary to its preferred zone.

Every step is safe on its own. The danger is in the order, the preconditions and what happens when one step half-succeeds.

The real dependencies are a live database cluster, SSH keys, several cloud SDKs, an alerting system, a secrets service and a running scheduler. For a long time, the only honest test of a change was to run it, which means the feedback loop was measured in maintenance windows.

That's the worst possible environment to add an AI assistant to. If the only checker is production, a faster code generator doesn't make you faster. It makes you riskier at a higher rate.

Everything that follows serves one goal: moving the moment you learn a change is wrong from the last step of the old loop to the second step of the new one.

1. Make verification cheaper than generation

The cost of a check decides how often it runs. A check that needs cloud credentials runs in CI, sometimes. A check that runs in two seconds on a laptop runs after every edit, and an AI assistant can run it itself.

So the first goal was a suite that needs nothing: no network, no credentials, no Airflow. The technique is to replace heavy modules in sys.modules before any source file imports them. Pytest loads conftest.py first, which makes it the right place:

# tests/conftest.py: runs before any test module is imported
import sys, types

class FakeTask:
    """Stands in for any operator: records its arguments and supports >>."""
    def __init__(self, **kwargs):
        self.kwargs = kwargs
    def __rshift__(self, other):
        return other
    def __rrshift__(self, other):
        return self

def install_stub(module_name, **attrs):
    module = types.ModuleType(module_name)
    module.__dict__.update(attrs)
    sys.modules[module_name] = module

install_stub("ops_lib.operators", ShellTask=FakeTask, PlaybookTask=FakeTask)
install_stub("ops_lib.secrets", get_secret=lambda name: "test-value")

The same pattern covers Airflow, the database driver, every cloud SDK and the secrets client. Each stub provides only the names touched at import time. Production code loads as usual and never knows the difference.

The result: 292 tests in about two seconds, a coverage gate that fails the build on regression, and a test toolchain of exactly two packages, pytest and pytest-cov.

Two seconds is not a vanity number. It changes who does the checking. The assistant runs the full suite after each edit, reads the failure and corrects itself before I see a diff. A twenty-minute integration suite breaks that loop; a two-second suite makes "run the tests" a reflex rather than a decision.

If verification is expensive, you ration it. If it's cheap, you delegate it.

The trade-off is real: stubs can drift from the libraries they imitate. Keep them minimal, stub only what's imported, and back them with a slower suite that runs against real software (principle 5 covers that).

2. Test the runbook, not just the functions

Unit tests verify the parts. Operational incidents rarely live in a single part. They live between parts: a step in the wrong order, a step that quietly disappeared, a workflow file that no longer loads.

Start with the cheapest high-value test there is. Airflow imports every DAG file to build its schedule, and a file that fails to import simply vanishes from the scheduler. One parametrised test imports every DAG under the stubs:

WORKFLOWS = sorted(p.stem for p in DAGS_DIR.glob("*.py"))
KNOWN_BROKEN = {"legacy_restart"}  # tracked defect: remove once fixed

@pytest.mark.parametrize("name", [
    pytest.param(n, marks=pytest.mark.xfail(strict=True, reason="known defect"))
    if n in KNOWN_BROKEN else n
    for n in WORKFLOWS
])
def test_workflow_imports(name):
    importlib.import_module(f"dags.{name}")

The detail worth copying is strict=True. It documents known breakage without hiding it, and it fails the build on the day someone fixes the DAG but forgets to update the list. Technical debt becomes a tracked, executable fact instead of tribal knowledge.

Make the code read like the runbook. The DAGs are built from small, composable templates, so the code that runs in production has the same shape as the procedure an engineer would follow by hand:

(
    node.mute_alerts()
    >> node.stop("database")
    >> node.upgrade_binary()
    >> node.start("database")
    >> node.check_replication()
    >> node.unmute_alerts()
)

This matters more with AI than without it. A model modifies code most reliably when the structure carries the intent. Fifty lines of raw operator wiring invite a plausible mistake; six lines that mirror the runbook make one obvious.

Then turn the runbook's rules into executable checks. Every runbook has rules that reviewers enforce by eye: mute alerts before stopping the database, step the primary down before touching it, confirm a backup before the first destructive step. Expressed against the task graph, those rules become tests:

# A sketch of the API we're building, not working code yet
@invariant("alerts are muted before the database stops and unmuted after")
def mute_wraps_stop(graph):
    for stop in graph.tasks(action="stop", service="database"):
        assert graph.upstream(stop).has(action="mute_alerts")
        assert graph.downstream(stop).has(action="unmute_alerts")

This is the layer we're building next. It needs the stub >> to record edges rather than pass them through, which is a few lines of work. Its payoff is large: a change that reorders safety steps, whether a human or a model wrote it, fails in two seconds instead of during a maintenance window.

3. Turn every incident into a test

Tests written from imagination encode what you expected to happen. Tests written from incidents encode what actually happened. The second kind is rarer and far more valuable, so the rule is simple: a production fix doesn't merge without a test that reproduces the failure.

Model fixtures on what production returns. Cluster status documents and the discovery step's output live in tests/fixtures/, shaped exactly like the real responses. Tests replay them rather than invent data. The snapshot-name bug from the opening is next in line: one fixture holding snapshot names in every format our environments produce.

Test the failure path as seriously as the happy path. One health check used to catch authentication and timeout errors and report the cluster type as UNKNOWN. The on-call engineer saw "Unrecognized configuration UNKNOWN" and had to guess. The fix carries the real cause into the error message, and a test asserts that it does. An error message is a user interface for the worst moment of someone's night. It deserves tests.

Treat cost as a correctness property. A backup step located a VM by listing every VM in a group, then reading one network interface per VM until an IP matched. In a large group, a single run spent five minutes on those reads and began collecting HTTP 429 throttling errors. The fix lists interfaces once, in one paginated call, and fetches only the matching VM.

Both versions return the same VM. Only the cost differs, and in production, cost is what fails first.

Note what a conventional test would have checked: "returns the right VM". The old code passed that. The meaningful assertion is about how many calls it takes:

def test_vm_lookup_lists_interfaces_once(cloud):
    vm = find_vm_for_ip(cloud, "10.0.3.17")
    assert vm.name == "host-a"
    cloud.network.list_interfaces.assert_called_once()
    cloud.compute.get_vm.assert_called_once_with("host-a")

That change shipped with tests for every new path: lookup by name, each fallback, the cache, the throttle logging. Coverage for the file went from 12% to 34% in one merge request. Exhaustive path coverage is exactly the kind of tedious, mechanical rigour an AI assistant does well, as long as you ask for it explicitly.

4. Treat the repository as the onboarding document

An AI assistant is a permanent new hire: fast, well read, and on its first day, every day. Whatever your team knows but hasn't written down doesn't exist for it. The same is true for the next engineer you hire, so the work pays off twice.

State the non-negotiables where they can't be missed. Our README states the rule in one sentence: every change ships with tests, and the suite stays green. It also explains how to run the suite, how branches are named and how a change moves from staging to production. An assistant that reads the README follows the process without being reminded.

Say which code is canonical. The README lists every DAG with a status (current, needs rework, TODO or utility) and names one of them as the reference that new work should copy. Without that signal, "add a step like the others" is ambiguous, and copying a deprecated pattern is a perfectly reasonable interpretation.

Delete dead code. I removed a deprecated upgrade DAG, an unused precheck DAG and a set of unused Ansible roles. In an AI-assisted codebase, dead code isn't inert. Every file is a potential example, and dead code is a set of examples you've stopped maintaining.

Agree on intent before implementation. Larger changes begin as a short written proposal in an openspec/ folder. An assistant can help draft it, but a human signs off before any code is written, and the implementation is reviewed against it. It's cheaper to correct a paragraph than a pull request.

Write commit messages for the next reader, who is increasingly a model. Ours follow one shape: what failed, why, what changed and how it was verified.

Failed: a backup step took five minutes and hit API throttling. Why: the VM lookup read one network interface per VM. Change: list interfaces once in a paginated call and fetch only the matching VM. Verified: tests cover every lookup path and assert the call count.

Six months on, git log explains the code better than any comment could, to people and to models alike.

5. Make every change reversible without a deploy

Tests prove a change is correct in the world you modelled. Production is a different world. So every risky change also needs a way back that doesn't depend on shipping more code.

Fail toward the known-good path. To cut API load further, snapshot listing moved to a cloud query service that answers for a whole group of resources in one call, from its own quota. The old listing remains as the fallback, and it's used in every case where the new path can't be fully trusted:

• the SDK isn't installed
• the query errors
• the result is empty, which more likely means missing permissions than no snapshots
• a timestamp can't be parsed

The last case is the instructive one. A snapshot assigned the wrong age could read as a fresh backup that doesn't exist, and a phantom backup is the one error this code must never make. The design principle is asymmetry: when the fast path is uncertain, pay for the slow path that's known to be right.

Every uncertain branch ends at the same place: the path that was already proven in production.

Put the switch in configuration, not in code. The new path is gated on an Airflow Variable that defaults to on. If production disagrees with the tests, an operator flips it off in the UI and the next run takes the old path. No deploy, no revert, no incident call to schedule one.

Decouple the change from the platform's release cycle. The worker image couldn't be rebuilt that week, so the new SDK is installed into a virtualenv for the single task that needs it. Nothing else pays the cost, and the change merged safely ahead of the image.

Give every destructive workflow a dry-run mode. Each upgrade and reboot DAG takes a test parameter. When it's true, the orchestration runs end to end (discovery, checks, ordering, branching), but every Ansible task that would touch a host is skipped. A dry run is a test you can point at reality.

The next step is a disposable environment: CI starts Airflow, a containerised replica set and SSH-reachable container "hosts", runs the real DAGs against them and tears everything down. The fast suite tells you a change is plausible. The slow one keeps the fast one honest.

The ladder is the whole strategy in one picture. The bottom rungs are where an AI assistant lives: cheap enough to run after every edit. The top rungs are where humans and real infrastructure take over.

Where the model excels, and where judgment stays human

The division of labour turned out to be sharp. The assistant is exceptional at reading, searching and being thorough. It knows nothing about production that you haven't told it.

Where it excelled:

• Reading at scale. It traced the snapshot-name bug from one log line to one condition in a large utility module.
• Recognising expensive patterns. Given the timings, it identified the one-read-per-VM lookup as the source of the throttling.
• Exhaustive rigour. Asked explicitly for "a test for every path this change adds", it covered the happy path, pagination, a missing SDK, an empty result and an unparseable timestamp.
• Precise commit messages. It writes problem, cause, fix and verification with a consistency most of us manage only on good days.

Where the tests caught the model. One change added diagnostic logging that imported a cloud SDK module the offline suite didn't stub. Three tests failed before they could even construct the object under test. The suite flagged it in seconds. The fix made the diagnostics optional, so an observability feature could never take down the work it observes, and added a proper stub so the new code was exercised rather than skipped.

Where a human had to decide:

• Production facts. A new warning fired when the cloud API's remaining-request budget ran low. Its threshold assumed a budget in the thousands; in production the header read a few hundred, so every response logged a warning and buried the signal it existed to raise. No amount of code reading reveals that. Only production logs do. The fix lowered the threshold and limited it to one warning a minute.
• Attribution under pressure. A discovery failure appeared just after a change landed, and the change was reverted. The actual cause was the run's input, which carried the wrong port. Reverting was still correct: restore a known-good state first, diagnose second. Small, well-tested changes are what make that kind of revert cheap.
• Which risk is acceptable. A model can propose five valid fixes. Choosing the one whose failure mode you can live with on a production database is engineering judgment, and it should remain so.

The leash, not the brakes

There's an intuition that the safe way to use AI on infrastructure code is to use less of it. Our experience points the other way. Tests aren't brakes on the model. They're the leash that lets it range further, because every step it takes is checked before it can matter.

The one-line fix from the opening took a minute to write. What made it safe to ship took months to build: a suite that runs anywhere, checks that understand the runbook, a repository that explains itself, and a way back when production disagrees.

The teams that get the most from AI won't be the ones that generate the most code. They'll be the ones that made verification cheap enough to run on every change.

Start with the cheapest rung: a test suite that runs offline in seconds. Everything else builds on it.

Build the leash first. Then let it run.