Break It on Purpose

An AI pipeline that catches its own bugs, explains them, and fixes them while a human makes two decisions.

By Denis Zontak, written with Claude. Originally published at zontakllc.com.

A bug in the search box

At 17:45 on October 24, 2026, the search box on our staging site started to crash. We had put the bug there on purpose: one line in a database read skipped archived clients, so a search that found their entries failed with Cannot read properties of undefined (reading 'name').

Nobody told the system. Forty-two seconds after the first error, a fix request was waiting in our pipeline. Thirty-one minutes after the first error, a pull request was ready for review: a one-line fix, and a test that proves it. A person made two decisions in between.

This is how that works, what went wrong on the way, and what it cost.

Why build it yourself

I built it because it's cool. I have full control over what it does and how it does it. As they say, homemade beats pretty much anything.

That control matters most where trust is thin. AI writes most of the code here, so the question is not whether it makes mistakes, but what catches them.

AI is a powerful tool. A system that can hallucinate, or lose track in a large context, is a reason not to trust it blindly. But in practice I do trust it: I trust that it can make a mistake, and that it can fix it too.

That is why there are only two human gates:

It can make a mistake and fix it, so I check the plan and the result, and let it work in between.

Two tools and two gates

Two small services do the work. Night Watch watches the site's logs and checks, opens incidents, asks an AI to triage them, and alerts by email, Telegram and Slack. Day Shift turns a request into a reviewed pull request: it writes a spec, builds in its own workspace, runs the project's checks, has a second agent review the change, and deploys to staging.

A person decides only two things: is this the right fix (the spec), and is it good enough to ship (the sign-off). The flow itself is plain code, every agent step is checked by something other than itself, and everything is written down, so the pipeline can stop and resume without doing work twice.

Drills: breaking staging on purpose

The staging copy of UseHourly has a drill switch. One command breaks one thing for at most 15 minutes: the health check answers 503, or answers slowly, or one data call fails. The switch only works on the staging build, only with a long token, and every start and stop writes a log line.

The first three drills taught more than any test suite:

Drill	What we saw	What we fixed
1	Caught in about a minute, and triage guessed the cause unprompted. Not labelled as a drill, and no email.	Night Watch read the request line instead of the app's own log line. The saved mail key was a pasted command.
2	Drill label and email worked, but 16 emails arrived in 4 minutes.	No catch-up mail on start, no reopening of old incidents, one thread per failure.
3	Exactly two emails: one after triage, one saying Healed. One Telegram message.	Nothing. This is the shape we wanted.

A drill is cheap, and it checks the parts no unit test reaches: the logs, the alerts, the person reading them.

We use drills like a unit test for the whole pipeline: how it responds to different errors and situations, and how we would handle them.

The flood of the second drill was a lesson of its own:

It was annoying at first, but I was also excited that the process worked and the alerts were coming in. A mixed bag, as they say.

The bug drill, minute by minute

A switch is not a bug, so the honest fix for a switch drill is "turn it off". For a real test we planted a real slip on a throwaway branch, as if a person had made it, and wrote down the right fix before anything ran. Then we let it go.

Time (EDT)	Step
17:45:24	First 500: a search hits an archived client's entry
17:46:02	Night Watch triage says code fix, citing the stack trace
17:46:06	Day Shift receives it as request R-2, 42 seconds after the first error
17:48:55	Spec v2 waits for a human. It found the place but not the cause, and wanted a weak test that hides the row
17:52:17	Sent back with one note: the cause is the new archived filter, and the entry must come back with its client name
17:53:23	Spec v3 matches the answer key. Approved
17:55:57	One-line fix; the reproduce test fails first, then the checks and code review pass
17:56:57	Fix deployed to staging
18:06–18:10	Replay: 10 of 10 searches return the entry with its client name
18:16:31	Pull request opened for sign-off

The fix matched the answer key line for line. The human did exactly what the gate is for: one send-back with the cause, one approval.

What surprised me most was how fast it was. Forty-two seconds from the first error to a fix request on my desk, and I hadn't touched anything.

What broke, and what we learned

Silence is not proof. Day Shift waited for Night Watch to say the incident was healed. But Night Watch closed a log incident only when it saw the errors stop, and with no traffic on staging there was nothing to see. After 15 minutes Day Shift decided "not healed" and built the same fix a second time. We closed the incident by hand, citing our replay as evidence, and the second build passed. The lesson: healing must be shown by replaying the request that failed, not by the absence of errors.

A spec writer must read the code it is fixing. The spec writer read the main branch, not the branch with the bug, so it had to guess the cause. It got the place right and the reason wrong. One human note fixed it, but the tool should not need that note.

Alerts need budgets. Twenty emails a day sounded generous until one bad start used sixteen of them. Drills and real incidents should not share one budget.

Small things hide in plain sight. A test that listed files with git found zero files when run from a git hook, and failed for a reason that had nothing to do with the code. A key saved as a pasted command looked fine until the first email bounced.

The numbers

The bug drill cost $2.45 of agent credit and took 31 minutes from the first error to a pull request. Without the two waits we now know how to remove, it would take about 12.

Measure	Value
First error to fix request	42 s
First error to pull request	31 min
Agent cost, bug drill (R-2)	$2.45
Agent cost, first feature ticket (R-1, CSV export)	$3.09
Human actions in the bug drill	1 send-back, 1 approve, 1 merge
Emails per drill, before and after the fix	16, then 2

We also loaded staging with 10,000 realistic accounts: 2.57 million entries and 66,881 invoices, 1.74 GB. Every page stayed under 0.2 s of data time for the heaviest account. The one slow page, an owner-only report, went from 2.9 s to 0.27 s after two small indexes.

What comes next

The next step is to stop counting on luck and start keeping score.

1. Honest healing: replay the failing request after a fix, and close a log incident after a quiet period that follows a deploy.
2. Synthetic users: short journeys on staging every few minutes (sign in, log time, search, open the week), so a bug shows up without real traffic.
3. A bug zoo: a set of planted bugs of different kinds, each with an answer key written before the run.
4. A nightly scorecard: one bug a night through the whole loop, graded on whether triage found the cause, whether the spec named it, whether the fix passed a hidden check, and what it cost.

The goal is one honest number, such as "healed 7 of 10, median 14 minutes, $3 each". Day Shift and Night Watch will be open source once that number is worth showing.

Next comes the next level: full automation, and resolving issues on its own. The sky's the limit, really. We'll start small and go big.

A word to other studios

If you have a website or an app that needs maintenance, that needs to be watched over diligently, you can benefit from these tools. They're elegant and pleasant to look at, and that's all I'll say. There's more, but that will be a surprise when you try it.