DEV Community: Denis Zontak

Break It on Purpose

Denis Zontak — Sun, 25 Oct 2026 00:29:37 +0000

An AI pipeline that catches its own bugs, explains them, and fixes them while a human makes two decisions.

By Denis Zontak, written with Claude. Originally published at zontakllc.com.

A bug in the search box

At 17:45 on October 24, 2026, the search box on our staging site started to crash. We had put the bug there on purpose: one line in a database read skipped archived clients, so a search that found their entries failed with Cannot read properties of undefined (reading 'name').

Nobody told the system. Forty-two seconds after the first error, a fix request was waiting in our pipeline. Thirty-one minutes after the first error, a pull request was ready for review: a one-line fix, and a test that proves it. A person made two decisions in between.

This is how that works, what went wrong on the way, and what it cost.

Why build it yourself

I built it because it's cool. I have full control over what it does and how it does it. As they say, homemade beats pretty much anything.

That control matters most where trust is thin. AI writes most of the code here, so the question is not whether it makes mistakes, but what catches them.

AI is a powerful tool. A system that can hallucinate, or lose track in a large context, is a reason not to trust it blindly. But in practice I do trust it: I trust that it can make a mistake, and that it can fix it too.

That is why there are only two human gates:

It can make a mistake and fix it, so I check the plan and the result, and let it work in between.

Two tools and two gates

Two small services do the work. Night Watch watches the site's logs and checks, opens incidents, asks an AI to triage them, and alerts by email, Telegram and Slack. Day Shift turns a request into a reviewed pull request: it writes a spec, builds in its own workspace, runs the project's checks, has a second agent review the change, and deploys to staging.

A person decides only two things: is this the right fix (the spec), and is it good enough to ship (the sign-off). The flow itself is plain code, every agent step is checked by something other than itself, and everything is written down, so the pipeline can stop and resume without doing work twice.

Drills: breaking staging on purpose

The staging copy of UseHourly has a drill switch. One command breaks one thing for at most 15 minutes: the health check answers 503, or answers slowly, or one data call fails. The switch only works on the staging build, only with a long token, and every start and stop writes a log line.

The first three drills taught more than any test suite:

Drill	What we saw	What we fixed
1	Caught in about a minute, and triage guessed the cause unprompted. Not labelled as a drill, and no email.	Night Watch read the request line instead of the app's own log line. The saved mail key was a pasted command.
2	Drill label and email worked, but 16 emails arrived in 4 minutes.	No catch-up mail on start, no reopening of old incidents, one thread per failure.
3	Exactly two emails: one after triage, one saying Healed. One Telegram message.	Nothing. This is the shape we wanted.

A drill is cheap, and it checks the parts no unit test reaches: the logs, the alerts, the person reading them.

We use drills like a unit test for the whole pipeline: how it responds to different errors and situations, and how we would handle them.

The flood of the second drill was a lesson of its own:

It was annoying at first, but I was also excited that the process worked and the alerts were coming in. A mixed bag, as they say.

The bug drill, minute by minute

A switch is not a bug, so the honest fix for a switch drill is "turn it off". For a real test we planted a real slip on a throwaway branch, as if a person had made it, and wrote down the right fix before anything ran. Then we let it go.

Time (EDT)	Step
17:45:24	First 500: a search hits an archived client's entry
17:46:02	Night Watch triage says code fix, citing the stack trace
17:46:06	Day Shift receives it as request R-2, 42 seconds after the first error
17:48:55	Spec v2 waits for a human. It found the place but not the cause, and wanted a weak test that hides the row
17:52:17	Sent back with one note: the cause is the new archived filter, and the entry must come back with its client name
17:53:23	Spec v3 matches the answer key. Approved
17:55:57	One-line fix; the reproduce test fails first, then the checks and code review pass
17:56:57	Fix deployed to staging
18:06–18:10	Replay: 10 of 10 searches return the entry with its client name
18:16:31	Pull request opened for sign-off

The fix matched the answer key line for line. The human did exactly what the gate is for: one send-back with the cause, one approval.

What surprised me most was how fast it was. Forty-two seconds from the first error to a fix request on my desk, and I hadn't touched anything.

What broke, and what we learned

Silence is not proof. Day Shift waited for Night Watch to say the incident was healed. But Night Watch closed a log incident only when it saw the errors stop, and with no traffic on staging there was nothing to see. After 15 minutes Day Shift decided "not healed" and built the same fix a second time. We closed the incident by hand, citing our replay as evidence, and the second build passed. The lesson: healing must be shown by replaying the request that failed, not by the absence of errors.

A spec writer must read the code it is fixing. The spec writer read the main branch, not the branch with the bug, so it had to guess the cause. It got the place right and the reason wrong. One human note fixed it, but the tool should not need that note.

Alerts need budgets. Twenty emails a day sounded generous until one bad start used sixteen of them. Drills and real incidents should not share one budget.

Small things hide in plain sight. A test that listed files with git found zero files when run from a git hook, and failed for a reason that had nothing to do with the code. A key saved as a pasted command looked fine until the first email bounced.

The numbers

The bug drill cost $2.45 of agent credit and took 31 minutes from the first error to a pull request. Without the two waits we now know how to remove, it would take about 12.

Measure	Value
First error to fix request	42 s
First error to pull request	31 min
Agent cost, bug drill (R-2)	$2.45
Agent cost, first feature ticket (R-1, CSV export)	$3.09
Human actions in the bug drill	1 send-back, 1 approve, 1 merge
Emails per drill, before and after the fix	16, then 2

We also loaded staging with 10,000 realistic accounts: 2.57 million entries and 66,881 invoices, 1.74 GB. Every page stayed under 0.2 s of data time for the heaviest account. The one slow page, an owner-only report, went from 2.9 s to 0.27 s after two small indexes.

What comes next

The next step is to stop counting on luck and start keeping score.

Honest healing: replay the failing request after a fix, and close a log incident after a quiet period that follows a deploy.
Synthetic users: short journeys on staging every few minutes (sign in, log time, search, open the week), so a bug shows up without real traffic.
A bug zoo: a set of planted bugs of different kinds, each with an answer key written before the run.
A nightly scorecard: one bug a night through the whole loop, graded on whether triage found the cause, whether the spec named it, whether the fix passed a hidden check, and what it cost.

The goal is one honest number, such as "healed 7 of 10, median 14 minutes, $3 each". Day Shift and Night Watch will be open source once that number is worth showing.

Next comes the next level: full automation, and resolving issues on its own. The sky's the limit, really. We'll start small and go big.

A word to other studios

If you have a website or an app that needs maintenance, that needs to be watched over diligently, you can benefit from these tools. They're elegant and pleasant to look at, and that's all I'll say. There's more, but that will be a surprise when you try it.

What it costs to start an open source project (my real numbers)

Denis Zontak — Wed, 21 Oct 2026 15:48:45 +0000

When I started working on UseHourly, I was just going to be the only user. But then somehow I decided it should be an open source project for other people to use. So in reality, since I'm hosting it, this project is no longer totally free for me. It's actually going to cost me money if people do start using it.

Here is what it actually costs me, line by line.

What I pay every month

What	Cost a month	Why
Cloudflare (hosting and database)	$5	The minimum paid plan. It runs the app and stores the data.
Resend (email)	$20	I upgraded to the Pro plan so I don't run into the email limits.
Domain (usehourly.app)	$1 for the first year, about $12 a year after that (about $1 a month)
Continuous integration (CircleCI)	$0 (free plan)	Runs the checks and deploys when I merge.
AI coding assistant (Claude)	$20 on the Pro plan, which should be enough for most people. I pay $100 for Max because I work on about five projects at the same time, so UseHourly is only part of that.	It wrote most of the code.
Running it (hosting, email, domain)	about $26 a month
Running it plus the assistant	about $46 a month on the $20 plan (about $126 with the $100 plan I use for all my projects)

What was free

Things I use that cost me nothing:

GitHub for the code, in a public repository.
CircleCI, on its free plan, for running the checks and deploying.
Automatic checks on every pull request (GitHub Actions), a security scan (CodeQL) and dependency updates (Dependabot).
A bot-protection widget for sign-up (Cloudflare Turnstile), ready but switched off.
npm, dev.to and Medium for sharing it.

The thing that surprised me: running UseHourly costs about $26 a month. The most expensive part is not the servers. It's the AI assistant I used to build it: $20 a month for most people. I pay $100 for the bigger plan, but I use it for about five projects at the same time, not just this one.

What I think it will cost with more users

I wanted to know what happens if people really do start using it, so I looked up the prices and made an estimate. These are estimates, not measurements. I assumed every user is active, about 20 days a month, with a small number of requests per day.

	Today	1,000 users	10,000 users
Cloudflare (hosting and database)	$5	about $5	about $5 to $6
Email (Resend Pro)	$20	$20	$20
Rough total a month	about $25	about $25	about $25 to $30

I was surprised by this. I thought the $5 plan would only handle about 100 users. By my estimate it covers thousands of active users, because the free amounts that come with the plan are large. The email plan is the cost that is already real. The prices are from Cloudflare's and Resend's own pricing pages.

What worries me

What worries me when many people sign up is that I'm going to end up with a bill of more than $100 for this thing. I guess I can afford it, but I'd rather not. I'd rather this be sponsored, really.

What I'll do if it gets expensive

If it got very expensive, I'll probably have to hold some kind of fundraiser. Like if it's over $100. I already have sponsorship links on my site. Hopefully I would just start by posting a message to all users, asking if they can donate a little bit to cover the monthly cost, if it's too large.

I built a small notice for exactly this. It's off right now. If I switch it on, people see a short note on the Today page saying what hosting costs, with a link to my sponsors page. It only shows inside the app. I don't send marketing email, and I don't want to start.

Conclusion

So that's what it costs to start an open source project, in my case. It's free to use, and it's not free for me. Right now it's about $26 a month to run, counting hosting, email and the domain, and my estimate says it stays about that even with thousands of users. If I'm wrong and it ever gets past $100, I'll say so openly, and I'll ask for help.

If you use it and it helps you, you can sponsor it on GitHub: https://github.com/sponsors/Zontak-LLC. And if you've paid for a free project of your own, I'd like to hear what it cost you.

This article was dictated by me. Claude, an AI assistant, helped put it in order, looked up the prices, made the cost estimate and drafted a few sentences, which I edited. The numbers about what I pay are mine.

Guardrails so AI-written code doesn't turn into spaghetti

Denis Zontak — Wed, 21 Oct 2026 14:05:14 +0000

I like open source. I've been using open source projects for many years, and there are some very, very good ones. I wanted to try building one myself, just to see how to set it all up.

At the time I was doing some freelance work for a friend at a company called MagSmith (magsmith.com), and I needed to manage my timesheets. I looked at some of the open source time trackers, the top three alternatives: solidtime, Kimai and TimeTagger, and they're okay. But I don't like it when a product says it's free but has a bunch of paid stuff on top. You want to use a feature and it ends up requiring an upgrade. I hate that.

So I wrote my own, the way I like it. It's called UseHourly, and I use it myself.

My first AI coding experience

My first AI coding experience came at work. We started using AI about a year ago. I started with an AI coding assistant in the IDE, and at that time AI was still not there. It couldn't produce code sophisticated enough for what we needed, and it made way too many mistakes. But it was helpful.

About six months ago, around the time when the Claude Opus model came out, it became good enough. It could implement the code completely from a Jira ticket, and then implement a list of tests by itself, like Cucumber tests for the integration tests. All by itself.

That was my first experience, and I love it.

Why I added guardrails

When I started UseHourly, I was kind of vibe coding it. Nothing was really going bad yet. It wasn't spaghetti code yet, because it's a reasonably small project. But I know that in a larger project it becomes a problem. I know this from experience.

So instead of letting it keep running code until it becomes spaghetti, I added some guardrails to let it know what not to do and what to do. Check for code smells, write good code, maintain good coding practices, enforcement, and stuff like that. Basically, how to keep a clean, tidy code base with well-written code and good test coverage. That's the goal.

What I did

First, I wrote the rules down in a file called CLAUDE.md, in the project. The assistant reads it every time it starts. It says how the project is built, how money and dates are stored, and what "done" means.

Second, I made one command that runs all the checks. It's the only definition of "the checks". I run it, the CI runs it, and the git hooks run it, so nobody can say "it passed on my side". If it fails, the work isn't done.

Third, I turned the rules into checks that fail the build. No any types, no ignoring the type checker, no TODO without an issue number, no copy-pasted code, no unused files, and a minimum test coverage that can only go up, never down. The AI can't talk its way past them. If it takes a shortcut, the build says no and it fixes it.

Fourth, I write the tests the way a person uses the app. At one point the AI rewrote a screen and a button did nothing. The tests passed, because none of them ever clicked it. So I added a test that clicks every button on every page, and a rule that every bug fix comes with a test that fails without the fix.

And now I review the code before I commit it, not after. A fresh reviewer reads the change cold, because the assistant that wrote the code is the worst reader of it.

Conclusion

So that's my story. I love open source, and I love working with AI coding assistants. But you can't just let the AI keep going. On a bigger project it turns into spaghetti, and I know that from experience.

So I put guardrails in place. Rules that tell it what to do and what not to do, and checks that make sure it does. The goal is a clean, tidy code base with well-written code and good test coverage.

I've turned these checks into a small tool, so anyone can add them to their own project. It's open source: https://github.com/Zontak-LLC/railing

I dictated this article and had Claude help me put it in order and check the facts against my code.