TL;DR
- For about a month, every time an AI wrote into our Backlog tickets, a second AI reviewed the write-up right after. 175 reviews produced 819 comments.
- Only 74 comments were high severity. Around 90% were minor: wording issues, or points the reviewer itself could not confirm.
- What worked was not checking everything. It was narrowing what gets checked, and keeping execution in human hands.
Why I added a reviewer
I am the only PM on a mobile app for beauty salons, working with offshore engineers. Writing investigation results and spec summaries into Backlog tickets had gradually become the AI's job.
As the volume grew, small errors started slipping in: broken links, wrong facts. A human re-reading everything does not scale, and I knew I would stop doing it.
On October 16, after asking for a one-off recheck of an AI-written investigation, I asked for the permanent version instead: whenever the AI posts to Backlog, a review should start automatically, every time.
How the loop works
When a ticket is created or its body is updated, and the body is at least 200 characters, a separate review-only AI starts on its own. The reviewer can read but never writes to Backlog. Fixing is done by the AI that wrote the text. The writer and the checker are different agents on purpose.
The first launch failed: the freshly created agent definition was not loaded into the running session. That day, another agent read the definition file and ran the reviews in its place, and the loop was live by the end of the day.
What 819 comments looked like
The period is October 16 to November 13, 2026. 87 tickets were involved, writes were detected 291 times, and 175 reviews actually ran.
Only 15 of the 175 reviews came back with nothing to say. The median was 4 comments per review.
The real errors
On the very first day, a high-severity comment appeared. In a table listing child tickets, one ticket's deadline was written as "none" when it actually had one. Every other deadline in the table was correct, which is exactly why a skim would miss it.
Other real ones: a rollback procedure pointing at the wrong tag, and "48 samples" in a write-up when the actual count was 40. Each read naturally as prose. All of them were factual errors.
Another day, a premise written into a screen-rework ticket contradicted the implementation ticket for the same feature. The writing AI fixed both tickets, and each fix triggered another review.
Most of it was minor, so I narrowed the scope
High severity was 74 of 819, under 10%. Low and needs-confirmation together were over 60%. Checking everything with the same effort was the wrong default.
On October 20 I stopped reviewing comments and kept only ticket creation and body updates, the places where facts get written down.
Honestly, I almost never read the review comments myself. The writing AI fixes its own text in the same session after each review. This was never a human review step. It is a loop where AIs correct each other.
Production database: drafts only, humans execute
The other write target was the production database, and the split there is stricter: the AI drafts SQL, a human executes it. Across the whole period, the AI connected to the database zero times.
On November 5, that split paid off. The pre-check SQL the AI wrote returned 0 rows when a human ran it in production, so the delete was not executed. The AI had misread what a datetime column meant. After changing how the target row was identified, a human ran the delete. Even the completion report got reviewed, and it received a high-severity comment: the SQL quoted in the report differed from the delete SQL posted earlier.
Drafting is where SQL help is fastest. I would like the AI to execute too, but building the safeguards for that is still on my list. For now, execution stays human. That is the honest current state.
The lines I kept
After a month, four criteria survived.
- Separate the scope an AI may write from the scope it may execute. Irreversible execution stays human.
- Have a different AI check the writing immediately after it happens.
- Do not check everything. Limit checks to where facts are written, and leave light writing out.
- If no human will read the check results, design the loop so the AIs close it themselves.
Letting an AI write is no longer unusual. The difference is in how you verify what it wrote, and where you draw the line between what it may do and what stays with a human.
Top comments (0)