I used five AI agents to decide how to rewrite 671 quiz questions. Three perspectives, one refuter, one synthesizer.
The shape came out of a five-agent cap, but nothing in it turned out to be waste. Generating independent proposals is not enough. You need something whose job is to kill them.
The refuter killed one of the three
One proposal went down for this:
It never ran the verification it most strongly demanded, on its own headline number — it had confused the denominator with a subset.
Three proposals side by side did not surface that. I didn't catch it either, reading them myself.
The refuter's instructions were explicit:
- Do not read this to agree with it
- Find the place where you can say "this will fail in execution"
- Suspect any number quoted without being run
The real win: the assumption all three shared
All three silently assumed that the metric they measured was the same thing as the problem that had been reported. Only the refuter went at that.
When I actually asked the operator, the answer was two different things entirely. Fixing only the headline metric would have missed the target.
Splitting the perspectives buys you independence of perspective. It does not buy you independence of assumption.
Don't let the synthesizer compromise
The synthesizer was told plainly: discard anything the refuter killed.
Allow compromise and a dead proposal comes back diluted. "Take the best of both" does not un-refute the reasoning that killed one of them.
Hand everyone the same measured data, up front
Let each agent do its own aggregation and they recount the same thing repeatedly and disagree on the results. I dumped the analysis to a file first and had all of them read it.
On top of that: write only numbers you produced by running something. No estimates.
One agent still got its own recount wrong. Because the report was fully reproducible, the refuter could determine that the agent was the thing that was off. Shared measured data decides who is wrong, not just that something is.
It also found errors in what I supplied
The refuter flagged two errors in the measured data I had handed them. Both were real.
Instruct it to suspect the premises it was given, and it catches your mistakes too.
Make "what to ask" a required output field
I required the synthesizer to emit open_question — things the operator should decide.
What came back: ask the operator which items they actually found confusing; ten minutes settles the direction. That pays for itself before you gamble 4 to 70 hours.
All three proposals started from one remark by the operator, and not one of them proposed going back and asking. It doesn't appear unless you require the field.
I hand-copied the work split and got a quarter of it wrong
This one is mine.
Distributing 116 sites across five agents, I hand-copied the script's assignment output into JSON. I got 28 of them — a quarter — wrong. Names that didn't exist went out; 28 real ones never did.
I caught it by diffing against the real data right after launch, before a single edit landed.
Three fixes:
- Diff what you handed out against what generated it, as the first step after launch
- Verify names down to file existence before handing them over (added an
assert) - Remove the transcription step entirely. The script now emits a string you paste as-is
The cause wasn't carelessness. It was that a copying step existed at all.
Fix the standard in a document before distributing
Before handing out 390 rewrites, I did 76 myself and wrote the pattern into a document — good examples, bad examples, and what must not be touched.
The thing that breaks first when you split work is consistency of standard. One dialect per agent is worse than no standard at all.
Machine checks are the gate; the writing standard travels as prose. I also stated: do not loosen the check to pass. If it feels too strict, report it — don't edit it. Never leave the gate adjustable from the inside.
If everyone falls in the same hole, it's your handoff
Six agents, one video cut each: every one of them hit "the font file isn't here," and solved it four different ways — bare font names, embedding a font from another project, fetching from the network.
I only found out because each report was required to include "where the instructions and reality diverged."
When all of them raise the same point, it isn't the agents. It's a gap in what you handed them.
The next round, I folded the previous four failures into the instructions. None of the four recurred. But sealing holes reveals the next ones — three agents independently fell into a different one.
Requiring those reports is how you learn what to hand over next time.
Takeaways
- Proposals alone don't find errors. Stand up an independent refuter
- Separate perspectives still share assumptions. Only the refuter attacks those
- Don't let the synthesizer compromise. Killed proposals return diluted
- Distribute the same measured data first. It decides who is wrong
- Tell it to suspect your premises. It will catch your errors
- Require an "open questions" field. It doesn't appear otherwise
- Never build a step that requires transcription. I got a quarter wrong
- Everyone in the same hole means the handoff is missing something
Separately from this, I build a desktop AI agent called Wisp. It stands on your desktop, answers when you talk to it, and runs commands when you ask — always showing you what it's about to do first.
Top comments (1)
The rule telling the synthesizer to discard killed proposals completely rather than compromising is the most important guardrail here.
When multiple agents run parallel proposals, people usually fall back to majority voting or merging. Parallel workers almost always inherit the same implicit assumptions, so compromise just averages the shared flaw into something plausible.
Dumping the raw measured data into a static file first is also the right pattern. The moment agents compute their own ad-hoc aggregations inside the prompt loop, you spend half your debugging time figuring out which agent miscounted instead of evaluating the actual reasoning.