I built a courtroom of AIs to judge ideas. Then I made it judge itself, and the verdict hurt.

Building a product has never been cheaper. One afternoon, an AI editor, and you have a landing page, a backend, and an app live. What got expensive is the only question that matters: did this deserve to exist?

The scarcity flipped and almost nobody noticed. The bottleneck used to be building. Today anyone builds. Now the bottleneck is knowing if anyone wants it. And since validating is work and building became a game, everyone skips validation. Out comes a flood of technically competent products that die not from the code, but because nobody on the other side wanted them.

I wanted to study orchestrating several LLMs, out of the "chatbot" cliche. So, I built a tool that does the opposite of the "AI validation" market: instead of handing you a flattering score and a generic report, it puts your idea on trial. Prosecutors attack, the defense holds the line, witnesses testify, a jury of different models votes, and a judge delivers the sentence. Live, like a courtroom.

And then came the only honest thing to do: I sat my own project in the defendant's chair.

It gave its own idea 81 (innocent). Lowest score: risk handling.

First instance: 81, innocent

I ran the trial on the very idea that gave rise to the tool. No cheating, because the verdict is not the opinion of a friendly LLM, it is calculated from the jurors' votes. I had no way to fudge the number.

It scored 81. Innocent. The judge, Claude Sonnet, recognized the product live with real payments, near-perfect timing, and genuine traction, but nailed the caveat: the lowest score was risk handling, with the note that "a tribunal" is still an easy layer to copy, and that I needed to prove the format reduces evaluation error, not just that it looks rigorous.

I could have stopped there and beaten my chest. But the product has an appeal. So I appealed. Live, recording. And this is where it gets good.

Appeal: I took the stand, hid my numbers, and tried to be clever

In the appeal, the defendant is interrogated. The prosecutor went for the jugular: how many users came back for a second trial? what is the margin per credit? where is the proof the adversarial format beats a single well-made report?

I refused to open a spreadsheet in the middle of my own trial. And I made a move I thought was genius: I used the demotion itself as a defense. "Look, the tribunal was merciless even with its owner. If it were rigged, it would have given me 100."

The judge did not fall for it. In the sentence, it wrote that the argument was "clever but circular": the platform was validating itself using its own verdict as proof. And that my refusal to show numbers "is honest, but costs dearly." Then the jury retreated on almost everything. Differentiation dropped from 4.1 to 3.7. Traction, from 4.3 to 3.6. Execution, from 4.4 to 4.0. Several jurors lowered their scores citing, word for word, that the "moat" was replicable in weeks and that I had brought no data.

On appeal, after I refused to show my numbers, the same judge dropped it to 78 (innocent with concerns).

The result: I fell from 81 to 78. I went from innocent to "innocent with concerns." The same judge that had acquitted me demoted me a full category, live, for dodging the question and trying to talk my way out.

There is no better proof the thing is not rigged. The tool I built to be merciless with ideas was merciless with me, and dismantled my best defense on top of it.

What I was really studying

The idea is the shell. Underneath, what interested me was the engineering problem, and it went deeper than I expected. A single judgment is 47 chained LLM calls. Making that work without stalling mid-show meant solving real things.

Orchestration and resilience: dozens of calls with dependencies, the jury voting in parallel, retries per step, and critical seats with a fallback list. If a juror goes down mid-trial, another takes the bench and the trial does not stop. It never throws an error screen in the middle of the show.

Real multi-LLM: the jury is not 12 clones of the same model. They are models from different houses, Anthropic, Google, OpenAI, xAI, Mistral, Amazon, DeepSeek, Qwen, Moonshot, MiniMax, Zhipu, and Meta, the way a real jury is a sample of society. And they read each other anonymously and change their votes. In the appeal, you can watch them lower their scores one by one, citing my evasiveness and the argument that the format is copyable.

Evidence, not hunches: before judging, the system searches the web and anchors the criticism in real pain with a linked source. The prosecution does not say "I think it is risky", it cites the Reddit post and the article that back the risk.

Before judging, it pulls real pain points from the web, each with a linked source.

Calculated verdict: the score is math over the jury's vote, not the judge's opinion. The judge only writes the sentence. That is why it could praise my execution and still put on record, in black and white, that my defense was circular.

The 12-model jury, one per company. The arrows are votes changing during the anonymous cross-review.

What stayed

Building got trivial. What got rare is judging honestly, including your own work. A tool that only flattered me would have been useless, and a fitting portrait of the bubble of products nobody asked for.

Making the machine criticize me with evidence, demote me when I dodged the question, and point out the fallacy in my defense, was worth more than any perfect score it could have given me.

And the irony: this started as just a weekend study. But I kept showing it to a few friends, and the reaction was always the same: this cannot stay in a drawer. I believed them, and it ended up going live for real. Today anyone can submit their own idea and watch the trial.

If you want to read the full trials, with every argument and every vote, here are both: the first instance (the 81) and the appeal (the drop to 78).

Want to see your idea (or a competitor's) in front of a jury that pulls no punches? The tribunal is open at tribunaldeideias.com.br.