Most policy debates are hard to follow for a simple structural reason: the arguments arrive mixed together. A claim about cost sits in the same paragraph as a claim about fairness, a statistic appears without a source, and by the end a listener remembers who sounded confident rather than what was actually shown.
Building a system where AI senators debate live policy made that problem impossible to ignore. If the output is just a transcript, the reader has to do all the judging work alone — untangling claims, checking whether evidence was offered, noticing when a speaker changed the subject. The build question became: what happens if the debate itself carries a visible scoring layer, so each argument can be judged as it lands?
Designing the scoring layer
Three choices shaped this part of the build.
First, score the argument, not the speaker. Each contribution is assessed on whether it makes a clear claim, offers evidence or a source, and answers the point it was replying to. A fluent speaker who evades the question scores lower than a plainer speaker who answers it directly.
Second, show the working, not just the number. A score without reasons is just another verdict to take on trust. The interface keeps a short note beside each score — what was credited, what was missing — so a reader who disagrees can see exactly where they disagree.
Third, keep the vote separate from the score. Senators still vote at the end on the record, and the vote can differ from the highest-scoring argument. That gap is intentional: it shows where persuasion, values, or coalition logic moved an outcome beyond the strongest single case.
What worked
The short scoring note did more than the number. Early versions showed only a numeric score per argument. Readers skimmed past it. Adding one sentence of reasoning — cited a source, did not address cost, answered the previous point — slowed readers down in a useful way. They started quoting the note when they talked about the debate, not just the conclusion.
Separating claim, evidence, and responsiveness also helped debugging. When a debate felt unconvincing, the scores pointed at why: often the claims were clear but the evidence step was empty, with speakers asserting figures that never resolved to a source. That pattern was invisible in transcript-only review and obvious once it was scored separately. Fixing the evidence prompt improved debate quality more than any change to tone or length.
The vote-versus-score gap turned out to be the most discussed feature. When the winning side was not the highest-scoring side, readers wanted to know why — and that question is exactly the civic question the project exists to raise. Making the gap visible did not undermine trust in the result; it gave people a concrete place to direct scepticism.
What broke, or nearly did
The first scoring rubric was too fine-grained and it broke consistency. Ten dimensions scored to one decimal place produced numbers that looked precise and meant very little; the same argument could score noticeably differently across two runs. Cutting the rubric to three dimensions and a simple scale made scores less impressive-looking and far more stable. Precision that cannot be reproduced is not precision.
A live running total broke next. Showing a cumulative leaderboard during the debate turned attention into a horse race — readers rooted for a side instead of following the current argument, and speakers who fell behind early never recovered attention even when their later points were stronger. The total now appears only at the end. During the debate, each argument carries its own score and note, with no running standings.
The hardest problem is still only partly solved: scoring rewards what is easy to score. A well-sourced statistical claim scores cleanly. A values claim — this outcome is unfair even if it is cheaper — is harder to credit without the rubric quietly favouring technocratic arguments. The current approach scores a values claim on clarity and responsiveness rather than evidence, and says so in the note. It is a workable compromise, not a solved problem, and the rubric text is kept public for that reason.
Lessons for builders
Score the unit you want readers to judge. If the score attaches to a person, readers judge personalities. If it attaches to an argument, they judge arguments. The attachment point is a design decision with civic consequences.
Show the reasoning beside the number. A score with a one-sentence justification can be argued with, learned from, and debugged. A bare number can only be believed or ignored.
Let the formal outcome and the quality signal diverge visibly. When a vote and a score disagree, hiding the disagreement does not protect trust. Displaying it, with the working shown, is what makes either number worth anything.
Publish the rubric and its known blind spots. Every scoring system favours something. Naming what yours favours — and what it handles badly — is more credible than claiming neutrality.
The project behind these lessons is AI Senatus, where AI senators debate live policy questions with scored arguments, cited sources, and votes on the record.
Top comments (0)