In debate, two judges can score the same speech differently. Students get a number with no reasons, and new judges have no standard to check themselves against. I'm building Saql, an Arabic debate-analysis engine that breaks a debate into arguments and objections, scores each on a written scale, and keeps the reasons for every score.
This post covers the methods it is built on, two design decisions I think matter most, the numbers that actually exist, and what I haven't proven.
The methods behind it
- Toulmin's model (claim, grounds, warrant, backing, qualifier, rebuttal) as the basis for scoring how an argument is constructed.
- Walton's argumentation schemes, with critical questions and burden of proof, as the basis for scoring reasoning quality.
- Pragma-dialectics as the basis for scoring how well a reply handles an objection.
- Multi-criteria decision analysis (MCDA), to organize the criteria and indicators.
- Behaviorally anchored rating scales (BARS): every level from 0 to 4 has a written description of what that score looks like.
- Test-retest reliability: in "precise mode" an evaluation runs three times, and scores that don't repeat are flagged.
- Human-in-the-loop adjudication: Saql never issues a final verdict. A human decides.
Decision 1: written anchors, not labels
Until 12 October 2026, the rubric levels in Saql were only labels ("absent", "weak", "partial", "competent", "strong") with empty descriptors, and the per-indicator anchors were recorded as unresolved. Only one indicator, RQ1, had written anchors. In commit 935fb96 I loaded the approved anchors for every indicator (methodology v2). The model's instructions moved into a standalone methodology document, and the prompt went from v1 to v3.
Here is what an anchor looks like. The indicator is RR2, "Response Relevance", which measures whether a reply is aimed at the objection that was actually raised:
| Level 1 | Level 3 |
|---|---|
| Same general topic, but it does not touch the objection. | Directed at the objection, with a minor digression. |
(The anchors are written in Arabic; the English above is my translation.)
The gap between the levels is what makes a disagreement interpretable. If one judge gives 1 and another gives 3, the question is precise: did the reply touch the objection? It is no longer "who is better?". When a human and the model use the same written anchors, you can ask whether the judge applied a description loosely, or whether the description itself needs tuning.
Some anchors also carry guard rules against style tricks. For indicator AC5, scope of the claim, cautious language alone does not raise the score, while an unqualified claim that is fully supported gets 4.
Decision 2: no overall score
Saql scores along five criteria:
| Criterion | What it covers | Indicators |
|---|---|---|
| AC | Argument Construction | AC1 to AC6, numeric 0 to 4 |
| RQ | Reasoning Quality | RQ1 to RQ5, numeric 0 to 4 |
| RR | Rebuttal Response | RR1 to RR5, numeric 0 to 4 |
| BF | Burden Fulfillment | BF1 to BF5, qualitative states, no number |
| CR | Claim Relevance | scored at the criterion level |
How Saql sees one argument. The claim rests on evidence through an inferential link (AC, RQ). An objection can target the claim, its evidence or its link, and the reply is measured only if the speaker had a chance to respond (RR). The claim's relevance to the case is measured separately (CR). Burden of proof (BF) is tracked per team as five obligations with qualitative states (satisfied, partial, unsatisfied, unspecified), never as a number, and is not derived from argument scores. The unit of analysis differs by criterion: the argument for AC, RQ and CR; the objection-and-reply exchange for RR; the team for BF. (Labels translated from Arabic.)
That is 16 numeric indicators, each with written anchors. The most important decision is that Saql does not combine them into one number. The code has no approved aggregation rule from indicators to criteria yet, so it doesn't invent one. An overall score would hide exactly what a student or a judge needs to see: whether an argument was clear but badly supported, or well supported but off target.
What the code does
- Each score is stored per indicator, together with its provenance: provider, model, prompt version, token count and latency.
- Each score links to its evidence references.
- Every change to a score is written to an audit trail.
- A human adjudicator always makes the final call.
The numbers that exist
| What | Number |
|---|---|
| Scale levels | 5 (0 to 4) |
| Criteria | 5 |
| Numeric indicators | 16 (AC1 to AC6, RQ1 to RQ5, RR1 to RR5) |
| Passes in precise mode | 3 per evaluation |
| Prompt versions | v1 to v3 |
| RQ1 pilot | 5 debates, 28 scoring units |
| Completed score rows in the pilot | 40, of which 39 have a numeric score (qwen2.5:7b: 28 rows, gemini-2.5-flash: 12 rows) |
The pilot covered one indicator (RQ1), used the first version of the prompt, and ran on two different models. It was a pipeline test, not a measurement of quality.
What I haven't proven
- No calibration yet. I have not compared Saql's scores with a panel of human judges at scale. When I do, I'll report exact agreement, agreement within one point, and quadratic weighted kappa, with the sample size.
- The anchors have not been validated against debate coaches. They are a careful first draft of what each score should mean.
- No aggregation rule. I deliberately left it open until there is evidence for one.
- No adoption yet. Some debate organizations have shown early interest, but no organization has adopted it or run a pilot.
How Saql is meant to be used
Saql is built for institutional ownership, not as a public app. The design assumes that a debate organization, a league or an academic program owns the instance, controls its data, and decides how its judges and coaches use it. That is why the scoring is transparent (written anchors, a record for every score, a human who always decides), and why I'm not promoting a public demo. A tool that scores people's reasoning should sit with an institution that is accountable for how it is used.
If you work with or advise a debate organization, a league or a university program and would like to talk about how this could work for your judges, leave a comment below or find me on LinkedIn: https://www.linkedin.com/in/iv7l
I'm a developer in Gaza building Fasil.
Top comments (1)
Author here. Saql scores Arabic debate arguments against written anchors (BARS), and deliberately refuses to combine the scores into one number. The part I'd most like feedback on is the missing aggregation rule: if you've built multi-criteria scoring for human judgment, I'd like to hear how you decided when (or whether) to aggregate.
Happy to answer questions about the anchors, the design choices, or the limits listed at the end.