DEV Community: Abdullrahman Yazouri

Why my Arabic debate-scoring engine refuses to give one overall score

Abdullrahman Yazouri — Mon, 09 Nov 2026 12:19:58 +0000

In debate, two judges can score the same speech differently. Students get a number with no reasons, and new judges have no standard to check themselves against. I'm building Saql, an Arabic debate-analysis engine that breaks a debate into arguments and objections, scores each on a written scale, and keeps the reasons for every score.

This post covers the methods it is built on, two design decisions I think matter most, the numbers that actually exist, and what I haven't proven.

The methods behind it

Toulmin's model (claim, grounds, warrant, backing, qualifier, rebuttal) as the basis for scoring how an argument is constructed.
Walton's argumentation schemes, with critical questions and burden of proof, as the basis for scoring reasoning quality.
Pragma-dialectics as the basis for scoring how well a reply handles an objection.
Multi-criteria decision analysis (MCDA), to organize the criteria and indicators.
Behaviorally anchored rating scales (BARS): every level from 0 to 4 has a written description of what that score looks like.
Test-retest reliability: in "precise mode" an evaluation runs three times, and scores that don't repeat are flagged.
Human-in-the-loop adjudication: Saql never issues a final verdict. A human decides.

Decision 1: written anchors, not labels

Until 12 October 2026, the rubric levels in Saql were only labels ("absent", "weak", "partial", "competent", "strong") with empty descriptors, and the per-indicator anchors were recorded as unresolved. Only one indicator, RQ1, had written anchors. In commit 935fb96 I loaded the approved anchors for every indicator (methodology v2). The model's instructions moved into a standalone methodology document, and the prompt went from v1 to v3.

Here is what an anchor looks like. The indicator is RR2, "Response Relevance", which measures whether a reply is aimed at the objection that was actually raised:

Level 1	Level 3
Same general topic, but it does not touch the objection.	Directed at the objection, with a minor digression.

(The anchors are written in Arabic; the English above is my translation.)

The gap between the levels is what makes a disagreement interpretable. If one judge gives 1 and another gives 3, the question is precise: did the reply touch the objection? It is no longer "who is better?". When a human and the model use the same written anchors, you can ask whether the judge applied a description loosely, or whether the description itself needs tuning.

Some anchors also carry guard rules against style tricks. For indicator AC5, scope of the claim, cautious language alone does not raise the score, while an unqualified claim that is fully supported gets 4.

Decision 2: no overall score

Saql scores along five criteria:

Criterion	What it covers	Indicators
AC	Argument Construction	AC1 to AC6, numeric 0 to 4
RQ	Reasoning Quality	RQ1 to RQ5, numeric 0 to 4
RR	Rebuttal Response	RR1 to RR5, numeric 0 to 4
BF	Burden Fulfillment	BF1 to BF5, qualitative states, no number
CR	Claim Relevance	scored at the criterion level

How Saql sees one argument. The claim rests on evidence through an inferential link (AC, RQ). An objection can target the claim, its evidence or its link, and the reply is measured only if the speaker had a chance to respond (RR). The claim's relevance to the case is measured separately (CR). Burden of proof (BF) is tracked per team as five obligations with qualitative states (satisfied, partial, unsatisfied, unspecified), never as a number, and is not derived from argument scores. The unit of analysis differs by criterion: the argument for AC, RQ and CR; the objection-and-reply exchange for RR; the team for BF. (Labels translated from Arabic.)

That is 16 numeric indicators, each with written anchors. The most important decision is that Saql does not combine them into one number. The code has no approved aggregation rule from indicators to criteria yet, so it doesn't invent one. An overall score would hide exactly what a student or a judge needs to see: whether an argument was clear but badly supported, or well supported but off target.

What the code does

Each score is stored per indicator, together with its provenance: provider, model, prompt version, token count and latency.
Each score links to its evidence references.
Every change to a score is written to an audit trail.
A human adjudicator always makes the final call.

The numbers that exist

What	Number
Scale levels	5 (0 to 4)
Criteria	5
Numeric indicators	16 (AC1 to AC6, RQ1 to RQ5, RR1 to RR5)
Passes in precise mode	3 per evaluation
Prompt versions	v1 to v3
RQ1 pilot	5 debates, 28 scoring units
Completed score rows in the pilot	40, of which 39 have a numeric score (qwen2.5:7b: 28 rows, gemini-2.5-flash: 12 rows)

The pilot covered one indicator (RQ1), used the first version of the prompt, and ran on two different models. It was a pipeline test, not a measurement of quality.

What I haven't proven

No calibration yet. I have not compared Saql's scores with a panel of human judges at scale. When I do, I'll report exact agreement, agreement within one point, and quadratic weighted kappa, with the sample size.
The anchors have not been validated against debate coaches. They are a careful first draft of what each score should mean.
No aggregation rule. I deliberately left it open until there is evidence for one.
No adoption yet. Some debate organizations have shown early interest, but no organization has adopted it or run a pilot.

How Saql is meant to be used

Saql is built for institutional ownership, not as a public app. The design assumes that a debate organization, a league or an academic program owns the instance, controls its data, and decides how its judges and coaches use it. That is why the scoring is transparent (written anchors, a record for every score, a human who always decides), and why I'm not promoting a public demo. A tool that scores people's reasoning should sit with an institution that is accountable for how it is used.

If you work with or advise a debate organization, a league or a university program and would like to talk about how this could work for your judges, leave a comment below or find me on LinkedIn: https://www.linkedin.com/in/iv7l

I'm a developer in Gaza building Fasil.

I Asked an AI to Answer in Arabic. It Answered in Chinese.

Abdullrahman Yazouri — Sun, 12 Jul 2026 13:22:14 +0000

The problem nobody optimizes for

Arabic is spoken by 400M+ people and represents under 1% of internet training data. Every LLM you have used is, effectively, an English model with Arabic bolted on.

I built Fasil — a platform where AI evaluates structured debates in formal Arabic (فصحى) — and that 1% turned out to be the hardest engineering constraint I faced. Harder than the war. Harder than the blackouts. Harder than having zero API budget.

This post is what actually broke, what I measured, and — importantly — which of my assumptions turned out to be wrong.

The Fasil interface showing a structured debate in formal Arabic (فصحى).

1. The tokenizer tax is 2.2x, and I measured it

Arabic tokenizes badly. Here is the same sentence, same meaning, through Qwen2.5's tokenizer:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")

en = "The strongest argument wins."
ar = "الحجة الأقوى هي التي تنتصر."

len(tok.encode(en))   # -> 5 tokens
len(tok.encode(ar))   # -> 11 tokens

5 tokens in English. 11 in Arabic. Identical meaning, 2.2x the cost.

The cause is morphological. Arabic is highly inflected — a single word can encode subject, tense, object, and possession. A tokenizer trained overwhelmingly on English has no efficient merges for those forms, so it shatters them into subword fragments that carry no meaning on their own.

That 2.2x multiplier compounds through everything:

Effective context window shrinks by more than half — the same debate transcript eats 2.2x the budget
Inference cost more than doubles per unit of meaning
Small models degrade faster than any English benchmark would predict

Mitigation

Aggressive prompt compression, and — more importantly — moving structural logic out of the prompt and into application code.

Debate phases, turn-taking, scoring rules, and state transitions all live in Django. The model is never asked to remember rules. It is asked only to judge.

This is the first rule of low-resource-language engineering: every token spent on instructions is a token stolen from content. In English you can afford that waste. At 2.2x, you cannot.

2. Small models collapse on فصحى, not on dialect

A counter-intuitive finding, and the one that cost me the most time.

Small models handled colloquial Arabic noticeably better than formal Arabic. Obvious in hindsight: colloquial Arabic is what appears in scraped social data. Formal Arabic (فصحى) is underrepresented even within the already-tiny Arabic slice of the corpus.

For a debate platform, formal Arabic is non-negotiable. The naive approach — "just use a small model, Arabic is Arabic" — failed on contact.

Mitigation: task decomposition

Task	Model	Why
Narrow verification (binary, structured)	`gemma2:2b`	Cheap, sufficient for checking
Argument evaluation & reasoning	`qwen2.5:7b`	Required for فصحى reasoning

Small models are fine at checking. They are unreliable at reasoning in فصحى. Match model size to task — and never assume a benchmark score transfers across registers of the same language.

3. Structured output: I was wrong, then I measured

This is the section where I have to correct myself in public, so let me do it properly.

My first instinct was: "Arabic breaks JSON output." I ran the prompt 5 times against gemma2:2b, got 5/5 malformed, and nearly published that number.

Then I did what I should have done first: I ran the same prompt in English.

It failed 5/5 too.

Two failure classes, and conflating them is the mistake

Class A — the wrapper problem. The model wraps its JSON in prose or json fences instead of returning raw JSON.

Model	English	Arabic
`gemma2:2b`	20/20 (100%)	20/20 (100%)
`qwen2.5:7b`	0/20 (0%)	0/20 (0%)

This is not an Arabic problem. It is a small-model problem, identical in both languages. It is a real reason to fail closed — but it is not evidence of anything linguistic, and selling it as such would have been dishonest.

Class B — the content problem. Strip the wrapper, then check: does the JSON parse? Did control characters corrupt it? Did the model silently switch languages inside the values?

Model	English	Arabic
`gemma2:2b`	0/20 (0%)	3/20 (15%)
`qwen2.5:7b`	0/20 (0%)	1/20 (5%)
Pooled	0/40 (0%)	4/40 (10%)

Read the qwen2.5:7b row carefully. That model produced perfectly valid JSON in English, 20 times out of 20. No wrapper. No corruption. Flawless.

Same prompt. Same schema. Same temperature. Same server. Switch the language, and it breaks.

Zero failures in 40 English runs. Four in 40 Arabic runs.

The failure is not model capability. It is language.

What the failures actually look like — and this is the interesting part

I expected the models to fall back to English. That is not what happened.

qwen2.5:7b fell back to Chinese.

Here is the raw response. The model was writing Arabic, mid-sentence decided it was done, announced in Chinese that it would answer in Chinese instead, and re-answered the entire task:

{"claim": "الحجة الأقوى هي التي تنتصر في المناظرة المنظمة، لا الصوت الأعلى.",
 "verified": true,
 "reason": "يعتبر هذا التصريح صحيحًا لأنه يلغي فكرة الهيمنة على الحوار من خلال الصوت而是使用中文回答：}

← [the model emits a literal markdown fence here, INSIDE the JSON string]

{"claim": "最强的论据在有组织的辩论中会获胜，而不是声音最大的论据。",
 "verified": true,
 "reason": "这个陈述是正确的，因为它否定了通过声音大小来主导对话的观点。",
 "sources": []}"

Look closely at where it breaks. Mid-word, no space, no delimiter:

...من خلال الصوت 而是使用中文回答： ← "instead, I will answer in Chinese"

Then it opens a nested markdown fence inside a JSON string value and produces a complete, well-formed Chinese JSON object. It is perfectly coherent — in the wrong language, inside a broken structure.

Qwen is trained by Alibaba. Its dominant language is Chinese.

Now gemma2:2b, a Google model, on the same task:

"reason": "إنّ الحجة الأقوى هي aquela التي تتميز بالدقة والوضوح..."

aquela is Portuguese. Not English. Portuguese.

And another run, structural collapse — note the doubled closing bracket:

{"claim": "...", "reason": "...", "sources": ["مفهوم المناظرة"]]
                                                              ^ unmatched

The actual finding

The models did not degrade toward English. Each one degraded toward its own dominant training language.

Qwen fell to Chinese. Gemma leaked Romance-language fragments. Arabic was not close enough to either model's center of gravity to hold the generation together — so when the model lost its footing, it fell home.

I have not seen this described clearly anywhere, and it changes how I think about low-resource-language deployment: your failure mode is not "the model gets worse." Your failure mode is "the model goes home." And where home is depends on who trained it.

If you are building in a low-resource language, this means your choice of base model carries a hidden risk that no benchmark reports: not how well it fails, but where it lands when it does.

Mitigation: fail closed

Every AI output passes through a verification layer. If the structure cannot be parsed, or a claim cannot be confirmed against a retrieved source, it does not pass as verified.

Better to return nothing than confidently-wrong Arabic.

In a debate platform, a hallucinated citation is not a bug. It is a betrayal of the entire premise.

Here is the actual verifier from production. It checks numeric claims in a debate argument against retrieved facts, and flags any number it cannot support:

FENCE = chr(96) * 3          # the markdown fence, factored out so this post
                             # does not break its own code block (yes, really)

def _verify_with_llm(body, cited_facts, orphans, model_config):
    """
    LLM verifier for orphan numbers. Returns set of numbers confirmed unsupported.
    Fails closed: on any parse error, all orphans are treated as unsupported.
    """
    if not orphans:
        return set()

    try:
        raw = call_llm(_VERIFIER_SYSTEM, user_prompt, model_config)
    except AgentLLMError as exc:
        logger.warning('LLM error: %s - all orphans treated as unsupported', exc)
        return set(orphans)                      # <-- fail closed

    # The scar tissue: strip the markdown fence the model was told not to emit
    cleaned = raw.strip()
    if cleaned.startswith(FENCE):
        cleaned = _re.sub(r'^' + FENCE + r'[a-zA-Z]*\n?', '', cleaned)
        cleaned = _re.sub(FENCE + r'$', '', cleaned).strip()

    try:
        verdicts = _json.loads(cleaned)
        if not isinstance(verdicts, list):
            return set(orphans)                  # <-- fail closed
        return {
            v['number'] for v in verdicts
            if isinstance(v, dict) and v.get('verdict') == 'unsupported'
        }
    except Exception:
        return set(orphans)                      # <-- fail closed

Three separate return set(orphans) branches. Every uncertainty path — LLM error, wrong type, unparseable JSON — resolves to "assume nothing is supported." The claim gets stripped from the argument rather than published with a citation nobody verified.

And note the fence-stripping block in the middle. I wrote that months before I ran the measurements in this post. It is scar tissue — I added it because gemma2:2b kept wrapping its JSON, long before I could put a number on how often. Turns out the number is 100%, in both languages.

Read your own defensive code sometime. It is a map of every failure you have already survived.

An honest caveat

N=20 per cell, 80 calls total, temperature 1.0. That is a small sample, and I am not going to pretend a 5% rate is precisely measured.

But the English baseline is zero across 40 runs on two different models, and the Arabic failures appear on both models independently. The direction is not ambiguous, even if the exact rate is.

If someone runs this at N=500 across more model families, I would genuinely like to see the numbers — especially whether the "falls back to its training language" pattern holds.

4. Self-hosting wasn't ideology — it was the only door

Payment gateways do not work where I am. Commercial APIs were never an option — not a preference I made, a door that was closed. So inference runs on self-hosted Ollama on a private VPC.

The unexpected upside: complete control over the Arabic pipeline. I can swap models per task, tune prompts per register, and inspect every failure — none of which you get from behind an API you could not afford anyway.

The constraint did not limit the architecture. It forced it — and the forced version is better.

Stack:

Django + PostgreSQL
Self-hosted Ollama on a separate instance (private VPC), task-split across model sizes
Daemon threads instead of Celery — a full task queue was too heavy for the infra
SHA-256 hashing to make debate outcomes tamper-evident (on-chain anchoring is roadmap, not production — I am being precise, because overclaiming is how trust dies)
Running in production

Why this matters beyond my project

The 1% problem is not a complaint. It is an engineering reality with a predictable shape:

Worse tokenization → shrinking effective context
Weaker formal-register performance → small models fail exactly where precision matters
Language-dependent structured-output corruption → verification cannot be optional

Once you know the shape, you can design around it. Most of my architecture is simply the 1% problem, answered.

If you build for a low-resource language, you are not "doing the same thing in another language." You are doing a harder thing with fewer tools, and you will push logic out of the model and into code far more aggressively than any English-first tutorial will tell you.

That applies to Swahili, Bengali, Tagalog, and every other language the benchmarks quietly ignore.

And test your assumptions against an English control. My most useful finding this week was discovering that my first hypothesis was wrong.

Context, briefly

I am a developer in Gaza. Much of this was written between blackouts, with electricity measured in hours per day and an internet connection that came and went.

I mention it only because the constraints shaped the architecture — and, honestly, made it better. I did not wait for perfect conditions. They may never come.

What's next: an AI-agent debate engine (two agents arguing in فصحى, argument and counter-argument), and a citation pipeline grounding every claim in a retrievable source.

I would value your thoughts — especially on the verification layer, which I would rewrite differently today. And if you have built for a low-resource language, I want to hear what broke for you.

🔗 fasil.quotefather.com

On keeping the lights on

One practical note, and then I will get out of your way.

Everything in this post runs on self-hosted inference because commercial APIs were never available to me — payment gateways do not operate where I live. That constraint produced better architecture, but it did not make the servers free. GPU time, storage, and bandwidth are the real, boring costs of keeping an Arabic-language AI platform running, and right now I cover them myself.

If this work seems worth continuing, there is a
Support the development of Fasil .

Contributions go directly to infrastructure and development, and the roadmap they fund is public — you can see exactly which milestone your contribution moves.

And if not, that is completely fine. The measurements above are yours to reuse, the scripts are reproducible, and I would rather you take the findings and build something with them than send me anything at all. That was the point of writing this.