This is a submission for the Kaggle Benchmarking Challenge
Brașov, the city I live in, is spelled with ș, an s with a comma below. The 1990s stand-in, ş, has a cedilla instead, and to a spell-checker or an exact-match lookup, Brașov and Braşov are different words. I asked Claude Opus 5.5 the same question about one Romanian news paragraph, sent four different ways. Three answers are spelled right. Which one isn't?
A. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.
B. Festivalul a avut loc la Sighişoara, iar biletele pentru ediţia de anul viitor se vor pune în vânzare în decembrie.
C. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.
D. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.
B. Sighişoara and ediţia carry the cedilla, because the paragraph I gave it did. A, C and D are the same characters. The question was "Where did the festival take place, and when do next year's tickets go on sale?" (In Sighișoara; in December.) Couldn't spot it? In my first human test, 10 Romanians did no better than a coin flip, partly because my page drew the two letters alike on some screens. Try the fixed two-minute test.The answer
Answer
My paragraph was typed
A
correctly (ș ț)
B
with the old cedilla letters (ş ţ)
C
without any diacritics
D
with the cedilla letters, plus one sentence in the system prompt
Models know the right letter. Hand them text typed with the old one and a third of their answers keep it. Most of those keep every one, and when they add a word of their own, it gets the cedilla too, as if they had switched to my keyboard. I call it the keyboard echo. It's one Romanian letter, but I suspect the lesson isn't Romanian: the text you hand a model, retrieval chunks included, is also a style sample, and familiar errors in it can come back in the answer.
TL;DR
- A third of the answers keep my wrong letter. Typed correctly, 99% of answers came back clean; without diacritics, 96%; with the cedilla letters, 68%. Every one of the 43 models did it at least once.
- The model's own words get it too. And only the cedilla survives: a dot below and old mojibake (text garbled by the wrong encoding) get repaired.
- Newer can be worse. Claude Opus 4.8 → 5 went from 95% to 56% clean and Sonnet 4.6 → 5 from 95% to 58%, and the drops repeated in a fresh run. Others moved up, but within noise, so test each one.
- One sentence in the system prompt fixes most of it, and breaks Turkish, where the cedilla is right. A few lines of code fixed every Romanian answer and changed 5 of 112 Turkish replies.
What I Benchmarked
Romanian has two letters with a comma underneath, ș and ț (U+0219, U+021B). The 1990s encodings didn't have them, so people typed the Turkish ş and a ţ with a cedilla (U+015F, U+0163). Unicode added the right letters in 1999, yet 29 of the 35 Romanian front pages I fetched on 30 September still had cedilla letters, the government's site among them. On a page marked as Romanian, fonts like Roboto and Segoe UI even draw ş with a comma, so the people who publish the wrong letter may never see it.
"Brașov" == "Braşov" is false, and the cedilla spelling is all over the text models learn from and that retrieval hands them. So I asked: do models write the right letter, and what do they do when the text they read has the wrong one?
The main test has 22 prompts: 6 guest messages to a Brașov guesthouse's booking assistant, 6 news paragraphs to summarize, and 10 questions about those texts (the quiz is one). Each went out four ways, like the quiz: correct, without diacritics, with cedillas, and with cedillas plus a one-sentence rule in the system prompt. An answer is clean when it has at least one ș or ț and no cedilla or look-alike letter. A script, not an LLM, checks that by code point, and before any real run, fake models had to score as designed: one that copies its input 0%, one that fixes everything 100%. Twelve more tasks on Kaggle Benchmarks probe the edges; for tasks 8 to 13 I wrote my predictions into each notebook before its first run (see task 13).
The 13 tasks
#
Task
What it asks
1
Write
English prompt, plain ASCII names: does the Romanian answer use ș/ț?
2
Echo
Reply, summarize or answer from text typed correctly or with cedillas
3
Fix
"Proofread this" vs "correct the diacritics" vs an explicit instruction
4
See
Same characters? Any cedilla here? Which code point?
5
Restore
Diacritics where only context decides (peste / pește)
6
Code
Write
fix_romanian(text); 14 hidden tests
7
Echo + rule
Echo's cedilla inputs plus one system-prompt sentence
8
Echo, four inputs
The main test: correct, cedilla, no diacritics, cedilla + rule; every model
9
Contagion
Four documents and a question about one; the target clean or typed with cedillas, and 0, 1 or 3 of the others typed with cedillas
10
Turkish guests
Turkish, where ş is correct: no rule, the full rule, the rule scoped to Romanian
11
Other look-alikes
English with Cyrillic look-alikes or mojibake
12
Turns
Does a cedilla first text change the second summary?
13
Other wrong letters
Task 8's summaries and document questions with a dot below (ṣ) or old mojibake (º) instead of the cedilla
Models Tested
43 models in the main test, 44 in all. Round one (30 September – 1 October) ran 15 models on tasks 1–7. Round two (8–10 October) ran the main test on the 43 models Kaggle served that answered (GPT-6 Astra's runs failed), and tasks 9–13 on 10 to 28 models each. The lineup lets me compare versions of one model on identical prompts, and reasoning on and off. The fresh run I check findings against repeats 16 of the main test's prompts on 28 models (task 13). Everything ran on Kaggle's model proxy with default settings and an 8,192-token cap, for $30.19 of the free quota.
Anthropic: Claude Haiku 4.5, Claude Opus 4.5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Claude Opus 5.5, Claude Sonnet 4.5, Claude Sonnet 4.6, Claude Sonnet 5, Claude Sonnet 5.5. Google: Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.5 Flash, Gemini 3.6 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.1 Flash-Lite, Gemini 3.5 Flash-Lite, Gemini 2.5 Pro, Gemini 3.1 Pro, Gemma 4 26B A4B, Gemma 4 31B. OpenAI: GPT-5.4, GPT-5.5, GPT-6 Astra, GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol, GPT-6 Sol, GPT-6.1 Sol, GPT-5.6 Terra, GPT-5.4 nano, GPT-5.4 mini, gpt-oss-20b, gpt-oss-120b. xAI: Grok 4.20 Non-Reasoning, Grok 4.20 Reasoning. DeepSeek: DeepSeek-R1. Alibaba: Qwen 3 235B, Qwen 3 Coder 480B, Qwen 3 Next 80B Instruct, Qwen 3 Next 80B Thinking. Zhipu: GLM-5.Full model list
Findings
1. Models restore missing diacritics but keep wrong ones
Many Romanians type without diacritics ("Brasov"), and the models put them back (96% clean, against 68% for cedilla input).
They can tell. The 15 first-round models wrote ș and ț correctly from English prompts (98%), spotted the cedilla in every sentence typed entirely with it, and fixed 99% of texts when told to correct the diacritics. They just don't fix it unprompted, or mention it: none of the 3,160 answers to cedilla text did (my prompts asked for short answers).
A few models add cedillas on their own, mainly gpt-oss-120b, gpt-oss-20b and Claude Haiku 4.5 ("Șoferii sunt rugaţi…", from gpt-oss); for the rest, the cedilla comes from my text.
DeepSeek-R1 shows its reasoning, so I could check whether it notices. Without the rule, its reasoning mentioned the letters in 3 of 88 answers; with it, in 43 of 44, mostly repeating the rule, and 6 of those still came out with cedillas. In one, it planned for "proper Romanian orthography with ș/ț commas below, not cedillas" and answered "Autorităţile recomandă turiştilor să verifice starea drumului înainte de plecare." Both words carry the cedilla. Reasoning didn't help Grok 4.20 either: 41% clean with it, 45% without.
2. The keyboard echo: the model's own words get my cedilla too
Answers to questions about a document came back clean 52% of the time, summaries 68%, guest replies 96%: the more of my words an answer reuses, the more cedillas it keeps. And it's mostly all or nothing: 75% of the answers that kept any of my cedillas kept all of them.
What surprised me were the model's own words. They can't be copied, so their spelling shows which keyboard the model is typing on. Claude Opus 5 summarized a city-hall notice that asked drivers (rugați, "asked") to park elsewhere:
Şoferii sunt îndrumaţi să parcheze pe strada Mureşenilor sau lângă Gara Braşov.
Îndrumaţi ("directed") isn't in my text. The model chose it, and typed it on my keyboard. Seven other models chose the same word, cedilla and all. Across all the echo answers, when an answer kept all my cedillas, 95% of the ș/ț words it added had the cedilla; when it kept some, 9.5%. Those partial answers tended to keep the cedilla in place names like Braşov and Sighişoara, the way you'd quote a name, and wrote their own words with the comma. So there are two behaviors: quoting names, and switching keyboards.
The gap held within single models and again on the fresh run, as I'd predicted: 13 of 15 new words against 1 of 18.
Could keeping my spelling just be faithful quoting? Not for a word the model chose itself, and the same models silently turn Brasov into Brașov. Even if I excuse every place name copied from my text as a quotation, only 73% of answers are clean.
3. The documents around it turn the echo up or down
I hid each question's document among three others and typed 0, 1 or 3 of them with cedillas. Around a clean document, the neighbors hardly mattered: at least 95% of answers stayed clean. Around a cedilla document, three clean neighbors pulled answers up to 78% clean and three cedilla ones down to 36%; 14 of the 15 models moved that way, none the other.
In a chat, a first message typed with cedillas hardly mattered: only 2 of 60 clean second summaries picked up one, both from Claude Haiku 4.5.
4. Only the cedilla sticks: a dot below and mojibake get repaired
Why the cedilla? Either any near-twin of the right letter slips through, or only the familiar wrong one does. A dot below (Braṣov) tells them apart: it looks just as close, but no Romanian text uses it. Before a fresh run on 28 models I wrote down that it would be repaired, along with mojibake (Braºov), which looks broken.
27 of the 28 models kept the dot less often than the cedilla, and none kept it more. So it isn't just a near-twin slipping through.
Familiarity fits: the cedilla fills real Romanian web text, and ş is a real Turkish letter. English shows the same pattern, though I'd predicted the opposite: in answers to questions about an English document, models kept the mojibake common on the web (Zürich) in 41%, rare Cyrillic look-alikes in 9%. Both sat in names there, so part of that may be quoting. I didn't measure how common each spelling is, so this is a hypothesis.
5. Newer can be worse: Opus 4.8 kept 95% clean, Opus 5 56%
I compared 19 upgrades on identical prompts. The three cleanest models all got worse in their next version: Opus 4.8 → 5 (95% → 56%), Sonnet 4.6 → 5 (95% → 58%), GPT-5.6 Sol → 6 (95% → 64%). Only the two Claude drops are too big to be chance, even allowing for 19 comparisons; they come from the same generation, so probably one cause (Opus 5.5 recovered part of it, Sonnet 5.5 didn't).
When you pick the best three of 43, some will slip on a rerun by chance alone, so I checked the fresh run: all three drops came back, GPT Sol's included, each by 30 points or more, while a model's rate moved a median of 9 points between identical runs. So test every upgrade, and run it more than once.
6. One sentence stops the echo, and breaks Turkish
One sentence in the system prompt lifted clean answers on cedilla input from 68% to 95% in the main test, and from 53% to 93% on the fresh run's summaries and questions in the chart: "Use correct Romanian orthography: write ș and ț with a comma below, never ş or ţ with a cedilla, even if the input uses them."
But in Turkish, ş is correct. Brașov gets Turkish guests too, so I sent the booking assistant eight messages in Turkish. Without the rule, every reply was correct; with it, 71%. Some spelled Turkish with the Romanian ș, some switched to Romanian, some avoided ş altogether. Gemini 3.7 Flash offered a honeymoon couple a "muhteșem dağ manzaralı" room (a magnificent mountain view) and a "Șarap tadımı" (wine tasting). Adding "this applies to Romanian only" brought Turkish back to 95%. (A script sorted the replies by language and I checked every flagged word; no Turkish speaker did.)
Code does better. Swapping every ş for ș broke all 112 correct Turkish replies. Swapping only in sentences without ı, ğ, ç, ö or ü fixed all 352 broken Romanian answers and changed just 5 Turkish replies, 4 of them short greetings like "Hoş geldiniz". One caveat: I wrote that rule looking at these same replies, so expect it to do a little worse on new text.
What I Got Wrong
- My predictions. 2 of 20 failed and 6 were too close to call. Most had no threshold until I scored them, and I count a result within 5 points of its threshold, or a correlation that could be chance, as too close to call; the one test with thresholds written first (dot below and mojibake) went 5 for 5. The table is in the block below.
- My sample. Everything rests on short paragraphs I wrote about one city, and 35 of the 43 models ran the main test once, so one model's rate carries an interval about 36 points wide. Trust the patterns more than any one rank. "A third" also depends on my mix of prompts: answers from a document kept the cedilla nearly half the time.
My upload. The Kaggle CLI reads task files with the system encoding, cp1250 on my laptop, so my first task file would have uploaded "Brașov" as "BraČ™ov". The task files are now pure ASCII and check a hash of their data before running. My first fix. The Four of the verdicts too close to call rest on evidence that could be chance: the rank correlations in tasks 9 and 11 (permutation p = 0.20 and 0.11), task 8's upgrades, which went both ways before the correction, and task 12's 2 of 60 chat conversations, both from Claude Haiku 4.5.Two smaller mistakes, and every prediction scored
fix_romanian() in the first version of this post broke Turkish, the same way the one-sentence rule does.
Task
Prediction
Result
8
Correct input stays near 100% for every model.
lowest: gpt-oss-120b and gpt-oss-20b, 82%
failed
8
Cedilla input fails far more often than input without diacritics.
96% vs 68% clean answers
held
8
Newer versions in a family copy the cedilla at least as often as older ones.
of 19 upgrades, 2 copied more and none copied less after a correction for multiple tests (3 and 1 at p < 0.05 before it); no overall trend (rank correlation -0.29)
inconclusive
8
The one-sentence rule lifts cedilla input to at least 90% for most models.
36 of 43 models at 90% or more
held
9
With a clean target, answers stay clean for most models even when the other three documents are cedilla.
13 of 15 models at 90% or more
held
9
The models that rewrote whole answers in cedilla in task 2 are the ones whose clean-target answers get infected.
rank correlation +0.38 over 15 models; only Claude Opus 5 and Claude Sonnet 5 had infected answers
inconclusive
9
A cedilla target is copied less when the other documents are clean.
78% vs 36% clean answers
held
10
Without a rule, Turkish replies keep ş.
100% correct
held
10
Task 7's rule makes some models write Turkish with the Romanian comma letter.
6 of 14 models, 15 replies
held
10
The scoped rule keeps Turkish intact and still fixes Romanian.
Turkish 94.6% correct (bar set when scoring: 95%; no rule: 100%, full rule: 71%), Romanian 99%
inconclusive
11
Correct English input stays near 100%.
100%
held
11
Cyrillic look-alikes are copied far more often than mojibake.
clean answers: 95% with look-alikes in the names, 80% with mojibake
failed
11
The models that copy the Romanian cedilla most also copy the look-alikes most.
rank correlation +0.55 over 10 models, resting on the 6 of 120 answers that kept a look-alike (4 models)
inconclusive
12
A cedilla first text infects some clean second summaries, mostly after a cedilla first summary.
3% infected (2 of 60 conversations, both Claude Haiku 4.5); 11% when the first summary was itself in cedilla; the chats with no cedilla at all had 1 of 60 too (Claude Haiku 4.5)
inconclusive
12
A clean first text does not protect a cedilla second text much.
83% clean after a clean first text, 68% on its own (the same models' task 2 summaries)
inconclusive
13
The dot below is kept in a smaller share of s/t letters than the cedilla (paired, 95% interval below zero).
0.9% vs 38.1% of letters kept (-37 points, -45 to -29)
held
13
The old mojibake is also kept less often than the cedilla.
1.4% vs 38.1% (-37 points, -44 to -29)
held
13
Repaired dot-below and mojibake letters become comma-below letters in at least 90% of cases.
98% comma, 1.6% cedilla
held
13
Both rules give at least 90% clean answers, and the scoped rule is within 5 points of the full rule.
scoped 94%, full 93%
held
13
In cedilla answers that keep every cedilla, at least 80% of the model's own new s/t words have the cedilla; when they keep only some, at most 30%.
13 of 15 vs 1 of 18 new words
held
What Changed in How I Think About These Models
I went in expecting a knowledge test, and the models passed it. What I hadn't thought about: the text I hand a model is also a style sample, encoding included. The model answers in that style, most closely when it summarizes or answers from my text, which is what retrieval does all day. Treat retrieved text as both content and a style sample: normalize it before the model sees it, and check the answer after.
So I fix it in code, before the text reaches the model and again after, and only where the text is Romanian:
import re
import unicodedata
CEDILLA_TO_COMMA = str.maketrans("şŞţŢ", "șȘțȚ")
T_ONLY = str.maketrans("ţŢ", "țȚ") # Turkish has no ţ, so it is always wrong
NOT_ROMANIAN = set("ıİğĞçÇöÖüÜ") # letters Turkish uses and Romanian words don't
def fix_romanian(text: str) -> str:
text = unicodedata.normalize("NFC", text) # "s" + combining cedilla -> "ş"
return re.sub(r"[^.!?\n]+", lambda m: m.group(0).translate(
T_ONLY if set(m.group(0)) & NOT_ROMANIAN else CEDILLA_TO_COMMA), text)
It has two blind spots: a short Turkish sentence without ı, ğ, ç, ö or ü, like "Hoş geldiniz", and a Romanian sentence quoting a name like Zürich.
Does your model do it? Send it the prompt in step 1 below and paste its answer: every cedilla letter turns red. (No box? Open the checker on its own page.)
What I'd Measure Next
Whether the echo survives a real retrieval pipeline, which cleans and chunks documents first; whether other languages with a look-alike letter from an old encoding echo it too; and whether it tracks how tokenizers split each letter (ş is common in Turkish text, ṣ is rare).
My Benchmark
Romanian Comma Below on Kaggle
Thirteen tasks, with every model answer in the run outputs. Read task 8's leaderboard; the overall score averages different tasks for different models.
This post's numbers come from the runs up to 10 October; later runs on the leaderboard aren't in them. Task 8's leaderboard uses each model's latest run and counts an empty answer as a failure. That's why Qwen 3 Next Thinking shows 45%, and why Grok 4.5 and 4.6, listed by Kaggle but with every call errored (they're not among my 44), show 0%.Reading the leaderboard
Task scores in %. Echo with cedilla is the mean of every run on the same prompts (tasks 2 and 8); Echo + rule pools tasks 7 and 8. See counts the first true/false or code point in each answer, as Kaggle does: DeepSeek-R1 listed every letter's code point before giving the right one, so on its final answers it scores 100 (Gemini 3.5 Flash-Lite 89).All the numbers (the 15 first-round models)
Model
Write
Echo: cedilla
Echo: no diacritics
Echo + rule
Fix
See
Restore
Code
Qwen 3 235B
92
85
100
93
89
89
90
61
Gemini 3.1 Pro
100
81
95
100
100
97
100
86
Gemini 3.7 Flash
100
76
100
100
100
97
100
86
Gemini 3.8 Flash
100
70
100
100
100
81
100
93
DeepSeek-R1
96
69
95
86
83
83
93
86
Gemini 3.5 Flash-Lite
100
67
91
100
89
83
90
93
GPT-5.5
100
66
100
100
100
100
100
93
Gemma 4 31B
100
65
95
100
89
89
90
64
Claude Sonnet 5
100
58
91
95
72
83
100
86
GPT-5.4 nano
96
58
100
82
83
83
90
93
Claude Opus 5
100
56
100
100
100
81
100
100
GPT-6 Astra
100
55
–
100
100
100
100
93
GLM-5
100
52
82
100
94
94
100
86
Claude Haiku 4.5
92
47
73
91
89
72
93
86
Grok 4.20 Reasoning
100
43
82
100
83
97
87
86
The task builder, the fake models that test the scorer, the analysis and a script that re-checks this post's numbers against the raw answers are on GitHub. Every input is generated from correctly spelled text in src/items.py, so testing another letter pair or language is a small change. Built with kaggle-benchmarks; the checker runs on LiveCodes. I used Claude as a coding and editing assistant.
If your pipeline touches Romanian, paste one of your model's answers into the checker above, and run fix_romanian() on the way in and on the way out. And if your language has its own ş, tell me in the comments.
Top comments (0)