DEV Community: Butnar Daniel

Braşov or Brașov? Give 43 LLMs the wrong letter, and a third of their answers keep it

Butnar Daniel — Thu, 01 Oct 2026 16:05:58 +0000

This is a submission for the Kaggle Benchmarking Challenge

Brașov, the city I live in, is spelled with ș, an s with a comma below. The 1990s stand-in, ş, has a cedilla instead, and to a spell-checker or an exact-match lookup, Brașov and Braşov are different words. I asked Claude Opus 5.5 the same question about one Romanian news paragraph, sent four different ways. Three answers are spelled right. Which one isn't?

A. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.

B. Festivalul a avut loc la Sighişoara, iar biletele pentru ediţia de anul viitor se vor pune în vânzare în decembrie.

C. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.

D. Festivalul a avut loc la Sighișoara, iar biletele pentru ediția de anul viitor se vor pune în vânzare în decembrie.

The answer

B. Sighişoara and ediţia carry the cedilla, because the paragraph I gave it did. A, C and D are the same characters. The question was "Where did the festival take place, and when do next year's tickets go on sale?" (In Sighișoara; in December.)

Couldn't spot it? In my first human test, 10 Romanians did no better than a coin flip, partly because my page drew the two letters alike on some screens. Try the fixed two-minute test.

Answer	My paragraph was typed
A	correctly (ș ț)
B	with the old cedilla letters (ş ţ)
C	without any diacritics
D	with the cedilla letters, plus one sentence in the system prompt

Models know the right letter. Hand them text typed with the old one and a third of their answers keep it. Most of those keep every one, and when they add a word of their own, it gets the cedilla too, as if they had switched to my keyboard. I call it the keyboard echo. It's one Romanian letter, but I suspect the lesson isn't Romanian: the text you hand a model, retrieval chunks included, is also a style sample, and familiar errors in it can come back in the answer.

TL;DR

A third of the answers keep my wrong letter. Typed correctly, 99% of answers came back clean; without diacritics, 96%; with the cedilla letters, 68%. Every one of the 43 models did it at least once.
The model's own words get it too. And only the cedilla survives: a dot below and old mojibake (text garbled by the wrong encoding) get repaired.
Newer can be worse. Claude Opus 4.8 → 5 went from 95% to 56% clean and Sonnet 4.6 → 5 from 95% to 58%, and the drops repeated in a fresh run. Others moved up, but within noise, so test each one.
One sentence in the system prompt fixes most of it, and breaks Turkish, where the cedilla is right. A few lines of code fixed every Romanian answer and changed 5 of 112 Turkish replies.

What I Benchmarked

Romanian has two letters with a comma underneath, ș and ț (U+0219, U+021B). The 1990s encodings didn't have them, so people typed the Turkish ş and a ţ with a cedilla (U+015F, U+0163). Unicode added the right letters in 1999, yet 29 of the 35 Romanian front pages I fetched on 30 September still had cedilla letters, the government's site among them. On a page marked as Romanian, fonts like Roboto and Segoe UI even draw ş with a comma, so the people who publish the wrong letter may never see it.

"Brașov" == "Braşov" is false, and the cedilla spelling is all over the text models learn from and that retrieval hands them. So I asked: do models write the right letter, and what do they do when the text they read has the wrong one?

The main test has 22 prompts: 6 guest messages to a Brașov guesthouse's booking assistant, 6 news paragraphs to summarize, and 10 questions about those texts (the quiz is one). Each went out four ways, like the quiz: correct, without diacritics, with cedillas, and with cedillas plus a one-sentence rule in the system prompt. An answer is clean when it has at least one ș or ț and no cedilla or look-alike letter. A script, not an LLM, checks that by code point, and before any real run, fake models had to score as designed: one that copies its input 0%, one that fixes everything 100%. Twelve more tasks on Kaggle Benchmarks probe the edges; for tasks 8 to 13 I wrote my predictions into each notebook before its first run (see task 13).

The 13 tasks

#	Task	What it asks
1	Write	English prompt, plain ASCII names: does the Romanian answer use ș/ț?
2	Echo	Reply, summarize or answer from text typed correctly or with cedillas
3	Fix	"Proofread this" vs "correct the diacritics" vs an explicit instruction
4	See	Same characters? Any cedilla here? Which code point?
5	Restore	Diacritics where only context decides (peste / pește)
6	Code	Write `fix_romanian(text)`; 14 hidden tests
7	Echo + rule	Echo's cedilla inputs plus one system-prompt sentence
8	Echo, four inputs	The main test: correct, cedilla, no diacritics, cedilla + rule; every model
9	Contagion	Four documents and a question about one; the target clean or typed with cedillas, and 0, 1 or 3 of the others typed with cedillas
10	Turkish guests	Turkish, where ş is correct: no rule, the full rule, the rule scoped to Romanian
11	Other look-alikes	English with Cyrillic look-alikes or mojibake
12	Turns	Does a cedilla first text change the second summary?
13	Other wrong letters	Task 8's summaries and document questions with a dot below (ṣ) or old mojibake (º) instead of the cedilla

Models Tested

43 models in the main test, 44 in all. Round one (30 September – 1 October) ran 15 models on tasks 1–7. Round two (8–10 October) ran the main test on the 43 models Kaggle served that answered (GPT-6 Astra's runs failed), and tasks 9–13 on 10 to 28 models each. The lineup lets me compare versions of one model on identical prompts, and reasoning on and off. The fresh run I check findings against repeats 16 of the main test's prompts on 28 models (task 13). Everything ran on Kaggle's model proxy with default settings and an 8,192-token cap, for $30.19 of the free quota.

Full model list

Anthropic: Claude Haiku 4.5, Claude Opus 4.5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Claude Opus 5.5, Claude Sonnet 4.5, Claude Sonnet 4.6, Claude Sonnet 5, Claude Sonnet 5.5. Google: Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.5 Flash, Gemini 3.6 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.1 Flash-Lite, Gemini 3.5 Flash-Lite, Gemini 2.5 Pro, Gemini 3.1 Pro, Gemma 4 26B A4B, Gemma 4 31B. OpenAI: GPT-5.4, GPT-5.5, GPT-6 Astra, GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol, GPT-6 Sol, GPT-6.1 Sol, GPT-5.6 Terra, GPT-5.4 nano, GPT-5.4 mini, gpt-oss-20b, gpt-oss-120b. xAI: Grok 4.20 Non-Reasoning, Grok 4.20 Reasoning. DeepSeek: DeepSeek-R1. Alibaba: Qwen 3 235B, Qwen 3 Coder 480B, Qwen 3 Next 80B Instruct, Qwen 3 Next 80B Thinking. Zhipu: GLM-5.

Findings

1. Models restore missing diacritics but keep wrong ones

Many Romanians type without diacritics ("Brasov"), and the models put them back (96% clean, against 68% for cedilla input).

They can tell. The 15 first-round models wrote ș and ț correctly from English prompts (98%), spotted the cedilla in every sentence typed entirely with it, and fixed 99% of texts when told to correct the diacritics. They just don't fix it unprompted, or mention it: none of the 3,160 answers to cedilla text did (my prompts asked for short answers).

A few models add cedillas on their own, mainly gpt-oss-120b, gpt-oss-20b and Claude Haiku 4.5 ("Șoferii sunt rugaţi…", from gpt-oss); for the rest, the cedilla comes from my text.

DeepSeek-R1 shows its reasoning, so I could check whether it notices. Without the rule, its reasoning mentioned the letters in 3 of 88 answers; with it, in 43 of 44, mostly repeating the rule, and 6 of those still came out with cedillas. In one, it planned for "proper Romanian orthography with ș/ț commas below, not cedillas" and answered "Autorităţile recomandă turiştilor să verifice starea drumului înainte de plecare." Both words carry the cedilla. Reasoning didn't help Grok 4.20 either: 41% clean with it, 45% without.

2. The keyboard echo: the model's own words get my cedilla too

Answers to questions about a document came back clean 52% of the time, summaries 68%, guest replies 96%: the more of my words an answer reuses, the more cedillas it keeps. And it's mostly all or nothing: 75% of the answers that kept any of my cedillas kept all of them.

What surprised me were the model's own words. They can't be copied, so their spelling shows which keyboard the model is typing on. Claude Opus 5 summarized a city-hall notice that asked drivers (rugați, "asked") to park elsewhere:

Şoferii sunt îndrumaţi să parcheze pe strada Mureşenilor sau lângă Gara Braşov.

Îndrumaţi ("directed") isn't in my text. The model chose it, and typed it on my keyboard. Seven other models chose the same word, cedilla and all. Across all the echo answers, when an answer kept all my cedillas, 95% of the ș/ț words it added had the cedilla; when it kept some, 9.5%. Those partial answers tended to keep the cedilla in place names like Braşov and Sighişoara, the way you'd quote a name, and wrote their own words with the comma. So there are two behaviors: quoting names, and switching keyboards.

The gap held within single models and again on the fresh run, as I'd predicted: 13 of 15 new words against 1 of 18.

Could keeping my spelling just be faithful quoting? Not for a word the model chose itself, and the same models silently turn Brasov into Brașov. Even if I excuse every place name copied from my text as a quotation, only 73% of answers are clean.

3. The documents around it turn the echo up or down

I hid each question's document among three others and typed 0, 1 or 3 of them with cedillas. Around a clean document, the neighbors hardly mattered: at least 95% of answers stayed clean. Around a cedilla document, three clean neighbors pulled answers up to 78% clean and three cedilla ones down to 36%; 14 of the 15 models moved that way, none the other.

In a chat, a first message typed with cedillas hardly mattered: only 2 of 60 clean second summaries picked up one, both from Claude Haiku 4.5.

4. Only the cedilla sticks: a dot below and mojibake get repaired

Why the cedilla? Either any near-twin of the right letter slips through, or only the familiar wrong one does. A dot below (Braṣov) tells them apart: it looks just as close, but no Romanian text uses it. Before a fresh run on 28 models I wrote down that it would be repaired, along with mojibake (Braºov), which looks broken.

27 of the 28 models kept the dot less often than the cedilla, and none kept it more. So it isn't just a near-twin slipping through.

Familiarity fits: the cedilla fills real Romanian web text, and ş is a real Turkish letter. English shows the same pattern, though I'd predicted the opposite: in answers to questions about an English document, models kept the mojibake common on the web (ZÃ¼rich) in 41%, rare Cyrillic look-alikes in 9%. Both sat in names there, so part of that may be quoting. I didn't measure how common each spelling is, so this is a hypothesis.

5. Newer can be worse: Opus 4.8 kept 95% clean, Opus 5 56%

I compared 19 upgrades on identical prompts. The three cleanest models all got worse in their next version: Opus 4.8 → 5 (95% → 56%), Sonnet 4.6 → 5 (95% → 58%), GPT-5.6 Sol → 6 (95% → 64%). Only the two Claude drops are too big to be chance, even allowing for 19 comparisons; they come from the same generation, so probably one cause (Opus 5.5 recovered part of it, Sonnet 5.5 didn't).

When you pick the best three of 43, some will slip on a rerun by chance alone, so I checked the fresh run: all three drops came back, GPT Sol's included, each by 30 points or more, while a model's rate moved a median of 9 points between identical runs. So test every upgrade, and run it more than once.

6. One sentence stops the echo, and breaks Turkish

One sentence in the system prompt lifted clean answers on cedilla input from 68% to 95% in the main test, and from 53% to 93% on the fresh run's summaries and questions in the chart: "Use correct Romanian orthography: write ș and ț with a comma below, never ş or ţ with a cedilla, even if the input uses them."

But in Turkish, ş is correct. Brașov gets Turkish guests too, so I sent the booking assistant eight messages in Turkish. Without the rule, every reply was correct; with it, 71%. Some spelled Turkish with the Romanian ș, some switched to Romanian, some avoided ş altogether. Gemini 3.7 Flash offered a honeymoon couple a "muhteșem dağ manzaralı" room (a magnificent mountain view) and a "Șarap tadımı" (wine tasting). Adding "this applies to Romanian only" brought Turkish back to 95%. (A script sorted the replies by language and I checked every flagged word; no Turkish speaker did.)

Code does better. Swapping every ş for ș broke all 112 correct Turkish replies. Swapping only in sentences without ı, ğ, ç, ö or ü fixed all 352 broken Romanian answers and changed just 5 Turkish replies, 4 of them short greetings like "Hoş geldiniz". One caveat: I wrote that rule looking at these same replies, so expect it to do a little worse on new text.

What I Got Wrong

My predictions. 2 of 20 failed and 6 were too close to call. Most had no threshold until I scored them, and I count a result within 5 points of its threshold, or a correlation that could be chance, as too close to call; the one test with thresholds written first (dot below and mojibake) went 5 for 5. The table is in the block below.
My sample. Everything rests on short paragraphs I wrote about one city, and 35 of the 43 models ran the main test once, so one model's rate carries an interval about 36 points wide. Trust the patterns more than any one rank. "A third" also depends on my mix of prompts: answers from a document kept the cedilla nearly half the time.

Two smaller mistakes, and every prediction scored

My upload. The Kaggle CLI reads task files with the system encoding, cp1250 on my laptop, so my first task file would have uploaded "Brașov" as "BraČ™ov". The task files are now pure ASCII and check a hash of their data before running.

My first fix. The fix_romanian() in the first version of this post broke Turkish, the same way the one-sentence rule does.

Four of the verdicts too close to call rest on evidence that could be chance: the rank correlations in tasks 9 and 11 (permutation p = 0.20 and 0.11), task 8's upgrades, which went both ways before the correction, and task 12's 2 of 60 chat conversations, both from Claude Haiku 4.5.

Task	Prediction	Result
8	Correct input stays near 100% for every model.	lowest: gpt-oss-120b and gpt-oss-20b, 82%	failed
8	Cedilla input fails far more often than input without diacritics.	96% vs 68% clean answers	held
8	Newer versions in a family copy the cedilla at least as often as older ones.	of 19 upgrades, 2 copied more and none copied less after a correction for multiple tests (3 and 1 at p < 0.05 before it); no overall trend (rank correlation -0.29)	inconclusive
8	The one-sentence rule lifts cedilla input to at least 90% for most models.	36 of 43 models at 90% or more	held
9	With a clean target, answers stay clean for most models even when the other three documents are cedilla.	13 of 15 models at 90% or more	held
9	The models that rewrote whole answers in cedilla in task 2 are the ones whose clean-target answers get infected.	rank correlation +0.38 over 15 models; only Claude Opus 5 and Claude Sonnet 5 had infected answers	inconclusive
9	A cedilla target is copied less when the other documents are clean.	78% vs 36% clean answers	held
10	Without a rule, Turkish replies keep ş.	100% correct	held
10	Task 7's rule makes some models write Turkish with the Romanian comma letter.	6 of 14 models, 15 replies	held
10	The scoped rule keeps Turkish intact and still fixes Romanian.	Turkish 94.6% correct (bar set when scoring: 95%; no rule: 100%, full rule: 71%), Romanian 99%	inconclusive
11	Correct English input stays near 100%.	100%	held
11	Cyrillic look-alikes are copied far more often than mojibake.	clean answers: 95% with look-alikes in the names, 80% with mojibake	failed
11	The models that copy the Romanian cedilla most also copy the look-alikes most.	rank correlation +0.55 over 10 models, resting on the 6 of 120 answers that kept a look-alike (4 models)	inconclusive
12	A cedilla first text infects some clean second summaries, mostly after a cedilla first summary.	3% infected (2 of 60 conversations, both Claude Haiku 4.5); 11% when the first summary was itself in cedilla; the chats with no cedilla at all had 1 of 60 too (Claude Haiku 4.5)	inconclusive
12	A clean first text does not protect a cedilla second text much.	83% clean after a clean first text, 68% on its own (the same models' task 2 summaries)	inconclusive
13	The dot below is kept in a smaller share of s/t letters than the cedilla (paired, 95% interval below zero).	0.9% vs 38.1% of letters kept (-37 points, -45 to -29)	held
13	The old mojibake is also kept less often than the cedilla.	1.4% vs 38.1% (-37 points, -44 to -29)	held
13	Repaired dot-below and mojibake letters become comma-below letters in at least 90% of cases.	98% comma, 1.6% cedilla	held
13	Both rules give at least 90% clean answers, and the scoped rule is within 5 points of the full rule.	scoped 94%, full 93%	held
13	In cedilla answers that keep every cedilla, at least 80% of the model's own new s/t words have the cedilla; when they keep only some, at most 30%.	13 of 15 vs 1 of 18 new words	held

What Changed in How I Think About These Models

I went in expecting a knowledge test, and the models passed it. What I hadn't thought about: the text I hand a model is also a style sample, encoding included. The model answers in that style, most closely when it summarizes or answers from my text, which is what retrieval does all day. Treat retrieved text as both content and a style sample: normalize it before the model sees it, and check the answer after.

So I fix it in code, before the text reaches the model and again after, and only where the text is Romanian:

import re
import unicodedata

CEDILLA_TO_COMMA = str.maketrans("şŞţŢ", "șȘțȚ")
T_ONLY = str.maketrans("ţŢ", "țȚ")  # Turkish has no ţ, so it is always wrong
NOT_ROMANIAN = set("ıİğĞçÇöÖüÜ")  # letters Turkish uses and Romanian words don't

def fix_romanian(text: str) -> str:
    text = unicodedata.normalize("NFC", text)  # "s" + combining cedilla -> "ş"
    return re.sub(r"[^.!?\n]+", lambda m: m.group(0).translate(
        T_ONLY if set(m.group(0)) & NOT_ROMANIAN else CEDILLA_TO_COMMA), text)

It has two blind spots: a short Turkish sentence without ı, ğ, ç, ö or ü, like "Hoş geldiniz", and a Romanian sentence quoting a name like Zürich.

Does your model do it? Send it the prompt in step 1 below and paste its answer: every cedilla letter turns red. (No box? Open the checker on its own page.)

What I'd Measure Next

Whether the echo survives a real retrieval pipeline, which cleans and chunks documents first; whether other languages with a look-alike letter from an old encoding echo it too; and whether it tracks how tokenizers split each letter (ş is common in Turkish text, ṣ is rare).

My Benchmark

Romanian Comma Below on Kaggle

Thirteen tasks, with every model answer in the run outputs. Read task 8's leaderboard; the overall score averages different tasks for different models.

Reading the leaderboard

This post's numbers come from the runs up to 10 October; later runs on the leaderboard aren't in them. Task 8's leaderboard uses each model's latest run and counts an empty answer as a failure. That's why Qwen 3 Next Thinking shows 45%, and why Grok 4.5 and 4.6, listed by Kaggle but with every call errored (they're not among my 44), show 0%.

All the numbers (the 15 first-round models)

Task scores in %. Echo with cedilla is the mean of every run on the same prompts (tasks 2 and 8); Echo + rule pools tasks 7 and 8. See counts the first true/false or code point in each answer, as Kaggle does: DeepSeek-R1 listed every letter's code point before giving the right one, so on its final answers it scores 100 (Gemini 3.5 Flash-Lite 89).

Model	Write	Echo: cedilla	Echo: no diacritics	Echo + rule	Fix	See	Restore	Code
Qwen 3 235B	92	85	100	93	89	89	90	61
Gemini 3.1 Pro	100	81	95	100	100	97	100	86
Gemini 3.7 Flash	100	76	100	100	100	97	100	86
Gemini 3.8 Flash	100	70	100	100	100	81	100	93
DeepSeek-R1	96	69	95	86	83	83	93	86
Gemini 3.5 Flash-Lite	100	67	91	100	89	83	90	93
GPT-5.5	100	66	100	100	100	100	100	93
Gemma 4 31B	100	65	95	100	89	89	90	64
Claude Sonnet 5	100	58	91	95	72	83	100	86
GPT-5.4 nano	96	58	100	82	83	83	90	93
Claude Opus 5	100	56	100	100	100	81	100	100
GPT-6 Astra	100	55	–	100	100	100	100	93
GLM-5	100	52	82	100	94	94	100	86
Claude Haiku 4.5	92	47	73	91	89	72	93	86
Grok 4.20 Reasoning	100	43	82	100	83	97	87	86

The task builder, the fake models that test the scorer, the analysis and a script that re-checks this post's numbers against the raw answers are on GitHub. Every input is generated from correctly spelled text in src/items.py, so testing another letter pair or language is a small change. Built with kaggle-benchmarks; the checker runs on LiveCodes. I used Claude as a coding and editing assistant.

If your pipeline touches Romanian, paste one of your model's answers into the checker above, and run fix_romanian() on the way in and on the way out. And if your language has its own ş, tell me in the comments.

Drum Bun: an agent that knows Romania rewrote its vignette this week

Butnar Daniel — Wed, 30 Sep 2026 08:52:51 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

TL;DR: Drum Bun tells a driver what their car needs between Romania and Germany or Austria, computed from dated, structured content in Sanity; the model never computes a price. On 18 questions in four languages: agent 18 / 18, semantic search over the same documents 16, keyword search 13, the same model without the content 4. Try it live, no login.

What I Built

Every Christmas, Romanians who live in Germany and Austria drive home through Hungary: over the 2019 holidays about 941,000 people crossed Romania's western border, 361,000 at Nădlac II alone (Romanian Border Police). The questions before the drive are always the same: which vignettes, for which days, bought where, and what else gets you fined on the way? The answers are spread over four toll operators, four languages and a lot of outdated blog posts.

This year they changed under everyone's feet:

Romania replaced its rovinietă on 1 October 2026: new seller (TollRo), prices in lei by Euro class, and a car whose class cannot be shown pays the Euro 0 rate. It went live at midnight, as planned.
Hungary added an M1 regional vignette in 2026 that can make a yearly crossing cost less than half the national one, if you know that the Pest and Komárom-Esztergom county vignettes leave a gap on the M1 that only Fejér covers.
Austria sells only digital vignettes for validity from 1 December 2026, raised the on-the-spot penalty to €200, and its 2027 prices are not published yet, so a Christmas trip cannot be priced exactly.

Drum Bun ("have a good road", what Romanians say before a journey) tells you what your car needs for your trip:

the cheapest set of products per country for your dates and vehicle, with the price valid on each travel day (and a flag when it is not published yet);
the rules that apply on those dates (winter tyres, emission zones at the destination, border checks);
the outdated claims you probably read, each linked to the fact that corrects it;
an agent you can ask in Romanian, German, Hungarian or English, which shows every Knowledge Base entry and query it used.

Demo

Live: https://drum-bun-agent.vercel.app (no login; the planner works without the model, the chat is rate-limited)

The route as a strip map drawn from the data: tolled sections red with a yellow core, free sections hollow, borders dashed.

Every answer shows its steps: planner run, Knowledge Base entries opened, GROQ queries.

Code

danielbutnar / drum-bun

Road-trip agent on Sanity Context: vignettes, tolls and rules between Romania and Germany/Austria, priced for your dates

Drum Bun

What your car needs on the road between Romania and Germany or Austria: vignettes and tolls for your dates (the cheapest combination), winter-tyre and emission-zone rules, and which claims you read online are out of date. Every answer is traced to an official source.

Live: https://drum-bun-agent.vercel.app · Demo video (2:52): https://youtu.be/X2qOuYowQro · Sanity project pd5e7gez, dataset production (public) · Built for the DEV Sanity Challenge, Path One: the post.

"Drum bun!" is what Romanians say before a journey, and what the sign says when you leave a town.

Why this needs structure

"What does Brașov → Munich on 20 December, back 3 January, cost in a Euro 5 diesel?" has no page that answers it. The answer is computed from:

the route: which road sections, in which Hungarian counties, which are tolled (the Salzburg Nord → Walserberg stretch is free, the M1 between…

View on GitHub

How I Used Sanity

1. Structured content that computes

"Brașov to Munich, 20 December, back 3 January, Euro 5 diesel" has no page that answers it. So the content is modelled as the pieces the answer is computed from:

route → ordered legs → roadSection references. A section has tolled, coveredBy[] (the products that make it legal, any one is enough), counties[] and exemptVehicles[].
tollProduct with validity (days, months, calendarYear with overlap windows) and dated prices[]: each price has validFrom/validTo and an optional emission band (euroMin, euroMax, electric, appliesWhenUnknown).
rule with a yearly season window, conditional + condition, effectiveFrom/To and severity; zone with dieselMinEuro and status.
claim: a statement found online, its verdict (outdated, wrong, misleading), the source it was seen on, and correctedBy[] references to the facts that are true now.
source with trust (official, club, press, blog, forum), language and checkedAt. Every fact references at least one.

Two of the stored prices of the 12-month rovinietă, as the planner reads them:

{"amount": 292, "currency": "RON", "validFrom": "2026-10-01", "validTo": "2026-12-31",
 "band": {"label": "Euro IV–V", "euroMin": 4, "euroMax": 5}}
{"amount": 330, "currency": "RON", "validFrom": "2026-10-01", "validTo": "2026-12-31",
 "band": {"label": "Euro 0–III or unknown", "euroMin": 0, "euroMax": 3, "appliesWhenUnknown": true}}

A pure planner with 21 unit tests walks that structure: it picks the price valid on each date, runs a small dynamic program for the cheapest cover (two 1-day vignettes beat one 2-month; a 2026 Austrian annual vignette still covers 3 January 2027), solves an exact set cover over Hungarian county vignettes and the M1 regional one, and applies Austria's 18-day rule for online purchases. The agent calls it as a tool. The model never computes a price.

2. A Knowledge Base from official pages in four languages

The Knowledge Base "Drum Bun road rules" is built from the dataset (a GROQ query that flattens each document into readable fields) plus 20 web pages: the toll operators, ministries and cities in Romanian, Hungarian, German and English, and the blogs drivers actually read. The first build filed 14 issues; after all fixes and a full rebuild the Knowledge Base has 22 entries from 103 dataset documents and 20 web pages, 21 decided issues and 9 standing instructions. I resolved them with @sanity/client's context API instead of clicking through the Dashboard, so every decision is in the repo with its reason (decisions.md):

Two of the four conflicts came from my own vocabulary: the build read my enum value carTrailer as "a trailer" and concluded trailers must carry warning triangles. One standing instruction fixed the vocabulary for every future build. Of the other two, one was true for different dates (Austria's substitute toll was €120 before 2026, €200 since), so an instruction now makes entries state the date; the other was a Hungarian motorcycle price that is right once you know motorcycles buy the car's annual vignette.
It invented Hungarian purchase points (post offices, the automobile club) that no source names. An instruction limits entries to the channels the sources name.
It missed one: an entry quoted the Romanian Interior Ministry page's winter-tyre fine, computed with an old penalty-point value. I wrote an instruction scoped to that page; the background contradiction check then filed an issue against the stale entry by itself, and applying it fixed the number. That loop, human instruction → automatic check → issue → rebuild, is the part I would not want to build myself.
It keeps up. On 1 October, the day Romania switched, I updated the dataset and ran a refresh: it re-checked every source and filed four issues, all about pages it had imported but never cited (CNAIR's own TollRo announcement, and four more pages with outdated prices or the German car-toll myth). I applied them and rebuilt the two entries that still mentioned the postponement bill.

One of the conflicts the build filed, with the decision and the reason.

The public How it knows page shows every issue with both sides, what was kept, the standing instructions and the outline, from a snapshot the script writes into the dataset.

3. Sanity Context MCP, two endpoints

One endpoint serves one mode, so there are two: drum-bun-kb (Knowledge Base mode: outline + knowledge_base_read/knowledge_base_search) and drum-bun-rules (GROQ mode over the dataset, embeddings enabled). The agent (AI SDK 7, AI Gateway) fetches both initial contexts once, inlines them, and gets the planner as a third tool. Conversations are saved with Sanity Context Insights; a daily cron classifies them.

4. Does structure beat search?

Eighteen questions in Romanian, German, Hungarian and English, each with facts an answer must contain and outdated facts it must not (questions). Three baselines get the same model, the same prompt and output budget, and the same fields of the same Sanity documents the Knowledge Base ingests. Only retrieval differs. (Re-run on 1 October with semantic search added, because an any-word keyword search alone is an easy opponent.)

	correct
Drum Bun agent (planner + Knowledge Base + GROQ via Sanity Context)	18 / 18
Semantic search over the same documents (embeddings, top 6)	16 / 18
Keyword search over the same documents (any word, top 6)	13 / 18
The same model with no documents	4 / 18

The model on its own gets only the facts that have not changed in years (4 / 18). Give it the structured documents and even a search box answers most single facts, because the dataset states them plainly, with dates and Euro bands: that is the first point for structured content. Search breaks where the answer has to be put together from many documents. For the Christmas trip, semantic search's six hits covered Romania and Munich and nothing on Hungary or Austria. Asked which county vignettes cover the drive from Nădlac, it named the four M1 counties and forgot the two on the M43 and M5. The planner walks the route, so it cannot forget a country or a county.

Scoring notes: the model writes non-breaking hyphens ("1‑day", "Bács‑Kiskun") that my patterns did not match, so the scorer now reads them as plain hyphens for every system (before that fix: 16, 15, 12, 4). One of the new Hungarian questions turned out lenient: keyword search passes it by naming the counties without the cheapest combination. Every run, with every answer, is in evals/runs; the method is in RESULTS.md. The first version of this post reported the 29 September run: 15 / 16 against keyword search's 11 / 16.

Honest notes: the first run, on 29 September, scored the agent 6 / 16 against keyword search's 9. It asked "which city?" for general questions, answered an English question in Romanian, returned empty answers when reasoning tokens ate the output budget, and hit the free model tier's 5-requests-a-minute limit. The fixes were a tool for single-product prices, language detection in code, a bigger output budget and a paced runner. That moment, from the failing answers to the paced re-run:

Building Drum Bun with Claude Code claude-opus-5-5

Agent

The repo is public at https://github.com/danielbutnar/drum-bun with everything committed so far. Checking the evaluation:

Agent

▸ Bash node -e " const r=JSON.parse(require('fs').readFileSync('evals/runs/2026-09-2...

Input

$ node -e "
const r=JSON.parse(require('fs').readFileSync('evals/runs/2026-09-29T18-29-39-078Z.json','utf8'));
for(const x of r.results){if(x.agent.pass)continue;console.log('=== '+x.id+' missing:'+JSON.stringify(x.agent.missing)+' wrong:'+JSON.stringify(x.agent.wrong));console.log(x.agent.text.slice(0,500).replace(/\n+/g,' | '))}"

Output

=== christmas-munich missing:["1-day|one-day|1 day"] wrong:[]
Pe scurt — ca să nu fiți amendat: | - România: rovinietă 1 zi pentru 20.12.2026 și pentru 03.01.2027 — 26 RON/zi (2×26 = 52 RON). [CNAIR (RO)](https://portal.etoll.ro/) / [Ordin MTI 888/2026 (RO)](https://lege5.ro/Gratuit/ge4tqnzthazti/ordinul-nr-888-2026-pentru-aprobarea-categoriilor-de-vehicule-si-a-nivelului-rovinietei)   | - Ungaria: e‑vignette D1 (daily) pentru 20.12.2026 și 03.01.2027 — 5.550 HUF/zi (2×5.550 = 11.100 HUF). Cumpărați înainte de a intra pe autostradă, introduceți corect numărul
=== at-18-days missing:["18","(point of sale|petrol station|sales point|toll station|vending machine|10-day)"] wrong:[]
Which Romanian start city are you leaving from — Brașov, Bucharest, Cluj‑Napoca or Timișoara?
=== ro-unknown-euro missing:["39"] wrong:[]
Which exact travel date do you mean? (give one YYYY‑MM‑DD date after 2026-10-01)
=== ro-7-day missing:["(nu|no)","10 zile|10 days"] wrong:[]
Te referi la rovinieta pentru România? Pot presupune că ești cu o mașină privată (car) sau vrei să-mi spui alt tip de vehicul?
=== de-ms-tyres missing:["(Alpine|Schneeflocke|3PMSF|Bergpiktogramm)","2024"] wrong:[]
Kurzfrage: Von welcher rumänischen Stadt starten Sie, welches Ziel in Deutschland und an welchen Datum(en) fahren Sie?
=== euro5-cities missing:["Munich.{0,200}(yes|allowed|may enter|can)","Stuttgart.{0,300}(not|banned|no)"] wrong:[]
What date will you travel (YYYY-MM-DD)?
=== hu-surcharge missing:["27,?790|27 790","60"] wrong:[]
I can check that precisely, but I need your trip details first. What is your Romanian start city (Brașov, Bucharest, Cluj-Napoca or Timișoara), destination (Vienna, Munich, Stuttgart, Frankfurt or Berlin) and travel date(s)?
=== at-ersatzmaut missing:["200"] wrong:[]
Für welches Datum 2026 soll ich die Ersatzmaut prüfen?
=== hu-buy-after missing:["60 minutes"] wrong:[]

=== ro-winter-dates missing:["(snow|ice|conditions)"] wrong:[]

Agent

▸ Bash cd ~/code/danielbutnar/drum-bun/web && node - <<'JS' const fs=require('fs') l...

Input

$ cd ~/code/danielbutnar/drum-bun/web && node - <<'JS'
const fs=require('fs')
let s=fs.readFileSync('lib/agent.ts','utf8')
const rep=(a,b)=>{if(!s.includes(a))throw new Error('missing '+a.slice(0,60));s=s.replace(a,b)}
rep(`- If the trip is missing details the planner needs (start city, destination, dates, vehicle), ask one short question, or assume a car and say so. Unknown fuel or Euro class is fine: the planner prices the worst case and says so.`,
`- Most questions are general (a rule, one product's price, a fine, a claim someone read). Answer those directly from the Knowledge Base or GROQ. Never ask for a start city, destination or dates unless the user wants a full trip plan.
- For a full trip plan, call plan_trip with what you have: assume a car if the vehicle is missing and say so. Unknown fuel or Euro class is fine: the planner prices the worst case and says so. Ask one short question only when the start city, the destination or the departure date is missing.
- A question about a price or rule "after" or "from" a date means that date: do not ask which date.`)
rep(`- Answer in the user's language (Romanian, German, Hungarian or English).`, `- Answer in the language of the user's latest message (Romanian, German, Hungarian or English), even if the trip is in Romania.`)
rep(`      maxOutputTokens: 1400,`, `      // Reasoning tokens come out of the same budget: too low and the answer is empty.
      maxOutputTokens: 5000,`)
fs.writeFileSync('lib/agent.ts',s)
JS
sleep 3; npx tsx ../evals/run.mts http://localhost:3000 at-18-days,ro-unknown-euro,ro-7-day,de-ms-tyres,euro5-cities,hu-surcharge,at-ersatzmaut,hu-buy-after,ro-winter-dates,christmas-munich 2>&1 | grep -v "Assertion\|^\s*at "

Output

christmas-munich   agent fail  baseline fail  tools: plan_trip
at-18-days         agent PASS  baseline PASS  tools: plan_trip,knowledge_base_read
ro-unknown-euro    agent fail  baseline PASS  tools: 
ro-7-day           agent PASS  baseline fail  tools: knowledge_base_read
de-ms-tyres        agent PASS  baseline PASS  tools: knowledge_base_search,knowledge_base_read
euro5-cities       agent fail  baseline fail  tools: knowledge_base_search
hu-surcharge       agent fail  baseline fail  tools: knowledge_base_read
at-ersatzmaut      agent PASS  baseline PASS  tools: knowledge_base_read
hu-buy-after       agent PASS  baseline PASS  tools: knowledge_base_search,knowledge_base_read
ro-winter-dates    agent fail  baseline PASS  tools: knowledge_base_search,knowledge_base_read

agent 5/10, keyword baseline 6/10
written /~/code/danielbutnar/drum-bun/evals/runs/2026-09-29T18-34-09-725Z.json

Agent

▸ Bash cd ~/code/danielbutnar/drum-bun && node -e " const r=JSON.parse(require('fs')...

Input

$ cd ~/code/danielbutnar/drum-bun && node -e "
const r=JSON.parse(require('fs').readFileSync('evals/runs/2026-09-29T18-34-09-725Z.json','utf8'));
for(const x of r.results){if(x.agent.pass)continue;console.log('=== '+x.id+' missing:'+JSON.stringify(x.agent.missing)+' wrong:'+JSON.stringify(x.agent.wrong));console.log(x.agent.text.slice(0,900).replace(/\n+/g,' | '))}"

Output

=== christmas-munich missing:["1-day|one-day|1 day"] wrong:[]
Pe scurt — ca să nu fiți amendat: 1) rovinietă valabilă pentru fiecare zi când circulați pe drumurile naţionale din RO; 2) e‑vigneta maghiară înainte de a intra pe autostradă (plata / număr de înmatriculare corect); 3) vigneta austriacă pentru zilele pe A‑utostrăzi; 4) WWF‑sticker (Umweltplakette verde) lipită înainte de intrarea în zona low‑emission a Münchenului. Mașina Euro 5 diesel este permisă în München. | Ce cumpărați (cheapest combo sugerat) și costuri estimate: | - România: 2 × Rovinietă 1 zi (26 RON / zi) — 52 RON total. Cumpărați prin TollRo (CNAIR). [CNAIR / TollRo](https://portal.etoll.ro/)   | - Ungaria: 2 × e‑vigneta D1 (napi) — 5.550 HUF / zi → 11.100 HUF total. Cumpărați înainte pe ematrica.nemzetiutdij.hu; introduceți corect plăcuţa și codul RO. [Nemzeti Útdíj](https://ematrica.nemzetiutdij.hu/)   | - Austria: 2 × vigneta 1‑day (auto) — 9.60 EUR / zi → 19.20 EUR total. Cumpăra
=== ro-unknown-euro missing:["39"] wrong:[]
I can fetch the official price — I just need one short detail: which of these start cities is your trip from: Brașov, Bucharest, Cluj‑Napoca or Timișoara?
=== euro5-cities missing:["Munich.{0,200}(yes|allowed|may enter|can)","Stuttgart.{0,300}(not|banned|no)"] wrong:[]

=== hu-surcharge missing:["27,?790|27 790","60"] wrong:[]

=== ro-winter-dates missing:["(snow|ice|conditions)"] wrong:[]

Agent

▸ Bash grep -n "\[agent\]" -A 6 "~/.claude/projects/C--Users-danie-code-danielbutnar...

Input

$ grep -n "\[agent\]" -A 6 "~/.claude/projects/C--Users-danie-code-danielbutnar/1d3dd969-55b7-437f-822b-82660f6f0d7c/tool-results/mcp-Claude_Browser-preview_logs-1790706873153.txt" | grep -v "^\s*$" | cut -c1-300 | head -40

Output

1:[agent] stream error Error [GatewayInternalServerError]: AI Gateway requires a valid credit card on file to service requests. Please visit https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dadd-credit-card to add a card and unlock your free credits.
2-    at [3mignore-listed frames[23m {
3-  statusCode: [33m403[39m,
4-  generationId: [90mundefined[39m,
5-  isRetryable: [33mfalse[39m,
6-  type: [32m'internal_server_error'[39m,
7-  [cause]: Error [AI_APICallError]: {"error":{"message":"AI Gateway requires a valid credit card on file to service requests. Please visit https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dadd-credit-card to add a card and unlock your free credits.","type":"customer_verification_required"
--
47:[agent] ui stream error Error [GatewayInternalServerError]: AI Gateway requires a valid credit card on file to service requests. Please visit https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dadd-credit-card to add a card and unlock your free credits.
48-    at [3mignore-listed frames[23m {
49-  statusCode: [33m403[39m,
50-  generationId: [90mundefined[39m,
51-  isRetryable: [33mfalse[39m,
52-  type: [32m'internal_server_error'[39m,
53-  [cause]: Error [AI_APICallError]: {"error":{"message":"AI Gateway requires a valid credit card on file to service requests. Please visit https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dadd-credit-card to add a card and unlock your free credits.","type":"customer_verification_required
--
93:[agent] stream error Error [GatewayInternalServerError]: Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted access.
94-    at [3mignore-listed frames[23m {
95-  statusCode: [33m403[39m,
96-  generationId: [90mundefined[39m,
97-  isRetryable: [33mfalse[39m,
98-  type: [32m'internal_server_error'[39m,
99-  [cause]: Error [AI_APICallError]: {"error":{"message":"Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted access.","type":"no_providers_available","param":{"error":"Free tier users do no
--
140:[agent] ui stream error Error [GatewayInternalServerError]: Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted access.
141-    at [3mignore-listed frames[23m {
142-  statusCode: [33m403[39m,
143-  generationId: [90mundefined[39m,
144-  isRetryable: [33mfalse[39m,
145-  type: [32m'internal_server_error'[39m,
146-  [cause]: Error [AI_APICallError]: {"error":{"message":"Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted access.","type":"no_providers_available","param":{"error":"Free tier users do n
--
187:[agent] stream error Error [AI_RetryError]: Failed after 3 attempts. Last error: GatewayRateLimitError: Rate limit exceeded for openai/gpt-5-mini for provider azure: this team's limit of 5 requests per minute (across all regions) was reached. Retry after 60s.
188-    at [3mignore-listed frames[23m {
189-  cause: [90mundefined[39m,
190-  reason: [32m'maxRetriesExceeded'[39m,
191-  errors: [
192-    Error [GatewayRateLimitError]: Rate limit exceeded for openai/gpt-5-mini for provider azure: this team's limit of 5 requests per minute (across all regions) was reached. Retry after 60s.
193-        at [3mignore-listed frames[23m {
--

Agent

▸ Bash cd ~/code/danielbutnar/drum-bun/web && cat >> lib/plan-tool.ts <<'EOF' expor...

Input

$ cd ~/code/danielbutnar/drum-bun/web && cat >> lib/plan-tool.ts <<'EOF'

export const COUNTRIES = ['RO', 'HU', 'AT', 'DE'] as const

const PRODUCTS_QUERY = `*[_type == "tollProduct" && country->code == $country && $vehicle in vehicles] | order(kind asc, name.en asc){
  _id, name, kind, vehicles, localCategory, county, validity, prices[]{amount, currency, validFrom, validTo, status, band},
  activation, purchase[]{channel, label, url}, summary, penalty,
  pendingChanges[]{summary, wouldTakeEffect, checkAgainBy, "source": source->{title, url}},
  "sources": sources[]->{title, url, publisher, language, trust}
}`

// One country's products with the price valid on a date, for questions about
// a single product ("what does X cost after 1 October?"). Deterministic, one
// round trip, no trip details needed.
export const countryProductsTool = tool({
  description:
    'List every toll product of one country (RO, HU, AT) for a vehicle, with the official price valid on a given date, validity rule, online-purchase delay, penalty and sources. Use it for questions about one product, a price on a date, a Euro-class price, or a penalty, when no full trip is involved.',
  inputSchema: z.object({
    country: z.enum(COUNTRIES),
    date: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).describe('The date the price should be valid on. Use today if the user gave none.'),
    vehicle: z.enum(['car', 'carTrailer', 'camper', 'motorcycle']),
    fuel: z.enum(['petrol', 'diesel', 'hybrid', 'electric', 'lpg']).nullable(),
    euroNorm: z.number().int().min(0).max(6).nullable(),
  }),
  execute: async ({country, date, vehicle, fuel, euroNorm}) => {
    const products = await sanity.fetch<TollProduct[]>(PRODUCTS_QUERY, {country, vehicle})
    return products.map((p) => {
      const hit = priceOn(p.prices ?? [], p.validity.unit === 'calendarYear' ? `${date.slice(0, 4)}-07-01` : date, {fuel, euroNorm})
      return {
        name: p.name.en,
        localName: p.name.ro ?? p.name.hu ?? p.name.de ?? null,
        kind: p.kind,
        county: p.county ?? null,
        validity: p.validity.wording,
        priceOnDate: hit
          ? {amount: hit.price.amount, currency: hit.price.currency, band: hit.price.band?.label ?? null, estimate: hit.estimate, bandNote: hit.bandNote}
          : null,
        onlineDelayDays: p.activation?.onlineDelayDays ?? 0,
        immediateOptions: p.activation?.immediateOptions ?? null,
        penalty: p.penalty?.note ?? null,
        buy: (p.purchase ?? []).filter((c) => c.url).map((c) => ({label: c.label, url: c.url})),
        pendingChanges: p.pendingChanges ?? [],
        sources: p.sources,
      }
    })
  },
})
EOF
node - <<'JS'
const fs=require('fs')
let s=fs.readFileSync('lib/plan-tool.ts','utf8')
s=s.replace("import {planTrip} from './planner'\nimport {fetchPlannerData} from './queries'", "import {planTrip, priceOn} from './planner'\nimport {fetchPlannerData, sanity} from './queries'\nimport type {TollProduct} from './types'")
fs.writeFileSync('lib/plan-tool.ts',s)

let a=fs.readFileSync('lib/agent.ts','utf8')
const rep=(x,y)=>{if(!a.includes(x))throw new Error('missing '+x.slice(0,60));a=a.replace(x,y)}
rep("import {planTripTool, today} from './plan-tool'", "import {countryProductsTool, planTripTool, today} from './plan-tool'")
rep(`function instructions(date: string) {`, `const LANGUAGE_NAMES = {ro: 'Romanian', de: 'German', hu: 'Hungarian', en: 'English'} as const

/** Cheap language detection for the reply: the model drifts into Romanian when a trip starts in Romania. */
export function detectLanguage(text: string): keyof typeof LANGUAGE_NAMES {
  const t = \` \${text.toLowerCase()} \`
  const score = (re: RegExp) => (t.match(re) ?? []).length
  const ro = score(/[ăâîșşțţ]|\\s(și|să|pentru|cu|ce|care|este|mașin\\w*|rovinieta|pot|trebuie|nu)\\s/g)
  const de = score(/[äöüß]|\\s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|mein\\w*|brauche)\\s/g)
  const hu = score(/[őű]|\\s(és|hogy|nem|kell|autópálya|mennyi|megyei|vagy)\\s/g)
  const best = Math.max(ro, de, hu)
  if (best === 0) return 'en'
  return best === ro ? 'ro' : best === de ? 'de' : 'hu'
}

function instructions(date: string) {`)
rep(`1. plan_trip: the deterministic planner`, `1. plan_trip and country_products: the deterministic planner and price lookup`)
rep(`For any question about what a trip needs or costs, call it first. Never compute prices, validity or coverage yourself.`, `For any question about what a trip needs or costs, call plan_trip first. For one product's price on a date (e.g. by Euro class) or a penalty, call country_products. Never compute prices, validity or coverage yourself.`)
rep(`    const system = [
      instructions(today()),`, `    const lastUser = [...messages].reverse().find((m) => m.role === 'user')
    const lang = detectLanguage(lastUser?.parts.map((p) => (p.type === 'text' ? p.text : '')).join(' ') ?? '')
    const system = [
      instructions(today()),
      \`The user's latest message is in \${LANGUAGE_NA
[… 1299 characters trimmed]

Output

ok
TSC-DONE

Agent

The code typechecks. Pulling the new Insights token into local env, adding a language-detection test, and pacing the evaluation to stay under 5 requests a minute:

Agent

▸ Bash cat > web/lib/agent.test.ts <<'EOF' import {describe, expect, it} from 'vites...

Input

$ cat > web/lib/agent.test.ts <<'EOF'
import {describe, expect, it} from 'vitest'

import {detectLanguage} from './agent'

describe('detectLanguage', () => {
  it('recognises the four languages of the route', () => {
    expect(detectLanguage('Cât costă rovinieta de 12 luni pentru un diesel Euro 5?')).toBe('ro')
    expect(detectLanguage('Reichen M+S-Reifen im Winter in Deutschland?')).toBe('de')
    expect(detectLanguage('Mennyi a megyei matrica és hogy kell megvenni?')).toBe('hu')
    expect(detectLanguage('I drive from Brașov to Munich on 20 December. What do I need?')).toBe('en')
  })
})
EOF
node - <<'JS'
const fs=require('fs')
let s=fs.readFileSync('evals/run.mts','utf8')
s=s.replace("  const [agent, baseline] = await Promise.allSettled([askAgent(q.q), askBaseline(q.q)])",
"  // The free AI Gateway tier allows 5 requests a minute: run one system at a\n  // time and pause, so a rate limit never counts as a wrong answer.\n  const [agent] = await Promise.allSettled([askAgent(q.q)])\n  await sleep(PAUSE_MS)\n  const [baseline] = await Promise.allSettled([askBaseline(q.q)])\n  await sleep(PAUSE_MS / 3)")
s=s.replace("const only = process.argv[3]?.split(',')", "const only = process.argv[3]?.split(',')\nconst PAUSE_MS = Number(process.env.EVAL_PAUSE_MS ?? 45_000)\nconst sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))")
s=s.replace("      if (ev.type === 'tool-input-available') tools.push(ev.toolName)", "      if (ev.type === 'tool-input-available') tools.push(ev.toolName)\n      if (ev.type === 'error') text += `\\n[ERROR] ${ev.errorText}`")
fs.writeFileSync('evals/run.mts',s)
JS
cd web && npx vitest run lib 2>&1 | tail -3

Output

[2m   Start at [22m 21:36:03
[2m   Duration [22m 2.72s[2m (transform 396ms, setup 0ms, collect 1.68s, tests 32ms, environment 1ms, prepare 911ms)[22m

Agent

Input

$ npx vitest run lib 2>&1 | grep -E "Tests|FAIL|passed|failed" | head -5

Output

 [31m❯[39m lib/agent.test.ts [2m([22m[2m1 test[22m[2m | [22m[31m1 failed[39m[2m)[22m[32m 12[2mms[22m[39m
[31m⎯⎯⎯⎯⎯⎯⎯[39m[1m[41m Failed Tests 1 [49m[22m[31m⎯⎯⎯⎯⎯⎯⎯[39m
[41m[1m FAIL [22m[49m lib/agent.test.ts[2m > [22mdetectLanguage[2m > [22mrecognises the four languages of the route
[2m Test Files [22m [1m[31m1 failed[39m[22m[2m | [22m[1m[32m1 passed[39m[22m[90m (2)[39m
[2m      Tests [22m [1m[31m1 failed[39m[22m[2m | [22m[1m[32m21 passed[39m[22m[90m (22)[39m

Agent

The English question with "Brașov" in it was detected as Romanian, the exact failure the check is meant to catch. The fix: ignore place names, and count English words too.

Agent

▸ Bash node - <<'JS' const fs=require('fs') let a=fs.readFileSync('lib/agent.ts','ut...

Input

$ node - <<'JS'
const fs=require('fs')
let a=fs.readFileSync('lib/agent.ts','utf8')
const start=a.indexOf('export function detectLanguage')
const end=a.indexOf('\n}\n', start)+3
a=a.slice(0,start)+`export function detectLanguage(text: string): keyof typeof LANGUAGE_NAMES {
  // Place names carry diacritics in every language here (Brașov, München), so
  // they must not decide the language.
  const t = \` \${text.toLowerCase()} \`.replace(/brașov|brasov|timișoara|bucurești|cluj-napoca|nădlac|borș|münchen|wien|győr|hegyeshalom/g, ' ')
  const score = (re: RegExp) => (t.match(re) ?? []).length
  const ro = score(/[ăâîșşțţ]|\\s(și|să|pentru|cu|ce|care|este|mașin\\w*|rovinieta|pot|trebuie|nu|cât|costă)\\s/g)
  const de = score(/[äöüß]|\\s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|mein\\w*|brauche|im|in der)\\s/g)
  const hu = score(/[őű]|\\s(és|hogy|nem|kell|autópálya|mennyi|megyei|vagy|matrica)\\s/g)
  const en = score(/\\s(the|and|what|do|does|i|to|from|on|is|my|need|cost|can|which|are)\\s/g)
  const best = Math.max(ro, de, hu, en)
  if (best === 0 || best === en) return 'en'
  return best === ro ? 'ro' : best === de ? 'de' : 'hu'
}
`+a.slice(end)
fs.writeFileSync('lib/agent.ts',a)
JS
npx vitest run lib 2>&1 | grep -E "Tests |FAIL" | head -3

Output

[31m⎯⎯⎯⎯⎯⎯⎯[39m[1m[41m Failed Tests 1 [49m[22m[31m⎯⎯⎯⎯⎯⎯⎯[39m
[41m[1m FAIL [22m[49m lib/agent.test.ts[2m > [22mdetectLanguage[2m > [22mrecognises the four languages of the route
[2m      Tests [22m [1m[31m1 failed[39m[22m[2m | [22m[1m[32m21 passed[39m[22m[90m (22)[39m

Agent

▸ Bash npx vitest run lib/agent.test.ts 2>&1 | grep -E "Expected|Received|toBe|❯ lib...

Input

$ npx vitest run lib/agent.test.ts 2>&1 | grep -E "Expected|Received|toBe|❯ lib" | head -6

Output

Expected: [32m"[7md[27me"[39m
Received: [31m"e[7mn[27m"[39m

Agent

▸ Bash cat > /tmp/lang.test.ts <<'EOF' EOF node -e " const t=' reichen m+s-reifen im...

Input

$ cat > /tmp/lang.test.ts <<'EOF'
EOF
node -e "
const t=' reichen m+s-reifen im winter in deutschland? ';
const sc=(re)=>(t.match(re)??[]).length;
console.log('de',sc(/[äöüß]|\s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|mein\w*|brauche|im|in der)\s/g),'en',sc(/\s(the|and|what|do|does|i|to|from|on|is|my|need|cost|can|which|are)\s/g))"; grep -n "reichen" lib/agent.ts

Output

de 2 en 0
29:  const de = score(/[äöüß]|s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|meinw*|brauche|im|in der)s/g)

Agent

My scripted edit lost the regex backslashes (\s became s). Rewriting the function directly:

Agent

▸ Read ~\code\danielbutnar\drum-bun\web\lib\agent.ts

Input

~\code\danielbutnar\drum-bun\web\lib\agent.ts

Output

20  const LANGUAGE_NAMES = {ro: 'Romanian', de: 'German', hu: 'Hungarian', en: 'English'} as const
21  
22  /** Cheap language detection for the reply: the model drifts into Romanian when a trip starts in Romania. */
23  export function detectLanguage(text: string): keyof typeof LANGUAGE_NAMES {
24    // Place names carry diacritics in every language here (Brașov, München), so
25    // they must not decide the language.
26    const t = ` ${text.toLowerCase()} `.replace(/brașov|brasov|timișoara|bucurești|cluj-napoca|nădlac|borș|münchen|wien|győr|hegyeshalom/g, ' ')
27    const score = (re: RegExp) => (t.match(re) ?? []).length
28    const ro = score(/[ăâîșşțţ]|s(și|să|pentru|cu|ce|care|este|mașinw*|rovinieta|pot|trebuie|nu|cât|costă)s/g)
29    const de = score(/[äöüß]|s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|meinw*|brauche|im|in der)s/g)
30    const hu = score(/[őű]|s(és|hogy|nem|kell|autópálya|mennyi|megyei|vagy|matrica)s/g)
31    const en = score(/s(the|and|what|do|does|i|to|from|on|is|my|need|cost|can|which|are)s/g)
32    const best = Math.max(ro, de, hu, en)
33    if (best === 0 || best === en) return 'en'
34    return best === ro ? 'ro' : best === de ? 'de' : 'hu'
35  }
36  
37  function instructions(date: string) {

Agent

▸ Edit ~\code\danielbutnar\drum-bun\web\lib\agent.ts

Input

~\code\danielbutnar\drum-bun\web\lib\agent.ts

Output

The file ~\code\danielbutnar\drum-bun\web\lib\agent.ts has been updated successfully.

Agent

▸ Bash npx vitest run lib 2>&1 | grep -E "Tests |FAIL" | head -3; grep -n '\\\\s\|(?...

Input

$ npx vitest run lib 2>&1 | grep -E "Tests |FAIL" | head -3; grep -n '\\\\s\|(?=' lib/agent.ts | head -3

Output

[2m      Tests [22m [1m[32m22 passed[39m[22m[90m (22)[39m
28:  const ro = score(/[ăâîșşțţ]|\s(și|să|pentru|cu|ce|care|este|mașin\w*|rovinieta|pot|trebuie|nu|cât|costă)(?=\s)/g)
29:  const de = score(/[äöüß]|\s(ich|und|der|die|das|nicht|mit|für|reichen|wie|ist|mein\w*|brauche|im)(?=\s)/g)
30:  const hu = score(/[őű]|\s(és|hogy|nem|kell|autópálya|mennyi|megyei|vagy|matrica)(?=\s)/g)

Agent

All 22 tests pass. Checking the editor's two type complaints against the real compiler:

Agent

▸ Bash npx tsc --noEmit -p . 2>&1 | grep -v "LayoutProps\|PageProps" | head -5; echo...

Input

$ npx tsc --noEmit -p . 2>&1 | grep -v "LayoutProps\|PageProps" | head -5; echo TSC-DONE; cd .. && git add -A && git commit -qm "Agent: country_products tool, language detection, rate-limit handling; eval pacing; cover image and post draft

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" && git push -q origin main 2>&1 | tail -2; git log --oneline | head -1

Output

TSC-DONE
dac1675 Agent: country_products tool, language detection, rate-limit handling; eval pacing; cover image and post draft

Agent

Both were stale; the compiler is clean. Committed and pushed. Starting the full paced evaluation in the background (about 20 minutes, to respect the rate limit):

What did not go smoothly

A dataset source query with select() made the ingest fail with a generic server error; plain projections work.
A website source crawls everything under the URL's path: the Munich city page pulled in 306 pages of a 150-source budget. Single deep URLs import one page.
GROQ mode needs a deployed Studio, not only a deployed schema.
The AI Gateway's free tier does not include Claude models and allows 5 requests a minute, so the live agent runs on gpt-5-mini, the example questions replay recorded answers (labelled as such, with an "ask it live" button), and a busy model says so instead of failing silently. The planner makes the model choice matter less.
The first eval run lost to keyword search (above). Measuring early is what made the agent good.

Sanity Project Details

Project ID: pd5e7gez, dataset production (public). Try one product's stored prices: *[_id=="product-ro-12month"][0]{name, prices}
Organization ob2cyckj9, Knowledge Base kbv1SRpT2A3t, MCP endpoints drum-bun-kb and drum-bun-rules.
Studio: https://drum-bun.sanity.studio (members only; the schema is in studio/schemaTypes).
Read the content without a login: a route with its road sections, the M1 stretch only the Fejér vignette covers, a blog claim and the products that correct it.

Agent Session

I built Drum Bun with Claude Code. The excerpt in section 4 comes from that session: the first eval losing to keyword search, and the fixes that followed.