A customer showed me what a machine translation tool had done to their project plan. It had turned the English word "execution" into Hinrichtung: putting a person to death. The plan now promised the Hinrichtung of the rollout phase in the third quarter.
That error is easy to laugh at. The harder ones sound perfectly professional.
Asked to analyse a sales-law case, a model gave me a clean, well-structured answer built around Wandlung, an old remedy for a defective purchase. The relevant concept for the question I asked was Rücktritt. The current § 437 BGB provides for withdrawal from the contract under specified conditions.
The answer looked competent. Its terminology gave me a reason to check the analysis.
That is what I wanted to test: whether a model uses the terms your work depends on, in the right domain and the right jurisdiction.
What German search taught me
I ran into the same problem while tuning German keyword search.
Without suitable text analysis, a search for Ventil misses Abgasrückführungsventil. Inflected forms such as auswuchten and ausgewuchtet need normalization. Related expressions with different roots need a domain dictionary.
BM25 scores the tokens it receives. The analysis pipeline decides whether those expressions can match at all.
That work meant combining exact matching for paragraph numbers and acronyms, compound decomposition, normalization of word forms and curated synonyms. Missing matches were inspectable. I could trace a failure to a token or a dictionary entry.
Semantic search bridges many of those gaps. A plausible nearest result still does not prove it preserved the distinction that matters. Arbeitssicherheit and IT-Sicherheit both concern Sicherheit. They answer different questions.
Those language layers became a small terminology test for model answers.
A benchmark can inherit its author's preferences
Each task contains a German domain question and a short list of expected terms. The score is the share of those terms found in the answer, after the scorer applies its matching rules.
That makes the score easy to inspect. It also hands the person writing the list considerable power over the result.
My first version put one vendor's models in first, second and third place. I had written the tasks with help from that same vendor's model.
For the next version, I required every expected term to trace to a cited provision of German or EU law. Eleven tasks survived, all about vehicle inspection procedure, with 47 expected terms between them.
The ordering changed.
| Model | Terms matched, out of 47 | Term coverage |
|---|---|---|
| GPT-6 Astra | 39 | 83% |
| Mistral Large 3 | 37 | 79% |
| GLM-5 | 36 | 77% |
| GPT-5.6 Sol | 36 | 77% |
| GPT-5.6 Terra | 35 | 74% |
| GPT-5.6 Luna | 34 | 72% |
| Claude Sonnet 5 | 33 | 70% |
| Claude Haiku 4.5 | 32 | 68% |
| Claude Opus 5 | 30 | 64% |
Recorded results from one run per model on eleven vehicle-inspection tasks. Percentages are rounded. These are terminology-coverage scores, not answer-accuracy scores.
The three models that led the earlier version came last here. The domain and the task length changed at the same time, so this does not isolate the effect of who wrote the questions. It was enough to stop me trusting the first version.
The gaps are also smaller than they look. One matched term is worth about 2.1 percentage points, and several terms belong to the same question, so 47 terms are not 47 independent tests. Five of the nine models sit within three terms of each other. Read the table as a reason to open the answers.
The scorer can be wrong, too
One answer made the limitation clear.
Asked how to classify a vehicle with defects in several groups, Opus 5 gave the governing rule and an additional escalation rule. It was more complete than an answer that received full marks. It still lost a term, because it wrote einzustufen where the source wrote eingeordnet.
The wording was acceptable. My scorer did not recognize it.
Manual review put roughly 6–30% of unmatched terms, depending on the model, in that category: acceptable synonyms the matching rules missed. The scorer errs in the other direction too. It does not penalize incorrect additional terms, so an answer can list several candidate provisions, include the expected one and collect the credit. A matched term can also sit inside a sentence that uses it wrongly. I spot-checked answers rather than validating every match in context.
Those two effects pull against each other. Missed synonyms undercount good answers, and speculative lists overcount weak ones. The number is not a competence measurement in either direction.
Holding the list fixed across a change keeps the comparison useful without removing the bias. Retrieval can raise the score by feeding the model the source's vocabulary. Fine-tuning can teach the preferred wording while the reasoning stays where it was.
What the score is good for: flagging answers for review, and tracking terminology coverage next to correctness.
Separate model knowledge from access to evidence
The next useful comparison runs the same questions under several conditions.
| Condition | What it helps investigate |
|---|---|
| German question without supporting material | What the model can answer unaided |
| The same question translated into English, with the German jurisdiction preserved, then the answer translated back | Whether a translation workflow helps or introduces errors |
| German question with the correct source passage supplied | Whether the model can use the evidence when it has it |
| German question with passages selected by the actual retrieval system | Whether the full workflow can find and use the evidence |
These comparisons point to different investments: a better source, a domain glossary, a different retrieval setup, or a different model. One score cannot tell you which.
Supplying the correct passage removes retrieval failure from the test. It also stops describing what a deployed system will deliver. Finding the right passage, handling document versions and using the passage correctly stay separate problems.
Translation needs its own check. My expectation is that it helps some mechanical questions and introduces ambiguity into jurisdiction-specific legal terms. A round trip changes more than the language, so an improvement there needs inspection before anyone credits it to domain knowledge.
These are the next experiments. The table above does not settle them.
The model you can deploy may be a different choice
For a manufacturing client, the shortlist also depends on where inference is allowed to run.
Amazon Bedrock's routing options decide that. Geographic inference profiles keep processing within the specified geography. Global profiles can route to supported commercial AWS Regions worldwide. The profile defines the possible destinations, and AWS documents the distinction.
Check the exact model and inference profile, including destination Regions, before treating a benchmark entry as a deployment candidate. Calling an endpoint in an EU Region does not by itself describe the routing policy.
So the comparison worth keeping records quality and deployability together: model version, invocation settings, permitted processing locations, latency, cost. A higher terminology score cannot buy back a configuration the customer is not allowed to use.
Start with ten questions from your own work
A small local evaluation is worth building, as long as the conclusions stay as local as the questions.
- Choose ten representative questions. Use ones your team already answers from service manuals, specifications or a clause library. Include routine cases that matter and failure cases that cost money.
- Write the answer rubric before you run anything. Record the source, the required concepts, acceptable wording, and the errors that would make an answer unusable. Require exact wording where exact wording matters.
- Freeze the first version. Run three or four candidate models several times with recorded settings. Repeated runs show variability on those questions. Anything broader needs more questions.
- Review hits as well as misses. A term every model misses might expose a shared weakness, an ambiguous question or a defect in the scorer. Questions every model passes still work as regression checks.
- Compare unaided, supplied-source and retrieved-source answers. Score correctness alongside term coverage. If you change the rubric after reading outputs, version it, rescore everything, and keep fresh questions for the next round.
The model calls are the cheap part. Expert review is what makes the scores worth reading.
I built this in the evenings, alongside my work on cloud and AI platforms for manufacturing clients. Eleven tasks are a starting point. Next come repeated runs and a better synonym dictionary, then a systematic check of whether the matched terms are used correctly.
The harness and the German language layers are at github.com/vidanov/de-bench.
Fluent German is easy to notice. Whether an answer preserves the distinction your work depends on takes a more specific test.
Top comments (0)