DEV Community: Wouter van Kemenade

The Executor-Plus-Gate Pattern: Why Cheap Models Need Stronger Verification

Wouter van Kemenade — Tue, 11 Aug 2026 15:01:02 +0000

Running LLM jobs over hundreds of items, the obvious shortcut is to collapse execution and verification into one model pass: one call, one output, ship it. It fails at scale, and a scoring system for 146 countries across 11 categories shows exactly why.

Each score runs 0 to 100 on a single canonical dataset, the overall rating is the arithmetic mean of those 11, and there are no per-country exceptions. One yardstick, applied identically everywhere.

Ask a cheap model to generate all 146 in one pass and you get speed with a hidden cost: drift. One country's "friendliness" score reads high because the model read it as social warmth rather than visa bureaucracy. Another's culture score inflates after the prompt happened to emphasize food over history. None of these are bugs, they're quiet inconsistencies, and at 146 items a 5% drift rate means seven countries silently failing the canonicity requirement while every individual score still looks reasonable.

The pattern

Step one: a cheap executor runs the mechanical pass. Fixed ruleset, all 146 countries in parallel batches, structured JSON out. No judgment calls, just apply rule X to field Y.

Step two: a stronger gate verifies before anything ships. Same scale everywhere? Any statistical outlier? Did a category get reweighted mid-run? This is judgment work, holding many items in view at once, and it's what a single combined pass can't do reliably.

A model doing both jobs at once optimizes for the wrong thing: it second-guesses the ruleset mid-run, adds nuance where the spec demanded consistency, and marks cases "exceptional" that shouldn't be. Splitting the two roles is faster and cheaper than one model trying to hold both contexts simultaneously.

Where the consistency requirement bites

The Country Comparison Tool's best-travel-months field works the same way: a month qualifies if it scores 70 or higher on a fixed weather index built from Open-Meteo data, no editorial override, no "tourists usually go in December anyway." One yardstick, all 146 countries, computed once and refreshed when the climate dataset updates. Add "let the model decide if a category doesn't apply here" and you've introduced exceptions. Add "one call to save tokens" and the 10th and 100th items get scored by two different reasoning paths without anyone noticing.

At five items you can eyeball drift and call it done. At 146 it's a data integrity problem, at 1,000 a regression nobody catches until a user does. The same split applies to any bulk LLM job against a fixed dataset: translation against a glossary, classification against a taxonomy. Collapse the layers and you get quiet inconsistency. Split them and drift gets caught before it ships.

Check out the Country Comparison Tool to see one canonical dataset and one scoring model produce consistent results across 146 countries.

Shipping i18n at Scale Without a Translation Platform

Wouter van Kemenade — Tue, 11 Aug 2026 15:00:45 +0000

Shipping a data-heavy tool across 5 locales, English, Dutch, German, French, Spanish, over 168 countries makes a translation platform feel inevitable. But a SaaS translation tool adds a dependency, a monthly bill, and a bottleneck: every content change means exporting strings, waiting on translators, importing segments, and validating five rollbacks.

Invert the model instead. Treat translation as deployment config, not a separate pipeline.

Glossary as config, executor plus gate

Store terminology and translation rules as a JSON glossary in version control. That glossary is the single source of truth for how a concept maps across locales.

Two agents do the work:

Executor (cheap model). Reads source content, applies the glossary mechanically, outputs all locales in parallel. Fixed rules only: substitute terms, apply locale formatting (date order, number separators), expand locale-aware blocks. No judgment calls.
Gate (stronger model). Validates every output before it ships. Checks for leaked source language, persona drift, and locale conventions the executor can't judge, catching the silent breakage a literal substitution introduces.

This decouples velocity (a cheap executor running in parallel) from quality (a gate doing deep inspection). Most content clears the gate on the first pass; regressions surface early instead of in production.

The scale that makes it matter

The Travel Safety Map applies this across 168 countries: 23 sections per profile (advisories, scams, emergency numbers, embassies, road safety), sourced from 5 government aggregators plus crime, safety and corruption indices. 152 of the 168 countries also ship in the 4 other locales, roughly 152 × 4 × 23, near 14,000 translation decisions per release. A glossary-driven executor clears that batch in minutes; the gate's real work is the custom advisory text and scam descriptions that rely on local context, not the 95% that just follows rules.

What breaks when you skip the gate

Three failure modes show up at that volume. Leaked source language: an English phrase slips through untranslated, and manual spot checks miss it in a 14,000-segment batch. Silent persona drift: German du/Sie, Spanish tú/usted, or French register shifts subtly correct-but-off in a way users notice before you do. Structural breakage: a URL, JSON key, or templated block gets translated when a glossary flag should have excluded it.

Why it transfers

Version control becomes the translation history: glossary changes are auditable commits, a release tag maps to a glossary version, rollback is a git checkout. The same executor-plus-gate split works for product tutorials, error messages, or any compliance copy across locales. i18n becomes a deployment problem you own, not a people problem you outsource, and the tradeoff is that there is no SaaS left to blame when a term is wrong.

See the pattern where the data density makes the case: 168 country safety profiles, 5 locales, 23 sourced sections each.

Why Route Ranking Needs Deterministic Scoring, Not a Language Model

Wouter van Kemenade — Tue, 11 Aug 2026 15:00:44 +0000

When you build a tool that compares travel routes: flight vs. train vs. bus, the obvious instinct is to let a language model read the variables and pick the best one. Cost per person, door-to-door time, comfort, CO2, frequency, group pricing, all fed into one prompt.

That approach breaks down fast at scale.

The LLM call problem

A model excels at explaining why a train beats a bus on one journey. It fails at consistent, reproducible ordering across 146 countries and thousands of route combinations.

Cost: rank five options per route across 7,000 routes and you're paying for millions of inference tokens to render a page. Consistency: the same route, queried twice, can return a different order because the model's read of "comfort" shifted between runs. Users refresh a bookmarked page and the ranking moves, which reads as a bug, not a feature. Latency: an API call per comparison turns a 200ms render into a round trip with retries and backoff.

The deterministic alternative

A scoring algorithm just weights the inputs:

score = (cost_weight × cost_normalized) + (time_weight × time_normalized) + (comfort_weight × comfort_normalized) + (co2_weight × co2_normalized)

Travel style, budget-first, comfort-first, middle ground, sets the weights. Rank by score. Same calculation every time, on every device, for every route. Change the travel style and the ranking recalculates client side, instantly, with no API call and no variance.

What deterministic costs you

The tradeoff is maintenance, not documentation. Every route needs real cost per person per mode, true door-to-door time including transfers, a normalized comfort metric, and estimated CO2. That data gets checked per country as routes change.

Groups complicate it further: a solo backpacker and a family of four hit different taxi and ferry pricing tiers, so the algorithm has to reweight by group size. Edge cases pile up too: a route with no flight option should just omit that mode, not return an empty ranking; seasonal gaps like ferries not running in winter need flags. None of this is a model call, it's structured logic you build and audit. Get a coefficient wrong and the ranking is wrong for everyone, silently, with no explanation attached.

The payoff

Instant re-ranking, no inference billing, the same result for the same inputs everywhere. And because the logic is deterministic, a wrong ranking is debuggable: trace the variable and weight that caused it. That is worth the maintenance burden. Model calls for a structured problem where users expect reproducible answers are a false economy.

Ready to see it in practice? Build or browse a route comparison here, and watch the ranking shift as you change your travel style.