The Scunthorpe Problem in Devanagari: My Filter Banned 'Annual', Then Approved Everything

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Everyone knows the Scunthorpe problem: a filter blocks a perfectly innocent word because a rude one is hiding inside it. It's a punchline. It's thirty years old. I hit it in 2026 anyway — and then, while fixing it, I fell through a trapdoor underneath it that I've never seen written up.

The first bug blocked something true. The second bug approved everything, silently, and my tests went green while it did.

Here's the one-line reframe I got out of it: \b is not a word boundary. It's an ASCII opinion about where words end.

The setup

I run a bilingual audio-narration pipeline. English text goes in, a matching Hindi translation goes in, and a text-to-speech pass turns both into audio. Generating audio costs real money, so before anything paid runs there's a free validation gate: check the Hindi file for untranslated leftovers, check that the page counts line up, and check that no banned language slipped in — the content is family-audience material, so mild slurs are a hard fail.

The banned-word check started life as the simplest thing that could possibly work:

const BANNED = [/साला/, /कमीना/]; // mild Hindi slurs

function hasBannedWord(line) {
 return BANNED.some((re) => re.test(line));
}

Substring match. Ship it. It even caught a real one during testing, which is exactly the kind of early success that stops you from looking closer.

Bug #1: the gate rejected a sentence about a meeting

A validation run came back NO-GO on a line that was, as far as I could see, completely clean. It was an ordinary sentence containing the Hindi word सालाना — "annual."

Look at the letters:

• साला — the slur (literally "brother-in-law"; used as a mild insult)
• सालाना — annual
• मसाला — masala, i.e. spice
• घोटाला — scam

The slur is a substring of all three innocent words. Devanagari is an abugida: consonants carry an inherent vowel and get modified by marks, so words routinely grow by appending syllables to something that was already a valid word. Accidental containment isn't an edge case in this script. It's Tuesday.

So: Scunthorpe. Embarrassing, but a solved problem, and I knew the fix from muscle memory.

The "fix" that broke everything quietly

const BANNED = [/\bसाला\b/, /\bकमीना\b/]; // ← anchor it to word boundaries

Re-ran the gate. GO. The "annual" sentence passed. The false positive was gone.

I almost committed it right there. The only reason I didn't is a habit I'd recommend to anyone touching a safety check: after you fix a false positive, always re-run the case the filter is actually supposed to catch. Not the bug. The purpose.

The genuinely-profane test line also passed.

The filter wasn't fixed. The filter was off.

Why \b does nothing in Devanagari

\b isn't a boundary between "words" in any linguistic sense. It's a zero-width assertion that fires at a position where exactly one side is a \w character — and in JavaScript, \w is hard-wired to [A-Za-z0-9_]. ASCII. That's the whole definition.

> /\w/.test("स")
false

Every Devanagari letter is a non-\w character. A space is also a non-\w character. Two non-word characters side by side means no transition, which means no boundary — anywhere in the string. /\bसाला\b/ isn't a stricter pattern; it's an unmatchable one. I hadn't tightened the filter, I'd asked it to match at a position that cannot exist.

And the u flag doesn't save you. Unicode mode fixes how the pattern is parsed, not what \w means. Here's the actual matrix, run in node:

Input	meaning	/साला/	/\bसाला\b/	/\bसाला\b/u	fixed
सालाना बैठक	annual meeting	❌ false positive	✅	✅	✅
मसाला	spice	❌ false positive	✅	✅	✅
वह साला था	actually profane	✅ caught	❌ missed	❌ missed	✅ caught
साला	actually profane	✅ caught	❌ missed	❌ missed	✅ caught

Two bugs on one line of code, pointing in opposite directions. The first one was loud — it stopped a build and made me look at it. The second one was polite. It agreed with me. It let everything through and reported success, and the only signal it ever produced was the absence of a complaint.

That asymmetry is the actual lesson. A validator that under-matches has no failure mode you can see. A test suite full of "does the gate accept good content?" cases would have stayed green forever.

The real fix

If \b won't tell you where a word ends, say it yourself — with lookarounds over the script's Unicode block, so the pattern only matches when neither neighbour is another Devanagari character:

const BANNED = [
 { pattern: /(?<![ऀ-ॿ])(साला|साले|साली)(?![ऀ-ॿ])/, label: "profanity (Hindi)" },
 { pattern: /(?<![ऀ-ॿ])(कमीना|कमीने|कमीनी)(?![ऀ-ॿ])/, label: "profanity (Hindi)" },
];

[ऀ-ॿ] is U+0900–U+097F, the Devanagari block. "Not preceded by a Devanagari char, not followed by one" is the boundary rule \b was pretending to give me. Note that the inflections have to be listed explicitly (साला/साले/साली) — that's the flip side of a strict boundary, and it's the correct trade: an over-tight pattern fails loudly when you find a form you missed, an over-loose one fails silently.

If you'd rather not hand-roll the boundary at all, the platform now knows how to segment these scripts properly:

const seg = new Intl.Segmenter("hi", { granularity: "word" });
const words = [...seg.segment("सालाना मसाला साला")].filter((s) => s.isWordLike).map((s) => s.segment);
// → [ 'सालाना', 'मसाला', 'साला' ]

Three words, correctly split, no regex opinions involved. Tokenize, then compare against a set. It's slower than a regex, but a preflight gate that runs once per file can afford correctness. (Intl.Segmenter is in every current browser and Node 16+.)

What I'd tell past me

1. \w and \b are ASCII contracts. The moment your input isn't ASCII, they aren't doing the thing their names imply. This is not a Hindi problem — Thai, Khmer, Japanese, Chinese and Lao will all humiliate the same regex, and several of those don't use spaces at all.
2. When you fix a false positive on a filter, re-test the true positive. The two failure directions live on the same line and only one of them ever files a bug report. This is the entire moral of the story.
3. Assert your guards can still fail. My gate now has a fixture whose only job is to be rejected. If the "should be blocked" case ever passes, the build breaks. A safety check with no red test is a decoration.
4. Suspect containment in abugidas and agglutinative languages. Substring matching is a bad primitive for banned-word lists in any script, and in some scripts it's a comedy routine.
5. Free preflight checks are worth writing carefully, because the whole reason this one exists is to stand between a mistake and a bill. A gate that always says GO costs more than no gate, because you trust it.

The bug that cost me an afternoon was the one that blocked the word "annual." The bug that could have cost me something real was the fix — the one that shipped green, passed review, and asked for nothing.

\b is not a word boundary. It's an ASCII opinion about where words end. In half the world's scripts, it's an opinion that resolves to "nowhere."