Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

A classifier was sending questions to the wrong bucket. Not occasionally. I measured it against twenty real inputs and seven of them landed somewhere they did not belong. Thirty-five percent.

The routing code was thirty lines long and did the obvious thing: lowercase the incoming text, then check whether any keyword for a category appears in it. First match wins.

Every one of those thirty lines was wrong in a way that only shows up in Turkish, and two of the three problems will bite you in any language whose case mappings are not one-to-one.

The symptom

Users type free-form questions. The router picks a topic so that the right prompt and the right reference data get loaded. When it picks wrong, the answer is confidently about the wrong subject, which is worse than no answer.

The reported complaint was "the answers feel shallow." The instinct is to blame the model. The model was fine. It was being handed the wrong context.

Trap 1: lowercasing a dotted capital I produces two code points

Here is the whole thing, measured rather than remembered:

$s = "İş";                              // "work", capital İ
$l = mb_strtolower($s, "UTF-8");

// original      : U+0130 U+015F
// mb_strtolower : U+0069 U+0307 U+015F
// mb_strlen     : 2 -> 3

U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE has no single-code-point lowercase form in the default Unicode case mapping. It maps to U+0069 LATIN SMALL LETTER I followed by U+0307 COMBINING DOT ABOVE. The string got longer. It still renders as "i̇ş" and looks entirely normal in your terminal, your editor, and your database client.

So this happens:

mb_strpos(mb_strtolower("İşten çıkarılır mıyım", "UTF-8"), "iş");   // false
mb_strtolower("İSTANBUL", "UTF-8") === "istanbul";                  // false

Your keyword is two code points. The haystack has three. No match, no error, no warning. The comparison returns false and the request goes to whatever category happens to be checked next.

The reason this is not an edge case: people capitalize the first word of a sentence. In Turkish a large number of everyday words begin with İ — the words for work, request, relationship, reputation, tender, transaction, need, cancellation. Any of them starting a sentence produces the three-code-point form. The broken form is the common form, and the version your test uses — all lowercase, typed by you — is the rare one.

That is why this survives a green test suite. You wrote "işten çıkarılır mıyım" in the fixture, in lowercase, by hand. It matches. The user wrote "İşten çıkarılır mıyım". It does not.

Unicode normalization will not save you

The reflex is to reach for Normalizer::normalize() and move on. I checked:

lower    : U+0069 U+0307 U+015F
NFC      : U+0069 U+0307 U+015F
NFKC     : U+0069 U+0307 U+015F
casefold : U+0069 U+0307 U+015F

NFC composes a base character with a combining mark when a precomposed code point exists. There is no precomposed "latin small letter i with dot above" — a lowercase i already has a dot, so Unicode never needed one. The sequence stays two code points under every composing form, and casefold() does not touch it either.

NFD is more interesting, because it goes the other way and decomposes everything:

NFD("i̇ş") : U+0069 U+0307 U+0073 U+0327

Now the cedilla under ş is its own code point too. That gives you one rule that handles both problems at once: decompose, then drop every nonspacing mark.

import unicodedata as u

def norm(s: str) -> str:
    s = u.normalize('NFD', s.lower())
    s = ''.join(c for c in s if u.category(c) != 'Mn')
    return s.replace('ı', 'i')      # dotless ı has no combining mark to strip

Measured output:

'İş'         -> 'is'
'İSTANBUL'   -> 'istanbul'
'Değişiklik' -> 'degisiklik'
'İlişki'     -> 'iliski'
'Ölüm'       -> 'olum'

In PHP, where the intl extension is frequently not loaded, the same thing without a normalizer:

function tr_norm(string $s): string {
    $s = mb_strtolower($s, 'UTF-8');
    $s = str_replace("\u{0307}", '', $s);        // the leftover dot from İ
    return strtr($s, [
        'ı'=>'i','ş'=>'s','ğ'=>'g','ü'=>'u','ö'=>'o','ç'=>'c',
        'â'=>'a','î'=>'i','û'=>'u',
    ]);
}

The single most important line is the U+0307 removal. Delete it and the bug is back.

Run your keyword list through the same function, too. Both sides of a comparison must see the same normalization, always. A keyword table normalized at author time and input normalized at request time will drift the moment somebody edits the table by pasting from a document.

The other direction, for readers who do not write Turkish

Turkish has two distinct letters where English has one: dotted İ/i and dotless I/ı. They are different letters, not styling. So the correct Turkish lowercase of I is ı, not i.

This is the well-known "Turkish I problem," and it usually appears as the mirror image of what I described. In Java, .NET, or any locale-sensitive toLowerCase() running under a tr-TR locale:

"INFO".toLowerCase()          // "ınfo" under tr-TR, "info" everywhere else
"FILE".toLowerCase()          // "fıle"

Code that lowercases a protocol keyword, a header name, a file extension, or an SQL identifier and then compares it against an ASCII literal breaks on a machine whose locale happens to be Turkish. The fix there is the opposite of the one above: pass an explicit invariant locale, toLowerCase(Locale.ROOT), and never let ambient locale decide the semantics of a comparison.

So there are two rules, pointing in opposite directions depending on what you are doing:

• Comparing protocol tokens and identifiers: force a locale-invariant mapping. Never fold, never guess.
• Comparing human text for search or classification: normalize aggressively, on both sides, with a function you control.

Confusing the two is how you end up with a system that is broken in one country and subtly wrong everywhere else.

Trap 2: unbounded substring matching

With normalization fixed, the router still misrouted. The second bug had nothing to do with Unicode: the check was strpos($text, $keyword) !== false, with no word boundary.

Short keywords swallow longer words that merely contain them. Real collisions from the keyword table:

Keyword	Means	Also matches	Which means
olum	death	olumlu, olumsuz	positive, negative
mal	goods, property	nor*mal, **mal*zeme	normal, material
is	work	k*isi, degisiklik, gis*e	person, change, counter
ask	love	m*ask*e	mask
din	religion	ay*din, **din*len	bright, rest

olum was the expensive one. "Olumlu" — "positive" — is among the most common words in any yes/no question. Every such question was routed to the death-and-inheritance category, because that category happened to sit near the top of the map.

Word boundaries in an agglutinative language are not just \b around a fixed string. Turkish stacks suffixes, and stems mutate: "lost" appears as kayıp, kaybettiğim, kaybolan — the stem itself shifts from kayıp to kayb. So match a stem plus a bounded run of following letters:

$pattern = '/(?<!\p{L})' . preg_quote($stem, '/') . '\p{L}{0,6}(?!\p{L})/u';

(?<!\p{L}) and (?!\p{L}) are Unicode-aware boundary assertions — plain \b reasons about \w, whose meaning depends on flags and locale. The {0,6} allows suffixes without letting the stem float in the middle of an unrelated word. And keep a stem list per concept rather than one string: kayıp|kayb|kaybol|kaybet, because sound changes will not occur to you at the keyboard, only in production.

Trap 3: map order is a priority ranking you did not know you wrote

foreach ($map as $topic => $words) {
    foreach ($words as $w) {
        if (matches($text, $w)) return $topic;   // first hit wins
    }
}

Whatever order the array literal happens to be in is your precedence rule. "Will my father's health improve" contains both a family keyword and a health keyword. Family was declared fourth, health sixth, so it went to the family bucket every time.

Two changes fix the class of problem rather than the instance:

1. Order specific before general, deliberately, with a comment saying so.
2. Stop returning on first hit. Collect every match, score them, take the best. A reasonable score is the number of distinct stems matched, weighted by stem specificity — a rare six-letter stem is worth more than a three-letter one that appears everywhere.

Scoring also gives you something first-match never can: a confidence number. When the top two topics score within noise of each other, you can fall back to a neutral path instead of committing to a coin flip.

Normalize for matching, never for identity

One caution, because the recipe above is lossy on purpose:

'ILIK'  -> 'ilik'
'ılık'  -> 'ilik'

Two different words collapse to the same key. That is exactly what you want for search and classification, and exactly what you must never do for identity.

If you normalize usernames or email local parts this way before a uniqueness check or a login comparison, you have created an account-collision path: two visually and semantically distinct identifiers now compare equal. The same applies to case-insensitive uniqueness in general — the database collation, the application-level check and the login comparison have to agree on one rule, and that rule should be locale-invariant, not a fuzzy fold.

Fold for relevance. Compare exactly for identity. Keep the two functions in separate files so nobody reuses one for the other by accident.

How to verify, because you cannot eyeball this

Every symptom in this article is invisible on screen. The three-code-point string looks identical to the two-code-point one. You cannot review your way to correctness here; you have to measure.

What I did, and would do again:

1. Write a table of 20–30 real inputs with the expected category, taken from actual traffic rather than invented. Run it, print the wrong-rate as a number. "Seven of twenty" is a fact you can act on; "seems better now" is not.
2. Sweep for collisions mechanically. For every keyword shorter than about five characters, generate common words containing it and feed them through the classifier. That is how olum inside olumlu surfaced — nobody was going to catch it by reading.
3. Print code points, not strings, whenever a comparison fails for no visible reason. printf("U+%04X ") over the characters takes ten seconds and ends the argument.
4. Re-run the same table after the fix and report the new number. A fix that is not re-measured is a hypothesis.

The short version

• mb_strtolower('İ') returns i + U+0307. The string gets longer, looks the same, and stops matching.
• NFC and casefold do not collapse it. NFD plus stripping nonspacing marks does.
• Normalize the input and the keyword list with the same function.
• Match stems with Unicode-aware boundaries, not bare substrings.
• Map order is silent precedence. Score instead of returning on first hit.
• Fold for search; compare exactly for identity.
• Measure the wrong-rate before and after, on real inputs.

None of this is exotic Unicode trivia. It is one line of case mapping and one missing word boundary, and together they were quietly discarding a third of the routing decisions in a system that looked, from the outside, like a model quality problem.

Written from production work at Alesta WEB.