Python's `\w+` Silently Breaks Devanagari Words

Have you ever used Python's regular expressions to tokenize text and assumed everything was working correctly?

I did, until I looked closely at what happened to Devanagari text.

Consider this simple example:

import re

text = "हिन्दी भाषा"

print(re.findall(r"\w+", text))

You might expect:

['हिन्दी', 'भाषा']

But Python actually returns:

['ह', 'न', 'द', 'भ', 'ष']

The vowel signs and virama disappear from the matches, splitting the written words into fragments.

This isn't a display problem. It's a consequence of how Python's regular expressions define a word character.

Why does this happen?

Python's \w matches Unicode alphanumeric characters and underscores. However, it does not include every Unicode combining mark.

Devanagari uses consonants, vowel signs, viramas and other marks to represent written words. Some of these marks are essential to the written form of a word, but they aren't matched by \w.

For example, the word हिन्दी contains consonants alongside combining marks. When a regular expression matches only the characters it considers word characters, those marks are excluded.

The result can be a sequence of fragments rather than the original word.

This matters if you use regex-based tokenization for:

• Word counts and vocabulary statistics
• Text preprocessing for NLP
• Search indexing
• Language analysis
• Building datasets for machine learning

If your tokenizer silently drops parts of words, the downstream results can be misleading.

Inspect the Unicode characters

Python's built-in unicodedata module makes it easier to investigate what's happening.

import unicodedata

for char in "हिन्दी":
    print(
        repr(char),
        unicodedata.name(char, "UNKNOWN"),
        unicodedata.category(char)
    )

Unicode categories help explain the behavior:

• Lo — Letter, Other; used for many Indic consonants.
• Mc — Mark, Spacing Combining.
• Mn — Mark, Nonspacing.

The consonants in the example belong to the Lo category, while the vowel signs and virama include combining marks.

Python's standard re module doesn't treat every combining mark as a \w character. That's why the regex can match the consonants while leaving out marks that belong to the same written word.

The important distinction is that Python isn't failing to recognize Devanagari text. The problem is that \w doesn't cover every character needed for this kind of tokenization.

There's another trap: Unicode normalization

Two strings can look identical but contain different sequences of Unicode code points.

For example:

import unicodedata

one = "\u0958"
two = "\u0915\u093c"

print(one == two)

print(
    unicodedata.normalize("NFC", one)
    == unicodedata.normalize("NFC", two)
)

Output:

False
True

The first string represents क़ using a single code point, while the second represents the same character using क followed by a nukta.

The strings differ before normalization, but their normalized forms compare equal.

Unicode normalization helps make canonically equivalent representations consistent. This can be useful when comparing, storing or processing text from different sources.

However, normalization does not solve tokenization by itself. Even normalized text can contain combining marks that a particular regular expression doesn't match.

A simple workaround and its limits

For a basic Devanagari text-processing experiment, we can normalize the input and explicitly include relevant Unicode ranges.

import re
import unicodedata

WORD = re.compile(
    r"[\w\u0900-\u0963\u0966-\u096F\u0971-\u097F]+"
)

def normalize(text: str) -> str:
    return unicodedata.normalize("NFC", text).strip()

def tokens(text: str) -> list[str]:
    return WORD.findall(normalize(text))

print(tokens("हिन्दी भाषा"))

Output:

['हिन्दी', 'भाषा']

The pattern combines \w with additional Devanagari code-point ranges so that the relevant marks can be included in a match.

This demonstrates one way to address the problem, but it isn't a complete tokenizer.

Unicode ranges can include characters that shouldn't necessarily be part of ordinary words. They can also require adjustments for different languages, scripts and text-processing requirements.

For a more comprehensive solution, consider a Unicode-aware regular-expression library such as Python's third-party regex package, which supports Unicode character properties. For serious NLP applications, a tokenizer designed for the target language may be more appropriate.

The right choice depends on the text you're processing and what you consider a word.

Test the behavior instead of assuming it

A few simple tests can help catch regressions.

def test_basic_words():
    assert tokens("हिन्दी भाषा") == ["हिन्दी", "भाषा"]

def test_empty_input():
    assert tokens("") == []

def test_english_words():
    assert tokens("hello world") == ["hello", "world"]

def test_punctuation():
    assert tokens("हिन्दी, भाषा!") == ["हिन्दी", "भाषा"]

def test_nukta_form():
    assert tokens("क़") == ["क़"]

These tests cover basic Devanagari words, empty input, English text, punctuation and a nukta-containing form.

You can extend them with danda (।), mixed-language text, unusual whitespace and examples from the language you're working with.

Passing a handful of tests doesn't prove that a tokenizer handles every possible input. It does, however, make its behavior explicit and gives you a foundation for testing more complex cases.

For serious NLP work, test against representative text from your target language rather than relying exclusively on a few hand-picked examples.

Final takeaway

Regular expressions are useful, but \w+ is not a universal solution for word tokenization.

When working with Devanagari or other scripts that use combining marks:

• Inspect the underlying Unicode representation.
• Normalize text where appropriate.
• Understand which characters your regex actually matches.
• Test with real examples from your target language.
• Choose a tokenizer that fits your application's requirements.

A regex that works perfectly for English text may behave very differently with another writing system.

The lesson isn't to avoid regular expressions. It's to understand their boundaries and verify their behavior before trusting the results.

Have you encountered a similar Unicode issue in Python? I'd love to hear how you handled it.