DEV Community: Kishlay Kashyap

Python's `\w+` Silently Breaks Devanagari Words

Kishlay Kashyap — Sun, 25 Oct 2026 10:07:54 +0000

Have you ever used Python's regular expressions to tokenize text and assumed everything was working correctly?

I did, until I looked closely at what happened to Devanagari text.

Consider this simple example:

import re

text = "हिन्दी भाषा"

print(re.findall(r"\w+", text))

You might expect:

['हिन्दी', 'भाषा']

But Python actually returns:

['ह', 'न', 'द', 'भ', 'ष']

The vowel signs and virama disappear from the matches, splitting the written words into fragments.

This isn't a display problem. It's a consequence of how Python's regular expressions define a word character.

Why does this happen?

Python's \w matches Unicode alphanumeric characters and underscores. However, it does not include every Unicode combining mark.

Devanagari uses consonants, vowel signs, viramas and other marks to represent written words. Some of these marks are essential to the written form of a word, but they aren't matched by \w.

For example, the word हिन्दी contains consonants alongside combining marks. When a regular expression matches only the characters it considers word characters, those marks are excluded.

The result can be a sequence of fragments rather than the original word.

This matters if you use regex-based tokenization for:

Word counts and vocabulary statistics
Text preprocessing for NLP
Search indexing
Language analysis
Building datasets for machine learning

If your tokenizer silently drops parts of words, the downstream results can be misleading.

Inspect the Unicode characters

Python's built-in unicodedata module makes it easier to investigate what's happening.

import unicodedata

for char in "हिन्दी":
    print(
        repr(char),
        unicodedata.name(char, "UNKNOWN"),
        unicodedata.category(char)
    )

Unicode categories help explain the behavior:

Lo — Letter, Other; used for many Indic consonants.
Mc — Mark, Spacing Combining.
Mn — Mark, Nonspacing.

The consonants in the example belong to the Lo category, while the vowel signs and virama include combining marks.

Python's standard re module doesn't treat every combining mark as a \w character. That's why the regex can match the consonants while leaving out marks that belong to the same written word.

The important distinction is that Python isn't failing to recognize Devanagari text. The problem is that \w doesn't cover every character needed for this kind of tokenization.

There's another trap: Unicode normalization

Two strings can look identical but contain different sequences of Unicode code points.

For example:

import unicodedata

one = "\u0958"
two = "\u0915\u093c"

print(one == two)

print(
    unicodedata.normalize("NFC", one)
    == unicodedata.normalize("NFC", two)
)

Output:

False
True

The first string represents क़ using a single code point, while the second represents the same character using क followed by a nukta.

The strings differ before normalization, but their normalized forms compare equal.

Unicode normalization helps make canonically equivalent representations consistent. This can be useful when comparing, storing or processing text from different sources.

However, normalization does not solve tokenization by itself. Even normalized text can contain combining marks that a particular regular expression doesn't match.

A simple workaround and its limits

For a basic Devanagari text-processing experiment, we can normalize the input and explicitly include relevant Unicode ranges.

import re
import unicodedata

WORD = re.compile(
    r"[\w\u0900-\u0963\u0966-\u096F\u0971-\u097F]+"
)

def normalize(text: str) -> str:
    return unicodedata.normalize("NFC", text).strip()

def tokens(text: str) -> list[str]:
    return WORD.findall(normalize(text))

print(tokens("हिन्दी भाषा"))

Output:

['हिन्दी', 'भाषा']

The pattern combines \w with additional Devanagari code-point ranges so that the relevant marks can be included in a match.

This demonstrates one way to address the problem, but it isn't a complete tokenizer.

Unicode ranges can include characters that shouldn't necessarily be part of ordinary words. They can also require adjustments for different languages, scripts and text-processing requirements.

For a more comprehensive solution, consider a Unicode-aware regular-expression library such as Python's third-party regex package, which supports Unicode character properties. For serious NLP applications, a tokenizer designed for the target language may be more appropriate.

The right choice depends on the text you're processing and what you consider a word.

Test the behavior instead of assuming it

A few simple tests can help catch regressions.

def test_basic_words():
    assert tokens("हिन्दी भाषा") == ["हिन्दी", "भाषा"]

def test_empty_input():
    assert tokens("") == []

def test_english_words():
    assert tokens("hello world") == ["hello", "world"]

def test_punctuation():
    assert tokens("हिन्दी, भाषा!") == ["हिन्दी", "भाषा"]

def test_nukta_form():
    assert tokens("क़") == ["क़"]

These tests cover basic Devanagari words, empty input, English text, punctuation and a nukta-containing form.

You can extend them with danda (।), mixed-language text, unusual whitespace and examples from the language you're working with.

Passing a handful of tests doesn't prove that a tokenizer handles every possible input. It does, however, make its behavior explicit and gives you a foundation for testing more complex cases.

For serious NLP work, test against representative text from your target language rather than relying exclusively on a few hand-picked examples.

Final takeaway

Regular expressions are useful, but \w+ is not a universal solution for word tokenization.

When working with Devanagari or other scripts that use combining marks:

Inspect the underlying Unicode representation.
Normalize text where appropriate.
Understand which characters your regex actually matches.
Test with real examples from your target language.
Choose a tokenizer that fits your application's requirements.

A regex that works perfectly for English text may behave very differently with another writing system.

The lesson isn't to avoid regular expressions. It's to understand their boundaries and verify their behavior before trusting the results.

Have you encountered a similar Unicode issue in Python? I'd love to hear how you handled it.

The Line-Equation Method for Pattern-Printing Programs

Kishlay Kashyap — Wed, 19 Aug 2026 10:56:06 +0000

Every programming course seems to have the same unit: print a triangle of stars, then a pyramid, and, if the syllabus is feeling ambitious, a diamond. The triangles are usually straightforward. You count. The diamond is where things often get messy: four sloping edges, a pile of possible <= and >= combinations, and a nested if that may work for one input and fail for another.

There is a more systematic way to do it. Instead of guessing the conditions, treat the loop indices as coordinates, treat the edges of the pattern as lines, and derive the boundary the same way you would in coordinate geometry.

The Grid as a Coordinate System

A nested loop like

for i in range(rows):
    for j in range(cols):

visits every cell in a grid. Let i be the row and j the column. That gives us a coordinate system, with one small difference from the usual Cartesian plane: row 0 is at the top, and i increases as you move downward.

Once you look at it that way, the pattern is just a region of the grid bounded by straight lines. Give me two points on one of those edges and I can write its equation.

The tool for that is the two-point form of a line:

(j − j₁) / (i − i₁)  =  (j₂ − j₁) / (i₂ − i₁)

Substitute the two known grid points, simplify the equation, and you end up with a relationship between i and j. That is exactly what a nested-loop if condition needs.

The simplest case: a diagonal

Take the classic lower-triangle pattern, with one star at the top and a full row at the bottom:

*
* *
* * *
* * * *

Its slanted boundary runs from (i, j) = (0, 0) to (n, n), where n is the last row index. Using the two-point formula gives:

(j − 0)/(i − 0) = (n − 0)/(n − 0)   →   j = i   →   i = j

That line divides the grid into two regions. To decide which side belongs to the pattern, pick a point that you already know is inside it. The bottom-left cell (n, 0) works. Since n > 0, it satisfies i > j, so the inclusive condition is i >= j.

That is the whole derivation behind the usual lower-triangle condition.

Two points, the other way

Now flip the triangle so that it gets narrower as you move downward. The boundary runs from (0, n) to (n, 0), which is the anti-diagonal:

(j − n)/(i − 0) = (0 − n)/(n − 0)   →   j − n = −i   →   i + j = n

Test the top-left corner (0,0). There, i + j = 0, which is less than n. So the triangle containing that corner is described by i + j <= n.

Choose the opposite side of the line instead and you get the complementary triangle, i + j >= n.

So with just two line equations, we already have two basic pattern families.

The real target: a diamond

A diamond is just four of those sloping boundaries combined. Put it inside a (2n + 1) × (2n + 1) grid by using the midpoint of each side of the bounding square as a vertex: (0, n), (n, 0), (n, 2n), (2n, n).

Now apply the two-point formula to each edge:

Edge	Through	Equation
top-left	(0, n) → (n, 0)	i + j = n
top-right	(0, n) → (n, 2n)	j − i = n
bottom-left	(n, 0) → (2n, n)	i − j = n
bottom-right	(n, 2n) → (2n, n)	i + j = 3n

The equations give us the boundaries. We still need to decide which side of each line belongs to the diamond.

There is a simple way to do that. Pick a point that is definitely inside the shape. The centre (n, n) is convenient.

At that point, i + j = 2n, which is greater than n and less than 3n. Also, j − i = 0 and i − j = 0, both less than n. Those comparisons tell us which direction to use for each inequality:

if (
    i + j >= n
    and j - i <= n
    and i - j <= n
    and i + j <= 3 * n
):
    print("*", end=" ")
else:
    print(" ", end=" ")

Four lines, four comparisons, one diamond. There is no need to guess at the conditions.

Getting the complement for free

Now suppose we want the opposite region: the diamond-shaped gap, with the corners of the grid filled instead.

It is tempting to flip every comparator and keep the and:

if (
    i + j < n
    and j - i > n
    and i - j > n
    and i + j > 3 * n
):  # prints nothing

That produces an empty grid. Not a nearly empty one. No cell satisfies the condition for any positive n.

The arithmetic is fine. The logic is the problem.

The filled diamond is an and of four conditions, so its exact complement follows from De Morgan's law: negate the whole expression, which means flipping each condition and replacing and with or.

if i + j < n or j - i > n or i - j > n or i + j > 3 * n:
    print("*", end=" ")

That is the actual complement. Every cell outside the diamond matches it, and no cell inside does.

Run the two patterns together for n=3 and they partition the 49-cell grid into 25 cells and 24 cells. They do not overlap, and no cell is left unclassified. That is simply P and not-P.

A common alternative is to relax the flipped comparators back to non-strict comparisons:

if i + j <= n or j - i >= n or i - j >= n or i + j >= 3 * n:
    print("*", end=" ")

That is a different shape from the strict complement above. It is not just another way of writing the same logic.

The two versions share the boundary lines with the diamond itself. For n=3, the two printed patterns overlap on 12 cells. When printed separately, though, the non-strict version often looks like the intended pattern because its own boundary stays closed instead of leaving a one-cell gap.

The useful part here is bigger than this particular diamond. When a shape is formed by intersecting several half-planes, its complement is formed by taking the union of the flipped conditions. Strict versus non-strict boundaries is a separate decision about which pattern gets the edge.

Beyond four lines

None of this depends on having exactly four edges. The same procedure works for any shape made from straight boundaries:

Mark the shape's vertices as grid coordinates.
Connect consecutive vertices and get one line equation for each edge using the two-point formula.
Decide the direction of each inequality by testing a known interior point. Then choose >= versus > or <= versus < depending on whether the boundary itself belongs to the shape.
and together all the conditions for one convex piece.
If the target shape is not a single convex piece, split it into several pieces and combine those pieces with or.
For the complement version, apply De Morgan's law to the complete Boolean expression rather than rebuilding the logic from scratch: flip every comparator and replace and with or at the corresponding level.

That fifth step is also useful for more complicated patterns, where the target shape cannot be represented as a single convex region and must instead be built from multiple pieces.

As a quick check, take the diamond and add two horizontal boundaries: i = a and i = 2n − a, where 0 < a < n. These cut off the top and bottom points. Keep the four diagonal conditions and add the two new ones:

if (
    i + j >= n
    and j - i <= n
    and i - j <= n
    and i + j <= 3 * n
    and i >= a
    and i <= 2 * n - a
):
    print("*", end=" ")

Now there are six conditions, but the method has not changed. Two more lines simply mean two more inequalities. The diamond has become a hexagon.

Relation to Existing Approaches

The basic idea of treating a grid position as a coordinate and using line equations to derive pattern conditions is not new. Dilip Kumar's "Pattern printing coding technique" explicitly uses the (i, j) grid representation and derives conditions such as i >= j for simple shapes.

A lot of other pattern-printing tutorials take a different route and build each shape directly from loops. GeeksforGeeks' diamond-pattern reference is one example. It constructs the diamond by handling the upper and lower parts separately, then controlling the spaces and stars with nested loops.

The difference here is mainly in how the condition is derived. Instead of trying combinations until the pattern looks right, you start with the boundary points, derive the line equations, determine the correct half-planes, and combine the resulting inequalities with Boolean logic.