How We Mapped 96 Minority Languages of Russia on the Internet – A Web Scraping Journey

The challenge: find 96 languages without a dictionary

Russia speaks dozens of minority languages – Tatar, Bashkir, Udmurt, even tiny ones like South Yukaghir. Some have no Wikipedia, no standard corpus, and almost no digital footprint. But they do exist online, scattered across forums, blogs, and VK communities.

We wanted to measure their online presence. As programmers, we quickly realised: no global corpus, no labelled dataset, and no simple way to ask a search engine “give me all pages in Chechen”.

So we built our own pipeline.

Step 1: Choose word markers – the hard way

You can’t download the whole internet (unless you’re Google). But search engines already crawled it. We used Yandex.XML – a limited API (10k requests/day) that returns search results page by page (each next page costs another request).

What do we search for? We need word markers:

• Frequent in the target language
• Unique – not appearing in Russian or other languages
• Free of special characters (because real users don’t type them)

That last point is killer. Most minority languages use Cyrillic extensions: ҙ, ә, ö, ҷ. There’s no standard keyboard for them. People write “менан” instead of “менән”. If we query with the “correct” spelling, we lose 90% of real texts.

We call this “household spelling” – like medieval birch‑bark manuscripts, but with modern laziness.

Step 2: Kill the noise

Even good markers return junk:

• Wikipedia duplicates (we can download those in one click, so exclude them)
• Linguistic papers with example sentences (Russian text containing one line of Rutul)
• Music sites with song titles but no lyrics
• Typos – e.g. Russian “пазух” (bosom) mistyped as “пазох”, which is a valid Khakas word meaning “again”

To filter, we used multi‑marker verification. If one marker appears on a domain but others from the same language don’t, it’s probably a false positive. That means extra API calls – expensive when you only have 10k/day.

Step 3: Classify domains by depth

We asked Yandex to categorise each domain into three tiers:

1. Language‑first – most pages in the minority language
2. Mixed – several pages, but minority language is not dominant
3. Giant platforms – YouTube, stihi.ru, etc., where minority content exists but is buried

Also, we specifically queried VK.com communities because social networks post‑2012 ate most standalone language sites.

Step 4: Detect the language (without a training corpus)

We downloaded pages and needed to separate Russian from, say, Tuvan. Classic n‑gram language detection requires a pre‑labelled corpus – which we didn’t have.

But we did have Russian – plenty of it. So we built a binary classifier: Russian vs. everything else. If a paragraph isn’t Russian, it’s our target. Not perfect, but good enough to build the first ever web corpora for these languages.

What we learned (the data science part)

We compared online metrics (Wikipedia articles, VK communities, text volume) with offline metrics (speaker count, regional economy).

Correlation (Spearman) among online metrics: >0.7

Correlation between online and offline: near zero

That means: how many people speak a language does not predict how alive it is on the internet. What matters is a small, active, passionate community.

Example: Bashkir has fewer speakers than Chechen, but far more online communities. Why? Chechnya’s internet infrastructure recovered later (1990s‑2000s conflicts delayed digital adoption).

A few geeky findings

• Post‑2012, almost all minority language activity moved to VK.com – Russian Facebook.
• Typical user: 19‑31 years old, lives in the region’s capital (Ufa for Bashkir, Izhevsk for Udmurt).
• Typical minority‑language post: ~5 words long. Most conversation is in Russian, but people sprinkle native words as identity markers.
• The minority language web is highly interconnected – sites in different languages link to each other. A rare case of “peoples united” online.

Data

Downloadable corpora and domain lists: web-corpora.net

And yes, there’s a VK community with cute memes in minority languages – featuring cats.

---

Takeaway for devs:

Web scraping isn’t just about avoiding robots.txt. It’s about understanding human behaviour – typos, keyboard limitations, social network migration, and why a language with 50 speakers can thrive online if those 50 are determined enough.