If you’ve ever scraped Wikipedia for training data, you probably assumed the text is “natural” – written by humans, balanced across topics, ready for your shiny language model.
Think again.
We analyzed Russian, Tatar, Bashkir, Mari, Mordvin, and Komi Wikipedias using a dead‑simple method: word frequency lists. The results are eye‑opening for anyone who handles text corpora.
The litmus test for a “natural” corpus
In any balanced, human‑written text collection, the most frequent words are function words – prepositions, conjunctions, pronouns, auxiliary verbs.
Example from the Russian National Corpus:
| # | Word | Part of speech | Frequency (ipm) |
|---|---|---|---|
| 1 | и (and) | conj | 35801 |
| 2 | в (in) | prep | 31374 |
| 3 | не (not) | part | 18028 |
Content words (nouns, adjectives, full verbs) appear much lower in the list.
So if your corpus’s top 20 is full of words like “river”, “basin”, “water”, “orchid” – you have a problem.
Russian Wikipedia passes (mostly)
The Russian Wikipedia (1.06M articles) top word forms:
| # | Word | Frequency |
|---|---|---|
| 1 | в (in) | 10,859,129 |
| 2 | и (and) | 5,761,105 |
| 3 | на (on) | 3,214,393 |
| 4 | с (with) | 2,439,469 |
Apart from the verb “быть” (to be), it’s all prepositions/conjunctions. ✅ Good enough for linguistic research.
But Tatar and Bashkir Wikipedia? Disaster.
Tatar Wikipedia (~50k articles) top 10:
| # | Word | Translation | Frequency |
|---|---|---|---|
| 1 | елга | river | 132,567 |
| 2 | бассейны | basin | 75,706 |
| 3 | су | water | 54,689 |
| 4 | буенча | by (postposition) | 48,838 |
| 5 | Русия | Russia | 48,722 |
The only “function word” is буенча at #4. “River” is 3x more frequent than the conjunction “and”.
Bashkir Wikipedia is identical: йылга (river), бассейны, һыу (water), Рәсәй (Russia) dominate the top.
Why? Auto‑generated articles
Most articles weren’t written by humans. They were created automatically from a template, filled with data from Russia’s State Water Registry.
Thousands of articles about rivers – each repeating “river”, “basin”, “water”, “located”, “km” – completely distort the frequency distribution.
For a programmer: Imagine scraping this corpus for word embeddings or a language model. Your embeddings would be obsessed with hydrology.
Mordvin Wikipedias: the Orchid syndrome
Erzya and Moksha (Mordvin) Wikipedias have ~1,100–1,500 articles. Their top words include “category”, “see”, “day”, “year” – but also:
| Erzya | Moksha |
|---|---|
| истэжо (also) | ди (and) |
| категория | и (and) |
| ван (see) | тыналста (from family) |
| чи (day) | касыкссь (plant) |
| ды (and) | панчф (flower) |
| ие (year) | орхидея (orchid) |
The explanation: most articles are automatically generated about orchids (family Orchidaceae). Hundreds of plant stubs, each repeating “orchid”, “flower”, “species”, “genus”.
They call this the Orchid Phenomenon – a corpus hijacked by bot‑generated content about a single narrow topic.
Average article length: a quick sanity check
| Wikipedia | Avg length (words) |
|---|---|
| Russian | 184.5 |
| Tatar | 58.5 |
| Bashkir | 56.8 |
| Komi‑Zyrian | 29.6 |
| Erzya | 34.5 |
| Moksha | 34.9 |
Short stubs = high risk of template‑driven, repetitive content.
What this means for you (the developer)
- Don’t trust Wikipedia dumps blindly – especially for low‑resource languages or small editions.
- Frequency analysis is your cheap, powerful friend – a simple count of top word forms can reveal massive bias.
- Auto‑generated content is everywhere – not just rivers and orchids, but also geographical coordinates, biological taxa, movie stubs.
- If you build a language model on such data, expect weird behaviour: over‑representation of “river”, “water”, “orchid”, “species” in any context.
The takeaway
“Wikipedia is free, but not always natural.”
For major languages (English, Russian, German) the corpus is surprisingly balanced. For many minority languages, the Wikipedia you scrape may be more botanical than linguistic.
Always profile your data. A frequency list takes 5 minutes to generate and can save you months of debugging weird model predictions.
Further reading: Orekhov, Reshetnikov (2014). “Towards an evaluation of Wikipedia as a linguistic source: a comparative study.” In Modern Russian Language on the Internet, Moscow.
Liked this? Share with your fellow NLP engineers. They’ll never look at Wikipedia the same way again.
Top comments (0)