DEV Community

Cover image for 🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened
Boris Orekhov
Boris Orekhov

Posted on

🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened

#word2vec #russian #nlp #gensim

TL;DR: I built a lemma‑based Word2Vec model (CBOW, 300d) on a huge corpus of Russian fiction (13B tokens → 7.3B after cleaning). You can load it with Gensim and explore semantic neighborhoods of words like слово, язык, речь. The model captures literary semantics without stop words or grammar tags. Check it out on Hugging Face.

Why another word2vec for Russian?

Most pre‑trained Russian word2vec models are trained on web crawls, news, or mixed corpora. That’s fine for general NLP, but fiction has its own semantic rules. Poetic metaphors, archaic vocabulary, and author‑specific styles shift vector spaces.

I wanted a model that:

  • Is trained exclusively on fiction (from classic to modern authors).
  • Uses lemmas (no inflection noise).
  • Is small enough to ship (197 MB for the main model, plus two 5.8 GB arrays).
  • Lets researchers and developers play with literary semantics.

So I built one. And I’m sharing it under the MIT license.

The Corpus by Numbers

Metric Value
Raw words before preprocessing 13.98 B
After lemmatization & stop‑word removal 7.36 B
Sentences (after cleaning short ones) 1.36 B
Paragraphs processed ~539 M

Stop words: removed using this list.

Lemmatizer: Yandex Mystem.

Sentence splitter: razdel.sentenize.

All lemmas are lowercase, dictionary form, no part‑of‑speech tags (saves time & space).

Training Details 

import gensim

data = gensim.models.word2vec.LineSentence('splitted_lemmed_lines.txt')
model = gensim.models.Word2Vec(
    data,
    vector_size=300,
    window=10,
    min_count=2,
    sg=0          # CBOW (faster, good for large corpora)
)
model.save('cbow_300_10.model')
Enter fullscreen mode Exit fullscreen mode
  • CBOW, not skip‑gram – slightly better syntactic capture for fiction.
  • Window size 10 – balances local and long‑range dependencies.
  • Min count 2 – very rare words become noise; keep it clean.

The training script is plain Gensim – no weird dependencies. You can retrain or fine‑tune if you have more data.

What’s Inside the Download?

When you clone from Hugging Face, you get:

  • cbow_300_10.model (197 MB) – the main Gensim model object
  • cbow_300_10.model.syn1neg.npy (5.8 GB) – negative sampling weights
  • cbow_300_10.model.wv.vectors.npy (5.8 GB) – the actual word vectors
  • README.md, tst.py – docs and a test script

Yes, the two .npy files are large. But you can load the model without loading both if you only need similarities (Gensim does lazy loading). Or use model.wv directly.

Quick Start – Get Similar Words 

import gensim

model = gensim.models.Word2Vec.load("cbow_300_10.model")

# Look at closest neighbours of "слово" (word)
for word, score in model.wv.most_similar("слово", topn=10):
    print(f"{word}: {score:.4f}")
Enter fullscreen mode Exit fullscreen mode

Output:

фраза: 0.7941
словечко: 0.6602
слог: 0.6322
реплика: 0.6015
словосочетание: 0.5928
изречение: 0.5818
высказывание: 0.5800
глагол: 0.5735
эпитет: 0.5615
сентенция: 0.5556
Enter fullscreen mode Exit fullscreen mode

Notice how epithet and verb pop up – the model clearly learned linguistic meta‑concepts from fiction.

Now try язык (language / tongue):

наречие (adverb) 0.6684
диалект 0.6095
латынь 0.5892
язычок (little tongue) 0.5698
алфавит 0.5039
грамматика 0.5027
суахили 0.4977
идиома 0.4952
иврит 0.4950
произношение 0.4927
Enter fullscreen mode Exit fullscreen mode

And речь (speech):

монолог 0.6400
спич 0.5914
тирада 0.5900
фраза 0.5652
диалог 0.5477
проповедь 0.5334
слово 0.5217
разглагольствование 0.5193
филиппика 0.5184
декламация 0.5147
Enter fullscreen mode Exit fullscreen mode

Cool Things You Can Do

  • Author style analysis – Compare vector spaces of Tolstoy vs. modern authors.
  • Semantic change over time – Not directly, but you can train separate models for 19th and 20th century subsets.
  • Genre classification – Fiction vs. non‑fiction (though this model is pure fiction, so it will be biased).
  • Creative writing tools – Find thematic associations, generate metaphor candidates.
  • Educational – Show how word geometry reflects literary concepts.

Known Limitations (Read Before You Complain)

  • No POS tags – All lemmas are ambiguous (e.g., стекло could be noun “glass” or past tense of “to flow”). That’s a conscious trade‑off for performance.
  • Only lemmas – No inflected forms. Want слова, слову, словом? You won’t find them; use the lemma слово.
  • Fiction bias – This model will perform poorly on legal documents, tweets, or technical manuals. But that’s the point.
  • Large memory footprint – The full model takes ~12 GB on disk. However, you can load just the vectors with KeyedVectors.load() to save RAM.

Comparison with Other Russian Word2Vec Models

Model Corpus Size POS Availability
This one Fiction, 13B words 300d No MIT, HF
w2v-russian-tolstoy Only Tolstoy 300d No MIT, HF
w2v-russian-19c-fiction-lemmas 19th century prose 300d No HF
RusVectores (web+news) Mixed, ~20B 300d Yes (tags) CC BY‑SA

If you need a general‑purpose model with grammatical info, go for RusVectores. If you work with literary analysis, this one is your friend.

Why You Should Star / Share / Fork

  • 📦 Ready to use – pip install gensim and model = gensim.models.Word2Vec.load(...)
  • 🧪 Reproducible – Full preprocessing script included in the HF repo.
  • 📚 Academically referenced – Several papers already used it (see README).
  • 🚀 Lightweight core – The 197 MB model loads in seconds; the big .npy files are optional.

Get the Model

Hugging Face: nevmenandr/w2v-russian-fiction

License: MIT

Clone with git lfs or download via huggingface_hub:

from huggingface_hub import snapshot_download
snapshot_download(repo_id="nevmenandr/w2v-russian-fiction", local_dir="./w2v_model")
Enter fullscreen mode Exit fullscreen mode

What I’d Love to See in the Comments

  • 🧩 Use cases – Are you doing computational literary studies? Building a Russian story generator? Tell me!
  • 🐛 Issues / improvements – Did you find a bug? A word that should be a lemma but isn’t?
  • 📊 Benchmarks – How does it perform on your task compared to other models?
  • 🧠 Funny analogies – Run model.most_similar(positive=['царь', 'женщина'], negative=['мужчина']) and share the result. (Hint: it’s not “царица” – fiction is weird.)

Final Code Snippet – Try It Yourself 

import gensim

model = gensim.models.Word2Vec.load("cbow_300_10.model")

# Find words similar to "поэт" (poet)
print("Neighbors of поэт:")
for w, s in model.wv.most_similar("поэт", topn=5):
    print(f"  {w}: {s:.4f}")

# Analogy: Москва : Россия = Париж : ?
result = model.wv.most_similar(positive=["Франция", "Москва"], negative=["Россия"], topn=1)
print(f"\nМосква / Россия ≈ Париж / {result[0][0]} (score {result[0][1]:.4f})")
Enter fullscreen mode Exit fullscreen mode

Run it. Play with it. Break it. Then tell me in the comments what you found.

Happy vector hunting! 🧙‍♂️

P.S. The model is called cbow_300_10.model – old‑school name, but it works like a charm.

Top comments (0)