DEV Community

Fortaki
Fortaki

Posted on

How I Built a 150 GB Multilingual & Code Dataset for Central Asian AI (And Fought Out-of-Memory Errors for 10 Hours)

#ai #machinelearning #opensource #llm

Hi Dev.to! While tech giants are competing to train LLMs on trillions of English tokens, there is a severe shortage of high-quality open-source datasets for Central Asian languages (Kyrgyz, Kazakh, Uzbek, Tajik). Technical corpora for these regions are scarce, and code-related datasets are practically non-existent.

As a developer, I decided to fix this myself. I spent weeks collecting, parsing, and cleaning data to build the Multilingual Code and Language Dataset — an open-source project that packs nearly 150 GB of uncompressed data, now available on Hugging Face.

But gathering the data was only half the battle. The real nightmare started when I tried to compress and upload it.

🛠️ The 10-Hour Archive Hell & Out-of-Memory Errors

When you have 150 gigabytes of raw text and source code, you can't just upload it directly. To save bandwidth for the community, compression was a must. I chose .7z with a maximum compression profile.

And that's when my PC started begging for mercy.

Compressing text at this scale requires massive amounts of RAM for dictionary sizes. The process took 10 agonizing hours. The compression tool kept crashing, my OS froze constantly, and the console kept throwing the dreaded Failed message due to Out-of-Memory (OOM) errors. I had to tweak parameters, reboot, catch the perfect timings, and pray my RAM sticks wouldn't melt.

In the end, I won. 150 GB of text was successfully tamed and compressed down to ~27.7 GB, making it easily downloadable.

📊 What’s Inside?

Once you download and uncompress the archive, you will find a highly structured dataset stored in clean .txt and .jsonl files:

  • 🇷🇺 Russian: 60 GB
  • 🇰🇬 Kyrgyz: 23 GB (A massive milestone for Kyrgyz NLP)
  • 🇰🇿 Kasakh: 22 GB
  • 🇺🇿 Uzbek: 7 GB
  • 🇹🇯 Tajik: 7 GB
  • 🇬🇧 English: 1.5 GB
  • 💻 Source Code (Python, C++, Rust, Go): 20 GB (~5 GB per programming language)

💡 Why This Dataset Matters

The core feature of this dataset is bridging the gap between programming source code and low-resource languages:

  1. Code LLM Fine-Tuning: Enhance models like StarCoder or CodeLlama to deeply understand technical specs, documentation, and comments written in Central Asian languages.
  2. Technical Translation: Train translation models that won't hallucinate or fail when converting complex IT terms (like Rust's thread-safety or Go's microservices) into local languages.
  3. Continual Pre-training: The sheer size of the national language corpora is more than enough for deep domain adaptation or training your own lightweight 1B–2.5B LLMs from scratch.

📦 Access the Dataset

The project is fully open-source and distributed under the CC BY 4.0 license. Feel free to use it for research, benchmarking, or commercial applications.

👉 Get it here on Hugging Face: https://huggingface.co/datasets/fortik11/Multilingual_dataset

I would love to hear your feedback! If you have any questions about data filtering, parsing pipelines, or want to collaborate on training models using this data, let's chat in the comments!

Top comments (0)