DEV Community: Servin Osmanov

Fine-tuning was the easy part: what shipping ASR for a low-resource language taught me

Servin Osmanov — Wed, 23 Sep 2026 19:49:45 +0000

The fine-tune took an hour and a half on a single consumer GPU. Everything that made that hour and a half mean anything took months.

This is the second post in a series about building speech tech for Crimean Tatar, a language with no datasets to speak of. The first one covered synthesis and how the project started; this one is the recogniser — much less a story about training than about measurement.

The headline, so you can decide whether to keep reading:

	WER	CER
the model I started from	34.6%	11.9%
+ fine-tuning (≈1.5 h, one GPU)	20.1%	9.4%
+ tuning how it decodes	17.0%	7.0%

Left to right: starting model, after fine-tuning, after decode-time tuning. The third bar cost zero training.

Now the parts worth your time.

I gave the model an exam before I tried to teach it

I already had a "working" Crimean Tatar recogniser: a large Whisper, fine-tuned on our language, which I'd been using on clean studio audio where it looked fine. I had read its transcripts and nodded. I had never scored it.

So the first thing I did here was not improve anything. I measured. 34.6% WER — every third word wrong. That's not "rough": it can't back archive search, can't produce training pairs for synthesis, and an assistant built on it mishears you constantly. I only knew that because I finally scored the thing I'd been eyeballing for a year.

One footnote on that baseline: 34.6% slightly understates it, because the model writes 1954 where the book spells out Biñ doquz yüz elli dört and the scorer counts four substitutions. Small, but if I don't say it, the number is dressed up.

Hold out whole recordings, not random clips

Standard practice is a random 90/10 split. Here that would have been meaningless. Adjacent clips come from the same recording, same reader, same microphone, same afternoon — and the segmenter routinely cuts one sentence across two neighbouring clips, so a random split drops half a sentence in train and the other half in eval. What you measure then isn't language ability. It's whether the model memorised a voice and a room.

So I held out two entire books, two readers the model had never heard: 893 clips, 1 h 52 min. One move removes the voice, the mic, the recording era and the vocabulary of a particular text at once. If your data has structure — speaker, session, document, device — split on it. Random splitting is only safe when clips are genuinely independent, and in low-resource audio they essentially never are.

That feels like enough caution. It wasn't.

Audit your eval set by content, not by filename

Before launching anything I checked the obvious thing: is any of my held-out eval material also sitting in training?

By filename: zero overlap. Not one collision. The two name sets looked like they came from different universes.

Then I compared the text instead: for every eval clip, does any training clip contain the same run of six consecutive words? Word 6-grams — cheap to build, cheap to hash, long enough that natural coincidence is rare at this corpus size.

Four audiobooks were in my material twice. Same recordings, segmented by two different tools at two different times, under completely unrelated names.

book	share of its clips with a twin in training
book I'd picked for eval	96.9% (651 / 672)
second eval book	58.8% (130 / 221)
third book	82.1% (769 / 937)
fourth book	69.8% (549 / 787)

Four books stored twice. Zero overlap by filename; the collision only showed up in the text.

Top row is the book I was about to grade the model on.

I wasn't panicked. I just understood two things at once: every measurement I'd ever taken on this data was void, and thank God it surfaced before training rather than after.

The counterfactual is the point. If I hadn't checked, nothing would have gone wrong. No crash, no warning, no weird loss curve. The model trains on recordings it will later be examined on — different cuts, same sentences, same voices — posts a great number, and I publish it. The number means nothing: the exam questions were handed out in advance. That's the nasty property of this bug class. It has no symptoms. Everything works, the result is better than expected — which is exactly what should make you suspicious.

And it generalises: in a low-resource language, re-cut copies of the same recordings are the norm, not the exception. There is little material, so you keep returning to it with new tooling, and every pass produces fresh filenames wrapping old content. Deduplicate by content. Filenames are not identity.

After cleaning I re-verified rather than trusting myself: zero eval clips in training, zero 6-gram matches. Measured and written down, not "I'm pretty sure I got them all".

Don't judge a fine-tune by its first checkpoint

Now the part everyone thinks is the project, which was the fastest and dullest bit of it.

I didn't retrain the model. I froze it and trained a small adapter alongside — 2% of parameters, 31M trainable against ~1.5B frozen. 15.5 hours of speech survived the cleanup, 3 epochs, 705 steps, 1 h 30 min on one home GPU, 10.4 GB peak VRAM, and a finished adapter of 126 MB. Result: 20.1% WER, 9.4% CER — a 42% relative cut in word errors.

I felt nothing at the time; a number moved as designed. The pride arrived later, when I ran the model over material it had no business handling — films, podcasts, noisy field recordings — and the transcripts came back near-perfect. That's when it stopped being a table row and became working ears.

Two things from the run are more useful than the headline number.

Checkpoint 100 was worse than the base model. I saved every 100 steps and scored all of them. The first checkpoint came out noticeably worse than what I started from: the adapter has half-learned the new behaviour, it's already interfering with the old one, and it isn't giving anything back yet. If I'd glanced at it and concluded the approach was a dud, I'd have killed a run that ended up cutting error by 42%. A dip at the first checkpoint is what partial adaptation looks like, not what failure looks like.

All the gains landed in epoch 1 of 3. Past roughly step 200 the numbers stopped moving; epochs two and three contributed nothing measurable. So "1.5 hours" is itself padded — half an hour would have done. Checkpoint often and score often; runs converge long before the schedule says so.

One detail that reassured me more than the average: both eval books improved, and the smaller, later, different-reader one improved more. That's the signal you want — the model didn't just learn one narrator's delivery.

Decode-time wins are real — decompose them anyway

Then error dropped again with no training at all.

A trained model has knowledge, but it also has a decoding policy. The default is greedy — take the locally most probable token, never reconsider. Fast, occasionally catastrophic: take one wrong turn and it can't back out, so it goes in circles. That's exactly what the worst eval clips were. Not mishearing — looping:

Qıznıñ qıznıñ qıznıñ qıznıñ …
– dedi Akimoviç, – dedi Akimoviç, – dedi Akimoviç, …

The fix is beam search: hold several candidate sequences and score whole hypotheses rather than committing token by token. Costs decode time, costs zero training. I swept 24 configurations on a separate selection set, not on the exam (more below), then ran the winner on the test set once. 17.0% WER, 7.0% CER. Not one weight changed. Decoding got 3.1× slower.

Here it would have been easy to write "beam search: −3 WER points" and move on. True, and half the truth. So I broke the win down.

409 word errors were recovered in total. 160 of them came from 3 clips out of 893. 0.34% of the material delivered 39% of the gain. Those are the loopers: on 41 reference words the greedy model emitted 175 words of repeated garbage. The remaining 890 clips improved by a modest, even 10%.

Both halves are real, and neither alone justifies a 3.1× slowdown. But they mean different things. The broad 10% is "the model got more careful". The three clips are "it stopped occasionally losing its mind". For archive transcription the second matters far more, because a clip containing 500 bytes of one repeated token isn't merely bad — it poisons anything you train on it later. After the change: zero looping clips, 0 of 893.

Decompose your aggregate wins. If most of your improvement lives in a handful of examples, you didn't improve accuracy; you fixed a failure mode. Those call for different follow-up work, and averaging hides the difference.

The obvious anti-loop knob broke real language

There's a more direct tool against repetition: forbid the model from emitting the same n-gram twice. On paper, exactly the right hammer.

I tested it. It was net harmful: across 255 selection clips it broke 17 and fixed 1, and 12 of the 17 it broke had been essentially perfect. The transcripts explain themselves. The constraint doesn't stop the model repeating — it forces it to repeat incorrectly:

book        – Qartbaba, qartbaba, – tez-tez çapıp kelgen kiçkene Aziz
unrestricted– Qartbaba! Qartbaba! – tez-tez çapıp kelgen kiçkene Aziz
restricted  – Qartbaba! Qartbabaa! – tez-tez çapıp kelgen kiçkene Aziz
harder      – Qartbaba! Qartbava! – tez-tez çapıp kelgen kiçkene Aziz

A child runs up calling "Qartbaba, qartbaba!" — grandpa, grandpa. The model heard it perfectly. The ban wouldn't let it write down what it heard, so it corrupted the second word to avoid matching the first. Elsewhere bağışla… bağışla… ("forgive me… forgive me…") came back the second time as the non-word Bağuşla; in a third case the model routed around the constraint by inventing a suffix, parçalarnıñ parça instead of parçalar. Doubled address and doubled pleading are ordinary Crimean Tatar. People talk that way.

This is my favourite finding of the stage, and it isn't really about decoding. You cannot fix a language with prohibitions. Any rule of the form "this never occurs" eventually collides with how people actually speak, and it breaks precisely that spot while leaving everything else intact — so you won't notice. A decoding default borrowed from an English-language tutorial encodes assumptions about English. Test it against your own language's rhetoric first.

Three sets, and the test set gets decoded once per decision

I've mentioned a "selection set" twice. Every number above rests on it. There are three sets, not two.

Train — the model learns here.
Dev / selection — 255 clips, half an hour, two short whole works pulled out of the training material. This is where I choose: which checkpoint, which of the 24 decode configs. Dozens of evaluations, as many as I liked.
Test — the two held-out books. Decoded exactly twice all stage: once on the winning checkpoint, once on the winning decode config. The selection rule was written down before I looked at any result.

Why so strict? Because if you run the test set repeatedly and take the best result each time, you're no longer selecting the best model — you're selecting the luckiest one. The number comes out pretty and never reproduces.

The dev set got the same cleanliness audit: zero shared clips, zero 6-gram matches. After the duplicate story I don't file that under paranoia.

Your dev set measures ranking, not absolute quality

The dev set was much easier than the test set. Same model: 17% WER on dev, 34.6% on test. Both honestly held out, neither seen in training. But the dev material comes from the same family of recordings as training, while the test books are two entirely foreign readers.

So dev is good for "config A beats config B" and useless for "the model errs at X%". Report the dev number as your result and you're not lying exactly — you're reporting the easier exam. Keep both in their lanes: dev ranks, test reports.

The tool I couldn't validate gets no number

One consequence of that discipline. Whisper ships a temperature-fallback mechanism against repetition: it detects circular output and retries. Precisely the tool this problem wants, and nearly free at decode time.

But my dev set contained zero looping clips. Nothing for it to act on — it looked like a config that does nothing at all. By my own selection rule I couldn't pick it, and I refused to sneak it onto the test set afterwards to see whether it would have helped. So I have no number for it, and I'm not inventing one. If you couldn't validate a tool on the set you're allowed to validate on, write down that it's unvalidated and keep it out of the results table.

17% in the studio. What about the world?

Everything above was measured on one kind of audio: one person, one mic, quiet room, reading a book. Honest for its own world, and it says little about the world a voice assistant lives in. So I took the same 893 clips and corrupted them 17 ways — noise, music, competing speech, reverb, telephone-band filtering, time-stretching. The reference text is known, so error is measurable in each.

Telephone band and aggressive compression are free. Through a telephone-band filter the model does a hair better than in the studio. Speech rate barely matters either: ±15% tempo costs at most 6%.
Competing speech is the enemy, by a wide margin. Babble costs roughly 2× what steady noise at the same loudness costs, ~4× what music costs. At equal loudness with the target, WER goes from 17% to 69% — the model stops working.
Reverb is second. A hallway-sized room multiplies error by 2.5×.

That points somewhere specific: the highest-leverage investment in robustness is not more data. It's source separation and microphone placement. More training hours will not fix a model whose input holds two people at equal volume.

One last check that made me happy: on an open Crimean Tatar audiobook set the model had never trained on, clean chapters landed at essentially the same number as my own exam. It didn't memorise my readers. It hears the language.

What I'd actually tell you

Training was 90 minutes. The months went into auditing the eval set by content, splitting by recording, checkpointing and scoring instead of trusting the schedule, decomposing wins instead of quoting averages, refusing to let an English-shaped default constrain our grammar, and keeping a test set clean enough that its number means something.

Fine-tuning is the part with a progress bar. It is not the part where the work is.

Second post in a series. Coming up: a synthesised voice reading an entire book — sixteen chapters, nearly two hours of audio — and what had to be verified chapter by chapter. Then the one where I rented somebody else's GPUs and set money on fire, computing almost nothing.

Audio samples and project page: https://ai.ana-yurt.dev/.

Two ways to help. If you speak Crimean Tatar, the highest-value contribution is fifteen minutes with headphones — listen to a clip, say whether the machine wrote it down correctly, fix a transcript. No programming needed, and ears like that caught things here that no automated check did; write to support@ana-yurt.com. If you'd rather fuel the work directly, there's ko-fi.com/anayurt.

What I learned building speech tech for a language with zero datasets

Servin Osmanov — Thu, 10 Sep 2026 00:44:51 +0000

The word was qara, "black". I fed it to an off-the-shelf TTS model, hit play, and got back a soft, polite, very Turkish kara.

In Crimean Tatar къ and к are two separate sounds, and къ sits inside words we say every day: yoq, qız, vaqıt, qadar. Getting it wrong is not an accent. It is the model quietly speaking a neighbouring language and hoping nobody notices.

I noticed. That one syllable is why I spent the next several months building speech models for Crimean Tatar on a single consumer GPU, in the evenings, with no dataset to start from. What follows is the engineering residue: what I would tell a solo developer about to do the same for their own low-resource language. Most of it cost me a week per lesson.

The chicken and the egg were both endangered

The standard way to build a speech dataset is to take audio and run ASR over it to get text. Except we had no usable ASR for Crimean Tatar, and you cannot build good ASR without a corpus. Classic chicken and egg, except the chicken is also endangered.

I stared at that loop longer than I want to admit before noticing I was holding the problem the wrong way round. Audiobooks come with the text. It is printed. It exists. There is nothing to recognise.

So I stopped trying to transcribe and started trying to align: known text, audio, find where each word lands, cut on those boundaries. Forced alignment instead of recognition. That flip is the reason the project exists at all, because it sidesteps the exact thing the language did not have.

If you take one thing from this post, take that one. Before training a recogniser, check whether the ground truth is already sitting on a shelf. If it is, you have an alignment problem, and alignment is enormously cheaper.

The run that lied to me for three and a half hours

The first big alignment run covered seven and a half hours of audio. It ran, it did not error, and it produced clean, confident, well-formatted output.

The first two hours were correct. The remaining three and a half hours were garbage. Not obviously-broken garbage either: plausible timestamps, plausible segments, looking exactly like the good ones. Out of seven hours I got a bit under two of usable material, and I only found out because I went listening.

I have come to think of a crash as a gift. A crash points at the line. A process that returns confident nonsense takes a week off your life, slowly.

What fixed it was not a code change but a granularity change. Instead of feeding the aligner three enormous blocks, I split the material into forty chapters and located each independently. Same audio, same text, same tooling, and over five hours of clean speech instead of under two.

That is the rule I kept: if a stage can fail silently, never run it on big units. Small units fail visibly, and they let you bin the bad ones instead of the whole batch.

Words do not line up. Letters do.

That chapter-matching step had its own trap waiting. Locating a chapter inside a book sounds like a string search problem, and my first instinct was word-level similarity. It found nothing. I tested it on a pair I knew matched, checked by hand, and the similarity came back under one percent.

The reason will bite anyone working outside English. Crimean Tatar is agglutinative, so the "same" word carries different suffixes in the two sources. Add two alphabets in circulation (Cyrillic and Latin), and word tokens simply do not line up.

Switching the comparison to short character sequences fixed it in one go: 38 of 40 chapters located. The characters survive the suffixes. The words do not. For anything morphologically rich or multi-script, character-level matching is the default and word tokens are an English-shaped assumption.

The base model had already decided

Back to qara. My first assumption was the obvious one: the model had never heard the sound. Easy to test, so I checked. In my data the letter q showed up in 88% of recordings, more often than k. The model had been shown the correct pronunciation thousands of times and had cheerfully ignored every one.

The problem sat underneath the fine-tuning. In the base model, before it ever saw a word of my language, "q" was already wired to /k/ by English, Spanish and French. Fine-tuning was not overwriting that prior, it was decorating it. Which leads somewhere unpleasant: more data would not have helped. Not ten more hours, not a hundred. The fix had to happen at the foundation.

So I went base-model shopping with a different question. Not "which base speaks a language close to mine" but "which base already produces the sound I keep losing".

Everybody's first suggestion is Turkish, the big well-resourced relative. But Turkish has no /q/ at all, only one k, which is exactly how you get kara. I verified that twice, because I did not trust my own ears the first time. Azerbaijani looks promising until you listen: its q leans towards /g/, then softens further into something breathy. The languages that keep the sound, with a dedicated letter of its own, are Kazakh, Uyghur and Bashkir.

The multilingual base I ended up choosing had roughly 1500 hours of Kazakh, 400 of Uyghur and 250 of Bashkir in its training mix, against 125 hours of Turkish. Exactly upside down from what "pick the closest big language" would tell you.

It still had no Crimean Tatar in it. I got around that at the writing-system level: our къ, гъ and нъ each map onto a single letter the model already knows (ҡ, ғ, ң), so you hand it Crimean Tatar text and it reads something familiar. One trick, two problems.

The nice part is that this is verifiable before you spend a single GPU-hour on training. Out of the box, with no fine-tuning at all, the new base pronounced our къ correctly 88–100% of the time. I was no longer teaching the model a new sound. I had stopped fighting it.

I took the Kazakh and Bashkir variants all the way to the end, because I wanted the comparison rather than the argument, and they tied to the third decimal place. I shipped Bashkir: at a certain point in training the Kazakh one returned a few empty phrases and let the /q/ slide back towards k. Given equal numbers, I take the one that breaks less often.

For a language with no data of its own, picking the base is a phonetics decision, not a taxonomy one. And you can smoke-test the prior in an afternoon.

Three times the data, and the number would not move

My favourite negative result of the project. The corpus kept growing: 5.9 hours, then 11.5, then 15.3. I tracked character error rate by synthesising speech and running it back through a recogniser, averaged over three runs each:

corpus	CER (mean of 3)
base, no Crimean Tatar at all	0.173–0.198
5.9 h	0.136
11.5 h	0.138
15.3 h	0.141

Two and a half times the data and the number goes slightly worse. Meanwhile the run-to-run spread on one unchanged voice was wide enough to swallow that whole difference, so the honest reading is that all three runs are indistinguishable by this metric. And yet in blind listening, not knowing which sample was which, I picked the newer voice every time.

The corpus grew two and a half times over. The automatic metric flatlined. Blind listening kept preferring the newer voice anyway.

The explanation matters more than the numbers. Intelligibility saturates early, around six hours. Prosody, breathing, the liveliness of a sentence keep improving well past the point where CER stops moving. The metric goes quiet exactly where the remaining quality lives, so anyone reading that table alone would have stopped collecting data, reporting the truth while making the wrong call.

The cheaper form of the same lesson: one measurement is not a measurement. The same voice gave one number on the first run and a noticeably different one averaged over three. I have deleted several "wins" that turned out to be variance wearing a hat. Measure three times, and never give a single automatic metric the only vote.

The bug you can only hear if you click one sentence

I was listening to a narrated chapter, clicked on sentence 195 to check something, and heard the wrong sentence. Not slightly wrong, a different sentence. The drift was beautifully linear:

sentence	timestamp offset
1	0 s
49	+10.7 s
98	+21.8 s
147	+33.0 s
195	+43.9 s

My first theory was the fashionable one: the model falls apart over long text. Wrong, and worth checking rather than assuming. Speaker similarity to the opening held at 0.96–0.98 throughout, and the tempo never shifted. The audio was fine.

The actual cause: two parts of my pipeline computed time independently, and one counted the pauses between phrases twice. Forty-three seconds is exactly the sum of those double-counted pauses across the chapter. Correct audio, wrong labels.

I did not debug this by reading code. I debugged it with a knife: cut the audio at the old timestamps and at the new ones, push both through recognition, compare. New marks came back at CER 0.008, so the text matches. Old marks came back at 0.712, so those are other sentences entirely. On the final line the old timestamp had run past the end of the file, and the recogniser dutifully "heard" its own boilerplate captions in the silence.

The part that should worry you: played straight through, the file sounds perfect. Every consumer of those timestamps, a word-highlighting reader or a training pipeline, would have inherited garbage from a file that passes any listening test you can run. Whole-file playback validates whole-file playback and nothing else.

The exam the model had already seen

The ASR side of the project produced numbers I liked:

	WER	CER
the model I started from	34.6%	11.9%
+ fine-tuning (about 1.5 h on one GPU)	20.1%	9.4%
+ tuning how the model decodes its answer	17.0%	7.0%

Left to right: starting model, after fine-tuning, after decoding changes.

The third row involved no training at all, not a single weight touched, only a more careful decoding setup, and it moved WER by three points. The entire adapter I added to that model is 126 MB.

Now the part that nearly ruined it. The same four audiobooks existed twice in my material. Cut differently, segmented differently, stored under unrelated filenames, zero overlap by name. For one book I had picked as an eval set, 96.9% of the segments had a twin sitting in the training data.

If I had missed it, I would have been examining the model on questions it had already been handed the answers to. A lovely number, published, believed, with the next six months built on top.

Filenames are not identity. I check for near-duplicates by content now, always, and I treat a suspiciously good eval score as a bug report. Same reflex later, when a synthesised voice read a whole book: I proved by measurement, not by reasoning, that the book's text never entered training. Zero word-sequence matches against a corpus of nearly one and a half million words. Otherwise it is not reading, it is reciting.

What it costs now

A synthesised voice read an entire book, Merdiven by Şamil Alâdin. Not a demo paragraph. All sixteen chapters, 15,444 words, one hour fifty-eight minutes of audio, word timings verified across 100% of the words, per-chapter CER median 0.022 (worst 0.027, best 0.015).

The timings got a control that was required to fail: cut 30 words per chapter at the recorded marks and recognise them, 447 of 480 matched; shift the same fragments by 0.4 seconds and 4 of 480 match. That gap is what makes the check worth running.

And the number I care about most: roughly three hours of machine time per two hours of finished audio, everything included. An audiobook in Crimean Tatar now costs one night of one home computer. Not a studio, not a year of waiting.

Earlier work is public if you want a starting point: the dataset and the model.

Where the months actually went

Training was hours. An hour and a half here, a few hours there. It is not the line item anybody thinks it is.

The months went into collecting and aligning data, repairing the parts that broke without saying anything, proving the evaluation was honest, re-measuring, and listening with headphones, blind, one sentence at a time.

That last one produced the finding I did not expect. Across the project the bugs split roughly half and half between instruments and a human ear. Instruments caught the drifting timestamps, which no amount of listening would ever have revealed. The ear caught breaths in wrong places, bad annotations, and this one:

I heard ниже where the book said нидже. The cause took a minute to find. My letter-mapping rule, the same trick that gave me ҡ and ғ, was also collapsing дж into ж. The model had been handed the string "ниже" and pronounced it flawlessly. Perfect execution of a wrong input.

The rule had erased a distinction the language genuinely has: дж in our own words like джан, оджа, нидже, versus ж in loanwords. After that collapse nothing downstream could have said them differently. That chapter held fifty-five such words. I caught exactly one, the case that happened to land on another real word. The other fifty-four sounded "a bit off", and the ear slides right over "a bit off".

Fixing it barely moved any metric. I fixed it anyway, because it is our language, not a leaderboard.

Which is the most useful thing I can hand anyone doing this for their own language: your cheapest and strongest tool is one native speaker, fifteen minutes and headphones. It beats the automation, and it means you do not have to be a programmer to contribute. You need to hear your language well.

First post in a series. Coming up: teaching a recogniser to hear Crimean Tatar; narrating a book chapter by chapter; running blind A/B tests so you stop fooling yourself; and the one where I set money on fire renting somebody else's GPUs and computed almost nothing.

Voice samples and project page: https://ai.ana-yurt.dev/blog/2026-09-kak-rodilas-ideya/.

Two ways to help. If you're a Crimean Tatar speaker, the highest-value contribution is fifteen minutes with headphones — blind listening and proofreading beat any tool in my stack; write to support@ana-yurt.com. If you'd rather fuel the work directly, there's ko-fi.com/anayurt.

My Journey Improving a TTS Model for the Crimean Tatar Language

Servin Osmanov — Fri, 07 Nov 2025 19:42:08 +0000

When you work with machine learning, success often hides behind hours of frustration, countless errors, and broken pipelines. This project — improving the Crimean Tatar TTS (Text-to-Speech) model — was exactly that kind of journey. What started as a small experiment to fine-tune an existing model turned into a full-scale debugging adventure that taught me more about data integrity, audio processing, and patience than any tutorial could.

In my previous article, Why Language Tech Matters: Developing AI Tools for Small Languages, I explored how AI can empower low-resource languages. This piece continues that journey with a hands-on look at improving TTS models.

The Starting Point: A Model That Worked — but Only Partially

My goal was simple: improve the voice model “Sevil” for the Crimean Tatar language. I had already worked with similar voices — “Arslan” and “Abibullah” — using Hugging Face datasets like speech-uk/tts-crh-arslan and speech-uk/tts-crh-abibullah.

The first training attempts went well — loss around 0.283, results acceptable. But something didn’t add up. The dataset had 1,566 audio files, yet the training logs showed only 415 were being used — about 26.5% of the total.

That meant almost three-quarters of my data was silently ignored.

At first, I thought it was a fluke. Then I realized it was a systemic problem in the Hugging Face datasets API when loading compressed audio from Parquet files.

Diagnosing the Problem

When I loaded the dataset through datasets.load_dataset(), most files threw errors:

Error: "Error while decoding audio"
Error: "Audio file appears to be empty"

That didn’t make sense — the audio bytes were clearly present in the Parquet files.

After checking manually with pandas.read_parquet(), I confirmed the data was there. The problem wasn’t in the files — it was in how the decoder handled them.

That’s when I realized: the datasets API couldn’t decode raw audio bytes correctly. The data was fine, but the pipeline was broken.

Turning Bytes into Sound

At this point, I tried everything.

I extracted the bytes manually, saved them as .raw files, and tried to load them with librosa and soundfile. Nothing worked.

Without proper metadata (sample rate, channels, encoding), the files were unreadable.

Eventually, I discovered the solution: FFmpeg.

By using known dataset parameters —

sample rate: 16000 Hz, channels: mono, format: PCM 16-bit little-endian —

I could convert all .raw files into clean .wav audio.

ffmpeg -f s16le -ar 16000 -ac 1 \
       -i sevil_0000.raw \
       sevil_0000.wav

And just like that — 1,566 files successfully converted.

No corruption. No decoding errors. 100% validation success.

It was the moment of breakthrough — the kind that makes you sit back, smile, and realize you’ve just solved a problem that haunted you for two days straight.

Training the Model — Again

With clean audio finally ready, I retrained the Sevil model from scratch, this time using all 1,566 recordings.

Training setup (based on my previous configs for “Arslan”):

num_train_epochs = 500
batch_size = 4
learning_rate = 1e-4
fp16 = True
warmup_steps = 2000
save_steps = 2000

The progress was promising:

Loss dropped from 1.14 → 0.80 → 0.50 → 0.27
The voice quality improved with every epoch
And then… it crashed at 78% completion.

The culprit? A familiar one for SpeechT5 users —

RuntimeError: torch.cat(): expected a non-empty list of Tensors

It turned out to be a bug in guided_attention_loss, a component that sometimes fails with uneven sequence lengths.

Fixing the Crash

Instead of starting over, I resumed training from the last checkpoint (step 16,000) and simply disabled guided attention:

model.config.use_guided_attention_loss = False

That one line saved the project.

Training resumed, completed 98% of the full cycle, and stabilized with a final loss of 0.267 — a small numerical improvement, but a big qualitative one.

The model became more consistent and robust across new data.

Comparing Results

Model	Data Used	Loss	Success	Notes
Sevil v1	415 files (26.5%)	0.276	47% test	Trained on partial data
Sevil v2	1,566 files (100%)	0.267	100% test	Fully trained, stable

The difference wasn’t just in metrics — it was in confidence.

Sevil v2 generalized better, produced smoother intonation, and maintained pronunciation consistency even on unseen words.

Lessons Learned

Never trust good metrics without checking data coverage.

My “good” baseline was trained on just 26% of the data.
FFmpeg is a lifesaver.

It solved what specialized libraries couldn’t.
Validate every single file.

Automation saved hours of manual checking.
Guided attention loss is optional.

Sometimes stability matters more than theoretical accuracy.
Document everything.

By keeping track of every attempt — successful or not — I could understand the full story, not just the happy ending.

Why This Project Matters

For me, this wasn’t just about fixing one dataset. It was about enabling a low-resource language — Crimean Tatar — to have a better voice in the digital world.

Improving the Sevil model means clearer pronunciation, smoother prosody, and better accessibility for learners and native speakers alike.

And for anyone working with custom TTS datasets:

Check your files, validate your data, and don’t give up when your model crashes at 78%.

That crash might be the best teacher you’ll ever have.

Author: Servin Osmanov

Lead Fullstack Python / ReactJS Engineer

AI researcher and TTS developer for low-resource languages

Project: servinosmanov/tts-crh-sevil-fixed on Hugging Face

My Journey as a Judge at CBIT Hacktoberfest 2025 — Lessons from the Other Side of the Table

Servin Osmanov — Thu, 30 Oct 2025 18:29:37 +0000

When I first started attending hackathons, I was always the one coding, building, and pitching. This year, for the first time, I got to experience the other side — as a judge at the CBIT Hacktoberfest Hackathon 2025.

It was an incredible 24-hour online event organized by the CBIT Open Source Community, celebrating open-source culture and collaboration as part of the global Hacktoberfest. The event gathered hundreds of students from universities across the world — developers, designers, and dreamers ready to learn, code, and share.

What the Event Was About

Hacktoberfest is all about celebrating open source, and this hackathon perfectly captured that spirit. The CBIT edition — now in its 8th year — encouraged participants to collaborate on innovative solutions using modern technologies while contributing to the open-source ecosystem.

Teams of 3–5 members worked virtually through Discord, tackling real-world challenges in just 24 hours. Despite the distance and time zones, the sense of connection and creativity was palpable.

The Judging Framework

Every project was evaluated based on a clear, balanced set of criteria (total: 50 points):

Criterion	Points	What It Measured
Innovation & Creativity	10	Original ideas and new approaches
Collaboration	5	Teamwork, communication, and shared contribution
Implementation	20	Technical soundness, scalability, and efficiency
Design	5	Usability and user experience
Presentation	10	Clarity, storytelling, and delivery

As a judge, I was looking for that spark — projects that combined solid implementation with a clear purpose.

What I Saw and Learned

Some projects were technically ambitious, pushing the boundaries of what’s possible in 24 hours. Others were beautifully simple, focusing on accessibility or social impact. A few used AI and automation in creative ways that genuinely surprised me.

But beyond the technology, what stood out was the teamwork. Many participants were strangers before the event — yet they managed to code, design, and present together like long-time collaborators. That’s the magic of hackathons.

The Judging Experience

Our panel included engineers and leaders from Electronic Arts, JP Morgan Chase, Oracle, Deliveroo, ServiceNow, EY, and more. Being part of such a diverse group gave every discussion depth. Each judge viewed “innovation” slightly differently — and that variety made our evaluations richer.

I appreciated the effort teams put into their presentations. Even short demos told full stories — from the problem statement to the final prototype. That’s where creativity met clarity.

Key Takeaways

Innovation isn’t always about technology. Sometimes it’s about empathy — understanding who you’re helping and why.
Good presentation matters. The best projects told their stories with confidence and focus.
Community is everything. Open-source hackathons like this show how technology connects us beyond borders.

As a developer, I’ve always believed in continuous learning. Judging this event reaffirmed that growth happens in many forms — whether you’re writing code or evaluating it.

Why Hackathons Matter More Than Ever

In a world where remote collaboration is the new normal, hackathons remain one of the most human ways to innovate. They combine competition, creativity, and community into one shared experience. For students and professionals alike, they’re the perfect place to experiment and grow.

I’m grateful to have played a small role in this journey — to witness ideas take shape, to learn from participants, and to see firsthand how open source continues to inspire a new generation of builders.

Author:
Servin Osmanov

Lead Software Engineer @ Anvaya Solutions Inc.

Judge at CBIT Hacktoberfest Hackathon 2025

Why Language Tech Matters: Developing AI Tools for Small Languages

Servin Osmanov — Sat, 25 Oct 2025 15:45:27 +0000

In a world where artificial intelligence is transforming how we communicate, the survival of small languages depends not just on cultural passion — but on technology.

While English, Chinese, or Spanish dominate the digital space, thousands of smaller languages remain digitally invisible. Without online presence, data, or digital tools, these languages risk extinction in the 21st century.

As a software engineer and cultural advocate, I’ve spent years developing digital tools for the Crimean Tatar language — a Turkic minority language spoken by less than 300,000 people worldwide. Through this work, I’ve learned that the intersection of AI and linguistics isn’t just a research topic; it’s a lifeline for cultural identity.

The Challenge: The Technology Gap for Small Languages

Mainstream AI models—from GPT-based chatbots to translation engines—are trained primarily on high-resource languages. This creates a serious imbalance: while global communication becomes easier for some, others are left behind.

For small linguistic communities, the lack of:

digital corpora,
standardized spelling systems,
and high-quality training datasets

makes it nearly impossible to integrate their languages into modern tools like voice assistants, translation services, or educational apps.

When a language isn’t “machine-readable,” it risks becoming irrelevant in the digital world — even if it’s still spoken at home.

Our Journey: Building Tools for the Crimean Tatar Language

In 2018, our team launched a series of educational and linguistic tools under the Qırımtatar Lugatı (Crimean Tatar Dictionary) project:

Our mission was to give people an accessible and modern way to learn, use, and preserve the Crimean Tatar language.

The apps quickly grew into more than just dictionaries — they became living platforms connecting speakers, learners, and educators. Yet, users wanted more: context-based search, on-the-fly translation, and natural voice pronunciation.

This inspired us to integrate artificial intelligence to make our tools more flexible and intelligent.

The Next Step: Bringing AI to Minority Languages

We are now integrating AI and machine learning into the Crimean Tatar dictionary project to expand its capabilities:

🧠 AI-assisted translation: dynamic, real-time translation between Crimean Tatar, Turkish, English, and Ukrainian.
🔊 Speech recognition and synthesis: allowing users to hear natural pronunciation and practice correct intonation.
📖 Adaptive learning: using AI to personalize vocabulary lessons based on user behavior and progress.
🪶 Data-driven NLP foundation: building scalable open datasets that will support future chatbots, voice assistants, and translation systems.

To achieve this, we’ve adopted and fine-tuned open-source speech synthesis models such as:

facebook/mms-tts-crh — part of Meta’s Massively Multilingual Speech project, providing a solid baseline for Crimean Tatar TTS.
robinhad/qirimtatar-tts — a community-driven model that helps us generate high-quality speech for educational and cultural content.

By leveraging these models, we’re building an AI-powered TTS system that brings Crimean Tatar audio resources to learners, teachers, and content creators for the first time.

Why This Matters

Language is more than communication — it’s collective memory.

When a language disappears, so does a unique worldview and cultural identity.

AI gives us a chance to reverse that process. With the right tools, small languages can become visible online, connect their communities, and survive in the digital era.

Projects like Qırımtatar Lugatı show that you don’t need a giant corporation to make an impact.

A small, passionate team with the right technical vision can give a digital voice — literally — to those who’ve been silent for too long.

How Developers and Researchers Can Help

If you’re a developer, linguist, or AI researcher, here are a few ways to make a difference:

Contribute to open-source datasets for small or low-resource languages.
Support multilingual NLP and TTS initiatives on Hugging Face or GitHub.
Collaborate with communities and educators who are digitizing endangered languages.
Help localize educational and cultural apps into underrepresented languages.

Every dataset, every model, and every contribution brings us closer to a more linguistically inclusive AI ecosystem.

Looking Ahead

Our roadmap for 2025 includes:

Full integration of AI-powered translation within the Qırımtatar Lugatı apps.
Implementation of voice features using our fine-tuned TTS models.
Launching an open API for developers who want to build tools for Crimean Tatar or similar minority languages.

Technology alone won’t preserve culture — but it can amplify it.

For small languages like Crimean Tatar, AI isn’t just a tool — it’s hope.

About the Author

Servin Osmanov is a Senior Full-Stack Engineer and founder of the Qırım.Online and Ana-Yurt.Com project — an initiative focused on preserving Crimean Tatar culture and language through modern technology.

LinkedIn: linkedin.com/in/servin-osmanov

GitHub: github.com/MrSerWin