Originally published on Medium. Cross-posted here for the dev community.
The problem, from a developer's point of view
Here's the spec a language learner would give you:
"Take any video. Let me loop one sentence at a time, repeat it out loud, and tell me which sounds I got wrong."
Sounds like a weekend project: grab captions, add a loop button, call a speech API. In practice, each of those three steps breaks in its own way for Japanese — no spaces between words, sentences that run for fifteen seconds, captions timed for reading rather than audio, and two writing systems for the same word.
We're the team behind Corodomo, a web and mobile app that turns videos into Japanese, English, Chinese and Korean lessons. This post walks through the approach we took for segmentation, timestamp sync, pronunciation scoring and dictation checking — not our full internals, but the decisions and the lessons, which should be useful to anyone working with speech, subtitles or learner feedback.
TL;DR
| Problem | Naive approach | What worked better |
|---|---|---|
| Split video into practice units | Use caption blocks | Combine linguistic boundaries (text) with pauses (audio), cap length |
| Loop timing | Caption start/end times | Word-level timing, snap to silence, small padding, user-editable subtitles |
| Pronunciation feedback | One overall score | Align attempt to reference, score per syllable, show both |
| Dictation checking | String diff | Normalise kana/kanji and punctuation first, then diff |
1. Segmentation: the unit of practice is a sentence
Learners don't shadow a video; they shadow a line until it feels natural. So the product depends on cutting audio into pieces someone can hold in short-term memory and repeat in one breath.
We generate our own transcript with speech recognition, because many videos have no subtitles and auto-captions are split by time, not meaning. We keep word-level timing, because every later step depends on it.
Then:
- No whitespace → you need a word segmenter anyway (we reuse it for the clickable dictionary).
- Long chained clauses (〜て, 〜けど, 〜から) → allow splits at clause boundaries when a sentence is too long.
- Speech ≠ punctuation → people pause mid-clause and add ね/よ after a breath.
Text-only splitting ignores where the speaker breathes. Pause-only splitting cuts every hesitation. Using both signals produced noticeably more natural loops in our testing.
2. Timestamp sync: loops must start and end on speech
Off by 300 ms and the learner loses the first mora, or hears the start of the next line. What helped:
- Work from word-level timestamps, not caption blocks.
- Nudge boundaries toward the nearest pause.
- Add small padding so the first consonant isn't clipped.
- Ship subtitle editing so learners can fix bad lines themselves.
Point 4 was the best return on effort. Users forgive a wrong boundary they can fix in two taps.
3. Pronunciation scoring: a number isn't feedback
"72/100" tells a learner nothing. For Japanese the natural unit is the syllable/mora, so each attempt shows per-syllable scores plus an overall score.
The high-level pipeline:
record attempt ──► align to reference text ──► assess each unit ──► aggregate
(per syllable) (sounds + pitch) (sentence score)
Lessons:
- Score against the target, not against a transcript of what the learner said — otherwise you're grading the recogniser's guesses.
- Show the reference (with optional furigana) next to the score.
- Design for repetition: an encouraging message and a next target keep people practising.
- Score pitch accent too. Each syllable's score includes whether its pitch (high/low) matches the reference, so はし with the wrong accent is flagged even when every sound is right.
- State the limits. A score is a guide, not a verdict; point learners to what to listen to again.
4. Dictation: don't diff raw strings
すこし vs 少し is not a listening error. しょう vs しよう is. So we normalise both strings (script, punctuation) before diffing, while keeping distinctions like small kana and long vowels that reflect what was actually heard. Answers in kana or kanji are both accepted. Then we highlight the difference side by side.
5. Takeaways for devs building learning tools
- Model the user's unit of work and build every pipeline stage around it.
- Fuse signals (text + audio) for boundaries.
- Make feedback granular and actionable.
- Give users an edit button for anything ML produces.
- Be explicit about what the model can't judge.
Try it / roast it
Corodomo runs on web, iPhone, iPad, Mac and Android. You can start learning without signing up; importing your own videos, pronunciation checks and AI speaking practice need a free account, and the free plan includes one hour of AI usage per month.
We're a two-person core team in Da Nang, Vietnam (plus a few part-time collaborators). If you've built anything with forced alignment, Japanese tokenisation or pronunciation assessment, I'd love to hear how you handled the edge cases — comments are open.
— Viet, founder of Corodomo
Top comments (0)