How We Built AI-Powered Japanese Shadowing Practice

Viet Tran — Sat, 24 Oct 2026 14:45:19 +0000

Originally published on Medium. Cross-posted here for the dev community.

The problem, from a developer's point of view

Here's the spec a language learner would give you:

"Take any video. Let me loop one sentence at a time, repeat it out loud, and tell me which sounds I got wrong."

Sounds like a weekend project: grab captions, add a loop button, call a speech API. In practice, each of those three steps breaks in its own way for Japanese — no spaces between words, sentences that run for fifteen seconds, captions timed for reading rather than audio, and two writing systems for the same word.

We're the team behind Corodomo, a web and mobile app that turns videos into Japanese, English, Chinese and Korean lessons. This post walks through the approach we took for segmentation, timestamp sync, pronunciation scoring and dictation checking — not our full internals, but the decisions and the lessons, which should be useful to anyone working with speech, subtitles or learner feedback.

TL;DR

Problem	Naive approach	What worked better
Split video into practice units	Use caption blocks	Combine linguistic boundaries (text) with pauses (audio), cap length
Loop timing	Caption start/end times	Word-level timing, snap to silence, small padding, user-editable subtitles
Pronunciation feedback	One overall score	Align attempt to reference, score per syllable, show both
Dictation checking	String diff	Normalise kana/kanji and punctuation first, then diff

1. Segmentation: the unit of practice is a sentence

Learners don't shadow a video; they shadow a line until it feels natural. So the product depends on cutting audio into pieces someone can hold in short-term memory and repeat in one breath.

We generate our own transcript with speech recognition, because many videos have no subtitles and auto-captions are split by time, not meaning. We keep word-level timing, because every later step depends on it.

Then:

No whitespace → you need a word segmenter anyway (we reuse it for the clickable dictionary).
Long chained clauses (〜て, 〜けど, 〜から) → allow splits at clause boundaries when a sentence is too long.
Speech ≠ punctuation → people pause mid-clause and add ね/よ after a breath.

Text-only splitting ignores where the speaker breathes. Pause-only splitting cuts every hesitation. Using both signals produced noticeably more natural loops in our testing.

2. Timestamp sync: loops must start and end on speech

Off by 300 ms and the learner loses the first mora, or hears the start of the next line. What helped:

Work from word-level timestamps, not caption blocks.
Nudge boundaries toward the nearest pause.
Add small padding so the first consonant isn't clipped.
Ship subtitle editing so learners can fix bad lines themselves.

Point 4 was the best return on effort. Users forgive a wrong boundary they can fix in two taps.

3. Pronunciation scoring: a number isn't feedback

"72/100" tells a learner nothing. For Japanese the natural unit is the syllable/mora, so each attempt shows per-syllable scores plus an overall score.

The high-level pipeline:

record attempt ──► align to reference text ──► assess each unit ──► aggregate
                     (per syllable)            (sounds + pitch)    (sentence score)

Lessons:

Score against the target, not against a transcript of what the learner said — otherwise you're grading the recogniser's guesses.
Show the reference (with optional furigana) next to the score.
Design for repetition: an encouraging message and a next target keep people practising.
Score pitch accent too. Each syllable's score includes whether its pitch (high/low) matches the reference, so はし with the wrong accent is flagged even when every sound is right.
State the limits. A score is a guide, not a verdict; point learners to what to listen to again.

4. Dictation: don't diff raw strings

すこし vs 少し is not a listening error. しょう vs しよう is. So we normalise both strings (script, punctuation) before diffing, while keeping distinctions like small kana and long vowels that reflect what was actually heard. Answers in kana or kanji are both accepted. Then we highlight the difference side by side.

5. Takeaways for devs building learning tools

Model the user's unit of work and build every pipeline stage around it.
Fuse signals (text + audio) for boundaries.
Make feedback granular and actionable.
Give users an edit button for anything ML produces.
Be explicit about what the model can't judge.

Try it / roast it

Corodomo runs on web, iPhone, iPad, Mac and Android. You can start learning without signing up; importing your own videos, pronunciation checks and AI speaking practice need a free account, and the free plan includes one hour of AI usage per month.

🔗 https://corodomo.com/en
🎬 Demo: https://youtu.be/XT-i86GCBtU

We're a two-person core team in Da Nang, Vietnam (plus a few part-time collaborators). If you've built anything with forced alignment, Japanese tokenisation or pronunciation assessment, I'd love to hear how you handled the edge cases — comments are open.

— Viet, founder of Corodomo

DEV Community: Viet Tran