Building sentence-level alignment for bilingual audio

肖邦强 — Sat, 24 Oct 2026 03:03:39 +0000

Most language-learning apps assume a text-first workflow: paste a paragraph, translate it, then generate audio. A harder and more useful case starts with two real recordings: the same story, lesson, or podcast delivered once in English and once in Mandarin or Japanese. The sentence boundaries do not match. The speaking speeds do not match. One track may have intro music, extra narration, or a different cut.

The goal for Bilingual Mix is to turn those two files into one alternating MP3: meaning-language sentence first, then the same idea in the target language. This post is about the alignment and rendering pipeline behind that output, and the tradeoffs that made it reliable enough to be usable.

1. Transcribe with timestamps, not just text

The server sends each file to Groq's Whisper-large-v3-turbo endpoint and requests both word and segment timestamps. Segment timestamps are useful as a first pass, but word timings matter because they let the later stages trim silence and verify that a repaired boundary still contains the words it claims to contain.

A transcript with text but no timing is not enough. The product needs four numbers for every candidate pair: start and end on track A, and start and end on track B.

2. Align short speech units, not whole subtitles

The original version used dynamic programming over entire ASR segments. It worked for clean, similarly paced recordings, but failed when one language split a sentence differently. A long segment on one side could correspond to two short segments on the other, and a single wrong grouping could push the rest of the path out of sync.

The current approach first turns each transcript into short speech units. Units are merged only within strict limits: about 6.5 seconds per unit, three units per candidate group, and an eight-second group maximum. Explicit noise utterances such as “um” or “嗯” are removed. This makes the alignment problem smaller without pretending that every ASR segment is a sentence.

3. Use multilingual embeddings, but keep time in the loop

Text alone is ambiguous. The same phrase can occur several times, and different dubs can describe the same moment with different wording. The service embeds candidate windows with Cloudflare Workers AI's multilingual BGE-M3 model, then compares both the original timestamps and normalized progress through each track.

The time distance is deliberately not just “closest timestamp wins.” Absolute time is useful when the tracks are tightly synchronized. Normalized progress is useful when one version runs consistently faster. The implementation evaluates both and keeps a candidate only when it is inside a time band of roughly 22–45 seconds.

There is also a minimum semantic similarity threshold. It becomes stricter as the time gap grows and is slightly stricter for one- or two-word utterances, where a chance embedding match is more likely. When a candidate merges several units, every individual unit must support the match; a strong phrase cannot carry a weak neighbor along with it.

4. Make the path monotonic

After candidates are generated, a dynamic-programming pass selects a monotonic path through both transcripts. A match advances both sides. Unmatched speech can be skipped on either side. That prevents the most damaging error mode: jumping backward or across large unrelated regions just because an embedding pair looked similar.

The selector also checks ambiguity. If two nearly equal candidates compete for one side of the same unit, the pair is treated as risky and can be excluded from the default result. That is important for repeated dialogue, refrains, and generic greetings.

5. Publish a conservative quality signal

The UI does not claim to show translation accuracy. It shows a conservative alignment score: the minimum of the mean semantic similarity and the duration coverage on either track. If one language loses a large section, the score drops even when the remaining pairs look confident.

The user still gets a review screen. They can listen to each original clip, inspect the proposed text, and uncheck bad pairs before any audio is generated. Automatic alignment is a draft, not a promise that every semantic match is correct.

6. Render locally with Web Audio and ffmpeg.wasm

Once the user confirms the pairs, the browser decodes both source files with Web Audio. The confirmed time windows are converted to PCM WAV clips, grouped into batches of about 12 pairs, encoded to 192 kbps MP3 with ffmpeg.wasm, and concatenated. The final concat is a stream copy because all batches use the same encoder settings.

This keeps the expensive audio work in the browser and makes it possible to generate the finished track without uploading a second copy of the audio for rendering. The server does the transcription and semantic alignment; the client performs the final cut and join.

What I would improve next

Two areas deserve more work. The first is better handling of overlapping speech and voice-over. The second is a more calibrated confidence model; the current score is deliberately conservative rather than a statistically calibrated probability.

For a focused language tool, that tradeoff is acceptable: a smaller result with visible coverage gaps is more useful than a longer track that confidently pairs the wrong sentences. The complete workflow, including free trial limits and sample tracks, is at Bilingual Mix.

If you are building anything with ASR timestamps, keep both absolute time and relative progress, make the matching path monotonic, and surface uncertainty before rendering. Those three decisions prevented a surprising number of silent failures.

DEV Community: 肖邦强