Japanese puts the verb at the end of the sentence. I built an all-local streaming speech translator and measured what that costs: early English-to-Japanese guesses share almost nothing with the final translation.
Scope note, first paragraph as promised: read speech (FLEURS test set) only, greedy decoding everywhere, Whisper-small plus two small MT backends (opus-mt + Qwen2.5-1.5B), no human listening test, no LLM judge. The headline asymmetry is measured on reference transcripts, so ASR error is excluded by construction.
Why word order is a latency problem
English is SVO: "The dog ate the food." Japanese is SOV: the verb comes last. A simultaneous translator hearing English must either guess the verb early (and flicker when wrong) or wait for it (and lag). Textbook linguistics; I wanted a number. So I built koe-bridge: Whisper-small ASR → MT (opus-mt-ja-en one way, local Qwen2.5-1.5B the other) → Kokoro / Style-Bert-VITS2 TTS, all on one RTX 3060, evaluated on FLEURS Japanese and English test utterances.
The cascade holds up (M1)
On 200 utterances per direction: ASR CER 0.091 (Japanese) / WER 0.104 (English); MT BLEU barely drops when fed ASR output instead of clean text (10.6 vs 18.1 ja→en; 24.1 vs 24.3 en→ja). A dead end worth recording: opus-mt-en-jap scored BLEU 0.0 — fluent, unrelated output with a verified-working encoder — so the en→ja arm uses Qwen instead (an honest NMT-vs-LLM comparison, not a planned one). Per-stage GPU cost: ASR ~0.35s, opus MT 1.39s, Qwen MT 0.38s, TTS ~0.2s. All per-row data is in the linked HF dataset.
First audio in ~1-2 seconds (M2)
A lockstep streaming sim (VAD segments → ASR → MT → TTS) on 20 utterances per direction: first-audio lag p50 1.86s ja→en, 1.26s en→ja (p95 3.32s / 2.22s). One real bug on the way: the silero VAD carries RNN hidden state across files, so segmentation was non-deterministic across runs (same file kept 87% then 98% of audio). Fixed with per-file state resets plus a regression test; M2 re-ran after the fix.
The wait-cost number (M3)
Translate the first 25/50/75% of each sentence and measure prefix-survival: how much of the early guess survives as a prefix of the full-sentence translation. On 100 reference transcripts per direction (no ASR error):
- EN→JA: 0.007 / 0.014 / 0.020 — early guesses are rewritten almost entirely.
- JA→EN: 0.077–0.134 / 0.088–0.184 / 0.132–0.204 depending on backend.
With the same backend (Qwen) both directions, matched 50% prefixes: 0.184 ja→en vs 0.014 en→ja, ~13x. The SVO→SOV direction pays the penalty, as predicted — a retranslation display would flicker badly in English-to-Japanese, so a real system must hold output until the verb (and pay the lag) instead. Two methodology notes: the first JA arm was invalid (word-splitting a language with no spaces made every prefix the full sentence — caught, fixed with character prefixes, re-run), and the pipeline's exact backend calling convention (prompt wording, >>en<< prefix) changed the numbers, so M1/M2/M3/demo now share one documented convention.
Negation flips both ways (follow-up)
The original spec question was polarity: Japanese negation also comes last, so prefix translation risks showing "yes" then flipping to "no". Measured on 40 negated FLEURS sentences per direction (reference transcripts, 10% prefix steps): polarity flips fall from ~0.7 at 10% prefixes to ~0.25 at 90%, median first-correct-polarity at 30% of the sentence, affirm-then-negate 0.65–0.70 — with no directional asymmetry this time. The worse finding: full-sentence translations lose explicit negation markers 22–25% of the time ("He did not set a figure" → an affirmative Japanese sentence). Small open backends don't just delay negation; they sometimes drop it. Also a deployment note: on fragments, opus hallucinates ("It's a dead end", "(Music)") while Qwen degrades into coherent partials.
What is still open
The demo (scripts/demo.py) translates one utterance end-to-end; its sample output mangles 地殻 (earth's crust) as "thin ebony", which tells you exactly how far small open MT backends go. M2 is a lockstep sim, not a live server. All code, results, and the eval dataset are linked below; TTS weights stay local (research-only voices, never published).
Top comments (0)