Most voice translation apps treat a conversation like a walkie-talkie. Person A taps a microphone button, speaks a paragraph, waits for an end-of-utterance timeout, and passes the phone across the table so Person B can tap their own button. In a loud taxi in Tokyo, a family dinner in Koh Samui, or a hotel check-in desk, that turn-taking ceremony breaks the rhythm of how people actually talk.
When we built Hey Interpreter for iPhone, Android, foldables, Apple Watch, and Wear OS, we wanted continuous sub-second two-way subtitles: put the phone flat on the table once, speak naturally in either language, and let both sides read live captions on their own half of the screen.
Shipping that experience across phones, hinges, and watch screens exposed five systems bugs that only show up outside an office Wi-Fi network. Here is how we solved them.
1. 180-Degree #000000 OLED Tabletop Split-Screen and Foldable Hinge Postures
When two people sit across a cafe table, a single portrait transcript forces one person to crane their neck or pass the handset back and forth. In Talk mode, the screen divides into two independent panes on a pure #000000 OLED canvas:
- Top pane (rotated 180 degrees): Faces your partner and renders every turn in their script and language.
-
Bottom pane (0 degrees): Faces you and renders every turn in your language, keeping owner controls (
Pause,Speak aloud,Clear, and language pills) on your side.
Using #000000 instead of dark gray (#121212) matters on OLED panels: unlit pixels draw near-zero current during a 45-minute dinner conversation, and on a night flight the screen blends into the table without casting glare upward.
Foldable hardware adds a physical constraint across the middle of that split. When a foldable sits half-opened in tabletop posture, placing buttons or subtitle baselines inside the crease makes text hard to read and taps unreliable.
-
On Android (
androidx.window.layout.FoldingFeature): We observeWindowInfoTrackerlayout updates forFoldingFeature.State.HALF_OPENEDwithFoldingFeature.Orientation.HORIZONTAL, readfeature.boundsin window coordinates, and reserve a 24dp crease gutter so the upper leaf is 100% partner-facing captions while owner chrome stays on the flat lower leaf. -
On iOS (
UIHingeInteraction): A zero-size UIKit bridge view observesUIHingeInteractionposture updates (falling back to slab layout on standard iPhones) and enforces the matching 24pt hinge clearance acrosstabletop(669x951pt) andbook(951x669pt) postures.
2. Bypassing Dead Captive-Portal Wi-Fi onto 5G/LTE for Live WebSockets
Translation apps get opened most often in airports, hotels, cafes, and train stations. Those are the exact places where a phone associates with a public Wi-Fi SSID that has valid DHCP and local DNS, yet blackholes WebSocket upgrades behind an unauthenticated captive portal.
Standard HTTP clients wait on the default route (Wi-Fi whenever associated). Waiting 15 seconds for a socket timeout while a taxi driver asks where you are going makes a live translator unusable.
When our live audio WebSocket or token handshake detects a Wi-Fi stall or captive-portal failure, we route the stream directly over cellular data without asking the user to toggle Wi-Fi off in system settings:
-
On iOS (
Network.framework):URLSessioncannot forbid Wi-Fi on a single WebSocket while Wi-Fi stays active system-wide, so we constructNWParameters.tlswithNWConnection:
let params = NWParameters.tls
params.prohibitedInterfaceTypes = [.wifi, .wiredEthernet]
This forces the TLS 1.3 WebSocket upgrade onto the pdp_ip0 5G/LTE interface while leaving system Wi-Fi untouched.
-
On Android (
ConnectivityManager): We request a cellularNetworkhandle viaNetworkRequest.Builder().addTransportType(NetworkCapabilities.TRANSPORT_CELLULAR).addCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)and bind OkHttp's socket factory to that cellularNetworkinstance.
3. Zero-Period Streaming Subtitle Grouping and Script-Aware Word Wrapping
Real-time speech-to-translation engines emit partial tokens every ~100ms before punctuation settles, and spoken conversation rarely arrives in tidy sentences. Scripts like Thai (ไทย), Japanese (日本語), and Chinese (中文) also omit ASCII spaces between words.
Rendering raw streaming turns directly causes layout jitter on every short backchannel, and compact surfaces like Apple Watch faces, Wear OS Tiles, or 4x2 / 2x1 Home Screen widgets clip mid-grapheme.
We built SubtitleFlowGrouper identically in Swift (HeyShared) and Kotlin (shared) to group token streams into stable blocks:
-
Acoustic turn merging: Consecutive turns in the same detected language merge into one
SubtitleParagraphuntil the spoken language shifts, an audio gap exceeds4,500ms(new paragraph), or a1,800mspause inserts clause separation. -
Space-free script segmentation: On web demo surfaces and compact caption views,
Intl.Segmenter(undefined, { granularity: "word" })(and mobile ICU word-boundary iterators) wrap Thai, Japanese, and Chinese on real lexical boundaries instead of splitting multi-codepoint grapheme clusters. -
Word-safe watch & widget budgets: Watch viewports pass a tight character budget (
SubtitleFlowGrouper.group(turns, lang, maxChars = 120).suffix(3)). For standalone wrist sessions on watchOS and Wear OS, 200ms 16 kHz mono PCM16 frames encode into 4-bit IMA ADPCM (~8 KB/s) off the render thread (WCSession.sendMessageDataon watchOS), keeping relay latency low enough for Digital Crown scrolling and wrist haptics.
4. +250ms PCM Mic Ducking During Speak Aloud TTS
In Talk mode, tapping Speak aloud on any translated bubble reads that line in the partner's language. If the live mic stays hot while AVSpeechSynthesizer or Android TextToSpeech plays through the loudspeaker, the mic captures its own speaker output and translates its own voice back into phantom echo turns. Tearing down the WebSocket on every utterance, however, adds handshake delay.
Our SpeechOutput coordinator handles this at the PCM frame layer:
object SpeechOutput {
private const val TAIL_MS = 250L
// Muted before the first TTS sample plays, plus 250ms after completion
}
private fun muteWhileSpeaking(pcm: ByteArray): ByteArray =
if (SpeechOutput.micMuted()) ByteArray(pcm.size) else pcm
-
Pre-sample mute:
SpeechOutput.started()runs beforeengine.speak(...), not inside the asynconStartcallback, because audio HAL buffers can emit the first phoneme before the callback lands on the main thread. -
Zero-filled PCM continuity: While muted, the recorder keeps streaming zeroed
ByteArray(pcm.size)frames so WebSocket keepalive and audio timestamps stay continuous. -
+250msacoustic tail (TAIL_MS = 250L): Room reflections and mixer drain outlast the TTS completion callback by 80-180ms. Holding zero-PCM silence for250msafteronDone/didFinishprevents self-translation echo in cars, tiled rooms, and tabletop sessions.
5. Why 60-Language Streaming Fills the Post-SayHi Gap
Two shutdowns left a gap in everyday two-way voice translation: Amazon retired SayHi Translate, and Microsoft shut down live Translator Conversation sessions. Meanwhile, built-in Apple Translate tops out at 41 languages (omitting Hebrew, Greek, Swedish, Romanian, Hungarian, Tagalog, Malay, Tamil, Bengali, and Swahili) and runs only on Apple hardware, while dedicated gadgets like Timekettle or Pocketalk cost $150-$350 and require charging another device.
Streaming recognition and translation across 60 languages simultaneously in Listen mode (auto-detecting whichever language is spoken in the room) and deterministic two-way routing in Talk mode lets one app cover iPhone, Android foldables, Apple Watch, and Wear OS without mic-button choreography.
Try It Live
- Web interactive split-screen demo & overview: heyinterpreter.com
- iOS & Apple Watch App Store: Hey Interpreter on the App Store
- Product profile & specs: Hey Interpreter on Awesome Tools
If you have hit quirky edge cases with UIHingeInteraction, NWParameters interface routing, or AVAudioEngine + TTS echo cancellation on mobile, drop a comment below. Happy to share notes from our test harness.
Top comments (0)