Audio Deadlines: Running On-Device TTS Without Audible Glitches

Gradium open-sourced Phonon, its on-device TTS engine (https://gradium.ai/phonon). That matters less because of the model than because of what it signals: on-device speech synthesis is now a dependency you can ship rather than a research demo. And a wave of teams is about to find out that the hard part of edge speech is not the model. It is that audio is the one component in your app with a hard real-time deadline, and nothing else in your background trained you for that.

Two clocks, and only one of them forgives you

A web request can take 900 ms and nobody dies. An audio callback cannot take 21 ms when it had 20. Once playback starts, the DAC consumes samples at a fixed rate forever, and your synthesis loop has to stay ahead of it. Miss by enough and the buffer underruns: you get a click, a pop, or a robotic stutter that users describe as "the app is broken".

Do the arithmetic once and keep it on the wall. At 24 kHz with a 20 ms mixer period, each callback needs 480 samples. Your synthesis loop is not measured in requests per second, it is measured in whether it produces 480 samples every 20 ms, including while the CPU is thermally throttled and the OS is doing a background sync.

This changes what you optimize. P50 latency is nearly irrelevant. What matters:

• Worst-case chunk time, because one slow chunk is an audible artifact.
• Sustained real-time factor over minutes, not seconds, because governors ramp down.
• Slack, the gap between your worst chunk and the buffer you pre-rolled.

Real-time factor is a mean, and means lie

RTF — synthesis wall time divided by audio duration produced — is the number everyone quotes. RTF 0.3 sounds great. It tells you the engine can run 3x faster than playback on a cold device with the screen on and nothing else running. That is not your user's device.

I have watched an engine hold RTF 0.55 for 60 seconds and drift to 1.1 by minute four, because the SoC dropped from the big cluster to the little cluster and stayed there. RTF measured over 5 seconds would have shipped that bug.

So measure the distribution and measure it long:

"""Deadline harness for an on-device TTS engine.

Reports whether the engine can keep the audio clock fed, not how fast it
is on average. Run it for 10+ minutes on a real device, not a laptop.
"""
import statistics
import time
from dataclasses import dataclass, field

SAMPLE_RATE = 24_000                    # engine output rate
BLOCK_MS = 20                           # one mixer/callback period
BLOCK_SAMPLES = SAMPLE_RATE * BLOCK_MS // 1000   # 480

@dataclass
class Run:
    chunk_ms: list[float] = field(default_factory=list)   # wall time per chunk
    audio_ms: list[float] = field(default_factory=list)   # audio produced per chunk

def measure(synth, texts, warmup=3):
    # Warm up: page in weights, let the JIT/kernel selection settle.
    for t in texts[:warmup]:
        synth(t)
    run = Run()
    for text in texts:
        t0 = time.perf_counter()
        pcm = synth(text)                                # mono float32 or int16
        run.chunk_ms.append((time.perf_counter() - t0) * 1000)
        run.audio_ms.append(len(pcm) / SAMPLE_RATE * 1000)
    return run

def report(run: Run, prebuffer_ms=400):
    rtf = sum(run.chunk_ms) / sum(run.audio_ms)
    p99 = statistics.quantiles(run.chunk_ms, n=100)[98]
    worst = max(run.chunk_ms)
    # Conservative bound for a stop-and-wait loop: a chunk longer than the
    # pre-roll plus one block drains the buffer before the next chunk lands.
    slack = prebuffer_ms + BLOCK_MS - worst
    print(f"RTF (mean)            {rtf:.3f}   # <1.0 is necessary, not sufficient")
    print(f"chunk p99             {p99:.0f} ms")
    print(f"worst chunk           {worst:.0f} ms")
    print(f"slack after prebuffer {slack:.0f} ms  {'OK' if slack > 0 else 'UNDERRUN'}")
    return rtf, p99, worst

If your engine pipelines synthesis and playback, the slack bound is pessimistic — that is fine, it is a bound. What you must not do is report RTF alone and call it green.

Prebuffer is your jitter absorber, and it costs you responsiveness

You cannot fix jitter with a faster model alone; you fix it with buffer. Pre-roll 300–500 ms of audio before you start the output stream, and that headroom absorbs a late chunk without an underrun.

The trade-off is real and it is user-facing. In a voice assistant, prebuffer is perceived as the delay between the user finishing a sentence and the assistant starting to speak. In a screen reader, it is a lag on every cursor move. There is no universal right answer, so pick per surface:

• Conversational: 150–250 ms prebuffer, tighter model, accept occasional degradation.
• Narration / media: 400–800 ms, and nobody notices.
• Accessibility: prebuffer short utterances fully before starting; a 3-word phrase is cheaper to synthesize whole than to stream and stitch.

The seam problem

Streaming TTS means chunking text and chunking audio, and both create seams. Sentence-level text chunking changes prosody across the join; frame-level vocoder chunking produces a discontinuity at the boundary unless you overlap and crossfade. A constant-power crossfade of 5–10 ms hides most of it. Under 3 ms you will hear a tick on plosives; over 20 ms you smear consonants.

Two failure modes I have seen:

1. Chunk boundary lands mid-word. Text splitter respects sentence boundaries in English, then a user types "e.g." or a German compound and you cut inside a token. The engine pronounces half a word with full confidence.
2. Crossfade applied to silence. Overlap-add on a chunk that ends in a natural pause attenuates the pause and makes the voice sound rushed. Gate the crossfade on frame energy, or normalize at the frame level.

Quantization changes prosody before it changes intelligibility

Int8 quantization on the acoustic model and vocoder usually preserves intelligibility longer than it preserves naturalness. WER measured through an ASR round-trip will stay flat while the voice starts sounding flat and slightly buzzy — and users notice that immediately even though your metric does not.

What I measure instead:

• F0 contour drift against a float32 reference on a fixed 50-sentence set. Mean absolute deviation in cents over voiced frames.
• Spectral distance on sustained vowels, which is where buzz shows up.
• A/B on device speakers, because the phone's tiny driver hides artifacts that earbuds expose. Test both.

Also: thread count is not free performance. Four threads finish a chunk faster but draw more power and heat the SoC, which is what caused the RTF drift in the first place. On a phone, 2 threads plus a larger prebuffer often beats 4 threads plus a small one over a 10-minute session.

Platform details that bite

• Android: AudioTrack has a low-latency path (PERFORMANCE_MODE_LOW_LATENCY) but it wants the device native sample rate. Synthesizing at 24 kHz and letting the framework resample to 48 kHz adds latency and a filter you did not choose. Match the device rate or resample yourself with a known-quality filter.
• iOS: AVAudioSession category and mode decide whether you get the low-latency route. playback with modeDefault is not the same as voicePrompt; test on hardware, not the simulator.
• Embedded Linux: ALSA period size is your callback period. Set period_size explicitly and check snd_pcm_sw_params for start thresholds; defaults vary by board and are frequently wrong.
• Page faults: memory-mapping multi-hundred-megabyte weights is correct, but the first utterance after a cold start can fault in enough pages to miss deadlines. Either warm the model at app start or mlock the hot region.

Fallbacks that don't sound like fallbacks

Decide your degradation ladder before launch, not during an incident:

1. Full-quality on-device voice.
2. Lower sample rate (48k → 24k) — barely perceptible, halves vocoder cost.
3. Shorter utterances with more prebuffer per chunk.
4. Server TTS, only if the network is up and the user is not on a metered connection.

Never switch voices mid-sentence. A voice change at a sentence boundary is a quirk; a voice change mid-clause sounds like corruption.

Pitfalls

• Reporting mean RTF and calling the engine "faster than real time".
• Benchmarking on a laptop or an emulator. Both lie about thermals and audio routing.
• Testing with a single short sentence. Long utterances keep the CPU hot

Comments

[arcwrightai]

The details here matter more than the headline, honestly.