I build Elsewhere, a browser-based ebook reader that reads DRM-free books aloud. The voice is made on the reader's own machine, inside the browser tab, with Piper neural voices running through ONNX Runtime Web on its WebAssembly backend. The text is not sent to a cloud speech API.
This post covers that part: how it is wired, what the thread count turned out to depend on, what the numbers look like, and a WebGPU experiment that stalled my laptop.
How it is wired
- ONNX Runtime Web runs the Piper model on its WASM backend.
- eSpeak NG turns the text into phonemes first.
- One speech worker holds one voice. Switching voice terminates that worker and starts another.
- ONNX Runtime starts its own threads inside the worker.
- A second, long-lived worker encodes MP3. More on that below.
Threads only exist if the page is cross-origin isolated
ONNX Runtime's WASM backend gets its extra threads from SharedArrayBuffer, and browsers only expose that to cross-origin isolated pages. Until I made the site isolated, there was no SharedArrayBuffer, so ONNX Runtime fell back to one thread.
The difference was large:
- Piper, a 37-second passage with the ljspeech voice: about 2.2 seconds on 4 threads, roughly 17 times real time, and roughly 7 times real time on 1 thread.
- Kokoro q8 on WASM: 0.56 times real time on one thread, 1.17 times on 4.
I used
Cross-Origin-Embedder-Policy: require-corpinstead ofcredentialless, because Safari only supportsrequire-corp.
The default thread count is the smaller of 4 and half the core count. I tested more on an M3 with 4 performance and 4 efficiency cores. With Kokoro, 4, 6, 7 and 8 threads all landed between 1.11 and 1.13 times real time, so I kept 4. I did not retest Piper at other thread counts, so for Piper I only know that 4 beats 1.
Voice files and the first load
Each Piper voice is 63 to 79 MB. The browser downloads it straight from Hugging Face and stores it in the Cache API, in a bucket called piper-voices. The write is all or nothing. If the reader hits Stop or the connection drops mid-download, no half-written model is left behind to break the next load.
The first load is the part readers feel:
- A real 64 MB download of one voice took 34 seconds on my connection.
- The same size of voice took 9.3 seconds in headless Chrome on a faster connection.
- Once the voice is cached, loading it takes about 0.8 to 1 second. The reader sees a progress notice during the download, and the Studio stays open while it runs. I have not tested what happens when the browser hits its storage quota or evicts the cache, so I cannot say how often a reader has to download a voice again.
The numbers
Speed here means seconds of audio produced per second of generation.
| Run | Setup | Audio | Generation time | Speed |
|---|---|---|---|---|
| Alice in Wonderland, about 27,000 words | Piper [voice, browser and machine to confirm] | 2 h 9 min | 13 min | about 9.9 times |
| The Waste Land, chapter I | Piper, voice Linda | about 4 min | about 25 s | about 9.6 times |
| The Waste Land, chapter I | Kokoro q4 on the CPU | 210 s | about 130 s | about 1.6 times |
| One 37 s passage | Piper ljspeech, 4 threads | 37 s | 2.2 s | about 17 times |
| One 37 s passage | Piper ljspeech, 1 thread | 37 s | not recorded | about 7 times |
What I have not measured: any phone, battery use, and Firefox. Safari works and so does Chrome. The only memory figure I have is for Kokoro, which uses about 1.3 GB per tab. Treat the table as results from my machine and nothing more.
Word timing and the MP3 worker
The read-along needs to know when each word is spoken. I patch the model in the browser to expose its phoneme durations, which costs 4 to 12 ms per model. That meant I did not need to host patched copies of the voices.
The second worker exists because of a measurement. I first encoded MP3 on the main thread with lamejs, which took about 90 ms per 10 seconds of audio. That was enough to make scrolling hitch. A single long-lived worker now does all the encoding.
The WebGPU detour
Piper is the voice that I shipped. Before that I tried a bigger model, Kokoro, and tried to run it on WebGPU. I never tried Piper on WebGPU. I assumed the dynamic shapes in VITS made it a poor fit, and the CPU was already fast enough.
My machine was a MacBook with an M3, 8 cores and 16 GB of unified memory. Kokoro fp32 on WebGPU ran at about 2.5 times real time (14.6 seconds for the passage), which beats the 1.17 times I got from the q8 model on 4 CPU threads. It also stalled the desktop.
- Memory. fp32 took all of the unified memory with no cap and blocked the GPU work that draws the screen. I measured sporadic 1 to 3 second frame stalls. I believe unified memory is the cause, but I never profiled it, so treat that as a guess.
- fp16. It produced gibberish.
- Fixes that did not fix it. The model already ran in a worker. Recycling it every 6 generations only freed ONNX Runtime's leaked GPU buffer pool. I then offered "Graphics card or Processor" as a setting, and finally removed WebGPU entirely. The processor path had no such problem. Across 468 seconds of Kokoro generation, 3 of 33,660 frames took longer than 50 ms.
That was not the end of Kokoro. On the CPU, even at q4 (2.15 times real time), readers still waited about 40 seconds for first audio. So, I removed it completely. Could probably offer it in a paid tier as "premium voices" that generates book audio in the cloud. Let's see.
What broke along the way
- A leak I introduced. The shared worker built a second ONNX session for a voice it already held, so it leaked one model per chapter.
- A load that did nothing. An earlier design ignored the load call once any model was loaded. A second run on a different backend kept the first model with no indication.
-
Safari and streams. Safari cannot iterate a stream with
for await, which broke the unzip step inside the phonemizer. I fixed it at bundle time. - int8 quantization. It shrank a voice by about 3 times and made it 2.5 to 3 times slower, so I did not use it.
Limits
- The tab has to stay open while narration is made.
- The voices are English only.
- I have never timed a phone run, so I will not claim that it works well on a phone.
- The timings come from one M3 laptop and a few runs. Treat them as a starting point.
Try it
Disclosure: I built Elsewhere, the app I mention here. It opens DRM-free ebooks, so Kindle store books will not open. The code is closed source and will stay that way. You can hear the voice and the read-along without an account at listenelsewhere.com/try.
A question for anyone who has tried it: has a VITS-style model with dynamic shapes run on WebGPU for you without starving the desktop on a unified-memory machine? I gave up on it, and I would like to know if I gave up too early.
Top comments (0)