DEV Community

MartinDelophy
MartinDelophy

Posted on

Running Chinese TTS Fully in the Browser: Migrating from Piper to Kokoro 1.1 FP16

#ai #webdev #opensource #javascript

When an AI video editor needs text-to-speech, the easiest solution is usually a hosted API: send the script to a server, wait a few seconds, and download the audio.

For Timeline Studio, we deliberately took a harder route. We want voice generation to run on the user's device, inside the browser, alongside the editable timeline.

In Timeline Studio v1.0.0, we replaced our previous Chinese Piper/VITS ONNX voices with a browser-ready FP16 build of Kokoro multi-lang v1.1. The new path provides two female and two male voices, handles Chinese text with inline English, and still performs synthesis locally through sherpa-onnx WASM.

If browser AI, WebAssembly, ONNX, or open-source video editing interests you, a GitHub star is greatly appreciated.

Why move away from Piper for Chinese?

Piper was a pragmatic starting point. Its ONNX models are relatively compact, the browser deployment path is well understood, and it allowed us to build a local TTS workflow without depending on a metered cloud service.

But the quality bar changes when TTS becomes part of a video editor.

It is no longer enough for a model to simply pronounce a sentence. Product demos, tutorials, explainers, and narrative videos need more natural pacing, clearer voice choices, and reliable handling of modern Chinese copy that often contains English names and technical terms.

Consider this sentence:

使用 Timeline Studio，让 AI video editing 直接在浏览器里完成。

Splitting it into separate Chinese and English clips creates avoidable problems: speaker identity can change, pacing can jump, and the sentence loses its natural context. We wanted one speaker to generate the complete utterance.

Kokoro multi-lang v1.1 gave us a better foundation for that experience.

Piper has not disappeared from Timeline Studio. It still powers browser voices for German, Spanish, French, Italian, and Brazilian Portuguese. This migration only replaces the Chinese Piper route.

A browser model is more than an ONNX file

Choosing a model was the easy part. Turning it into a dependable browser feature required much more work.

A server can assume a controlled filesystem, abundant memory, a long-running process, and predictable model storage. A browser must deal with:

  • first-use download size;
  • memory and bandwidth pressure;
  • main-thread responsiveness;
  • storage quotas;
  • interrupted or incomplete downloads;
  • regional model availability;
  • immutable model versions;
  • cache migration across application releases.

A model that produces one successful sample on a developer machine is a demo. A product feature must download, verify, initialize, cache, reuse, upgrade, and fail clearly.

Converting Kokoro 1.1 to FP16

We converted the selected Kokoro multi-lang v1.1 bundle to FP16 and packaged it for the sherpa-onnx WASM TTS runtime.

The resulting browser pipeline looks like this:

User script
    ↓
Chinese/English text normalization
    ↓
Voice ID → speaker ID
    ↓
Kokoro multi-lang v1.1 FP16
    ↓
sherpa-onnx WASM in a Web Worker
    ↓
Float32 PCM samples
    ↓
WAV encoding in the browser
    ↓
Timeline Studio asset library
Enter fullscreen mode Exit fullscreen mode

FP16 reduces storage, transfer, and runtime memory-bandwidth pressure relative to FP32 while preserving the voice quality we need.

It is not magic. The first run still requires a substantial model download, and synthesis speed depends on the device, browser, available memory, and script length. The goal was not to pretend the cost had disappeared, but to make a higher-quality multilingual TTS model practical in a local browser workflow.

Keeping inference off the main thread

Model initialization and synthesis are compute-heavy operations. Running them on the main thread would make an editor feel frozen.

Timeline Studio creates a dedicated worker:

worker = new Worker("/kokoro-multilang.worker.js");

worker.postMessage({
  type: "init",
  baseUrls,
});
Enter fullscreen mode Exit fullscreen mode

The UI thread handles the script, selected voice, progress display, and timeline state. The worker downloads and verifies the runtime bundle, initializes sherpa-onnx WASM, creates the offline TTS session, and generates the samples.

A synthesis request is intentionally small:

worker.postMessage({
  type: "generate",
  requestId,
  text,
  sid,
  speed,
});
Enter fullscreen mode Exit fullscreen mode

The generated sample buffer is returned as a transferable object:

self.postMessage(
  {
    type: "result",
    requestId: message.requestId,
    samples: audio.samples,
    sampleRate: audio.sampleRate || tts.sampleRate,
  },
  [audio.samples.buffer],
);
Enter fullscreen mode Exit fullscreen mode

Transferring the underlying ArrayBuffer avoids copying a potentially large block of PCM data.

The worker stays alive after initialization, so repeated generations in the same editing session reuse the warm runtime instead of downloading and initializing the model again.

Four voices, one shared model

We selected four speakers with distinct roles:

Timeline Studio voice Upstream speaker Character
Qinglan / 晴岚 zf_001 Natural, clear female voice
Ruoxi / 若溪 zf_073 Softer female voice
Yunzhou / 云舟 zm_009 Steady, natural male voice
Jingche / 景澈 zm_010 Younger, brighter male voice

Internally, product voice IDs map to the four speaker slots:

const SPEAKER_IDS = Object.freeze({
  zh_f_qinglan: 0,
  zh_f_ruoxi: 1,
  zh_m_yunzhou: 2,
  zh_m_jingche: 3,
});
Enter fullscreen mode Exit fullscreen mode

All four voices share the same FP16 model. Switching speakers does not trigger another full model download.

We also provide a real preview generated by the matching speaker for every selectable voice. A voice card should never play a placeholder sample from a different speaker.

Keeping mixed Chinese and English in one utterance

Before synthesis, Timeline Studio normalizes the text while preserving Han characters, Latin characters, numbers, whitespace, and common punctuation:

const text = normalized
  .replace(
    /[^\p{Script=Han}\p{Script=Latin}0-9\s，。！？；：、,.!?;:()\-]/gu,
    "",
  )
  .replace(/[ \t]+/g, " ")
  .replace(/ *\n+ */g, "。")
  .replace(/[。]{2,}/g, "。")
  .trim();
Enter fullscreen mode Exit fullscreen mode

The important product rule is that mixed-language copy remains one linguistic utterance. We do not split a Chinese sentence simply because it contains "WebGPU", "API", or an English product name.

That preserves speaker identity, punctuation-driven pauses, and editing simplicity.

Parallel downloads with integrity checks

The browser bundle includes the WASM binary, JavaScript runtime, wrapper code, model data, and supporting resources. Large data is divided into parts described by a manifest.

The worker downloads manifest entries in parallel:

const entries = [
  ...manifest.runtime.files,
  ...manifest.runtime.data.parts,
];

const resources = await Promise.all(
  entries.map((entry) =>
    fetchAndVerify(baseUrl, entry, onChunk),
  ),
);
Enter fullscreen mode Exit fullscreen mode

Each entry includes its expected byte length and SHA-256 digest. The worker verifies individual downloads, reassembles the model data in manifest order, and verifies the completed payload again before creating the TTS session.

A successful HTTP status is not enough. Proxies, partial caches, and interrupted connections can all return incomplete model data. Integrity verification prevents corrupted weights from reaching inference.

Hugging Face and ModelScope mirrors

Model availability is a product issue, especially for a browser application serving users in different regions.

We mirror the voice bundle in repositories we control on both Hugging Face and ModelScope, pinned to immutable provider revisions.

Timeline Studio prefers ModelScope for Chinese and domestic sessions and Hugging Face elsewhere. If the preferred route fails, it attempts the other mirror.

The two providers use different URLs and revisions for the same artifacts, so we canonicalize them to one internal cache identity. Otherwise, a browser could store two copies of the same large model after a network route changes.

The source may change; the model identity should not.

Cache migration matters

Timeline Studio also uses local models for captions, music, vision, voice conversion, and other features. Browser storage cannot be treated as unlimited.

Before loading the Kokoro bundle, the application preflights available storage. During upgrades, it:

  • removes legacy FP32 Kokoro files;
  • avoids duplicate Piper caches;
  • migrates unchanged files to the new canonical revision;
  • deletes changed manifest or model parts;
  • evicts stale voice families when capacity is tight.

This is not glamorous work, but it separates a one-release demo from a browser AI application that can continue evolving.

TTS is only the beginning of the workflow

Timeline Studio is not a standalone text-to-speech page.

A generated WAV becomes an editable media asset. The user can audition it, regenerate it, place it on the voiceover track, trim and move it, generate captions, mix it with source audio and music, and export the final project to MP4 or WebM.

The current voice routing is deliberately model-specific:

Language or use case Browser model
Chinese and mixed Chinese/English Kokoro multi-lang v1.1 FP16
English Kokoro 82M ONNX
German, Spanish, French, Italian, Brazilian Portuguese Piper/VITS ONNX

We prefer specialized, verified paths over claiming that one model is best for every language.

What this migration delivered

The final change included more than replacing two entries in a voice picker:

  • Kokoro multi-lang v1.1 replaced the Chinese Piper route;
  • the model was converted and packaged as FP16;
  • sherpa-onnx WASM runs synthesis locally;
  • a persistent Web Worker keeps the editor responsive;
  • two female and two male voices share one model;
  • mixed Chinese/English scripts remain one utterance;
  • model parts download in parallel and are SHA-256 verified;
  • Hugging Face and ModelScope provide pinned fallback mirrors;
  • both providers share one cache identity;
  • storage preflight and cache migration support future releases;
  • generated audio enters the editable video timeline instead of ending as a demo file.

Closing thoughts

Running AI in the browser is not simply a matter of moving an ONNX file to the frontend.

A production-quality local feature needs a model delivery system, an isolated runtime, integrity checks, regional routing, cache ownership, upgrade behavior, honest progress reporting, and a clear place in the user's workflow.

That engineering work is less visible than a model benchmark, but it is what turns local inference into a usable creative tool.

Timeline Studio is open source under the MIT License:

If this work is useful to you, please consider starring the repository, opening an issue, or contributing.

Top comments (0)