ONNX-Based Text-to-Speech (TTS) Integration for HarmonyOS Applications

Read the original article：ONNX-Based Text-to-Speech (TTS) Integration for HarmonyOS Applications

Requirement Description

This guide shows how to integrate Sherpa-Onnx Text-to-Speech functionality into HarmonyOS applications, providing offline voice synthesis with circular buffer audio streaming. The implementation uses a worker thread architecture for TTS generation and AudioRenderer for audio playback, converting sample text to natural-sounding speech.

Background Knowledge

The Sherpa-Onnx TTS system is a high-performance, offline neural text-to-speech solution that runs ONNX models locally on devices. This eliminates the need for internet connectivitiy and provides privacy-preserving voice synthesis.

The implementation includes;

VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) Models: Pre-trained neural networks for high-quality speech synthesis.

Multi-Speaker Support: Different voice personalities (0-N speakers) for contextual announcements.

Circular Buffer Architecture: Efficient real-time audio streaming without memory overflow.

Worker Thread Processing: Background TTS generation to prevent UI blocking.

AudioRenderer Integration: HarmonyOS native audio playback with precise sample rate control.

This TTS implementation can be used for;

• Converting any text content to speech
• Offline voice announcements and notifications
• Accessibility features for text reading
• Interactive voice responses in applications

Implementation Steps

Add Sherpa-ONNX Dependency

• Add **''sherpa_onnx: "1.12.11"** to your **oh-package.json5** dependancies.
• Configure worker thread in build-profile.json5 under buildOption.sourceOption.workers.

Download and Setup TTS Models

• Downlad a TTS model from the sherpa-onnx's pre-trained models*.*
• Place the model folder in src/main/resources/rawfile/.

Create TTS Worker Thread

• Implement NonStreamingTtsWorker.ets with model initialization and text processing. (Example below)
• Configure model paths and parameters. (model name, lexicon, data directories)
• Setup callback system for progress reporting and smaple streaming.

Initialize Audio Rendering System

• Create AudioRenderer with appropriate AudioStreamInfo. (sample rate, channels, format)
• Setup circular buffer Circular Buffer for smooth audio playback.
• Implement audio callback for real-time sample processing.

Create TTS Service Class

• Implement a service to handle text-to-speech generation.
• Manage TTS worker communication and audio playback.
• Handle different text inputs and speech parameters.

Integrate with UI Components

• Add TTS controls to your application interface.
• Implement loading states and progress indicators.
• Handle TTS initialization and error states.

Code Snippet / Configuration

Worker Thread Configuration (build-profile.json5)

{
  "apiType": "stageMode",
  "buildOption": {
    "sourceOption": {
      "workers": [
        "./src/main/ets/workers/NonStreamingTtsWorker.ets"
      ]
    }
  },
//Rest of your code
}

TTS Worker Implementation (NonStreamingTtsWorker.ets)

Initialize TTS model and handle generation requests;

import { OfflineTtsConfig, OfflineTts, listRawfileDir, TtsInput, TtsOutput } from 'sherpa_onnx';

function initTts(context: Context): OfflineTts {
  modelDir = 'vits-piper-en_US-libritts_r-medium';  //Your Model's Directory here
  modelName = 'en_US-libritts_r-medium.onnx';   //Your Model's Name
  dataDir = 'espeak-ng-data';  //Data Directory

  copyRawFileDirToSandbox(context, modelDir)
  let sandboxPath: string = context.getApplicationContext().filesDir;

  const config: OfflineTtsConfig = new OfflineTtsConfig();
  config.model.vits.model = modelDir + '/' + modelName;
  config.model.vits.tokens = modelDir + '/tokens.txt';
  config.model.vits.dataDir = sandboxPath + '/' + modelDir + '/' + dataDir;
  config.model.numThreads = 2;

  return new OfflineTts(config, context.resourceManager);
}
//Handle TTS generation Request
workerPort.onmessage = (e: MessageEvents) => {
  const msgType = e.data['msgType'] as string;

  if (msgType == 'tts-generate') {
    const input: TtsInput = new TtsInput();
    input.text = e.data['text'] as string;
    input.sid = e.data['sid'] as number || 0;
    input.speed = e.data['speed'] as number || 1.0;

    tts.generateAsync(input).then((ttsOutput: TtsOutput) => {
      workerPort.postMessage({
        'msgType': 'tts-generate-done', samples: Float32Array.from(ttsOutput.samples),
      });
    });
  }
}

Audio Renderer Setup

Configure audio output for speech playback;

private setupAudioRenderer(): void {
    const audioStreamInfo: audio.AudioStreamInfo = {
      samplingRate: this.sampleRate,
      channels: audio.AudioChannel.CHANNEL_1,
      sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
      encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW,
    };

    const audioRendererInfo: audio.AudioRendererInfo = {
      usage: audio.StreamUsage.STREAM_USAGE_MUSIC,
      rendererFlags: 0,
    };

    const audioRendererOptions: audio.AudioRendererOptions = {
      streamInfo: audioStreamInfo,
      rendererInfo: audioRendererInfo,
    };

    audio.createAudioRenderer(audioRendererOptions, (err, renderer) => {
      if(!err && renderer) {
        this.audioRenderer = renderer;
        this.audioRenderer.on('writeData', this.audioPlayCallback);
      } else {
        //Failed to initialize error here
      }
    });
  }

Main Thread TTS Controller

Manage worker communication and audio playback;

@Component
export struct TtsController{
  @State private isGenerating: boolean = false;
  private workerInstance?: worker.ThreadWorker;
  private readonly scriptUrl: string = 'entry/ets/workers/NonStreamingTtsWorker.ets';
  private sampleBuffer: CircularBuffer = new CircularBuffer(16000 * 5);

  aboutToAppear(): void {
    this.workerInstance = new worker.ThreadWorker(
      'entry/ets/workers/NonStreamingTtsWorker.ets'
    );
    this.workerInstance.onmessage = (e:MessageEvents) => {
      if (e.data['msgType'] == 'tts-generate-done') {
        this.isGenerating = false;
        const samples: Float32Array = e.data['samples'] as Float32Array;
        this.sampleBuffer.push(samples);
        this.setupAudioRenderer();
        this.audioRenderer?.start();
      }
    };
  }

  generateSpeech(text: string) : void {
    this.isGenerating = true;
    this.workerInstance?.postMessage({
      msgType: 'tts-generate',
      text: text,
      sid: 0,
      speed: 1.0,
    });
  }
}

Audio Playback Callback

Convert and output audio samples for playback;

private audioPlayCallback = (buffer: ArrayBuffer) => {
    const numSamples = buffer.byteLength / 2;
    const samples: Float32Array = this.sampleBuffer.get(this.sampleBuffer.head(), numSamples);
    const int16Samples = new Int16Array(buffer);

    for (let i = 0; i < numSamples; ++i) {
      let s = samples[i] * 32767;
      int16Samples[i] = Math.max(-32768, Math.min(32767, s));
    }
  };

Basic Usage Example

Button("Speak Text")
      .onClick(()=>{
        this.ttsController.generateSpeech(this.sampleText);
      })
      .enabled(!this.ttscontroller.isGenerating)

Dependencies (oh-package.json5)

"dependencies": {
    "sherpa_onnx": "^1.12.11"
  }

Limitations or Considerations

HarmonyOS API Version: Requires HarmonyOS API 11 or higher for AudioRenderer and worker thread support

Model Size: TTS models are typically 50-100MB, consider storage limitations on wearable devices.

Processing Power: Real-time TTS generation requires sufficient CPU resources, may affect battery life.

Sample Rate Compatibility: Ensure TTS model sample rate matches AudioRenderer configuration. (common rates: 16000, 22050, 44100 Hz)

Written by Can Kankaya