Android lightweight TTS with Kokoro

About a year ago Hexagrad released Kokoro, a high-performance, open-source text-to-speech (TTS) model. It is extremely efficient and uses only 82 million parameters to deliver very natural and human-sounding speech.

Its small footprint enables it to run locally on Android. It could be an interesting alternative to Android’s platform TTS API which relies on cloud-based synthesis for better quality voices or on older robotic sounding system voices.

ONNX Runtime

To use it on Android, you can export it to the ONNX (Open Neural Network Exchange) format and then use the ONNX Runtime for Android for inference. ONNX is backed by Microsoft but supported by a dynamic open source community.

Initially created to simplify the deployment of a model in the cloud while maximizing the inference performances, ONNX supports various ML frameworks (PyTorch, TensorFlow, JAX, etc…). The model is passed in its original format to the appropriate ONNX converting library which converts it into a file combining both the weights and the model computation graph into a single .onnx file.

This file is then passed to the ONNX runtime which leverages the targeted platform’s hardware accelerators to maximize inference performance.

In our example we want to run Kokoro on Android, so we’ll use the ONNX Runtime for Android. Also, it’s worth noting that we skipped the conversion step.

Implementation

Our inference pipeline to generate speech from text using the Kokoro on Android looks like this:

Let's review the steps to follow to add it to your Android app. First add ONNX Runtime to your gradle dependencies:

dependencies {
    implementation("com.microsoft.onnxruntime:onnxruntime-android:1.23.2")
}

Then initialize an ONNX Runtime session (OrtSession):

import ai.onnxruntime.OrtEnvironment
import ai.onnxruntime.OrtSession
import android.content.Context
import java.io.File

class KokoroRunner(private val context: Context) {
    private val env = OrtEnvironment.getEnvironment()

    private val session: OrtSession by lazy {
        val modelFile = File(context.filesDir, "kokoro.onnx")

        val options = OrtSession.SessionOptions().apply {
            addConfigEntry("nnapi.flags", "USE_FP16")
            addConfigEntry("nnapi.use_gpu", "true")
            addConfigEntry("nnapi.gpu_precision_loss_allowed", "true")
        }

        env.createSession(modelFile.absolutePath, options)
    }
}

Before using the session to send the text to the model for speech synthesis, we need to:

1/ Clean up the text to, for example, remove abbreviations and convert the numbers into text:

“Mr. Smith saw 1,234 users on version 3.14.”

becomes:

“Mister Smith saw one thousand two hundred thirty four users on version three point one four.”

2/ Convert the English text string into phonemes via the IPA (International Phonetic Alphabet).

“Mister Smith saw one thousand two hundred thirty four users on version three point one four.”

becomes:

“ˈmɪstɚ smɪθ sɔ ˈwʌn ˈθaʊzənd tu ˈhʌndɹəd ˈθɝdi fɔɹ ˈjuzɚz ɑn ˈvɝʒən θɹi pɔɪnt wʌn fɔɹ”

As an example of implementation, you can review KokoroPhonemeConverter.kt.

Once we have the phonemes we can convert them into tokens that the model will understand. We are also loading the voice style by converting the af_sky.npy file (the American female voice named “Sky”) into a tensor. You can read more about the other Kokoro voices available here.

import ai.onnxruntime.OnnxTensor

val normalizedText: String = normalize(text)
val phonemes: String = phonemize(normalizedText)
val tokenIds: LongArray = tokenize(phonemes)
val style: Array<FloatArray> = loadVoiceStyle("af_sky.npy")

fun runKokoro(
    tokenIds: LongArray,
    style: Array<FloatArray>,
    speed: Float = 1.0f
): FloatArray {
    val paddedTokens = longArrayOf(0L) + tokenIds + longArrayOf(0L)

    OnnxTensor.createTensor(env, arrayOf(paddedTokens)).use { tokenTensor ->
        OnnxTensor.createTensor(env, style).use { styleTensor ->
            OnnxTensor.createTensor(env, floatArrayOf(speed)).use { speedTensor ->

                val inputs = mapOf(
                    "tokens" to tokenTensor,
                    "style" to styleTensor,
                    "speed" to speedTensor
                )

                session.run(inputs).use { outputs ->
                    return (outputs[0].value as FloatArray).copyOf()
                }
            }
        }
    }
}

In the app, once ONNX Runtime returns FloatArray, to play it you need to:

• clamp each float sample to [-1f, 1f],
• convert each sample to a signed 16-bit PCM,
• pack it as little-endian bytes.
• And finally stream those bytes through Android's AudioTrack API.

For more details, look at the code here.

This covers the high-level implementation for now. I may publish a complete project sample later, but in the meantime, if you experimented with Kokoro on device, I’d love to hear about your experience!