DEV Community

SoftwareDevs mvpfactory.io
SoftwareDevs mvpfactory.io

Posted on • Originally published at mvpfactory.io

Wiring Whisper.cpp to Android's AudioRecord API

#webdev #programming 
---
title: "Wiring Whisper.cpp to Android's AudioRecord: Sub-100ms On-Device ASR with a JNI Ring Buffer"
published: true
description: "Wire AudioRecord PCM buffers directly to whisper.cpp via JNI, use a ring-buffer for overlapping capture and inference, and hit real-time factors below 0.3x on mid-range Android devices."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://mvpfactory.co/blog/whisper-cpp-android-audiorecord-jni
---
Enter fullscreen mode Exit fullscreen mode

What We Are Building

By the end of this tutorial you will have a streaming speech recognition pipeline that skips the WAV file entirely. We wire AudioRecord's raw PCM buffers straight into whisper.cpp through a JNI bridge, front it with a lock-free ring buffer, and offload the encoder to the GPU via GGML's Vulkan backend. The result: sub-100ms perceived latency on mid-range Android hardware.

Let me show you a pattern I use in every production on-device ML project — the bottleneck is almost never the model, it is the data pipeline around it.

Prerequisites

Make sure your environment matches before wiring this up:

Requirement Minimum Recommended
Android NDK r25c r26b
Android API level 26 (Oreo) 28+
whisper.cpp b1.5.0+ latest main
CMake 3.22 3.26
Device GPU OpenCL 2.0 / Vulkan 1.1 Vulkan 1.2+

Vulkan support is available on most Android devices shipping since 2019. API 26 is the floor for AudioRecord features used here.

Step 1 — Replace the WAV Round-Trip With a Direct JNI Push

Here is the minimal setup to get this working. Most teams treat whisper.cpp as a batch processor: record audio → write WAV → pass path to JNI → wait → read string. That pipeline has a filesystem round-trip baked in at every utterance. Drop it entirely.

// Kotlin — AudioRecord capture loop
val bufferSize = AudioRecord.getMinBufferSize(16000, CHANNEL_IN_MONO, ENCODING_PCM_16BIT)
val recorder = AudioRecord(MIC, 16000, CHANNEL_IN_MONO, ENCODING_PCM_16BIT, bufferSize)

val shortBuffer = ShortArray(bufferSize / 2)
recorder.startRecording()

while (isCapturing) {
    val read = recorder.read(shortBuffer, 0, shortBuffer.size)
    if (read > 0) {
        // S16→float32 conversion happens in native code — zero Kotlin allocation
        WhisperBridge.pushSamples(shortBuffer, read)
    }
}
Enter fullscreen mode Exit fullscreen mode 
// C++ — JNI receive, convert S16→float32, write to ring buffer
extern "C" JNIEXPORT void JNICALL
Java_com_example_WhisperBridge_pushSamples(
    JNIEnv* env, jobject, jshortArray samples, jint count) {

    jshort* raw = env->GetShortArrayElements(samples, nullptr);

    std::vector<float> pcm(count);
    for (int i = 0; i < count; i++) {
        pcm[i] = raw[i] / 32768.0f;
    }
    env->ReleaseShortArrayElements(samples, raw, JNI_ABORT);

    ring_buffer_write(g_ring, pcm.data(), count);
}
Enter fullscreen mode Exit fullscreen mode

No file I/O. No intermediate WAV header. This one change cuts latency 40–60% with zero model changes required.

ring_buffer_write and friends represent a thread-safe lock-free SPSC circular buffer. In production use moodycamel's readerwriterqueue or a custom fixed-size atomic queue.

Step 2 — Enable the GGML Vulkan Backend

The docs do not emphasize this enough, but choosing the wrong GPU backend costs you significant throughput. Use Vulkan if your minimum API is 24+ and your targets are post-2019 devices. Use CLBlast only for older Mali or Adreno GPUs where Vulkan drivers are absent or unstable. Never enable both — GGML selects one at init time and the extra binary size buys you nothing.

cmake -DWHISPER_VULKAN=ON \
      -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
      -DANDROID_ABI=arm64-v8a \
      -DANDROID_PLATFORM=android-26 \
      ..
Enter fullscreen mode Exit fullscreen mode 
whisper_context_params cparams = whisper_context_default_params();
cparams.use_gpu = true;  // GGML selects best available backend
ctx = whisper_init_from_file_with_params(model_path, cparams);
Enter fullscreen mode Exit fullscreen mode

Step 3 — Set Window and Stride for Your Use Case

Here is the gotcha that will save you hours: get window and stride wrong and you will see repeated or dropped words regardless of model quality.

const int WINDOW_SAMPLES = 16000 * 8;  // 8s window
const int STRIDE_SAMPLES = 16000 * 2;  // 2s stride

while (running) {
    if (ring_buffer_available(g_ring) >= WINDOW_SAMPLES) {
        float window[WINDOW_SAMPLES];
        ring_buffer_peek(g_ring, window, WINDOW_SAMPLES);

        whisper_full(ctx, wparams, window, WINDOW_SAMPLES);
        ring_buffer_consume(g_ring, STRIDE_SAMPLES);

        emit_transcript(whisper_full_get_segment_text(ctx, 0));
    } else {
        std::this_thread::sleep_for(std::chrono::milliseconds(20));
    }
}
Enter fullscreen mode Exit fullscreen mode

A 2-second stride on an 8-second window gives each inference run 6 seconds of shared context with the previous one — that meaningfully improves word boundary accuracy at segment edges. Tune by task:

  • Command recognition: 3–4s window, 1s stride
  • Continuous transcription: 8–10s window, 2–3s stride

Gotchas

Benchmark on your actual target hardware. On a Pixel 6a (Tensor G1), the Vulkan path hits 80–150ms at P50. A Snapdragon 8 Gen 2 runs 20–30% faster. A Snapdragon 695 lands at 150–250ms. The delta is not trivial — lock in your build configuration only after profiling on devices representative of your real users.

Do not enable both Vulkan and CLBlast. GGML picks one at init; the second backend inflates binary size with zero runtime benefit.

Do the S16→float32 conversion in native code. Doing it in Kotlin allocates per-buffer. Over a continuous capture session that GC pressure is measurable.

The ring buffer capacity matters. Size it for ~30 seconds of float32 samples at 16kHz. Too small and your write thread blocks; too large and you carry stale audio into inference windows.

Results

Approach Latency (P50) Real-Time Factor
File-based batch (WAV round-trip) 800–1200ms ~1.2x
Direct JNI, CPU only 300–500ms ~0.6x
Direct JNI + Vulkan GPU encoder 80–150ms ~0.25x
Direct JNI + ring buffer overlap 60–100ms perceived <0.3x effective

Benchmarked on Pixel 6a, Android API 33, whisper-base.en Q5_K quantized (~57MB).

Conclusion

The WAV round-trip is the easiest win in any on-device ASR pipeline — implement the direct JNI push pattern and you cut latency without touching the model. Layer on GGML Vulkan for encoder acceleration, then tune your ring buffer window and stride to match your use case. That combination gets you to sub-100ms perceived latency on hardware your users actually own.

Top comments (0)