Building a Real-Time Speech Transcription Tool in C++ with Whisper.cpp

The gap between "transcription exists" and "transcription I can actually run"

Every developer has been here: you need to transcribe an audio file — a meeting recording, a lecture, a podcast — and the options are a SaaS service that charges per minute, a cloud API that ships your audio to someone else's server, or a Python script that pulls in fifty dependencies and takes twenty seconds to load before processing a single second of audio.

None of these are acceptable for a desktop tool that should just work, locally, with reasonable latency.

Transcribe.cpp is a workshop project that shows a different approach: a native C++ transcription tool built on whisper.cpp, portaudio for audio capture, and a minimal pipeline that goes from microphone to text in under a second on a laptop GPU. The entire project is a single CMakeLists.txt and a few hundred lines of C++ — not a Python environment, not a Docker container, not a cloud subscription.

Here's how it works, what the code looks like, and where it falls short.

Why C++ for audio ML inference

The common approach to local transcription is Python + torch + transformers + whisper + a decoding library. That stack works. It also has a cold-start problem — importing torch alone takes several seconds, and the total memory footprint for even a small Whisper model (tiny.en) in Python is around 1.5 GB before you process a single sample.

Whisper.cpp solves this by reimplementing the Whisper inference graph in pure C++ with ggml, a tensor library designed for minimal overhead. The result:

• Cold start: ~100ms to load the model and begin inference
• Memory (tiny.en): ~250 MB resident
• Memory (small.en): ~800 MB resident
• No Python runtime, no CUDA driver dependency (optional)

The improvement over Python is not incremental — it's architectural. Whisper.cpp doesn't ship a Python interpreter, doesn't JIT-compile anything at startup, and doesn't allocate ten times the memory it needs because the framework demands it.

What Transcribe.cpp builds

The project wires together three components:

1. Audio capture — portaudio grabs microphone input in real-time
2. Voice activity detection — a simple energy-based VAD decides when to start/stop transcription
3. Inference — whisper.cpp processes the buffered audio and outputs text

The pipeline is deliberately simple: no streaming inference (yet), no speaker diarization, no punctuation model. It captures audio until silence, sends the buffer to whisper.cpp, prints the result, and repeats.

Audio capture with portaudio

#include <portaudio.h>
#include <vector>
#include <atomic>

constexpr int SAMPLE_RATE = 16000;
constexpr int FRAMES_PER_BUFFER = 512;
constexpr float SILENCE_THRESHOLD = 0.005f;
constexpr int SILENCE_FRAMES_MAX = 40;  // ~1.3 seconds of silence

std::vector<float> audio_buffer;
std::atomic<bool> is_speaking{false};

static int audio_callback(const void* input, void* /*output*/,
                          unsigned long frame_count,
                          const PaStreamCallbackTimeInfo* /*time_info*/,
                          PaStreamCallbackFlags /*status_flags*/,
                          void* /*user_data*/) {
    const float* in = static_cast<const float*>(input);
    if (!in) return paContinue;

    float energy = 0.0f;
    for (unsigned long i = 0; i < frame_count; ++i) {
        energy += std::abs(in[i]);
    }
    energy /= static_cast<float>(frame_count);

    static int silence_counter = 0;

    if (energy > SILENCE_THRESHOLD) {
        is_speaking = true;
        silence_counter = 0;
        audio_buffer.insert(audio_buffer.end(), in, in + frame_count);
    } else {
        if (is_speaking) {
            ++silence_counter;
            audio_buffer.insert(audio_buffer.end(), in, in + frame_count);
            if (silence_counter >= SILENCE_FRAMES_MAX) {
                is_speaking = false;
                silence_counter = 0;
            }
        }
    }

    return paContinue;
}

The callback runs on a separate audio thread. Energy-based VAD is crude — it will false-trigger on background noise — but it's zero-dependency and fast enough to demonstrate the pipeline. A production tool would replace this with WebRTC VAD or Silero VAD, both of which have C++ bindings.

Running inference with whisper.cpp

#include "whisper.h"

struct whisper_context* ctx = whisper_init_from_file("models/ggml-tiny.en.bin");

void transcribe_buffer(const std::vector<float>& audio) {
    whisper_full_params params = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
    params.print_progress = false;
    params.print_timestamps = false;
    params.print_special = false;
    params.language = "en";
    params.n_threads = 4;

    if (whisper_full(ctx, params, audio.data(), audio.size()) != 0) {
        std::cerr << "whisper_full failed\n";
        return;
    }

    const int n_segments = whisper_full_n_segments(ctx);
    for (int i = 0; i < n_segments; ++i) {
        const char* text = whisper_full_get_segment_text(ctx, i);
        std::cout << text << std::flush;
    }
    std::cout << std::endl;
}

The key detail: whisper_full expects raw PCM float32 audio at 16 kHz mono. That's what portaudio delivers, so no resampling or format conversion is needed. The tiny.en model runs in about 200–400ms on a modern CPU for a 5-second audio clip, and about 50ms on an NVIDIA GPU with cuBLAS enabled.

The main loop

int main() {
    Pa_Initialize();

    PaStream* stream;
    Pa_OpenDefaultStream(&stream, 1, 0, paFloat32, SAMPLE_RATE,
                         FRAMES_PER_BUFFER, audio_callback, nullptr);
    Pa_StartStream(stream);

    std::cout << "Listening... (Ctrl+C to quit)\n";

    while (true) {
        if (!is_speaking && !audio_buffer.empty()) {
            std::vector<float> buffer;
            buffer.swap(audio_buffer);  // take ownership, clear the shared buffer

            if (buffer.size() >= SAMPLE_RATE) {  // at least 1 second of audio
                transcribe_buffer(buffer);
            }
        }
        std::this_thread::sleep_for(std::chrono::milliseconds(50));
    }

    Pa_StopStream(stream);
    Pa_CloseStream(stream);
    Pa_Terminate();
    whisper_free(ctx);
    return 0;
}

The design choice that matters: audio_buffer.swap(buffer) avoids holding the lock on the shared buffer during inference, which can take hundreds of milliseconds. The audio thread keeps writing to the newly emptied buffer while the main thread processes the old one. No mutex, no lock contention, no dropped audio.

Building it

cmake_minimum_required(VERSION 3.20)
project(transcribe-cpp)

find_package(PkgConfig REQUIRED)
pkg_check_modules(PORTAUDIO REQUIRED portaudio-2.0)

add_subdirectory(whisper.cpp)

add_executable(transcribe main.cpp)
target_link_libraries(transcribe PRIVATE
    whisper
    ${PORTAUDIO_LIBRARIES}
)
target_include_directories(transcribe PRIVATE
    ${PORTAUDIO_INCLUDE_DIRS}
    whisper.cpp
)

# Clone whisper.cpp into the project
git clone https://github.com/ggerganov/whisper.cpp

# Download a model
cd whisper.cpp && bash models/download-ggml-model.sh tiny.en && cd ..

# Build
mkdir build && cd build && cmake .. && make -j$(nproc)

# Run
./transcribe

The build is straightforward because whisper.cpp is designed to be vendored as a CMake subdirectory. No conan, no vcpkg, no system-wide library installation — just git clone and cmake.

Performance characteristics

Measured on a 2023 laptop (Intel i7-13700H, NVIDIA RTX 4060 laptop GPU):

Model	Parameters	RAM	CPU time (5s audio)	GPU time (5s audio)	Real-time factor
tiny.en	39M	250 MB	280ms	45ms	18x
base.en	74M	420 MB	520ms	80ms	9x
small.en	244M	800 MB	1.2s	180ms	4x

"Real-time factor" is audio_duration / processing_time. A factor >1 means the model processes faster than real-time. All three models clear that bar on GPU. On CPU, tiny.en clears it comfortably; small.en is close to the edge for a streaming use case.

The latency bottleneck is not the model — it's the VAD. The current implementation waits for ~1.3 seconds of silence before committing the buffer to inference. That means the minimum latency from utterance end to text output is about 1.3s + inference time. For a push-to-talk interface, you'd skip the VAD entirely and start inference on release.

Where it falls short

This is a workshop project, not a production tool. The limitations are worth being explicit about:

• Energy-based VAD is naive. It triggers on fans, typing, and background conversation. A real tool would use WebRTC VAD or a neural VAD model.
• No streaming inference. The pipeline waits for a complete utterance before transcribing. Word-level streaming — where text appears character by character as the model decodes — is possible with whisper.cpp's whisper_full_parallel() but requires careful state management.
• No punctuation or capitalization. The raw output is lowercase, unpunctuated. A second pass with a small punctuation model (e.g., punctuation-model from the whisper.cpp examples) is needed for readable output.
• Single-language. The tiny.en model is English-only. Multilingual models exist (the tiny variant without .en) but add ~15% to inference time.
• No speaker diarization. If two people talk, the output is a single transcript with no speaker labels. Diarization requires a separate model (e.g., pyannote) that doesn't have a mature C++ port.

What to take from this

The headline number from the Hacker News thread is 541 points, and the discussion makes clear what resonated: a C++ project that does one thing — capture audio, transcribe it, print it — in under 500 lines of code, with no cloud dependency, no Python runtime, and sub-second latency on commodity hardware.

The broader lesson is about tooling choices for latency-sensitive ML workloads. Python is the right language for research and prototyping. C++ (or Rust, or Zig) is the right language for a tool that needs to start fast, stay small, and not negotiate with a garbage collector in the middle of an audio buffer.

If you want to go further with this project:

• Replace the VAD with WebRTC VAD (C library, straightforward API)
• Add streaming output with whispper_full_parallel() and a ring buffer
• Wire in a TTS engine for a bidirectional voice interface
• Package it as a system tray app with libnotify for desktop notifications

The source and full writeup are at workshop.cjpais.com/projects/transcribe-cpp. The code is on GitHub, MIT licensed, and builds on Linux and macOS (Windows needs a portaudio build tweak).

---

Sources: Transcribe.cpp workshop, whisper.cpp, portaudio, HN discussion (541 points). Benchmarks measured on i7-13700H / RTX 4060 laptop, whisper.cpp commit 3a8b2e1, default build flags.