Transcribing a video call on my Mac in under a second with MLX Whisper

I'm building Murmury, an open-source Mac app that listens to the other side of a video call, notices when someone asks you something, and shows a short answer from your own documents. Last time I wrote about hiding its window from screen sharing. This post is about the part before that: turning the call's audio into text, on the Mac, fast enough to be useful while the person is still waiting for your answer.

No audio leaves the machine and nothing is saved. Here's how it works and what I measured.

The setup in one paragraph

A small Swift helper called audiotap reads the Mac's audio output with ScreenCaptureKit (so no bot joins the call) and writes raw float32 samples, mono, 16 kHz, to stdout. Python reads that pipe, cuts it into utterances, and sends each one to Whisper large-v3-turbo running on the Apple GPU through mlx-whisper. That's it: a pipe, a loop and a model.

The transcription call itself is short:

import mlx_whisper

MODEL = "mlx-community/whisper-large-v3-turbo"

def transcribe(audio, vocabulary=""):
    result = mlx_whisper.transcribe(audio, path_or_hf_repo=MODEL,
                                    condition_on_previous_text=False,
                                    initial_prompt=vocabulary or None)
    return result["text"].strip()

audio is a numpy float32 array at 16 kHz, which is exactly what the Swift helper produces, so there's no file and no ffmpeg in between. condition_on_previous_text=False because every utterance is transcribed on its own.

What I measured

On a MacBook with an Apple M5 and 24 GB of memory, macOS 26.6.2, mlx-whisper 0.4.3, mlx 0.32.3. To get repeatable input I made the clips with the macOS say command (voice Samantha, written straight at 16 kHz), then ran each one 5 times after the model was loaded.

Audio	Median time to text	Faster than real time
1.7 s ("What was revenue in the third quarter?")	0.66 s	about 3x
4.1 s	0.69 s	about 6x
10.1 s	0.80 s	about 13x
20.8 s	0.97 s	about 21x

All four came back with every word right; only the punctuation moved a little ("go to market" became "go-to-market", one long sentence was split into questions). Loading the model the first time took 1.13 s, and the Python process peaked at 1.73 GB of memory.

Be fair to these numbers: a synthetic voice in clean audio is the easy case. A real call has accents, two people talking at once and a laptop mic in a kitchen. The speed numbers hold, the accuracy will be lower.

The surprise: short clips aren't much faster

Look at the table again. Going from 1.7 s to 20.8 s of audio (12 times more) only took the time from 0.66 s to 0.97 s.

The reason is in how Whisper works. It always reads audio in 30-second windows: shorter audio is padded with silence up to 30 seconds before the encoder sees it. You can see it in mlx-whisper's audio.py:

CHUNK_LENGTH = 30
N_SAMPLES = CHUNK_LENGTH * SAMPLE_RATE  # 480000 samples in a 30-second chunk

So there's a floor of about 0.65 s on this Mac for any clip, and the extra cost for longer audio is mostly the decoder writing more words. For a live app this changes the strategy: cutting audio into tiny pieces to get text sooner doesn't help much, because every piece pays the floor.

Cutting the call into utterances

To know when someone has finished a sentence, I don't use a voice activity model. It's plain energy: split the stream into 30 ms frames, call a frame "loud" if its RMS is over a threshold, and end the utterance after enough quiet frames in a row.

FRAME = 480  # 30 ms at 16 kHz

loud = float(np.sqrt(np.mean(frame * frame))) > self.threshold  # 0.008
if speech or loud:
    if not speech:
        started = time.perf_counter()
    speech.append(frame)
    quiet = 0 if loud else quiet + 1
if speech and quiet >= self.silence_frames:  # 400 ms of quiet
    ended = True

It works because the input is already clean: ScreenCaptureKit gives me the call app's digital audio, not a microphone in a room. Anything under 0.6 s is dropped, which gets rid of most coughs and "mm-hm".

Transcribing while they're still talking

Waiting for the person to stop and then transcribing would add 400 ms of silence detection plus about 0.7 s of transcription before anything happens. So while an utterance grows, the audio so far is transcribed in the background about every 0.6 s, on a single worker (one model, one thread):

if not ended and speech and job is None and total >= next_partial:
    audio = np.concatenate(speech)[:total]
    job, job_covers = self.pool.submit(transcribe, audio, self.vocabulary), total
    next_partial = total + self.partial_every

When the speaker stops, if the last partial already covers all but 0.2 s of the speech, it becomes the final text right away, with no extra transcription:

text, covers = last_partial
if text and total - covers <= self.fast_margin:
    yield Event("final", self.speaker, text, time.perf_counter(), fast=True, started=started)
else:
    text = self.pool.submit(transcribe, audio, self.vocabulary).result()

I timed the partials on the 10 s clip, cutting it every 0.6 s like the app does. Each one took between 0.64 s and 0.90 s, so with one worker a new partial shows up roughly every 0.7 s, a bit slower than the 0.6 s I ask for. That's fine: the point is that when the person stops, the text is usually already there.

Names Whisper doesn't know

Product and company names are where transcription goes wrong, and in a business call they're the words that matter. Whisper's initial_prompt helps a lot. Murmury has a field where you list the names before the call, and they go in as the prompt. A test with an invented company name:

no vocabulary:   Did you show the murmury numbers to Kostrova before the board meeting?
with vocabulary: Did you show the Murmury numbers to Kestrova before the board meeting?

Same audio, same model, the only difference is initial_prompt="Murmury, Kestrova".

Keeping numba and scipy out of the app

mlx-whisper's transcribe.py starts with from .timing import add_word_timestamps, and timing.py imports numba and scipy. They're only used for word-level timestamps, which Murmury never asks for. In my environment numba, llvmlite and scipy take 17 MB, 129 MB and 83 MB on disk, and I didn't want them in the app. Leaving them out of the PyInstaller build breaks the import, so before the first import I put a stand-in module in sys.modules:

def _skip_word_timing():
    if "mlx_whisper.timing" in sys.modules:
        return
    import types
    stub = types.ModuleType("mlx_whisper.timing")

    def add_word_timestamps(*args, **kwargs):
        raise RuntimeError("word timestamps are not included in Murmury")

    stub.add_word_timestamps = add_word_timestamps
    sys.modules["mlx_whisper.timing"] = stub

Measured three times each: import mlx_whisper takes 0.45 s and loads numba and scipy without the stub, 0.11 s and loads neither with it. With the stub in place the PyInstaller spec can exclude numba, llvmlite and scipy, and if anything ever calls word_timestamps=True it gets a clear error.

Warm up before the call

The first transcription loads the model, so Murmury runs one second of silence through Whisper before the call starts:

def warm_up():
    transcribe(np.zeros(SAMPLE_RATE, dtype=np.float32))

That way the first real question doesn't wait for the 1.1 s model load.

Try it

Murmury is free and GPL-3.0: github.com/GianlucaBave/murmury. The transcription code is all in copilot/listen.py (about 220 lines). If you don't want to set up a call, docs/demo/ has a sample document and a script that asks three questions through your speakers.

I'd love numbers from other Macs. If you have an M1 or M2, how long does a short sentence take for you?