Scoring Local AI Music with a Number — A Whisper-Based QA Pipeline that Measures Lyric-Match Rate on an M1 Mac

📝 This is an English write-up of a piece originally published in Japanese on Zenn (canonical). All figures and numbers are real measurements from my own runs.

The takes pile up, but my listening time doesn't

Once I started running AI music generation locally, the first wall I hit wasn't quality itself. It was volume.

My generation engine is ACE-Step 1.5 — a music-generation model (with a 0.6B LM) that runs as a local API on Apple Silicon via MLX. Producing English-vocal AI songs, the number of takes needed to finalize a single song piles up fast: v1 / v2 by parameter and seed, times two variants each. Four takes per song is normal. This batch came to 32 takes total.

Here's the problem. Listen to each take and yes, "the melody is good." The backing sounds fine too. But a certain fraction of takes don't sing the lyrics correctly — vowels smear into different words, word endings drop, whole phrases get skipped. If you get carried away by a good melody, you miss these.

Hand-auditing all 32 takes every batch burns a whole day just on finalization. Your ears tire. By the third song, judgment drifts. Betting quality assurance entirely on human concentration was broken by design.

What I wanted was a cheap pre-filter that mechanically thins the candidates before I listen.

The idea: whether it sings can be measured by transcription

Song quality has many axes — pitch, rhythm, audio fidelity, expression. Quantifying all of them at once is hard. But the single thing troubling me most was "does it sing the lyrics correctly?" That one axis turns out to be surprisingly easy to put a number on.

The idea is simple. Transcribe the generated audio, then compute its match rate against the intended lyric text. If it sings well, the transcription lands close to the original lyrics; if it breaks down, they diverge. Collapse that into a single number — the match rate — and I can rank all 32 takes mechanically.

For transcription I chose mlx-whisper: an Apple-Silicon-native Whisper implementation that runs fully locally, no cloud upload. No API bills, no upload waits. Generation is local via ACE-Step; the scoring gate's entry point is local via mlx-whisper. The entire quality gate closes on my own machine. For anyone running a lot of AI music solo, that's decisive — free, and repeatable as many times as I like.

Of course Whisper's transcription isn't perfect. But at this stage I don't need a perfect transcript — I need enough reproducibility to compare takes against each other. Transcribe every take with the same model under the same conditions, and it holds up for side-by-side comparison.

The pipeline: separate generation from scoring

Here's the QA flow I built.

Four stages:

1. Generate — produce takes with ACE-Step 1.5. v1 / v2 × 2 variants per song.
2. Transcribe — run each output through mlx-whisper to text.
3. Compute lyric-match rate — compare the transcript against the intended lyrics. This is the automated cutoff gate.
4. Independent rubric scoring — score the candidates that pass the cutoff against a rubric.

One thing I deliberately insisted on: do not let the generating model score its own output.

Ask a generation model "how was that song?" and its evaluation gets pulled toward its own convenience — it grades itself softly. That's the classic "the maker also being the judge" situation, and it can't be trusted as a quality gate. So scoring runs as a separate process, against an independent rubric. Physically decoupling generation from evaluation is the heart of this gate.

Finally, only the low-scoring takes go to a human ear for the final call. No full audit. Human concentration is concentrated only on what the machine flagged as suspect. Total listening dropped sharply.

Measured: narrowing 32 takes down to 8

Here are the actual numbers.

• Total takes generated: 32 (v1 / v2 × 2 variants per song)
• Best takes finalized through three quality gates (①lyric-match ②independent rubric ③human ear): 8
• Mean lyric-match rate of the 8 finals: 83.9%
• Mean independent rubric score of the 8 finals: 94.1 / 100
• Sent back for rework after the gates: 1 time
• Each song runs 100–120 seconds (measured with afinfo)

The funnel shows a wide entrance (32) and a narrow exit (8). Roughly a 4-takes-in, 1-out yield; the other three fall out because — good melody notwithstanding — the lyrics broke, or the rubric bar wasn't cleared. This "over-produce, then cut" premise is the reality of volume production. Which is exactly why the cutting step has to be automated.

I measured length with afinfo for a reason: the length a generation model reports doesn't necessarily match the actual playback length of the file. If you're going to talk about quality with numbers, read the length from the real file, not an estimate. Obvious — but fill that gap with a guess and the credibility of every other number erodes.

Retraction: a low match rate did NOT mean a bad song

Read this far and it looks like "cut off by match rate and you're done." I believed that myself for a while: low match rate = doesn't sing = bad song.

That was wrong.

There was a cluster whose match rate came out clearly low. Investigating, the common factor was in the content of the songs: songs that sing the characters themselves — like reciting the alphabet — rather than words. This type measured a match rate as low as 75.3%, distinctly below the 83.9% mean of the 8 finals.

But listen with your ears and those songs sang just fine. What had broken wasn't the song — it was Whisper's transcription.

The reason, on reflection, is obvious: Whisper is trained to transcribe natural-language speech. It's strong on contextual words and sentences. But utterances with thin linguistic context — singing "A, B, C…" one letter at a time — are a structural weakness. Transcription wobbles, and the match rate against the source text drops. So that 75.3% reflected the limit of the measuring instrument, not a failure of the song.

That was an important lesson. Automated scoring is a pre-filter, not the final verdict. Had I mechanically discarded every low-match take, I'd have killed genuinely good, well-sung takes as collateral damage of my instrument's quirks.

So I locked the operating rule: before discarding a low-score take, always confirm by ear. High scores you can trust (it matches because it sings). But a low score can mean either "doesn't sing" or "can't be measured," so I don't let the machine's verdict become an automatic death sentence. The correct use of a numeric gate is to reduce human listening, not to replace it.

Environment gotcha: grab Python as an arm64 native

One footing-level note. MLX-family tools assume Apple-Silicon-native (arm64) execution. Miss that and they either don't run, or run slow.

A common trap is grabbing an x86_64 Python under Rosetta. In environments where Homebrew, pyenv, and multiple Pythons coexist, the x86_64 python can get prioritized unintentionally. MLX assumes native execution, so in this state you don't get its real performance.

The fix is simple — confirm your Python is arm64 before you start:

python3 -c "import platform; print(platform.machine())"

If it returns arm64, you're good. If it prints x86_64, that's a Python through Rosetta, so switch environments before running MLX. When transcription is inexplicably slow or MLX misbehaves, type this one line before suspecting the model or the code. If the foundation is tilted, no amount of tuning on top pays off.

Takeaway: the number is a pre-filter; the ear is the final verdict

With a fully free, local toolkit (ACE-Step 1.5 to generate, mlx-whisper to transcribe), I built a quality gate that puts a number on "does it sing." I thinned 32 takes by machine, finalized 8, and cut total listening dramatically. Lessons:

• For volume QA, automate the pre-filter first. A design that hand-audits everything depends on concentration and breaks. Prepare a single mechanically-rankable number (here, lyric-match rate) up front.
• Separate scoring from generation. Don't let the generation model grade itself. An independent rubric in a separate process is what makes the gate trustworthy.
• The number is a pre-filter, not the final verdict. For cases where the instrument (Whisper) is structurally weak — like songs that sing letters — low scores appear. Always confirm a low score by ear before discarding.
• Verify the foundation first. If you use MLX, arm64-native Python is mandatory. Confirm platform.machine() is arm64 before you begin.

The value of automation isn't "removing humans" — it's "concentrating human judgment where judgment is actually needed." The numeric gate narrows the entrance to listening; the ear at the exit stays in human hands to the end. That division of labor is, for now, my working answer for keeping quality up while mass-producing local AI music.