Evaluating Voice Cloning Models on CPU: A Practical Benchmark of Pocket TTS, Kokoro, Audio8, and XTTS-v2

If you ship TTS or voice cloning, you eventually need a straight answer: which model sounds natural, stays intelligible, actually clones the reference speaker, and still runs at a usable speed on CPU. This post walks through a full objective evaluation of four models on a CPU-only box, how the dataset and metrics were chosen, what broke along the way, what the numbers mean, and how you can clone the project and extend it with Neo.

Why This Evaluation Exists

Vendor demos and single-speaker samples are easy to game. A fair comparison needs many speakers, gender balance and accent diversity, held-out text that never appears in the reference clip, metrics that separate cloning fidelity, naturalness, intelligibility, and latency, and an honest split between true zero-shot cloners and fixed-voice TTS.

This case study was produced end to end by Neo, an autonomous AI engineering agent you can run from VS Code or Cursor. From a high-level goal, Neo planned the harness, integrated the models, built the data pipeline, ran the sweep, fixed a real data-quality bug, regenerated reports, and pushed the repo.

Models Under Test

That split matters. Only Audio8 and XTTS do genuine zero-shot cloning here. Pocket TTS and Kokoro are still useful as CPU quality and speed baselines, but their speaker-similarity scores are not cloning-fidelity evidence.

Metrics

All scoring is automatic. No MOS listening panel in this run.

Speaker similarity: WavLM-Large x-vector embeddings, cosine similarity, both streams at 16 kHz. Higher is closer to the reference speaker. In practice this metric saturated near 1.0 for all four models, so it is a sanity check more than a ranking signal.

Naturalness (UTMOSv2): Predicted MOS on a 1-5 scale. Higher is more natural. This is where fixed-voice Kokoro pulled ahead of the true cloners.

Intelligibility (WER / CER): faster-whisper small (CTranslate2, CPU int8) → normalized text → jiwer. Lower is better. Hypotheses were spot-checked against real generated WAVs so WER was not a silent fallback.

Latency: Wall-clock seconds and RTF (wall / audio duration). Peak memory for XTTS is not trustworthy in the parent process because XTTS runs in an isolated subprocess (venv_xtts).

Dataset and Bias Control

Early Setup: LibriSpeech

The first passes used LibriSpeech test-clean (small speaker counts, few sentences). That is fine for wiring scorers and wrappers. It is not enough if someone will argue the benchmark is biased or too narrow.

Final Design: VCTK Sweep

Final numbers come from VCTK (British English multi-speaker corpus; HF mirror badayvedat/VCTK):

Speakers: 22, 11 female and 11 male, for gender balance.

Accents: 11 represented: English, Scottish, Irish, Northern Irish, Indian, Welsh, Canadian, American, Australian, South African, and New Zealand, to avoid one-accent conclusions.

Test sentences per speaker: 12 unique held-out texts for enough text diversity for WER.

Manifest rows: 264, one row per speaker x sentence.

Result rows: 1056, 4 models x 264.

Reference clip: ~10-13 s per speaker, enough signal for cloning and conditioning.

Leakage: 0. Test text never appears in the reference transcript and test utt_ids are disjoint from the reference.

Different corpus from LibriSpeech also reduces "we only ever tested American read speech" risk.

The Data Bug Neo Found and Fixed

VCTK reuses elicitation sentences across recordings. The first manifest builder took the first N non-reference utterances without deduplicating by transcript text. Result: only 6 unique texts per speaker out of 12 rows, distinct audio files but repeated text pairs, and WER that looked artificially strong on short repeated lines.

Neo diagnosed that during score validation (not during the first green run), then fixed it:

1. Deduplicate candidates with a seen_test_text set in scripts/build_vctk_manifest.py
2. Expand the per-speaker candidate pool from 30 to 60 so 12 unique texts still fit after reference selection
3. Enforce separation on both utt_id and text
4. Rebuild the manifest, re-run all 1056 evaluations, regenerate aggregates and the report

Post-fix check: 12 unique test texts per speaker, 0/264 leakage rows, all 1056 rows status=OK.

That is the main non-trivial failure mode in this project. Integration pain (XTTS transformers pin, separate venv, CPU device for UTMOSv2, torchvision CPU wheel) was real but expected plumbing. The sentence-dedupe bug is the one that would have quietly misled an expert reader.

Harness Architecture

manifest_vctk.jsonl
        │
        ▼
   run_eval.py
   ├── wrappers/   pocket_tts, kokoro, audio8, xtts (subprocess)
   ├── energy gate (reject silent / empty audio)
   └── scorers/    speaker_sim, naturalness (UTMOS), wer, timing
        │
        ▼
output/results.csv → aggregated_results.csv
                   → zero_shot_comparison.*
                   → report.md

XTTS lives in venv_xtts and is invoked through xtts_subprocess_entry.py so Coqui's transformers pin does not break the main env. Audio8 uses ONNX registration against a short reference clip plus transcript. Kokoro and Pocket TTS take documented fixed voices when true cloning is unavailable.

Hardware for the published run: CPU only, 8 cores, ~62 GB RAM, no GPU.

Results

All Four Models (264 Unique Sentences Each)

All 1056 rows passed the energy gate with non-null metrics.

Highlights:

Kokoro-82M leads on naturalness (UTMOS 3.690) and is by far the fastest (RTF 0.124, ~0.51 s wall-clock). Audio8 leads on intelligibility (WER 0.0037, CER 0.0008) among all models. XTTS is the slowest (RTF 5.311, ~21.26 s wall-clock) and has the highest WER (0.0091). Speaker similarity is essentially tied across all models (0.9856-0.9882), so it does not discriminate well at this scale.

Zero-Shot Only: Audio8 vs XTTS (528 Rows)

Restricted to the two true zero-shot cloners:

Per-speaker UTMOS: Audio8 ahead on 13/22 speakers, XTTS on 9/22. Naturalness is not a clean sweep either way. Intelligibility and speed favor Audio8 on this box.

What Changed After the Sentence Fix

With duplicated elicitation text, WER looked too good (for example Kokoro ~0.0017). After unique texts, Kokoro WER moved to ~0.0051 and XTTS to ~0.0091. The ranking story stayed directionally similar, but the absolute WER numbers became honest for a multi-sentence claim.

LibriSpeech vs VCTK also mattered for naturalness among cloners: on the smaller LibriSpeech set Audio8 led XTTS by a larger UTMOS margin; on VCTK they are effectively tied. That shift is a reason not to overfit conclusions to one small speaker pool.

How to Read the Outcome

Speaker similarity: Do not pick a winner from WavLM cosine here. Everything sits near 0.99. Use it to confirm a real same-speaker condition, not to rank models.

Naturalness: Kokoro is the clear leader (UTMOS ~3.69) if you can accept a fixed voice. Among true cloners, Audio8 and XTTS are a coin flip on average MOS.

Intelligibility: Audio8 posts the best WER overall (~0.0037), including among true cloners. XTTS is still strong in absolute terms (~0.009) but clearly behind on this ASR-based measure.

Latency on CPU: Kokoro is comfortably real-time (RTF ~0.12). Pocket TTS is also real-time (~0.24). Audio8 is about 1.4x real-time. XTTS is about 5.3x real-time (~21 s wall per utterance on this machine). If your product is interactive CPU TTS, XTTS is a hard sell without a GPU path.

Practical Recommendations

Best sounding fixed voice and speed on CPU: Kokoro-82M. Highest UTMOS and lowest RTF.

True zero-shot cloning with intelligibility and CPU speed: Audio8. Best WER and much faster than XTTS.

True zero-shot when naturalness is the only lens: Audio8 or XTTS. The UTMOS gap is noise-level.

Who clones identity best: Speaker similarity saturated near 1.0 for all models. This metric does not discriminate well at this scale.

Caveats: objective-only (no human MOS), Whisper small for WER, Pocket/Kokoro not true cloners in this run, XTTS peak memory not measured in-process, no paired bootstrap or significance test on the deltas.

Setup and Installation

This harness is CPU-only and uses two separate virtual environments:

• ./venv: main harness environment (torch, torchaudio, onnxruntime, librosa, soundfile, numpy, pandas, matplotlib, jiwer, datasets, faster-whisper, UTMOS, WavLM)
• ./venv_xtts: isolated environment for XTTS-v2 (runs in a separate subprocess so its memory footprint is measured independently)

# From the project root
cd /home/azureuser/clone_eval

# Create the main venv
python3 -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt

# Create the XTTS venv
python3 -m venv ./venv_xtts
source ./venv_xtts/bin/activate
pip install TTS  # Coqui XTTS-v2

Model weights are downloaded on first run into models/ (Audio8_TTS ~1 GB, kokoro ONNX ~311 MB, voices bin ~27 MB). These are not committed to the repo.

How to Run the Harness

Run the full evaluation sweep with the VCTK manifest:

./venv/bin/python run_eval.py --manifest data/manifest_vctk.jsonl --results output/results.csv

This produces:

• output/results.csv: per-model x per-speaker x per-sentence raw metrics (1056 rows)
• output/aggregated_results.csv: averaged per-model metrics
• output/zero_shot_comparison.csv + output/zero_shot_comparison_summary.txt: Audio8 vs XTTS zero-shot comparison
• output/report.md: the full written report
• output/audio/: generated sample WAVs (transparency artifact)

Regenerating the Manifest and Reports

Regenerate the VCTK manifest:

./venv/bin/python scripts/build_vctk_manifest.py
# writes data/manifest_vctk.jsonl (264 rows)

Regenerate the zero-shot comparison:

./venv/bin/python scripts/zero_shot_comparison.py
# writes output/zero_shot_comparison.csv + output/zero_shot_comparison_summary.txt

Project Layout

clone_eval/
├── run_eval.py                     # Harness driver
├── requirements.txt                # Base CPU dependencies
├── wrappers/                       # Per-model inference wrappers
│   ├── pocket_tts_wrapper.py
│   ├── kokoro_wrapper.py
│   ├── audio8_wrapper.py
│   ├── xtts_wrapper.py
│   └── xtts_subprocess_entry.py    # XTTS runs in a separate subprocess
├── scorers/                        # Metric scorers
│   ├── speaker_sim.py              # WavLM cosine similarity
│   ├── naturalness.py              # UTMOS
│   ├── wer.py                      # faster-whisper + jiwer
│   └── timing.py                   # RTF / wall-clock
├── scripts/                        # Manifest builders + comparison
│   ├── build_manifest.py
│   ├── build_vctk_manifest.py
│   ├── extract_vctk_metadata.py
│   ├── zero_shot_comparison.py
│   └── ...
├── test_suite/                     # Verification scripts
├── data/
│   ├── manifest.jsonl              # LibriSpeech manifest
│   └── manifest_vctk.jsonl         # VCTK manifest (264 rows)
├── output/
│   ├── results.csv                 # 1056 raw rows
│   ├── aggregated_results.csv
│   ├── zero_shot_comparison.csv
│   ├── zero_shot_comparison_summary.txt
│   └── report.md                   # Full report
└── plans/                          # Planning docs

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a CPU-only, fully automated evaluation harness that benchmarks four TTS and voice-cloning models across a statistically defensible, multi-speaker, multi-accent dataset sweep. NEO planned and produced the files in this repository: the harness driver, four model wrappers including an isolated XTTS subprocess, four metric scorers covering speaker similarity, naturalness, intelligibility, and timing, manifest builders with deduplication and leakage controls, comparison scripts, and the full report generation pipeline.

The result is a fully working evaluation harness that produces 1056 rows of objective metrics across 22 speakers, 11 accents, and 4 models, with a data bug caught and fixed mid-run and all results verified against real generated WAVs.

How You Can Use This With NEO

Run it before choosing a TTS model for a product.
Point the harness at your own reference clips and get objective CPU latency, naturalness, and intelligibility numbers across many speakers instead of trusting vendor demos or single-speaker samples. The VCTK manifest covers 22 speakers and 11 accents out of the box.

Validate whether a zero-shot cloner actually preserves speaker identity at scale.
Single cherry-picked demos are easy to game. Running the harness across 22 speakers with 12 unique held-out sentences per speaker gives you a real picture of how consistently a model clones across accent diversity and text variation.

Use the per-speaker CSV to find where a model breaks down.
Every metric for every speaker and every sentence is in output/results.csv. Loading it in pandas and grouping by accent or speaker reveals whether a model fails on specific speaker types before it reaches production.

Use the data quality pattern in your own eval pipelines.
The sentence deduplication fix — catching repeated elicitation text across a speaker's rows — is the kind of silent bug that makes WER look better than it is. The seen_test_text logic in scripts/build_vctk_manifest.py is a direct template for applying the same check to any manifest-based eval you run.

Final Notes

Kokoro leads on naturalness and speed but does not clone. Among the true zero-shot cloners, Audio8 beats XTTS on intelligibility and CPU latency while naturalness is too close to call. The results only became trustworthy after a sentence deduplication fix and a full re-run. Run the harness on your own prompts before making a product decision.

The code is at https://github.com/gauravvij/voice-clone-eval
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code