Identifying Speakers by Voice Quality Alone — Labeling Unknown Audio Sources

📝 Originally published (in Japanese) at forge.workstyle.tech.

When you are growing an audio dataset, you will inevitably run into the "Who's voice is this again?" problem. The anchors (target speakers for voice conversion) in this app were no exception. As I collected more material, I found anchors that had lost their labels—they were just numbered, like spk7 through spk18.

Files had missing names, some were renamed halfway through, and the mapping between the raw source directories and the extracted anchors became disconnected. For various reasons, the result is the same: you end up with "voice data that exists, but whose identity is unknown."

If left alone, this kind of material tends to become "dead data"—ignored because the source is unclear. But the clue actually lies within the sound itself. Since we know the vocal characteristics, we can just match them by voice quality. This article is a record of how I re-labeled unknown anchors by relying solely on speaker embedding similarity and rebuilt the anchor_sources.json file.

The Idea: Matching by Voice, Not by Label

Here is the breakdown of what I wanted to achieve:

• I have a large amount of raw source material with labels (wav files where the speaker's name is in the filename).
• On the other hand, I have unlabeled anchors (spk7, etc.).
• I want to match the two using voice quality to restore the correct names to the anchors.

This is essentially classical speaker verification. Fortunately, this app already uses campplus speaker embeddings for its anchor definitions. By measuring the cosine similarity within the same embedding space, I can determine "which raw source file is most similar to this anchor."

The Source Side: Creating Robust Speaker Vectors

First, I create a "gallery" to act as the comparison target. I extract one speaker vector from each raw source file. However, taking just a single 4-second window is unstable; if the window hits silence, a breath, or a stutter, the resulting vector won't represent that person's voice accurately.

Instead, I extract embeddings from 12 distributed windows across the entire file, L2-normalize them, and then average them. I discard silent chunks based on their amplitude.

def _robust_embed(campplus, y16):
    win = int(WIN_S * SR16); guard = int(5 * SR16)
    hi = len(y16) - guard - win
    embs = []
    for p in np.linspace(guard, hi, N_WIN).astype(int):   # N_WIN=12 windows
        seg = y16[p:p + win]
        if float(np.max(np.abs(seg))) < 1e-3:             # Discard silence
            continue
        e = _embed(campplus, seg)
        embs.append(e / (np.linalg.norm(e) + 1e-9))       # Normalize each window
    v = np.mean(embs, axis=0)
    return v / (np.linalg.norm(v) + 1e-9)                 # Re-normalize the mean

The key is the sequence: "Normalize, then average, then re-normalize the average." If you average without normalizing first, the result is pulled too heavily toward windows that happen to have a large norm. By converting them to unit vectors before averaging, you create a robust speaker vector that treats each window equally.

The Anchor Side: Reusing Existing Embeddings

For the anchors being compared, there is no need to calculate new embeddings. Since this project already saves all anchor embeddings in anchor_embeddings.npz via precompute_anchors.py, I just need to load and normalize them.

d = np.load(os.path.join(DATA, "anchor_embeddings.npz"), allow_pickle=True)
names = [str(x) for x in d["names"]]
embs = d["embeddings"].astype(np.float64)
emap = {n: embs[i] / (np.linalg.norm(embs[i]) + 1e-9) for i, n in enumerate(names)}

By using the same vectors used for generation to perform the estimation, I maintain consistency: "Selection, definition, and estimation all using the same ruler."

Matching: Top 3 Candidates and Confidence Levels

Finally, I calculate the dot product (which equals cosine similarity since they are normalized) between the target anchor vector and the entire gallery, then look at the top results. If the gallery is stored as a matrix G, a single G @ v operation gives the similarity scores for all source files.

sims = G @ emap[t]
order = np.argsort(sims)[::-1][:3]
cand = "  ".join(f"{glabels[i]}={sims[i]:.3f}" for i in order)
conf = "○" if sims[order[0]] >= 0.55 else ("△" if sims[order[0]] >= 0.45 else "×")
print(f"{t}: {conf} {cand}")

The important thing is not to declare the result based on "only the top result." I list the top 3 candidates and use the best score to categorize the confidence into three levels:

• ○ (≥0.55): Same level as the single-speaker threshold. Can be considered certain.
• △ (0.45〜0.55): A candidate, but a gray zone where a human should decide by looking at the top 3.
• × (<0.45): No match in the gallery. Either a different person or the source material is missing.

This threshold of 0.55 is the same as homo-thresh used to identify single speakers during anchor selection. I standardized the criteria for "closeness that can be called the same speaker" across both selection and labeling.

Why Not Just Pick "Number 1"?

When people hear "automation," they often want to immediately adopt the top result, but in speaker verification, that leads to errors. There are real people with very similar voice qualities (e.g., deep-voiced men, high-pitched bright women), and source material recorded under similar conditions can look similar. If the scores for 1st and 2nd place are close, it is a signal that "the machine cannot decide."

Therefore, I output the top 3 candidates with their scores and decided that anything marked as △ or below must be reviewed by a human. This isn't a stopgap; it is a division of labor. The machine handles the vast number of obvious ○ cases, and the human only looks at the small number of truly ambiguous ones. This is orders of magnitude more efficient than listening to every single clip.

Once an estimation is confirmed, I write it back to anchor_sources.json (the mapping table of Anchor Name $\to$ Raw Source Filename). Having this mapping makes everything downstream much easier. Quality audits (audit_anchor_quality.py) and distribution visualizations (plot_anchor_distribution.py) can now use this table to display spk24 by its actual speaker name. Data that could only be referred to by numbers has been transformed into an asset that can be discussed by name.

Pitfalls and Lessons Learned

• Don't trust a single-window embedding. A speaker vector should be a normalized average of multiple windows. If you are skewed by a single window containing silence or a stutter, the same person will look like someone else.
• Explicitly show confidence and hand gray zones to humans. Automatically adopting the #1 result will definitely cause issues with speakers of similar voice qualities. I created a division of labor between machine and human using "Top 3 + 3 levels of confidence."
• Use the same embeddings and the same thresholds for estimation and definition. By using the same campplus and 0.55 threshold for both, I prevent the discrepancy where "the machine says they are similar, but they don't sound similar in production."
• Labels can be recovered from the voice. Even if metadata is lost, the voice itself is primary information. Before discarding data because the source is unknown, it is worth questioning if you can match them by voice quality.

Summary

• I restored labels for unknown anchors using nothing but cosine similarity of speaker embeddings.
• I created noise-resistant speaker vectors by calculating the normalized average of 12 windows per source file.
• I reused the existing anchor_embeddings.npz for the anchors, ensuring estimation and definition use the same "ruler."
• I implemented a division of labor: Obvious cases are handled by the machine (○ ≥0.55), while ambiguous cases are sent to humans (△ ≥0.45).
• By writing the restored labels back to anchor_sources.json, audits and visualizations can use names instead of numbers, turning dead data into a valuable asset.