DEV Community: zi zong

Your AI Voice App Has 500+ Voices. How Should Users Find the Right One?

zi zong — Thu, 24 Sep 2026 03:42:52 +0000

Ten voices can fit in a simple list. At one hundred, filters start earning their place. At one thousand, discovery becomes a system of its own.

The hard question is no longer:

Can the system generate speech?

It is:

Can someone find the right voice without opening dozens of unrelated samples?

This looks like a UI problem, but the interface is only the visible part. Underneath it are decisions about data modeling, retrieval, ranking, and content management.

If I were building an AI voice application with a growing catalog, I would treat voice selection as a pipeline of its own, not as a dropdown placed in front of the Generate button.

Start with a voice schema that supports decisions

Before working on search, I would define what a voice means inside the application. A minimal record might look like this:

{
  "voice_id": "voice_0042",
  "languages": ["en", "ja"],
  "style": "calm",
  "pace": "medium",
  "use_cases": ["narration", "education"],
  "source_type": "shared",
  "preview_url": "/previews/voice_0042.mp3",
  "usage_notes": "Suitable for non-character narration"
}

The audio file is only one part of that record. The metadata is what makes the voice findable.

voice_id gives the system a stable identifier.
languages removes incompatible results early.
style describes the general delivery.
pace separates slower narration from faster conversational speech.
use_cases groups voices by the work they suit.
source_type distinguishes private, shared, and synthetic assets.
usage_notes carries context that does not fit neatly into a category.

Without those fields, a catalog eventually turns into a folder full of audio. With them, the application has something it can query.

Filter before reaching for semantic search

It is tempting to turn every discovery feature into an embedding problem. I would start with ordinary structured filters.

Suppose the request is:

Language: Japanese
Style: Calm
Use case: Education

The first pass can be quite literal:

all voices
↓
Japanese voices
↓
calm Japanese voices
↓
calm Japanese voices suitable for education

Ranking can happen after that reduction. This approach is easier to inspect and debug. If someone asks why a result appeared, the answer comes from fields in the record rather than an opaque similarity score.

Preview belongs inside the retrieval loop

Voice search does not behave like image search. A thumbnail often tells you enough to reject an image; a voice usually has to be heard.

Preview is therefore part of retrieval, not a secondary feature:

search
↓
filter
↓
preview
↓
shortlist
↓
generate test

That affects the design of each result. A useful voice card might show:

Voice name
Language and style
Preview duration
Play button
Usage notes

There is no need to expose every stored field. The card only needs enough information to help someone decide whether the sample is worth playing.

Keep the first ranking function boring

After filtering, I would try a small deterministic score before adding a recommendation system:

function scoreVoice(voice, query) {
  let score = 0;

  if (voice.languages.includes(query.language)) {
    score += 4;
  }

  if (voice.style === query.style) {
    score += 3;
  }

  if (voice.use_cases.includes(query.useCase)) {
    score += 2;
  }

  if (voice.pace === query.pace) {
    score += 1;
  }

  return score;
}

const rankedVoices = voices
  .map((voice) => ({
    ...voice,
    score: scoreVoice(voice, query)
  }))
  .sort((a, b) => b.score - a.score);

The scoring is intentionally plain. Early on, predictable ranking is often more useful than a clever system that nobody can explain. Once real selection data exists, the weights and fields can change without replacing the whole pipeline.

Metadata should reflect actual selection behavior

Another trap is adding fields because the schema looks more complete with them.

Labels such as these sound reasonable:

Warm
Bright
Calm
Energetic
Conversational
Professional

But a label earns its place only when it changes a decision. If people never filter by Bright, or cannot agree on what it means, storing it may add noise instead of structure.

The better question is: what information causes someone to choose one voice over another?

While thinking through this interface, I found it useful to look at existing catalogs, including the FreeVoiceClone voice library, with that question in mind. The number of available voices matters less than whether the interface provides enough information to compare two candidates.

A catalog becomes easier to use when its metadata follows real selection behavior rather than an imagined taxonomy.

Provenance belongs in the same data model

Voice assets are not quite like fonts or icons. A voice may be tied to a person's identity, permission, or a particular source, so provenance cannot be left for a separate policy page.

A record may eventually need fields such as:

{
  "source_type": "shared",
  "visibility": "public",
  "permission_status": "verified",
  "intended_use": ["narration", "education"],
  "created_at": "2026-09-24"
}

The exact schema will depend on the product. What matters is keeping provenance and intended use beside the asset they describe.

If a voice is easy to discover but its origin is unclear, retrieval has solved only half of the problem.

A voice catalog also needs lifecycle states

Once several people can create, review, and publish voices, the catalog needs lifecycle information as well:

draft
↓
reviewed
↓
published
↓
deprecated

The corresponding record could include:

{
  "status": "published",
  "reviewed_by": "user_102",
  "version": 3,
  "updated_at": "2026-09-24"
}

These fields become useful as soon as audio or metadata can change. Without them, even basic questions are awkward to answer:

Which version is public?
Who reviewed it?
Did the metadata change after the audio?
Should an older voice still appear in search?

This is a familiar content-management problem. The content just happens to be audio.

Search becomes part of the product architecture

As the catalog grows, voice search starts affecting the whole experience. A basic architecture might look like this:

Voice assets
    ↓
Metadata store
    ↓
Filter layer
    ↓
Ranking
    ↓
Preview
    ↓
User selection
    ↓
Generation

Generation is near the end of this flow. A user may spend more time finding and checking a voice than producing the final audio, which changes where engineering effort should go.

Add semantic search when structured filters stop being enough

I would add semantic search after users begin expressing needs that fixed fields cannot capture well, for example:

A calm voice that sounds suitable for a documentary.

Or:

Friendly, but not too energetic.

An embedding-based layer or an LLM query parser could translate those requests into structured constraints:

{
  "style": ["calm", "professional"],
  "use_case": ["narration", "documentary"],
  "energy": "low"
}

The existing filter and ranking pipeline can still do most of the work. Semantic search becomes an input layer instead of a replacement for the rest of the system.

The harder product work starts after generation works

AI voice products are often demonstrated through generation quality. That makes sense: realism is immediate and easy to hear.

Once generation is good enough, a different set of questions takes over:

How should voices be organized?
How can users search and compare them?
Is the source of each voice clear?
Which fields belong in ranking, and which only belong in the details view?

These are product and data-design questions, not model questions.

With ten voices, almost any interface works. With one thousand, metadata, filtering, preview, ranking, provenance, and lifecycle state are part of the core system.

Generation creates the asset. Information architecture makes it usable.

A Practical Workflow for Reusing One Voice in Multilingual Videos

zi zong — Fri, 18 Sep 2026 05:52:47 +0000

Producing a video in one language is usually straightforward. Making the same video in three or four languages is where the workflow starts to drag.

The edit may stay mostly the same, but the voice track does not. If each translation requires a fresh recording, even a minor script change means recording, exporting, and syncing the audio again.

That is a lot of repeated work for a short video.

I started looking at the problem less as "How do I generate AI speech?" and more as:

How can the same voice be reused across multiple versions of a video without rebuilding the audio workflow every time?

The basic workflow is:

voice sample → translated script → cloned speech → video editor

The steps are simple. Getting useful results depends on a few details.

1. Start with a clean voice sample

Source audio has a noticeable effect on the result.

A short recording from a quiet room is usually more useful than a longer clip with background music, room echo, or several people speaking.

Before using a sample, check:

Is there only one person speaking?
Is the voice clearly louder than the background noise?
Does it contain normal speech rather than shouting or whispering?

Studio-quality audio is not required. The recording just needs to be clean enough for the voice to stand apart from everything else.

2. Test with a short sentence first

Pasting the entire translated script into a tool may save a step initially, but it makes problems harder to isolate.

Start with one or two sentences. Listen for pronunciation, pacing, and wording that sounds awkward when spoken.

A sentence that fits neatly in English may become much longer after translation. The generated voice can sound fine while the timing no longer fits the original video.

It is quicker to revise a short test than to regenerate a two-minute track.

3. Treat voice cloning as one step in the workflow

The cloning step can be fairly simple. A browser-based option such as FreeVoiceClone can take a voice sample and generate speech from another piece of text.

Most of the work sits around that step:

Prepare the original voice sample.
Translate the script.
Edit the translation for spoken delivery.
Generate a short test.
Check its pronunciation and pacing.
Generate the full section.
Import the audio into the video editor.

The third step is easy to overlook. A machine translation may be technically correct and still sound stiff when read aloud. Voice generation will reproduce that awkward wording; it will not repair it.

Sometimes a small rewrite helps more than changing the audio settings.

4. Expect the timing to change between languages

This is usually the main editing problem. Different languages take different amounts of time to express the same idea.

A six-second sentence in the original might take four seconds in one translation and eight in another.

For a talking-head video, a small playback-speed adjustment or a different cut may be enough. With a screen recording, adjusting the visuals around the new voice track is often easier than forcing the speech into the original timing.

Short social videos leave less room. In that case, translate for meaning rather than word for word, then shorten the sentence where needed. The result often sounds more natural too.

5. Keep the generated audio editable

Avoid generating the whole script as one long audio file. Split it into sections, for example:

intro
main point 1
main point 2
ending

If a sentence needs to change, only that section has to be replaced.

This matters more once several language versions are involved. A small update to the source script no longer requires rebuilding every voice track from the beginning.

Where this workflow works well

This approach fits content that changes often, including:

short product demos
tutorials
educational clips
social media videos
internal training videos
localized versions of the same content

It is a weaker fit when the recording relies heavily on acting, emotion, or precise lip synchronization.

Voice cloning can remove some repetitive recording, but localization still needs editorial work. The script, pronunciation, timing, and final cut all need review.

Review the result like an editor

It is easy to focus on whether the generated voice "sounds real." That is only one part of the finished video.

Check the full edit:

Does the wording sound natural in that language?
Are the pauses in sensible places?
Does the audio match what is happening on screen?
Is the pace comfortable?
Are names and technical terms pronounced correctly?

A slightly imperfect voice with good timing often works better than a realistic voice with awkward pacing.

Final thoughts

For multilingual video, voice cloning is most useful when it removes repeated recording from the production process.

The workable setup is fairly ordinary: a clean source sample, an edited translation, short test generations, and a timeline that can accommodate timing changes. Once those pieces are in place, adding another language takes less rework than recording the full video again.