I built an AI voice chat app with custom characters — here's the stack and everything that broke

I got tired of chatbots that can only talk. I wanted one that sounds like the character, keeps its personality over a whole conversation, and answers me in my own language.

So I built IboChat: a Flask app where every character replies in text and out loud, in 24+ languages, and where you can create your own character in a couple of minutes. It's live at chat.emreguzel.ca — everything below is the real story of how it's built and what went wrong along the way.

The shape of the thing

The stack is boring on purpose:

• Flask, one file, serving pages + JSON API + audio endpoints
• GLM (glm-4.5-air) through the Z.ai API as the brain
• faster-whisper running locally for speech-to-text
• SQLite + Werkzeug hashing for accounts, auto-created on startup
• Docker behind Cloudflare

No React, no build step, no microservices. The interesting parts are the voice pipeline and the character system.

How a message travels

You type or talk. If you talk, faster-whisper turns it into text on the server CPU. The backend then builds the prompt the same way every single time: the character's personality prompt, a window of recent history, your language code. The model never sees a naked message — that discipline is what keeps a character from drifting into generic-assistant mush halfway through a chat.

The reply comes back in character, then the fun part starts: making it speak.

The voice pipeline: three tiers because no engine does it all

This is where most of my hair loss happened. I needed "sounds like the character" AND "works in any language" AND "doesn't fall over". One engine couldn't do all three, so /speak tries three paths in order:

1. GPT-SoVITS — best cloned voices, ~5–18 seconds to synth, but English/Chinese/Japanese/Korean only, and it permanently eats ~3.9GB of VRAM once loaded.
2. OmniVoice Studio — any language, ~30–40s, through an OpenAI-compatible endpoint.
3. Edge TTS — Microsoft's free neural voices. Nothing cloned, but 24 languages, zero latency, never fails.

The last one is the safety net that means the app always has something to say.

The bug that took me three days: one GPU, two voice models

GPT-SoVITS and OmniVoice both live on the same 4GB GPU. For a while, loading one starved the other and audio came out corrupted — I genuinely thought I had a bad CUDA install and reinstalled PyTorch twice. The actual fix was orchestration, not hardware:

• Flask launches GPT-SoVITS on demand, never at boot
• A watchdog thread kills it after 120 idle seconds
• OmniVoice's VRAM gets flushed before every call so the two models stop wrestling

Cold starts, or the silence of death

The voice model takes ~30–50 seconds to load. If a user sends a message during that window, the request just... hangs. My first version held the request open. Now the server probes whether the model is ready, and if not it falls through to the next voice tier instead of making a human wait in silence. Users hear a slightly less fancy voice; they don't hear a timeout.

The small fights

• Rate limits. The AI endpoint rate-limits under load. Retries with exponential backoff — a chatty user sees a pause, never an error.
• Mobile autoplay. Browsers refuse to auto-play audio before the user has touched the page. First reply plays after the first tap. Annoying, unfixable, fine.
• Voice cloning from YouTube. Cloning a character voice is automated now: yt-dlp grabs a clean clip, ffmpeg trims it, OmniVoice builds the profile. Cached per character so it only happens once. Getting clips that were clean enough was its own evening of pain.
• The endpoint is called /api/gemini. Early version used Google Gemini. It proxies to Z.ai now. I left the name because renaming would break my own frontend and, honestly, it's a nice little tombstone of the architecture's history.

Cost

One thing I'll say for anyone budgeting this: with a cheap model (glm-4.5-air class) and a text-only reply per message, the LLM side of a conversation costs a rounding error. Voice is free (edge-tts) or electricity (the two GPU models).

Try it / what's next

It's live: chat.emreguzel.ca. Characters are clearly labeled as AI parodies.

Next up: giving characters long-term memory between sessions, and better mid-conversation language switching.

If you build something with voice + AI, drop a link in the comments — I want to see how other people solve the one-GPU-two-models problem.

---

Originally published on my blog. I post new C, Python and AI-agent guides there regularly.