Voice AI Architecture: Building Scalable TTS Systems

Why a Scalable TTS Architecture Matters

Text‑to‑speech (TTS) is no longer a novelty. From interactive voice assistants to real‑time captioning, developers are asked to serve high‑quality, low‑latency audio to thousands (or millions) of users. Building a monolithic script that calls a cloud endpoint works for a prototype, but it quickly hits bottlenecks when traffic spikes, latency budgets tighten, or you need to support multiple voices and languages.

In this article we’ll walk through a production‑ready architecture for voice AI that:

1. Decouples synthesis from request handling.
2. Scales horizontally with containers or serverless functions.
3. Supports voice cloning and custom voice models without rewriting code.

Along the way we’ll use ElevenLabs – a state‑of‑the‑art TTS service that offers natural‑sounding voices and an easy‑to‑use REST API. (You can try it here: https://try.elevenlabs.io/kr07zfuqn1bp)

---

1. Core Building Blocks

Block	Responsibility	Typical Tech
API Gateway	Auth, rate‑limiting, routing	AWS API GW, Kong, Nginx
Request Processor	Validate payload, queue jobs	FastAPI, Express, Go HTTP
Job Queue	Buffer bursts, guarantee delivery	RabbitMQ, SQS, Redis Streams
Worker Pool	Call the TTS engine, store results	Python workers, Node workers, Celery
Storage	Persist audio files & metadata	S3, GCS, Azure Blob
Cache	Serve recent audio quickly	CloudFront, Cloudflare, Redis
Monitoring	Latency, error rates, cost	Prometheus, Grafana, OpenTelemetry

The flow is simple:

Client → API GW → Processor → Queue → Worker → TTS Service → Storage → CDN → Client

Because the heavy lifting (the actual synthesis) lives in the Worker stage, you can spin up dozens of workers on demand without touching the front‑end code.

---

2. Picking the Right TTS Engine

There are three main options:

Option	Pros	Cons
Open‑source (e.g., Coqui TTS, Mozilla TTS)	Full control, no per‑call cost	GPU‑heavy, harder to maintain, voice quality varies
Commercial SaaS (ElevenLabs, Google Cloud TTS, Azure Speech)	High fidelity, built‑in voice cloning, auto‑scaling	Pay‑per‑use, vendor lock‑in
Hybrid (run a small open‑source model for cheap fallback)	Cost‑effective for low‑quality needs	Adds complexity

For most developers who need natural‑sounding voices and quick voice cloning, ElevenLabs is a solid choice. Their API returns audio in a single HTTP request, supports multiple languages, and lets you upload a few minutes of reference audio to create a custom voice. You can get started instantly with the affiliate link: https://try.elevenlabs.io/kr07zfuqn1bp

---

3. A Minimal FastAPI + ElevenLabs Worker

Below is a Python 3.10+ example that shows how a worker can pull jobs from a Redis stream, call the ElevenLabs API, and store the MP3 in an S3 bucket.

# worker.py
import os
import json
import aiohttp
import aioboto3
import asyncio
import redis.asyncio as aioredis

ELEVEN_API_KEY = os.getenv("ELEVEN_API_KEY")
S3_BUCKET = os.getenv("S3_BUCKET")
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
STREAM_NAME = "tts_requests"

async def synthesize(text: str, voice_id: str = "EXAVITQu4vr4xnSDxMaL"):
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json",
        "Accept": "audio/mpeg"
    }
    payload = {"text": text, "model_id": "eleven_monolingual_v1"}
    async with aiohttp.ClientSession() as session:
        async with session.post(url, json=payload, headers=headers) as resp:
            resp.raise_for_status()
            return await resp.read()   # MP3 bytes

async def store_audio(audio_bytes: bytes, key: str):
    session = aioboto3.Session()
    async with session.client("s3") as s3:
        await s3.put_object(Bucket=S3_BUCKET, Key=key, Body=audio_bytes,
                            ContentType="audio/mpeg")
    return f"s3://{S3_BUCKET}/{key}"

async def process_job(msg_id, data):
    payload = json.loads(data["payload"])
    text = payload["text"]
    voice = payload.get("voice_id")
    audio = await synthesize(text, voice_id=voice)
    s3_key = f"tts/{msg_id}.mp3"
    url = await store_audio(audio, s3_key)
    # Mark job as done (optional: push result to another stream)
    print(f"✅ Saved {url}")

async def worker():
    r = aioredis.from_url(REDIS_URL)
    while True:
        msgs = await r.xreadgroup(groupname="tts_workers",
                                  consumername="worker_1",
                                  streams={STREAM_NAME: ">"}, 
                                  count=5, block=5000)
        if not msgs:
            continue
        for stream, entries in msgs:
            for msg_id, fields in entries:
                try:
                    await process_job(msg_id, fields)
                except Exception as exc:
                    print(f"❌ Failed {msg_id}: {exc}")
                finally:
                    await r.xack(STREAM_NAME, "tts_workers", msg_id)

if __name__ == "__main__":
    asyncio.run(worker())

Key takeaways

• The worker is stateless – you can run dozens of identical containers behind a Kubernetes Deployment.
• Using aiohttp and aioboto3 keeps the I/O non‑blocking, which is essential when you have many concurrent syntheses.
• The ElevenLabs endpoint is called with a single POST that returns the MP3 directly, simplifying error handling.

---

4. Scaling the Worker Pool

Container‑First Approach

# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY worker.py .
CMD ["python", "worker.py"]

Deploy with a simple Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tts-worker
spec:
  replicas: 3   # increase based on CPU/RAM & traffic
  selector:
    matchLabels:
      app: tts-worker
  template:
    metadata:
      labels:
        app: tts-worker
    spec:
      containers:
        - name: worker
          image: ghcr.io/yourorg/tts-worker:latest
          env:
            - name: ELEVEN_API_KEY
              valueFrom:
                secretKeyRef:
                  name: elevenlabs-secret
                  key: api-key
            - name: S3_BUCKET
              value: your-audio-bucket
            - name: REDIS_URL
              value: redis://redis:6379
          resources:
            limits:
              cpu: "500m"
              memory: "256Mi"

Kubernetes will automatically schedule new pods when you bump replicas. For bursty traffic you can add a Horizontal Pod Autoscaler (HPA) that watches the length of the Redis stream.

Serverless Alternative

If you prefer not to manage containers, you can spin up an AWS Lambda (or Cloudflare Workers) that processes a single message from SQS. The same synthesize function works; just replace the storage code with a pre‑signed S3 URL.

---

5. Voice Cloning & Custom Voices

ElevenLabs makes voice cloning as easy as uploading a few seconds of reference audio. After you’ve created a custom voice in their dashboard, you’ll receive a voice_id that you can pass to the API.

{
  "text": "Hello, this is my brand new voice!",
  "voice_id": "my_custom_voice_123"
}

When you store the voice_id alongside user profiles, your API can automatically select the correct voice per request, enabling personalized assistants or audiobook narration with a single line of code.

Tip: Cache the voice_id mapping in Redis for fast lookup; a miss‑cache hit can be handled by a fallback generic voice.

---

6. Observability & Cost Controls

Metric	Why It Matters
Latency (ms)	Determines user experience; aim for < 500 ms end‑to‑end.
Success Rate	Detect API throttling or audio corruption early.
Cost per 1k chars	ElevenLabs charges per character; keep an eye on usage spikes.
Queue Depth	High depth signals under‑provisioned workers.

Prometheus exporters can be added to the FastAPI processor and the worker. For cost alerts, you can use the ElevenLabs usage endpoint (if available) or simply monitor the total characters processed in Redis.

# Example: expose worker metrics
pip install prometheus-client

from prometheus_client import Counter, start_http_server

synth_counter = Counter("tts_requests_total", "Number of TTS requests processed")
error_counter = Counter("tts_errors_total", "Number of failed syntheses")

# Increment inside process_job
synth_counter.inc()
# In exception block:
error_counter.inc()

Run start_http_server(8000) in the worker entry point and scrape it with Prometheus.

---

7. Putting It All Together

1. Front‑end (React, Flutter, etc.) calls your API Gateway with { text, voiceId? }.
2. Processor validates and pushes a JSON payload onto a Redis stream.
3. Workers (containerized, autoscaled) read from the stream, invoke ElevenLabs, store the MP3 in S3, and optionally push a result URL to a response queue.
4. Client polls or receives a webhook with the audio URL, which is served via a CDN for sub‑second playback.

With this pattern you get:

• Horizontal scalability – add more workers, no code changes.
• Fault tolerance – if a worker crashes, the message stays in the stream.
• Voice flexibility – switch between built‑in and custom voices with a single field.

---

8. Next Steps

• Experiment: Use the free tier of ElevenLabs to generate a few samples.
• Add a cache: Store recently requested phrases in Redis to avoid duplicate syntheses.
• Implement auth: JWT or API keys at the gateway to protect your usage.
• Explore multi‑region deployment: Keep latency low for global users by replicating the storage bucket and CDN.

---

Ready to give it a spin?

ElevenLabs provides a generous free quota and a simple API that fits perfectly into the architecture we just built. Grab your API key and start cloning voices today: https://try.elevenlabs.io/kr07zfuqn1bp

Happy coding, and may your applications sound as smooth as your code!