Self-hosting an OpenAI-compatible TTS API with Kokoro-82M (no GPU needed)

Great Sage — Mon, 20 Jul 2026 00:03:47 +0000

Most self-hosted TTS is a tradeoff: espeak/piper are fast and free but sound robotic, and anything that sounds actually good usually means a GPU box or an ElevenLabs bill. Kokoro-82M is the first open model I've used that dodges both problems — 82M parameters, runs fine on CPU, and the output is close enough to ElevenLabs-tier for narration and voiceover work that I stopped reaching for paid APIs.

The part that made it actually usable for me is Kokoro-FastAPI — a wrapper that exposes Kokoro behind an OpenAI-compatible /v1/audio/speech endpoint. If you already have code that talks to OpenAI's TTS API, you don't rewrite anything, you just swap the base URL:

from openai import OpenAI

client = OpenAI(
    base_url="https://your-app.up.railway.app/v1",
    api_key="your-api-key",
)

response = client.audio.speech.create(
    model="kokoro",
    voice="af_bella",
    input="Hello world!",
)
response.stream_to_file("output.mp3")

What you get out of the box:

OpenAI-compatible POST /v1/audio/speech
multi-language: English, Japanese, Chinese
output formats: MP3, WAV, Opus, FLAC, M4A, PCM
streaming with configurable chunk sizes
weighted voice combinations, e.g. af_bella(2)+af_sky(1) for a 67/33 blend
word-level timestamped captions via POST /dev/captioned_speech
a built-in web UI (/web) and API docs (/docs) so you can poke at it without writing a client

Tradeoff worth knowing up front: the image I run is the CPU build, so it's not going to touch a GPU pipeline on raw throughput or handle heavy concurrent load. For narration, scripts, voiceover, IVR prompts — anything not real-time-at-scale — it's been solid.

Deploying it

I maintain a one-click Railway template for this — it's just the stock ghcr.io/remsky/kokoro-fastapi-cpu image, nothing patched. Full disclosure: I get a kickback if you deploy through it.

Deploy: https://railway.com/deploy/kokoro-tts-api?referralCode=Z1xivh&utm_medium=integration&utm_source=template&utm_campaign=generic

If you'd rather not go through a referral link, the same image runs anywhere Docker does:

docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:latest

Upstream repo (all credit to the actual authors): https://github.com/remsky/Kokoro-FastAPI

Happy to answer questions on memory sizing or voice quality in the comments — it's been running steady for me without needing much tuning.

Grading free-form Japanese answers with an LLM: Django + pydantic-ai

Great Sage — Sun, 19 Jul 2026 01:03:49 +0000

I'm studying for the JLPT (aiming for N2 this cycle), and the thing that kept bugging me about every prep app I tried is that they're all multiple choice. You tap A/B/C/D, you get a green checkmark, and you feel like you're learning. But multiple choice trains recognition, not recall. You can pattern-match your way to a passing-looking score without being able to actually produce a single sentence.

So I started building BetterLingo — a JLPT study app whose daily practice is free-form. No multiple choice. You write a real answer, and an LLM grades it. This post is about the part that was actually interesting to build: grading a free-form Japanese answer reliably enough that a learner trusts the score.

The problem with "just ask the model to grade it"

The naive version is one prompt: "Here's the question, here's the student's answer, give it a score out of 100." It works in a demo and falls apart in real use:

Scores are unstable — same answer, different run, ±20 points.
The model hallucinates a rubric that doesn't match the question type.
You get back prose you have to regex, and sometimes it just... doesn't give a number.
No structured breakdown, so the learner can't see why they lost points.

For a study tool that has to be near-zero cost per grade, run reliably, and give feedback a learner believes, none of that flies.

Structured output with pydantic-ai

The single biggest reliability win was forcing the model into a typed schema instead of free text. I use pydantic-ai, so the grade is a Pydantic model:

from pydantic import BaseModel, Field
from pydantic_ai import Agent


class GradeBreakdown(BaseModel):
    grammar: int = Field(ge=0, le=100)
    vocabulary: int = Field(ge=0, le=100)
    naturalness: int = Field(ge=0, le=100)
    meaning: int = Field(ge=0, le=100)


class Grade(BaseModel):
    score: int = Field(ge=0, le=100, description="Overall score")
    breakdown: GradeBreakdown
    feedback: str = Field(description="Two or three sentences, specific")
    example_answer: str = Field(description="A model answer in Japanese")


grader = Agent(
    "openai:qwen3-next-80b",  # via an OpenAI-compatible endpoint
    output_type=Grade,
    system_prompt=GRADER_SYSTEM_PROMPT,
)

Because the output is a validated model, I never parse prose. If the model returns something off-schema, pydantic-ai catches it and retries automatically. The ge/le bounds mean a score is always a real 0–100 int, never "about 85".

Four question types, four rubrics

"Free-form" isn't one task. The daily set mixes four types, and each needs a different grading emphasis:

Answer — respond to a prompt in Japanese. Graded on meaning + naturalness.
Fill in the blank — one right-ish token, graded strictly on grammar/collocation.
Translate EN→JP — meaning fidelity first, then naturalness.
Translate JP→EN — comprehension check; grammar of the English matters less.

The trick that stabilized scores was not having one universal rubric. Each type gets its own system prompt and its own weighting of the breakdown fields. A perfectly natural answer that misses the point of a translation task shouldn't score the same as one that nails the meaning.

Keeping grading cost near zero

This runs on my own study sessions daily, and I want it to scale without a bill that scales with it. So the two LLM jobs are split by cost profile:

Grading (high volume, runs every practice item) uses Qwen3-Next-80B on NVIDIA's free inference tier. Structured output means even a smaller/cheaper model stays on-rails.
Tutor + hints (lower volume, needs to be sharp) use Claude Sonnet via OpenRouter.

The tutor — I call her Rami — is the other half of the free-form bet. She sees your actual attempt: the question, what you wrote, the score, the per-field breakdown, the feedback. So when you ask "why is my answer wrong," she's responding to your specific mistake, not reciting a generic grammar FAQ. That context handoff is only possible because the grade is structured data, not a paragraph.

The boring stack (on purpose)

The rest is deliberately unexciting so I spend my time on the learning loop, not plumbing:

Django 6 + HTMX + Tailwind — no JS build step, server-rendered.
Postgres for everything.
granian for ASGI.
A traditional SM-2 SRS review mode and kanji study with level gates, for when free-form recall is too heavy and you just want to drill.

What I've learned so far

Structured output is the reliability lever. Switching from "parse the prose" to "the grade is a typed object" removed almost all the flakiness in one move.
One rubric per task type beats one clever universal prompt.
A cheap model with tight constraints often beats an expensive model with a loose prompt, especially for high-volume grading.
Passing structured feedback into the tutor makes the tutor feel like it's actually reading your work — because it is.

It's still rough — I'm the primary user, fixing things as I hit them in my own N2 prep. If you're building anything that grades or evaluates free-form user input with an LLM, I'm happy to compare notes on schema design and rubric splitting. And if you're prepping for the JLPT yourself, betterlingo.io is the thing I'm building — I'd genuinely like the feedback.

DEV Community: Great Sage