The AI speaking tutor is four API calls in a trenchcoat

gao chong — Fri, 06 Nov 2026 09:19:42 +0000

I build an English vocabulary app for Vietnamese learners. One feature — "speak, get scored, get a spoken reply" — turned out to be four sequential network hops, and most of the interesting bugs lived in the seams between them.

I'll only publish numbers I actually measured in production. Where I don't have a number, I say so instead of guessing.

One turn is four hops, not one request

record (client)
  -> upload the clip to temporary storage, get back an internal URL
  -> speech recognition: submit job -> poll -> download transcript
  -> LLM writes the reply, with the conversation history attached
  -> TTS renders the reply as audio

The surprise: the ASR model does not accept audio inlined in the request body. It takes a URL. So the upload step isn't an optimisation, it's mandatory — and it comes with a short-lived link (48 hours in my config), which means every piece of processing has to close inside the session.

The ASR call is also asynchronous. There is no POST-and-get-your-transcript path. You submit, then poll until the result file exists. If you're used to writing synchronous request/response code, this polling loop feels like a wart. It buys you the ability to not hold an open connection for tens of seconds.

Latency the user perceives is the sum of all four hops plus the network between them. Nobody experiences "the ASR step is slow"; they experience "the app is slow".

History length is a latency bug that never throws

Because every turn re-sends the whole conversation, latency grows with turn count. It degrades smoothly, which is exactly why it's hard to catch: no error, no timeout, just a gradually worse app.

// was: 20
private static final int MAX_HISTORY_MESSAGES = 10;

I cut it to 10 messages for the free-form speaking tutor, and 16 for the scenario dialogues — a scripted scenario is usually over in four to six turns, and extra history past that mostly causes the model to restate points it already made. Same place, I cap max_tokens at 1000 (speaking) and 600 (scenarios): long enough for a conversational reply, short enough that the model can't give a speech.

Three small parameters with non-obvious behaviour

language_hints reads only the first element. I used to send a two-language list and assumed the system would balance between them. It doesn't — the second value is silently ignored. If you want multi-language behaviour, you implement the selection logic yourself and send exactly one hint.

enable_thinking: false. Turning off hidden reasoning cuts time-to-first-byte. Correct for short conversational turns; the trade is that answers are less "cooked". Don't leave it on by default just because the vendor's example does.

temperature: 0.8. Higher than the safe textbook value. Language practice needs varied phrasing, not reproducible phrasing.

"Please speak slowly and clearly" costs 15–19%

I added an instruction to the TTS call asking for slower, clearer output. Measured effect: the same text produces 15–19% longer audio. Users wait longer, the bill is bigger.

I kept it, because comprehension improved enough to justify it. The point isn't the decision, it's that a single innocuous-looking instruction string is a cost and latency decision. If you use these, measure audio duration across a sample — don't eyeball one clip and call it fine.

The scoring model that graded silence

This is the one I'd tell anyone choosing a speech-scoring model: one model returned high confidence scores for a silent recording. It wasn't scoring pronunciation, it was guessing that a score should exist.

If your model-selection process is "run some samples and see if the scores look reasonable", you will pick this one, because its output looks great. My selection criteria changed permanently: feed it garbage first, real speech second. A silent file, a noise-only file, a recording of someone reading the wrong sentence. If it still hands out high scores, it's out.

Rate limits are a cost brake, not a "fair use policy"

Real numbers, currently in production:

Path	Limit
LLM reply generation	3 req / min / user
Speech recognition	20 / min / user, 40 / min / IP
Text-to-speech	30 / min / user, 60 / min / IP
Pronunciation scoring	6 / min
Pronunciation scoring (HTTP ceiling)	read 90s — measured p50 2.3s, worst ~3.2s

The per-IP numbers exist because per-user alone isn't enough: a lot of users sit behind one NAT (cafés, dorms) and would evict each other if I counted only by IP. I learned that expensively.

For new users I chose a total turn allowance rather than "N turns per day". A serious learner needs to talk a lot in the first few days to find out whether the app fits them; someone who bounces after three days isn't retained by a daily allowance anyway. The cost is that I lose the ability to nurture people slowly with a small daily drip. That's a business decision wearing an engineering costume, and you should know that before you copy it.

What I still haven't measured

So this isn't an advert with extra steps:

No per-region cost per turn — the vendor invoice aggregates, and I haven't split it.
No peak-hour load test. The latency figures above come from normal usage.
No assessment of ASR behaviour on weak connections, which is the common case for learning on a bus.

If you're building something like this, measure those three before writing a post about it.

The original long version, with the full parameter dump: thuoctu.91word.com/devlog-luyen-noi-ai.html

DEV Community: gao chong