[AI in Practice] Gemini 3.8 Flash TTS Launch: I built a "Learn Japanese with MVs" Web App and burned through my daily quota.

Recap

Every time I see a new Gemini feature, my first thought is "Can I connect it to my LINE Bot?"

The Gemini API release notes from 9/22 stated that Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS are officially launched (GA), and the official blog simultaneously posted Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS. As usual, I listed a bunch of LINE Bot ideas: bedtime stories in parents' voices, turning group chats into radio dramas, morning dual-host podcasts...

Halfway through the list, I realized that what actually struck me most wasn't a bot, but the fact that "directing tone sentence-by-sentence" is particularly suitable for teaching pronunciation. So the topic changed to a Web App:

1. Pick a song, get the lyrics, and translate them into Chinese simultaneously.
2. If it's Japanese or Korean, attach the phonetic transcription, then transform into a language teacher to teach you how to read it sentence by sentence.
3. Turn it into a "learning languages through songs" tool.

As it turned out, there was nothing to complain about regarding the quality of the TTS itself; what really took time were the things surrounding it that weren't in the documentation.

---

What is Gemini 3.8 Flash TTS

Two models were released in this GA:

Model	Positioning
gemini-3.8-flash-tts	Flagship model, emphasizes vocal performance and character creation, allows sentence-by-sentence performance control
gemini-3.8-flash-lite-tts	Cheap, fast, suitable for high-volume generation

Compared to the previous generation, there are three things I find truly useful:

• Voice design: Generate a voice using a text description, for example, "a 60-year-old British-accented astronomer with a warm tone." The generated voice gets a voice_id for repeated use.
• Voice replication: Replicate a person's voice using a 10–30 second recording. The prerequisite is that the voice owner must record a consent statement, and the output audio will carry a SynthID watermark and C2PA markers.
• Sentence-by-sentence performance control: Every segment of text can include a style (e.g., "speak slowly, pronounce every syllable clearly"), and it supports tags like <laughs>, <sigh>, as well as dialogue between up to two people.

The actual call structure is different from the past generate_content, moving to two sets of APIs: interactions and voices:

# Design a voice using text description (one-time, save for reuse)
voice = client.voices.create(
    store=True,
    voice={
        "model": "gemini-3.8-flash-tts",
        "type": "prompted",
        "display_name": "Song Lingo Japanese Teacher",
        "gender": "female",
        "language_code": "ja-JP",
        "prompted": {"input": "A warm, patient Japanese language teacher in her early 30s from Tokyo..."},
    },
)

# Use this voice to read a sentence, with style controlling tone and speed
interaction = client.interactions.create(
    model="gemini-3.8-flash-tts",
    input=[{
        "type": "user_input",
        "content": [{
            "type": "text",
            "text": "こんにちは。今日はいい天気ですね。",
            "annotations": [{"type": "speech_metadata", "style": "speaking slowly and clearly"}],
        }],
    }],
    response_format={"type": "audio"},
    generation_config={"speech_config": [{"voice": voice.id}]},
)

A few technical details to know beforehand that will save a lot of trouble:

Item	Specification
Output Format	WAV, 16-bit PCM, mono, 24 kHz; also supports streaming
Languages	Over 100 types, including Simplified Chinese, Traditional Chinese, Cantonese; Taiwanese was not seen
Custom Voice Limit	200 per project, kept for 1 year; or choose self-managed voicekey_, valid for 7 days
Voice replication Regional Restrictions	Not available in Illinois, Texas, EEA, UK, Switzerland, India
Price	Not mentioned in announcements or docs
Tier 1 Quota	100 requests per day (this becomes the main character later)

---

Things to Handle Before Starting: Lyrics Copyright

"Enter song name, automatically crawl lyrics" is the most intuitive way, but it gets stuck in two places:

• Copyright: Lyrics sites pay for licenses to display them. Crawling them to display full text and adding translations (legally considered a derivative work) carries infringement risks if launched publicly.
• Technical: Even using Gemini with Google Search, Gemini has mechanisms to prevent reproducing copyrighted text, often returning finishReason: RECITATION.

Then I thought of another way: Throw the YouTube MV URL directly to Gemini and ask it to transcribe the lyrics. Technically feasible, as the Gemini API accepts public YouTube URLs.

But let's be clear: Obtaining lyrics in a different way doesn't make it legal. Whether crawled from a site, typed manually, or transcribed by AI from an MV, it's the same protected text. What determines the risk is how you use it.

So my decision was: This is a personal learning tool; lyrics only exist in the local output/ folder, and this folder was added to .gitignore from the first commit, never appearing on GitHub. During development, I also required Claude Code to only print statistics when verifying data, never printing lyrics to the terminal or writing them to any logs.

---

Architecture: Three Python Scripts + One Next.js

YouTube URL
  ─transcribe.py─▶ Lyrics and timeline
  ─annotate.py──▶ Phonetics, translation, word breakdown, grammar points
  ─speak.py────▶ Teacher demonstration (normal / slow)
  ─Next.js─────▶ Learn sentence-by-sentence with MV

Transcription: Prioritizing On-Screen Subtitles for Gemini

transcribe.py sends the YouTube URL as file_data to gemini-3.8-flash, using structured output to require start/end times, original text, Hiragana for Japanese, and an "unclear" flag for each sentence.

The most critical line in the prompt: If there are lyric subtitles on the MV screen, prioritize using the subtitles. Transcribing singing is much harder than speech due to accompaniment, elongated notes, and harmonies; Japanese also has homophone issues (hearing "kimi" without knowing if it's "君" or "きみ"). With subtitles, Gemini looks at the screen and listens simultaneously, greatly improving accuracy.

I tested with three songs:

Song	Language	Lyrics Source	Sentence Count (Unique)
Yuuri 〈Betelgeuse〉	Japanese	Audio	40 (23)
Yuuri 〈Christmas Eve〉	Japanese	Screen Subtitles	45 (35)
Take That 〈Back for Good〉	English	Audio	37 (26)

The structure was complete: no empty sentences, no backward time, no continuous repetition of the same sentence (a common symptom of model errors). The only thing that made me uneasy was that every sentence was marked as "certain", even for the two based on audio. The model is too confident; I'll fix this later with another method.

Annotation: Don't Leave All Phonetics to the LLM

annotate.py adds Traditional Chinese translation, word-by-word breakdown (reading, POS, meaning), a grammar point, and a pronunciation tip to each sentence. Choruses repeat, so Gemini is only called for unique sentences, saving about 30-40%.

Phonetics is the most interesting part of this section. My original idea was to use existing libraries, which are more reliable than LLMs, but both failed:

• pykakasi (Japanese) transliterates the particle "は" as ha, when the correct pronunciation is wa. Just looking at Hiragana, it can't tell if "は" is a particle.
• korean-romanizer (Korean) transliterates "감사합니다" as gamsahapnida. The official Revised Romanization follows actual pronunciation, which should be gamsahamnida; this package doesn't handle nasalization.

The final division of labor:

• Japanese: Gemini handles word segmentation and provides reading/POS for each word; the program then uses pykakasi to convert to Romaji and corrects particles は, へ, を to wa, e, o based on POS.
• Korean: Hangul is phonetic; I simply asked Gemini to output according to Revised Romanization rules.

Reason and Solution: Libraries are good at "deterministic conversion" but bad at "judgment requiring context"; LLMs are the opposite. Handing the parts requiring understanding (segmentation, POS) to the LLM and the deterministic parts (Kana to Romaji) to the program uses both where they excel.

Cross-checking with Two Independent Results

As mentioned, the transcription results marked everything as "certain," which isn't very believable. My fix: Gemini already gave a full-sentence reading during transcription, and then gave individual word readings during annotation. These two were generated separately; sentences that don't match are likely Kanji misreadings.

In practice, each of the two Japanese songs had 1 mismatch (1/23, 1/35). The program adds needs_review to them, so the webpage can remind the user to proofread.

This trick doesn't cost any extra API calls; it just compares two existing results.

Teacher: "Designed" via Voice Design

speak.py designs a teacher for each language, e.g., for Japanese: "A gentle and patient Japanese teacher from Tokyo in her early 30s." Created once, voice_id saved for reuse. Two audio clips per sentence:

	Normal Speed	Slow Speed
Length	2.5–6.1s, avg 4.3s	4.4–9.7s, avg 7.1s
Slow / Normal		At least 1.24x, up to 2.34x

Slow speed relies solely on a style description without adjusting playback speed, and it's "speaking slowly and clearly," not stretching a normal speed clip. This is what impressed me most about this TTS.

Web App: Next.js 16

The webpage uses Next.js, with the server-side reading data directly from ../output/. The screen shows the embedded MV on the left, teaching cards below (Kana/Romaji/Translation/Breakdown/Grammar/Tips above Kanji, plus "Teacher," "Slow," "Original" buttons); the right side is the sentence list, auto-switching with the MV.

Upon running create-next-app, I encountered something interesting. The generated AGENTS.md wrote in the first line:

This is NOT the Next.js you know

Meaning this version has breaking changes, and agents should read node_modules/next/dist/docs/ before coding. Claude Code followed suit; params became Promises, and RouteContext types for route handlers were found there. Documentation written specifically for AI by frameworks is a practice I think will become increasingly common.

---

Quota: Tier 1 is Only 100 Times Per Day

This is the most important constraint of the project, so I'll mention it first.

gemini-3.8-flash-tts in Tier 1 has 100 requests per day. A song with 30+ sentences, each with normal and slow versions, takes about 50–70 requests. At this rate, only one or two songs can be processed per day.

So I changed the batch generation in speak.py to generate only when the user clicks play, then cache it:

Action	Consumes Gemini?
Viewing lyrics, translation, breakdown	No
Playing MV, clicking "Original"	No, that's YouTube
Playing cached demo	No
First time playing a demo	1 TTS call (Normal and Slow counted separately)
Adding new song	~2 Flash calls, no TTS

Audio files are named using the hash of the sentence content. Repeated choruses naturally share the same audio. Quota is only spent on sentences actually practiced.

---

Pitfall 1: Program "Stuck," Actually Sleeping for Seven Hours

While batch generating for the second song, progress stopped at 56/70.

The status was strange: Python process alive, CPU 0%, 4 workers waiting on Google servers. First guess was no timeout, so I added 60s timeout and retries. After restarting, only 1 segment was added in ten minutes, and this time there wasn't even a network connection.

Turning off SDK auto-retry and looking at the response revealed the answer:

429 Rate limit exceeded for model gemini-3.8-flash-tts
(limit: 100 requests per day on Tier 1). Please retry in 7h12m16s
retry-after: 25936

Daily quota was exhausted, and the SDK, upon receiving a 429, obediently waited 25,936 seconds to retry according to Retry-After. From the outside, it looked like a program that consumed no CPU, had no network, and would never end.

Worse, my initial attempt to use HttpRetryOptions(attempts=1) to disable retries failed completely. The reason is that the interactions API uses a different HTTP client in the SDK (module name _gaos), which ignores HttpRetryOptions. I had to change its own config:

client = genai.Client()
# This API doesn't respect HttpRetryOptions; if not disabled, it sleeps for hours on daily quota limits
client.interactions.sdk_configuration.retry_config.max_retries = 0

Reason and Solution: "Respecting Retry-After" is good for per-minute limits but a disaster for daily quotas. Now the program handles retries itself: transient errors retry up to 3 times; if a 429 contains "per day," all workers stop immediately, print the wait time, and keep existing audio files.

---

Pitfall 2: Fields in Python SDK Don't Exist in REST

This was the most expensive lesson.

After switching to "generate on play," the work moved from Python to Next.js server-side using fetch for REST. Python version:

audio = base64.b64decode(interaction.output_audio.data)

The TS version followed suit with json.output_audio?.data. The problem: The output_audio field doesn't exist in the REST response at all.

Checking the SDK source code revealed that output_audio is a convenience field calculated by the Python SDK in a pydantic validator: it looks for a step with type: "model_output" in the steps array, then extracts the item with type: "audio" from that step's content. The real REST response looks like this:

steps[] → { type: "model_output", content[] → { type: "audio", data: "<base64>" } }

So this happened:

1. Gemini successfully generated audio every time, counting against the quota.
2. My program couldn't read the audio, returned 502, and didn't save the file.
3. Every time a user clicked play, plus multiple Range requests from the browser, TTS was called again.

Within about an hour, the 100-request quota was exhausted, and not a single new file was in output/audio/.

To make matters worse, when this code went live, the daily quota was already exhausted, so the success format couldn't be verified. Claude Code noted "success path unverified" in the report, but we let it go live anyway.

Reason and Solution: Fixed in two layers.

• Parse according to SDK logic: Look for steps, then the old outputs format, and finally output_audio. No quota to verify via API, so I fed the same mock response to the Python SDK parser and the new TS function to ensure consistency. Later, when a quota slot opened, the first English audio was generated and saved (5.04s, 24 kHz), finally verifying it.
• Prevent double billing on failure: For 60 seconds after a failed generation, return the last error directly; after a 429, block all audio until the Retry-After time.

// A failed generation may still have been billed, so don't let repeated clicks or the
// browser's parallel range requests retry it: replay the error for a while instead.
const recentFailures = new Map<string, { error: TtsError; until: number }>();

What this bug taught me: Failure doesn't mean no cost. As long as the request reaches the model and the model runs, even if your program fails at the last parsing step, the bill still counts. Programs that consume quota must see a real success response before going live, and failure paths should default to "this might have already been billed."

---

Pitfall 3: Quotes in .env

Next.js first API call returned:

400 API key not valid. Please pass a valid API key.

The same key worked fine in Python. Checking the .env format (length and start/end chars only) revealed my key was GEMINI_API_KEY="..." with double quotes. Python's python-dotenv automatically strips quotes; the Node-side reader didn't, sending the quotes as part of the key.

Also caught another issue: Gemini's error response is sometimes an array [{ "error": ... }] instead of an object, so the original program couldn't even read the error message, showing a blank "400:" on screen.

Reason and Solution: Changed .env reading logic to match python-dotenv (allow export prefix, strip quotes), and handled both error formats. A small thing, but these "one side handles it for you" differences are easy to miss when sharing a config file between languages.

---

Pitfall 4: Naming Audio by Line Number Causes Misalignment on Proofreading

Transcription errors are inevitable, so I added a proofreading interface: edit text, reading, or translation directly, or click "✓ OK"; after editing text, "Re-analyze whole song" reruns annotate.py (1 Flash request) while preserving manual edits.

Before starting, I found a problem: Audio files were named by "index of unique sentence," e.g., 028_normal.wav. If a lyric line is edited, subsequent sentence numbers might shift, causing existing audio to map to the wrong sentence.

Reason and Solution: Changed to naming by the hash of the sentence content:

def clip_name(text: str, speed: str) -> str:
    """Clips are keyed by line content so editing a lyric only invalidates that line's audio."""
    return f"{hashlib.sha1(text.encode()).hexdigest()[:16]}_{speed}.wav"

Editing one line only invalidates that line's audio; others are unaffected. All 103 generated audio files were moved to new filenames, so no quota was wasted. Python and TS both calculated the hash for the same Japanese segment to ensure consistency before going live.

---

Pitfall 5: Some MVs Don't Allow Embedding

Found some videos wouldn't play. YouTube returns error 101 or 150 when "owner does not allow playback on other sites."

Now when these errors are detected, the video area swaps to a thumbnail with an explanation and an "Open on YouTube (start from here)" button; "Original" also opens a new tab at that timestamp. Teacher demos and teaching cards remain available.

A small pitfall with thumbnails: High-res maxresdefault.jpg isn't available for every video, but YouTube doesn't return 404; it returns a 120x90 gray default image, so <img onError> doesn't trigger, resulting in a large gray area.

Reason and Solution: Check image width after loading; if under 121 pixels, swap to the guaranteed hqdefault.jpg.

---

Development Workflow: Let Claude Code Verify, but Don't Let It See Lyrics

The whole project was done with Claude Code, from checking changelogs and discussing product direction to coding and deployment planning. A few things I think we did right:

• 邊做邊推上 GitHub: Commit and push at every stage, explaining "why" in messages, not just "what."
• 驗證只看統計數字: Checking transcription quality via sentence counts, time flow, repetitions, Kanji in readings; checking audio via length, slow-motion ratio, anomalies. No need to print lyrics.
• 測試用假歌曲: Use a fake song for testing proofreading, re-analysis, and embedding errors (e.g., "駅まで歩く"), deleting it after testing.
• 用無頭 Chrome 實際點按鈕: Server-side tests aren't enough; used Puppeteer for play, proofread, and add flows. The "no sound" issue was narrowed down to the server-side after confirming Chrome playback was fine.

There were mistakes too, as mentioned in the pitfalls:

• Going live with quota-consuming code before verifying success format (Pitfall 2).
• Background "Add Song" status stayed in memory; if files were deleted, it would say "Complete" without redoing. Caught this while writing "resume from break" tests; changed to rely on disk files.

---

Results and Benefits

	Number
commit	11
Songs added	3 (2 Japanese, 1 English)
Demo clips generated	104
Single demo generation time	~5s
Adding a new song	~48s (Transcribe 14s, Analyze 34s)
API usage per song	~2 Flash calls, demos generated on demand

Current features:

• YouTube URL addition, background transcription and analysis.
• Sentence-by-sentence learning: Kana, Romaji, Translation, Breakdown, Grammar, Tips.
• Teacher demos, normal and slow, generated on first play.
• Proofreading interface: Edit lyrics, one-click confirm, re-analyze whole song.
• Fallback UI for non-embeddable videos.

Next Step: Move to Cloud Run

Currently only runs locally. Plan to deploy to GCP:

• Cloud Run for Next.js, containerized with Python and uv.
• GCS for audio and song data. Use signed URLs for playback; GCS supports Range, so Safari works.
• Secret Manager for API keys.
• IAP to lock the service to my Google account. This is essential: public access would drain my quota, and full lyrics/translations would move beyond "personal learning."

Key Takeaways

SDK convenience fields are not part of the API. output_audio is Python-only. Check real responses when switching languages or using REST.

"Respecting Retry-After" depends on the limit type. Good for per-minute, bad for daily. Fail fast and explain why for quota errors.

Failed requests might have been billed. Assume cost on failure for paid operations and prevent retry storms. A 60s failure cache would have saved a day's quota.

Use LLMs and deterministic programs where they excel. Gemini for segmentation/POS, pykakasi for Kana-to-Romaji. Cross-check independent results to find errors for free.

Think through copyright before the first line of code. Different acquisition methods don't change legality. Use .gitignore, stats-only verification, and IAP to keep it a personal tool.

Code at kkdai/song-lingo (lyrics data not in repo). Official docs for Speech generation and Voice design. Reminder for REST users: audio is in steps[].content[], not output_audio.

Comments

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍‌