DEV Community: Evan Lin

[AI in Action] Gemini 3.8 Live Upgrade: Paying Off Technical Debt and How My Test Script Fooled Me Three Times

Evan Lin — Tue, 29 Sep 2026 08:39:05 +0000

Recap

I have a LINE Bot that I use every day, linebot-helper-python. It provides summaries and social media copy for URLs, allows continuous questioning for YouTube links, and handles bookmarks and location queries. One of its features is a LIFF voice assistant: it opens a webpage, connects to the Gemini Live API via WebSocket, and supports push-to-talk or hands-free conversation. The transcribed content is then summarized into a message and pushed back to the LINE chatroom.

In mid-September, Google released Gemini 3.8 Live. After reading the announcement, what I wanted to do was slightly different from what they were trying to sell.

The announcement focused on capabilities: Speech to Speech Index ranked first (82.6 points), Big Bench Audio at 97.7%, automatic detection of 97 languages with the ability to switch mid-conversation, tool calling executing in the background without interrupting the flow, and real-time visual input. There is also a Gemini 3.8 Live Extended Thinking model that can reason while speaking.

My first thought was: Now I can finally remove that preview exception.

The Real Motivation: A Preview Model That Tripped Me Up Twice

In my previous post about agentic video, I mentioned that my intelligent dialogue feature was broken for a while, and no one noticed. The reason was that loader/chat_session.py had gemini-3-pro-preview hardcoded. After that model was removed from Vertex AI, the actual path returned a 404, and the exception handling in that file simply raised the error.

After fixing it, I added a set of guards using tests to enforce that "Model IDs can only appear in config/agent_config.py" and "No preview or experimental models allowed." But those guards had two exceptions:

# Live API only supports gemini-3.1-flash-live-preview; TTS is a separate model family.
# These two places must maintain preview models and are outside the scope of the guards.
EXEMPT_FILES = {"services/voice_live.py", "tools/tts_tool.py"}

The reason for the voice exception was "The Live API only has this one model available." At the time, that was true. But once that sentence is written into a comment, no one checks if it's still valid—the last loophole for the same type of bug sat there in plain sight for months.

The significance of 3.8 Live, therefore, isn't just the capability upgrade. Its model name does not contain -preview, which means that exception can be removed.

Ask the API First, Not the Documentation

This time, I didn't read the documentation first. Instead, I asked the API directly what actually exists:

curl -s "https://generativelanguage.googleapis.com/v1beta/models?key=$KEY&pageSize=200" \
  | jq -r '.models[]?.name' | rg -i 'live'


models/gemini-3.5-transcribe-live
models/gemini-3.1-flash-live-preview
models/gemini-3.8-live
models/gemini-3.8-live-extended-thinking
models/gemini-3.5-live-translate-preview

One command gave me five confirmed model names, which is much more reliable than digging through documentation. gemini-3.8-live is right there, with a clean name.

I also happened to find gemini-3.5-live-translate-preview, which I didn't know about. I was originally thinking about "whether to implement a real-time interpretation mode," and it seems that's a specialized model rather than a brute-forced general model. I'll look into that later.

Next, I queried the metadata:

	`3.1-flash-live-preview`	`3.8-live`
inputTokenLimit	131072	131072
outputTokenLimit	65536	65536
supportedGenerationMethods	`bidiGenerateContent`	`bidiGenerateContent`

The numbers are identical. At this point, I had initial confidence that it would be a "painless swap," but this was just metadata, not behavior.

Pitfall 1: Warning Against a Non-existent Risk

When I first evaluated this, I solemnly noted: "The announcement didn't mention Vertex AI, and the decision for this project is to switch entirely to Vertex, so it's uncertain if 3.8 Live will be available."

It sounded very professional. Then I looked at my own code:

client = live_genai.Client(
    api_key=GOOGLE_AI_API_KEY,
    vertexai=False,
    http_options={"api_version": "v1beta"},
)

vertexai=False. The voice path has used the Gemini API with an AI Studio key from the very beginning; it doesn't use Vertex at all—and 3.8 Live was released on the Gemini API.

I was warning my own repo about a risk that didn't exist in my repo. The project as a whole indeed "switched entirely to Vertex AI," but voice was the exception, and I was the one who wrote that exception.

Cause and Solution: Project-level decision records can become a memory shortcut. "We use Vertex for everything" is a correct summary, but summaries erase exceptions. Checking the source code takes thirty seconds and is much cheaper than reasoning based on impressions.

Pitfall 2: The Control Group Saved Me

After confirming the model existed, I wrote a throwaway script for testing. The key design was: Don't write a separate config; directly import the existing build_live_config() and build_voice_tools() from the project. I wanted to test "Can the configuration I'm currently running work?", not "Can a separate configuration I wrote work?".

The first run looked like this:

=== gemini-3.8-live ===
  [handsfree] {'connected': True, 'audio': True, ..., 'error': None}
  [PTT] {'connected': True, 'audio': False, ...,
               'error': 'APIError: 1007 None. Precondition check failed.'}

The push-to-talk (PTT) mode was blocked. If I had only tested 3.8, I would have concluded: "3.8 Live doesn't support PTT; cannot upgrade."

But I also ran the old model:

=== gemini-3.1-flash-live-preview ===
  [PTT] {..., 'error': 'APIError: 1007 None. Precondition check failed.'}

Exactly the same error. Since both models failed in the same way, it wasn't a model issue; it was a script issue.

The actual error was: In PTT mode, the production environment sends PCM audio, but I sent text for convenience. Inserting text input between activity_start and activity_end is an invalid combination.

After switching to real 16kHz PCM:

=== gemini-3.8-live ===
  [PTT] {'connected': True, 'audio': True, 'out_tx': True, 'in_tx': True,
         'resume': True, 'events': ['audio','in_tx','out_tx','resume','turn_complete']}

Everything was there, and it matched the old model item by item.

Cause and Solution: The only thing I did right this time was keeping a control group. When testing something new, testing the old thing in the same way costs almost nothing, but it allows you to distinguish between "the new thing is broken" and "my test is written incorrectly." These two things look identical.

Pitfall 3: Sine Waves Aren't Human Voices

In the second version of the script, I used math.sin to generate a 180Hz PCM segment as audio input. PTT mode tested smoothly, but both models timed out in hands-free mode.

I almost wrote this off as "hands-free behavior pending confirmation." But a moment's thought revealed why: hands-free mode relies on Gemini's own Voice Activity Detection (VAD), and a pure sine wave isn't speech. The VAD correctly judged that "no one is talking," so it never triggered.

So it wasn't a "detected problem"; it was not detected at all. Two timeouts looked like data, but they were actually two empty spaces.

This later evolved into a checklist. I separated what the script tested from what it didn't:

Tested: Connection, config acceptance, PTT audio round-trip, bidirectional transcripts, resumption handle, turn_complete, tool calling trigger with correct parameters.

Not tested: Real human Chinese speech recognition, automatic VAD in hands-free mode, interrupting while speaking, reconnection after a ten-minute connection recycle, whether the resumption handle actually restores context, Google Search grounding actually taking effect, the second half of slow task delegation pushed back to LINE, voice quality, and latency.

Of the eight items, eight were not tested. The script ran beautifully, but it touched the protocol layer, not what a user would encounter.

Comparison Table

After going through the process, the final comparison looks like this. Both columns are results from hitting the real API using the project's existing config generator:

Validation Item	`3.1-flash-live-preview`	`3.8-live`
PTT (activity signal + PCM chunks)	Audio / Bidirectional Transcripts / resume / turn_complete	Same
`session_resumption`	Yes	Yes
`context_window_compression`	Accepted	Accepted
Bidirectional `AudioTranscriptionConfig`	Yes	Yes
Voice `Aoede`	Yes	Yes
`google_search` and `function_declarations` in separate Tools	Accepted	Accepted
Tool calling actually triggered	Both tool parameters correct	Same
`api_version="v1beta"`	Required	Still required

In other words, gemini-3.8-live can be swapped directly into the existing configuration.

Extended Thinking Has a Required Field

I also tested gemini-3.8-live-extended-thinking, and the connection was immediately blocked:

APIError: 1007 None. Thinking level must be specified for this model.

After adding thinking_config, both LOW and HIGH could connect. So it's not broken; it just has an extra required parameter.

I decided not to adopt it because the current build_live_config() doesn't send thinking_config. If I just swapped the model name, the voice assistant would crash at the connection step. Using it would require modifying the config generator, which is a separate task.

But I couldn't "keep" the knowledge that it shouldn't be set, so I wrote it as a test:

# Live API (bidiGenerateContent) support list. Confirmed by connection test on 2026-09-16.
# Deliberately excludes gemini-3.8-live-extended-thinking: this model mandates thinking_level,
# and connections without thinking_config will be blocked (1007).
LIVE_CAPABLE = {"gemini-3.8-live"}

Last time I wrote "Live API only supports a certain model" in a comment, that sentence expired for months without anyone noticing. This time, I wrote it as an assertion that will fail.

What Changed

The actual code changes were small, about eighty lines across seven files:

config/agent_config.py: Added VOICE_MODEL module constant and AgentConfig.voice_model. A detail here—I deliberately didn't just put it in get_agent_config(), because that function requires GOOGLE_CLOUD_PROJECT, and services/voice_live.py needs the model ID at import time. If I went that route, imports would fail if the project environment variable wasn't set.
services/voice_live.py: Removed the model literal entirely and pulled it from the config.
tests/test_model_config.py: Reduced EXEMPT_FILES from two files to one; included voice_model in the existing "no preview" and "must be in the verified capable list" guards.

The last guard originally looked like this:

def test_voice_live_model_unchanged():
    """Live API only supports gemini-3.1-flash-live-preview; changing it will break the voice assistant."""
    assert VOICE_MODEL == "gemini-3.1-flash-live-preview"

It asserted a string. Strings expire, and when they do, the test is still green—it only gets in your way when you want to upgrade; it doesn't save you when a model is taken down.

By changing it to a list of capable models, it guards against the category of "this value must be a verified Live model," rather than a specific value.

The exception for tools/tts_tool.py remains. I checked, and all three TTS models on the API (2.5-flash-preview-tts, 2.5-pro-preview-tts, 3.1-flash-tts-preview) are still in preview, with no GA versions to swap to. The exceptions were reduced from two to one, but not zeroed out.

Deployment Order: Config First, Code Second

Environment variables were set before merging:

gcloud run services update linebot-helper-python --region us-central1 \
  --update-env-vars VOICE_MODEL=gemini-3.8-live

This step had absolutely no effect at the time. The live version was still running the old image, which had the model hardcoded in voice_live.py and wouldn't even read this variable.

But the order was correct. Merging triggers Cloud Build for automatic deployment. As soon as the new image goes up, it reads the already-in-place configuration, ensuring there isn't a window where "the code is up but the config isn't."

This approach has a side effect that I find more valuable than the upgrade itself: Rollback doesn't require re-deployment.

gcloud run services update linebot-helper-python --region us-central1 \
  --update-env-vars VOICE_MODEL=gemini-3.1-flash-live-preview

If real-world testing feels off, a single command switches it back without waiting for a build. Changing the model ID from hardcoded to an environment variable bought me this flexibility.

Then I Tested It for Real, and the Results Overturned My Observations

There was one thing in the script I always found suspicious. During the tool calling round, the old model would speak a filler phrase before waiting for the tool result, while 3.8 did not.

My judgment at the time was cautious: this might be the "tools execute in the background without interrupting the flow" mentioned in the announcement, or it might just be my loop closing early. Automation couldn't tell; it required a human talking to know if it was an improvement or an awkward silence.

After merging and deploying, I spoke a few sentences via LIFF. The answer: There is no difference from before; it's very smooth.

So that difference doesn't exist to a human ear. It was a byproduct of my loop returning as soon as it got turn_complete. I was observing the shape of my own script.

In this article, my script lied to me three times: the PTT incident made the new model look broken, the sine wave incident made "not tested" look like "tested," and this time made a non-existent difference look like a new capability. None of them were model issues.

296 Passing Tests Don't Prove Voice Works

This is the part I find most worth writing down.

After the upgrade, the tests went from 293 to 296, all green. But this number has almost nothing to do with "whether voice works on 3.8."

tests/test_voice_live.py has 32 tests covering PTT disabling auto-VAD, activity signals, transcript forwarding, tool calling execution, resumption handles, connection recycling, and interruptions. It looks comprehensive. But they all use FakeLiveSession and don't connect to the internet. If the model changes from 3.1 to 3.8, not a single one of these 32 tests will change color.

They test "Is what I'm sending correct?", not "What is the other side returning?".

The three layers of coverage look like this:

Layer	What it tests	Who runs it	Repeatable
Repo test suite (296)	Outgoing config and message format	CI on every push	Yes
Throwaway probe (3)	Protocol layer compatibility with real API	Ran only once	No
Real human conversation	Understanding, smoothness	Me	No

The middle layer is the only automated test that actually touched 3.8, and it's not in the repo, CI doesn't run it, and no one will know the next time a model is taken down.

The newly added guard only blocks "setting it to a known bad value"; it doesn't block "Google taking 3.8-live down"—which is what actually happened last time.

That loophole is still open. Turning the probe into a smoke test that requires a key is the solution; I haven't done it yet.

The code is at kkdai/linebot-helper-python, and this change is in PR #24. The official announcement is Gemini 3.8 Live, which mentions that all AI-generated audio carries a SynthID watermark—I did not verify this.

The announcement spent the most space on real-time visual input, which I didn't touch at all this time—currently, the LIFF getUserMedia is hardcoded to video: false. That will probably be the next post.

[AI in Practice] Shadowing Assessment with Gemini 3.5 Transcribe: Let Song Lingo Listen to You Read Lyrics

Evan Lin — Tue, 29 Sep 2026 08:38:38 +0000

Recap

In the first post, I used Gemini 3.8 Flash TTS to create Song Lingo: paste a YouTube MV URL, Gemini transcribes the lyrics, adds furigana, translation, and grammar, and then a teacher designed with voice design demonstrates pronunciation sentence by sentence. In the second post, I deployed it to Cloud Run and locked it with IAP so only I can use it.

After deployment, I opened a roadmap on GitHub with three items: mobile layout, shadowing score, and flashcards. The mobile layout is finished; this post is about the second item.

The teacher demonstrates, but I never know if I'm pronouncing it correctly. Listening to the teacher ten times is not as good as saying it once and being told where you're wrong. This is a key step in Song Lingo's evolution from "listening to songs and reading lyrics" to "practicing pronunciation."

What is Gemini 3.5 Transcribe

Google released Gemini 3.5 Transcribe on 8/26, which consists of two models specialized for speech-to-text:

Model	Usage
`gemini-3.5-transcribe`	Recorded audio files, transcribed in full
`gemini-3.5-transcribe-live`	Real-time streaming via Live API WebSocket, transcribing while speaking

Key features:

85+ languages, with auto-detection and support for code-switching.
Word-level timestamps: Each word has a start and end time (only available in the full transcription version).
Speaker diarization: Identifies who is speaking each sentence.
Custom vocabulary: Up to 1,000 proper nouns to improve recognition of specialized terms.
Two modes: smart handles self-corrections, removes filler words, and auto-formats; verbatim transcribes word-for-word.

The official blog cites Artificial Analysis: average Word Error Rate (WER) of 2.6% for full transcription and 4.0% for streaming; compared to the previous generation Chirp 3, the time to get final results is 70% faster. Pricing has not been announced.

The actual call method is the same as TTS via the interactions API, but audio files must first be uploaded using the Files API:

uploaded = client.files.upload(file=audio_path, config={"mime_type": "audio/m4a"})
interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": uploaded.uri, "mime_type": uploaded.mime_type}],
    generation_config={
        "transcription_config": {
            "language_codes": ["ja-JP"],
            "mode": {"type": "verbatim", "timestamp_granularities": ["word"]},
        }
    },
)

Many formats are supported; audio/webm happens to be the format for Chrome recordings, and audio/m4a corresponds to AAC recorded by iPhone Safari.

Which Model to Choose

For shadowing scores, there were three candidates:

Model	Suitable for	Use in Shadowing
`gemini-3.5-transcribe`	Transcribing after recording the whole segment	✅ Has word-level timestamps, takes about 5s in testing
`gemini-3.5-transcribe-live`	Real-time streaming	A line of lyrics is only a few seconds; transcribing after finishing is enough. Maintaining WebSockets behind Cloud Run and IAP is much more troublesome.
`gemini-3.8-flash` and other multimodal models	Listening to audio directly to provide text feedback	Results vary every time, no guaranteed word-level timestamps, not suitable as the core for scoring.

I finally chose gemini-3.5-transcribe. As a specialized speech-to-text model, the results are stable, it provides word-level info, and it only uses about 40 tokens per call, not consuming the 100-call daily quota for TTS.

See Real Responses Before Coding

The most expensive lesson from the first post was: the web version of TTS followed the Python SDK's way of reading output_audio, but that field didn't exist in the REST response, burning a whole day's quota in an hour.

So this time, I wrote a rule directly in the roadmap issue: Verify the parsing logic with real API responses before going live.

Test audio was generated using macOS's built-in say command with sentences I wrote myself: one saying "今日は晴れです" (Today is sunny) and another intentionally saying "今日は雨です" (Today is rainy):

say -v Kyoko -o ok.aiff "今日は晴れです"
afconvert -f m4af -d aac ok.aiff ok.m4a # Same AAC format as iPhone recordings

Then I intercepted the raw HTTP content sent and received by the SDK and printed the response structure. I discovered two things not clearly explained in the documentation that would have caused errors if followed blindly.

`output_text` is assembled by the SDK again

Documentation says transcription results are in interaction.output_text. The actual REST response looks like this:

steps[] → { type: "model_output",
            content[] → { type: "text", text: "今日は晴れです。",
                          annotations[] → { type: "word_info", text: "今日",
                                            start_index: 0, end_index: 6,
                                            start_offset: "0.100s", end_offset: "0.400s" } } }

output_text, like output_audio last time, is a convenience field the Python SDK assembles from steps. Knowing this beforehand, I read directly from steps[].content[].

Word positions are calculated in UTF-8 bytes

The start_index for the word "今日" is 0 and end_index is 6, not 0 to 2. Positions are calculated using UTF-8 bytes, where one CJK character takes 3 bytes. If sliced according to JavaScript string indices, Japanese text would be completely misaligned.

These two things can only be known by looking at real responses. This time, spending two API calls meant no more guessing after going live.

How to Determine "Correct Pronunciation"

Use verbatim, not smart

The official smart mode handles self-corrections and removes filler words. While a plus for meeting minutes, it's a minus for shadowing: if you mispronounce and then correct yourself, smart might only keep the corrected version; if you miss a particle, it might fill it in smoothly. Shadowing needs "what you actually said," so use verbatim.

Don't use custom vocabulary

Setting the original text as custom vocabulary seems like it would improve accuracy, but it makes the model biased toward hearing the correct answer, making it harder to catch mistakes. Also, documentation states custom vocabulary and word-level timestamps cannot be used together.

Compare both pronunciation and writing

Transcription results might be in Kanji "今日" or Hiragana "きょう"; both are correct. Comparing only text would mark correct pronunciations as wrong.

So for Japanese, I compare both ways simultaneously; it counts as correct if either matches:

Pronunciation: Convert transcription to Hiragana using pykakasi and compare with the pre-labeled pronunciation.
Writing: Compare text directly to prevent pykakasi from misreading a Kanji.

The comparison itself uses Python's built-in difflib.SequenceMatcher for character alignment, mapping results back to each word. English and Korean are compared word-by-word.

Distinguish between "Mispronounced" and "Missed"

The first version only checked "if the word's characters matched." Consequently, "晴れ" in "今日は雨です" was marked as missed, when it was actually replaced by another word.

After changing to record alignment results for each character (match, replaced, deleted), they can be distinguished:

Scenario	Transcription Result	Judgment
Correct	今日は晴れです。	All correct, 100 points
Replaced a word	今日は雨です。	"晴れ" mispronounced, 75 points
Skipped middle word	今日はです	"晴れ" missed, 75 points
Stopped halfway	今日は	"晴れ" "です" missed, 50 points
Transcribed as Hiragana	きょうははれです	All correct
Katakana with full-width symbols	キョウハ　ハレデス！	All correct
Added filler words before/after	えっと今日は晴れですね	All correct

These are unit tests that don't call the API, allowing the comparison logic to be tested instantly after any modification.

Clarifying Limitations First

Speech-to-text checks " whether others can understand which word you are saying," not fine-grained pronunciation scoring:

Slightly non-standard but still recognizable words will be judged as correct. The model guesses the most likely word, which is the nature of speech-to-text.
Details like long vowels, geminate consonants (っ), and pitch accent are only caught if they are wrong enough to become a different word.

Therefore, the card description explicitly states: "Only checks if the word is recognizable; pitch and vowel length are not within the scoring scope." If more detailed feedback is needed later, the recording and the teacher's demo can be sent to gemini-3.8-flash for a textual explanation, but that's a separate feature.

Pitfall 1: Button says "Recording" before the microphone is ready

The first version's flow was: Click "Shadow" -> Button immediately changes to "■ Stop and Score" -> Request microphone.

The problem is that on the first use, the browser pops up a microphone permission prompt. At this point, the button already says "Stop and Score," but the mic hasn't started recording. If clicked again, the program thinks it hasn't started and requests the microphone again instead of stopping.

Cause & Solution: Added a "Preparing Microphone" state. After clicking, the button shows "Preparing Microphone..." and is temporarily disabled. Only when MediaRecorder actually starts recording does it switch to "Stop and Score."

By the way, the issue originally specified "Hold to record, release to send," but I changed it to "Click to start, click to stop" during implementation because of this permission prompt: if holding, the finger would definitely release when clicking the permission button, making the first attempt fail. Plus, long-pressing on iPhone easily triggers text selection, and saying a whole lyric takes several seconds; clicking twice is more user-friendly than holding.

Pitfall 2: Headless Chrome on Mac can't access the microphone

For end-to-end testing, I've been using Puppeteer with headless Chrome. Chrome has a set of test flags to "use an audio file to fake a microphone":

--use-fake-ui-for-media-stream
--use-fake-device-for-media-stream
--use-file-for-fake-audio-capture=ok.wav

However, getUserMedia just hung without responding. I tried 48kHz wav, not specifying a file, pre-granting mic permissions—nothing worked. It couldn't even get a basic fake device; it seems macOS's microphone permission mechanism blocks headless Chrome.

Cause & Solution: Instead of getting stuck on the test environment, I changed the approach. Before the page loads, I replace getUserMedia to decode a test audio file and play it via Web Audio, creating a real audio stream:

navigator.mediaDevices.getUserMedia = async () => {
  const ctx = new AudioContext({ sampleRate: 48000 });
  const buffer = await ctx.decodeAudioData(testWavBytes.buffer);
  const src = ctx.createBufferSource();
  src.buffer = buffer;
  const dest = ctx.createMediaStreamDestination();
  src.connect(dest);
  src.start();
  return dest.stream;
};

Except for the "system microphone" part, the entire flow—recording (MediaRecorder to webm/opus), uploading, transcribing, comparing, and displaying results—is real. The test result was 100 points, all four words green, only one scoring request sent, and no residual files in the server's temp folder.

The bug from Pitfall 1 was also caught in the first version of this test: the button showed "Stop and Score," but MediaRecorder was never created.

Privacy and Quota

Shadowing involves two sensitive things: my recordings and API quota.

Design	Implementation
Recordings not saved	Only written to the server's temp folder during scoring, then deleted immediately along with the folder (in a `finally` block); not stored in buckets or logged.
Don't trust lyrics from the browser	The server reads the original text and pronunciation itself based on song ID and line number.
No double billing	Only one scoring process at a time; SDK auto-retry is disabled.
Stop when quota is exhausted	After receiving a 429, requests are rejected during the `Retry-After` period without calling Gemini.
Inaccessible to unauthenticated users	The shadowing API is also behind IAP; unauthenticated requests are blocked at the IAP layer before reaching the app, so no quota is consumed.

Sending recordings to Gemini for transcription is necessary for the feature itself, but on the service side, the recording only exists for the few seconds of that call.

Side note: Making PWA installable behind IAP

The first item on the roadmap was the mobile layout: MV fixed at the top, cards swipeable left/right, thumb-accessible bottom control bar, plus a PWA that can be added to the home screen. The layout itself isn't special, but there's a pitfall with PWA behind IAP worth noting.

When browsers fetch the PWA manifest, they don't include cookies by default. Behind IAP, requests without cookies are redirected to the Google login page, so the manifest can't be read, and the PWA can't be installed.

The HTML solution is to add crossorigin="use-credentials" to the <link rel="manifest">. However, looking at the Next.js 16 source code, I found its built-in manifest link only adds this attribute in Vercel preview environments:

crossOrigin: !manifestOrigin && process.env.VERCEL_ENV === 'preview' ? 'use-credentials' : undefined

Cause & Solution: Instead of using Next's built-in app/manifest.ts, I used a regular route to provide the manifest and manually placed the link with use-credentials in the <head>. Icons in the manifest were also embedded as data URLs to prevent the browser from making another cookie-less request.

Results and Benefits

	Figures
Commits this time	2 (Mobile layout & PWA, Shadowing score)
Total repo commits	18
Closed issues	2 / 3 (roadmap: flashcards remaining)
One shadowing score	Approx. 9–10s (upload ~2s, transcription ~4s, rest is Python startup)
Usage per score	Approx. 40 tokens, doesn't consume TTS quota
Calls spent verifying API	5–6 transcribe calls

The workflow for learning a song on mobile is now: View card → Listen to teacher → Press shadow and speak → See which words were wrong → Listen to your recording vs. teacher → Swipe to next line.

Key Takeaways

"See real responses first" must be a rule, not a memory. I explicitly wrote this in the issue and made two API calls before starting. As a result, I knew about output_text and UTF-8 bytes before writing a single line of code.

The officially promoted mode isn't always best for your use case. smart mode is a plus for most scenarios, but shadowing needs exactly what it removes. Think about whether you want "organized results" or "what actually happened" before choosing a mode.

Features that improve accuracy might actually be counterproductive. Custom vocabulary biases the model toward the correct answer, while scoring needs to catch errors.

UI state should reflect real state, not assumed state. When the button says "Recording," the mic isn't actually on yet. This gap is most obvious during the first authorization, which is exactly when users are most likely to click randomly.

When the test environment hangs, change the approach. Instead of fighting macOS mic permissions, replace the "system microphone" segment and let the rest run for real.

Clarify capabilities upfront. Speech-to-text checks "understandability," not pronunciation. Writing this limitation on the screen is more honest than letting users think a 100-point score means perfect pronunciation.

Code is at kkdai/song-lingo, shadowing implementation is in shadow.py and web/app/api/songs/[id]/lines/[index]/shadow/. Official resources: Gemini 3.5 Transcribe Launch Post, Transcribe Documentation.

Hands-on with the official google-cloud-developer plugin: Stop AI agents from hallucinating gcloud commands

Evan Lin — Mon, 28 Sep 2026 15:54:43 +0000

Recently, while browsing Google Cloud's developer environment setup documentation, I noticed something new amidst the usual gcloud CLI, Cloud Shell, and Cloud Workstations:

agy plugin install https://github.com/google/skills/plugins/cloud/google-cloud-developer

agy is the Google Antigravity CLI. But what really made me stop wasn't this new CLI, but the URL following it: Google has started packaging its own product knowledge as agent plugins and publishing them on GitHub.

As it turns out, when I checked my own Claude Code, this plugin was already installed on my machine. So, in this post, we won't bother installing the Antigravity CLI first; instead, we'll use what's already available to see what it actually does.

What's inside this plugin?

Let's take it apart. The entire package is actually quite small: 5 skills, 1 set of routing rules, and 1 MCP server, totaling 1549 lines of markdown with no code at all:

skills/gcloud/ # Safety guardrails and syntax validation for gcloud commands (272 lines + 340 lines of reference docs)
skills/retrieving-developer-knowledge/ # Querying official documentation (103 lines + 182 lines of reference docs)
skills/finding-google-skills/ # Fetching skills from a remote directory on demand (134 lines)
skills/google-cloud-recipe-onboarding/ # Onboarding for a user's first project (228 lines)
skills/google-cloud-recipe-auth/ # Credentials and ADC selection (260 lines)
rules/google-cloud-discovery.md # Skill routing table (30 lines)

Interestingly, four sets of manifests are packed into the same directory:

.claude-plugin/plugin.json # Claude Code
.codex-plugin/plugin.json # Codex
gemini-extension.json # Gemini CLI / Antigravity
plugin.json # Universal format

The universal plugin.json starts like this:

{
  "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
  "name": "google-cloud-developer",
  "version": "1.1.2",
  "author": { "name": "Google LLC" },
  "license": "Apache-2.0"
}

agent-plugins.org is a cross-tool plugin schema. Google didn't create a proprietary format for its own Antigravity; instead, they made one package that four different types of agents can consume. In other words, you can use this content even without installing the Antigravity CLI.

Installation: Three Paths to Choose From

Antigravity CLI (As per documentation)

agy plugin install https://github.com/google/skills/plugins/cloud/google-cloud-developer

Claude Code

claude plugin marketplace add google/skills
claude plugin install google-cloud-developer@google-plugins

Universal

npx skills add google/skills

After installation, you need to enable the API:

gcloud services enable developerknowledge.googleapis.com --project=YOUR_PROJECT_ID

To confirm where it's installed:

$ ls ~/.claude/plugins/cache/google-plugins/google-cloud-developer/
1.1.2

Test 1: Can guardrails stop hallucinated flags?

The core of this plugin is the gcloud skill. Its opening paragraph is very blunt:

All pre-existing knowledge of gcloud commands, flags, flag values, and positional argument syntax is stale and prone to hallucination.

In plain English: The gcloud syntax you (the model) remember is outdated; do not write it from memory. Consequently, it sets a hard rule: before taking action, you must run gcloud help <leaf command>. Furthermore, parent-level help doesn't count; you must validate down to the lowest-level sub-command. It even explicitly forbids using web searches to check syntax; gcloud help is the sole authority.

Is this rule necessary? Let's test it with Cloud Run's scaling flags:

$ for f in --min-scale --max-scale --min-instances --max-instances; do
    gcloud help run deploy | rg -q -- "$f" && echo "$f exists" || echo "$f does not exist"
  done
--min-scale does not exist
--max-scale does not exist
--min-instances exists
--max-instances exists

--min-scale / --max-scale were the syntax for Knative and early Cloud Run, but they are no longer used. However, these strings are everywhere in old articles and Stack Overflow answers online, so models are very likely to spit out a command that looks reasonable but fails immediately upon execution. By running gcloud help first, this problem disappears.

By the way, gcloud run deploy doesn't actually have a --dry-run flag; it only has --async. The skill's rule is: "If the help output lists --dry-run or --validate-only, you must run it once first." If it's listed, run it; if not, skip it.

Test 2: How much context does a blind list consume?

There is a rule in the skill that I initially thought was a bit tedious:

DO NOT execute any list command without including at least one data reduction flag (--limit, --filter, or --format).

A quick measurement shows why. I have a project running 52 Cloud Run services:

# A. Blind list
$ gcloud run services list --project=YOUR_PROJECT_ID --format=json | wc -c
281918

# B. Projecting only the two required fields (still 52 services, none missing)
$ gcloud run services list --project=YOUR_PROJECT_ID \
    --format="json(metadata.name,status.url)" | wc -c
7963

281,918 bytes vs. 7,963 bytes—a 35x difference. In terms of tokens, that's roughly 70k vs. 2k. In other words, a list command without --format would consume 70k tokens, and 97% of what was cut out consists of annotations, conditions, revision templates, and a bunch of timestamps.

The skill also teaches a very practical "probe schema before querying full data" trick:

gcloud <GROUP> <RESOURCE> list --limit=1 --format=json

First, grab one entry to see what the JSON looks like. After confirming the field paths, then construct the --filter and --format.

Other execution rules follow the same logic:

Run only one command at a time; no && chaining.
Prohibit pipes, $( ), and redirection (the reason being to keep commands readable for human review).
Always add --project=<PROJECT_ID>; do not rely on the default values of the active config.
Always add --quiet to avoid getting stuck on interactive confirmations in environments without a TTY.

Test 3: Which commands is it forbidden to run?

This is the section I find most worth copying. The skill contains an explicit denylist of operations that must never be executed automatically without explicit human authorization:

Any changes to IAM policy / role / binding (risk of privilege escalation or locking oneself out).
gcloud * delete (irreversible resource destruction).
gcloud billing * (risk of billing spikes).
gcloud organizations * (organization-level settings affect everyone).
gcloud kms * (potential to permanently lock data).
gcloud infra-manager deployments apply (automatically running IaC might delete a whole batch of resources).
Proactively enabling APIs is also on the list, the reason being that enabling an API might trigger billing.

I find that last one particularly interesting. Enabling an API sounds harmless, but the skill explicitly states "assume necessary APIs are enabled"; if they need to be enabled, go back and ask the human. This level of conservatism shows they have seriously thought about what happens when an agent is connected to a production environment.

Test 4: Developer Knowledge—Querying docs instead of memory

The plugin includes an MCP server:

{
  "mcpServers": {
    "developer-knowledge": {
      "type": "streamable-http",
      "url": "https://developerknowledge.googleapis.com/mcp"
    }
  }
}

Here I encountered the first pitfall. Just because an MCP server is declared in the configuration file doesn't mean it actually connects. The tools listed on my end were only authenticate and complete_authentication; the answer_query and search_documents mentioned in the documentation didn't appear at all, meaning it was stuck in the authentication process.

Interestingly, the skill itself had already anticipated this, stating in black and white:

A declared server is not always a connected server. Some clients cannot complete the MCP handshake with this server and expose no answer_query, search_documents or get_documents tool at all, even though the plugin declares one.

It then provides a REST fallback. You can use the gcloud credentials you already have to make the call, without needing to apply for a separate API key:

curl -s -X POST "https://developerknowledge.googleapis.com/v1:answerQuery" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "X-Goog-User-Project: $(gcloud config get-value project)" \
  -H "Content-Type: application/json" \
  -d '{"query": "How do I deploy a container to Cloud Run with gcloud?"}'

HTTP 200, and the returned data is more sophisticated than I expected:

$ jq '.answer | keys' response.json
["answerText", "citations", "references"]

$ jq '.answer.references | length' response.json
10

$ jq -r '.answer.references[].documentReference.documentChunk.document.uri' response.json | sort -u
https://docs.cloud.google.com/run/docs/deploying
https://docs.cloud.google.com/run/docs/quickstarts/deploy-container
https://docs.cloud.google.com/sdk/gcloud/reference/run/deploy
https://docs.cloud.google.com/run/docs/configuring/services/containers
https://docs.cloud.google.com/run/docs/create-jobs
...

It doesn't just provide an answer; it attaches 10 official documents as sources, and citations use character ranges to map to the reference index:

{"startIndex": 447, "endIndex": 505, "sources": [{"referenceIndex": 3}, {"referenceIndex": 7}]}

Characters 447 to 505 in the answer come from the 3rd and 7th documents. With this level of granularity in citations, verifying the truth of the answer becomes much easier.

To query precise syntax, use another endpoint. I tested it with Cloud Run IAM permission strings:

curl -s -G "https://developerknowledge.googleapis.com/v1/documents:searchDocumentChunks" \
  --data-urlencode "query=cloud run invoker IAM permission run.routes.invoke" \
  --data-urlencode "pageSize=3" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "X-Goog-User-Project: $(gcloud config get-value project)"

The retrieved permission strings:

run.routes.invoke
run.services.create
run.services.delete
run.services.get
run.services.list
run.services.update

These service.resource.verb format strings are where models are most likely to miss a character or add an extra 's'. Checking them is much more reliable than guessing.

A small pitfall to note: the skill's own fallback documentation says the response should be read from documentChunks[].content, but the actual field returned is results[].content. This minor discrepancy is telling in itself. Even official documentation can be out of sync with the actual API, which is why the "verify before acting" rule is indeed useful.

Test 5: How 137 skills avoid consuming context

finding-google-skills solves another problem: Google's skill directory is massive, but it's impossible to preload everything.

Its approach is to keep only a single page routing table and fetch the remote directory only when actually needed:

$ curl -sSL https://raw.githubusercontent.com/google/skills/main/index.json -o index.json
$ jq '.skills | length' index.json
137
$ wc -c index.json
84476

137 skills, 84KB. Stuffing it all into the context is too wasteful, so the skill teaches you to filter with jq before reading:

$ curl -sSL https://raw.githubusercontent.com/google/skills/main/index.json \
  | jq -r '.skills[] | select((.name+" "+.description)|test("cloud run";"i")) | .name'
cloud-run-basics
cloud-run-alert-configuration
cloud-monitoring-metric-selection
google-cloud-global-frontend-configuration
...

Select up to three of the most relevant ones, and only then fetch their entrypoints. This "remote directory, load on demand" pattern is, in my opinion, the most reasonable solution for handling a large number of skills currently.

One more detail: this skill explicitly forbids using curl -k to bypass certificate verification when authentication fails, even specifically naming PowerShell's ServicePointManager callback. The reason given is excellent: "You are about to execute commands based on what you fetch; an unverified directory is worse than no directory at all."

Five Practical Scenarios

Scenario 1: Connecting an agent to production, but only allowing it to look, not touch

This is the one I wanted most. Previously, I didn't quite dare point Claude Code at a production project because it might suddenly decide to "clean up" a batch of resources. With the denylist, the boundaries become very clear.

Help me check which Cloud Run services in the production project haven't had a new revision in over three months.

It will list, filter, and organize the data into a table for you, but it will stop and ask when it encounters a delete operation. What it saves is the time spent flipping through documents and assembling commands; the decision-making power remains in your hands. This boundary is very well-defined.

Scenario 2: Starting a GCP project from scratch and deploying a LINE Bot to Cloud Run

I want to start a new GCP project to deploy a LINE Bot. Help me go through the process from creating the project, linking billing, enabling APIs, to deploying to Cloud Run.

google-cloud-recipe-onboarding will guide you through the entire sequence: billing → project → enabling APIs → deployment. It verifies syntax before each command and will definitely stop to ask you before enabling APIs (because it's on the denylist). This is perfect for people who only start a new project once in a long while and have to re-google everything each time.

Scenario 3: Credential Hell—Works locally, but 403 on Cloud Run

This is probably the most common "hitting a wall" scenario in GCP.

My service runs fine locally, but after deploying to Cloud Run, calling Vertex AI keeps returning a 403. Help me clarify which credential is the issue.

google-cloud-recipe-auth is responsible for clarifying the relationships and applicable scenarios for user credentials, ADC, and service accounts, while Developer Knowledge is responsible for looking up the precise IAM permission strings. Combined, these two are much faster than digging through five-year-old answers on Stack Overflow.

Scenario 4: Cloud Resource Inventory and Billing Health Check

List the names, URLs, and last deployment times of all Cloud Run services in this project, and identify those that might no longer be in use.

The key here is that 35x difference measured earlier. Without guardrails, the raw JSON of 52 services would be enough to fill the agent's context with junk data before it even begins the actual analysis. With --format projection and the habit of "probing schema with --limit=1 first," 52 services only consume 2k tokens, allowing it to handle several times more resources.

Scenario 5: Avoiding the model's old memories when asking about new features

For products that iterate very quickly like Gemini API, Cloud Run, and Firebase, the model's training data is bound to be outdated.

How do I set min instances and startup CPU boost in Cloud Run now?

With Developer Knowledge, the answer will include links to official documents and character-level citation ranges. More importantly, if it can't find the information, it will say so directly:

Presenting recalled documentation as a retrieved result is the worst available outcome, because nothing in the reply distinguishes it from a real lookup.

"I didn't find it; this is an answer from memory" is far more valuable than a flag that sounds smooth but is incorrect.

Summary

From my testing, the things that left the deepest impression on me were actually two things unrelated to functionality.

First, the entire package is 1549 lines of markdown, without a single line of code. All security is implemented through text-based rules: verify syntax first, limit output, explicit denylist, and honest reporting of failure. This means you can also use the same method to write your own team's runbooks and security rules as skills: the barrier to entry is much lower than imagined.

Second, four sets of manifests are squeezed into the same directory. Google didn't create a proprietary format for Antigravity; instead, they made the same package usable across Claude Code, Codex, and Gemini CLI. Skills are becoming a common format that isn't tied to a specific tool.

If you have a GCP project and are using a coding agent, this plugin takes about five minutes to set up. Just for the fact that it won't hallucinate flags or touch IAM on its own, it's well worth it.

Reference Links

[AI in Action] Gemini Agentic Video: 4 Hidden Prerequisites and My LINE Bot Integration Process

Evan Lin — Sat, 26 Sep 2026 17:26:14 +0000

Background

I have a LINE Bot that I use every day, linebot-helper-python. If you send it a URL, it returns a summary and social media copy for four platforms; if you send a YouTube link, it returns a video summary. It also handles bookmarks, location queries, voice assistants, and more. It runs on Cloud Run and uses Vertex AI.

In late August, Google published a post Introducing agentic video in Gemini. After reading it, my first thought was: this could allow my bot to do something it currently can't—ask questions about a video, rather than just providing a summary.

As it turned out, the parts not mentioned in the announcement were more worth documenting than the parts that were.

What is Agentic Video?

Originally, Gemini's video reading was "static": no matter what you asked, it would cram every frame and every audio segment into the context at a fixed sampling rate. For a two-hour video, the video itself could take up hundreds of thousands of tokens, even if you just wanted to ask, "Did he mention pricing?"

The Agentic mode flips this: it lets the model decide which segments to load. It first scans the transcript, determines that the answer might be around 1:24, and then only pulls in the frames for that specific segment.

The official announcement cited figures for long-form video scenarios: 88% fewer tokens, 66% lower costs, and 7% higher accuracy. To enable it, you just add a parameter to the Part:

video_part = types.Part(
    file_data=types.FileData(file_uri=..., mime_type="video/mp4"),
    media_processing="AGENTIC", # or "STATIC"
)

Three models are supported: gemini-3.7-flash, gemini-3.6-flash, and gemini-3.5-flash-lite. Input sources can be YouTube URLs, Cloud Storage URIs, or embedded base64.

What can it actually do?

I tested it with the two-hour Google I/O ‘25 keynote, asking "Did he mention pricing? If so, please give me the timestamp." The response was:

The video mentions information related to pricing and subscription plans. It mainly appears around 1:24:40 to 1:26:10 in the video...

It then accurately described the differences between the Google AI Pro and Google AI Ultra plans. When I asked where the Android XR glasses segment was, it replied 1:36:31 to 1:50:30, and the content matched perfectly.

This ability to precisely locate information within a two-hour video is where I think the real value of this feature lies. Anyone can do summaries, but "when in this long video did he talk about X" is something few tools currently do well.

Prerequisites: Four parameters, missing one won't trigger an error

This was the most expensive lesson this time, so I'm putting it first.

To make agentic video actually work on Vertex AI, four things must be true simultaneously:

#	Condition	What happens if missing
1	`api_version="v1beta1"`	Agentic is only available in v1beta1; using `v1` silently falls back to STATIC
2	`media_processing="AGENTIC"`	Defaults to STATIC; if not set, it's not enabled
3	Model is on the supported list	Unsupported models silently downgrade to STATIC
4	`thinking_level` is set	See the next section; this is the trickiest one

The keyword is silent. If any of these are missing, the API will return a 200, the answer will still be generated, and everything will look perfectly normal—except for the bill. No error message will tell you that agentic mode didn't take effect.

There are also two environmental prerequisites:

SDK Version: The media_processing field was added in google-genai 2.20.0. I verified this by downloading versions one by one; it's not in 2.19.0, but it is in 2.20.0. If the version is too old, this parameter is discarded as an unknown field, again without an error.
Vertex Exclusive: The multi-turn conversation examples in the official documentation are written for the Gemini Developer API. The behavior on Vertex is different, as I'll explain later.

My approach was to write a test for each of these four items, asserting the parameters sent to the SDK:

def test_thinking_level_is_low(captured):
    youtube_tool.summarize_youtube_video(VIDEO_URL)
    config = captured["call_kwargs"]["config"]
    assert config.thinking_config is not None
    assert str(config.thinking_config.thinking_level).upper().endswith("LOW")

The only reason these four tests exist is that "if these four things are wrong, the program still runs, the answer still comes out, but the cost silently changes." This kind of bug cannot be caught by manual review.

Pitfall 1: Video context cannot be preserved on Vertex AI

My original design was this: the user pastes a link to get a summary, clicks "Ask about this video" to enter Q&A mode, and then every subsequent follow-up question uses the context from the previous round, avoiding the need to re-process the video.

The official documentation indeed says this: the response will carry tool_call / tool_response parts; put them back into the history as-is, and the next round won't need to re-process the video.

In practice, I couldn't get them.

The parts returned by Vertex only contained a bare thought_signature, with no tool_call or tool_response. Passing the history back resulted in:

400 INVALID_ARGUMENT: Invalid thought signature.

I tried three different serialization methods, all failed. Then I tried not serializing at all and passing the response object back directly, which also failed. So the problem wasn't my serialization; it was the platform.

After setting thinking_level to LOW, the error stopped, but the tool_use_prompt_token_count for the second round was 2,163, exactly the same as the 2,163 in the first round. The video was completely re-processed; no context was preserved.

Cause and Solution: That multi-turn mechanism was written for the Gemini Developer API. Since it can't be preserved on Vertex, don't pretend it can: switch to a stateless re-query approach where every question is an independent call.

This decision actually made the implementation simpler: no need to serialize history, no need to handle thought signatures, and the session only needs to store three fields: "which video is this user currently asking about." The complex version I originally planned was scrapped.

By the way, I was originally worried that "history serialization might exceed the Firestore single-document 1 MiB limit." In reality, it was only 1.5–7 KiB, so it was never an issue. The things you spend time worrying about are often not where the real problems occur.

Pitfall 2: Thinking tokens are the main cost driver, and they are unstoppable

I had to correct my causal inference three times to get this one right, and the process was more interesting than the conclusion.

Version 1 Conclusion (Wrong)

Initial spike, same 10-minute video:

Mode	in	out (incl. thinking)	Cost
STATIC	54,546	291	$0.0171
AGENTIC, no thinking_level set	2,216	37,574	$0.0946
AGENTIC + `thinking_level="LOW"`	2,216	638	$0.0023

It seemed clear: agentic mode removed the video from the input (54,546 → 2,216), but the model used "thinking" to navigate, and thinking tokens are billed as output ($2.50/M, which is 8.3x the input price). So, not setting thinking_level was 5.5x more expensive than static, but setting it made it 7.4x cheaper.

Then I ran a two-hour video, also with LOW set, and the thinking tokens were 359,961, costing $0.91 for a single call.

My conclusion: "On long videos, thinking_level is ignored."

Version 2 Conclusion (Also Wrong)

During the implementation phase, the agent responsible for that task ran a validation as I requested and hit a threshold, stopping to report: for the same video and the same settings, the thinking tokens for five consecutive runs were:

0, 0, 34911, 35122, 37410

It even pulled the DEBUG logs to confirm that every request sent indeed carried v1beta1 + AGENTIC + thinking_level=LOW, so it wasn't a client-side omission.

So it wasn't the video length—I had run each setting only once and mistaken sampling noise for causality.

The agent proposed a hypothesis: it might be related to prompt complexity, because the previous batch used a one-sentence short prompt, while the problematic batch used the official long prompt. Reasonable, so I had it re-test with the official prompt.

Version 3 (The Reality)

27 calls, 6 times for each of the three settings plus 3 times for the Q&A path:

Setting	thinking peak	tool_use>0	Average Cost
`thinking_level="LOW"`	0/6	✓	$0.00146
`thinking_budget=0`	0/6	✓	$0.00137
Not set at all	0/6	✓	$0.00137
`STATIC`	0/6	✗ (=0)	$0.01659

All zero peaks. But the previous batch using the same official prompt had peaks in 3 out of 4 runs.

So prompt complexity wasn't it either. The behavior of the three client-side settings was identical; the only difference was when they were called.

Cause and Solution: This is server-side non-determinism, and the client has no leverage to control it. I also confirmed two things: thinking_budget and thinking_level cannot be used together (the server returns a 400); STATIC is the only deterministic option, but its tool_use_tokens is 0. That's not "the same feature in a stable mode," it's turning off agentic mode entirely.

My decision was to keep thinking_level="LOW" and add a warning log:

# In testing, thinking tokens only fall into two groups: ~0 or ~35,000-37,000, with no values in between.
# The cause is server-side non-determinism, which no client setting can prevent.
THINKING_TOKENS_WARN_THRESHOLD = 5000

I kept LOW not because it's more stable (it isn't), but because there was no measurable difference between the three settings, and changing it would mean swapping verified behavior for unverified behavior. The truly valuable reinforcement was turning that 60x cost event from something invisible into something searchable in Cloud Run logs. If you can't stop it, at least make it visible.

So the honest cost picture is: normally about $0.0014 per call, with unpredictable and unpreventable peaks about 60x higher. The "66% cost reduction" mentioned in the announcement holds true when there's no peak, but flips when there is.

The thinking_level parameter in my implementation is required, with no default value:

def _generate_video(youtube_url: str, prompt: str, *, thinking_level: str) -> dict:

Giving it a default value is just an invitation for someone to omit it, and omitting it leaves no trace.

Integrating into the LINE Bot: What the feature looks like

The flow is as follows:

User pastes a YouTube link
  → Receives summary + social media copy (existing feature)
  → A new button appears: "🎬 Ask about this video"
  → After clicking, the user types a question: "Did he mention pricing?"
  → Answer comes back with timestamps
  → User pastes a new URL → Automatically exits video mode

Three new modules, each independently testable:

tools/youtube_tool.py (modified) — The only place communicating with the Gemini video API, with the four required parameters centralized in one function.
services/video_qa.py (new) — Remembers "User → Video" mapping, with a 30-minute TTL.
services/usage_meter.py (new) — Records tokens and costs for each call.

main.py only adds two integration points: a message interceptor and a postback branch.

Decision order is the behavioral contract

When a user types in video mode, the decision order is:

Exit condition check → Quota check → Call Gemini

The exit check must come before the quota check. If a user pastes a new URL to change the topic, they shouldn't be blocked because their video Q&A quota is exhausted. Those are two unrelated things. If the order is reversed, a person could get stuck in a mode where every message is rejected and they can't get out.

This rule has its own dedicated test because it's a contract, not an implementation detail.

Use heuristics for exiting mode, not LLM

To determine if a user wants to leave video mode, I used the simplest method: if the message contains a URL or starts with /, exit.

I deliberately avoided using an LLM to judge intent because that would require an extra Gemini call for every message, and the cost of a false positive is low (the user just asks again). It's not worth the cost and latency.

There's a detail worth noting in the implementation: this check directly calls the same find_url() used by the main path. Initially, I wrote my own regex, but during review, someone suggested broadening it to support URLs without schemes (www.youtube.com/...). I checked what the main path actually used and found its regex couldn't catch those either: broadening it unilaterally would only create a discrepancy: video mode would exit, but the main path wouldn't treat it as a URL, leaving the user with a useless chat reply instead of a useless video reply.

By switching to the shared function, the two are always consistent; if find_url is broadened in the future, video mode will automatically follow suit.

Development Process: Using subagents to catch my mistakes

For this implementation, I used Claude Code's subagents: a plan split into 8 tasks, with each task assigned to a fresh agent for implementation, followed by another agent for review. I only acted as the coordinator and arbitrator.

The benefit of this arrangement is that the reviewer doesn't have the implementer's attachment. An agent that just wrote the code is likely to think it's correct; a reviewer agent only sees the diff, the requirements, and the report, without the baggage of "I thought about this for a long time."

The things they caught were far more valuable than I expected.

They caught three of my incorrect causal inferences

As mentioned in the thinking token section, the first correction happened when the implementation agent hit the threshold I set and stopped to report (I wrote "stop if you see over 30,000, don't just commit silently" in the instructions, and it followed them). The second was when it proposed the prompt complexity hypothesis and designed an experiment to rule it out.

The third was the most interesting. While updating the documentation, I replaced the debunked theory with "gemini-3.5-flash-lite is the only model with stable costs." The reviewer agent pointed out that the 27 non-deterministic experiments were run on 3.5-flash-lite, and the peaks were measured on it, so that sentence directly contradicted the paragraph it was citing.

I had replaced one baseless claim with another baseless claim, and I did it three times (in the spec, the config file comments, and the test docstrings).

It caught a Critical bug that missed eight rounds of review

The final branch review caught one thing: the entry button for the entire feature wouldn't even show up.

The button was attached to a carousel message, but four text messages were appended afterward. LINE only renders the quickReply of the last message, so the button was buried. A user pasting a YouTube link would receive the summary and four pieces of copy, but see no button.

Eight rounds of task reviews had passed it because each round only saw "the button is correctly attached to the carousel"; no one saw that four more messages would be added later. The tests missed it too—those tests called the handler directly, skipping the message assembly part.

This is something a single-task review is structurally unable to see; it only becomes visible when the entire path is laid out.

After fixing it, I verified it again with mutation testing: moving the attachment back to the original position caused only the new multi-URL test to fail, while the original single-URL test still passed, which is exactly why it couldn't catch the bug.

Mistakes I made myself

No matter how good the process is, it can't stop me from writing the wrong things in the instructions:

In the design document and instructions for the agents, I wrote "if reply fails, it will switch to push," and even repeated it twice in conversation. In reality, that mechanism doesn't exist: I had misread a sentence in a docstring describing "message overflow handling" as "failure fallback" and never verified it. The final branch review caught it.
A code snippet I provided had an UnboundLocalError where a variable was only assigned in one branch. The implementation agent corrected it.
I told it to add the entry button in a place that didn't exist: that postback handler was dead code; nothing in the repo would generate the corresponding button. It found the actual reachable path itself and attached it there.
I requested "reverting numerical rounding" without first calculating how it related to existing test tolerances, causing that task to go in a circle. The implementation agent showed me the math instead of just picking a side.

It's a bit embarrassing to write down, but these are exactly the real outputs of this process: every single error was caught before merging.

Results and Benefits

	Figures
Commits	18
Tests	192 → 275 passed
Converged model literals	28 places
Fixed online failures	1 (Smart Dialog 404)
Actual cost	Approx. $6–8 (mostly on three rounds of cost measurement)

I also completed an item that had been on the roadmap for a long time: token and cost logging. This was originally "do it when there's time," but after the thinking token pitfall, it became a necessary component: since peaks can't be prevented, you must at least be able to see that they happened after the fact.

A few things I'm taking away

Check constraints first, then design. This time, almost every design decision was forced by constraints: Vertex not preserving context forced stateless queries, uncontrollable thinking tokens forced warning logs, and the inability to use thinking_budget and thinking_level together scrapped an entire option. It's much easier to check constraints thoroughly before starting than to design first and hit a wall later.

A single observation is not a conclusion. My most expensive mistake was taking one observation, making a causal judgment, and then writing that judgment into the design document, where it spread to code comments, environment variable descriptions, and decision records. By the time I realized it was wrong, it took three rounds just to clean up the residue—and I even managed to generate new incorrect theories in the middle. Overturning a conclusion is much more work than establishing one.

Tests guard bug classes, not strings. When the gemini-3-pro-preview deprecation broke a flagship feature, if I had just changed the model name, it would have happened again next time. By changing it to "no preview models allowed," that entire class of problem was truly blocked.

Silent failures deserve dedicated tests. For those four required parameters, if any one is wrong, the program still runs and the answer still comes out. This won't show up in error logs or be caught by code reviews; only the bill will tell you—and by the time the bill tells you, a month has usually passed.

The code is at kkdai/linebot-helper-python, and this implementation is in PR #20. The official documentation is Video understanding, but that version is for the Gemini Developer API; if you are using Vertex AI like me, treat the multi-turn conversation section as a reference, not a guarantee.

[Steam Deck Guide] Fix Windows .exe Patch Path Errors: The Ultra-Stable "Relay Transfer Method"

Evan Lin — Sat, 26 Sep 2026 17:25:46 +0000

When playing various PC games on the Steam Deck, you often encounter situations where you need to install "community localization patches," "fan-made fix updates," or "MOD expansion packs." However, many patches released by developers or localization groups are packaged as Windows .exe installers that require the player to manually select the game directory during execution.

This often leads to a major pain point under Steam Deck's SteamOS (Linux-based architecture): The installer's folder selection window simply cannot find where the game is installed.

This article will analyze the core cause of this problem and share a "Relay Relocation Method" that is currently the most intuitive, has the highest success rate, and does not require entering complex commands.

Why do patch programs get "lost"?

SteamOS uses a compatibility layer (Proton or Wine) to simulate a Windows environment to run programs. Two underlying mechanisms easily cause installers to get stuck:

Hidden Folder Restrictions: By default, Steam installs all games in /home/deck/.local/share/Steam/steamapps/common/. In Linux systems, folders starting with a dot (.) are hidden folders. The "Browse Folder" browsers built into most Windows patches cannot recognize or display these hidden paths.
Sandbox Permission Isolation: If you are used to using Bottles or certain Flatpak applications to launch .exe files, these tools are subject to sandbox permission controls by default and cannot directly read deep system files of the user.

Core Solution: Three-Step Relay Relocation

Instead of fighting with hidden folder settings and Linux commands, the cleanest approach is to first move the game folder to a non-hidden public directory (such as Downloads or Desktop), complete the patch injection, and then overwrite it back to the original path.

Step 1: Copy the game directory to a relay path

Long-press the power button and switch to Switch to Desktop.
Open the Steam Library, right-click on the target game ➔ Manage ➔ Browse local files.
The file manager (Dolphin) will open the directory where the game is located. Click the address bar to go up one level back to the common folder.
Right-click on "the folder for that game" and select Copy.
Enter Downloads or Desktop from the left shortcut bar, right-click in a blank space, and select Paste One Folder.

Step 2: Run the patch and select the path via the Z: drive

Launch the patch .exe via Bottles, Protontricks, or by adding the patch as a non-Steam game.
When the installer pops up the "Please select the game installation directory" window, click open My Computer ➔ Z: drive (Z:\ corresponds to the root directory of the Steam Deck).
Expand the following public paths in order:
- Z:\home\deck\Downloads (or Desktop)
After selecting the folder, press confirm to start the patch writing process. Since the path is completely transparent, the installer can successfully extract the replacement files into it.

Step 3: Overwrite back to the original path and confirm the startup file

After the patch installation is complete and the window is closed, return to the Downloads or Desktop directory from earlier.
Enter the patched game folder, press Ctrl + A to select all files, and right-click to select Copy.
Return to the original Steam game folder (Steam Library ➔ Right-click game ➔ Manage ➔ Browse local files).
Right-click in a blank space and select Paste. When the system prompts for overwriting, select Overwrite / Write Over (Overwrite All).
Critical Check: Some patches generate a separate executable after installation (e.g., Game_Patch.exe or Game_ZH.exe). If this is the case, please back up and rename the original game launcher (e.g., Game.exe) and rename the new executable generated by the patch back to the default Steam startup filename.
Clean up temporary files: Once everything is confirmed to be working, you can delete the relay folder in Downloads or Desktop.

Notes and Troubleshooting Tips

Linux Case Sensitivity: Linux systems treat Data and data as completely different directories. If the folder extracted from the patch has different capitalization than the original game, please manually merge the contents to avoid generating dual folders that prevent the game from reading the patch resources.
Animation and Encoding Compatibility: If you encounter missing text boxes, missing Japanese characters, or a black screen during opening animations after patching, change the game's Properties ➔ Compatibility in Steam to use GE-Proton (which can be installed via ProtonUp-Qt in the Discover store). This usually solves most issues with custom video decoders.

By using this "Relay Relocation" technique, you can accurately complete installations on the Steam Deck and enjoy the best gaming experience, regardless of how old the interface or how tricky the path of a Windows one-time patch program may be.

[AI in Practice] Gemini 3.8 Flash TTS Launch: I built a "Learn Japanese with MVs" Web App and burned through my daily quota.

Evan Lin — Sat, 26 Sep 2026 17:25:33 +0000

Recap

Every time I see a new Gemini feature, my first thought is "Can I connect it to my LINE Bot?"

The Gemini API release notes from 9/22 stated that Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS are officially launched (GA), and the official blog simultaneously posted Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS. As usual, I listed a bunch of LINE Bot ideas: bedtime stories in parents' voices, turning group chats into radio dramas, morning dual-host podcasts...

Halfway through the list, I realized that what actually struck me most wasn't a bot, but the fact that "directing tone sentence-by-sentence" is particularly suitable for teaching pronunciation. So the topic changed to a Web App:

Pick a song, get the lyrics, and translate them into Chinese simultaneously.
If it's Japanese or Korean, attach the phonetic transcription, then transform into a language teacher to teach you how to read it sentence by sentence.
Turn it into a "learning languages through songs" tool.

As it turned out, there was nothing to complain about regarding the quality of the TTS itself; what really took time were the things surrounding it that weren't in the documentation.

What is Gemini 3.8 Flash TTS

Two models were released in this GA:

Model	Positioning
`gemini-3.8-flash-tts`	Flagship model, emphasizes vocal performance and character creation, allows sentence-by-sentence performance control
`gemini-3.8-flash-lite-tts`	Cheap, fast, suitable for high-volume generation

Compared to the previous generation, there are three things I find truly useful:

Voice design: Generate a voice using a text description, for example, "a 60-year-old British-accented astronomer with a warm tone." The generated voice gets a voice_id for repeated use.
Voice replication: Replicate a person's voice using a 10–30 second recording. The prerequisite is that the voice owner must record a consent statement, and the output audio will carry a SynthID watermark and C2PA markers.
Sentence-by-sentence performance control: Every segment of text can include a style (e.g., "speak slowly, pronounce every syllable clearly"), and it supports tags like <laughs>, <sigh>, as well as dialogue between up to two people.

The actual call structure is different from the past generate_content, moving to two sets of APIs: interactions and voices:

# Design a voice using text description (one-time, save for reuse)
voice = client.voices.create(
    store=True,
    voice={
        "model": "gemini-3.8-flash-tts",
        "type": "prompted",
        "display_name": "Song Lingo Japanese Teacher",
        "gender": "female",
        "language_code": "ja-JP",
        "prompted": {"input": "A warm, patient Japanese language teacher in her early 30s from Tokyo..."},
    },
)

# Use this voice to read a sentence, with style controlling tone and speed
interaction = client.interactions.create(
    model="gemini-3.8-flash-tts",
    input=[{
        "type": "user_input",
        "content": [{
            "type": "text",
            "text": "こんにちは。今日はいい天気ですね。",
            "annotations": [{"type": "speech_metadata", "style": "speaking slowly and clearly"}],
        }],
    }],
    response_format={"type": "audio"},
    generation_config={"speech_config": [{"voice": voice.id}]},
)

A few technical details to know beforehand that will save a lot of trouble:

Item	Specification
Output Format	WAV, 16-bit PCM, mono, 24 kHz; also supports streaming
Languages	Over 100 types, including Simplified Chinese, Traditional Chinese, Cantonese; Taiwanese was not seen
Custom Voice Limit	200 per project, kept for 1 year; or choose self-managed `voicekey_`, valid for 7 days
Voice replication Regional Restrictions	Not available in Illinois, Texas, EEA, UK, Switzerland, India
Price	Not mentioned in announcements or docs
Tier 1 Quota	100 requests per day (this becomes the main character later)

Things to Handle Before Starting: Lyrics Copyright

"Enter song name, automatically crawl lyrics" is the most intuitive way, but it gets stuck in two places:

Copyright: Lyrics sites pay for licenses to display them. Crawling them to display full text and adding translations (legally considered a derivative work) carries infringement risks if launched publicly.
Technical: Even using Gemini with Google Search, Gemini has mechanisms to prevent reproducing copyrighted text, often returning finishReason: RECITATION.

Then I thought of another way: Throw the YouTube MV URL directly to Gemini and ask it to transcribe the lyrics. Technically feasible, as the Gemini API accepts public YouTube URLs.

But let's be clear: Obtaining lyrics in a different way doesn't make it legal. Whether crawled from a site, typed manually, or transcribed by AI from an MV, it's the same protected text. What determines the risk is how you use it.

So my decision was: This is a personal learning tool; lyrics only exist in the local output/ folder, and this folder was added to .gitignore from the first commit, never appearing on GitHub. During development, I also required Claude Code to only print statistics when verifying data, never printing lyrics to the terminal or writing them to any logs.

Architecture: Three Python Scripts + One Next.js

YouTube URL
  ─transcribe.py─▶ Lyrics and timeline
  ─annotate.py──▶ Phonetics, translation, word breakdown, grammar points
  ─speak.py────▶ Teacher demonstration (normal / slow)
  ─Next.js─────▶ Learn sentence-by-sentence with MV

Transcription: Prioritizing On-Screen Subtitles for Gemini

transcribe.py sends the YouTube URL as file_data to gemini-3.8-flash, using structured output to require start/end times, original text, Hiragana for Japanese, and an "unclear" flag for each sentence.

The most critical line in the prompt: If there are lyric subtitles on the MV screen, prioritize using the subtitles. Transcribing singing is much harder than speech due to accompaniment, elongated notes, and harmonies; Japanese also has homophone issues (hearing "kimi" without knowing if it's "君" or "きみ"). With subtitles, Gemini looks at the screen and listens simultaneously, greatly improving accuracy.

I tested with three songs:

Song	Language	Lyrics Source	Sentence Count (Unique)
Yuuri 〈Betelgeuse〉	Japanese	Audio	40 (23)
Yuuri 〈Christmas Eve〉	Japanese	Screen Subtitles	45 (35)
Take That 〈Back for Good〉	English	Audio	37 (26)

The structure was complete: no empty sentences, no backward time, no continuous repetition of the same sentence (a common symptom of model errors). The only thing that made me uneasy was that every sentence was marked as "certain", even for the two based on audio. The model is too confident; I'll fix this later with another method.

Annotation: Don't Leave All Phonetics to the LLM

annotate.py adds Traditional Chinese translation, word-by-word breakdown (reading, POS, meaning), a grammar point, and a pronunciation tip to each sentence. Choruses repeat, so Gemini is only called for unique sentences, saving about 30-40%.

Phonetics is the most interesting part of this section. My original idea was to use existing libraries, which are more reliable than LLMs, but both failed:

pykakasi (Japanese) transliterates the particle "は" as ha, when the correct pronunciation is wa. Just looking at Hiragana, it can't tell if "は" is a particle.
korean-romanizer (Korean) transliterates "감사합니다" as gamsahapnida. The official Revised Romanization follows actual pronunciation, which should be gamsahamnida; this package doesn't handle nasalization.

The final division of labor:

Japanese: Gemini handles word segmentation and provides reading/POS for each word; the program then uses pykakasi to convert to Romaji and corrects particles は, へ, を to wa, e, o based on POS.
Korean: Hangul is phonetic; I simply asked Gemini to output according to Revised Romanization rules.

Reason and Solution: Libraries are good at "deterministic conversion" but bad at "judgment requiring context"; LLMs are the opposite. Handing the parts requiring understanding (segmentation, POS) to the LLM and the deterministic parts (Kana to Romaji) to the program uses both where they excel.

Cross-checking with Two Independent Results

As mentioned, the transcription results marked everything as "certain," which isn't very believable. My fix: Gemini already gave a full-sentence reading during transcription, and then gave individual word readings during annotation. These two were generated separately; sentences that don't match are likely Kanji misreadings.

In practice, each of the two Japanese songs had 1 mismatch (1/23, 1/35). The program adds needs_review to them, so the webpage can remind the user to proofread.

This trick doesn't cost any extra API calls; it just compares two existing results.

Teacher: "Designed" via Voice Design

speak.py designs a teacher for each language, e.g., for Japanese: "A gentle and patient Japanese teacher from Tokyo in her early 30s." Created once, voice_id saved for reuse. Two audio clips per sentence:

	Normal Speed	Slow Speed
Length	2.5–6.1s, avg 4.3s	4.4–9.7s, avg 7.1s
Slow / Normal		At least 1.24x, up to 2.34x

Slow speed relies solely on a style description without adjusting playback speed, and it's "speaking slowly and clearly," not stretching a normal speed clip. This is what impressed me most about this TTS.

Web App: Next.js 16

The webpage uses Next.js, with the server-side reading data directly from ../output/. The screen shows the embedded MV on the left, teaching cards below (Kana/Romaji/Translation/Breakdown/Grammar/Tips above Kanji, plus "Teacher," "Slow," "Original" buttons); the right side is the sentence list, auto-switching with the MV.

Upon running create-next-app, I encountered something interesting. The generated AGENTS.md wrote in the first line:

This is NOT the Next.js you know

Meaning this version has breaking changes, and agents should read node_modules/next/dist/docs/ before coding. Claude Code followed suit; params became Promises, and RouteContext types for route handlers were found there. Documentation written specifically for AI by frameworks is a practice I think will become increasingly common.

Quota: Tier 1 is Only 100 Times Per Day

This is the most important constraint of the project, so I'll mention it first.

gemini-3.8-flash-tts in Tier 1 has 100 requests per day. A song with 30+ sentences, each with normal and slow versions, takes about 50–70 requests. At this rate, only one or two songs can be processed per day.

So I changed the batch generation in speak.py to generate only when the user clicks play, then cache it:

Action	Consumes Gemini?
Viewing lyrics, translation, breakdown	No
Playing MV, clicking "Original"	No, that's YouTube
Playing cached demo	No
First time playing a demo	1 TTS call (Normal and Slow counted separately)
Adding new song	~2 Flash calls, no TTS

Audio files are named using the hash of the sentence content. Repeated choruses naturally share the same audio. Quota is only spent on sentences actually practiced.

Pitfall 1: Program "Stuck," Actually Sleeping for Seven Hours

While batch generating for the second song, progress stopped at 56/70.

The status was strange: Python process alive, CPU 0%, 4 workers waiting on Google servers. First guess was no timeout, so I added 60s timeout and retries. After restarting, only 1 segment was added in ten minutes, and this time there wasn't even a network connection.

Turning off SDK auto-retry and looking at the response revealed the answer:

429 Rate limit exceeded for model gemini-3.8-flash-tts
(limit: 100 requests per day on Tier 1). Please retry in 7h12m16s
retry-after: 25936

Daily quota was exhausted, and the SDK, upon receiving a 429, obediently waited 25,936 seconds to retry according to Retry-After. From the outside, it looked like a program that consumed no CPU, had no network, and would never end.

Worse, my initial attempt to use HttpRetryOptions(attempts=1) to disable retries failed completely. The reason is that the interactions API uses a different HTTP client in the SDK (module name _gaos), which ignores HttpRetryOptions. I had to change its own config:

client = genai.Client()
# This API doesn't respect HttpRetryOptions; if not disabled, it sleeps for hours on daily quota limits
client.interactions.sdk_configuration.retry_config.max_retries = 0

Reason and Solution: "Respecting Retry-After" is good for per-minute limits but a disaster for daily quotas. Now the program handles retries itself: transient errors retry up to 3 times; if a 429 contains "per day," all workers stop immediately, print the wait time, and keep existing audio files.

Pitfall 2: Fields in Python SDK Don't Exist in REST

This was the most expensive lesson.

After switching to "generate on play," the work moved from Python to Next.js server-side using fetch for REST. Python version:

audio = base64.b64decode(interaction.output_audio.data)

The TS version followed suit with json.output_audio?.data. The problem: The output_audio field doesn't exist in the REST response at all.

Checking the SDK source code revealed that output_audio is a convenience field calculated by the Python SDK in a pydantic validator: it looks for a step with type: "model_output" in the steps array, then extracts the item with type: "audio" from that step's content. The real REST response looks like this:

steps[] → { type: "model_output", content[] → { type: "audio", data: "<base64>" } }

So this happened:

Gemini successfully generated audio every time, counting against the quota.
My program couldn't read the audio, returned 502, and didn't save the file.
Every time a user clicked play, plus multiple Range requests from the browser, TTS was called again.

Within about an hour, the 100-request quota was exhausted, and not a single new file was in output/audio/.

To make matters worse, when this code went live, the daily quota was already exhausted, so the success format couldn't be verified. Claude Code noted "success path unverified" in the report, but we let it go live anyway.

Reason and Solution: Fixed in two layers.

Parse according to SDK logic: Look for steps, then the old outputs format, and finally output_audio. No quota to verify via API, so I fed the same mock response to the Python SDK parser and the new TS function to ensure consistency. Later, when a quota slot opened, the first English audio was generated and saved (5.04s, 24 kHz), finally verifying it.
Prevent double billing on failure: For 60 seconds after a failed generation, return the last error directly; after a 429, block all audio until the Retry-After time.

// A failed generation may still have been billed, so don't let repeated clicks or the
// browser's parallel range requests retry it: replay the error for a while instead.
const recentFailures = new Map<string, { error: TtsError; until: number }>();

What this bug taught me: Failure doesn't mean no cost. As long as the request reaches the model and the model runs, even if your program fails at the last parsing step, the bill still counts. Programs that consume quota must see a real success response before going live, and failure paths should default to "this might have already been billed."

Pitfall 3: Quotes in `.env`

Next.js first API call returned:

400 API key not valid. Please pass a valid API key.

The same key worked fine in Python. Checking the .env format (length and start/end chars only) revealed my key was GEMINI_API_KEY="..." with double quotes. Python's python-dotenv automatically strips quotes; the Node-side reader didn't, sending the quotes as part of the key.

Also caught another issue: Gemini's error response is sometimes an array [{ "error": ... }] instead of an object, so the original program couldn't even read the error message, showing a blank "400:" on screen.

Reason and Solution: Changed .env reading logic to match python-dotenv (allow export prefix, strip quotes), and handled both error formats. A small thing, but these "one side handles it for you" differences are easy to miss when sharing a config file between languages.

Pitfall 4: Naming Audio by Line Number Causes Misalignment on Proofreading

Transcription errors are inevitable, so I added a proofreading interface: edit text, reading, or translation directly, or click "✓ OK"; after editing text, "Re-analyze whole song" reruns annotate.py (1 Flash request) while preserving manual edits.

Before starting, I found a problem: Audio files were named by "index of unique sentence," e.g., 028_normal.wav. If a lyric line is edited, subsequent sentence numbers might shift, causing existing audio to map to the wrong sentence.

Reason and Solution: Changed to naming by the hash of the sentence content:

def clip_name(text: str, speed: str) -> str:
    """Clips are keyed by line content so editing a lyric only invalidates that line's audio."""
    return f"{hashlib.sha1(text.encode()).hexdigest()[:16]}_{speed}.wav"

Editing one line only invalidates that line's audio; others are unaffected. All 103 generated audio files were moved to new filenames, so no quota was wasted. Python and TS both calculated the hash for the same Japanese segment to ensure consistency before going live.

Pitfall 5: Some MVs Don't Allow Embedding

Found some videos wouldn't play. YouTube returns error 101 or 150 when "owner does not allow playback on other sites."

Now when these errors are detected, the video area swaps to a thumbnail with an explanation and an "Open on YouTube (start from here)" button; "Original" also opens a new tab at that timestamp. Teacher demos and teaching cards remain available.

A small pitfall with thumbnails: High-res maxresdefault.jpg isn't available for every video, but YouTube doesn't return 404; it returns a 120x90 gray default image, so <img onError> doesn't trigger, resulting in a large gray area.

Reason and Solution: Check image width after loading; if under 121 pixels, swap to the guaranteed hqdefault.jpg.

Development Workflow: Let Claude Code Verify, but Don't Let It See Lyrics

The whole project was done with Claude Code, from checking changelogs and discussing product direction to coding and deployment planning. A few things I think we did right:

邊做邊推上 GitHub: Commit and push at every stage, explaining "why" in messages, not just "what."
驗證只看統計數字: Checking transcription quality via sentence counts, time flow, repetitions, Kanji in readings; checking audio via length, slow-motion ratio, anomalies. No need to print lyrics.
測試用假歌曲: Use a fake song for testing proofreading, re-analysis, and embedding errors (e.g., "駅まで歩く"), deleting it after testing.
用無頭 Chrome 實際點按鈕: Server-side tests aren't enough; used Puppeteer for play, proofread, and add flows. The "no sound" issue was narrowed down to the server-side after confirming Chrome playback was fine.

There were mistakes too, as mentioned in the pitfalls:

Going live with quota-consuming code before verifying success format (Pitfall 2).
Background "Add Song" status stayed in memory; if files were deleted, it would say "Complete" without redoing. Caught this while writing "resume from break" tests; changed to rely on disk files.

Results and Benefits

	Number
commit	11
Songs added	3 (2 Japanese, 1 English)
Demo clips generated	104
Single demo generation time	~5s
Adding a new song	~48s (Transcribe 14s, Analyze 34s)
API usage per song	~2 Flash calls, demos generated on demand

Current features:

YouTube URL addition, background transcription and analysis.
Sentence-by-sentence learning: Kana, Romaji, Translation, Breakdown, Grammar, Tips.
Teacher demos, normal and slow, generated on first play.
Proofreading interface: Edit lyrics, one-click confirm, re-analyze whole song.
Fallback UI for non-embeddable videos.

Next Step: Move to Cloud Run

Currently only runs locally. Plan to deploy to GCP:

Cloud Run for Next.js, containerized with Python and uv.
GCS for audio and song data. Use signed URLs for playback; GCS supports Range, so Safari works.
Secret Manager for API keys.
IAP to lock the service to my Google account. This is essential: public access would drain my quota, and full lyrics/translations would move beyond "personal learning."

Key Takeaways

SDK convenience fields are not part of the API. output_audio is Python-only. Check real responses when switching languages or using REST.

"Respecting Retry-After" depends on the limit type. Good for per-minute, bad for daily. Fail fast and explain why for quota errors.

Failed requests might have been billed. Assume cost on failure for paid operations and prevent retry storms. A 60s failure cache would have saved a day's quota.

Use LLMs and deterministic programs where they excel. Gemini for segmentation/POS, pykakasi for Kana-to-Romaji. Cross-check independent results to find errors for free.

Think through copyright before the first line of code. Different acquisition methods don't change legality. Use .gitignore, stats-only verification, and IAP to keep it a personal tool.

Code at kkdai/song-lingo (lyrics data not in repo). Official docs for Speech generation and Voice design. Reminder for REST users: audio is in steps[].content[], not output_audio.

[AI in Practice] Deploying Song Lingo to Cloud Run: Making a Private Lyrics Website Just for Me

Evan Lin — Sat, 26 Sep 2026 17:24:45 +0000

Recap

In the previous post, I used Gemini 3.8 Flash TTS to build Song Lingo: you paste a YouTube MV URL, Gemini transcribes the lyrics, adds furigana, translation, and grammar notes, and then a teacher designed via voice design reads it to you line by line.

It has only ever run on my own computer, but I want to be able to use it on my phone. So the goal of this post is simple: Move it to Cloud Run so it can be used on mobile.

However, this website has a unique characteristic: the page contains full lyrics and translations.

Requirement: How accurate does "Only I can see it" need to be?

When a typical side project is deployed, it's at most a bit embarrassing if others see it. Song Lingo is different; making it public has two practical consequences:

Copyright: I spent a whole section in the last post explaining that this is a personal learning tool, and the lyrics only exist on my own computer. Once the site is public, the nature changes from "personal learning" to "providing lyrics to the public."
Cost: Adding a song calls Gemini Flash, and playing the demo audio for the first time calls TTS. If anyone can use it, it means anyone can spend my quota.

So the goal isn't just to "have a login function," but to ensure at every layer, from start to finish, that only my account can access the lyrics and APIs.

I first compared two solutions:

	A. IAP + In-app Verification	B. No external access, use `gcloud run services proxy`
Supported Devices	Any browser, including mobile	Only computers logged into `gcloud`
Setup Difficulty	Medium	Low
Potential for Error	Low	Lowest, no public entry point at all

Option B is the safest, but it doesn't work on mobile, which is the whole reason for this deployment. So I chose A.

Vulnerability in Version 1: Signed URLs

My initial plan was this: store audio files in GCS, and when playing, the API generates a short-lived signed URL and redirects the browser to it. The advantage is that the audio doesn't pass through Cloud Run, saving bandwidth, and GCS itself supports Range requests.

Halfway through writing, while reconsidering "how to accurately ensure only I can see it," I realized this was a vulnerability:

Within the expiration period, anyone who gets the signed URL can download it directly, completely bypassing IAP.

It is essentially an anonymous bearer token. If the URL appears in browser history, is pasted somewhere, or is recorded by an extension, others can bypass all the previous login checks.

Cause & Solution: "Only I can access" is a chain; its strength depends on the weakest link. The final approach was not to use signed URLs. Audio files are always read by Cloud Run and then sent to the browser, and every playback must first pass IAP and in-app verification. An audio segment is about 250KB; the traffic cost of this extra hop is negligible.

Architecture

Component	Approach
Container	One image containing both Node 22 and uv/Python; Next.js directly calls the original Python scripts
Song Data & Audio	Private Cloud Storage bucket, mounted as the `/data` folder
API key	Secret Manager, provided as environment variables
Access Control	IAP + In-app verification of the IAP signature
Instances	Max 1, min 0; CPU always allocated

Reasons for several decisions:

Mount the bucket as a folder instead of rewriting the code to call the GCS API. The program originally read/wrote to the output/ folder; after mounting, I just need to point SONG_DATA_DIR to /data, requiring almost no code changes.
Run only 1 instance. To avoid duplicate audio generation, pausing calls when the quota is exhausted, and tracking the progress of adding new songs, these states are stored in memory. Also, the mounted bucket doesn't have cross-instance locking. For personal use, 1 is enough, and this ensures the mechanisms to prevent duplicate billing from the previous post remain effective.
CPU always allocated (--no-cpu-throttling). "Adding a new song" continues transcription and analysis in the background after the response is sent. Cloud Run defaults to throttling the CPU after a response is sent, which would stall background tasks.

Four Layers of Protection

The final access control consists of four layers. If any single layer fails, the others still hold:

Cloud Run Permissions: --no-allow-unauthenticated, only the IAP service account can call this service.
IAP: Only accounts granted roles/iap.httpsResourceAccessor can pass, which is only me.
In-app Verification: Every request verifies the IAP assertion header and compares it against the email.
Private Bucket: Public access is strictly prohibited; only the service account dedicated to this service can read/write.

The third layer might seem redundant since IAP is already in front. But it protects against "IAP layer misconfiguration": someone accidentally adding --allow-unauthenticated, IAP being turned off, or ingress settings being changed. If that happens, the third layer is the last line of defense.

Verifying IAP Signatures in Next.js 16

Next.js 16 renamed middleware to proxy.ts, which runs in the Node.js runtime by default, making it perfect for this:

export async function proxy(request: NextRequest) {
  if (!process.env.K_SERVICE) return NextResponse.next();

  const result = await checkIapAssertion(request.headers.get("x-goog-iap-jwt-assertion"), {
    audience: process.env.IAP_AUDIENCE,
    allowedEmails: process.env.ALLOWED_EMAILS,
  });
  if (!result.ok) {
    console.warn(`[auth] rejected ${request.method} ${request.nextUrl.pathname}: ${result.reason}`);
    return new NextResponse(result.reason, { status: result.status });
  }
  return NextResponse.next();
}

Two design highlights:

Use K_SERVICE to determine if running on Cloud Run. This environment variable is automatically set by Cloud Run. It doesn't exist during local development, so local work is unaffected, and there's no need for an extra "disable verification" switch (which would eventually be accidentally pushed to production).
Reject all if configuration is missing. If IAP_AUDIENCE or ALLOWED_EMAILS is not set, all requests return 500. A configuration error results in "the site won't open" rather than "the site is open to the public."

The verification itself follows IAP documentation: ES256 signature, issuer is https://cloud.google.com/iap, audience is /projects/PROJECT_NUMBER/locations/REGION/services/SERVICE_ID, and the public key is fetched from Google's JWK endpoint.

How to Test Before Going Live

Since there's no real IAP to hit locally, I made the key source a replaceable parameter and tested 12 scenarios locally using my own generated ES256 keys:

Scenario	Result
Allowed account (including case sensitivity)	Pass
Other accounts, no email	403
Wrong audience, wrong issuer, forged signature, expired, gibberish, missing header	401
Missing audience, empty allowlist	500

Then I ran the production build in three modes:

Mode	Result
Local (no `K_SERVICE`)	All 200
On Cloud Run, but forgot settings	All 500
On Cloud Run, settings complete, but missing or forged header	All 401

Pitfall 1: Another Claude window already did half the work

Halfway through coding, two files I didn't create appeared in git status: Dockerfile and .dockerignore.

It turns out I accidentally had two Claude Code windows open. Both were discussing song-lingo, and the other one had also talked about deployment and already written a Dockerfile using a different approach: mounting GCS as a folder, while I was currently writing a whole storage abstraction layer to change all reads/writes to GCS API calls.

After comparing, the mounting approach was clearly better: it required almost no code changes, and the only advantage of my abstraction (version checking on write) wasn't really needed for a single-instance, single-user scenario. So I deleted my abstraction and switched to mounting.

But that wasn't all. After deploying, I found the service was already on revision 2. Revision 1 had been created earlier that day, and IAP was already enabled. That window hadn't just written a Dockerfile; it had actually deployed once.

Cause & Solution: Two agents in the same repo and same GCP project were working independently, unaware of each other. Nothing went wrong this time because I checked git status before acting and checked the revision list and service settings after deploying, rather than assuming "I am the first." My habit going forward: Only do one task in one window, and always check what's already in the cloud before starting a deployment.

Pitfall 2: Personal Gmail projects, IAP returns 502 for every request

After deploying and enabling IAP, I opened the URL and got a 502.

Initially, I thought the program crashed, but a key clue was in the response headers:

HTTP/2 502
x-goog-iap-generated-response: true

Empty Google Account OAuth client ID(s)/secret(s).

x-goog-iap-generated-response: true means this error was returned by IAP itself; the request never reached my code. The message indicates IAP has no OAuth client to use.

The reason is that my project doesn't belong to any organization; it was created with a personal Gmail account. IAP defaults to using a Google-managed OAuth client, which only supports accounts within an organization. For such projects, you must create your own OAuth client:

Google Auth Platform (OAuth consent screen): Set User Type to External.
Credentials → Create OAuth client ID → Web application, set the redirect URI to https://iap.googleapis.com/v1/oauth/clientIds/CLIENT_ID:handleRedirect.
Use gcloud iap settings set to apply the client ID and secret to this service.

I executed step 3 in my own terminal rather than in the Claude Code chat, so the client secret wouldn't appear in any chat history.

What if the consent screen is already public?

My project contains many other services, and the OAuth consent screen was already set to public for other apps. My first reaction was, "If it's public, can anyone log in? Should I switch back to testing mode?"

The answer is no need to change, and you shouldn't:

Layer	Responsibility	Impact of being Public
OAuth Consent Screen	Confirm "Which Google account are you"	Any account can complete the login step
IAP Access Permissions	Confirm "Can this account use this service"	Unaffected
In-app Verification	Re-confirm signature and email	Unaffected

The consent screen is only responsible for "identifying who you are." The actual decision of "whether you can enter" is made by the subsequent two layers. Switching back to testing mode would instead affect other services sharing the same consent screen: only test users could log in, and authorization would expire in about 7 days.

Cause & Solution: When you see a 502, check the response headers first. x-goog-iap-generated-response tells you directly if the problem is with IAP or your code. Personal Gmail projects need their own OAuth client; whether the consent screen is public does not affect who can use the service.

Pitfall 3: Ensuring `.env` and lyrics aren't uploaded

gcloud run deploy --source . uploads the entire folder to Cloud Build. The local .env has API keys, and output/ has all the lyrics; these must absolutely not be uploaded.

gcloud defaults to using .gitignore if .gcloudignore is missing, and both of these were already in .gitignore. But "should be excluded" wasn't enough, so I explicitly wrote a .gcloudignore and used gcloud's own command to list what would actually be uploaded:

gcloud meta list-files-for-upload .

The result was 48 files; .env, output/, node_modules, and .next were all 0.

Cause & Solution: For commands that send files out, use the tool's own listing feature to see "what is actually being sent" rather than relying on your own inference of ignore rules.

Small things that lead to misjudgment

zsh wildcards: Counting objects with gcloud storage ls -r gs://bucket/** resulted in 0. zsh tries to expand ** as a local wildcard first; if it finds nothing, it errors out, and the query is never sent. Adding quotes fixed it: 116 objects, perfectly matching local.
Fields not found by --format don't error: Querying bucket settings with --format='value(iamConfiguration.publicAccessPrevention)' gave empty output. The field name changed in newer gcloud versions, but it doesn't error; it just silently gives you a blank. Switching to JSON output revealed public_access_prevention: enforced.

The commonality here: "0" and "blank" do not mean "none" or "no problem." When verifying security settings, if you get an empty result, suspect the query itself first.

Verification: Every layer must be tested

The post-deployment verification checklist, every item was actually run:

Check	Result
Open homepage, API, audio without logging in	IAP returns 302, redirects to Google login
Attach a forged IAP signature header	Still 302, IAP doesn't accept external signatures
Add a new song via POST without logging in	IAP returns 401
Anonymous access to bucket files and listing	403
Cloud Run invocation permissions	Only IAP service account
Bucket public permissions	No `allUsers` or `allAuthenticatedUsers`
My account logs in and plays	Normal, in-app verification 0 rejections
Other Google account logs in	"You don’t have access"

The last two can only be tested in a browser. Confirming "my account can use it" is actually the most critical: the audience format was filled according to documentation; if it were wrong, I would pass IAP but be blocked by my own code. I confirmed this by searching Cloud Run logs for [auth] rejected, which returned 0.

I also saw something in the logs that made me very happy: I added a new song directly in the cloud, and the entire workflow—transcription, analysis, demo audio generation, and playing from the bucket—worked perfectly within the container.

Costs and To-dos

Item	Estimate
Cloud Run	Scales to 0 when not in use, no charge; CPU always allocated only charges during instance uptime
Cloud Storage	~27MB, less than US$0.01 per month
Cloud Build	A few minutes per deployment, within free tier
Gemini API	Same as local, billed by usage

There are two more things I need to do in the Console:

Restrict API key usage to only call the Generative Language API.
Set budget alerts. If all the previous protections fail, this is the final insurance to cap losses and ensure I'm notified.

Also note: the bucket and local output/ are now two independent sets of data. Songs added in the cloud won't automatically sync back; use gcloud storage rsync when needed.

Key Takeaways

"Only I can access" is a chain, not a door. No matter how well IAP is set up, a single signed URL can bypass it. When checking access control, list all paths through which data can exit, not just the entrance.

When a setting is missing, the system should fail closed, not open. In-app verification rejects everything if settings are missing. If a future deployment misses an environment variable, the result will be that I can't open the site and will notice immediately, rather than the site silently being open to the public for weeks.

Look at who returned the error. For a 502, the x-goog-iap-generated-response header narrows the problem down from "the whole service" to "IAP OAuth configuration."

Suspect the query first for empty results. An object count of 0 or a blank configuration field might be a malformed query rather than the resource not existing. In security verification, the cost of such a misjudgment is particularly high.

Do one task in only one agent window. Two Claude Code windows acting on the same project simultaneously were only prevented from overwriting each other because I checked the current state before acting.

Keep secrets out of the conversation. OAuth client secrets and API keys were handled in my own terminal. The AI was only responsible for giving me the commands, ensuring secrets never entered the chat history.

The code is at kkdai/song-lingo. The deployment section of the README has complete commands and a verification checklist (values replaced with placeholders). Related documentation: IAP for Cloud Run, Verifying IAP assertion headers, IAP custom OAuth configuration.

[AI in Practice] Building a Dynamic LINE Group Buying Bot with Edit and Unsend Webhooks

Evan Lin — Thu, 03 Sep 2026 04:27:12 +0000

When Messages Can Be Taken Back: Building a "Shapeshifting" LINE Group Buying Bot with Edit and Unsend Webhooks

Author: Evan Lin, LINE Taiwan Developer Relations Team Lead

On August 20, 2026, LINE announced the open free trial of the "Edit Message" feature in LINE Labs.

For general users, this is a very intuitive feature: if you find a typo, a wrong date, or want to adjust your tone after sending a message, you no longer need to go through the "Unsend, retype, resend" process. You can simply edit the original text message.

But when I saw this feature, the first thing I thought of was something else:

If a user changes a message that has already been processed by a LINE Bot, does the Bot know?

The answer is: Yes. The LINE Messaging API provides the Edit event; and when a user unsends a message, there is also an Unsend event that can be received.

In this article, I want to share how I turned these two types of webhook events into a LINE Bot Demo with a story, as well as several easily overlooked but very important details during implementation.

The complete sample code is available on GitHub:

https://github.com/kkdai/linebot-edit-unsend

Starting with "Edit Message" in LINE Labs

To try out message editing in LINE Labs currently, you need to meet the following conditions:

Update the mobile version of LINE to version 26.12.0 or higher.
Go to "Home → Settings → LINE Labs".
Enable "Edit Messages".

In standard one-on-one and group chats, text messages can be edited within 15 minutes of being sent; for Keep Notes, it's within 6 days. Photos, videos, voice messages, files, and stickers cannot be edited at this time. After modification, the chat room will display "Edited," but it does not provide a way to view or restore old versions.

There is also a limitation directly related to Bot development: currently, one-on-one chats with LINE Official Accounts do not support message editing. Therefore, to test the Messaging API's Edit event, the Bot must be added to a group chat.

For features and activation methods, please refer to the LINE Taiwan Newsroom Announcement.

From UI Features to Bot Data Consistency

In the past, the common process for a Bot receiving a text message was:

Receive message webhook.
Parse the text.
Write to the database or trigger subsequent processes.
Reply with the processing result.

If a message cannot be modified after being sent, this model is very simple. But when messages can be edited, the original text is not necessarily the user's final intent.

For example, a user sends:

Pearl Milk Tea / Half Sugar / Less Ice / 1

The Bot has already recorded it as one drink costing 60 TWD. A few seconds later, the user directly edits the original message:

Pearl Milk Tea / Micro Sugar / No Ice / 2

If the Bot does not handle the Edit event, the chat room sees two cups, but the backend still records one cup. The world on the screen and the world inside the Bot become disconnected.

This is the most practical value of the Edit event: it's not just a notification that "the text has changed," but an opportunity for the service to synchronize with the user's latest intent.

Group Buying Transformation: Giving the API Demo a Story

To demonstrate both edit and unsend simultaneously, I set the Bot as "Shapeshifting Store Manager - Dan Dan," responsible for saving the office from low energy at 3:00 PM.

The Demo flow is as follows:

The group leader enters Start Group Buy 50 Lan 15:20 Deadline.
Alice enters Pearl Milk Tea / Half Sugar / Less Ice / 1.
Alice edits the same message to Pearl Milk Tea / Micro Sugar / No Ice / 2.
The Bot receives the Edit event and updates the quantity, sweetness, ice level, and total amount.
Bob unsends his message after placing an order.
The Bot receives the Unsend event, deletes Bob's order, and recalculates.
Enter Current Orders to view the latest summary via Flex Message.

This scenario is perfect for a Demo because everyone knows the two most common things in group buying: changing your mind after ordering, and suddenly not wanting to drink after ordering.

More importantly, the audience can directly see from the quantity and amount whether the webhook actually affected the backend state, rather than the Bot just replying "Edit received."

Structure of the Edit Event

The type of the Edit event is messageEdited, which carries the edited text, timestamp, reply token, and message ID:

{
  "type": "messageEdited",
  "replyToken": "950e63e8f46542ab89f645b4c2a1180a",
  "message": {
    "type": "text",
    "id": "610830548529053697",
    "text": "Pearl Milk Tea / Micro Sugar / No Ice / 2"
  },
  "timestamp": 1776914799524,
  "source": {
    "type": "group",
    "groupId": "Ca56f94637c...",
    "userId": "U4af4980629..."
  }
}

The most critical design point is: the message.id in the edit event is the same as the message ID of the original message event.

Therefore, we can directly use the message ID as the order ID:

case webhook.MessageEvent:
    h.handleMessage(e)
case webhook.MessageEditedEvent:
    h.handleEdit(e)
case webhook.UnsendEvent:
    h.handleUnsend(e)

Add an order when the original message event is received; when MessageEditedEvent is received, use the same message ID to find the order and overwrite the content.

Additionally, the Edit event has its own reply token, which is different from the original message event's reply token, so the Bot can directly reply "Order successfully transformed" to this specific edit.

If your Bot also uses the Mark as Read API, note that the Edit event does not carry a markAsReadToken. You cannot simply apply the standard message event processing flow. These field differences are well-suited for a full test using webhook fixtures after upgrading the SDK.

For complete fields, please refer to Messaging API: Edit event.

The First Trap: Edit Events May Not Arrive in Order

This is the most easily overlooked part of this implementation.

A user might edit the same message multiple times in quick succession, and multiple messageEdited webhooks are not guaranteed to arrive in the order they were edited. Official documentation recommends using the largest timestamp to represent the latest state.

Therefore, you cannot simply adopt the "last received webhook"; you should adopt the "webhook with the latest timestamp":

if timestamp <= current.UpdatedAt {
    return summarize(buy), errStaleEdit
}

current.UpdatedAt = timestamp
buy.Orders[messageID] = current

Assuming the second edit arrives first and the first edit arrives later, this logic prevents old content from overwriting new content.

At the same time, the LINE Platform might resend webhooks, so the example also uses webhookEventId for deduplication. These two mechanisms handle different problems:

webhookEventId: Prevents the same event from being processed twice.
timestamp: Prevents different edit events from updating data in the wrong order.

Both need to be handled.

The Second Trap: Edited Content May No Longer Be a Valid Order

Users don't necessarily just change the quantity. They might also change the original order to something like:

I suddenly don't want to drink anymore

In this case, the new message can no longer pass the order format validation. If the backend continues to keep the old order, it will still cause state inconsistency.

The approach in this Demo is: clear the old order content, mark this record as invalid and temporarily remove it from the total, and then ask the user to continue editing to fix it.

parsed, err := parseOrder(text)
if err != nil {
    current.Item = ""
    current.Sugar = ""
    current.Ice = ""
    current.Quantity = 0
    current.UnitPrice = 0
    current.Valid = false
    current.UpdatedAt = timestamp
    buy.Orders[messageID] = current
    return summarize(buy), err
}

Actual products can adopt different strategies, such as putting it into a "waiting for manual confirmation" state; the key is not to silently continue using old content that no longer exists in the chat room.

Unsend Event: Beyond Technology, User Intent

The content of the Unsend event is simpler than the Edit event:

{
  "type": "unsend",
  "source": {
    "type": "group",
    "groupId": "Ca56f94637c...",
    "userId": "U4af4980629..."
  },
  "unsend": {
    "messageId": "610830548529053697"
  }
}

It only tells us which message ID was unsent; it does not re-attach the message content, nor does it have a reply token.

Since we already use the message ID as the order ID, deletion is straightforward:

delete(buy.Orders, messageID)

But what's truly important here is not delete(), but how to respect the user's intent of "I want to take back this content."

Official documentation specifically reminds service providers that after receiving an Unsend event, they should handle it carefully so that the target message cannot be seen or used in the future. Therefore, this Demo will not quote the unsent item, sweetness, or other original text in the Bot's reply, but will only say:

💨 An order has been safely withdrawn, and the saved content has been deleted.

Since the Unsend event has no reply token, if you want to proactively notify the group, you can only use a push message. This means it will count towards message usage, so quota and cost should be considered when designing a formal service.

For full details, please refer to Messaging API: Unsend event.

Making State Changes Visible with Flex Message

While plain text can show results, to make the Demo understandable at a glance, I highly recommend using Flex Message to display the current order.

The card this time includes:

Store name and deadline
Group buy status (In progress or Closed)
Each member's item, sweetness, ice level, quantity, and subtotal
Total quantity and total amount
"Refresh Order" button

After every addition, edit, or unsend, the Flex Message is regenerated from the current state. This way, the quantity and amount before and after an edit change immediately, and the line item disappears after an unsend.

To avoid the Flex Message becoming too large, the example displays a maximum of eight orders, with the rest summarized. Formal products can switch to carousels, LIFF pages, or add pagination queries as needed.

SDK and Execution Environment Versions

This example uses:

github.com/line/line-bot-sdk-go/v8 v8.22.0
Go 1.25

Older versions of the SDK may already have UnsendEvent, but they might not include the new MessageEditedEvent type. After upgrading the SDK, you also need to check the required Go version; when upgrading from the old example this time, the Go toolchain and CI workflow both needed adjustment.

This is a common but easily missed upgrade issue: being able to compile locally doesn't mean Cloud Build or GitHub Actions are still using the same version.

State Issues When Deploying to Cloud Run

This Demo is deployed on Google Cloud Run. The channel secret and channel access token are injected via Secret Manager and are not committed to the repository.

To keep the example simple, current orders are stored in the program's memory. This brings two limitations:

If the instance restarts or scales to zero, orders will disappear.
When multiple instances exist simultaneously, each instance will have a different order state.

Therefore, the Demo environment sets the maximum instances to 1 to avoid webhooks for the same group buy being split. But this is only suitable for demonstration, not a complete solution for a production environment.

Formal services should use Redis, Firestore, or other shared storage, and further handle:

Atomic updates and concurrency control
Webhook idempotency
Data retention periods
Cross-system deletion after unsend
Push Message failure retries
The boundary between audit logs and privacy requirements

Especially for unsend, if message content has already been sent to search indices, analysis platforms, or other downstream services, deleting only the main database is incomplete. Data flow design should know where content went from the start to be able to truly complete a deletion.

How to Run This Demo

First, please enable webhooks in the LINE Developers Console and allow the Bot to join group chats. Point the webhook URL to the /callback of your deployed service.

Then, enter the following in the group:

Start Group Buy 50 Lan 15:20 Deadline
Pearl Milk Tea / Half Sugar / Less Ice / 1
Current Orders

Then directly edit the second message:

Pearl Milk Tea / Micro Sugar / No Ice / 2

You will see the Bot reply "Order successfully transformed," and the quantity and total amount in the Flex Message will also update.

Finally, unsend that order message, and the Bot will delete the content and show a new order summary.

The menu and prices in the example are fixed Demo data to make amount changes clearly visible. For complete startup instructions, environment variables, and test commands, please refer to the README in the repository:

https://github.com/kkdai/linebot-edit-unsend

What Else Can This Be Applied To?

Group buying is just one easy-to-understand story. The same event model can be extended to:

Itinerary Bots: Update the itinerary when the meeting time is edited, and cancel the activity when the message is unsent.
Task Bots: Edit the assignee or deadline, and remove the task when the message is unsent.
Reservation Bots: Modify the time slot or number of people, and cancel the reservation when the message is unsent.
Announcement Bots: Update announcement content, and synchronize the removal of copies on other channels after unsend.
Interactive Story Bots: Rewrite the plot when a choice is edited, and return to the previous node when a message is unsent.

As long as a Bot has ever converted "a message" into some kind of system state, it's worth re-checking: when the message is edited or unsent, should that state also change?

Conclusion

Message editing looks like an improvement to the chat interface, but from a Bot developer's perspective, it actually changes the lifecycle of an event.

A message is no longer just the moment it is "sent." It might be updated, or it might be withdrawn. Backend services need to understand these events, maintain the correct order, and respect the user's latest intent.

This time, I used an afternoon tea group buy story to string together Edit event, Unsend event, Flex Message, and Cloud Run. I hope this small example helps everyone master the new features faster and start thinking about how their own LINE Bots should respond after a message is changed.

Feel free to refer to the code, and I look forward to seeing everyone create more interesting applications:

https://github.com/kkdai/linebot-edit-unsend

References

[AI in Practice] Gemini 3.5 Transcribe: Real-time Transcription and Speaker Diarization in a macOS Meeting Translation App

Evan Lin — Fri, 28 Aug 2026 15:53:34 +0000

Previously

I have a macOS App I use myself, gemini-live-translate-macos. It uses ScreenCaptureKit to directly capture audio from a specified App, eliminating the need for virtual sound cards like BlackHole. It then sends the audio to the Gemini Live API for real-time translation, outputting Traditional Chinese subtitles while playing Chinese audio. I've written two posts about the development process: the first one was about building it from scratch using AGY CLI, and the second one was about using Claude Code to take it from "functional" to "user-friendly."

The starting point for this new addition was simple: I saw a document for "Real-time Transcription" added to the Live API. Since I was already connected to the Live API, I thought adding a pure transcription mode would just be a matter of changing a few parameters.

However, after checking the documentation, I realized that Google released two models with very similar names but very different capabilities at once. The specific feature I actually wanted (speaker diarization) wasn't available at all on the model I originally thought it was.

Two Models with Names Differing by Only Two Words

Let's lay out the differences first; this is the part I spent the most time figuring out:

	`gemini-3.5-transcribe-live`	`gemini-3.5-transcribe`
API Used	Live API (WebSocket streaming)	Interactions API (Standard HTTP request)
Usage Scenario	Transcribe while speaking	Upload the whole file after recording
Speaker Diarization	Not supported	Up to 8 speakers
Word-level Timestamps	Not supported	Supported
Audio Length	10 minutes per session	1 hour (30 mins with diarization)
Smart Mode	`SMART` available	`smart` is mutually exclusive with diarization
Interim Subtitles	Has `interimInputTranscription`	Not applicable

The official documentation on the Live page's limitations section is very blunt:

Speaker diarization is not supported in live streaming sessions. For speaker diarization, use the non-streaming Audio transcription endpoint.

So, "seeing who is saying what in real-time" is currently impossible. For speaker diarization, you must record it and send the whole thing after the meeting. This limitation determined my entire subsequent architecture.

The Real-time One: Interim is the Key

The setup shape is different from the original translation model. responseModalities must be TEXT, and transcription parameters are placed under setup.inputAudioTranscription:

{
  "setup": {
    "model": "models/gemini-3.5-transcribe-live",
    "generationConfig": { "responseModalities": ["TEXT"] },
    "inputAudioTranscription": {
      "languageCodes": [],
      "mode": "SMART"
    },
    "realtimeInputConfig": {
      "automaticActivityDetection": { "disabled": false }
    }
  }
}

Leaving languageCodes empty enables automatic language detection; filling it with BCP-47 codes like ["en-US"] gives it a language preference. mode has two values: VERBATIM keeps everything word-for-word, while SMART removes filler words like "uh" and "um" and automatically adds formatting. For meeting minutes, I chose SMART, which is much cleaner to read.

There is also a customVocabulary where you can stuff technical terms, with a limit of 1000 items, though the documentation suggests staying under 100 for best results. I didn't implement this yet; I'll wait until I encounter names that are constantly misheard.

The response side adds a field that the original translation model didn't have:

interimInputTranscription: Tentative results while speaking, which will be overwritten by subsequent content.
inputTranscription: The finalized text when the speaker pauses or the turn ends.

This distinction affects how the UI is written, which I'll discuss later.

The Batch One: Uses a Completely Different API

This is the easiest place to trip up. gemini-3.5-transcribe doesn't use generateContent; it uses the Interactions API:

POST https://generativelanguage.googleapis.com/v1beta/interactions

{
  "model": "gemini-3.5-transcribe",
  "input": [
    { "type": "audio", "uri": "YOUR_FILE_URI", "mime_type": "audio/wav" }
  ],
  "generation_config": {
    "transcription_config": {
      "language_codes": [],
      "mode": {
        "type": "verbatim",
        "diarization_mode": "speaker",
        "timestamp_granularities": ["word"]
      }
    }
  }
}

diarization_mode: "speaker" is the switch for speaker diarization, but it can only be paired with verbatim. In other words: if you want speaker diarization, you have to give up the benefits of SMART mode's filler word removal; you can't have both.

I didn't include timestamp_granularities at first because it wasn't in the short example in the documentation. Without it, the entire feature silently fails—a process I'll describe in the final section.

Audio must first be uploaded to the Files API to get a file URI, which is then included in the request. The official documentation doesn't provide a base64 embedding example. The response looks like this:

{
  "steps": [{
    "type": "model_output",
    "content": [{
      "type": "text",
      "text": "Hello world",
      "annotations": [
        { "type": "word_info", "text": "Hello", "speaker": "spk_1",
          "start_offset": "0.100s", "end_offset": "0.450s" }
      ]
    }]
  }]
}

Speaker labels are in the word-level annotations as IDs like spk_1, spk_2. The model doesn't know who is who. The documentation also notes a maximum of 8 speakers, and "attribution for 3 or more people is experimental."

Phase 1: Adding Pure Transcription Mode to the App

This part was simpler than I expected because the App was already parsing inputTranscription and outputTranscription (the translation model returns both for bilingual subtitles). The real change was in the mode branching.

The original connection layer determined it like this:

let isTranslateModel = modelName.contains("live-translate")

A boolean split into two paths. Now needing three, I switched to an enum:

enum LiveMode {
    case translate // gemini-*-live-translate-*: output translated audio + bilingual subtitles
    case transcribe // gemini-*-transcribe-live: text only
    case general // other Live models: rely on systemInstruction for interpretation

    static func from(modelName: String) -> LiveMode {
        if modelName.contains("transcribe") { return .transcribe }
        if modelName.contains("live-translate") { return .translate }
        return .general
    }
}

I also took the opportunity to extract the setup config generation and server response parsing from GeminiLiveConnection into pure functions. That file was getting a bit bloated; extracting these reduced it by over sixty lines, and the extracted parts can be tested directly.

A Preventive Trade-off: Duplicate Words

When parsing responses, I made a decision worth noting. Besides inputTranscription, the parts in modelTurn might also carry the same text. If both are collected, the same sentence will appear twice in the subtitles and export file.

To be honest, I haven't actually seen this happen, but while reading the documentation and old code, I noticed both paths would lead to didReceiveOutputTranscription. I blocked it to avoid the risk:

// Content for pure transcription mode is already provided by inputTranscription.
// Receiving text from modelTurn again would cause the same sentence to appear twice.
if mode != .transcribe,
   let text = part["text"] as? String, !text.isEmpty {
    events.append(.outputTranscription(text))
}

The downside of this preventive defense is that if it never actually happens, this line is dead code that no one understands. So I wrote it as a test ("transcribe mode ignores modelTurn text"), ensuring the behavior is locked in and the intent is documented.

How to Run Tests in a Project Without a Testing Framework

This project doesn't use an Xcode project file; it uses a build_app.sh that calls swiftc to compile all .swift files into a .app. No SwiftPM means no swift test and no XCTest.

My solution was to use the same trick again: since the extracted parts are pure functions, I wrote a main.swift as an assertion runner. I compile it with those pure functions into an executable and use the exit code to determine success or failure.

swiftc -sdk "$SDK_PATH" -target "${ARCH}-apple-macos13.0" \
  -o "${BUILD_DIR}/run_tests" \
  LiveSetupConfig.swift TranscriptFormatter.swift \
  WAVRecorder.swift AudioChunker.swift \
  GeminiTranscribeService.swift GeminiSummaryService.swift \
  Tests/main.swift

The assertion function itself is less than ten lines:

func checkEqual<T: Equatable>(_ actual: T?, _ expected: T, _ name: String) {
    if actual == expected {
        passedCount += 1
    } else {
        failures.append("\(name) (Actual: \(String(describing: actual)), Expected: \(expected))")
    }
}

This obviously can't compete with a real testing framework—no setup/teardown, no parallel execution, and it won't tell you which line failed. But it runs, it prevents regressions, and it doesn't require converting the whole project to SwiftPM just for testing. This phase ended with 37 assertions; after adding speaker diarization, it grew to 98.

Phase 2: Speaker Diarization Requires More Changes Than Expected

"Adding speaker diarization" sounds like calling one more API, but it actually involves three more tasks: saving audio to a file, uploading it, and then connecting to a completely different API. And because batch transcription might take several minutes, it can't block the original stop process.

The final data flow looks like this:

graph TD
    A[ScreenCaptureKit captures PCM] --> B[Gemini Live API<br/>Real-time Subtitles]
    A --> C[WAVRecorder<br/>Synchronous writing]

    D[Press Stop] --> E[Phase 1: Immediate Output]
    E --> E1[Transcript md]
    E --> E2[AI Meeting Summary]
    E --> E3[Meeting Record Webpage]

    D --> F[Phase 2: Background Execution]
    F --> F1[Collect file → Split if > 30 mins]
    F1 --> F2[Upload to Files API]
    F2 --> F3[Interactions API Batch Transcription]
    F3 --> F4[Delete cloud copy]
    F4 --> F5[Write diarized.md]
    F5 --> F6[Rerun summary with speaker info + infer names]
    F6 --> F7[Regenerate webpage]
    F7 --> F8[Delete local recording]

Splitting it into two phases was intentional. Every step in Phase 2 could fail (upload timeout, quota exhausted, response format mismatch). If it were hooked into the middle of the original process, a failure would mean losing even the basic transcript and summary. Now, Phase 1 remains untouched, and Phase 2 is just appended; if it fails, you just lose one output.

The recording format was also conveniently available: the audio sent to the Live API is already resampled to 16kHz mono 16-bit PCM. Writing that same data to disk results in valid WAV content—just add a 44-byte header, no re-encoding required.

The same defensive mindset was applied to parsing: parseDiarized assumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire output_text so the transcript isn't completely empty. This fallback actually came in handy, though not in the way I expected, as I'll explain in the last section.

Major Pitfalls and Solutions

Pitfall 1: Chunking cannot be "fill 30 minutes and leave a remainder"

The audio limit for speaker diarization is 30 minutes, but meetings often last over an hour, so chunking is necessary.

My first idea was the most intuitive: fill 30 minutes for one segment and leave the rest as the last segment. While writing tests, I realized this approach had two holes.

A 60-minute and 5-second meeting would be split into 30 mins, 30 mins, and 5 seconds. Sending that 5-second tail for transcription is meaningless and costs an extra upload and API call. What if I merge the tail into the previous segment? Then that segment becomes 30 mins and 5 seconds, exceeding the API limit, and the whole segment gets rejected.

Cause & Solution: Switch to even splitting. Calculate how many segments are needed (round up), then distribute the total length evenly:

let count = (usable + maxChunkBytes - 1) / maxChunkBytes
guard count > 1 else { return [0..<usable] }

var ranges: [Range<Int>] = []
var start = 0
for index in 1...count {
    var end = usable * index / count
    end -= end % blockAlign // Align to 16-bit sample boundary
    if index == count { end = usable }
    ranges.append(start..<end)
    start = end
}

60 mins and 5 seconds becomes two segments of 30 mins and 2.5 seconds? No, that would exceed the limit. Actually, it's ceil(3605 / 1800) = 3, split into three segments of 20 minutes each. There will never be a tiny remainder segment, and it will never exceed the limit.

The end -= end % blockAlign is also necessary. 16-bit mono uses 2 bytes per sample. If you cut on an odd byte, the entire subsequent audio stream's bytes will be shifted by one, resulting in noise when played.

Pitfall 2: spk_1 in Segment 2 is not the same as spk_1 in Segment 1

A problem I thought of only after chunking. Each segment is an independent API call. The model doesn't know what happened in the previous segment, so spk_1 in Segment 2 has no relation to spk_1 in Segment 1; they could be different people.

If you just concatenate the three segments, the reader will naturally assume spk_1 is the same person throughout. This is worse than having no speaker labels: it gives a false sense of certainty.

Cause & Solution: Whenever chunking occurs, append the segment number to the ID to create a namespace:

static func qualifiedLabel(chunkIndex: Int, chunkCount: Int, speaker: String) -> String {
    guard chunkCount > 1 else { return speaker }
    return "Seg\(chunkIndex + 1)-\(speaker)"
}

Insert an explanatory line at the segment boundaries during export:

---

> Segment 2 (Speaker IDs are not continuous with the previous segment)

These IDs are also used for the text sent to the AI for name inference. By sharing the same vocabulary, the model has a chance to link the same person across segments—if someone is called "Evan" in both Segment 1 and Segment 3, it can map them individually rather than being forced to assume the IDs are identical.

Pitfall 3: Joining words results in "H e l l o W o r l d"

The batch API returns word-level annotations, which you have to join into sentences yourself. For English, it's intuitive: join with spaces.

The problem is Chinese. Gemini's Chinese tokens joined with spaces look like "Ni hao shi jie wo men jin tian" (Hello world we today), looking like a word segmentation exercise.

Cause & Solution: When joining characters, check the properties of the characters on both sides. If either side is CJK (Chinese, Japanese, Korean), don't add a space; also, don't add a space before punctuation:

private static func needsSpace(after previous: Character, before next: Character) -> Bool {
    if isCJK(previous) || isCJK(next) { return false }
    if next.isPunctuation { return false }
    return true
}

isCJK checks Unicode blocks, covering CJK Unified Ideographs, Kana, Hangul, and full-width characters.

The punctuation rule was added later. Originally I only blocked CJK, but testing ["Hello", ",", "world"] revealed it became Hello , world. You'd never notice this without writing tests because it doesn't "break"—it's just a bit ugly.

Pitfall 4: Two asynchronous processes racing to write the same summary

Phase 1 generates an AI meeting summary after stopping. Phase 2 generates another one after getting the diarized transcript (this time with speaker info, so the "Assignee" field in action items can be filled).

Normally, Phase 2 is definitely slower—it has to upload over 100MB of audio and wait for transcription. But "normally slower" isn't a guarantee. If the Phase 1 summary API happens to hang and retry, while the Phase 2 audio is only one minute long and finishes quickly, the order will reverse. The late-returning old summary from Phase 1 would overwrite the diarized result.

Cause & Solution: Phase 2 waits for Phase 1 to finish before writing:

// Wait for Phase 1 summary to land, otherwise it might return after us and overwrite the diarized result
await minutesTask?.value
guard !Task.isCancelled else { return }

A one-line fix, but you have to first realize that "these two things actually have no guaranteed order." This kind of race almost never appears in testing; it only happens on a day with particularly bad network, leaving the user with a confusingly reverted meeting record.

Pitfall 5: During the TDD RED phase, the test itself crashed

While writing tests for the WAV header, I followed TDD rules: write the test first, create a stub returning an empty Data(), and run it to see it fail. Instead of failing, the entire test program crashed:

Swift/arm64e-apple-macos.swiftinterface:41299: Fatal error:
UnsafeRawBufferPointer.load out of bounds
Trace/BPT trap: 5

My test was trying to read the 24th byte to check if the sample rate was 16000, but the stub returned empty Data, causing an out-of-bounds read.

Cause & Solution: Zero-pad before reading:

let produced = WAVRecorder.header(dataByteCount: 64000)
checkEqual(produced.count, 44, "WAV header is 44 bytes")

// Pad with zeros if length is insufficient, so subsequent fields report failure instead of crashing the test
let header = produced + Data(repeating: 0, count: max(0, 44 - produced.count))

This is a small thing, but it highlighted something I usually ignore: the test program itself must be resilient to the object under test being completely broken. If a test ends in a crash during the RED phase instead of reporting a failure, you only know "something broke," not which of the twelve fields were wrong. After padding with zeros, a single run lists all twelve expected values, allowing me to implement them by following a list rather than guessing.

Pitfall 6: Scroll anchor fails in the new mode

There's a structural difference between real-time transcription and translation modes: translation mode accumulates words into the "current sentence" and pushes to history only when punctuation is reached; transcription mode's inputTranscription is a finalized sentence that goes straight to history.

The original auto-scroll was written like this:

proxy.scrollTo("currentLine", anchor: .bottom)

The ID currentLine was attached to the display block for the "current sentence." In transcription mode, the sentence goes to history as soon as it's finalized, and that block immediately disappears—the scroll target no longer exists, so the screen stays put.

Cause & Solution: Use a bottom anchor that always exists:

Color.clear
    .frame(height: 1)
    .id("bottomAnchor")

This bug had no error message or crash; the "new feature just wouldn't scroll," and it only became apparent when there was enough content to exceed the screen. I found it by reading the view's conditional branches, not by running it.

Pitfall 7: Letting AI guess names—the key is stopping it from hallucinating

Once you have spk_1 and spk_2, the natural desire is to replace the IDs with real names. This is actually feasible—meetings often contain clues like "Evan, how's the progress on your end?" or "I'm Sarah, in charge of frontend." If you give the tagged transcript to the model, it can make the connection.

But this is also where hallucinations are most likely. Models are happy to "infer" a name from tone, job content, or speaking frequency and present it with the same confidence as a fact in the meeting minutes.

Cause & Solution: Three methods combined.

First, the schema explicitly allows null and requires evidence:

properties["speakers"] = [
    "type": "ARRAY",
    "items": [
        "type": "OBJECT",
        "properties": [
            "label": ["type": "STRING"],
            // Must return null if no clues are found, rather than forcing a name
            "name": ["type": "STRING", "nullable": true],
            "evidence": ["type": "STRING", "nullable": true]
        ],
        "required": ["label"]
    ]
]

Second, the prompt sets strict rules:

Only fill in the name if there is a clear address, roll call, or self-introduction in the transcript or meeting notes. Explain in 'evidence' which sentence led to this conclusion. Do not speculate on names based on tone, job content, or speaking frequency. If no clear clues are found, both name and evidence must return null.

Third, remove the confidence score. I originally designed a confidence field but later removed it. The reason is that a model's self-assessment of confidence is inherently unreliable, and evidence already fully serves this role: if there's evidence, the inference succeeded; if not, it didn't. An extra confidence score just makes people think it's more credible than it is—"It says 0.7 here, so it's probably 70% accurate"—but that 0.7 doesn't come from any real probability distribution.

The output looks like this, with evidence for successful inferences and honest admissions for failures:

## Participants
- **Evan** (spk_1) — Evidence: Addressed as "Evan, how's the progress on your end?" in Segment 3
- spk_2 — Insufficient clues in transcript and notes to identify name

Keeping the ID in parentheses is also intentional. Seeing **Evan (spk_1)** tells you it was inferred; if it's wrong, you can verify it yourself. If it just said **Evan**, it would look like an absolute fact.

Another source of clues: I also send the meeting notes the user wrote on the spot. Notes often already contain a list of participants, which significantly increases the success rate. But a safeguard is needed here—the prompt must clearly state "Notes can only be used to map speaker names; do not treat note content as something someone said in the summary or action items," otherwise your own memos might turn into someone else's speech.

Privacy and Cleanup: Don't leave behind what you don't want to keep

Batch transcription inherently creates two more pieces of data than real-time streaming: the local recording file and the copy uploaded to Google. Both must have a clear disposal plan.

Recording files are only created if the option is checked. If speaker diarization isn't enabled, no file is written, and nothing stays on the disk.
Local recordings are deleted upon successful transcription. A one-hour meeting is about 115MB, which adds up quickly.
Files are kept if transcription fails. This is an intentional exception: if it fails, the recording is kept, and the status bar tells you the path so you can retry or handle it yourself. Deleting it then would be actual data loss.
Cloud copies are deleted immediately after use. Files API files expire automatically after 48 hours, but "it will disappear eventually" and "I'm sure it's gone now" are two different things.

// Delete the cloud copy immediately after use, don't wait 48 hours for auto-expiration
await GeminiFilesUploader.delete(apiKey: apiKey, name: uploaded.name)

Results and Benefits

The numbers for both phases combined:

	Pure Transcription Mode	Speaker Diarization	Post-testing Fixes
New Files	2	4	0
Total Assertions	37	98	108
Commit	`b0e12c5`	`486ba8f`	`686899a`, `1cc41c3`

GeminiLiveConnection.swift, the file that originally did everything, was reduced by over sixty lines after setup generation and response parsing were extracted. Those two pure function modules are now guarded by over 20 tests. This was an unexpected benefit: the decoupling done to make things testable was itself a refactoring I had put off for a long time.

First, confirm which model has the feature you want. I initially assumed "Real-time Transcription" would have speaker diarization—they're both transcription models, right? The only difference should be real-time vs. batch. I only found out otherwise after checking the docs, and the difference isn't just a parameter switch; it's architectural: for diarization, you must record, upload, use a different API, accept a 30-minute limit, and give up SMART mode. If I hadn't checked first, I would have hit a whole new subsystem while expecting to just "change a few parameters."

Limitations often dictate the architecture. Almost every design decision this time was forced by limitations: the 30-minute limit forced chunking, chunking forced ID namespaces, the slowness of batching forced two-phase export, and the mutual exclusivity of smart and diarization forced me to choose between a clean transcript and speaker labels. Checking limitations before designing is much easier than designing and then hitting a wall.

The real-time transcription half has been tested in the field. The screenshot above is the result of feeding it a Japanese video; gemini-3.5-transcribe-live automatically detected Japanese and output a Japanese transcript directly without translation, and the meeting record webpage was generated as usual after stopping. Leaving languageCodes empty for auto-detection actually works.

The speaker diarization half had issues during field testing, and in a way I didn't expect: the file was generated, the transcript was correct, and the program reported no errors, but there was no speaker separation. This part deserves its own section.

Postscript: Documentation examples might give you an empty result without errors

After posting, I ran speaker diarization on a real conversation. meeting-2026-08-28-11-54-diarized.md was generated, the content was complete, not a word was missing, but there were no speaker labels from beginning to end—just one continuous block of text.

No error messages, the status bar showed success, and the file had everything it should. This kind of failure is the hardest to debug because it looks like success.

Create a Minimal Reproducible Example First

The first problem was that I had no evidence—the App didn't save the raw response, and the recording of that meeting was automatically deleted because "transcription succeeded." To re-run it, I'd have to start another meeting with no guarantee of reproduction.

So instead of guessing what broke, I first found a way to get a raw response. macOS's built-in say command can use different voices, so I used it to synthesize a two-person conversation:

say -v Alex -o a1.aiff "Hi Samantha, did you finish the quarterly report yesterday?"
say -v Samantha -o a2.aiff "Yes Alex, I sent it to the whole team this morning."
say -v Alex -o a3.aiff "Sure, I will look at the budget section this afternoon."

for f in a1 a2 a3; do afconvert -f WAVE -d LEI16@16000 -c 1 $f.aiff $f.wav; done

The three segments joined together are 17 seconds long with two speakers, in the exact same format as the App's recordings (16kHz mono 16-bit). Then I used curl to go through the entire upload and transcription process, dumping the full JSON.

This step took less than five minutes, but it turned "starting another meeting, running for ten minutes, and not being sure of reproduction" into "changing one field, running for ten seconds, and seeing the difference immediately." A minimal reproducible example is worth the time, especially when the original reproduction path is expensive.

The dumped response looked like this:

{
  "steps": [{
    "content": [{ "text": "Hi Samantha, did you finish...", "type": "text" }],
    "type": "model_output"
  }]
}

811 bytes, complete transcript, zero annotations. So the problem wasn't my parsing; it was the request.

Root Cause 1: Word-level annotations must be explicitly enabled

Using the same audio and the same uploaded file, I changed just one field in the request:

Request Content	Objects with `speaker` in response
Only `diarization_mode: "speaker"`	0
Removed `language_codes: []`	0
Added `timestamp_granularities: ["word"]`	41

Speaker IDs are attached to word_info annotations, and word-level annotations must be explicitly requested in the request to be returned. No request, no annotations; no annotations, no speakers.

I didn't include this field originally because I copied the shortest Python example from the documentation—that example only had type and diarization_mode. The full REST example elsewhere in the documentation actually has timestamp_granularities, but by then I already "knew" how to write it and didn't look back.

The most frustrating part is that it doesn't report an error. The API returns 200, gives you the full transcript, and the status is completed. If it had returned an error like "You requested speaker diarization but didn't enable word annotations," I would have fixed it in five minutes.

Root Cause 2: The actual label format differs from the documentation

After adding the field, annotations appeared, but they looked different from what I expected:

{"text":"Hi,","start_offset":"0.100s","end_offset":"0.500s","speaker":"spk:0","type":"word_info"}

spk:0. A colon, and starting from 0. The documentation consistently uses spk_1, spk_2.

My display logic was written like this:

if let range = speaker.range(of: "spk_"), let number = Int(speaker[range.upperBound...]) {
    return "Speaker \(number)"
}
return speaker // ← Fallback prints the raw ID if no match

spk:0 doesn't match spk_, so it fell straight into the fallback, displaying **spk:0**: on the screen.

My tests completely missed this bug for a simple reason: the test data was written according to the documentation. The documentation was wrong, so the tests were wrong, and the green light told me everything was fine. This is the main thing I want to record: for external API tests, you are actually testing "my understanding of this API," not the API itself. If your understanding is wrong, the test will faithfully protect that error.

I didn't fix it by changing spk_ to spk:; that would just be betting in a different direction. Instead, I stopped parsing the number in the ID and used the order in which the speaker first appeared in that segment:

/// The actual format of the ID is determined by the API (actual returns spk:0, spk:1, while official docs say spk_1),
/// so we don't parse the number in the ID. Instead, we number them based on their first appearance in this segment.
static func speakerOrder(in chunk: [DiarizedSegment]) -> [String: Int] {
    var order: [String: Int] = [:]
    for segment in chunk where !segment.speaker.isEmpty {
        if order[segment.speaker] == nil {
            order[segment.speaker] = order.count + 1
        }
    }
    return order
}

It won't break even if the format changes again because it doesn't look at the format at all.

My own fallback made failure look like success

I wrote this earlier and was quite proud of it:

parseDiarized assumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire output_text so the transcript isn't completely empty.

This fallback did work, and the effect was as expected: the user got a complete transcript with nothing lost.

But it also hid the failure. If I hadn't had that fallback, -diarized.md would have been empty or not generated at all, and I would have known immediately that something was wrong. With it, I got a file that looked perfectly normal, just missing the feature I wanted—the very feature that was the sole reason for enabling it.

I haven't fully figured out the balance here. Graceful degradation isn't wrong; the mistake is not speaking up after degrading. My current approach is to keep the fallback but change the status bar message when it's triggered, explicitly stating "Speaker info not obtained this time" instead of the usual "Diarized transcript saved."

After the Fix

Running the full parsing with a real response, both segments spoken by Alex correctly returned to Speaker 1:

**Speaker 1**: Hi, Samantha. Did you finish the quarterly report yesterday?
**Speaker 2**: Yes, Alex. I sent it to the whole team this morning. Could you review the budget section?
**Speaker 1**: Sure. I will look at the budget section this afternoon and get back to you.

The quality of the speaker diarization itself is good. Tests increased from 98 to 106, with the extra eight being regression tests for these two root causes—this time, the test data wasn't copied from the docs but clipped from real responses.

Three more details learned from field testing:

The REST endpoint only accepts snake_case; sending generationConfig is explicitly rejected with Unknown parameter 'generationConfig'. Did you mean 'generation_config'?. This is a great error message, a hundred times more useful than the silent failure above.
Annotations include start_index and end_index, mapping directly to positions in content.text. Using these to split strings is much more accurate than my heuristic for joining CJK/English words, meaning the entire joinWords could be removed. I haven't done this yet.
The state of uploaded files is immediately ACTIVE; 17 seconds of audio didn't go through a PROCESSING phase. My polling logic seems redundant for short audio, but I don't know if it's needed for long audio, so I'll keep it for now.

This article thus has two endings, and I've decided to keep both. The first part said "Speaker diarization hasn't been verified yet; I'll update after I run it," and then I actually ran it, and it broke. If I had edited the first part and only kept the fixed version, this would have been a smooth "I did X, and it worked" post—but what actually happened was "I did X according to the docs, it failed silently, and it took me half an hour to find out why." The latter is much more useful to readers.

The code is at kkdai/gemini-live-translate-macos. The two official documents are Live transcription and Audio transcription. I recommend reading the limitations section on the speaker diarization page thoroughly before starting.

[Go in Practice] Writing Modern Go with AI: Testing JetBrains go-modern-guidelines and Refactoring a 1,039-line main.go

Evan Lin — Thu, 27 Aug 2026 04:13:56 +0000

Background

In the AI era, even I delegate most of my code optimization or writing tasks to AI. However, due to factors in model training data, too many writing styles are outdated. This results in code that cannot utilize features of the latest Go versions, which is quite a pity.

Fortunately, JetBrains released go-modern-guidelines, a very useful plugin. It makes your AI Agent smarter and teaches it how to use the latest syntax to optimize your Golang code.

What is go-modern-guidelines?

The problem it aims to solve: Models have a knowledge cutoff, Go doesn't

The positioning of this project is very straightforward: Provide contemporary Go writing specifications for AI agents so they don't write outdated Go due to knowledge cutoffs.

The problem has two layers. The first layer is easy to understand: training data has a cutoff. Anything added to the standard library after that cutoff won't be used because the model hasn't seen it. The project's own example is errors.AsType[T] (Go 1.26); if the model hasn't seen it, it naturally won't write it.

The second layer is more subtle, which the project calls frequency bias: even if the model "knows" the new way, the old way appears overwhelmingly more often in the training data. In ten years of Go code on the internet, interface{} appears far more than any, and sort.Slice far more than slices.SortFunc. Models perform probabilistic prediction; the one that wins by majority vote is usually the old one.

I really saw this second point in this refactoring. The original project had this snippet:

// The oauth2 library can return an error containing "invalid_grant"
// when the refresh token is expired, revoked, or otherwise invalid.
if err != nil {
    errorStr := err.Error()
    // Basic substring check to avoid importing "strings"
    for i := 0; i <= len(errorStr)-13; i++ {
        if errorStr[i:i+13] == "invalid_grant" {
            return true
        }
    }
}

A hand-rolled string search, with a comment specifically explaining "to avoid importing strings". strings is in the standard library; the cost of importing it is zero. What this code actually needed was just one line: strings.Contains(err.Error(), "invalid_grant").

How it works: Two commands, a list that grows with Go versions

The tool itself is a CLI with only two subcommands:

list [--go-version <version> | --file-path <path>]
    Returns a list of guidelines supported by this Go version, sorted from newest to oldest.

explain <id>...
    Returns detailed explanations and before/after examples for specific guidelines.

The key point of list is that it provides different answers based on the Go version. You can pass a file path directly, and it will look up for go.mod, go.work, or fall back to the local Go toolchain:

$ go-modern-guidelines list --file-path ~/Documents/linebot-file/main.go

My project's go.mod specifies go 1.24.0, so it returned 45 guidelines. Change the version number, and the count changes:

Go Version	Guideline Count
1.21	32
1.22	37
1.23	41
1.24	45
1.25	46
1.26	48
1.27	54

This design is intentional: it only suggests syntax that your project version can actually use. This is crucial for AI agents; otherwise, it might happily suggest errors.AsType[T], and your CI would fail because it's running on Go 1.24.

Looking at the differences between versions, it's essentially a condensed list of Go's recent features:

$ diff <(list --go-version 1.21) <(list --go-version 1.22)
> range_over_int: Use for i := range n when iterating from 0 to n-1.
> loopvar_capture: Do not add redundant loop-variable copies before closures or
  taking addresses; Go 1.22 gives each iteration its own variables.
> cmp_or: Use cmp.Or to pick the first non-zero value from a fallback chain.
> reflect_type_for: Use reflect.TypeFor[T]() instead of reflect.TypeOf((*T)(nil)).Elem().
> http_servemux_patterns: Use method-aware ServeMux patterns and r.PathValue for
  path parameters.

list provides a one-line summary; use explain for detailed instructions when you're ready to work. Output looks like this:

$ go-modern-guidelines explain cmp_or

cmp_or:
  Since: Go 1.22

  Summary:
    Use cmp.Or to pick the first non-zero value from a fallback chain.

  Details:
    cmp.Or returns the first non-zero value from its arguments. It is concise
    for simple fallback chains, but remember that all arguments are evaluated
    before the call.

  Examples:

  Before:
    name := os.Getenv("NAME")
    if name == "" {
      name = "default"
    }

  After:
    name := cmp.Or(os.Getenv("NAME"), "default")

Note the last sentence in the Details section: "all arguments are evaluated before the call." This is the real trap of cmp.Or—if your fallback source is an expensive function call, writing cmp.Or(a(), b()) will execute both. This kind of "you can use it, but know the cost" reminder is much more useful than just telling you to change the syntax.

The two-layer information design is actually to save context

This list/explain layering might look like just interface design, but it's actually for the AI agent's context window. 45 guidelines, each with a one-line summary, take about 1000 tokens; but if every guideline included full explanations and before/after examples, just stuffing this list would burn tens of thousands of tokens.

So the workflow is: first list to scan everything, determine which guidelines are relevant to the current code, and then only call explain for those. In this case, I actually only explained six guidelines.

graph TD
    A[Prepare to modify Go code] --> B[list --file-path main.go]
    B --> C[Parse Go version from go.mod]
    C --> D[Return 45 guidelines available for that version<br/>one-line summary each]
    D --> E{Which ones are relevant to this code?}
    E -->|Pick candidates| F[explain cmp_or min_max ...]
    F --> G[Get detailed explanations and before/after]
    G --> H[Actually apply to code]
    E -->|None relevant| I[Write in original way]

There's one rule in the skill documentation written with particular emphasis: Do not pipe the output of list to head, tail, or grep, as you might miss important guidelines. I violated this rule on my first try, which I'll discuss in the "Pitfalls" section.

Installation

For Claude Code, it's two lines:

/plugin marketplace add JetBrains/go-modern-guidelines
/plugin install modern-go-guidelines@goland-claude-marketplace

Once installed, it triggers automatically for Go-related tasks, or you can call it manually: /modern-go-guidelines:use-modern-go. Cursor, Junie, and Codex have their own installation methods; other agents can use npx skills add JetBrains/go-modern-guidelines. The project is licensed under Apache 2.0.

On the first run, the wrapper script will automatically install the CLI to the local cache directory:

go-modern-guidelines: installing github.com/JetBrains/go-modern-guidelines@v0.1.1
  into /Users/xxx/.cache/go-modern-guidelines/v0.1.1

This project: a main.go that grew to 1039 lines

First, some background. The architecture of linebot-file isn't complex:

graph LR
    A[LINE App] -->|Send file| B[LINE Platform]
    B -->|webhook| C[Cloud Run]
    C -->|Read token| D[(Firestore)]
    C -->|Upload/Query| E[Google Drive API]

Users authorize with /connect_drive, tokens are stored in Firestore, and files sent to the chatroom are automatically uploaded to a folder structure like LINE Bot Uploads/YYYY-MM/. Features were added incrementally, and everything was piled into main.go, where the main() function itself took up 564 lines.

What the health check found

This section isn't directly related to go-modern-guidelines—that tool manages whether the "writing style is contemporary," not whether the "logic is correct." But these are the things that actually bite users, so I'll record them anyway.

1. Code that compiles but will never execute

This is the most interesting one. The original event handling looked like this:

switch e := event.(type) {
case webhook.MessageEvent:
    switch message := e.Message.(type) {
    case webhook.TextMessageContent:
        // ...
    case webhook.FileMessageContent:
        // ...
    case webhook.FollowEvent: // ← Note the indentation level here
        if s, ok := e.Source.(*webhook.UserSource); ok {
            bot.LinkRichMenuIdToUser(s.UserId, richMenuConnect)
        }
    }
}

webhook.FollowEvent was written inside the inner switch. The inner switch evaluates e.Message, which is of type MessageContentInterface—a follow event can never be the content of a message.

Why did it compile? Go does check type switches; if a case type cannot possibly implement that interface, the compiler reports impossible type switch case. The problem lies in the SDK's interface definition:

type MessageContentInterface interface {
    GetType() string
}

It only requires a GetType() string. And FollowEvent happens to have this method (all event types do), so in the type system, it "can" be a MessageContentInterface. The compiler allows it, but it never matches at runtime.

Actual consequence: When new users add the bot as a friend, the Rich Menu for guiding authorization was never bound. This feature had probably been broken for a long time because it doesn't report an error; it just quietly does nothing.

2. Group messages cause a panic

userID := e.Source.(webhook.UserSource).UserId

Unchecked type assertions appeared six times. As long as the bot is pulled into a group and someone sends an image, this line panics.

By the way, several other places in the same file used e.Source.(*webhook.GroupSource) (pointer). Checking the SDK's UnmarshalSource, it returns a value, not a pointer, so those assertions with , ok were always false—also dead code. Two different wrong ways in the same file, in opposite directions.

3. /recent_files returns folders

// When uploading: file is placed in LINE Bot Uploads/YYYY-MM/
monthFolderID, _ := findOrCreateFolder(srv, "2026-08", mainFolderID)
srv.Files.Create(&drive.File{Parents: []string{monthFolderID}})

// When querying: only look under LINE Bot Uploads
query := fmt.Sprintf("'%s' in parents and trashed=false", mainFolderID)

Files are stored in monthly subfolders, but the query only looks at the root folder. In the Google Drive data model, a folder is also a type of file, so this query does return things—it returns the 2026-08, 2026-07 folders themselves.

4. User input directly concatenated into the Drive query

query := fmt.Sprintf("... and name contains '%s'", searchQuery)

No escaping. If a user searches for it's, that single quote breaks the query syntax; thinking further, extra query conditions could be injected. The fix is to properly write an escaping function, noting that the order cannot be reversed:

// Backslashes must be escaped first, otherwise the backslash added 
// to escape a quote will be escaped again in the second round of processing.
func escapeDriveQuery(s string) string {
    s = strings.ReplaceAll(s, `\`, `\\`)
    return strings.ReplaceAll(s, `'`, `\'`)
}

5. /quit treated as a search command

} else if (len(message.Text) > 13 && message.Text[:13] == "/search_files") ||
          (len(message.Text) > 2 && message.Text[:2] == "/q") {
    commandPrefixLen := 0
    if ... {
        commandPrefixLen = 14 // Length of "/search_files "
    } else if ... {
        commandPrefixLen = 3 // Length of "/q "
    }
    searchQuery = message.Text[commandPrefixLen:]

Manual string slicing, and it hardcoded "must be followed by a space." If a user types /quit, the first two characters are /q, so it becomes a search for it.

What go-modern-guidelines actually changed

Back to the topic. Out of those 45 guidelines from list, these were the ones actually applied in this change:

`http_servemux_patterns`: Also removed a piece of manual path checking

The original way was to have all requests go to the same handler and then determine the path manually:

http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
    // LINE Platform must POST to the webhook URL
    if req.URL.Path != "/" {
        http.NotFound(w, req)
        return
    }
    // ...
})

After Go 1.22, ServeMux patterns support methods and exact paths:

mux := http.NewServeMux()
// "/{$}" only matches the root path, not all paths beneath it
mux.HandleFunc("POST /{$}", webhookHandler)
mux.HandleFunc("GET /oauth/callback", oauthCallbackHandler)
// Cannot be called /healthz, see Pitfall 5 for the reason
mux.HandleFunc("GET /health", healthHandler)

The {$} syntax is key: "/" in ServeMux is a subtree pattern that consumes all paths beneath it, which is why the original code needed that manual check. "/{$}" only matches the root path itself, so the check is no longer needed. I also added method restrictions and a health check endpoint while I was at it.

The health check endpoint issue came up later, but that was discovered after deployment; I'll save that for Pitfall 5.

`cmp_or`: Three fallback segments turned into three lines

// Before
port := os.Getenv("PORT")
if port == "" {
    port = "5000"
}

// After (also changed default to 8080 to align with Dockerfile EXPOSE and Cloud Run convention)
port := cmp.Or(os.Getenv("PORT"), "8080")
richMenuConnect = cmp.Or(os.Getenv("RICH_MENU_CONNECT"), defaultRichMenuConnect)
richMenuMain = cmp.Or(os.Getenv("RICH_MENU_MAIN"), defaultRichMenuMain)

This exactly fits the usage conditions reminded by explain: all three parameters are os.Getenv and constants, and evaluating them all has no side effects.

`strings_cut_prefix_suffix`: Replacing manual string slicing

The command parsing from point five earlier, message.Text[:13], was replaced with a proper parsing function:

func parseCommand(text string) (name, arg string, ok bool) {
    text = strings.TrimSpace(text)
    if !strings.HasPrefix(text, "/") {
        return "", "", false
    }

    name, arg, _ = strings.Cut(text, " ")
    switch name {
    case cmdConnect, cmdReconnect, cmdDisconnect, cmdRecent, cmdSearch, cmdSearchShort:
        return name, strings.TrimSpace(arg), true
    }
    return "", "", false
}

Changing to strings.Cut to first slice out the full command name and then using a switch for comparison structurally eliminated the /quit bug—the sliced name is /quit, which isn't in the allowed list, so it returns false directly.

`slices_sort_func` + `min`: Fixing that fake sorting

The original search results looked like this after deduplication:

// Remove duplicates and sort by creation time (newest first)
uniqueFiles := make(map[string]*drive.File)
for _, file := range files {
    if _, exists := uniqueFiles[file.Id]; !exists {
        uniqueFiles[file.Id] = file
    }
}

result := make([]*drive.File, 0, len(uniqueFiles))
for _, file := range uniqueFiles {
    result = append(result, file)
}

if len(result) > 10 {
    result = result[:10]
}

The comment says "sort by creation time (newest first)", but in reality, there was no sorting action at all—map iteration order is random, and then it just truncated the first 10 items. So users got 10 random items, not the 10 newest ones.

// createdTime returned by Drive is an RFC 3339 UTC string; direct string comparison is the correct chronological order
func sortAndTrimFiles(files []*drive.File, limit int) []*drive.File {
    slices.SortStableFunc(files, func(a, b *drive.File) int {
        return cmp.Compare(b.CreatedTime, a.CreatedTime)
    })
    return files[:min(len(files), limit)]
}

The min built-in function (Go 1.21) saves an if here.

`any`, `errors_is`: Small details

Changing map[string]interface{} to map[string]any; no need to say more about such one-liners.

`crypto/rand.Text()`: A suggestion that requires a version upgrade first

Original way to generate OAuth state:

func generateState() string {
    b := make([]byte, 16)
    rand.Read(b) // Error ignored
    return base64.URLEncoding.EncodeToString(b)
}

crypto/rand.Text(), added in Go 1.24, returns a random string directly, won't fail, and the output is base32 (A-Z, 2-7), which is naturally URL-safe—perfect for states and Firestore document IDs:

func generateState() string {
    return rand.Text()
}

But there's a prerequisite for this one, which I'll discuss below.

Major Pitfalls and Solutions

Pitfall 1: The skill explicitly tells you not to grep, and I did exactly the opposite the first time

The skill documentation is clear:

Do not pipe the output through head, tail, grep, sed, or any other truncating/filtering command. Important guidelines may otherwise be missed.

The first time I called it, I typed:

$ go-modern-guidelines list --file-path main.go 2>&1 | tail -60

Purely a reflex to avoid flooding the screen with long output. In hindsight, I realized this was dangerous on two levels: first, list explicitly states it is sorted from newest to oldest, so tail gets exactly the oldest batch; second, I only got away with it this time because the go.mod specified 1.23, totaling 41 lines, which is less than 60, so tail -60 printed everything.

Reason and Solution: Pure luck. If the project had been Go 1.27 (54 guidelines), tail -60 still wouldn't have truncated; but if I had typed head -20 or grep slices, I would have missed entire batches of items with absolutely no hint that I missed anything. This kind of "output truncated but looks normal" failure is the hardest to detect. Just read the full output honestly; it's only 45 lines.

Pitfall 2: Your go.mod might not allow the syntax suggested by the tool

rand.Text() appeared in the suggestion list, but the project's go.mod at the time was:

module github.com/kkdai/linebot-file

// +heroku goVersion go1.21
go 1.23.0

toolchain go1.24.3

The go 1.23.0 line determines the language version, which is different from the toolchain. The tool answers based on the version it can parse, but to actually use rand.Text(), you have to modify go.mod.

It's not just about brainlessly changing one line; you have to ensure everything in the chain aligns: the toolchain was already go1.24.3, the Dockerfile used golang:1.24-alpine, both were fine. However, the CI had an issue—.github/workflows/go.yml hardcoded go-version: '1.22', which is older than what go.mod required; it was currently only not breaking because of Go's automatic toolchain download mechanism.

Reason and Solution: Updated go.mod to go 1.24.0, cleared out that outdated // +heroku goVersion go1.21 line (this project has long been running on Cloud Run), and changed the CI to use go.mod as the single source of truth:

- uses: actions/setup-go@v5
  with:
    # Use go.mod as the single source of truth to avoid CI and project version inconsistency
    go-version-file: go.mod

Pitfall 3: After changing go.mod, the tool's answers changed

This was the most interesting discovery this time. After upgrading go.mod, I ran list again before writing tests and found four new items at the top of the list:

testing_t_context: Use t.Context() when a test function needs a context tied to
                   the test lifetime.
json_omitzero: Use omitzero on JSON-tagged bool, numeric, struct, and time
                   fields whose zero value should be omitted...
testing_b_loop: Use b.Loop() for the main loop in benchmark functions.
strings_split_seq: Use strings or bytes SplitSeq and FieldsSeq helpers...

These four are exactly what was added in Go 1.24. And testing_t_context directly changed the test I was currently writing:

// Before
srv, err := drive.NewService(context.Background(),
    option.WithEndpoint(server.URL), option.WithoutAuthentication())

// After — context bound to test lifetime, automatically canceled when test ends
srv, err := drive.NewService(t.Context(),
    option.WithEndpoint(server.URL), option.WithoutAuthentication())

Reason and Solution: The output of this tool changes according to the project state; it's not a static document. Upgrade the version or switch projects, and the answers change. So the correct usage isn't to check once before starting and be done, but to rerun it when the nature of the changes shifts—in my case, I reran it at the junction of "finished main program, starting tests," and happened to catch testing_t_context. If I had only checked at the very beginning, I would have missed it.

Pitfall 4: The tool manages syntax, not architecture—and architectural pitfalls are deeper

This is the reverse: where go-modern-guidelines doesn't and shouldn't have an opinion.

Originally, file uploads were done synchronously in the webhook handler: downloading a video from LINE and then uploading to Drive could take dozens of seconds. LINE expects a response within a certain time; if it times out, it retries, and retries would cause duplicate uploads of the same file.

The standard advice for this is almost reflexive: return 200 first, and throw the rest into a goroutine. I thought the same at first, but halfway through writing, I remembered something—this service runs on Cloud Run, which by default only allocates CPU during request processing. Once the response is sent, that goroutine will be throttled by the CPU, becoming a black hole that looks like it's working but actually has no idea when it will finish. This is worse than synchronous processing; at least synchronous processing fails honestly.

Reason and Solution: Changed to use the webhook's event ID for deduplication so that retries don't cause duplicate uploads, while keeping synchronous processing:

// handledEvents remembers recently processed webhook event IDs. LINE will resend
// requests it considers failed; without this protection, resending would upload the same file again.
type handledEvents struct {
    mu sync.Mutex
    seen map[string]time.Time
}

func (h *handledEvents) markHandled(id string) bool {
    if id == "" {
        return true // Without an ID, there's no way to deduplicate, so treat as a new event
    }
    // ... clear expired ones, then check for duplicates
}

Added a fallback of "if the reply token expires, use push message" so that the user is still notified after a large file finishes uploading.

This is a compromise; the real solution is to use Cloud Tasks or Pub/Sub. I've added it to the project roadmap, including the reason "can't just use goroutines"—otherwise, the next person taking over (likely me in three months) will probably hit the same pitfall again.

Pitfall 5: ServeMux was written correctly, but Cloud Run won't let you use it

After the PR was merged, I checked Cloud Build with gcloud; the status was SUCCESS, the new revision was Ready, and all traffic was switched over. It looked like a job well done.

Poked the endpoints:

GET / 405 ← method-aware ServeMux in effect
POST / No signature 400 ← signature verification in effect
GET /nope 404 ← {$} exact match in effect
GET /healthz 404 ← ?

The first three were correct, but the health check returned 404.

At first, I thought I wrote the route wrong, but I only realized something was off after printing the response content—it was a Google-branded HTML error page (Error 404 (Not Found)!!1, with the Google robot image), not Go's 404 page not found plain text. This meant the request never even reached my program.

Checking the Cloud Run request logs confirmed this:

15:44:54 GET 400 /oauth/callback
15:44:36 GET 404 /nope
15:44:36 POST 400 /
15:44:36 GET 405 /

I sent five requests, but there were only four in the log. The two /healthz requests didn't even have a record.

Scanning various common health check paths narrowed it down significantly:

/healthz 404 GFE(Google) ← Intercepted
/healthz/ 404 app(Go) ← Only one slash difference
/health 404 app(Go)
/readyz 404 app(Go)
/livez 404 app(Go)
/_ah/health 404 app(Go)
/status 404 app(Go)
/ping 404 app(Go)
/healthcheck 404 app(Go)

Only the exact path /healthz was intercepted by Google Frontend; even adding a slash allowed it to reach the app normally. I looked it up and found this is a known behavior of Cloud Run, which Streamlit and n8n have also encountered.

Reason and Solution: Changed the endpoint to /health, a one-line fix. The annoying part is that this pitfall doesn't make a sound—go vet doesn't speak, tests don't speak, CI is all green, build succeeds, Cloud Run shows Ready, and even request logs leave no trace. The only way to find it is to actually poke the endpoint and notice that the returned 404 looks different from the one your program returns.

So when fixing it, I left a comment in the code and wrote a section in the README:

// Not "/healthz": Cloud Run's frontend reserves that exact path and
// answers it with its own 404, so the request never reaches us.
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {

Without this line, the next person who sees /health and thinks "this isn't the convention, it should be healthz" (very likely myself) will change it back.

Pitfall 6: I thought I added context to all external calls, but I missed an entire path

While fixing /healthz, I scanned the code again and found something even more embarrassing.

One change I was very satisfied with earlier was "using context throughout the process, with timeouts for all external calls." Firestore had it, Drive had it. Then I grepped all external calls:

webhook.go:312 blob.GetMessageContent(messageID)
line.go:73 bot.ReplyMessage(...)
line.go:94 bot.PushMessage(...)
line.go:118 bot.LinkRichMenuIdToUser(...)

Four LINE calls, none of which took a context. I had to dig into the SDK to find out why:

c := &MessagingApiAPI{
    channelToken: channelToken,
    httpClient: http.DefaultClient, // ← Timeout is zero, meaning no timeout
}

And the method signatures generated by the SDK don't accept context, so the timeout I wrapped in the outer handler had absolutely no effect on these four calls. If the LINE side hangs, the goroutine just hangs indefinitely.

Reason and Solution: The problem wasn't that I didn't know to add a timeout, but that the memory of "I've added context" overrode the fact of "whether this SDK actually accepts context." When modifying Drive and Firestore, I added .Context(ctx) all the way down very smoothly—so smoothly that I didn't stop to think about which other external calls didn't look like that.

The SDK provides injection points:

bot, err = messaging_api.NewMessagingApiAPI(accessToken,
    messaging_api.WithHTTPClient(&http.Client{Timeout: lineAPITimeout})) // 10 seconds

blob, err = messaging_api.NewMessagingApiBlobAPI(accessToken,
    messaging_api.WithBlobHTTPClient(&http.Client{Timeout: lineBlobTimeout})) // 5 minutes

I gave the blob side 5 minutes because it needs to download videos sent by users.

Along with a more hidden trap. The SDK provides something that looks exactly like what I wanted:

func (call *MessagingApiAPI) WithContext(ctx context.Context) *MessagingApiAPI {
    call.ctx = ctx
    return call
}

It directly overwrites the field of a shared structure and then returns the same pointer. My bot is a package-level shared variable; if multiple requests come in simultaneously and each calls WithContext, it's a standard data race. The name sounds like a functional option, but the behavior is mutation. I also left a comment for this line in the code to prevent someone from changing it later thinking it's more precise than WithHTTPClient.

Pitfall 7: Bugs that don't make a sound need to be actively hit by tests

There was another one found in the same round. The uploadParents function is responsible for listing all YYYY-MM month folders, originally written like this:

r, err := srv.Files.List().Q(query).Fields("files(id)").Context(ctx).Do()

No PageSize set. The Drive API defaults to 100 items per page; anything beyond that requires asking again with a nextPageToken. One folder is generated per month, so after 100 months—about 8 years and 4 months—the oldest folders would disappear from the scope of searches and /recent_files. There would be no error, no warning, just fewer results.

Reason and Solution: Changed to use Pages() to iterate through all pages. What I really want to talk about is the next step. I didn't quite trust the pagination logic I just wrote, so I wrote a mock server that returns a nextPageToken and then temporarily changed the implementation back to only fetch the first page to see if the test would fail:

--- FAIL: TestUploadParentsPagesThroughAllSubfolders
    uploadParents() = [root_id month_1 month_2], want [root_id month_1 month_2 month_3]

Confirmed it would fail before restoring the implementation. This step took less than two minutes, but without it, I would only have a "ran and passed" test without knowing if it was actually testing anything. Bugs like silent truncation don't reveal themselves; if the test is also a green illusion, you have nothing.

Results and Benefits

First, the most direct numbers. Originally, main.go was 1039 lines and main() was 564 lines. After splitting into six files:

File	Lines	Responsibility
`main.go`	94	Startup, environment variable checks, routing
`config.go`	74	Constants and shared state
`webhook.go`	344	Event dispatching, command parsing, command handling
`line.go`	164	LINE message assembly
`drive.go`	183	Drive query/upload
`auth.go`	264	OAuth, token, revocation

main() went from 564 lines to 62 lines. Interestingly, the total lines of main code barely changed (1039 → 1123); what significantly increased was the tests: from 88 lines to 468 lines, and the number of tests from 1 to 16, all passing under -race.

In terms of performance, the search function was originally "find the root folder, then check each monthly subfolder one by one," which is 1 + N Drive API calls; after changing to concatenate all parents with or into a single query, it's fixed at 2 calls.

The value of go-modern-guidelines isn't that it taught me syntax I'd never seen. I generally knew about cmp.Or, min, and slices.SortFunc; the problem is that I don't actively think of them while coding—especially when modifying an existing file, where the surrounding old syntax creates a kind of gravity, making it natural to continue writing in the same style. A sentence in the skill documentation hits the mark:

If a guideline applies, follow it even when nearby code or repository convention uses an older pattern.

This sentence is fighting the frequency bias mentioned earlier, and it applies to humans too.

Its boundaries are also very clear. Those five bugs that actually bite users—the wrong-level switch, the panicking type assertion, the query returning folders, the unescaped query, the misjudged /quit—none of them were caught by go-modern-guidelines; that's not its defensive scope. It manages whether "this Go code is contemporary enough," not whether "this logic is correct." Treat it as a supplement to a linter, not a replacement for code review.

The first half of this article was written before deployment. The /healthz pitfall was discovered after the article was finished and the PR merged, when I checked the build status on gcloud. The situation was: all 45 guidelines checked, everything applicable applied, 17 tests passed under -race, all three CI checks green, Cloud Build SUCCESS, and Cloud Run showing Ready with 100% traffic switched. In this entire row of green lights, not a single one told you an endpoint was dead.

I wrote in the previous post about handling Cloudflare that "a successful build cannot be taken as verification," and I thought I remembered it, but I still paid tuition in the same place this time, just on a different layer—last time the build succeeded but the container crashed on startup; this time the container ran fine but was eaten by the outer infrastructure layer. Tools manage syntax, tests manage logic, CI manages whether these two have regressed, but none of them manage "what happens when this thing is deployed to that specific environment." That part you have to poke yourself.

Keep in mind that the output changes with the project state. The phenomenon in Pitfall 3 where "four more suggestions appeared after upgrading go.mod" was the most practical realization this time. It's not a static document to be checked once, but a query interface that answers based on the project's current state. When the nature of the changes shifts (from main program to tests, language version upgrade, project switch), it's worth running again.

Finally, this change is in PR #4, the two fixes added after deployment are in #5 and #6, and the code is at kkdai/linebot-file. The source code for go-modern-guidelines is at JetBrains/go-modern-guidelines, licensed under Apache 2.0.

[LINE Bot Debugging] A 403 Error and Three Underlying Issues: npm Version Drift, Node.js Requirements, and Cloudflare Verificati

Evan Lin — Thu, 27 Aug 2026 02:46:14 +0000

Previously

"URL retrieval failed, help me check the logs."

I receive this kind of report about once every week or two. Usually, it's because a website has added another layer of anti-crawling protection. A quick check and adding a fallback method to bypass it usually settles it. This time started the same way: a user sent an acm.org URL, and the summary feature spat out an error.

Upon investigation, I realized it wasn't that simple. The URL retrieval feature had been silently broken for three days. The problem wasn't the target website; it was an unlocked npm package version in my own Docker image. One day, it quietly upgraded to a new version that required a newer Node.js to run, while the Node.js in the container was still stuck on the version installed six months ago.

First, let's look at the logs:

gcloud logging read 'resource.type="cloud_run_revision" AND \
  resource.labels.service_name="linebot-helper-python" AND \
  textPayload:"acm.org"' --limit=50 --freshness=2d \
  --format="table(timestamp,severity,textPayload)"


ERROR:loader.url:All methods failed for URL: https://www.acm.org/articles/people-of-acm/2026/russ-cox
WARNING:loader.url:cloudscraper failed ...: 403 Client Error: Forbidden for url: ...
WARNING:loader.url:httpx failed ...: Client error '403 Forbidden' for url: ...
WARNING:loader.url:singlefile failed ...: SingleFile exited with code 1
ERROR:loader.singlefile:SingleFile loading failed ...: SingleFile exited with code 1

This Bot has a fallback chain for grabbing webpage content: first try singlefile (full rendering with headless Chromium), if that fails switch to httpx, and if that fails switch to cloudscraper (a tool specifically for bypassing Cloudflare). It only reports an error to the user if all three fail. It's understandable for httpx and cloudscraper to get a 403; acm.org has protection. But singlefile shouldn't fail; it's designed as the last line of defense to bypass such protections.

Scrolling down, I found the actual error for singlefile:

Error: WebSocket is not available, Node.js 22.4.0 or later is required
    at file:///usr/local/lib/node_modules/single-file-cli/lib/deno-polyfill.js:132:8
Node.js v18.20.4

The container was running Node 18, but single-file-cli required 22.4 or later. Checking the logs, this error first appeared three days ago, not today. This means for the past three days, if any of the first two methods were blocked for a URL, the whole process would fail. Users would only receive a message saying "Unable to read content from the URL," with no indication of the underlying cause.

Why Run Cloud Build Three Times Instead of Deploying Immediately After Changes

The rest of this article will feature three cycles of "modify Dockerfile → build with Cloud Build → run and see." Initially, I didn't plan on doing this; I thought changing a version number and seeing the build pass would be enough to deploy. I later found this assumption was wrong every single time.

The reason is that for commands like npm install -g single-file-cli, a successful build only means "npm found the package and could install it," not "the package works when it runs." npm's check for the engines field is just a warning by default and won't fail the build. Whether the package actually crashes won't be known until it tries to launch a headless Chromium and establish a WebSocket connection. There's a gap in between that the build log doesn't show at all.

So this time, I didn't stop at "build passed." Every time I changed a version, I ran a shell script directly in Cloud Build to use the installed single-file in the container to actually fetch the acm.org URL that was failing. This habit later saved me twice: if I had deployed just because the build succeeded, the Dockerfile changes would have eventually re-enacted the exact same crash in the production environment, just with a different error message.

System Architecture

After the fix, the URL retrieval failure handling looks like this:

graph TD
    A[User sends URL] --> B[loader/url.py fallback chain]
    B --> C1[singlefile: headless Chromium]
    C1 -->|Content fetched| D{is_challenge_page check}
    D -->|Is Cloudflare challenge page| E[Treat as failure, move to next method]
    D -->|Is real content| F[Return to Gemini]
    C1 -->|Crash or timeout| E
    E --> C2[httpx]
    C2 -->|403 or challenge page| C3[cloudscraper]
    C2 -->|Real content| F
    C3 -->|403 or challenge page| G[All three failed]
    C3 -->|Real content| F
    G --> H[Throw Chinese error message to user]

Originally, only the "crash or timeout" path was considered a failure; the is_challenge_page branch was newly added this time, and I'll explain why later.

Core Implementation

1. Dockerfile: Locking Versions for Both Node.js and single-file-cli

Before the fix, this RUN command looked like this:

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        nodejs \
        npm \
        git \
        chromium \
        ffmpeg \
    && npm install -g single-file-cli \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

Both problems were in these lines. apt-get install nodejs pulls the version from the built-in Debian apt source, which is stuck on v18. npm install -g single-file-cli had no version lock, so every image rebuild would pull the latest version currently on npm. Separately, these aren't severe, but together they are a ticking time bomb: I have no control over when single-file-cli increases its Node.js requirement, while my container version was fixed.

After the fix:

# Lock both Node.js and single-file-cli versions to avoid silently pulling incompatible new versions during future rebuilds:
# - Node.js: Switched to NodeSource 24.x (Debian's built-in v18 is too old), locking the exact version.
# The ws/simple-cdp dependency of single-file-cli requires a global CloseEvent, which is only available in Node 24 
# (not in Node 22, even with --experimental-websocket, as verified by testing). This was the root cause of the acm.org scraping failure.
# - single-file-cli: Locked at 2.0.83, verified to work correctly under Node 24.
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        gnupg \
        git \
        chromium \
        ffmpeg \
    && mkdir -p /etc/apt/keyrings \
    && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
    && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
    && apt-get update && apt-get install -y --no-install-recommends nodejs=24.19.0-1nodesource1 \
    && npm install -g single-file-cli@2.0.83 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

The exact version numbers nodejs=24.19.0-1nodesource1 and single-file-cli@2.0.83 weren't chosen at random; they were determined through three rounds of Cloud Build testing. I'll explain how I arrived at these numbers in the "Major Pitfalls" section below.

2. Detecting Cloudflare Challenge Pages

After singlefile stopped crashing, I thought it was over until I actually used it to fetch acm.org:

== single-file fetch acm.org (headless chromium) ==
OK: 18400 bytes
<title>Attention Required! | Cloudflare</title>

It "succeeded," but what it fetched was the Cloudflare human verification page, not the article itself. The problem was that this feature had no detection for this situation. While httpx and cloudscraper would call raise_for_status() on 4xx/5xx and fail immediately, the page rendered by headless Chromium is always an HTTP 200 (the browser displays whatever it gets). This layer of check didn't exist for singlefile. The verification page text would be sent to Gemini, and the user would receive a nonsensical summary completely unrelated to the original article, appearing as a normal response without any error prompts.

I added the detection logic in loader/html.py:

# Common characteristic strings for Cloudflare/WAF human verification pages. 
# singlefile uses headless Chromium to render pages, and unlike httpx/cloudscraper, 
# it doesn't call raise_for_status() for 4xx/5xx. It treats fetching a challenge page 
# as a "successful fetch"—this specifically intercepts that case. If detected, it must 
# be treated as a failure so the fallback chain in loader/url.py moves to the next method, 
# preventing challenge page content from being sent to Gemini.
CHALLENGE_PAGE_MARKERS = (
    "Attention Required! | Cloudflare",
    "Just a moment...",
    "Checking your browser before accessing",
    "cf-browser-verification",
    "Enable JavaScript and cookies to continue",
    "DDoS protection by Cloudflare",
)

def is_challenge_page(text: str) -> bool:
    """Check if the fetched content is a Cloudflare/WAF challenge page rather than real content."""
    return any(marker in text for marker in CHALLENGE_PAGE_MARKERS)

All three loaders (httpx, cloudscraper, singlefile) now use this check. If a challenge page is caught, it's treated as a failure, allowing the fallback chain to try the next method:

if is_challenge_page(resp.text):
    raise RuntimeError(f"httpx got a bot-challenge page instead of real content: {url}")
return parse_html(resp.text, markdown=markdown)

Major Pitfalls and Solutions

Pitfall 1: Don't Take Version Numbers in Error Messages at Face Value

Initially, seeing Node.js 22.4.0 or later is required, I naturally upgraded the container to Node 22, and the build passed smoothly. I thought I was done until I actually ran single-file to fetch acm.org and got a completely different error:

ReferenceError: CloseEvent is not defined
    at #onClose (.../single-file-cli/node_modules/simple-cdp/mod.js:334:36)
Node.js v22.23.2

The ws package that single-file-cli depends on tries to new CloseEvent(...) when handling connection close events, which requires a global CloseEvent class in the execution environment. I ran a probe directly in the container:

node -e "console.log('WebSocket', typeof WebSocket, 'CloseEvent', typeof CloseEvent)"
# WebSocket function CloseEvent undefined

node --experimental-websocket -e "console.log('CloseEvent', typeof CloseEvent)"
# CloseEvent undefined

Node 22.23.2 has a global WebSocket (which is what the "22.4.0 or later" error message was actually checking), but it doesn't have CloseEvent, even with the experimental flag. Testing again with Node 24.19.0:

node -e "console.log('CloseEvent', typeof CloseEvent)"
# CloseEvent function

Cause and Solution: The internal version check in single-file-cli (the "Node 22.4 or later" in deno-polyfill.js) only checks the first API it uses; it doesn't mean all APIs used by its dependencies are present in that version. The real blocker was the downstream CloseEvent, which only became globally available in Node 24. Treat version numbers in error messages as a starting point, not the final answer. You only know where it will actually get stuck by running it.

Pitfall 2: The "Latest Version" on npm is a Moving Target

After upgrading to Node 22, I thought it was over until a user asked to "lock the versions while you're at it to avoid future issues." Checking the current single-file-cli info on npm:

npm view single-file-cli engines
# { deno: '>=2.2', bun: '>=1.2', node: '>=24.0.0' }

The latest version requires Node 24 or higher, not the 22 I had just fixed. Checking the release times, I realized there were two consecutive version jumps within the same week:

npm view single-file-cli time --json | grep -E "2\.0\.83|2\.1\.0"
# "2.0.83": "2025-11-29T23:34:16.671Z"
# "2.1.0": "2026-08-16T15:56:28.819Z"

Version 2.0.83 was from last November, with engines set to node >= 20, and it worked stably for over six months. Version 2.1.0, released on August 16th, suddenly raised the bar to node >= 24. I first observed the crash on August 20th, which aligned perfectly with an image rebuild: npm install -g single-file-cli without a specified version had silently pulled the 2.1.0 version released just days prior.

Cause and Solution: npm install -g <package> (without a version) in CI/CD or a Dockerfile essentially means "every rebuild might install something different." This becomes a problem the moment a package maintainer raises the Node version requirement. Locking versions isn't about being conservative; it's about changing "when this container breaks" from "whenever the package author feels like it" to "whenever I decide to upgrade." I eventually chose 2.0.83—not the latest, but the last stable version explicitly compatible with Node 20—rather than trying to meet the new Node 24 requirement of 2.1.3.

Pitfall 3: Build Success Does Not Equal Runtime Success

I mentioned this earlier in "Why Run Cloud Build Three Times," but here is how it actually looked. After locking Node 24 + single-file-cli 2.0.83, gcloud builds submit finished successfully, and Successfully built was printed. If I had only looked at the build success message, I would have mistakenly thought the problem was solved. But the version that first passed the build was actually the Node 22 version, which would have exploded at runtime due to the CloseEvent issue in Pitfall 1. Build success itself does not prove the content of the container actually works.

I added an extra step to the Cloud Build configuration to run the newly built image immediately:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'nodefix-verify', '.']
- name: 'nodefix-verify'
  entrypoint: 'bash'
  args:
    - '-c'
    - |
      node -v
      single-file --version
      single-file --browser-executable-path=/usr/bin/chromium \
        --browser-args='["--no-sandbox","--disable-dev-shm-usage"]' \
        "https://www.acm.org/articles/people-of-acm/2026/russ-cox" /tmp/out.html \
        && echo "OK: $(wc -c < /tmp/out.html) bytes"

The first time I ran this, it caught the CloseEvent is not defined error from Pitfall 1. Only on the second run (Node 24 + 2.0.83) did I finally see OK: 18400 bytes.

Cause and Solution: npm install only prints a warning for engines mismatches by default; it doesn't fail the command. docker build success only means the exit code of every command was 0. Together, this means there is no necessary correlation between "build passed" and "this thing works correctly." Including "actually run the target scenario" in the verification process is the only way to bridge this gap.

Pitfall 4: Fetching Content Doesn't Mean Fetching the Right Content

When the verification in Pitfall 3 printed OK: 18400 bytes, I initially thought the problem was solved until I saw that within those 18,400 bytes, the <title> said Attention Required! | Cloudflare. I realized that even if single-file-cli works perfectly and renders the page with headless Chromium, Cloudflare might still block the request because it detects a headless browser, returning a challenge page. To single-file-cli, this challenge page looks exactly like a normal article: it's HTML, it can be saved, and it counts as a "success."

After adding the is_challenge_page check, the first round of tests didn't all pass:

FAILED test_load_html_with_httpx_raises_on_challenge_page
FAILED test_load_html_with_cloudscraper_raises_on_challenge_page

| The logic seemed correct, yet the httpx/cloudscraper tests failed to catch the challenge page. Upon investigation, I found that load_html_with_httpx in loader/html.py converts HTML to markdown (using markdownify) before returning it. markdownify does not preserve tags like <title> which are in the <head> and not displayed as body text. My detection string "Attention Required! | Cloudflare" happened to be in that discarded <title>, so it wasn't in the text after the markdown conversion. Even worse was cf-browser-verification, which in a real Cloudflare page is a CSS ID (<div id="cf-browser-verification">). It only exists in HTML attributes, so neither markdown conversion nor plain text extraction (get_text()) would include the attribute value as text content. |

Cause and Solution: Move the checkpoint from "processed text" to "raw HTML":

# Some challenge page features (like cf-browser-verification) only appear in HTML attributes, 
# which markdownify/get_text won't catch. So, check the raw HTML before conversion.
if is_challenge_page(resp.text):
    raise RuntimeError(f"httpx got a bot-challenge page instead of real content: {url}")
return parse_html(resp.text, markdown=markdown)

I made the same adjustment for singlefile. Originally, it used BeautifulSoup to convert the file to plain text before checking. I changed it to read the raw bytes and check them first; only if it passes is it handed over to BeautifulSoup:

with open(f, "rb") as fp:
    raw_html = fp.read()

if is_challenge_page(raw_html.decode("utf-8", errors="ignore")):
    raise RuntimeError(
        f"SingleFile got a bot-challenge page instead of real content: {url}")

soup = BeautifulSoup(raw_html, "html.parser")
text = soup.get_text(strip=True)

The lesson here is: when writing protection logic, you must be clear about which layer of data you are checking. The same content, after different conversions (HTML → markdown, HTML → plain text), will lose information asymmetrically: some words are visible, while others only exist in attributes, and both extraction methods might miss them.

Results and Benefits

After the changes, I ran a real end-to-end test, calling the load_url() function actually used in Python within the container against the same acm.org URL:

INFO:loader.url:Trying singlefile for URL: ...
ERROR:loader.singlefile:SingleFile loading failed ...: SingleFile got a bot-challenge page instead of real content: ...
WARNING:loader.url:singlefile failed ...
INFO:loader.url:Trying httpx for URL: ...
WARNING:loader.url:httpx failed ...: 403 Forbidden
INFO:loader.url:Trying cloudscraper for URL: ...
WARNING:loader.url:cloudscraper failed ...: 403 Forbidden
ERROR:loader.url:All methods failed for URL: ...
Raised as expected: Exception 無法從網址讀取內容，請確認網址是否正確或稍後再試 (Unable to read content from the URL, please verify the URL or try again later)

All three methods still failed; the Cloudflare protection behind acm.org was not bypassed. That hasn't changed. What changed is how it failed: previously, the singlefile crash dragged down the entire fallback chain, leaving the user unaware that three methods had even been attempted. Now, all three methods fail cleanly on their own, and the user receives a clear error message instead of a nonsensical summary that is actually a Cloudflare challenge page.

Locking versions solves "will it break again in the future," not "can the acm.org URL be fetched." It's easy to conflate these two at first: once versions are locked and single-file-cli no longer crashes, it's tempting to think the problem is solved. But the Cloudflare hurdle was never the scope of this fix, nor is there a simple solution (bypassing it would likely require browser fingerprint spoofing, which has a poor ROI). What was achieved was the complete separation of "crawler blocked" and "my own container is broken." The latter should no longer recur due to npm version drift.

"Build success" cannot be treated as verification passed. This was the biggest takeaway. If I had only looked for red text in the build log after every Dockerfile change, I would have missed at least two issues: the CloseEvent crash with Node 22, and the fact that Cloudflare challenge pages were being sent as body text even when singlefile worked perfectly. Both are "build okay, crash at runtime" types of problems that only become visible when running against a target URL.

Protection logic must align with the actual data layer received. The issue in Pitfall 4 where markdownify swallowed the <title> would have likely gone unnoticed until a user reported the exact same problem again, had I not written a test and seen it DID NOT RAISE.

In total, four files were modified: Dockerfile, loader/html.py, loader/singlefile.py, plus a new test tests/test_challenge_page_detection.py. All 150 tests in the project passed. The code is available at kkdai/linebot-helper-python.

[Gemini API in Action] Adding a "Detailed Research Report" Button to a LINE Bot: Using Google Search Grounding to Turn Summaries

Evan Lin — Thu, 20 Aug 2026 14:00:07 +0000

Background

My LINE Bot has always had a summary feature: you drop a URL in, it crawls the content, generates a summary, and attaches a social media post draft along with a button to save it as a bookmark. This feature has been around since 2024, but it has always only solved the "what is this about" problem. I often find myself wanting to know three other things:

What is the background context of the things discussed in this article? Have there been counter-arguments from others? Are the numbers mentioned sourced, or are they just the author's own claims?

Summaries can't answer these because the input for a summary is only the article itself. The model has no other materials; if you ask it for a "critical analysis," it can only circle around the original text or start hallucinating.

Google Search Grounding fills exactly this gap. I used it as a search assistant in a previous article; back then, the purpose was to answer questions. This time, I wanted to try another approach: give an existing article to the model, let it search for information outside the article on its own, and then look back to review the article.

The result is a new "📄 Detailed Research Report" button on the summary card. About one to two minutes after clicking it, the Bot pushes a web link to you.

Main Repo: https://github.com/kkdai/linebot-helper-python

Why Grounding instead of building your own search pipeline

Before Grounding, to let a model read real-time information from the web, you had to build a pipeline yourself: first, ask the model to extract keywords from the article, use those keywords to call a search API, crawl the search result pages one by one, stuff them into the prompt, and then ask the model to summarize. This involves three or more API calls, any of which could fail, and the quality of the extracted keywords directly determines whether the retrieved information is useful.

Grounding integrates this entire process into the model. You simply attach a google_search tool in the GenerateContentConfig, and the model handles the rest: it decides whether to search, what to search for, how many times to search, and judges which results are worth using.

For the "Research Report" topic, the model deciding what to search for is particularly valuable. When writing the prompt, I don't know what article the user will provide, so I naturally can't write the specific keywords to search. But after the model reads the article, it knows; it will look for the context of the topic and check if there are opposing views.

Another advantage I care about is that citations are returned. The grounding_metadata in the model's response contains the actual web pages it referenced, including titles and URLs. This means phrases like "according to other reports" in the report aren't just the model speaking from memory; there are corresponding web pages you can click to verify. For information-based products, this makes a huge difference.

The code to extract sources is in loader/langtools.py, written defensively because these fields don't exist at all if no search was triggered:

def _extract_grounding_sources(response) -> list:
    """Extract citations from grounding metadata (same approach as chat_session)."""
    sources = []
    try:
        if getattr(response, 'candidates', None):
            candidate = response.candidates[0]
            metadata = getattr(candidate, 'grounding_metadata', None)
            chunks = getattr(metadata, 'grounding_chunks', None) if metadata else None
            for chunk in chunks or []:
                web = getattr(chunk, 'web', None)
                if web:
                    sources.append({
                        'title': getattr(web, 'title', '') or '',
                        'uri': getattr(web, 'uri', '') or '',
                    })
    except Exception as e:
        logging.warning(f"Failed to extract grounding sources: {e}")
    return sources

System Architecture

The entire flow starts from the button on the summary card, goes through a re-crawl and a grounding call, and ends with a temporary webpage.

graph TD
    A[User sends URL] -->|Summary Flex Bubble| B[📄 Detailed Research Report Button]
    B -->|Postback with bookmark doc id| C[Verify bookmark ownership]
    C -->|Immediate Reply: Researching| D[LINE Chatroom]
    C -->|Background Task| E[load_url: Re-crawl original text]
    E --> F[Gemini + Google Search Grounding]
    F -->|Markdown + Citations| G[render_report_page to HTML]
    G -->|Store in memory ReportStore| H[Get uuid report_id]
    H -->|Push Link| I[GET /reports/:id Temporary Webpage]

The button carries the bookmark's document ID, not the URL itself. This follows the existing "Save Bookmark" mechanism. The benefit is that using the doc ID allows verifying that the bookmark actually belongs to the user before generating the report. Conversely, if Firestore isn't connected or the doc ID can't be retrieved, this button won't appear.

Core Implementation

1. Generating Research Reports with Grounding

The key to generate_research_report() isn't the code, but the prompt. I explicitly ask the model to search proactively and require it to label which information comes from the search and which comes from the original text:

    prompt = f"""You are a rigorous research analyst. Please write a detailed research report based on the following article content,
in Traditional Chinese (Taiwan usage), Markdown format (starting from ## level, do not include the main article title).

Required Structure:
## Executive Summary (3-5 sentences explaining what this is about and why it matters)
## Background Context (The history and context of this topic, combined with relevant information you searched for)
## Core Arguments & Evidence (Organize the article's claims and supporting evidence point by point, labeling the strength of evidence)
## Data & Fact Summary (Key numbers, dates, people, and organizations from the text, using tables or lists)
## Counter-perspectives & Critique (Search for related reports, compare other viewpoints; point out blind spots, assumptions, or controversies in the article)
## Further Questions (3-5 questions worth investigating further)

Requirements:
- Please proactively search for supplementary background and comparative information outside the article, and label in the text whether the information comes from search or the original text.
- Be specific rather than abstract; clearly label unsupported inferences as "speculation".
- Use full-width punctuation, avoid AI-sounding clichés.

Original URL: {url}

Article Content:
{text}"""

The phrases "label the strength of evidence" and "clearly label unsupported inferences as speculation" are the parts of the prompt I care about most. Without them, every sentence in the report would sound equally confident, and the reader wouldn't be able to distinguish what the article said, what the model added from search results, and what it inferred itself.

The part for attaching the tool is very short; whether tools is provided or not is the difference between having grounding or not:

    def _call(with_grounding: bool):
        client = _get_vertex_client()
        tools = [types.Tool(google_search=types.GoogleSearch())] if with_grounding else None
        return client.models.generate_content(
            model="gemini-3.1-flash-lite",
            contents=prompt,
            config=types.GenerateContentConfig(
                temperature=0.4,
                tools=tools,
                max_output_tokens=16384,
                labels={"client_id": "info_helper"},
            )
        )

    try:
        try:
            response = _call(with_grounding=True)
        except Exception as e:
            logging.warning(
                f"Grounded research call failed, retrying without tools: {e}")
            response = _call(with_grounding=False)

I used two layers of try because grounding involves external searches, so the failure rate is naturally higher than pure text generation. When the tool call fails, instead of returning "Generation failed," it's better to retry once with the same prompt but without the tool. In this case, the user gets a pure article analysis without comparative views or sources, but at least they have something. This degradation is intentional, not an accident.

2. Where to put the report

The report is a full Markdown document, often thousands of words long, which can't fit into a LINE message. Making it a Flex Message isn't suitable either because it contains tables and multi-level headings. So, I turned it into a webpage.

But then I had to decide: should these reports be stored in a database?

I chose not to. The reports are only stored in memory and disappear as soon as the Cloud Run instance is recycled:

class ReportStore:
    def __init__ (self, ttl_seconds: float = DEFAULT_REPORT_TTL_SECONDS):
        self.ttl = ttl_seconds
        self._reports: Dict[str, dict] = {}
        self._lock = Lock()

    def put(self, html: str) -> str:
        report_id = uuid.uuid4().hex
        with self._lock:
            self._purge_expired()
            self._reports[report_id] = {
                "html": html,
                "created_at": time.time(),
            }
        return report_id

The report_id uses uuid.uuid4().hex because this URL has no login protection; anyone with the link can open it, so the ID must be unguessable. The page itself also includes <meta name="robots" content="noindex"> to prevent search engines from indexing people's reading history.

3. Expired Pages

Since reports disappear, "link expiration" is not an exception but the normal end for every report. So the route is written like this:

@app.get("/reports/{report_id}")
def serve_research_report(report_id: str):
    """Temporary research report page: returns expired page (404) after expiration or instance restart."""
    html = report_store.get(report_id)
    if html:
        return HTMLResponse(html)
    return HTMLResponse(render_expired_page(), status_code=404)

I also made it clear in the message pushed to the user, not pretending it's a permanent link:

⏳ This is a temporary page, kept for about 24 hours (invalidated after the service sleeps). Please copy the content if you need to save it.

Major Pitfalls and Solutions

Pitfall 1: Grounding tools and response_schema cannot be used together

I encountered this pitfall earlier when building a map restaurant search. At that time, I naturally thought: since I want to get a restaurant list from the model, I'll use structured output, attach response_mime_type="application/json" and response_schema, and get correctly typed data directly to save myself from parsing it.

The result was an immediate API error. The fix then was to remove the schema (commit a2c8745) and instead ask the model to output JSON text, which I then parsed myself.

Reason and Solution

Google Search Grounding and structured output are mutually exclusive. When the model is performing grounding, it needs to freely intersperse searching, thinking, and citing; this process cannot be simultaneously constrained to a fixed JSON schema.

So when making the research report, I abandoned the idea of "returning a structured object" from the start and let the model output Markdown plain text directly, writing the structure into the "Required Structure" section of the prompt instead of the schema.

Looking back, this limitation actually made things simpler. The report is meant to be a long-form text for humans to read; Markdown is its most natural form. If forced into JSON fields, it would just have to be stitched back into an article during rendering anyway. The only fields that truly need a strict structure are the citations, and those can be taken from grounding_metadata, which never needed a schema to begin with.

Pitfall 2: Three-second timeout, plus Gemini is synchronous and blocking

LINE Webhook requires an HTTP 200 response within three seconds, but this feature needs to re-crawl the original text and then wait for the grounding call to finish, which takes one to two minutes in total.

Let's talk about the part fewer people notice. client.models.generate_content() is a synchronous blocking call. Even if you wrap it in an async def, it will still block the entire event loop. While one user is generating a report for those two minutes, messages from other users will also be blocked.

Reason and Solution

Split it into two parts: reply and push, each with its own responsibility:

    url = doc.get("url", "")
    await line_bot_api.reply_message(
        event.reply_token,
        [TextSendMessage(text="🔬 Starting in-depth research on this article (approx. 1-2 mins). I'll send you the report link once finished.")])

    try:
        crawled_text = await load_url(url)
        # Gemini call is synchronous and blocking; offload to a thread to avoid blocking other tasks on the event loop
        result = await asyncio.to_thread(generate_research_report, crawled_text, url)

First, use reply_message to say "Starting research," finishing the webhook request within three seconds. The heavy lifting is moved to the background, and the synchronous Gemini call is offloaded to a thread using asyncio.to_thread. Once finished, use push_message to proactively send the link.

The phrase "approx. 1-2 mins" is also intentional. If a user clicks a button and nothing happens for thirty seconds, they'll start to suspect it's broken and click it again. Explaining how long to wait upfront is cheaper than explaining it afterward.

Pitfall 3: Who can see whose report

The button carries the bookmark's doc ID. If I simply used this ID to query data and generate a report, anyone who could construct a postback could read the content of bookmarks saved by others.

Reason and Solution

Verify ownership during the query. Both parameters for get_bookmark(user_id, doc_id) are required. If not found, treat it as expired without telling the user "this exists but doesn't belong to you":

    doc = svc.get_bookmark(user_id, doc_id) if (doc_id and svc.available) else None
    if not doc:
        await line_bot_api.reply_message(
            event.reply_token,
            [TextSendMessage(text="⚠️ Data has expired. Please send the URL again and try once more.")])
        return

Results and Benefits

In practice, this button changes more than just "making the summary longer."

Summaries and research reports answer different questions. A summary tells you what the piece is about, suitable for quickly deciding whether to read it. A research report tells you if the piece is correct, how others view it, and which numbers are sourced. That's why I made it two layers instead of making the summary longer: the cheap layer runs every time, and the expensive layer is there for when you really want to dive deep.

Citations make the report verifiable. At the bottom of the report is a "📚 References" list, all from grounding_metadata, which are the actual web pages the model read. If you see something in the "Counter-perspectives & Critique" section that differs from the original text, you can click directly to the original report. This is why I think grounding is more worthwhile than building your own search API pipeline: what you save isn't just code, but the traceability of "where did this sentence come from."

There's still something to see even if search fails. After degrading to pure article analysis, the report will lack background context and comparative views, but the executive summary, core arguments, and data organization sections will still be there.

Temporary webpages save more than expected. No need to set up a database, no need to write cleanup schedules, and no need to design a report list page. A dict plus a lock is all it takes, with security maintained by unguessable UUIDs. The trade-off is that links will expire, which I've made clear in the push message. Reading behavior is usually concentrated in the few minutes after receiving a link; I don't think it's worth the overhead of a full persistence system for the few cases where someone wants to save it long-term.

If long-term storage is really needed later, the current architecture isn't hard to change: the ReportStore interface only has put and get methods. Replacing it with a Firestore implementation wouldn't require any changes to the upper layers.

The entire feature adds up to about five hundred lines, a third of which are tests. The code is at kkdai/linebot-helper-python, and the design document is in docs/superpowers/specs/2026-08-15-research-report-design.md. Feel free to check it out if you're interested.

DEV Community: Evan Lin

[AI in Action] Gemini 3.8 Live Upgrade: Paying Off Technical Debt and How My Test Script Fooled Me Three Times

Recap

The Real Motivation: A Preview Model That Tripped Me Up Twice

Ask the API First, Not the Documentation

Pitfall 1: Warning Against a Non-existent Risk

Pitfall 2: The Control Group Saved Me

Pitfall 3: Sine Waves Aren't Human Voices

Comparison Table

Extended Thinking Has a Required Field

What Changed

Deployment Order: Config First, Code Second

Then I Tested It for Real, and the Results Overturned My Observations

296 Passing Tests Don't Prove Voice Works

[AI in Practice] Shadowing Assessment with Gemini 3.5 Transcribe: Let Song Lingo Listen to You Read Lyrics

Recap

What is Gemini 3.5 Transcribe

Which Model to Choose

See Real Responses Before Coding

output_text is assembled by the SDK again

Word positions are calculated in UTF-8 bytes

How to Determine "Correct Pronunciation"

Use verbatim, not smart

Don't use custom vocabulary

Compare both pronunciation and writing

Distinguish between "Mispronounced" and "Missed"

Clarifying Limitations First

Pitfall 1: Button says "Recording" before the microphone is ready

Pitfall 2: Headless Chrome on Mac can't access the microphone

Privacy and Quota

Side note: Making PWA installable behind IAP

Results and Benefits

Key Takeaways

Hands-on with the official google-cloud-developer plugin: Stop AI agents from hallucinating gcloud commands

What's inside this plugin?

Installation: Three Paths to Choose From

Test 1: Can guardrails stop hallucinated flags?

Test 2: How much context does a blind list consume?

Test 3: Which commands is it forbidden to run?

Test 4: Developer Knowledge—Querying docs instead of memory

Test 5: How 137 skills avoid consuming context

Five Practical Scenarios

Scenario 1: Connecting an agent to production, but only allowing it to look, not touch

Scenario 2: Starting a GCP project from scratch and deploying a LINE Bot to Cloud Run

Scenario 3: Credential Hell—Works locally, but 403 on Cloud Run

Scenario 4: Cloud Resource Inventory and Billing Health Check

Scenario 5: Avoiding the model's old memories when asking about new features

Summary

Reference Links

[AI in Action] Gemini Agentic Video: 4 Hidden Prerequisites and My LINE Bot Integration Process

Background

What is Agentic Video?

What can it actually do?

Prerequisites: Four parameters, missing one won't trigger an error

Pitfall 1: Video context cannot be preserved on Vertex AI

Pitfall 2: Thinking tokens are the main cost driver, and they are unstoppable

Version 1 Conclusion (Wrong)

Version 2 Conclusion (Also Wrong)

Version 3 (The Reality)

Integrating into the LINE Bot: What the feature looks like

Decision order is the behavioral contract

Use heuristics for exiting mode, not LLM

Development Process: Using subagents to catch my mistakes

They caught three of my incorrect causal inferences

It caught a Critical bug that missed eight rounds of review

Mistakes I made myself

Results and Benefits

A few things I'm taking away

[Steam Deck Guide] Fix Windows .exe Patch Path Errors: The Ultra-Stable "Relay Transfer Method"

Why do patch programs get "lost"?

Core Solution: Three-Step Relay Relocation

Step 1: Copy the game directory to a relay path

Step 2: Run the patch and select the path via the Z: drive

Step 3: Overwrite back to the original path and confirm the startup file

Notes and Troubleshooting Tips

[AI in Practice] Gemini 3.8 Flash TTS Launch: I built a "Learn Japanese with MVs" Web App and burned through my daily quota.

Recap

What is Gemini 3.8 Flash TTS

Things to Handle Before Starting: Lyrics Copyright

Architecture: Three Python Scripts + One Next.js

`output_text` is assembled by the SDK again

Pitfall 3: Quotes in `.env`

Pitfall 3: Ensuring `.env` and lyrics aren't uploaded